Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper aims to ground language to vision via cross-modal contrastive learning to build an explainable shared semantic space. This is achieved in two parts (i) align visual and language representations with MS COCO dataset (ii) Retrieve visual objects with language queries through a cross-modal attention module and infer the visual relations between the retrieved objects through a bilinear operator on Visual Genome dataset. This allows model's language encoder to become a stand-alone language model that can encoder embedding concepts in a visually grounded semantic space. The model enables compositional language understanding based on visual knowledge and multimodal image search with queries. The authors show experiments on clustering, compositional reasoning, relationship with human-defined norms, and image search to showcase the usefulness of the model.
main_review: The paper targets a less-studied problem of grounding language in visual semantic space and uses a two-step process to achieve that. Specifically, for the first step the paper uses a CLIP-style [1] contrastive loss to align visual and language representations using MS COCO dataset. For the second step, which seems useful in helping ground the object relations to generate visually ground object representation for the text, the model uses a cross-attention module similar to VILBERT [2] model, by passing language stream as a query and visual output as a key and value to generate a language conditioned visual representation which is basically visually grounded object representation for the object present in the text. A bilinear relation module is applied to the ground object representation for predicting the relationship between the two objects which is trained using a contrastive learning approach. The idea of aligning using MS COCO dataset uses CLIP-style contrastive loss and is not new while the idea of improving the object semantic grounding using Visual Genome dataset and contrastive loss over relations is new and insightful. Comments on the paper are as follows:

1) The paper doesn't provide a concrete way of evaluating the usefulness of these embeddings on the downstream vision-and-language tasks in a quantitative way. I was expecting to see some experiments similar to CLIP paper on how exactly the better grounding can help achieve a better general model for downstream tasks. The paper does cover a qualitative study on multimodal image search but didn't do extensive experiments on image or text retrieval which this model would have empowered.The supplementary material provides some results on retrieval on MS-COCO dataset but the number seem to very low compared to state-of-the-art model which seems surprising (the appendix only have qualitative figure for comparison and no table). This leaves the reader with incomplete knowledge on the actual practical usefulness of the model.
2) The section 4.3 is complicated and complex to understand in the way it is written. I would suggest that authors to either improving the writing or remove that section for the sake of the clarity of the paper. It took me some time to understand how figure 6 was working and what it was supposed to show after reading section 4.3.
3) The paper does evaluate self-attention on top of VGG16 feature extractor they use for imagenet classification accuracy (in appendix) but never evaluate their final visual stream on imagenet to check how grounding helps.
4) The paper also evaluates on GLUE tasks in appendix but it looks like the performance is decreasing on most tasks which is counter-intuitive as learning about world through images should maintain or improve original BERT model. I would suggest that authors take a look at [3] which aims to learn multimodal and unimodal tasks using a single transformer architecture.
5) The paper doesn't provide any ablations on not using "visual grounding of object relations" stage or only using that stage. These ablations are necessary and insightful to get a sense to how much visual grounding of object relations is helping the model in comparison to just using CLIP-style loss. Ideally, if it helps, it could also help improve the already strong CLIP model as well. Furthermore, the ablation study on only using "visual grounding of object relations" could help understand how much CLIP-style loss helps in overall grounding of the language to vision and vice versa.
6) Similarly, the ablations of Loss_l and Loss_v alone are missing to understand impact of two modalities on contrastive loss. The paper also doesn't provide ablations on how much data helps in how much performance. Ideally, a study could be conducted on using percentage of coco dataset to showcase if more data can help in the grounding process. The paper also doesn't mention any hyperparameter tuning or otherwise.
7) There is a lot going on in the appendix and some of the things there can make it into the paper (such as ablations on visual grounding loss's components relations and object prediction loss) while some the things like I mentioned can be moved to supplementary.

The paper is a good attempt in a less-explored direction but fails to provide reader with useful insights on actual practical help of doing the visual grounding as GLUE results drop, multimodal retrieval results are not impressive among others. Though qualitative results make sense around clustering and probing, the model at large doesn't look useful from the provided information in the manuscript for the downstream tasks.

[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." arXiv preprint arXiv:2103.00020 (2021).

[2] Lu, Jiasen, et al. "Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks." arXiv preprint arXiv:1908.02265 (2019).

[3] Hu, Ronghang, and Amanpreet Singh. "UniT: Multimodal Multitask Learning with a Unified Transformer." arXiv preprint arXiv:2102.10772 (2021).
limitations_and_societal_impact: The paper doesn't have a broader impact statement included.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper analyzes the linguistic embedding in the joint image-language space learned by cross-modal contrastive learning. Comparing with the standard word embedding, the grounded language embedding is human explainable. Specifically, the PCA of the grounded langauge embedding is aligned with human rating. When clustering the language embedding, the cluster of grounded language embedding is more semantically coherent than the un-grounded language embedding. Furthermore, the paper showed that the grounded langauge embedding could benefit visual-linguistic downstream tasks.
main_review: Novelty:

1. Method: The main focus of this paper is to show that the grounded linguistic embedding is more aligned with human perception. Therefore, in terms of the method, the paper uses a pretty standard two stream vision and language model to learn the embedding. The description of the approach is clean and easy to follow.

2. Experiment:the analysis conducted in this paper is pretty interesting. However, this paper is not the first one to conduct similar experiments. 

  2.a In terms of 'Clustering of word representations', ViCo (Gupta et, al., 2019) conducted similar experiments (Sect. 4.1) and showed the grounded embedding is better than the un-grounded. 

  2.b In terms of the Multimodal image search, Ailem et. al, 2018 conducted similar experiments. They showed that the grounded embedding can help the image retrieval model to achieve better performance than the un-grounded.
  
Therefore, the experiments and analysis are very interesting. However, they are not surprising given the previous research.

---
Experiment:

1. Word coverage rate: The experiments in Sect. 4.1 and 4.4 are very interesting. However, I wonder whether all the tokens in those two datasets are covered by MS-COCO? Specifically, MS-COCO is annotated with image and text. The text is descriptive sentence which describe the object/event in the image. In Sect. 4.1 the dataset contains word like 'emotions' and 'mythical_beasts' (Fig. 3). I wonder whether those words are covered by the MS-COCO or Visual Genome?

2. In Sect. 4.1, does the ungrounded word embedding show similar behavior? Specifically, could the PCA on the ungrounded word embedding provide any useful interpretation?

3. Sect. 4.5. I am not quite sure how did the image-text retrieval conduct? I could understand everything till L236. But I don't know hwy a query image and a query text need to be combined. How to use the combined representation to conduct image retrieval.

4. Sect. 4.5. As the MS-COCO is a dataset defined for image retrieval, I wonder why the standard image retrieval task and metrics (Recall) is not used/reported?

---
The rating is based on the novelty and the questions in experiment. I would happy to raise the score if the concerns are resolved.

Gupta T. et. al., ViCo: Word Embeddings from Visual Co-occurrences, ICCV 2019.
Ailem, M. et. al., A Probabilistic Model for Joint Learning of Word Embeddings from Texts and Images. EMNLP 2018. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a contrastive learning framework to learn visually grounded, and as a result, semantically interpretable language representations. It proposes two contrastive losses which make use of multiheaded cross-modal attention and bilinear interaction to compute similarity scores between subject and object representations, which can then we used in the contrastive loss.

They perform thorough experiments with the resulting representation to illustrate how well they correlate to human ratings along different language axes such as concrete-abstract and compositionality, as well as qualitative multimodal image search examples.
main_review: This paper provides a meaningful contribution that naturally follows recent image-text representations learned with a contrastive loss (e.g. CLIP/ALIGN). The writing and structure of the paper is strong and clear, it was easy to read. The authors also clearly motivate the work, as having more semantically meaningful language representations is something very desirable, especially from an ethical and interpretability perspective. 

The overall approach and results are well illustrated, I think they could be reproduced fairly easily (although time consuming if associated code is not released). There are a comprehensive and interesting set of experiments on the language representations, which include analyzing representations with PCA and their relationship to SemCat words, use for predicting binary semantic features, how well they cluster according to SemCat categories, how they capture compositional relationships, and how search results change along an image to text feature continuum.

There are some remaining questions I hope can be answered:
1. In Section 4.1, how many principal components are there?
2. L173-175, Bert is discussed as comparison. What about vision-language transformers or other vision-language language models? It seems like the comparison is a bit incomplete/unfair given that there are vl language representations out there which may have better captured the semantic space than Bert (e.g., ViLBERT, LXMERT, VisualBERT, Oscar, MT-GrOVLE etc)
3. The illustration of the significance levels in Figure 4 are pretty hard to follow - why are there overlapping brackets with different *? Shouldn't each box plot be associated with only one of the significance values?
4. Typo at L225, I believe "shape" should be "sharp"
limitations_and_societal_impact: The authors have not fully addressed limitations of their work. The method, motivation, and qualitative results are important alone, but the goal in having visually grounded and interpretable language representations is to then also use them in multimodal tasks (and replace current representations which are not grounded nor interpretable). The hope is that the representations proposed in this work perform as well as current state of the art or better, but this isn't demonstrated with any downstream experiments. E.g., the multimodal image search section provides a very cool illustration (Figure 7), but then no quantitative experiments on retrieval are provided. Of course the main contribution of this paper concerns interpretability, but connecting the work to real world applications would be an important point to include in the discussion.

As far as societal impact, this paper only provides positive impact, which is great to see!
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper implements a model which learns word representations from
text grounded in images. The model is composed of a BERT-based text
stream, a VGG-based image stream, and visual relation classification
module. The text and image components are pre-trained and fine tuned
on the COCO dataset of captioned images as well as on Visual Genome
relations. The main contribution of the paper is the large number of
analyses carried out on the induced grounded word embeddings, compared
to plain BERT embedddings. The paper shows some evidence that the
principal components of the grounded embedding space correspond to
interpretable semantic dimensions (i.e. concrete-abstract), that
words' semantic features can be predicted based on embeddings. The
paper also claims that the model is capable of compositional
understanding.

main_review: ## Strengths

The main idea of visual grounding  has been of interest in
computational linguistics for a long time, and has been investigated
computationally many times. This paper main contributions are the use
of state-of-the-art neural architectures and pretrained models, and an
in-depth study of the nature of the learned embeddings. The results
range from compelling (the interpretable principal components) to
suggestive but preliminary (compositionality), but overall show
convincingly that in this particular incarnation of grounding does
make a major contribution to make the word representations more
human-like.

## Weaknesses

The discussion of and comparison to earlier related work is very
incomplete. Given the analytical focus of the paper quantitative
comparisons to alternative models are not absolutely necessary but it
would be good to at least see a more in-depth related work
section. There are a lot of papers that could be mentioned here; 
a small sample follows:


- Bordes, P., Zablocki, E., Soulier, L., & Piwowarski, B. (2019,
  November). Incorporating Visual Semantics into Sentence
  Representations within a Grounded Space. In Proceedings of the 2019
  Conference on Empirical Methods in Natural Language Processing and
  the 9th International Joint Conference on Natural Language
  Processing (EMNLP-IJCNLP) (pp. 696-707).

- Kiros, J., Chan, W., & Hinton, G. (2018, July). Illustrative
  language understanding: Large-scale visual grounding with image
  search. In Proceedings of the 56th Annual Meeting of the Association
  for Computational Linguistics (Volume 1: Long Papers) (pp. 922-933).

- Chrupała, G., Kádár, Á., & Alishahi, A. (2015, July). Learning
  language through pictures. In Proceedings of the 53rd Annual Meeting
  of the Association for Computational Linguistics and the 7th
  International Joint Conference on Natural Language Processing
  (Volume 2: Short Papers) (pp. 112-118).

limitations_and_societal_impact: Limitations are not explicitly discussed. The preliminary nature and
small scale of some of the experiments (e.g. section 4.4) should be
acknowledged. The word "explainable" suggests that the architecture is
designed to be explainable, but in fact it doesn't seem to be the case: rather, 
analyses are carried out which provide some insight into the learned 
representations.
review_title: null

Meta-review: This paper investigates cross-modal contrastive learning for semantic representations and finds that grounded language embeddings are more semantically coherent than un-grounded ones. The reviewers generally like this direction and the paper is well-written. The authors should really heed the feedback from the reviewers, however, and in particular: the authors should make *very* clear that this is not a methods paper, but an analysis paper. Occasionally the paper reads too much like it is claiming a new method, which is problematic because a) the comparison to other models is lacking; and b) the paper is not adequately positioned in the existing literature. There are many papers that have explored visually-grounded semantic embeddings in the past, several of them already quite old, and this work should be positioned accordingly.
Decision: Accept (Poster)