Title: When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?
Authors: Mert Yuksekgonul
Federico Bianchi
Pratyusha Kalluri
Dan Jurafsky
James Zou
Venue: ICLR 2023
Keywords: vision-language models
clip
contrastive learning
retrieval
vision-language pretraining
multimodal representation learning

Abstract: Despite the success of large vision and language models (VLMs) in many downstream applications, it is unclear how well they encode the compositional relationships between objects and attributes. Here, we create the Attribution, Relation, and Order (ARO) benchmark to systematically evaluate the ability of VLMs to understand different types of relationships, attributes, and order information. ARO consists of \emph{Visual Genome Attribution}, to test the understanding of objects' properties; \emph{Visual Genome Relation}, to test for relational understanding; and \emph{COCO-Order \& Flickr30k-Order}, to test for order sensitivity in VLMs. ARO is orders of magnitude larger than previous benchmarks of compositionality, with more than 50,000 test cases. We present the settings where  state-of-the-art VLMs behave like bags-of-words---i.e. when they have poor relational understanding, can blunder when linking objects to their attributes, and demonstrate a severe lack of order sensitivity. VLMs are predominantly trained and evaluated on large scale datasets with rich compositional structure in the images and captions. Yet, training on these datasets has not been enough to address the lack of compositional understanding, and evaluating on these datasets has failed to surface this deficiency. To understand why these limitations emerge and are not represented in the standard tests, we zoom into the evaluation and training procedures. We demonstrate that it is possible to perform well on image-text retrieval over existing datasets without using the composition and order information. This further motivates the value of using ARO to benchmark VLMs. Given that contrastive pretraining optimizes for retrieval on large datasets with similar shortcuts, we hypothesize that this can explain why the models do not need to learn to represent compositional information. This finding suggests a natural solution: composition-aware hard negative mining. We show that a simple-to-implement modification of contrastive learning significantly improves the performance on tasks requiring understanding of order and compositionality. 


Review discussion:
Strengths: (1) Bring up an interesting and critical issue of not sensitive to compositional order of the current large VL models
(2) Four new tasks based on existing dataset that do not need human annotations to validating the aforementioned issue
(3) a simple yet effective treatment to the issue by adding compositional-aware hard examples
(4) Good results on the newly proposed tasks
Weakness:
(1) Hard to control each example in the 4 tasks, for example, swapping adj for the caption "a cute girl is walking a little dog", it would be good to have human check if the semantic is actually not valid
The paper is of good quality and the author promise to release code
NO.
Dear Reviewer y4fM,
Thank you for your detailed review! You evidently spent a substantial time reviewing our paper, and we are thankful for your efforts during the challenging 2 week review period. We appreciate your comments that we “bring up an interesting and critical issue” and that we have “a simple yet effective treatment”.
**Semantic Validity:** First, we note how we get the attributes for the Visual Genome Attribution dataset. We mine the attributes “that are not mutual” between two objects.
Namely, if “small” is a common attribute between the two objects, we do not include it.
Hence, when the attributes in the test cases are swapped, it is not very likely to include common attributes. We argue that the probability of the caption being semantically valid after swapping is very low.
**Validation study:** We took your suggestion to use human validation and ran the following verification study:
To estimate the error probabilities, we randomly sampled 100 instances from the VG-Attribution dataset and 100 instances from the VG-Relation dataset.
Next, we recruited 2 reviewers, who were asked to pick between 4 options for each image: Caption 1, Caption 2, None, or Both. The order of captions is randomly swapped.
Results:
VG Attributions: 99/100 samples correct caption; 1/100 samples false caption, 0/100 labeled as None/Both.
VG Relations: 93/100 samples correct caption; 4/100 samples false caption, 3/100 labeled as None 0/100 labeled as both.
Overall, we believe these demonstrate that the semantic validity of a permuted caption for the given image is a low-probability event. We are posting a subset of our dataset at https://anonymous.4open.science/r/aro-benchmark-samples-65E0/  for you to investigate if desired. The full dataset will be released upon deanonymization.
Thank you very much for your thoughtful review! We believe your comments led us to demonstrate the validity of our tasks. Please let us know if you have further questions, we are happy to follow up.
Strength:
1) This paper is well-written and easy to follow.
2) The discovery that existing visual-language models could not handle complex relationship understanding seems to be well-supported.
Weakness:
1) The proposed benchmark seems to be very similar as the Winoground, which is also cited in the related work. This, to some extent, limits the novelty of this work to the community. Could authors test the proposed composition-aware hard negatives on the Winoground to see if the method can improve these natural images capturing object relationship?
2) The proposed composition-aware hard negatives needs to know the targeted composition beforehand, which greatly limits the generalization and the potential effectiveness of this work.
3) "Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation" by Qin et al is closely related to this work and should be cited in Related work.
See above.
NO.
Dear Reviewer LKvK,
Thank you for your detailed review! You evidently spent a substantial time reviewing our paper, and we are thankful for your efforts during the challenging 2 week review period. We appreciate your comments saying “the paper is well-written and easy to follow”, and finding our claims well-supported.
**Comparison to Winoground**: We agree that Winoground is a wonderful first step to study this problem. We first refer to a recent paper “Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality” (Diwan et al. EMNLP, 2022). Fundamentally, Winoground intertwines several challenges beyond compositionality such as ambiguous images and tests requiring common-sense reasoning and world knowledge. As quick examples, see at https://huggingface.co/datasets/facebook/winoground/viewer/default/test sample id 44 ("the heavy oncoming traffic is contrasted with the light outgoing traffic") / 70 ("the person with the lighter shoes is holding an emoji and the person with the darker shoes isn't"). In our work, we aimed to isolate this effect and focus on testing for compositionality, without overloading with other kinds of knowledge.
Hence, we not only scale the number of samples but also propose a much more specific and targeted testbed.
Overall, Winoground is a great test to evaluate broad Visio-linguistic understanding;
but due to its small size (400 examples) and the set of non-compositional difficulties, our dataset offers a complementary evaluation.
We tested NegCLIP on Winoground and we find that it marginally improves the performance (30.5% = 30.5% overall score, 11.5% > 10.5% image score, 8% = 8% text score where Overall Score is matching all of the pairs correctly, Image Score is retrieving the correct caption for each image; see the Winoground paper for details) over CLIP. In fine-grained subsets, we observe both improvements (Relation Tag: Text 26% > 22% (N=233), Group 7% > 6%; Series Tag Text 26% > 9% (N=23)) and some performance losses. Due to the smaller sample size of Winoground, it is difficult to say if the changes are statistically significant.
**Limitation of targeted compositions**: We agree that this approach could be further improved. We’d like to propose the vision analogues. We want vision classifiers to be invariant to rotations or flips, hence many models have them as data augmentations during training. Our approach is very similar in that we are encoding our inductive biases in augmentations.
Yet, we definitely agree that further research could improve this, e.g. with Language Model Priors!
In fact, to back this up, we refer to the latest developments in text-to-image models. Imagen performs better at the compositional tasks than DALL-E (Saharia et al. 2022).
Even though the methodologies are mostly similar, we believe the main difference is that Imagen is using T-5 as the text encoder, versus DALL-E which uses CLIP! We believe this is related to our findings, and as you also suggested, there should be an interesting avenue for research in getting the best of both worlds, we added this as a comment to our paper. We believe our paper makes a case for further research in this venue.
**Citation**: Thank you for pointing us to this paper! We added the corresponding citation both in related works and patch permutations.
Thank you very much for your thoughtful review! We believe your comments led us to have a more complete paper. Please let us know if you have further questions, we are happy to follow up.
Strengths:
- The discussion of the problem and presentation of the dataset and solution is extremely clear; the paper is very well-written.
- The paper presents a clear problem with existing systems and the drawback of using retrieval as an evaluation method. The presented benchmark is offering a "bare minimum" sort of evaluation for such systems.
Weaknesses:
- The fonts in some of the figures is really small. I also strongly suggest putting numbers on the actual bar charts; it's difficult to interpret the results otherwise.
- The numbers for the experiment in 2.3 (evaluating COCO/Flickr30k on perturbations of captions) should be in the main paper, not the appendix.
- It is suggested that the reason these models ignore word ordering so much is that they really are trained as keyword identifiers, as required for image retrieval, and there's no reason to learn ordering.
However, what happens if you incorporate better priors on the captions? I would imagine that a large language model would place very low probability on most of the perturbed captions (I could be wrong, though), and a VLM that uses features from a general large language model would be able to distinguish the obviously grammatically incorrect examples from the true caption.
- I was hoping the dataset would be a bit more of a scaled-up Winoground dataset, because Winoground directly tests all four settings (perturbation of relations in text and perturbation of relations in image).
However, this evaluation set only seems to test perturbation of relations in text.
Questions:
- Why are there so few relations and attribute pairs in ARO, as described in Section 2.1?
- Practically, how are some of the perturbations done? With operations on top of the parse tree?
- Does the experiment in Section 4 backprop through both text and image features in CLIP?
Quality:
- Experimental setup is thorough. My main concern would be that this becomes a new benchmark to achieve such that people interpret success on it to imply good compositional understanding. I think it works as a "bare minimum" evaluation, but performing well on it doesn't necessarily confirm a model is performing correct compositional reasoning (partially because I imagine using a LLM prior will downweight the grammatically incorrect or semantically implausible perturbations, for example).
Clarity:
- Paper is very clearly written, with a few small comments on readability of figures.
Originality:
- Relatively original; the experiments are new as far as I can tell, as is the proposed training method.
However, in terms of a dataset, there are many that evaluate compositionality with synthetically generated data (GQA, CLEVR), or could be adapted to evaluate more semantically plausible alternatives for compositionality. Missing citation for the NLVR(2) corpora (Suhr et al. 2017/2019) which also evaluate compositionality with both true and false image pairs.
NO.
Dear Reviewer U4Mo,
Thank you for your detailed review! You evidently spent a substantial time reviewing our paper, and we are thankful for your efforts during the challenging 2 week review period. We appreciate your kind comments, finding our “ solution is extremely clear”; and “the paper is very well-written”.
**Format and citations**: We updated the paper to include the numbers from the tables in Figure 1-2 to have the results. Please see the updated figures that include concrete numbers in addition to the plots. We also added the citations we missed, thank you for pointing them out.
**Having language priors**: We are very excited about this comment. We imagine that there will be a fruitful research avenue on merging the power of language models with multimodal training.
In fact, to back this up, we refer to the latest developments in text-to-image models. Imagen performs better at the compositional tasks at DALL-E (Saharia et al. 2022).
Even though the methodologies are mostly similar, we believe the main difference is that Imagen is using T-5 as the text encoder, versus DALL-E which uses CLIP! We believe this is tightly related to our findings, and as you also suggested, there should be an interesting avenue for research in getting the best of both worlds, we added this as a comment to our paper. We believe our paper makes a case for further research in this venue.
**Comparison to Winoground**: We agree that Winoground is a wonderful first step to study this problem. We first refer to a recent paper “Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality” (Diwan et al. EMNLP, 2022). Fundamentally, Winoground has further challenges beyond compositionality such as ambiguous images and tests requiring common-sense reasoning and world knowledge. As quick examples, see at https://huggingface.co/datasets/facebook/winoground/viewer/default/test sample id 44 ("the heavy oncoming traffic is contrasted with the light outgoing traffic") / 70 ("the person with the lighter shoes is holding an emoji and the person with the darker shoes isn't") or more. Here, we aimed to isolate this effect and focus on testing for compositionality, without overloading with other kinds of knowledge.
Hence, we not only scale the number of samples but also propose a much more specific and targeted testbed.
Overall, Winoground is a great test to evaluate broad Visio-linguistic understanding;
but due to its size (400 examples) and the set of non-compositional difficulties, our dataset offers a complementary evaluation.
**Number of relations / attribute pairs**: In Appendix A.1, we describe the dataset generation procedure. To keep the annotation quality high, we filter the objects that are small in the image and hence may be hard to see (i.e. we require both height and width to be at least 1/4th of the image), and this eliminates a large set of attributes/relations from Visual Genome.
**How perturbations are done**: We use Spacy to do part-of-speech tagging, and then perform the perturbations using POS tags. We will release the code that we used to generate the perturbations.
**Backprop through both text and image features in CLIP?**: Yes, we backprop through both text and images.
**Tracking the progress / "bare minimum" evaluation**: We agree with this sentiment.
For instance, we do not expect a model to perform well in Winoground without performing well on our benchmark, as ours is a more isolated and focused evaluation. It was also indeed an intriguing surprise that the models are even not doing well with this bare minimum (grass is eating the horse vs horse is eating the grass)!
Hence, we believe our evaluations are important to track the progress.
On the other hand, we believe our methodology to generate test cases from Visual Genome can be used to implement more complex tasks, which can be helpful for future work.
Overall, we enjoyed reading your comments and discussion, and hope that our comments clarify your questions. Please let us know if you have further questions or comments, and thank you for being very thorough with your review!
There is no major weakness. I will just note two issues I noticed which do not affect my rating of the paper.
1.
In terms of the three challenges presented in the paper, it seems they can be addressed by explicitly considering how the challenge is formed (e.g., adding shuffled captions during training as hard negatives).
However, this does not imply that the problem is solved. This reminds me of what is discussed in Teney et al., 2020, where once we know how the “challenge” dataset is constructed, we could use such information to train a model that performs well on the challenge dataset;
but the model will likely fail on cases not foreseen during dataset creation.
I wonder what should be the intended use of the benchmark. If simply adding shuffled captions can solve the challenges pretty well, what should be the next move?
Teney et al., On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law.
2. I would have appreciated more discussions on relation to negative mining and contrastive learning.
Minor comments:
1. Most tables are in the appendix.
While figures send a strong message, I would appreciate having a few tables in the main paper.
2. NegCLIP has two improvements: generating neg captions and sampling hard-neg images. Is there an ablation study on this?
3. What’s the batch size of fine-tuning CLIP?
The paper is clear and easy to follow.
NO.
Dear Reviewer J2a5,
Thank you for your detailed review! You evidently spent a substantial time reviewing our paper, and we are thankful for your efforts during the challenging 2 week review period. We are very excited about your approval, and kind comments saying our work has “no major weakness”, and has “easy-to-follow yet important findings”.
**Tracking the progress**: We share your sentiment that performing well on our tasks will not be the end goal, and we view our benchmark as a first, and isolated step. To motivate this, we refer to the concurrent work where Winoground has been suggested to include broader challenges, such as Text Difficulties / Complex Reasoning and more (“Why is Winoground Hard? Investigating Failures in Visuolinguistic Compositionality”. Diwan et al. EMNLP, 2022). In this sense, we believe our work isolates these effects and focuses on tracking the progress focusing on compositional sensitivity.
**What’s next**: We also note that there is still further improvement to expect, both in our challenges and definitely on Winoground! Broadly speaking, hard negative mining is an augmentation strategy to encode our inductive biases, but certainly does not mean inheriting all the desired properties of language. We believe there is a very exciting area of future work that aims to inherit (and evaluate!) the language model priors to improve the understanding of VLMs. We hope that our work motivates this branch.
**Minor comments**: We updated the paper to include the numbers from the tables in Figure 1. Please see the updated figures that include concrete numbers in addition to plots.  We are sorry to miss the batch size in the text, it is $N=1024$ and we updated the paper to reflect this. In preliminary experiments, we observed that having both neg captions and neg images presents the best results. We will add the results of the ablation to the camera-ready version.
We again thank you for your detailed comments, exciting intellectual query on what is the bigger picture, and for helping us make our paper better. Please let us know if you have further questions or comments, we are happy to follow up!

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: *Summary*: This paper studies vision-language models that are trained using a contrastive alignment objective for the image and language representations. The authors find interesting and practically important failure modes of such models - insensitivity to word order, Subject-object relations, and other compositional failures. They propose a benchmark to evaluate these models since current benchmarks do not reveal these failures. Finally, they propose a hard negative mining approach that improves results.

*Strengths*: (1) The central finding of failure modes for vision-language models is stated and evaluated clearly. I believe this paper was written really well and easy to follow. (2) The authors propose a benchmark to evaluate the failure modes which is useful for future work. (3) The authors propose an automatic way to create compositional hard negatives that improve performance. 

*Weaknesses*: (1) The benchmark proposed in this work, as noted by other reviewers, is not extensive as it only tests for perturbations in the text. (2) The automatic hard negative mining process  relies on extra information which limits its broad applicability.
justification_for_why_not_higher_score: N/A
justification_for_why_not_lower_score: The problem studied in this paper is very relevant for visual representation learning. The authors have done a good job of finding failure modes for most vision-language models and proposing a sensible benchmark to evaluate it. I believe this deserves attention from the ML community at large given the increased interest in this field.

Decision: Accept: notable-top-5%