Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper reveals an interesting and important failure pattern of the pre-trained vision-language models (VLMs): they are insensitive to object attributes, relations, and even word orders. They created several tests from Visual Genome, COCO caption, and Flickr30K, and show several representative VLMs (BLIP, CLIP, etc.) are insensitive to:

1. Relation (prepositions and verbs): e.g., differentiate between “the **man** is behind the **tree**’ vs “the **tree** is behind the **man**”

2. Attribute: e.g., differentiate between “the **crouched man** and the **open door**” and “the **open man** and the **crouched door**”

3. Word order: whether the model can tell if a sentence differentiates a sentence with shuffled words

The paper goes on discussing what causes the models to be insensitive to such compositional structures in the images and captions. They provide an intuitive hypothesis: the contrastive (retrieval) objective in the pre-training does not encourage the model to learn such compositional structures, because “the datasets are not designed to contain many images with captions containing similar words that must be differentiated” (or from my understanding, the probability of being able to sampling such “hard negatives” is very small). They also provided an experiment backing it up: models do not need order information to do well on current image-caption retrieval benchmarks.

They further show that if we purposely generate negative captions by swapping the word orders and sample hard negative images using clip similarity, the model does much better on the proposed evaluation benchmarks.
strength_and_weaknesses: There is no major weakness. I will just note two issues I noticed which do not affect my rating of the paper.

1.  In terms of the three challenges presented in the paper, it seems they can be addressed by explicitly considering how the challenge is formed (e.g., adding shuffled captions during training as hard negatives). However, this does not imply that the problem is solved. This reminds me of what is discussed in Teney et al., 2020, where once we know how the “challenge” dataset is constructed, we could use such information to train a model that performs well on the challenge dataset; but the model will likely fail on cases not foreseen during dataset creation.

   I wonder what should be the intended use of the benchmark. If simply adding shuffled captions can solve the challenges pretty well, what should be the next move?

Teney et al., On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law.

2. I would have appreciated more discussions on relation to negative mining and contrastive learning. 

---

Minor comments:

1. Most tables are in the appendix. While figures send a strong message, I would appreciate having a few tables in the main paper.

2. NegCLIP has two improvements: generating neg captions and sampling hard-neg images. Is there an ablation study on this?

3. What’s the batch size of fine-tuning CLIP?
clarity,_quality,_novelty_and_reproducibility: The paper is clear and easy to follow.
summary_of_the_review: Overall, this is a well-presented paper with an easy-to-follow yet important finding: performance on the current VL benchmarks could be misleading (even though these benchmark data contain images and captions of rich compositional structures) and current VLMs are very close to bag-of-word models.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper probes the compositional generalization ability of large pre-trained vision-and-language models, such as CLIP. They introduce a benchmark dataset of synthetically generated caption perturbations, called ARO. They show that out-of-the-box performance on ARO is poor, and that existing models do not seem to distinguish captions with ordering perturbation. They also propose a method for training these models to be more robust compositionally, by mining and generating hard negative examples to fine-tune with.
strength_and_weaknesses: Strengths:
- The discussion of the problem and presentation of the dataset and solution is extremely clear; the paper is very well-written. 
- The paper presents a clear problem with existing systems and the drawback of using retrieval as an evaluation method. The presented benchmark is offering a "bare minimum" sort of evaluation for such systems.

Weaknesses:
- The fonts in some of the figures is really small. I also strongly suggest putting numbers on the actual bar charts; it's difficult to interpret the results otherwise.
- The numbers for the experiment in 2.3 (evaluating COCO/Flickr30k on perturbations of captions) should be in the main paper, not the appendix.
- It is suggested that the reason these models ignore word ordering so much is that they really are trained as keyword identifiers, as required for image retrieval, and there's no reason to learn ordering. However, what happens if you incorporate better priors on the captions? I would imagine that a large language model would place very low probability on most of the perturbed captions (I could be wrong, though), and a VLM that uses features from a general large language model would be able to distinguish the obviously grammatically incorrect examples from the true caption.
- I was hoping the dataset would be a bit more of a scaled-up Winoground dataset, because Winoground directly tests all four settings (perturbation of relations in text and perturbation of relations in image). However, this evaluation set only seems to test perturbation of relations in text.

Questions:
- Why are there so few relations and attribute pairs in ARO, as described in Section 2.1? 
- Practically, how are some of the perturbations done? With operations on top of the parse tree?
- Does the experiment in Section 4 backprop through both text and image features in CLIP? 
clarity,_quality,_novelty_and_reproducibility: Quality:
- Experimental setup is thorough. My main concern would be that this becomes a new benchmark to achieve such that people interpret success on it to imply good compositional understanding. I think it works as a "bare minimum" evaluation, but performing well on it doesn't necessarily confirm a model is performing correct compositional reasoning (partially because I imagine using a LLM prior will downweight the grammatically incorrect or semantically implausible perturbations, for example).

Clarity:
- Paper is very clearly written, with a few small comments on readability of figures.

Originality:
- Relatively original; the experiments are new as far as I can tell, as is the proposed training method. However, in terms of a dataset, there are many that evaluate compositionality with synthetically generated data (GQA, CLEVR), or could be adapted to evaluate more semantically plausible alternatives for compositionality. Missing citation for the NLVR(2) corpora (Suhr et al. 2017/2019) which also evaluate compositionality with both true and false image pairs.
summary_of_the_review: This paper introduces a new benchmark and training method for evaluating compositionality of VLMs. The compositionality is mostly evaluated by perturbing captions of existing datasets in several ways. Experiments find that SOTA VLMs perform poorly (i.e., they cannot easily distinguish between perturbed and true captions). The paper is very well written. My main concern is that the kinds of perturbations in the evaluation set may be easy to reject simply by considering priors from an LLM, something which is not evaluated in this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a benchmark for evaluating the ability of vision-language models with regard to relations, attributions and order information. By testing existing visual-language models, they show that the exiting models have poor relationship understanding. They further propose a simple modification contrastive loss to help models become more sensitive to orders and compositionality.
strength_and_weaknesses: Strength:
1) This paper is well-written and easy to follow.
2) The discovery that existing visual-language models could not handle complex relationship understanding seems to be well-supported.

Weakness:
1) The proposed benchmark seems to be very similar as the Winoground, which is also cited in the related work. This, to some extent, limits the novelty of this work to the community. Could authors test the proposed composition-aware hard negatives on the Winoground to see if the method can improve these natural images capturing object relationship?

2) The proposed composition-aware hard negatives needs to know the targeted composition beforehand, which greatly limits the generalization and the potential effectiveness of this work.

3) "Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation" by Qin et al is closely related to this work and should be cited in Related work.
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: This work investigates the ability of existing visual-language models on relationship understanding and constructed a new complementary benchmark for this task. The proposed composition-aware hard negatives have major limitations but good to see they have effects on the targeted compositionality. I am leaning towards accepting this work and curious about the results on Winoground.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes four additional task for validating whether current large VL models trained with contrastive losses are sensitive to detailed attributes and relationships of the objects. The tasks include genome attributes and relations and COCO and Flickr captions. The paper found that most large VL models can not respond to the changes applied to the attributes and captions. In order to address this issue, the paper propose to add  composition-aware hard negatives during training. The resulting model obtains similar results on the original downstream task and good results on the 4 proposed new tasks. 
strength_and_weaknesses: Strengths: (1) Bring up an interesting and critical issue of not sensitive to compositional order of the current large VL models 
(2) Four new tasks based on existing dataset that do not need human annotations to validating the aforementioned issue
(3) a simple yet effective treatment to the issue by adding compositional-aware hard examples
(4) Good results on the newly proposed tasks


Weakness: 
(1) Hard to control each example in the 4 tasks, for example, swapping adj for the caption "a cute girl is walking a little dog", it would be good to have human check if the semantic is actually not valid 

clarity,_quality,_novelty_and_reproducibility: The paper is of good quality and the author promise to release code
summary_of_the_review: The paper proposes 4 new tasks to measure whether a large VL model trained using contrastive loss are sensitive to compositional orders. After finding that most of them perform bad on these 4 new task, the paper propose a simple fix by adding compositional aware hard negative. These validation of 4 new tasks and the compositional hard negatives help the model be more compositional aware without much suffering on the original  downstream tasks. My largest concern is that when using the 4 tasks as validation dataset, the dataset should eb further validated by human to check if the semantic is valid or not.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: *Summary*: This paper studies vision-language models that are trained using a contrastive alignment objective for the image and language representations. The authors find interesting and practically important failure modes of such models - insensitivity to word order, Subject-object relations, and other compositional failures. They propose a benchmark to evaluate these models since current benchmarks do not reveal these failures. Finally, they propose a hard negative mining approach that improves results.

*Strengths*: (1) The central finding of failure modes for vision-language models is stated and evaluated clearly. I believe this paper was written really well and easy to follow. (2) The authors propose a benchmark to evaluate the failure modes which is useful for future work. (3) The authors propose an automatic way to create compositional hard negatives that improve performance. 

*Weaknesses*: (1) The benchmark proposed in this work, as noted by other reviewers, is not extensive as it only tests for perturbations in the text. (2) The automatic hard negative mining process  relies on extra information which limits its broad applicability.
justification_for_why_not_lower_score: The problem studied in this paper is very relevant for visual representation learning. The authors have done a good job of finding failure modes for most vision-language models and proposing a sensible benchmark to evaluate it. I believe this deserves attention from the ML community at large given the increased interest in this field.
Decision: Accept: notable-top-5%