SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality
Authors: Cheng-Yu Hsieh
Jieyu Zhang
Zixian Ma
Aniruddha Kembhavi
Ranjay Krishna
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: vision-language models
contrastive training
CLIP
compositional understanding
compositionality
dataset artifacts
TL;DR: We show that existing benchmarks for vision-language compositionality are hackable. We present SugarCrepe, a new benchmark that remedies this vulnerability to faithfully evaluate a vision-language model's compositionality.

Abstract: In the last year alone, a surge of new benchmarks to measure $\textit{compositional}$ understanding of vision-language models have permeated the machine learning ecosystem.
Given an image, these benchmarks probe a model's ability to identify its associated caption amongst a set of compositional distractors.
Surprisingly, we find significant biases in $\textit{all}$ these benchmarks rendering them hackable. This hackability is so dire that blind models with no access to the image outperform state-of-the-art vision-language models.
To remedy this rampant vulnerability, we introduce $\textit{SugarCrepe}$, a new benchmark for vision-language compositionality evaluation.
We employ large language models, instead of rule-based templates used in previous benchmarks, to generate fluent and sensical hard negatives, and utilize an adversarial refinement mechanism to maximally reduce biases. We re-evaluate state-of-the-art models and recently proposed compositionality inducing strategies, and find that their improvements were hugely overestimated, suggesting that more innovation is needed in this important direction.
We release $\textit{SugarCrepe}$ and the code for evaluation at: https://github.com/RAIVNLab/sugar-crepe.

CREPP: Fixing Hackable Benchmarks for Vision-Language Compositionality
Cheng-Yu Yu Hsieh'; Jieyu Zhang'; Zixian Ma', Aniruddha Kembhavi', Ranjay Krishna
*University of Washington, Allen Institute for Artificial Intelligence
zixianma, ran @cs. washington. edu, anik@allenai.om org
Abstract
the last year alone, a surge of new benchmarks to measure compositional understanding of vision-language models have permeated the machine learning ecosystem. Given an image, these benchmarks probe a model's ability to identify its associated caption amongst set of compositional distractors. Surprisingly, we find significant biases in all these benchmarks rendering them hackable. This hackability is so dire that blind models with no access to the image outperform state-of-the-arr vision-language models. To remedy this rampant vulnerability, we introduce SUGARCREPF,
benchmark for vision-language compositionality evaluation. We employ large language models, instead of rule-based templates used in previous benchmarks, generate fluent and sensical hard negatives, and utilize an adversarial refinement mechanism to maximally reduce biases. We re-evaluate state-of-the-ee models and recently proposed compositionality inducing strategies, and find that their improvements were hugely overestimated, suggesting that more innovation is needed in this important direction. We release SUGARCREPE and the code for evaluation at: https: //t/t/gttb////////////////t com Satm/RAAIWLab/eeee sugar- crepe.
Introduction
Scholars today herald compositionaliiy as fundamental presupposition characterizing both human perception and linguistic processing [10]. Through compositional reasoning, humans can comprehend new scenes and describe those scenes by composing known atoms [19. 17, 3. 9]. For instance, compositionality allows people to differentiate between a photo of "a girl in white facing a man in black" and "a girl in black facing man in white". For while now, vision-language research has sought to develop models that can similarly comprehend scenes and express them through compositional language [22, 20, 29, 15].
Given its importance, a surge of new benchmarks have been proposed to evaluate whether vision- language models exhibit compositionality. Recently, Winoground [45], VL-CheckList [53]. ARO [49], CREPE [30], and Cola [38] have entered the machine learning zeitgeist. Evaluation mostly done through image-to-text retrieval task formulation [53, 49, 30]: by measuring how often models pick the description, "a girl in white facing a man in black" when presented with an image of it, and avoid choosing the incorrect hard negative description, "a girl in black facing man in white".
In this work, we uncover a crucial vulnerability in not just one but all these image-to-text compo- sitionality benchmarks: We find that a blind model that never looks at the image, can identify the correct caption and avoid choosing the supposed "hard negatives". This blind model outperforms wide array of pretrained vision- vision-language models across the suite of benchmarks [36, 18, 14]. We explain this undesired hackability in existing benchmarks by showcasing that there exists a significant distributional gap between the positive and hard negative captions. For instance, in the ARO bench-
The authors contribute equally this work.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

mark [49], human-generatee generated positive captions differ drastically from the hard negative texts generated by randomly shuffling words in the positive captions. As new research has begun to propose methods that claim to improve compositionality on these benchmarks [49, 38), we find it critical to highlight our findings and propose a solution.
We propose a solution to existing hackable benchmarks by introducing SUGARCREPE, a new benchmark faithfully evaluate compositionality. In curating SUGARCREPE, we identify two main biases that result in the distributional gap between positive and hard negatives; and employ mechanisms to fix the shifts. In particular, we find the current procedure in generating hard negatives introduces descriptions that are (1) not plausible and (2) non-fluent For example, while the caption sensical fluent caption, benchmarks often have non- non-plaun non-plausible
"olives and grapes on a plate" hard negatives like "olives and grapes inside plate" or simply incomprehensible ones like "right has word another word. There is a words" (see Table for more examples). We mitigate such biases by first leveraging a modern large language model, ChatGPT [32], to generate plausible and natural hard negative texts instead of relying on simple rule-based templates employed by existing benchmarks [30, 491. Then, we subsample the dataset through adversarial refinement process to ensure the identified biases are maximally removed by drawing on recent dataset de- biasing work [50, 41, 233. Taken together, this workflow is where SUGARCREPE derived its name: Synthetic yet Unbiased Generation with Adversarially Refined Compositional REPresentation Evaluation. We qualitatively and quantitatively verify through both human and automatic evaluations that SUGARCREPE effectively fixes these biases.
With SUGARCREPE, we re-evaluate recent methods proposed to improve compositionality. Specif- ically, we focus on one prominent approach that aims to improve compositionality through data augmentation. This method trains models by generating compositional hard negatives and injecting them within a training batch [13, 49]. Unfortunately, we observe that the effectiveness of this simple data augmentation approach is hugely overestimated when evaluated on existing benchmarks, leading to limited improvements on SUGARCREPE. Finally, we evaluate a wide variety of 17 pretrained CLIP models [36, 18, 14], and find that current models still lack compositionality. Our results suggest that to improve compositionality. future work may need more innovative techniques.
2 Related Work
We situate our paper amongst existing work on vision-language compositionality, and debiasing datasets for model evaluation.
Evaluating lasiunnl compositionality. Recent works have introduced benchmarks to evaluate the compositionality of vision-language models [36]; they find that current models exhibit little compositional understanding [49, 45, 53, 30, 38] despite their remarkable performance on downstream tasks [36, 25, 43, 1, 47, 48, 52]. Models have a hard time discerning between text containing the same words ordered differently [45]. Models also fail to link objects to their attributes, or understand the relationship between objects [53, 49, 38]. Our work finds that many of the benchmarks used to evaluate compositionality have hackable biases; blind models that do not even look at the image outperform state-of-the-an vision-language models.
Improving vision-language compositionality. To enhance vision-languan models' composi- tionality, new proposals suggest training strategies that utilize additional data, models, and/or losses [49, 5, 38, 13, 44]. Amongst them, one prominent approach is to explicitly train the models to distinguish hard negatives from the correct captions [49, 13]. While these approaches appear to improve compositionality on benchmarks, it is unclear if these models achieve such improvements by actually acquiring compositional understanding or by exploiting biases in these datasets. We answer this question in our evaluation.
Debiasing dataset for faithful model evaluation. Several prior manuscripts have pointed out that biased datasets could lead to an overestimation of models' true capabilities [16]. They have proposed
dataset de- de-biasing methods to enable more faithful model evaluations [39, 50, 41, 23, 34]. For instance, adversarial filtering [50] iteratively trains an ensemble of classifiers on different training splits and uses them to filter out "easy" negatives for each instance. Building upon adversarial filtering, AFLite (41, 23] filters data instances in more light- manner without retraining
2 We use biases and artifacts interchangeably in the paper.

---

Table I: Existing compositionality benchmarks rely on snccduurall--eneraate hard negatives which often do not make logical sense or are not fluent due to grammatical errors.
Dataset
Nonsensical Hard Negatives
Non fluent Hard Negatives
CREPE [30]
Olives and grape inside plate.
A door with punes not in room: the door has windows.
the flowets.
Right
another word. A shell' with books in somathing. There is no background.
words.
A huir wearing a nocklave, with her lady on a lable.
ARO [-99
The grass cuting the horse,
At brown cal a looking dog sitting is and white buthtu.
gay bathtu looking white
Scene cemarkable ball blue green hehind chair.
Green ball with a remarkable chuir behind a blue scene.
Books the looking people arc.
VL. (63]
Sheep is handwood.
An man fishing a food from a wrapper using pas at open.
Emply zebwas.
It heasing city.
The bush speaking in the gardon.
An grouping subddunng at room access.
model each iteration and leads benchmarks that more accurately represent the underlying tasks. We use adversarial refinement to remove biases that creep into the generation of compositionality benchmarks.
3 Limit and biases of current compositionality benchmarks
A majority of existing compositionality benchmarks for vision-language models formulate the evaluation task as image-to-text retrieval [53, 49, 30]. We focus these benchmarks and discuss others [45, 38] in Appendix B. Given an image. the model is probed to select text that correctly describes the image from a pool of candidates. Unlike standard retrieval tasks where the negative (incorrect) candidates differ lot from the positive (correct) text, compositionality benchmarks intentionally design hard negative texts that differ minimally from the positive text, in order to test whether the model understands the fine-grained atomic concepts that compose the scene.
Existing hard negative generation process introduces undesirable biases. Existing benchmarks generate hard negative texts through rule-based programmatic procedures [53, 49,30) 30], which produce hard negatives by replacing word of specific type (an object, attribute, or relation) in the original text, by swapping two words, or by shuffling the word order. We find that such procedures introduce unintentional biases in the generated hard negatives (see Table 1); specifically, we observe two major types of undesirable artifacts: (1) nonsensical artifacts, and (2) non-fluent artifacts. In order to quantitatively measure these biases, we utilize Vera [27], a plausibility estimation model, to characterize the nonsensical bias. Specifically, we define Vera(T") to be the plausibility score of a caption T. where a higher score suggests more sensical the caption is. Similarly, to capture the non-fluent bias, we leverage a grammar-check model [31] that assigns high scores, Grammar(7)) to more grammatically correct texts. In Figure I, we find that Vera and the grammar model assign higher scores to positive texts, suggesting that many hard negatives are nonsensical and not fluent.
Dataset biases render current compositionality benchmarks ineffective. Given the heavily- skewed score gaps, we show that blind models (i.e., Vera and the grammar model) that simply select the higher-scoring texts as positives and admittedly do not possess any vision-language compositionality, can achieve state-of-the-arr performances on existing benchmarks. We compare the the blind models against 17 pretrained CLIP models from three sources: OpenAI's in-house Weblmage Text dataset [36], LAION [42], and Datacomp [14]. We plot the performances of the blind models and the best-performing CLIP models from each category (Figure 2). Blind models achieves state- siieeoof-ta the performances on 9 out of 10 existing benchmark tasks. We provide full evaluation results in Appendix D.
4 SUGARCREPE
We introduce SUGARCREPE, new benchmark for faithful evaluation of vision-language models' compositionality based on the image-text pairs of COCO [26]. SUGARCREPE presents two key contributions over existing benchmarks: (1) it drastically reduces the two identified dataset biases (Sec. 4.1, and (2) covers broad range of fine-grained types of hard negatives (Sec. 4.2). We present summary comparison on compositionality benchmarks in Appendix B.
3

---

CREPE
ARO
VL-CheckList
Hard Negative
VG Relation
Object
VG Attribution
Swap
Attribute
COCO Order
Negate
Flickr Order
Relation
-1.0
-0.5
0.0
0.5
1.0
-1.0
0.5
0.0
0.5
1.
-0.5
0.0
0.5
1.0
Vera Score Gap (Positive
Hard Negative)
Hard Negative
Atomic
VG Relation
Object
VG Attribution
Swap
Attribute
COCO Order
Negate
Flickr Order
Relation
0.0
0.5
-1.0
-.5
0.0
0.5
0.5
0.0
0.5
Grammaa Score Gap (Positive Hard Negative)
Figure 1: Top row: We define Vera score gap as the score difference between the positive and hard gap distribution lies the positive
negative texts: Vera(TP) Vera(TT"). The entire Vera spectrum, indicating that the semplate-generatee hard negative texts usually have low plausibility. Bottom row: Similarly, Grammar score gap is defined by: Grammar(TP) Grammar(T")) On grammar score, we also find that the distribution largely rests on the positive side, suggesting that most hard negative texts in existing benchmarks exhibit grammatical errors.
DataComp
CREPE
ARO
VL-Checklist
M Flickr Order
Recallian
50
80
25
40
Atomic
Negate
VG Relation VG Actribution COCO Order
Object
Attribute
Relation
Figure 2: Blind commonsense Vera model and Grammar model outperform saate-of-the-arr CLIP models on nearly all existing benchmarks by exploiting the nonsensical and fl-ult artifacts. This suggests that existing benchmarks are hackable and ineffective in measuring compositionality.
4.1 SUGARCREPE generation workflow alleviates dataset biases
The generation procedure of SUGARCREPE consists of three main stages, centered around creating sensical and fluent hard negatives that close the distributional gaps to the positive texts, and ensuring balanced distribution on the score gaps make the final dataset robust the identified biases.
Stage Generate sensical and fluent hard negatives with large language model. Observing the capability of modern large language models in generating fluent and plausible texts, we leverage ChatGPT [32] to generate hard negative texts where we explicitly instruct it to avoid commonsense (logical) and fluency (grammatical) errors. To guide ChatGPT in re-writing given positive text into its hard negative counterparts, we provide few-shot demonstrations written by the authors and leverage its in-context learning ability [4] to generalize unseen texts. Figure 3 shows example demonstration used and an actual hard negative generated. We detail all the prompt templates in Appendix C.2. Table 3 shows the comparisons between hard negatives generated from ChatGPT in SUGARCREPE and that from existing benchmarks.
Stage 2: Filter false negatives with human validation. A generated text is considered valid hard negative only if incorrectly describes the corresponding image. For example, given an image with positive caption "a man and child sitting a sofa". compositional change that replaces "child" with "girl" may still result a correct caption. To ensure the validity of the hard negatives in SUGARCREPE, filter out false negatives by manually examining the generated hard negatives and their corresponding images.
Stage 3: De-bias dataset with adversarial refinement. While ChatGPT yields more sensical and fluent text, there is no guarantee that bias between positive and negative texts is negligible. Following dataset de-biasing work [50, 41, 23], we develop an adversarial refinement mechanism that maximally reduces the undesirably exploitable artifacts SUGARCREPE. Specifically, our goal to ensure that performance improvements on SUGARCREPH cannot be achieved by exploiting the

---

Algorithm Adversarial Refinement
civen an Laput sentence describing a scene, your task
1s to:
Require: Text-only model Mi and M2: Num- ber of grids K; A set of candidates D (T, TP.T)) JEE[N] where I,, T and 7, are i-th image, positive caption, and negative cap- tion.
1. Locate the noun worda in the sentence.
2. Randonly pick
word.
3. Replace the selected noun word with new noun word
to nake new centence.
The new sentence most sest the following thres
requirenents:
Ensure: A subset D C D
1. The new sentance must be describing scene that ia
1: Calculate the model score gap for each candi- date g. Mj (T!) (TP) and gf
as
at poseible from original scene.
2. The new sentence must be fluent and gramatically correct.
M2 - (T
sentence make logical sense.
2: Split the 2D space [- [ to K
Here are sone exaxples:
equal-size grids.
Original centence: nan is kitchen naking pizzas.
3: Place each candidate
grid based the
Nouns: ["man", "ksitchen", "pizzar"]
score gaps g." and g!?
Selected noun! man
4: Initialize D = 0
New sentence: wonan in kischen making pizzas.
5: for each pair of grid (G,, GJ) symmetric about the original point (0. 0) do
Original centerce:
seated wall and birds
besides her
6:
if IG, > |G then
Nouns: ('wonan', , 'wall', "birds") "
7:
Sample [G]] candidates from Gy and
Selected noun: wall
noun:
put them to D.
New sentance: wonan seated banch and bi
8:
Put candidates in GÝ to D.
besides her.
9:
else
10:
Sample [Gy] candidates from Gj and
Figure 3: Example prompt (black) and actual hard negative (green) generated from ChatGPT.
put them to D.
11: Put candidates in G, D.
identified nonsensical and non-fluent biase.. To accomplish this, we characterize the biases again with the commonsense and grammar models [27, 31J, and subsample the dataset to ensure symmetric score gap distributions on both the positive and negative sides, as shown in Figure 4. We note the symmetry around zero implies that the commonsense and grammar scores can no longer be used to infer the ground truth positive texts. We provide the adversarial refinement algorithm in Algorithm 1.
4.2 SUGARCREPE covers a broad range of hard negative types
To test different aspects of vision-language models' compositional understanding, we follow CREPE [30] to consider various forms of hard negatives, and follow VL-CheckList [53] and ARO [49] to consider different fine-grained categories of the atomic concepts. In total, SUGARCREPE covers 7 fine-grained types of hard negatives, as shown in Table 2. We introduce the dataset taxonomy below, starting from the form of the hard negatives to its different f (iner-grainel variants.
The REPLACE form. Given a positive text describing a scene, we generate a REPLACE hard negative by replacing an atomic concept the original text with a new concept that makes the text mismatch with the original scene. Based on the type of the atomic concept-objer attribute, or relation-we further categorize REPLACE hard negatives into REPLACE- OBII, REPLACE-ATT, and REPLACE-REL.
The SWAP form. Different from REPLACE, SWAP does not introduce new concepts in the hard negatives, but SWAP hard negative is generated by swapping two atomic concepts of the same category in the positive text. We further categorize SWAP into SWAP-OBJ and SWAP-ATT, and omit swapping two relationships since it generally results in nonsensical texts.
The ADD form. Similar to the REPLACE form, but instcad of replacing an atomic concept with a new one, generate an ADD hard negative by adding new atomic concept to the positive text that makes it mismatch with the original scene. We only further categorize ADD into ADD-OBJ (adding object concept) and ADD-ArT (adding attribute concept), as adding new relationship concepts to the positive texts often make them highly implausible.
Dataset overview. The final evaluation set of SUGARCREPE consists of 512 examples, where the numbers for each fine-grained type are listed in Table Each example is an image-to-text retrieval task composed of an image, a positive text, and a hard negative. On SUGARCREPE. random chance performance has an average accuracy of 50%. We note that ARO and CREPE additionally consider SHUFFLE (randomly shuffling words in sentence) and NEGATE (adding negation keywords "no/not"
5

---

Table 2: We report the number of hard negative captions of all types in SUGARCREPE.
REPLACE
SWAP
ADD
Object
Attribute
Relation
Object
Attribute
Object
Attribute
# negative captions
1,652
788
1,406
246
666
2,062
692
Table 3: We present example positive texts and their hard negatives in ARO+CREPE (generated
using existing procedures) and SUGARCREpE (generated with ChatGPT). SUGARCREPE brings significant improvements in commonsense and fluency.
Hand Negutive Type
Test Type
Commoneense
Fluency
Original
Two adult bears play fight the water.
A man sitting in from of lapoo computer
RUPLACE
ARO&CREPE
Two adult hears play Sigtt the soda.
A man sitting around froat of laptop computes
SUGARCREPE
A Book of ducks play fight in the water.
A mun standing in fromt of liptop computer
Original
wuman standing behind fence looking elephant.
Man swinging lennis mekktt while group of people watches. people wasches.
ARO+CREPE
standing behind
looking elepham.
swinging
racket while
SUGARCREER
elephant standing behind fonce looking
Group people swinging tennis neket while warches.
Original
loddy hear next suuffd fish.
A rod fire hydrent city sidewalk.
NEGATE / ADD
ARO+CREPE
A soddy bear nest to a stuffed fish. There is no roddy bear.
A red fire not hydram on a city sidewalk.
SEGARCREFE
A ieddy bear and staffed fish and a sobor toy.
A rod tie hydrant and a trash can on a city sidewalk.
to sentence) hard negatives. We however omit them in SUGARCREPE SHUFFLE is very unlikely to be plausible and fluent, and NEGATE introduces irreducible keyword artifacts [30]."
5 Evaluations
In this section, we qualitatively and quantitatively compare SUGARCREPE existing benchmarks (Sec. 5.1), re-evaluate recent methods proposed to improve compositionality of vision-language models (Sec. 5.2), and comprehensively evaluate wide array of pretrained CLIP models (Sec. 5.3).
To systematically and fairly compare SUGARCREPE with existing benchmarks, we normalize the benchmarks by reproducing their data generation workflow using COCO [26] as in SUGARCREPE. We utilize source code from CREPE [30] to generate REPLACE, SWAP, NEGATE hard negatives and take SHUFFLE hard negatives released in ARO [49]. We refer to this reproduced dataset as ARO+CREPE. In addition, we standardize the evaluation task as retrieving the correct caption from two possible choices, i.e., positive text and hard negative. This normalization sets the positive texts fixed for all benchmarks, including SUGARCREPE.
5.1 SUGARCREPE significantly reduces dataset biases
SUGARCREPE generates more sensical and fluent hard negatives. We validate that SUGARCREPE generates higher quality hard negative texts by leveraging ChatGPT than previous rule-based ap- proaches. Qualitatively, in Table 3, we observe that the hard negatives in SUGARCREPE are more sensical and fluent compared to hard negatives in ARO+CREPE. We report human evaluation results in Appendix D.2 that show on average of 35% of examples, hard negatives in SUGARCREPE have strictly higher quality than ARO+CREPE in terms of commonsense and fluency. For instance, on SWAP, humans judge that SUGARCREPE wins 68% over ARO+ ARO+CREPE and ties on 28% of examples in terms of commonsense. Quantitatively, in Table 4, we compare the commonsense and grammar scores averaged over the hard negative texts in both ARO+CREPE and SUGARCREPE. We see SUGAR C CREPE has much higher average scores than ARO+CREPE. Additionally, pairwise com- parisons show that SUGARCREPE has higher commonsense and grammar scores than ARO+CREPE on 86% of examples on average.
SUGARCREPE disentangles the identified exploitable biases. We show that the final SUGARCREPE evaluation set maximally reduces the identified biases that could be exploited undesirably to achieve improvements benchmark. Figure 4 visualizes the Vera/ Ceram/rrammr score gap distributions. We compare the distributions between ARO+CREPE and SUGARCREPE (before and after adversarial refinement). First, We see that by leveraging ChatGPT, the hard negative texts in SUGARCREPE already have lower biases than ARO+CREPE before adversarial refinement, i.e., the score gap distribution is more centered around zero. Furthermore, see that after adversarial refinement, the
One can easily infer hard negatives from whether the text contains negation keywords "no/uot".

---

Table 4: We compare the commonsense and grammar hard negatives in ARO+CREPE and SUGARCREPE. We report both their respective average scores and the ratio where SUGARCREPE has higher score than ARO+CREPE in pairwise comparison. Overall, SUGARCREPE has hard negatives with better commonsense and grammar.
Average Score
Hard negatie Type
Metric
ARO-CREPE
SUGARCREPE
Pairwise Better Ratio
Commonsense
37.46
50.21
77.71
REPLACE
Grammar
86.85
Commonsense
23.09
41.57
78.76
SWAP
Grammar
45.67
80.46
87.02
Commonsense
25.24
50.20
87.24
NEGATE/ ADD
ARO+ CREPE
SugarCrepe (No Refinement)
SugarCrepe
Hard Negative
Shuffle
Replace
Replace
Replace
Swap
Swap
Swap
Negate
Add
Add
005
0.0
1.0
-0.5
0.0
1.0
-0.5
0.5
1.0
Vera Score Gap (Positive
Hard Negative)
Hard Negative
Shuffle
Replace
Replace
Replace
Swap
Swap
Swap
Negate
Add
Add
-0.5
0.0
-1.0
-0.5
1.0
-1.0
-0.5
0.5
1.0
Grammar Score Gap (Positive Hard Negative)
Figure 4: We compare the Vera (top row) and Grammar (bottom row) between ARO+CREPE (leftmost column), SUGARCREPE without adversarial refinement (middle), and SUGARCREPE (rightmost). Top row: We see that Vera score gap distribution shifts from the positive spectrum to more centered around zero from ARO+CREPE to SUGARCREPE without refinement. After adversarial refinement, zero on SUGARCREPE. Bottom row: Similarly, from ARO+CREPE to SUGARCREPE, we see the Grammar score gap distribution shifts from the positive spectrum centered around zero.
gap distributions
ensure the score gap distribution is centered around
score gap distributions the final SUGARCREPE evaluation set are symmetric around zero. This implies that the previously identified artifacts can no longer be exploited to infer the positive texts. As result, show that the previous commonsense and grammar attacks that are extremely successful on existing benchmarks do not work on SUGARCREPE. As shown in Table 6, these blind models now consistently rank the last SUGARCREPE as compared to other pretrained CLIP models.
5.2 Re-evaluating recent methods for improving compositionality
Given the vulnerability of existing compositionality benchmarks, it is unclear whether recently proposed methods that show state-of-the-ar performances on these benchmarks are indeed effective. Thus, we re-evaluate these methods with SUGARCREPE.
Hard negative augmented training. Specifically, we focus on evaluating one common data- augmentation approach considered [49, 13], where the core idea is explicitly create hard negatives and train the model to distinguish them. We broadly refer to this training scheme as NEGCLIP following [49]. We evaluate two NEGCLIP training schemes: finetuning and training from scratch. For finetuning, in addition to taking the model released in [49], we finetune another three NEGCLIP models (using ViT-B/32 following [49]) with three respective types of hard negatives (i.e., REPLACE, SWAP, NEGAtE) generated using CREPE's [30] source code. For training from scratch, we use RN50 as the base model and train variants of NEGCLIP by augmenting the training examples with different types of hard negatives. We perform both training and finetuning COCO [26].

---

Improvements are overestimated due to unintentionally overfitting. In Table 5, we first see that NEGCLIP finetuned models show significant improvements on ARO+CREPE, boosting the performance more than 10% compared to standard CLIP finetuning on 11 out of 16 cases (high- lighted in green). The lifts are especially large when the hard negative type used in finetuning matches that used in evaluation, where NEGCLIP finetuned models can achieve near human-level performances. For instance, by finetuning with REPLACE hard negatives, NEGCLIP reaches 94% on ARO+CREPE evaluated with REPLACE hard negatives (human performance is 95%). While the results on ARO+CREPE suggest that NEGCLIP seemingly sufficient in equipping models however that the improvements brought by NEGCLIP much smaller on SUGARCREPE. In fact, none of the improvements on SUGARCREPE is larger than 10%, and the best performing NEGCLIP finetuned models still have large gaps to human-level performances, 6.g., best NEGCLIP model lags behind human by 23% on SUGAR CREPE'S SWAP hard negatives. Similarly, when trained from scratch, we observe the same trend that NRGCLIP's improvements are much larger on ARO+CREPE than on SUGARCREPE. The improvements on ARO+CREPE are again most pronounced when the training and testing hard negative type matches.
with strong compositionality,
We attribute the stark contrast in NEGCLIP's effectiveness on ARO+CREPE and SUGARCREPE to model's s unintentional overfitting: The NEGCLIP models learned to exploit artifacts that can be used to easily distinguish hard negatives from positives on ARO+CREPE, instead of actually improving compositionality. Thus, when evaluated on SUGARCREPE where the artifacts are removed, the improvement from NEGCLIP drastically reduces. These results imply that NEGCLIP's effectiveness overestimated existing benchmarks, and we may still need further innovations to fundamentally improve a model's compositionality.
Table 5: Re-evaluating hard negative augmented training shows that the method's s improvements on existing benchmarks (ARO- CREPP))) are hugely overestimated, particularly when the test hard negative type matches the one used in training, which can be attributed to overfitting the artifacts.
Color notations:
Gains compared to standard CLIP (finetuned / from scratch) > 10%
ARO+CREPE
SUGARCREPE
Model
Training
Hard Negative Used
REPLACE
SWAp
NEGATE
SHUFFLE
REPLACE
SWAP
ADD
Human
95.33
100
99.33
96.00
98.67
99.50
99.00
Pretraincd
N/A
71.58
76.89
80.76
63.27
75.09
CLIP finctuned
N/A
77.06
68.81
61.19
63.04
84.76
70.83
85.58 58
VIT-B/32
REPLACE
94.51
90.04
85.06
88.15
88.27
74.89
90.16
NEGCLIP finetuned
SWAP
82.88
94.48
77.57
87.00
85 55.54
76.21
86.56
NEGATE
71.24
68.91
99.54
64.28
84.97
70.29
85 85.84
Released in [49]
85.72
94.35
83.51
90.45
85.36
75.33
87.29
CLIP scratch
$9.96
55.36
69.54
RN50
REPLACE ACP
66.51
60.90
NEGCLIP from scratch
SWAP
72.33
92.29
64.51
73.31
71.93
NEGATE
60.29
99.45
72.74
Sw NEG
86.30
88.60
99.34
75.26
67.69
5.3
Comprehensive evaluations on existing pretrained vision-language models
We present four key findings in our evaluation over 17 pretrained CLIP models on SUGARCREPE, with results reported in Table and visualized in Figure 5.
The best pretrained CLIP models demonstrate some compositional understanding but still have overall large rooms for improvements. Table 6 shows that the largest pretrained CLIP models, e.g., OpenAl's RN50x64, LAION's xlm-roberta-laagee Siaarge-VTT---HLL ViT-H-14, and DataComp's ViT- L-14, achieve near-human performance on REPLACE- OBJ. However, on REPLACE-OBA. smaller models pretrained on small datasets still suffer from big drops in performance - 23% and 43% respectively for DataComp's 's small and medium models compared to humans. Additionally, on nearly all other hard negative types, there are clear gaps (larger than 10%) between the best model performances and human performances, showing an overall large room for improvements in current models' compositionality.
'I Appendix D.3, we provide further results on training NEGCLIP with hard negatives filtered with our adversarial refinement mechanism.
8

---

Table 6: Our evaluation of pretrained CLIP models on SUGARCREPE shows that they demonstrate compositionality on some hard negatives but are far from human performance on others, especially on SWAP hard negatives or ones perturbing attributes and relations (also illustrated in Figure 5: lower overall performance on SWAP, and lower performances on altributes/relations compared to objects). We additionally evaluate recently introduced GPT-4V [33]. While it demonstrates strong results, there is still gap to human-level performance.
REPLACE
SWAP
AnD
Onjectt
Objject
Attribute
Objjcct
Average
Text only model
Granmaa [31]
Yen [27]
50.00
49.39
+9.52
50.00
50.00
49.35
50.00
10.19
50.00
40.40
49.42
50.00
50.00
49.57
$0,00
09.42
KN50
1002
41.37
$0.55
69.99
61
08.41
74.56
09.05
21,83
RN101
170
00.49
63.08
67.07
56
75.46
70.00
71.06
VIT 32
151
98.92
90.08
09.20
61.35
63.95
17.21
68,79
73.00
OpenAl [76)
VIT 32 negelip
4003
151
92.68
83.91
76.45
75.3 20
75.35
58.50
82.50
82.46
RN9054
175
92.08
62,94
57.57
05 04
03.35
79.34
70.04
74.64
R6
201
03.46
82.11
69.20
63.01
65.77
60.70
75.87
75.73
VIT 14
428
34.07
79.19
65.15
60.16
62.31
73.32
71.53
72.96
RN
635
09.49
63.50
70.63
83.37
73.99
26.33
roberta-Vi --2-
212
92.
64.90
72.81
00.00
11MS
K7.16
79.91
78.78
VIT---14
92.05
81.64
g-1
95.76
72,80
91.31
52.08
90.14
96.57
81.91
366
63.41
slmveddottb-aggg--i VIT-----
86.13
32
133
151
56,90
60.35
54.43
DatsCoomp
Kodium:
large: VIT-B-16
16
151
150
92.68
79.82
63.94
56.10
57.66
64.34
64.38
78.61
64.37
73.31
VIT-L-14
13B
428
95.52
84.52
69.99
65.04
66.82
91.03
$4.97
79.70
OpenAA
[S]
96.31
SugarCrepe Recall81
Hard Negative Type Replace
Hard Negative Type Swap
Hard Negative Type Add
Fine-Grained Type
1.00
0.97 onf
Attribute
Object
0.88
Rolation
0.75
0.
0.81
0.50
0.9
0.78
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
0.00
0.25
0.50
0.75
ImageNet Zero-Shot Top-1 Acc.
ImageNet Zero-Shot Top-1 Acc.
ImageNet Zero-Shot Top-1 Acc.
Figure 5: We plot pretrained vasion-language models' zero-shot top-1 accuracy on ImageNet versus their retrieval recall@1 1 on SUGARCREPE, where is the Pearson correlation coefficient. This plot suggests that models' ImageNet zero-shot accuracy positively correlates with their compositionality.
All models struggle at identifying SWAP hard negatives, regardless of their pertaining dataset and model size. Among the three types of hard negatives, SWAP hard negatives present the biggest challenge to the pretrained CLIP models, even though humans can easily tell them apart from the positive captions. We observe in Table 6 that all models demonstrate low performance on both SWAP-OBJ and SWAP-ATT hard negatives regardless of their pretraining dataset and model sizes, with the difference from human performance reaching from 27% to 50%.
Existing models are object-centric, struggling compose aftributes and relations. We find that existing pretrained models are lot better at composing objects than attributes or relations (Table 6). This finding holds for both REPLACE and ADD hard negatives but not the most difficult SWAP negatives, where models perform equally poorly on both SWAP-OBJ and SWAP-ATT. On REPLACE hard negatives, even though most models achieve human-level performance on REPLACE-OFE -OBI, they all suffer from drop in performance on REPLACE-ALE ATT and REPLACE-RELE where the drop is as large as 15% and 29% respectively. Similarly, on ADD hard negatives, all models except for DataComp's 's vialI-ViT-B- VIT-B-32 experience a decrease in performance from ADD-OBJ to ADD-ATT, with the largest difference reaching 10%.
Models' performance on SUGARCREPE correlates with their ImageNet zero-shot accuracy. We show in Figure 5 that there a positive correlation between models' performance on SUGARCREPE and their zero-shot accuracy on ImageNet. This correlation is moderate on SWAP-OBJ and ADD-ATT (Pearson correlation coefficient 0.78 and r = 0.75 respectively) and strong on all other hard negatives (r > 0.8).

---

Discussions
Our investigation reveals significant biases present in existing benchmarks for the compositional comprehension capability of vision- rision-language models. The severity of this vulnerability is exemplified by text-only models without access to the image outperforming rision-language models. To address this, we introduce SUGARCREPE, a novel benchmark for evaluating the compositionality of vision- language understanding. Unlike previous benchmarks that relied on rule-based templates, we leverage large language models to generate less biased negatíves and employ adversarial filterigg mechanisms to minimize biases. Through reassessment of state-of-the-art models and recently proposed compositionality inducing mechanisms, we uncover significant overestimation of their advancements, underscoring the need for further innovation.
6.1 Limitation and future work
Scope of the compositionality benchmarks and vision-language models. We focus our scope on compositionality benchmarks formulated as image-to-text retrieval task. While this is currently the most prevailing evaluation framework, future research can characterize compositionality evaluation as text-to-image retrieval problem, as in the initial efforts considered by [38, 45]. More importantly, we hope our work can guide future efforts in creating and ensuring faithful compositionality benchmarks in text-to toxt-to-image form. In addition, we focus our evaluations on contrastively learned vision-language models [36]. Future work should include and characterize the compositionality of modern generative lainu-la models [1, 7, 24, 46].
Potential biases imposed by language models. In this work, we identify two human interpretable dataset biases, the nonsensical and non-fluent biases, which may not cover all dataset artifacts that could possibly be exploited by a model. By leveraging ChatGPT in generating hard negatives, the generated captions may also exhibit hard to detect biases imposed by the language model, e.g., watermarks [21]. Future work may utilize more sophisticated adversarial filtering techniques that train models to detect and remove spurious dataset artifacts beyond human comprehension [51, 23).
Shifts in language model behavior. Our work leverages ChatGPT to generate hard negatives. However, recent work has pointed out that the underlying model behind these APIs may change, resulting in model behavior shifts [6, 28]. We discuss how this potential model behavior shift may affect our proposed dataset construction pipeline. Specifically, while there may be variances on the quality of the generated texts, we note that our employed adversarial refinement mechanism can ensure that the final evaluation set is free of the identified artifacts. In the case when ChatGPT improves and generates sighlttquality captions, the refinement mechanism will filter out less examples and we can more efficiently create the final evaluation set. On the other hand, if ChatGPT degrades and shifts towards generating less fluent and plausible captions, the refinement mechanism will filter out more generated examples and we would need to generate more candidates in order to create an evaluation set of the same desired size. As result, while the efficiency of the proposed dataset construction pipeline depends on quality of the language model used, our pipeline ensures the generated set does not contain the identified biases. In the large language model era, see these capable models as productive tools one can leverage to efficiently process and create data. We do however deem careful manual and automatic filtering technique, necessary to ensure
validation mechanisms, such that the ultimate goal is properly achieved.
6.2 Societal impact
As vision-language models such as CLIP [36] are becoming the foundation models for many down- stream applications [40, 37J, it is imperative to understand the limitations of these models to avoid misuses and undesirable outcomes [8. 2). Compositionalityy benchmarks probe model's understand- ing of finer- grer-grained concepts, and hence allow us to identify blind spots [49, 53, 30] of seemingly
powerful models deemed by standard classification and retrieval benchmarks [11, 26]. Our work further alleviates common artifacts in existing compositionality benchmarks that result in overestima- tion of model's capability. We hope our proposed benchmark SUGARCREEP leads to more faithful assessment of vision- lisguanlan model's 's compositionality. and can hence guide more accurate usages of the models. Nevertheless, we note that strong performances on SUGARCREPE do not imply perfect models. We envision SUGARCREPP being one of the many benchmarks used to comprehensively understand the abilities of vision-language models from various aspects.
10

---

References
[1]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning, Advances in Neural Information Processing Systems, 35:23716-23736. 2022.
[2] Federico Bianchi, Pratyusha Kalluri, Esin Durmus, Faisal Ladhak, Myra Cheng. Debora Nozza, Tatsunori Hashimoto, Dan Jurafsky, James Zou, and Aylin Caliskan. Easily acces- sible text-to- text-to-image generation amplifies demographic stereotypes at large scale. arXiv preprint arXiv:2211.03759 2022.
[3] Léon Bottou. From machine learning to machine reasoning. Machine learning, 94(2):133-14, 2014.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020.
[5] Paola Cascante- Bonnilaa, Khaled Shehada, James Scale Smith, Sivan Doveh, Donghyun Kim, Rameswar Panda, Gül Varol, Aude Oliva, Vicente Ordonez, Rogerio Feris, and Leonid Karlinsky. Going beyond nouns with vision & language models using synthetic data, 2023.
[6] Lingjiao Chen, Matei Zaharia, and James Zou, How is chatgpt's behavior changing over time? arXiv preprint arXiv:2307.09000 2023.
[7] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.0677 06794, 2022.
[8] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of ext-to--mmaee generative transformers. arXiv preprint arXiv:2202.04053 2022.
[9] Noam Chomsky and Morris Halle. Some controversial questions in phonological theory. Journal of linguistics, 1(2:997-33,, 1965.
[10] MJ Cresswell. Logics and languages. 1973.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[12] Anuj Diwan, Layne Berry, Eunsol Choi, David Harwath, and Kyle Mahowald. Why is winoground hard? investigating failures in visuolinguistic compositionality. arXiv preprins arXiv:2211.0076 2022.
[13] Sivan Doveh, Assaf Arbelle, Sivan Harary, Eli Schwartz, Roei Herzig, Raja Giryes, Rogerio Feris, Rameswar Panda, Shimon Ullman, and Leonid Karlinsky. Teaching structured vision & language concepts to vision & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2657-2668. 2023.
[14] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.14100 2023.
[15] Madeleine Grunde-MeLnn McLaughlin, Ranjay Krishna, and Maneesh Agrawala. Agqa: benchmark for compositional Computer Vision and Pattern Recognition, 2021.
reasoning. In Proceedings of the IEEE/CVF Conference on
[16] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah Smith. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02333 023224, 2018.

---

[17] Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795,
2020.
[18] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.
[19] Theo MV Janssen and Barbara H Partee. Compositionality, In Hlandbook of logic and language, pages 417-473. Elsevier, 1997.
[20] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10236-10247, 2020.
[21] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301.10226 2023.
[22] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017.
[23] Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. Adversarial filters of dataset biases. In International Conference on Machine Learning, pages 1078-1088. PMLR, 2020.
[24] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2:
bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023.
[25] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language- image pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu. and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 12888- 12900. PMLR, 2022.
[26] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[27] Jiacheng Liu, Wenya Wang, Dianzhuo Wang. Noah A Smith, Yejin Choi. and Hannanch general-purpose
Hajishirzi. Vera: A general- purpose plausibility estimation model for commonsense statements. arXiv preprint arXiv:2305.00 03695, 2023.
(28) Yang Liu, Yuanshun Yao, Jean-Francoir Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy 1lms: a survey and guideline for evaluating large language models' alignment. arXiv preprint arX5::2308.05374. 2023.
[29] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In Computer Vision-ECCY 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 852-869. Springer, 2016.
[30] Zixian Ma, Jerry Hong, Mustafa Omer Gul, Mona Gandhi, Irena Gao, and Ranjay Krishna. Crepe: Can vasion-language foundation models reason compositionally? arXiv preprint arXiv:2212.07796. 2022.
[31] John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nip. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119-126, 2020.
12

---

[32] OpenAl. Chatgpt. 2022.
[33] OpenAl. Gpt-4v(ision) system card. 2023.
[34] Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[35] Mahima Pushkamn,, Andrew Zaldivar, and Oddur Kjartansson. Data cards: Purposeful and transparent dataset documentation for responsible ai. 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1776-1826, 2022.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages $748-8763 PMLR, 2021.
[37] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint urXiv:2204.06/2 2022.
[38] Arijit Ray, Filip Radenovic, Abhimanyu Dubey, Bryan A. Plummer, Ranjay Krishna, and Kate Saenko. Cola: How to adapt vision-language models to compose objects localized with attributes?, 2023.
[39] Yuval Reif and Roy Schwartz. Fighting bias with bias: Promoting model robustness by amplifying dataset biases. arXiv preprint arXiv:2305. 18917, 2023.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[41] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 64(9):99-106, 2021.
[42] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[43] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 15638-15650, 2022.
[44] Harman Singh, Pengchuan Zhang, Qifan Wang, Mengiiao Wang, Wenhan Xiong, Jingfei Du, and Yu Chen. Coarse-to-fine contrastive learning image-text-graph space for improved vision-language compositionality. 2023.
[45] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 5238-5248, 2022.
[46] Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. arXiv preprint arXiv:2306.00 07915, 2023.
13

---

[47] Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo. Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Yuan. Omnivl: One foundation model for image-language and video-language tasks. arXiv preprint urXin:2209.07526, 2022.
[48] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442. 2022.
[49] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou, When and why vision-language models behave like bags-of-words, and what do about it? In International Conference on Learning Representations, 2023.
[50] Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv: 05326, 2018.
[51] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.0788 2019.
[52] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18123-18133, 2022.
[53] Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects. attributes and relations, arXiv preprint arXh:22207.00221 2022.
14

---

A Implementation details
A.1 Hardware information
All experiments are run on a machine with an Intel(R) Xeon(R) CPU E5-2678 v3 with a 512G memory and two 48G NVIDIA RTX A6000 GPUs.
A.2 Dataset sources
We obtain existing datasets from their original sources released by the authors. refer readers to these sources for the dataset licenses. To the best of our knowledge, the data we use does not contain personally identifiable information or offensive content.
CREPE [30]: We obtain CREPE dataset from its official repository $
ARO [49]: We obtain ARO dataset from its official repository 6
VL-CheckList [53]: We obtain VL-CheckList dataset from its official repository 7,
COCO [26]: We obtain COCO from its official project website
A.3 Software configuration
Models. We detail the sources of the pretrained models we use the paper, and the hyper-parameterss used in training own models.
Vera model [27]] We obtain pretrained Vera model released by its author 9.
Grammar model [31]: We obtain the Grammar model released by the authors 10,
All pretrained CLIP models: We obtain all pretrained CLIP models' weights from Open- CLIP
We obtain weights for pretrained NEGCLIP released by the authors 12 For training from scratch and finetuning, we train RN50 and ViT-B/32 based on OpenCLIP codebase and apper-parameterss as the following: number of warmup steps is 1000, batch size is 256, learning rate is le-4, weight decay is 0.1, number of epochs is 30. We augment the original CLIP loss with hard negative captions following NEGCLIP [49].
NEGCLIP models:
Evaluations. We base our evaluation framework on OpenCLIP [18]. We follow all default hyper- parameters used for evaluating models.
B Vision-language compositionality benchmarks
We provide an overview of existing vision-language compositionality benchmarks below, with Table 7 summarizing the dataset comparisons.
B.1 Image-to-text formulation
A majority of current benchmarks formulate the evaluation task as image-to-text retrieval problem. These benchmarks generate hard negative texts procedurally through rule-based templates, where each benchmark considers different types of hard negatives.
VL-Checklist [53]. VL-CheckList aims at evaluating rision-languee models' understanding of different objects, attributes, and relationships. contains REPLACE hard negatives generated by
htttpe///gthab..cc////lllbbbbbb httt::://ggthaab...mmm/rrrr///// language waw.-age-momlelleeeeeeeeeee models larlls-ree arr--rreooo bows
http:p///.gggttab...co///maaaaaa Checklist
hhcrps///coodaaasee..rr//
http:////uggggggeee.c//////etttr wwwe--accaeed-ooo CoLA
models- bows
15

---

Table 7: Summary on vision- languaee compositionality benchmarks. SUGARCREPE considers image-to-tex formulation to enable larger scale evaluation set. In addition, SUGARCREPE considers a wide range of hard negative types. SHUFFLE and NEGATE are omitted as they introduce inevitable biases discussed in Sec. 4.2.
Hard Negative Text Type
Benchmark
Task Formulation
Scale
SHUFFLE
REPLACE
SWAP NEGATE
ADD
VL-CCeckLList [53]
Image-to-Teas Text
1000
ARO [49]
Image-to-
> 1000
CREPE [30)
Image-to-Text
> 1000
/
Winoground [45]
Image-lo- Tengee-o-TTex/-e TTexx-to--maagee 400
Cola [38]
Text-to-Image
210
N/A
SUGARCREPE
Image-to- Text
> 1000
replacing atomic parts of the positive texts with other foils. VL-CheckList further breaks the hard negatives down into more granular categories based on the type of the replaced atomic part, i.e., object, attribute, relationship.
ARO [49]. ARO focuses on models' understanding of different relationships, attributes, and order information. It considers SWAP and SHUFFLE hard negatives. SWAP hard negatives are generated by swapping two words in the positive texts; on the other hand, SHUFFLE hard negatives are generated by shuffling words in the positive texts. ARO further divides SWAP hard negatives into attribute or relationship type.
CREPE [30]. CREPE is large-scale evaluation benchmark that includes three types of hard negatives: REPLACE, SWAP and NEGATE. REPLACE and SWAP hard negatives are generated as in VL-CheckList and ARO. In addition, NEGATE hard negatives are generated by adding negation keywords (i.e., not or no) to the original positive texts. The hard negatives are not further divided into fine-grained types (object, attribute, or relations).
B.2 Text-to-image formulation
Complementary to image-to-text formulation, compositionality can as well be evaluated by probing model to select image that best matches given text description, against other hard negative images as distractors. Unlike hard negative texts, hard negative images are more difficult to obtain and thus current text-to-image compositionality benchmarks smaller at scale.
Winoground [45]. Winoground is small dataset manually curated by human annotators. Each example in the dataset contains two images and two matching captions, where both captions contain identical words that appear in different orders. Note that Winoground be used for either image- to-text or text-to-image retrieval. While the original intention for Winoground is to evaluate vision- language compositionality. recent work [12] has pointed out that solving the tasks in Winoground requires not just compositional lasion-language understanding, but additionally a suite of other abilities such commonsense reasoning, distinguishing visually difficult images.
Cola [38]. Cola tests vision-language model's ability to select an image that correctly matches given caption, against another distractor image with the same objects and attributes but the wrong composition. The image pairs mined from existing datasets. As result, the final evaluation set is relatively small in size (210 examples in total).
We deem text-to-image evaluation as important image-to-text evaluation. Future work can explore approaches to generate or mine compositional hard negative images at scale, as preliminarily explored in [38, 49].
16

---

C SUGARCREPE
C.1
Taxonomy
Figure 6 shows the taxonomy of SUGARCREPE. We first categorize the hard negatives based on their forms: REPLACE, SWAP, and ADD. We then further divide each type of hard negatives into finer-grained sub-categories based on the type (object, attribute, or relation) of the atomic concept altered. SUGARCREPE covers a total of 7 fine-graind hard negative types.
SUGARCHEPE
REPLACE
SWAP
ADD
REPLACE-OBU
REPLACE-AFT
REPLACE-REL
SWAP-OU OBJ
SWAP-AIT
ADD-OBa
ADD-AIT
Figure 6: Taxonomy of hard negatives considered in SUGARCREPE.
C.2 Hard negative generation procedure and templates
To generate hard negatives in SUGARCREPF, come up with three different prompt templates for the three hard negative types considered: REPLACE, SWAP, and ADD. Each template consists of task instruction for generating the corresponding type of hard negatives and several (7 more) few-shot demonstrations. We describe the general generation procedure and example prompt templates below and refer readers to our dataset repository for the full prompts used 13
Generating REPLACE hard negatives. To best leverage ChatGPT's capabilities, we devise a three- step workflow generate REPLACE hard negatives: (1) We prompt ChatGPT in locating the desired atomic concepts (e.g., objects) in the sentence; (2) We prompt ChatGPT to generate a new concept to replace randomly selected old concept: (3) We let ChatGPT compose new sentence by replacing the old concept with the new one. For steps (1) and (3), we prompt CharGPT with a temperature of 0.0 to get stable outputs. For step (2), however, we diversify the outputs by prompting ChatGPT with a higher temperature of 1.5. With this design, we are able to generate diverse REPLACE hard negatives. Figure shows the example templates and outputs for REPLACE hard negatives.
Generating SWAP hard negatives. To generate SWAP hard negatives, which do not require any new concepts, we simply prompt ChatGPT once with a temperature of 0.0. Unlike REPLACE, SWAP hard negatives are only possible when there are at least two atomic concepts of the same category, i.e., either object or attribute. Thus, our prompt first queries ChatGPT whether it is possible to swap two atomic concepts in the input sentence to generate a new description. Only if the answer is yes, will ChatGPT then proceed identify two swappable concepts and compose the corresponding new sentence by swapping the two concepts. Figure 8 shows the example templates and outputs for SWAP hard negatives.
Generating ADD hard negatives. Similar to the REPLACE, we also employ a three-step prompting procedure generate ADD hard negatives. The only difference in the procedure is that we prompt ChatGPT to add the generated new concept to the original caption, instead of using it to replace an old concept. Figure 9 shows the example templates and outputs for ADD hard negatives.
C.3 Adversarial refinement
We detail the adversarial refinement procedure below. Given a text model M, we denote its output score for the positive and negative caption of i-th image as (p:) and (n,).. If M > M (n.). then the model could identify the correct caption for the i-th image without referring to it. For test set to be unattackable given the text model M, the expectation of M's identifying the correct caption should be as close to random guess as possible; in particular, we hope that [M (p:) > M (n:)] = 0.5. To achieve this for both the grammar model Mj and plausibility model M2, we first calculate the score
13 Phttps:/ :/gg:::///ttttt htt.aa/.........aaaa/aaaaaa----a
17

---

cives in tepet sescence dasoribing sce0s, your caKE
dives in teput sastence dascribing scene, your ta22
to:
cbjecte
Gooase the nous sarda is the centesos.
if there
adjective werda,
2. Randenly pice one neus vord.
exply Hit.
3. heplace the selested soon word wint a new nout word
2. Fandoaly plak one adjeotive word.
5. Peplace the selected adjersive
The seu sentence sust nees she fol bowing three requiieeental
sau
most rees the fol towting three
dierrerett paseible from the origiiaa The sustence be floent crumactioaly corsect.
nie nestance wont be desoribing come that difturent possible tron the origine;
2. Zae now sestence sest be flaest and goumattionll correct.
3. Ihe nuN austance eart make Lopinel sunse.
Burv cuumple:::
Buze use sces uumples:
original sentence: san is in kitcter naking pizzza. selected noun:
"piszas")
Adjections: ["bias"]
Sav zoin: weasn
selecced adjective: blue
Sav seccence: NOEan ds kitchen making pizzas.
Sov adjestive: red
Original aentence:
meated oz vall and hirds
besiiess fer
Original centence: the kischen is olean and ready for us set.
Rounsi "kooan" ( Thieds!)
Adjenstves
"neady")
bench
del adjenttoos olean
Ree sensenne: soeen dwaeed on a bench and birde
tee adjecstons dirny
bactdee her.
fou sensennes The kischen 16 diviy and ready for to
(a) REPLACE-OBI.
(b) REPLACE-ATT
Gives liput sestence describing acans, your taok
Pind aetter spactal relationehipe betsnee
cbjects in the sectence. If there are no zuch
ralationthipe, return aepty 15st.
deplace the selected relationship with relationship make now centerce.
sev sentence nust nees following
1. The new sessence wiss be biigg soene that in as derent an prosible from the origital scese.
graeati
3. Ehe auscence make Jopical
Buce uce scea cxappees:
Ceiginal suntence: The dining table the kitchan bes a bost oz deutt on it.
kalationthipe: ["eaes", "oa"]
relationehig: for feow
Rtc sectence: The dining cable for from the littctes har bevl of frats ez 58.
couple
shite
Sel antinnhige: [-in"]
telected rolat Senthip1
relationekig actelde
bockets ousside shite
(c) REPLACE-REIL
Figure 7: Example prompt templates (black) and outputs (green) from ChatGPT for REPLACE hard negatives.
difference
g. (1) M, M, (n.) and of M2(p.) - M2(n.), where the range of both g()) and
(22) is [-1, Then split the 2D space of the joint range of (1) and and g into 100 100 equal grids, E (-0.04, 0.06]} and (()(1) ( (( (-0.02, -0.04), (2) (0.04, }. we preserve the same number of data for both grids, therefore we ensure that for the resultant set, (P. > Mn+)) = 0.5 and
and for each pair of symmetric grids, e.g. (1) (1) E 0.04).
E, (P.) > M2(n.)] = 0.5.
C.4 Dataset construction cost
We provide a high-level overview to the cost used to build SUGARCREPE by utilizing OpenAl's ChaiGPT API for generating hard negatives. In building SUGARCREPE, we use approximately 40 API calls to generate hard negatives for each COCO test caption, including all different fine-grained types of hard negatives. This amounts to total of 25, 000 x 40 1, 000, 000 API calls to ChatGPT. With each API call costing around $0.0005, it took roughly $500 to build SUGARCREPE.
18

---

your
Gives ss input venteses describing acene, your tesk te to fires laeate use maggaail noun persese in the the new sentance rvop follewing three requireectts:
deccance desoribing differess oajenss, and then
ssapaable adyent
thes make new contence.
following
L. The new sentence nust describing dicfacent scmee fron the imput sessence.
1. Ehe sustence must be describing ditterent
scons fron this input vistance.
The
fluess gesestiiolly
2. The now sustence must be flount und gromaticully
correct:
3. The now sentenee must nake logical sems.
2. The tastence rake lagical
Assvez question sbether genecating such
you
To copplate the tack, you sbestd:
quention possible 2. Dasput the avapaable ivee
gezarating
suscance La possibie using Tes oe So.
3. nerput the copprable nou phrasee.
3. Bwop tien to make new sertence.
Kare are zone exsuples:
esappies:
Iuput: cesting laptop next a parsus, sussence to ymerete
input: à giel in pink shirt holding biue unbceils. Ie possibls swap attribstes in the input ventescs to generate rentaton and uskee lagtcal concel Yoe swarpable atsributee: plnk, blue cotper: girl in Siee shirt holding a pisk iabrella.
snsp sestunce that different
tcom the unput sentesee and eates logical senset Yeu
swagpable noun phranss: laptop, paraon
cutpes:
lspsop.
Input: A plate of donata with parazn in the bankground.
with blue tootstruush
beushing
posaiale seep phrases
that diffarent
different Ezom. the blue cutpec: A pizk with bise abirt brushing her seats with grees testhbruab.
to it possiale to seap attribstes in the ingut sentesee to genezate new aestence
From the regur. centerne and eales loglnal aoneet Yos sesppable nown parsnee: i plate donoti, parsor Cutpec: parson with plate of donuta the backgsound.
sentesca
svappable atseibutes:
(a) SWAP-OBI.
(b) SWAP-ATT.
Figure 8: Example prompt templates (black) and outputs (green) from ChatGPT for SWAP hard negatives.
ligut describing your
Gives an Input seccence descriting scene, your task
Find the objacts in the suntesce.
objects
Bandunly pica dbject.
1. donazate a bew object that's is sot in the sectance,
4. kdd the new object next to the selected cbject to
Randanly piee sbject.
4. Add the new actrihate next the selected abject to nate now sectence.
this abject that's not contance.
The sou centeine russ ween sit fall louting thres requireeental
the sav zentence requizeenta::
rast follexing three
slearly
The nox sectence rest describe cleariy ane and diffecent scese.
3. Yhe nnd sentence went be Floeet and gnalratinally correot.
diffarent
be 3. The new aestence wont wake logloa) eense.
goams
correct.
3. The BEN ausbence mast make Jopizel sense.
Burv ucu complie::
exmpless
Original sentence: An elephast stunting under the shade
Criginel suntence: lorge waite aispleze and pecaco en lsc.
Sov abjess: aquireel
0bjects:
telecsed jeject: eleptant
"nelephuss", "shode treet)
slsphant equiezel atanding
Suv auscence: largu white and hiue sisplane und
Sov abttebute: siae
Solucted abject: sieplane
shude
Original sentence: bench at the beach next to the zua
beach
Original centence: three people eiding borass on
Sov abject: unbeulla
colected ject: bench
Selected object: people sew astrlbutee eldecly Bow secsence) 7aree olderly people riding horses or
people', "borsen', "baack"]
teu sanienne: In tatbirlla and a beoch, the besob next
(a) ADD-OBJ.
(b) ADD-ATT.
Figure 9: Example prompt templates (black) and outputs (green) from ChatGPT for ADD hard negatives.
C.5 Dataset information
We host SUGARCREPE on Github 14 The data card [35] for SUGARCREPE, containing detailed dataset documentation, is available at the dataset repository 15, We provide a summary below.
Dataset documentation. SUGARCREPE is a benchmark for faithful vision- vision-language compositionality evaluation. Given an image, model is required to select the positive text that correctly describes the image, against another hard negative text distractor that differs from the positive text only by small compositional changes. Each example consists of three fields:
14 whetpe:://github.com///tt wtggghhhwb.ccc//..//laaaaaagaaae crepe
15 http://ghhhhubooo///l/llll//agga
.dff
19

---

f filename:
The id to an image
caption: Positive text correctly describing the image
negative_caation: Hard negative text incorrectly describing the image
Maintenance plan. We are committed to maintain the dataset to address any technical issues. We actively monitor issues in the repository.
Licensing. We license our work using MIT License 16 All the source data we use is publicly released by prior work [26].
Author statement. We the authors will bear all responsibility in case of violation rights.
D Detailed evaluation results
D.1 Full evaluation results existing benchmarks
We provide the full evaluation results over 17 pretrained CLIP models as well as 2 text-only models, Vera [27] and the Grammar model [31]. existing compositionality benchmarks in Table 8. We see that the text-only models, arguably without any vision-languae compositionality, outperform most of the pretrained CLIP models, achieving state-of-the-an performances many benchmark tasks. This implies that current benchmarks fail to faithfully reflect a model's 's vision-language compositionality.
Table Blind models (i.e., Vera and Grammar model) outperform all 17 existing pretrained CLIP models nearly all existing benchmark tasks. This implies that current benchmarks fail to faithfully measure a model's vingon-language compositionality.
YL-Checklst
Source
Medd
Atomic
Seap
Nesste
VG XG-Relation
YG-Attiibtii
COCO-Oricr
Flicke30k-Odder
Object
Altrilttt
Relation
Tot-oale mode)
1771
66.11
59
44.39
06.35
26.24
25.22
0.51
$1.59
63.27
20.39
34.55
$7.25
05.24
73.81
Opeat)(3)
VIT-5-11 11
01.30
26.55
75.78
11.13
01.35
37.
17 18
$7.00
08.00
77 (1
23.05
0.38
$2.13
30-25
34.76
05
09.33
76
V1
36.35
25.00
34224
15.00
VI-41-14
23.20
23.22
15.34
50.35
0235
23.79
30.05
$3.10
05.95
71.15
23.70
34.76
30.30
$1.00
61.30
25.59
30.10
56.17
09.03
23.65
31.34
30.31
47.46
15.11
stm-mbens howe BIT-B-33
14
00.90
13.76
47:53
19.35
33.55
30.32
56
70.41
63:01
sias-edens lage VI 11-14
24.15
23.99
2005
46.14
57.84
2605
31.00
$7.59
70.25
63.80
manDCop (14)
Jargo VIT.B-1
18.15
20.35
17.36
11.86
50.45
15.13
53.31
21.49
76.14
54.73
65.73
66.81
x-1 s-gergel L-14
21.03
22.57
16.35
48.51
60.3
23.99
39.33
86.06
67.00
67.90
D.2 SUGARCREPE human evaluation
To compare the quality of the hard negatives generated in SUGARCREPE to those in current bench- marks (i.e., ARO+CREPE), we randomly sample 100 examples for each the hard negative types: REPLACE, SWAP, and NEGATE/ / ADD. Fach example is organized to consist of (1) the original posi- tive text, (2) its hard negative ARO+CREPE, and (3) its hard negative in SUGAR SUGARCREPE, For each example, human user rates whether the hard negative in ARO+CREPE or that in SUGARCREPE better (or tie) in terms of commonsense and grammatical correctness, respectively. Note that we compare NEGATE in ARO+CREPE to ADD in SUGARCREPE, probe a model's understanding of the existence or not of an atomic concept. Table shows that hard negatives in SUGARCREPE are much more sensical and fluent than that in ARO+CREPE across all three different types. For instance, SUGARCREPP has 68% more sensical and 46% more fluent hard negatives than ARO+CREPE on SWAP.
both hard negatives are intended to
D.3 Additional NEGCLIP results
In this section, we conduct preliminary experiments to answer whether models' performances on SUGARCREPE would increase hugely if the models are trained with hard negatives generated through
16
20

---

Table 9: Human evaluation results the comparisons between hard negatives in ARO+CREPE and SUGARCREPE. We report the counts (out 100 sampled examples) that the human user considers better or tie, w.r.t. both commonsense and grammatical correctness.
Human counts of better examples
Hard-negative Type
Evaluation
ARO+CREPE
SUGARCREPE
Tic
Commonsense
11
29
60
REPLACE
Grammar
4
33
63
Commonsense
4
68
28
SWAP
Grammar
4
46
50
Commonsense
26
73
NEGATE/ ADD
Grammar
35
64
the same procedure as how we create SUGARCREPE. Since generating hard negatives for training with ChatGPT would incur substantial cost, create hard negatives using proxy method. In particular, we start with camplate-generatee hard negatives on the COCO training set and apply our adversarial refinement technique remove the biases. We use this adversarially refined dataset for NEGCLIP training. We show the results in Table 10. While we observe that the method improves over vanilla CLIP training without hard negatives, it performs similarly to NegCLIP and does not saturate the performance on SugarCrepe. This suggests that while the adversarial refinement mechanism prevents SugarCrepe from being attacked as an evaluation benchmark, leveraging the approach alone for training does not saturate the performance on SugarCrepe, Future work may characterize how LLMs could be used generate better hard negatives for training to genuinely improve vision-language models' compositionality.
Table 10: Model performances SUGARCREPE when trained with hard negatives generated through similar procedure as how SUGARCREPE created.
SUGARCREPE
Model
Hard negative
REPLACE
SWAP
ADD
CLIP without hard negatives
N/A
69.54
60.33
67.63
NEGCLIP with template hard negatives
REPLACE
74.32
62.65
72.92
NEGCLIP with adversarial refined hard negatives
REPLACE
73.37
61.40
72.84
NEGCLIP with template hard negatives
SWAP
73.31
68.35
71.93
NEGCLIP with adversarial refined hard negatives
SWAP
72.07
65.13
69.68
NEGCLIP with template hard negatives
NEGATE
72.74
60.89
70.47
NEGCLIP with adversarial refined hard negatives
NEGATE
72.70
60.75
68.70
21


Reviews:
[{"review_id": "7Jas1oBaBX", "reviewer": "Reviewer_eSt3", "summary": "", "questions": "", "limitations": "The paper has adequately addressed the limitations and potential negative societal impact (Appendix A).", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper is well-motivated by the finding of biases in existing benchmarks (Section 3). Quantitative results of blind models give solid evidence of such biases (Section 3).\n2. The proposed method to construct more plausible and fluent hard negatives is well-suited to address the bias problem.\n3. The evaluation of existing vision-language models (e.g., NegCLIP (Section 5.2) and 17 CLIP models (Section 5.3)) on the new benchmark is extensive and comprehensive. The experiments also show good insights into which specific types of hard negative models suffer more compositionality problems.\n4. The paper is well-written and easy to follow.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Ekiryv85Mr", "reviewer": "Reviewer_iLag", "summary": "", "questions": "", "limitations": "Yes, the author adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- This work successfully reveals what previous visual compositional reasoning benchmarks are tested on: some shortcuts, including grammatical patterns drawn from the template-based negative construction procedure, but not genuinely vision-language compositionality.\n\n- The proposed method of leveraging large language models to generate candidate negatives is intuitive, easy to implement, and works well compared to templated-based approaches.\n\n- Experimental results on CREPE show that the original version of the CREPE dataset has the hard negative problems mentioned. SUGARCREPE is a nice fix to this, meanwhile highlighting the difficulty of compositional visual reasoning.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "0S0rmA9X8S", "reviewer": "Reviewer_6iRd", "summary": "", "questions": "", "limitations": "Large language models (LLMs) are increasingly used to generate annotations. However, this raises a methodological question since the benchmarks used to evaluate these models, in part rely on these models. This raises the question about how sustainable and robust is the use of LLMs to generate evaluation data.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper observes an important limitation of existing compositional benchmarks. The paper is well organized, with empirical evidence motivating the work and convincing empirical results:\n* The paper proposes strong evidence exhibiting the limits of existing evaluation benchmarks, in particular ARO and CREPE. \n* The paper introduces a method to limit the biases in these datasets by generating hard negatives for image-to-text generation benchmark using LLMs. \n* Using this method, the authors can re-evaluate the effectiveness of some approaches for compositional evaluation.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Helz7oTT0Y", "reviewer": "Reviewer_bMrd", "summary": "", "questions": "", "limitations": "Yes, the authors have detailedly addressed the potential limitations of the work in the appendix.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Authors choose to utilize language models to generate hard negatives with an adversarial refinement mechanism to refine synthesis. This approach is innovative and practical.\n2. SUGARCREPE re-evaluates SOTA VLMs to show that the past improvements were overestimated, urging improvements in recently proposed compositionality-inducing strategies.\n3. The paper is well-presented with clear details of prompts and textual operations for the proposed framework.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This work points out that existing compositional vision-language understanding benchmarks are hackable by containing non-plausible and non-fluent biases, which even enables language-only models to achieve state-of-the-art performances. Then the authors introduce SugarCrepe, a new benchmark to evaluate vision-language compositionality. The authors leverage large language models to generate plausible and fluent hard negatives followed by adversarial filtering. By re-evaluating existing methods on the SugarCrepe, the paper finds that existing methods’ compositionality is overestimated when evaluated on previous hackable benchmarks. This work provides a challenging benchmark and interesting findings. Please make sure to add the promised results and related discussion in the final revision.
Decision: Accept (Poster)