SlowGuess/NeurIPS_2023_OCR huggingface dataset:
SUGARCREPE: Fixing Hackable Benchmarks for Vision-Language Compositionality
Cheng-Yu Hsieh<sup>1*</sup>, Jieyu Zhang<sup>1*</sup>, Zixian Ma<sup>1</sup>, Aniruddha Kembhavi<sup>2</sup>, Ranjay Krishna<sup>1,2</sup>
<sup>1</sup>University of Washington, <sup>2</sup>Allen Institute for Artificial Intelligence
{cydhsieh, jieyuz2, zixianma, ranjay}@cs.washington.edu, anik@allenai.org
In the last year alone, a surge of new benchmarks to measure compositional understanding of vision-language models have permeated the machine learning ecosystem. Given an image, these benchmarks probe a model's ability to identify its associated caption amongst a set of compositional distractors. Surprisingly, we find significant biases in all these benchmarks rendering them hackable. This hackability is so dire that blind models with no access to the image outperform state-of-the-art vision-language models. To remedy this rampant vulnerability, we introduce SUGARCREPE, a new benchmark for vision-language compositionality evaluation. We employ large language models, instead of rule-based templates used in previous benchmarks, to generate fluent and sensical hard negatives, and utilize an adversarial refinement mechanism to maximally reduce biases. We re-evaluate state-of-the-art models and recently proposed compositionality inducing strategies, and find that their improvements were hugely overestimated, suggesting that more innovation is needed in this important direction. We release SUGARCREPE and the code for evaluation at: https://github.com/RAIVNLab/sugar-crepe.
Scholars today herald compositionality as a fundamental presupposition characterizing both human perception and linguistic processing [10]. Through compositional reasoning, humans can comprehend new scenes and describe those scenes by composing known atoms [19, 17, 3, 9]. For instance, compositionality allows people to differentiate between a photo of "a girl in white facing a man in black" and "a girl in black facing a man in white". For a while now, vision-language research has sought to develop models that can similarly comprehend scenes and express them through compositional language [22, 20, 29, 15].
Given its importance, a surge of new benchmarks have been proposed to evaluate whether vision-language models exhibit compositionality. Recently, Winoground [45], VL-CheckList [53], ARO [49], CREPE [30], and Cola [38] have entered the machine learning zeitgeist. Evaluation is mostly done through an image-to-text retrieval task formulation [53, 49, 30]: by measuring how often models pick the description, "a girl in white facing a man in black" when presented with an image of it, and avoid choosing the incorrect hard negative description, "a girl in black facing a man in white".
In this work, we uncover a crucial vulnerability in not just one but all these image-to-text compositionality benchmarks: We find that a blind model that never looks at the image, can identify the correct caption and avoid choosing the supposed "hard negatives". This blind model outperforms a wide array of pretrained vision-language models across the suite of benchmarks [36, 18, 14]. We explain this undesired hackability in existing benchmarks by showcasing that there exists a significant distributional gap between the positive and hard negative captions. For instance, in the ARO bench
* The authors contribute equally to this work.
mark [49], human-generated positive captions differ drastically from the hard negative texts generated by randomly shuffling words in the positive captions. As new research has begun to propose methods that claim to improve compositionality on these benchmarks [49, 38], we find it critical to highlight our findings and propose a solution.
We propose a solution to existing hackable benchmarks by introducing SUGARCREPE, a new benchmark to faithfully evaluate compositionality. In curating SUGARCREPE, we identify two main biases<sup>2</sup> that result in the distributional gap between positive and hard negatives; and employ mechanisms to fix the shifts. In particular, we find the current procedure in generating hard negatives introduces descriptions that are (1) not plausible and (2) non-fluent. For example, while the caption "olives and grapes on a plate" is a sensical fluent caption, benchmarks often have non-plausible hard negatives like "olives and grapes inside a plate" or simply incomprehensible ones like "right has word another word. There is a words" (see Table 1 for more examples). We mitigate such biases by first leveraging a modern large language model, ChatGPT [32], to generate plausible and natural hard negative texts instead of relying on simple rule-based templates employed by existing benchmarks [30, 49]. Then, we subsample the dataset through an adversarial refinement process to ensure the identified biases are maximally removed by drawing on recent dataset debiasing work [50, 41, 23]. Taken together, this workflow is where SUGARCREPE derived its name: Synthetic yet Unbiased Generation with Adversarily Refined Compositional REpresentation Evaluation. We qualitatively and quantitatively verify through both human and automatic evaluations that SUGARCREPE effectively fixes these biases.
With SUGARCREPE, we re-evaluate recent methods proposed to improve compositionality. Specifically, we focus on one prominent approach that aims to improve compositionality through data augmentation. This method trains models by generating compositional hard negatives and injecting them within a training batch [13, 49]. Unfortunately, we observe that the effectiveness of this simple data augmentation approach is hugely overestimated when evaluated on existing benchmarks, leading to limited improvements on SUGARCREPE. Finally, we evaluate a wide variety of 17 pretrained CLIP models [36, 18, 14], and find that current models still lack compositionality. Our results suggest that to improve compositionality, future work may need more innovative techniques.
We situate our paper amongst existing work on vision-language compositionality, and debiasing datasets for model evaluation.
Evaluating vision-language compositionality. Recent works have introduced benchmarks to evaluate the compositionality of vision-language models [36]; they find that current models exhibit little compositional understanding [49, 45, 53, 30, 38] despite their remarkable performance on downstream tasks [36, 25, 43, 1, 47, 48, 52]. Models have a hard time discerning between text containing the same words ordered differently [45]. Models also fail to link objects to their attributes, or understand the relationship between objects [53, 49, 38]. Our work finds that many of the benchmarks used to evaluate compositionality have hackable biases; blind models that do not even look at the image outperform state-of-the-art vision-language models.
Improving vision-language compositionality. To enhance vision-language models' compositionality, new proposals suggest training strategies that utilize additional data, models, and/or losses [49, 5, 38, 13, 44]. Amongst them, one prominent approach is to explicitly train the models to distinguish hard negatives from the correct captions [49, 13]. While these approaches appear to improve compositionality on benchmarks, it is unclear if these models achieve such improvements by actually acquiring compositional understanding or by exploiting biases in these datasets. We answer this question in our evaluation.
Debiasing dataset for faithful model evaluation. Several prior manuscripts have pointed out that biased datasets could lead to an overestimation of models' true capabilities [16]. They have proposed dataset de-biasing methods to enable more faithful model evaluations [39, 50, 41, 23, 34]. For instance, adversarial filtering [50] iteratively trains an ensemble of classifiers on different training splits and uses them to filter out "easy" negatives for each instance. Building upon adversarial filtering, AFLite [41, 23] filters data instances in a more light-weight manner without retraining a
2 We use biases and artifacts interchangeably in the paper.
Table 1: Existing compositionality benchmarks rely on procedurally-generated hard negatives which often do not make logical sense or are not fluent due to grammatical errors.
Dataset	Nonsensical Hard Negatives	Non-fluent Hard Negatives
CREPE [30]	Olives and grape inside a plate. Ground in a basket on the flowers. A hair wearing a necklace, with her lady on a table.	A door with panes not in a room; the door has windows. Right has word another word. There is a words. A shelf with books in something. There is no background.
ARO [49]	The grass is eating the horse. A gray bathtub is looking at a white cat. Green ball with a remarkable chair behind a blue scene.	At brown cat a in looking a gray dog sitting is and white bathtub. Scene with remarkable a ball blue a green behind chair. Books the looking at people are.
VL-CheckList [53]	Sheep is hardwood. Empty zebras. The bush speaking in the garden.	An man fishing a food from a wrapper using a paw at a open. It heaving at a city. An grouping subduing at a room access.
model at each iteration and leads to benchmarks that more accurately represent the underlying tasks. We use adversarial refinement to remove biases that creep into the generation of compositionality benchmarks.
3 Limit and biases of current compositionality benchmarks
A majority of existing compositionality benchmarks for vision-language models formulate the evaluation task as image-to-text retrieval [53, 49, 30]. We focus on these benchmarks and discuss others [45, 38] in Appendix B. Given an image, the model is probed to select text that correctly describes the image from a pool of candidates. Unlike standard retrieval tasks where the negative (incorrect) candidates differ a lot from the positive (correct) text, compositionality benchmarks intentionally design hard negative texts that differ minimally from the positive text, in order to test whether the model understands the fine-grained atomic concepts that compose the scene.
Existing hard negative generation process introduces undesirable biases. Existing benchmarks generate hard negative texts through rule-based programmatic procedures [53, 49, 30], which produce hard negatives by replacing a word of specific type (an object, attribute, or relation) in the original text, by swapping two words, or by shuffling the word order. We find that such procedures introduce unintentional biases in the generated hard negatives (see Table 1); specifically, we observe two major types of undesirable artifacts: (1) nonsensical artifacts, and (2) non-fluent artifacts. In order to quantitatively measure these biases, we utilize Vera [27], a plausibility estimation model, to characterize the nonsensical bias. Specifically, we define  $\mathrm{Vera}(T)$  to be the plausibility score of a caption  $T$ , where a higher score suggests more sensual the caption is. Similarly, to capture the non-fluent bias, we leverage a grammar-check model [31] that assigns high scores,  $\mathrm{Grammar}(T)$ , to more grammatically correct texts. In Figure 1, we find that Vera and the grammar model assign higher scores to positive texts, suggesting that many hard negatives are nonsensical and not fluent.
Dataset biases render current compositionality benchmarks ineffective. Given the heavily-skewed score gaps, we show that blind models (i.e., Vera and the grammar model) that simply select the higher-scoring texts as positives and admittedly do not possess any vision-language compositionality, can achieve state-of-the-art performances on existing benchmarks. We compare the blind models against 17 pretrained CLIP models from three sources: OpenAI's in-house WebImageText dataset [36], LAION [42], and Datacomp [14]. We plot the performances of the blind models and the best-performing CLIP models from each category (Figure 2). Blind models achieves state-of-the-art performances on 9 out of 10 existing benchmark tasks. We provide full evaluation results in Appendix D.1.
4 SUGARCREPE
We introduce SUGARCREPE, a new benchmark for faithful evaluation of vision-language models' compositionality based on the image-text pairs of COCO [26]. SUGARCREPE presents two key contributions over existing benchmarks: (1) it drastically reduces the two identified dataset biases (Sec. 4.1), and (2) it covers a broad range of fine-grained types of hard negatives (Sec. 4.2). We present a summary comparison on compositionality benchmarks in Appendix B.
Figure 1: Top row: We define Vera score gap as the score difference between the positive and hard negative texts:  $\mathrm{Vera}(T^{\mathrm{p}}) - \mathrm{Vera}(T^{\mathrm{n}})$ . The entire Vera score gap distribution lies on the positive spectrum, indicating that the template-generated hard negative texts usually have low plausibility. Bottom row: Similarly, Grammar score gap is defined by:  $\mathrm{Grammar}(T^{\mathrm{p}}) - \mathrm{Grammar}(T^{\mathrm{n}})$ . On grammar score, we also find that the distribution largely rests on the positive side, suggesting that most hard negative texts in existing benchmarks exhibit grammatical errors.
Figure 2: Blind commonsense Vera model and Grammar model outperform state-of-the-art CLIP models on nearly all existing benchmarks by exploiting the nonsensical and non-fluent artifacts. This suggests that existing benchmarks are hackable and ineffective in measuring compositionality.
4.1 SUGARCREPE generation workflow alleviates dataset biases
The generation procedure of SUGARCREPE consists of three main stages, centered around creating sensical and fluent hard negatives that close the distributional gaps to the positive texts, and ensuring a balanced distribution on the score gaps to make the final dataset robust to the identified biases.
Stage 1: Generate sensical and fluent hard negatives with a large language model. Observing the capability of modern large language models in generating fluent and plausible texts, we leverage ChatGPT [32] to generate hard negative texts where we explicitly instruct it to avoid commonsense (logical) and fluency (grammatical) errors. To guide ChatGPT in re-writing a given positive text into its hard negative counterparts, we provide few-shot demonstrations written by the authors and leverage its in-context learning ability [4] to generalize to unseen texts. Figure 3 shows an example demonstration used and an actual hard negative generated. We detail all the prompt templates in Appendix C.2. Table 3 shows the comparisons between hard negatives generated from ChatGPT in SUGARCREPE and that from existing benchmarks.
Stage 2: Filter false negatives with human validation. A generated text is considered a valid hard negative only if it incorrectly describes the corresponding image. For example, given an image with a positive caption "a man and a child sitting on a sofa", a compositional change that replaces "child" with "girl" may still result in a correct caption. To ensure the validity of the hard negatives in SUGARCREPE, we filter out false negatives by manually examining the generated hard negatives and their corresponding images.
Stage 3: De-bias dataset with adversarial refinement. While ChatGPT yields more sensual and fluent text, there is no guarantee that the bias between positive and negative texts is negligible. Following dataset de-biasing work [50, 41, 23], we develop an adversarial refinement mechanism that maximally reduces the undesirably exploitable artifacts in SUGARCREPE. Specifically, our goal is to ensure that performance improvements on SUGARCREPE cannot be achieved by exploiting the
Figure 3: Example prompt (black) and actual hard negative (green) generated from ChatGPT.
identified nonsensical and non-fluent biases. To accomplish this, we characterize the biases again with the commonsense and grammar models [27, 31], and subsample the dataset to ensure symmetric score gap distributions on both the positive and negative sides, as shown in Figure 4. We note the symmetry around zero implies that the commonsense and grammar scores can no longer be used to infer the ground truth positive texts. We provide the adversarial refinement algorithm in Algorithm 1.
4.2 SUGARCREPE covers a broad range of hard negative types
To test different aspects of vision-language models' compositional understanding, we follow CREPE [30] to consider various forms of hard negatives, and follow VL-CheckList [53] and ARO [49] to consider different fine-grained categories of the atomic concepts. In total, SUGARCREPE covers 7 fine-grained types of hard negatives, as shown in Table 2. We introduce the dataset taxonomy below, starting from the form of the hard negatives to its different finer-grained variants.
The REPLACE form. Given a positive text describing a scene, we generate a REPLACE hard negative by replacing an atomic concept in the original text with a new concept that makes the text mismatch with the original scene. Based on the type of the atomic concept—object, attribute, or relation—we further categorize REPLACE hard negatives into REPLACE-OBJ, REPLACE-ATT, and REPLACE-REL.
The SWAP form. Different from REPLACE, SWAP does not introduce new concepts in the hard negatives, but a SWAP hard negative is generated by swapping two atomic concepts of the same category in the positive text. We further categorize SWAP into SWAP-OBJ and SWAP-ATT, and omit swapping two relationships since it generally results in nonsensical texts.
The ADD form. Similar to the REPLACE form, but instead of replacing an atomic concept with a new one, we generate an ADD hard negative by adding a new atomic concept to the positive text that makes it mismatch with the original scene. We only further categorize ADD into ADD-OBJ (adding object concept) and ADD-ATT (adding attribute concept), as adding new relationship concepts to the positive texts often make them highly implausible.
Dataset overview. The final evaluation set of SUGARCREPE consists of 7,512 examples, where the numbers for each fine-grained type are listed in Table 2. Each example is an image-to-text retrieval task composed of an image, a positive text, and a hard negative. On SUGARCREPE, random chance performance has an average accuracy of  $50\%$ . We note that ARO and CREPE additionally consider SHUFFLE (randomly shuffling words in a sentence) and NEGATE (adding negation keywords "no/not"
Table 2: We report the number of hard negative captions of all types in SUGARCREPE.
REPLACE	SWAP	ADD
Object	Attribute	Relation	Object	Attribute	Object	Attribute
# negative captions	1,652	788	1,406	246	666	2,062	692
Table 3: We present example positive texts and their hard negatives in ARO+CREPE (generated using existing procedures) and SUGARCREPE (generated with ChatGPT). SUGARCREPE brings significant improvements in commonsense and fluency.
Hard-Negative Type	Text Type	Commonsense	Fluency
REPLACE	Original	Two adult bears play fight in the water.	A man sitting in front of a laptop computer.
ARO+CREPE	Two adult bears play fight in the soda.	A man sitting around front of a laptop computer.
SUGARCREPE	A flock of ducks play fight in the water.	A man standing in front of a laptop computer.
SWAP	Original	A woman standing behind a fence looking at an elephant.	Man swinging tennis racket while group of people watches.
ARO+CREPE	A fence standing behind a woman looking at an elephant.	Group swinging tennis racket while man of people watches.
SUGARCREPE	An elephant standing behind a fence looking at a woman.	Group of people swinging tennis racket while man watches.
NEGATE / ADD	Original	A teddy bear next to a stuffed fish.	A red fire hydrant on a city sidewalk.
ARO+CREPE	A teddy bear next to a stuffed fish. There is no teddy bear.	A red fire not hydrant on a city sidewalk.
SUGARCREPE	A teddy bear and a stuffed fish and a robot toy.	A red fire hydrant and a trash can on a city sidewalk.
to a sentence) hard negatives. We however omit them in SUGARCREPE as SHUFFLE is very unlikely to be plausible and fluent, and NEGATE introduces irreducible keyword artifacts [30].<sup>3</sup>
In this section, we qualitatively and quantitatively compare SUGARCREPE to existing benchmarks (Sec. 5.1), re-evaluate recent methods proposed to improve compositionality of vision-language models (Sec. 5.2), and comprehensively evaluate a wide array of pretrained CLIP models (Sec. 5.3).
To systematically and fairly compare SUGARCREPE with existing benchmarks, we normalize the benchmarks by reproducing their data generation workflow using COCO [26] as in SUGARCREPE. We utilize source code from CREPE [30] to generate REPLACE, SWAP, NEGATE hard negatives and take SHUFFLE hard negatives released in ARO [49]. We refer to this reproduced dataset as ARO+CREPE. In addition, we standardize the evaluation task as retrieving the correct caption from two possible choices, i.e., a positive text and a hard negative. This normalization sets the positive texts fixed for all benchmarks, including SUGARCREPE.
5.1 SUGARCREPE significantly reduces dataset biases
SUGARCREPE generates more sensical and fluent hard negatives. We validate that SUGARCREPE generates higher quality hard negative texts by leveraging ChatGPT than previous rule-based approaches. Qualitatively, in Table 3, we observe that the hard negatives in SUGARCREPE are more sensical and fluent compared to hard negatives in ARO+CREPE. We report human evaluation results in Appendix D.2 that show on an average of  $35\%$  of examples, hard negatives in SUGARCREPE have strictly higher quality than ARO+CREPE in terms of commonsense and fluency. For instance, on SWAP, humans judge that SUGARCREPE wins  $68\%$  over ARO+CREPE and ties on  $28\%$  of examples in terms of commonsense. Quantitatively, in Table 4, we compare the commonsense and grammar scores averaged over the hard negative texts in both ARO+CREPE and SUGARCREPE. We see SUGARCREPE has much higher average scores than ARO+CREPE. Additionally, pairwise comparisons show that SUGARCREPE has higher commonsense and grammar scores than ARO+CREPE on  $86\%$  of examples on average.
SUGARCREPE disentangles the identified exploitable biases. We show that the final SUGARCREPE evaluation set maximally reduces the identified biases that could be exploited undesirably to achieve improvements on a benchmark. Figure 4 visualizes the Vera/Grammar score gap distributions. We compare the distributions between ARO+CREPE and SUGARCREPE (before and after adversarial refinement). First, we see that by leveraging ChatGPT, the hard negative texts in SUGARCREPE already have lower biases than ARO+CREPE before adversarial refinement, i.e., the score gap distribution is more centered around zero. Furthermore, we see that after adversarial refinement, the
3One can easily infer hard negatives from whether the text contains negation keywords "no/not".
Table 4: We compare the commonsense and grammar scores on hard negatives in ARO+CREPE and SUGARCREPE. We report both their respective average scores and the ratio where SUGARCREPE has higher score than ARO+CREPE in pairwise comparison. Overall, SUGARCREPE has hard negatives with better commonsense and grammar.
Hard-negative Type	Metric	Average Score	Pairwise Better Ratio
ARO+CREPE	SUGARCREPE
REPLACE	Commonsense	37.46	50.21	77.71
Grammar	76.79	88.96	86.85
SWAP	Commonsense	23.09	41.57	78.76
Grammar	45.67	80.46	87.02
NEGATE / ADD	Commonsense	25.24	50.20	87.24
Grammar	65.09	90.07	95.03
Grammar Score Gap (Positive - Hard Negative)
Figure 4: We compare the Vera (top row) and Grammar (bottom row) score gap distributions between ARO+CREPE (leftmost column), SUGARCREPE without adversarial refinement (middle), and SUGARCREPE (rightmost). Top row: We see that Vera score gap distribution shifts from the positive spectrum to more centered around zero from ARO+CREPE to SUGARCREPE without refinement. After adversarial refinement, we ensure the score gap distribution is centered around zero on SUGARCREPE. Bottom row: Similarly, from ARO+CREPE to SUGARCREPE, we see the Grammar score gap distribution shifts from the positive spectrum to centered around zero.