SlowGuess/NeurIPS_2023_OCR huggingface dataset:
score gap distributions on the final SUGARCREPE evaluation set are symmetric around zero. This implies that the previously identified artifacts can no longer be exploited to infer the positive texts. As a result, we show that the previous commonsense and grammar attacks that are extremely successful on existing benchmarks do not work on SUGARCREPE. As shown in Table 6, these blind models now consistently rank the last on SUGARCREPE as compared to other pretrained CLIP models.
5.2 Re-evaluating recent methods for improving compositionality
Given the vulnerability of existing compositionality benchmarks, it is unclear whether recently proposed methods that show state-of-the-art performances on these benchmarks are indeed effective. Thus, we re-evaluate these methods with SUGARCREPE.
Hard negative augmented training. Specifically, we focus on evaluating one common data-augmentation approach considered in [49, 13], where the core idea is to explicitly create hard negatives and train the model to distinguish them. We broadly refer to this training scheme as NEGCLIP following [49]. We evaluate two NEGCLIP training schemes: finetuning and training from scratch. For finetuning, in addition to taking the model released in [49], we finetune another three NEGCLIP models (using ViT-B/32 following [49]) with three respective types of hard negatives (i.e., REPLACE, SWAP, NEGATE) generated using CREPE's [30] source code. For training from scratch, we use RN50 as the base model and train variants of NEGCLIP by augmenting the training examples with different types of hard negatives. We perform both training and finetuning on COCO [26].
Improvements are overestimated due to unintentionally overfitting. In Table 5, we first see that NEGCLIP finetuned models show significant improvements on ARO+CREPE, boosting the performance more than  $10\%$  compared to standard CLIP finetuning on 11 out of 16 cases (highlighted in green). The lifts are especially large when the hard negative type used in finetuning matches that used in evaluation, where NEGCLIP finetuned models can achieve near human-level performances. For instance, by finetuning with REPLACE hard negatives, NEGCLIP reaches  $94\%$  on ARO+CREPE evaluated with REPLACE hard negatives (human performance is  $95\%$ ). While the results on ARO+CREPE suggest that NEGCLIP is seemingly sufficient in equipping models with strong compositionality, we however see that the improvements brought by NEGCLIP are much smaller on SUGARCREPE. In fact, none of the improvements on SUGARCREPE is larger than  $10\%$ , and the best performing NEGCLIP finetuned models still have large gaps to human-level performances, e.g., best NEGCLIP model lags behind human by  $23\%$  on SUGARCREPE's SWAP hard negatives. Similarly, when trained from scratch, we observe the same trend that NEGCLIP's improvements are much larger on ARO+CREPE than on SUGARCREPE. The improvements on ARO+CREPE are again most pronounced when the training and testing hard negative type matches.
We attribute the stark contrast in NEGCLIP's effectiveness on ARO+CREPE and SUGARCREPE to model's unintentional overfitting: The NEGCLIP models learned to exploit artifacts that can be used to easily distinguish hard negatives from positives on ARO+CREPE, instead of actually improving compositionality. Thus, when evaluated on SUGARCREPE where the artifacts are removed, the improvement from NEGCLIP drastically reduces. These results imply that NEGCLIP's effectiveness is overestimated on existing benchmarks, and we may still need further innovations to fundamentally improve a model's compositionality.<sup>4</sup>
Color notations: Gains compared to standard CLIP (finetuned / from scratch)  $>10\%$ .
Table 5: Re-evaluating hard negative augmented training shows that the method's improvements on existing benchmarks (ARO+CREPE) are hugely overestimated, particularly when the test hard negative type matches the one used in training, which can be attributed to overfitting the artifacts.
Model	Training	Hard Negative Used	ARO+CREPE	SUGARCREPE
REPLACE	SWAP	NEGATE	SHUFFLE	REPLACE	SWAP	ADD
Human	95.33	100	99.33	96.00	98.67	99.50	99.00
ViT-B/32	Pretrained	N/A	75.71	71.58	76.89	72.06	80.76	63.27	75.09
CLIP finetuned	N/A	77.06	68.81	61.19	63.04	84.76	70.83	85.58
NEGCLIP finetuned	REPLACE	94.51	90.04	85.06	88.15	88.27	74.89	90.16
SWAP	82.88	94.48	77.57	87.00	85.54	76.21	86.56
NEGATE	77.24	68.91	99.54	64.28	84.97	70.29	85.84
Released in [49]	85.72	94.35	83.51	90.45	85.36	75.33	87.29
RN50	CLIP from scratch	N/A	69.93	59.96	55.36	68.78	69.54	60.33	67.63
NEGCLIP from scratch	REPLACE	89.04	66.51	60.90	75.23	74.32	62.65	72.92
SWAP	72.33	92.29	64.51	84.84	73.31	68.35	71.93
NEGATE	70.09	60.29	99.45	69.03	72.74	60.89	70.47
REP + SW + NEG	86.30	88.60	99.34	82.93	75.26	67.69	73.08
5.3 Comprehensive evaluations on existing pretrained vision-language models
We present four key findings in our evaluation over 17 pretrained CLIP models on SUGARCREPE, with results reported in Table 6 and visualized in Figure 5.
The best pretrained CLIP models demonstrate some compositional understanding but still have overall large rooms for improvements. Table 6 shows that the largest pretrained CLIP models, e.g., OpenAI's RN50x64, LAION's xlm-roberta-large-ViT-H-14, and DataComp's ViT-L-14, achieve near-human performance on REPLACE-OBJ. However, on REPLACE-OBJ, smaller models pretrained on small datasets still suffer from big drops in performance —  $23\%$  and  $43\%$  respectively for DataComp's small and medium models — compared to humans. Additionally, on nearly all other hard negative types, there are clear gaps (larger than  $10\%$ ) between the best model performances and human performances, showing an overall large room for improvements in current models' compositionality.
4In Appendix D.3, we provide further results on training NEGCLIP with hard negatives filtered with our adversarial refinement mechanism.
Table 6: Our evaluation of pretrained CLIP models on SUGARCREPE shows that they demonstrate compositionality on some hard negatives but are far from human performance on others, especially on SWAP hard negatives or ones perturbing attributes and relations (also illustrated in Figure 5: lower overall performance on SWAP, and lower performances on attributes/relations compared to objects). We additionally evaluate recently introduced GPT-4V [33]. While it demonstrates strong results, there is still gap to human-level performance.
Source	Model	Data Size	Model Size (M)	REPLACE	SWAP	ADD	Average
Object	Attribute	Relation	Object	Attribute	Object	Attribute
Human	100	99	97	99	100	99	99	99
Text-only model	Vera [27]	49.39	49.62	49.36	49.19	49.40	49.42	49.57	49.42
Grammar [31]	50.00	50.00	50.00	50.00	50.00	50.00	50.00	50.00
OpenAI [36]	RN50	102	91.77	80.58	69.99	61.79	68.47	74.54	69.65	73.83
RN101	120	92.49	83.88	67.07	56.50	65.92	75.46	70.09	73.06
ViT-B-32	151	90.92	80.08	69.20	61.38	63.96	77.21	68.79	73.08
ViT-B-32-negclip	400M	151	92.68	85.91	76.46	75.20	75.38	88.80	82.80	82.46
RN50x4	178	92.68	82.99	67.57	65.04	63.36	79.34	70.09	74.44
RN50x16	291	93.46	82.11	69.20	63.01	65.77	80.70	75.87	75.73
ViT-L-14	428	94.07	79.19	65.15	60.16	62.31	78.32	71.53	72.96
RN50x64	623	94.49	83.50	70.63	61.79	66.67	83.27	73.99	76.33
LAION [42]	roberta-ViT-B-32	212	92.86	84.90	72.40	63.01	71.02	87.34	79.91	78.78
ViT-H-14	2B	986	96.49	84.77	71.76	67.48	73.12	92.05	85.84	81.64
ViT-g-14	1367	95.76	85.03	72.40	63.01	71.17	91.51	82.08	80.14
ViT-bigG-14	2540	96.67	88.07	74.75	62.20	74.92	92.19	84.54	81.91
xlm-roberta-base-ViT-B-32	366	93.16	84.01	69.20	63.41	67.57	87.78	81.07	78.03
xlm-roberta-large-ViT-H-14	5B	1193	96.85	86.04	72.05	63.82	72.07	93.11	86.13	81.44
DataComp [14]	small:ViT-B-32	13M	151	56.90	56.85	51.99	50.81	50.00	53.93	60.55	54.43
medium:ViT-B-32	128M	151	77.00	69.54	57.68	57.72	57.06	66.73	64.88	64.37
large:ViT-B-16	1B	150	92.68	79.82	63.94	56.10	57.66	84.34	78.61	73.31
xlarge:ViT-L-14	13B	428	95.52	84.52	69.99	65.04	66.82	91.03	84.97	79.70
OpenAI	GPT-4V [33]	96.31	93.53	90.26	83.13	90.09	91.59	91.76	90.95
Figure 5: We plot pretrained vision-language models' zero-shot top-1 accuracy on ImageNet versus their retrieval recall@1 on SUGARCREPE, where  $r$  is the Pearson correlation coefficient. This plot suggests that models' ImageNet zero-shot accuracy positively correlates with their compositionality.
All models struggle at identifying SWAP hard negatives, regardless of their pertaining dataset and model size. Among the three types of hard negatives, SWAP hard negatives present the biggest challenge to the pretrained CLIP models, even though humans can easily tell them apart from the positive captions. We observe in Table 6 that all models demonstrate low performance on both SWAP-OBJ and SWAP-ATT hard negatives regardless of their pretraining dataset and model sizes, with the difference from human performance reaching from  $27\%$  to  $50\%$ .
Existing models are object-centric, struggling to compose attributes and relations. We find that existing pretrained models are a lot better at composing objects than attributes or relations (Table 6). This finding holds for both REPLACE and ADD hard negatives but not the most difficult SWAP negatives, where models perform equally poorly on both SWAP-OBJ and SWAP-ATT. On REPLACE hard negatives, even though most models achieve human-level performance on REPLACE-OBJ, they all suffer from a drop in performance on REPLACE-ATT and REPLACE-REL, where the drop is as large as  $15\%$  and  $29\%$  respectively. Similarly, on ADD hard negatives, all models except for DataComp's small:ViT-B-32 experience a decrease in performance from ADD-OBJ to ADD-ATT, with the largest difference reaching  $10\%$ .
Models' performance on SUGARCREPE correlates with their ImageNet zero-shot accuracy. We show in Figure 5 that there is a positive correlation between models' performance on SUGARCREPE and their zero-shot accuracy on ImageNet. This correlation is moderate on SWAP-OBJ and ADD-ATT (Pearson correlation coefficient  $r = 0.78$  and  $r = 0.75$  respectively) and strong on all other hard negatives ( $r > 0.8$ ).
Our investigation reveals significant biases present in existing benchmarks for the compositional comprehension capability of vision-language models. The severity of this vulnerability is exemplified by text-only models without access to the image outperforming vision-language models. To address this, we introduce SUGARCREPE, a novel benchmark for evaluating the compositionality of vision-language understanding. Unlike previous benchmarks that relied on rule-based templates, we leverage large language models to generate less biased negatives and employ adversarial filtering mechanisms to minimize biases. Through reassessment of state-of-the-art models and recently proposed compositionality inducing mechanisms, we uncover a significant overestimation of their advancements, underscoring the need for further innovation.
6.1 Limitation and future work
Scope of the compositionality benchmarks and vision-language models. We focus our scope on compositionality benchmarks formulated as image-to-text retrieval task. While this is currently the most prevailing evaluation framework, future research can characterize compositionality evaluation as text-to-image retrieval problem, as in the initial efforts considered by [38, 45]. More importantly, we hope our work can guide future efforts in creating and ensuring faithful compositionality benchmarks in text-to-image form. In addition, we focus our evaluations on contrastively learned vision-language models [36]. Future work should include and characterize the compositionality of modern generative vision-language models [1, 7, 24, 46].
Potential biases imposed by language models. In this work, we identify two human interpretable dataset biases, the nonsensical and non-fluent biases, which may not cover all dataset artifacts that could possibly be exploited by a model. By leveraging ChatGPT in generating hard negatives, the generated captions may also exhibit hard to detect biases imposed by the language model, e.g., watermarks [21]. Future work may utilize more sophisticated adversarial filtering techniques that train models to detect and remove spurious dataset artifacts beyond human comprehension [51, 23].
Shifts in language model behavior. Our work leverages ChatGPT to generate hard negatives. However, recent work has pointed out that the underlying model behind these APIs may change, resulting in model behavior shifts [6, 28]. We discuss how this potential model behavior shift may affect our proposed dataset construction pipeline. Specifically, while there may be variances on the quality of the generated texts, we note that our employed adversarial refinement mechanism can ensure that the final evaluation set is free of the identified artifacts. In the case when ChatGPT improves and generates higher-quality captions, the refinement mechanism will filter out less examples and we can more efficiently create the final evaluation set. On the other hand, if ChatGPT degrades and shifts towards generating less fluent and plausible captions, the refinement mechanism will filter out more generated examples and we would need to generate more candidates in order to create an evaluation set of the same desired size. As a result, while the efficiency of the proposed dataset construction pipeline depends on quality of the language model used, our pipeline ensures the generated set does not contain the identified biases. In the large language model era, we see these capable models as productive tools one can leverage to efficiently process and create data. We do however deem careful validation mechanisms, such as our manual and automatic filtering technique, necessary to ensure that the ultimate goal is properly achieved.
6.2 Societal impact
As vision-language models such as CLIP [36] are becoming the foundation models for many downstream applications [40, 37], it is imperative to understand the limitations of these models to avoid misuses and undesirable outcomes [8, 2]. Compositionality benchmarks probe a model's understanding of finer-grained concepts, and hence allow us to identify blind spots [49, 53, 30] of seemingly powerful models deemed by standard classification and retrieval benchmarks [11, 26]. Our work further alleviates common artifacts in existing compositionality benchmarks that result in overestimation of a model's capability. We hope our proposed benchmark SUGARCREPE leads to more faithful assessment of a vision-language model's compositionality, and can hence guide more accurate usages of the models. Nevertheless, we note that strong performances on SUGARCREPE do not imply perfect models. We envision SUGARCREPE being one of the many benchmarks used to comprehensively understand the abilities of vision-language models from various aspects.
[47] Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for image-language and video-language tasks. arXiv preprint arXiv:2209.07526, 2022.
[48] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442, 2022.
[49] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? In International Conference on Learning Representations, 2023.
[50] Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint arXiv:1808.05326, 2018.
[51] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019.
[52] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123-18133, 2022.
[53] Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations. arXiv preprint arXiv:2207.00221, 2022.
A.1 Hardware information
All experiments are run on a machine with an Intel(R) Xeon(R) CPU E5-2678 v3 with a 512G memory and two 48G NVIDIA RTX A6000 GPUs.
A.2 Dataset sources
We obtain all existing datasets from their original sources released by the authors. We refer readers to these sources for the dataset licenses. To the best of our knowledge, the data we use does not contain personally identifiable information or offensive content.
- CREPE [30]: We obtain CREPE dataset from its official repository  ${}^{5}$  .
- ARO [49]: We obtain ARO dataset from its official repository  ${}^{6}$  .
- VL-CheckList [53]: We obtain VL-CheckList dataset from its official repository  ${}^{7}$  .
COCO [26]: We obtain COCO from its official project website.
A.3 Software configuration
Models. We detail the sources of the pretrained models we use in the paper, and the hyper-parameters used in training our own models.
- Vera model [27]: We obtain pretrained Vera model released by its author.
- Grammar model [31]: We obtain the Grammar model released by the authors  $^{10}$ .
- All pretrained CLIP models: We obtain all pretrained CLIP models' weights from OpenCLIP<sup>11</sup>.
- NEGCLIP models: We obtain weights for pretrained NEGCLIP released by the authors  $^{12}$ . For training from scratch and finetuning, we train RN50 and ViT-B/32 based on OpenCLIP codebase and set hyper-parameters as the following: number of warmup steps is 1000, batch size is 256, learning rate is 1e-4, weight decay is 0.1, number of epochs is 30. We augment the original CLIP loss with hard negative captions following NEGCLIP [49].
**Evaluations.** We base our evaluation framework on OpenCLIP [18]. We follow all default hyperparameters used for evaluating models.
B Vision-language compositionality benchmarks
We provide an overview of existing vision-language compositionality benchmarks below, with Table 7 summarizing the dataset comparisons.
B.1 Image-to-text formulation
A majority of current benchmarks formulate the evaluation task as image-to-text retrieval problem. These benchmarks generate hard negative texts procedurally through rule-based templates, where each benchmark considers different types of hard negatives.
VL-Checklist [53]. VL-CheckList aims at evaluating vision-language models' understanding of different objects, attributes, and relationships. It contains REPLACE hard negatives generated by
$^{5}$ https://github.com/RAIVNLab/CREPE  
 $^{6}$ https://github.com/mertyg/vision-language-models-are-bows  
 $^{7}$ https://github.com om-ai-lab/VL-CheckList  
 $^{8}$ https://cocodataset.org/  
 $^{9}$ https://huggingface.co/liujch1998/vera  
 $^{10}$ https://huggingface.co/textattack/distilbert-base-uncased-CoLA  
 $^{11}$ https://github.com/mlfoundations/open Clip  
 $^{12}$ https://github.com/mertyg/vision-language-models-are-bows
Table 7: Summary on vision-language compositionality benchmarks. SUGARCREPE considers image-to-text formulation to enable larger scale evaluation set. In addition, SUGARCREPE considers a wide range of hard negative types. SHUFFLE and NEGATE are omitted as they introduce inevitable biases discussed in Sec. 4.2.
Benchmark	Task Formulation	Scale	Hard Negative Text Type
SHUFFLE	REPLACE	SWAP	NEGATE	ADD
VL-CheckList [53]	Image-to-Text	>1000	✓
ARO [49]	Image-to-Text	>1000	✓	✓
CREPE [30]	Image-to-Text	>1000	✓	✓	✓
Winoground [45]	Image-to-Text / Text-to-Image	400	✓
Cola [38]	Text-to-Image	210	N/A
SUGARCREPE	Image-to-Text	>1000	✓	✓	✓
replacing atomic parts of the positive texts with other foils. VL-CheckList further breaks the hard negatives down into more granular categories based on the type of the replaced atomic part, i.e., object, attribute, or relationship.
ARO [49]. ARO focuses on models' understanding of different relationships, attributes, and order information. It considers SWAP and SHUFFLE hard negatives. SWAP hard negatives are generated by swapping two words in the positive texts; on the other hand, SHUFFLE hard negatives are generated by shuffling words in the positive texts. ARO further divides SWAP hard negatives into attribute or relationship type.
CREPE [30]. CREPE is a large-scale evaluation benchmark that includes three types of hard negatives: REPLACE, SWAP and NEGATE. REPLACE and SWAP hard negatives are generated as in VL-CheckList and ARO. In addition, NEGATE hard negatives are generated by adding negation keywords (i.e., not or no) to the original positive texts. The hard negatives are not further divided into fine-grained types (object, attribute, or relations).
B.2 Text-to-image formulation
Complementary to image-to-text formulation, compositionality can as well be evaluated by probing a model to select an image that best matches a given text description, against other hard negative images as distractors. Unlike hard negative texts, hard negative images are more difficult to obtain and thus current text-to-image compositionality benchmarks are smaller at scale.
Winoground [45]. Winoground is a small dataset manually curated by human annotators. Each example in the dataset contains two images and two matching captions, where both captions contain identical words that appear in different orders. Note that Winoground can be used for either image-to-text or text-to-image retrieval. While the original intention for Winoground is to evaluate vision-language compositionality, recent work [12] has pointed out that solving the tasks in Winoground requires not just compositional vision-language understanding, but additionally a suite of other abilities such as commonsense reasoning, or distinguishing visually difficult images.
Cola [38]. Cola tests a vision-language model's ability to select an image that correctly matches a given caption, against another distractor image with the same objects and attributes but in the wrong composition. The image pairs are mined from existing datasets. As a result, the final evaluation set is relatively small in size (210 examples in total).
We deem text-to-image evaluation as important as image-to-text evaluation. Future work can explore approaches to generate or mine compositional hard negative images at scale, as preliminarily explored in [38, 49].
C SUGARCREPE
C.1 Taxonomy
Figure 6 shows the taxonomy of SUGARCREPE. We first categorize the hard negatives based on their forms: REPLACE, SWAP, and ADD. We then further divide each type of hard negatives into finer-grained sub-categories based on the type (object, attribute, or relation) of the atomic concept altered. SUGARCREPE covers a total of 7 fine-graind hard negative types.
Figure 6: Taxonomy of hard negatives considered in SUGARCREPE.
C.2 Hard negative generation procedure and templates
To generate hard negatives in SUGARCREPE, we come up with three different prompt templates for the three hard negative types considered: REPLACE, SWAP, and ADD. Each template consists of task instruction for generating the corresponding type of hard negatives and several (7 or more) few-shot demonstrations. We describe the general generation procedure and example prompt templates below and refer readers to our dataset repository for the full prompts used  $^{13}$ .
Generating REPLACE hard negatives. To best leverage ChatGPT's capabilities, we devise a three-step workflow to generate REPLACE hard negatives: (1) We prompt ChatGPT in locating the desired atomic concepts (e.g., objects) in the sentence; (2) We prompt ChatGPT to generate a new concept to replace a randomly selected old concept; (3) We let ChatGPT compose a new sentence by replacing the old concept with the new one. For steps (1) and (3), we prompt ChatGPT with a temperature of 0.0 to get stable outputs. For step (2), however, we diversify the outputs by prompting ChatGPT with a higher temperature of 1.5. With this design, we are able to generate diverse REPLACE hard negatives. Figure 7 shows the example templates and outputs for REPLACE hard negatives.