SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Generating SWAP hard negatives. To generate SWAP hard negatives, which do not require any new concepts, we simply prompt ChatGPT once with a temperature of 0.0. Unlike REPLACE, SWAP hard negatives are only possible when there are at least two atomic concepts of the same category, i.e., either object or attribute. Thus, our prompt first queries ChatGPT whether it is possible to swap two atomic concepts in the input sentence to generate a new description. Only if the answer is yes, will ChatGPT then proceed to identify two swappable concepts and compose the corresponding new sentence by swapping the two concepts. Figure 8 shows the example templates and outputs for SWAP hard negatives.
Generating ADD hard negatives. Similar to the REPLACE, we also employ a three-step prompting procedure to generate ADD hard negatives. The only difference in the procedure is that we prompt ChatGPT to add the generated new concept to the original caption, instead of using it to replace an old concept. Figure 9 shows the example templates and outputs for ADD hard negatives.
C.3 Adversarial refinement
We detail the adversarial refinement procedure below. Given a text model  $M$ , we denote its output score for the positive and negative caption of  $i$ -th image as  $M(p_i)$  and  $M(n_i)$ . If  $M(p_i) > M(n_i)$ , then the model could identify the correct caption for the  $i$ -th image without referring to it. For a test set to be unattackable given the text model  $M$ , the expectation of  $M$ 's identifying the correct caption should be as close to random guess as possible; in particular, we hope that  $E_i[M(p_i) > M(n_i)] = 0.5$ . To achieve this for both the grammar model  $M_1$  and plausibility model  $M_2$ , we first calculate the score
$^{13}$ https://github.com/RAIVNLab/sugar-crepe
(a) REPLACE-OBJ.
(b) REPLACE-ATT.
(c) REPLACE-REL.
Figure 7: Example prompt templates (black) and outputs (green) from ChatGPT for REPLACE hard negatives.
difference  $g_{i}^{(1)} = M_{1}(p_{i}) - M_{1}(n_{i})$  and  $g_{i}^{(2)} = M_{2}(p_{i}) - M_{2}(n_{i})$ , where the range of both  $g^{(1)}$  and  $g^{(2)}$  is  $[-1, 1]$ . Then we split the 2D space of the joint range of  $g^{(1)}$  and  $g^{(2)}$  into  $100 \times 100$  equal grids, and for each pair of symmetric grids, e.g.,  $\{(g^{(1)}, g^{(2)}) | g^{(1)} \in (0.02, 0.04], g^{(2)} \in (-0.04, 0.06]\}$  and  $\{(g^{(1)}, g^{(2)}) | g^{(1)} \in (-0.02, -0.04], g^{(2)} \in (0.04, -0.06]\}$ , we preserve the same number of data for both grids, therefore we ensure that for the resultant set,  $E_{i}[M_{1}(p_{i}) > M_{1}(n_{i})] = 0.5$  and  $E_{i}[M_{2}(p_{i}) > M_{2}(n_{i})] = 0.5$ .
C.4 Dataset construction cost
We provide a high-level overview to the cost used to build SUGARCREPE by utilizing OpenAI's ChatGPT API for generating hard negatives. In building SUGARCREPE, we use approximately 40 API calls to generate hard negatives for each COCO test caption, including all different fine-grained types of hard negatives. This amounts to a total of  $25,000 \times 40 = 1,000,000$  API calls to ChatGPT. With each API call costing around $0.0005, it took roughly $500 to build SUGARCREPE.
(a) SWAP-OBJ.
Given an input sentence describing a scene, your task is to first locate two swappable noun phrases in the sentence, and then swap them to make a new sentence. The new sentence must meet the following three requirements: 1. The new sentence must be describing a different scene from the input sentence. 2. The new sentence must be fluent and grammatically correct. 3. The new sentence must make logical sense. To complete the task, you should: 1. Answer the question of whether generating such a new sentence is possible using Yes or No. 2. Output the swappable noun phrases. 3. Swap them to make a new sentence. Here are some examples: Input: A cat resting on a laptop next to a person. Is it possible to swap noun phrases in the input sentence to generate a new sentence that is different from the input sentence and makes logical sense? Yes Swappable noun phrases: laptop, person Output: A cat resting on a person next to a laptop. Input: A plate of donuts with a person in the background. Is it possible to swap noun phrases in the input sentence to generate a new sentence that is different from the input sentence and makes logical sense? Yes Swappable noun phrases: a plate of donuts, a person Output: A person with a plate of donuts in the background.
(b) SWAP-ATT.
Given an input sentence describing a scene, your task is to first locate two swappable adjectives in the sentence describing different objects, and then swap them to make a new sentence. The new sentence must meet the following three requirements: 1. The new sentence must be describing a different scene from the input sentence. 2. The new sentence must be fluent and grammatically correct. 3. The new sentence must make logical sense. To complete the task, you should: 1. Answer the question of whether generating such a new sentence is possible using Yes or No. 2. Output the swappable adjectives. 3. Swap them to make a new sentence. Here are some examples: Input: A girl in a pink shirt holding a blue umbrella Is it possible to swap attributes in the input sender to generate a new sentence that is different from the input sentence and makes logical sense? Yes Swappable attributes: pink, blue Output: A girl in a blue shirt holding a pink umbrella! Input: A girl with a green shirt brushing her teeth with a blue toothbrush. Is it possible to swap attributes in the input sender to generate a new sentence that is different from the input sentence and makes logical sense? Yes Swappable attributes: green, blue Output: A girl with a blue shirt brushing her teeth with a green toothbrush.
Figure 8: Example prompt templates (black) and outputs (green) from ChatGPT for SWAP hard negatives.
(a) ADD-OBJ.
Given an input sentence describing a scene, your task is: 1. Find the objects in the sentence. 2. Randomly pick one object. 3. Generate a new object that's not in the sentence. 4. Add the new object next to the selected object to make a new sentence. The new sentence must meet the following three requirements: 1. The new sentence must describe a clearly new and different scene. 2. The new sentence must be fluent and grammatically correct. 3. The new sentence must make logical sense. Here are some examples: Original sentence: An elephant standing under the shade of a tree. Objects: ["elephant", "shade of a tree"] Selected object: elephant New object: squirrel New sentence: An elephant and a squirrel standing under the shade of a tree. Original sentence: A bench at the beach next to the sea Objects: ['bench', 'beach', 'sea'] Selected object: bench New object: umbrella New sentence: An umbrella and a bench at the beach next to the sea.
(b) ADD-ATT.
Given an input sentence describing a scene, your task is:  
1. Find the objects in the sentence.  
2. Randomly pick one object.  
3. Generate a new plausible but uncommon attribute for this object that's not in the sentence.  
4. Add the new attribute next to the selected object to make a new sentence.  
The new sentence must meet the following three requirements:  
1. The new sentence must describe a clearly new and different scene.  
2. The new sentence must be fluent and grammatically correct.  
3. The new sentence must make logical sense.  
Here are some examples:  
Original sentence: A large white airplane and a person on a lot.  
Objects: ["airplane", "person"]  
Selected object: airplane  
New attribute: blue  
New sentence: A large white and blue airplane and a person on a lot.  
Original sentence: three people riding horses on a beach  
Objects: ['three people', 'horses', 'beach']  
Selected object: three people  
New attribute: elderly  
New sentence: Three elderly people riding horses on a beach.
Figure 9: Example prompt templates (black) and outputs (green) from ChatGPT for ADD hard negatives.
C.5 Dataset information
We host SUGARCREPE on Github  $^{14}$ . The data card [35] for SUGARCREPE, containing detailed dataset documentation, is available at the dataset repository  $^{15}$ . We provide a summary below.
Dataset documentation. SUGARCREPE is a benchmark for faithful vision-language compositionality evaluation. Given an image, a model is required to select the positive text that correctly describes the image, against another hard negative text distractor that differs from the positive text only by small compositional changes. Each example consists of three fields:
$^{14}$ https://github.com/RAIVNLab/sugar-crepe
15https://github.com/RAIVNLab/sugar-crepe/blob/main/data_card.pdf
- filename: The id to an image
- caption: Positive text correctly describing the image
- negative Caption: Hard negative text incorrectly describing the image
Maintenance plan. We are committed to maintain the dataset to address any technical issues. We actively monitor issues in the repository.
Licensing. We license our work using MIT License  $^{16}$ . All the source data we use is publicly released by prior work [26].
Author statement. We the authors will bear all responsibility in case of violation of rights.
D Detailed evaluation results
D.1 Full evaluation results on existing benchmarks
We provide the full evaluation results over 17 pretrained CLIP models as well as 2 text-only models, Vera [27] and the Grammar model [31], on existing compositionality benchmarks in Table 8. We see that the text-only models, arguably without any vision-language compositionality, outperform most of the pretrained CLIP models, achieving state-of-the-art performances on many benchmark tasks. This implies that current benchmarks fail to faithfully reflect a model's vision-language compositionality.
Table 8: Blind models (i.e., Vera and Grammar model) outperform all 17 existing pretrained CLIP models on nearly all existing benchmark tasks. This implies that current benchmarks fail to faithfully measure a model's vision-language compositionality.
Source	Model	CREPE	ARO	VL-Checklist
Atomic	Swap	Negate	VG-Relation	VG-Attribution	COCO-Order	Flickr30K-Order	Object	Attribute	Relation
Text-only model	Vera [27]	43.70	70.80	66.15	61.71	82.59	59.81	63.52	82.48	73.99	85.72
Grammar [31]	18.15	50.88	9.77	59.55	58.38	74.33	76.26	57.95	52.35	68.50
OpenAI [36]	RN50	26.47	28.32	31.25	53.87	63.37	44.89	52.46	86.85	68.30	75.95
RN101	27.63	32.74	12.50	52.43	62.93	29.86	39.34	86.44	67.93	71.75
RN50x4	26.24	28.32	9.51	51.59	62.27	29.39	34.56	87.23	68.74	73.81
ViT-B-32	22.31	26.55	28.78	51.12	61.33	37.14	47.18	87.00	68.80	77.04
RN50x16	26.36	29.65	9.38	52.13	62.71	29.95	34.26	86.95	69.34	76.83
RN50x64	26.82	30.09	23.57	51.00	62.56	40.54	46.74	87.71	68.61	74.97
ViT-L-14	26.36	25.66	24.74	53.34	61.50	36.11	45.08	87.86	68.27	75.89
LAION [42]	ViT-H-14	23.70	25.22	16.54	50.33	62.93	25.79	30.96	85.39	68.46	71.13
ViT-g-14	23.70	24.78	20.70	51.60	61.20	25.59	30.10	86.07	69.43	71.03
ViT-bigG-14	23.58	24.78	17.97	51.61	61.89	25.24	30.22	84.66	67.80	66.48
roberta-ViT-B-32	22.66	21.24	20.31	47.46	62.00	24.77	30.76	85.71	68.82	65.90
xlm-roberta-base-ViT-B-32	21.16	20.80	12.76	47.93	59.73	23.85	30.32	86.06	70.41	63.01
xlm-roberta-large-ViT-H-14	24.16	23.89	20.05	46.14	57.84	26.05	31.00	87.89	70.25	63.89
DataComp [14]	small:ViT-B-32	13.64	27.88	14.84	50.83	50.17	13.35	14.02	68.72	58.80	57.00
medium:ViT-B-32	16.42	20.35	11.33	50.45	54.04	16.44	16.26	78.43	63.53	62.94
large:ViT-B-16	18.15	17.26	17.06	48.82	53.21	21.49	26.44	84.73	65.72	64.81
x-large:ViT-L-14	21.62	22.57	16.28	48.54	60.03	23.19	29.52	86.66	67.01	67.93
D.2 SUGARCREPE human evaluation
To compare the quality of the hard negatives generated in SUGARCREPE to those in current benchmarks (i.e., ARO+CREPE), we randomly sample 100 examples for each of the hard negative types: REPLACE, SWAP, and NEGATE / ADD. Each example is organized to consist of (1) the original positive text, (2) its hard negative in ARO+CREPE, and (3) its hard negative in SUGARCREPE. For each example, a human user rates whether the hard negative in ARO+CREPE or that in SUGARCREPE is better (or tie) in terms of commonsense and grammatical correctness, respectively. Note that we compare NEGATE in ARO+CREPE to ADD in SUGARCREPE, as both hard negatives are intended to probe a model's understanding of the existence or not of an atomic concept. Table 9 shows that hard negatives in SUGARCREPE are much more sensual and fluent than that in ARO+CREPE across all three different types. For instance, SUGARCREPE has  $68\%$  more sensual and  $46\%$  more fluent hard negatives than ARO+CREPE on SWAP.
D.3 Additional NEGCLIP results
In this section, we conduct preliminary experiments to answer whether models' performances on SUGARCREPE would increase hugely if the models are trained with hard negatives generated through
$^{16}$ https://github.com/RAIVNLab/sugar-crepe/blob/main/License
Table 9: Human evaluation results on the comparisons between hard negatives in ARO+CREPE and SUGARCREPE. We report the counts (out of 100 sampled examples) that the human user considers better or tie, w.r.t. both commonsense and grammatical correctness.
Hard-negative Type	Evaluation	Human counts of better examples
ARO+CREPE	SUGARCREPE	Tie
REPLACE	Commonsense	11	29	60
Grammar	4	33	63
SWAP	Commonsense	4	68	28
Grammar	4	46	50
NEGATE / ADD	Commonsense	1	26	73
Grammar	1	35	64
the same procedure as how we create SUGARCREPE. Since generating hard negatives for training with ChatGPT would incur substantial cost, we create hard negatives using a proxy method. In particular, we start with template-generated hard negatives on the COCO training set and apply our adversarial refinement technique to remove the biases. We use this adversarially refined dataset for NegCLIP training. We show the results in Table 10. While we observe that the method improves over vanilla CLIP training without hard negatives, it performs similarly to NegCLIP and does not saturate the performance on SugarCrepe. This suggests that while the adversarial refinement mechanism prevents SugarCrepe from being attacked as an evaluation benchmark, leveraging the approach alone for training does not saturate the performance on SugarCrepe. Future work may characterize how LLMs could be used to generate better hard negatives for training to genuinely improve vision-language models' compositionality.
Table 10: Model performances on SUGARCREPE when trained with hard negatives generated through similar procedure as how SUGARCREPE is created.
Model	Hard negative	SUGARCREPE
REPLACE	SWAP	ADD
CLIP without hard negatives	N/A	69.54	60.33	67.63
NEGCLIP with template hard negatives	REPLACE	74.32	62.65	72.92
NEGCLIP with adversarial refined hard negatives	REPLACE	73.37	61.40	72.84
NEGCLIP with template hard negatives	SWAP	73.31	68.35	71.93
NEGCLIP with adversarial refined hard negatives	SWAP	72.07	65.13	69.68
NEGCLIP with template hard negatives	NEGATE	72.74	60.89	70.47
NEGCLIP with adversarial refined hard negatives	NEGATE	72.70	60.75	68.70