SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2 reports ITR performance (i.e., Recall at 1, 5, and 10) on COCO-CFs and human-evaluated-COCO-CFs for pre-trained BridgeTower and Flava models. The percentages enclosed within parentheses indicate the change in performance of a model on an evaluated dataset versus the performance of that model on MS-COCO (baseline). We observe that the performance of BridgeTower and Flava decreases significantly (up to  $51\%$  and  $57\%$ , respectively) compared to the baseline's performance on both COCO-CFs and human-evaluated-COCO-CFs. These results demonstrate that COCO-Counterfactuals can serve as a challenging test set for SOTA multimodal vision-language models.
4.2.2 Image-text Matching
Typically, during pre-training for image-text matching (ITM), multimodal models learn to differentiate actual image-text pairs from alternative images or captions which are randomly sampled from a dataset. By design, our COCO-CFs have the potential to make this task significantly more challenging by requiring models to also differentiate between minimally-edited image or text candidates. We measure the magnitude of this increased difficulty by comparing the difference in ITM scores between actual image-text pairs and their corresponding counterfactual or randomly sampled alternatives.
Let  $(C_o, I_o)$  denote an original image-text pair from MS-COCO,  $I_o^s$  denote our synthetically-generated image corresponding to  $C_o$ , and  $(C_c, I_c^s)$  denote our corresponding synthetically-generated counterfactual image-text pair in COCO-Counterfactuals. We further denote  $(C_r, I_r)$  as a different original image-text pair randomly sampled from MS-COCO such that  $I_o \neq I_r$ . For a given pre-trained
multimodal model, we compute the following metrics using its ITM scoring function  $\mathcal{G}$ :
$$
\operatorname {I R} _ {r} = \mathcal {G} \left(C _ {r}, I _ {r}\right) - \mathcal {G} \left(C _ {r}, I _ {o}\right) \quad \operatorname {I R} _ {c} = \mathcal {G} \left(C _ {c}, I _ {c} ^ {s}\right) - \mathcal {G} \left(C _ {c}, I _ {o} ^ {s}\right)
$$
$$
\mathrm {T R} _ {r} = \mathcal {G} \left(C _ {r}, I _ {r}\right) - \mathcal {G} \left(C _ {o}, I _ {r}\right) \quad \mathrm {T R} _ {c} = \mathcal {G} \left(C _ {c}, I _ {c} ^ {s}\right) - \mathcal {G} \left(C _ {o}, I _ {c} ^ {s}\right)
$$
$\mathrm{IR}_r$  and  $\mathrm{TR}_r$  scores can be viewed as measuring the confidence of a model's image or text retrieval (respectively) over two real image-text pairs from MS-COCO. Similarly,  $\mathrm{IR}_c$  and  $\mathrm{TR}_c$  scores measure image or text retrieval confidence, but using matched image-text pairs from COCO-Counterfactuals dataset. For all metrics, values greater than zero indicate that a model scores the correct image-text pair as more similar than its random or counterfactual alternative. Larger positive values can be viewed as indicating greater confidence in the model's correct discernment between the alternatives.
Figure 3 plots the distribution of these four metrics for three pre-trained multimodal models: CLIP, BridgeTower, and ViLT. All three models exhibit a significant negative distribution shift when presented with counterfactual alternatives rather than random alternatives, demonstrating the increased difficulty of COCO-Counterfactuals for existing models. A significant number of COCO-Counterfactuals are also incorrectly scored (i.e., have ITM score difference less than zero) by all three models. Even in cases where the counterfactual alternatives can be correctly discerned, we posit that the much smaller values of  $\mathrm{IR}_c$  and  $\mathrm{TR}_c$  may improve the efficiency of training through the increased difficulty of the ITM task.
4.2.3 Discussion
When used as a test set, COCO-Counterfactuals by design evaluate the robustness of models to minimal changes in paired image-text data. Table 2 and Figure 3 show that existing models perform significantly worse when evaluated on COCO-Counterfactuals. Additionally, we find that training these same models on COCO-Counterfactuals produces an average relative improvement of  $24.3\%$  in image-text retrieval performance on withheld counterfactual examples (see Table 5 of Appendix A.1). These results point to the usefulness our dataset for evaluating and improving the robustness of multimodal models to counterfactual changes.
5 COCO-Counterfactuals for Training Data Augmentation
This section aims to evaluate whether COCO-Counterfactuals can serve as an alternative to real data for training data augmentation in low-resource scenarios. We train a fully unfrozen pre-trained CLIP model with its contrastive loss using various combinations of real data from MS-COCO and COCO-CFs datasets (see Appendix B.3 for additional training details). In order to investigate the robustness of models trained on COCO-CFs, we evaluate them on OOD datasets for image-text retrieval and image recognition. For baselines, we report the performance of pre-trained CLIP (i.e., without any additional training) as well as a CLIP model which has been additionally trained using only real data from MS-COCO. We repeat each of our training experiments with 25 different random seeds and report both the mean and standard deviation of performance measured across all random seeds. We also validate the statistical significance of performance improvements obtained by models trained on COCO-CFs using one-tailed t-tests.
5.1 Image-text Retrieval
To evaluate OOD performance on the image-text retrieval task that CLIP was trained for, we use the 1K test set of Flickr30k dataset (Young et al., 2014). Table 3 reports the zero-shot performance of the baselines as well as CLIP trained with varying amounts of the original MS-COCO and COCO-CFs datasets. We observe that all CLIP models trained with COCO-Counterfactuals outperform pre-trained CLIP by an average of 5 points, based on the mean performance across text and image retrieval settings. Additionally, our best model trained with 20,894 COCO-Counterfactuals provides statistically significant improvements relative to training only on the real MS-COCO dataset across all settings. We also found that COCO-Counterfactuals improve in-domain performance on the MS-COCO test set, which we detail in Appendix A.5.
To investigate the potential impact of COCO-Counterfactuals which were labeled incorrectly by human annotators, we repeated our training data augmentation experiments using only image-text pairs which were correctly matched in our human evaluation study (Section 4.1). Overall, we found
Training dataset	|Dtrain|	% CFs	Text Retrieval	Image Retrieval	Mean
R@1	R@5	R@10	R@1	R@5	R@10
None (pre-trained CLIP)	0	0%	67.1	89	93.8	69.4	90.6	94.9	84.13
MS-COCO	13,928	0%	77.900.4	93.790.2	97.110.1	75.140.4	93.720.2	96.690.2	89.060.1
MS-COCO + COCO-CFs	13,928	50%	76.660.5	94.530.3	96.840.2	75.750.4	93.600.2	96.960.2	89.050.1
MS-COCO + COCO-CFs	34,820	60%	78.280.4	94.720.3	97.270.2	76.130.5	93.850.2	96.910.2	89.530.2
MS-COCO + COCO-CFs	41,784	67%	77.750.5	94.510.3	97.030.2	76.380.3	94.010.2	96.790.2	89.410.1
Table 3: Image-text retrieval performance on the OOD Flickr30k 1K test set for pre-trained CLIP and CLIP models trained on varying amounts of data from MS-COCO and COCO-CFs datasets.  $\left| D_{\text{train}} \right|$  indicates the total number of image-text pairs used for training, while % CFs indicate the percentage of those image-text pairs which were sampled from COCO-CFs. Results report mean over 25 different random seeds, with standard deviation as a subscript. Best results are in boldface. Results which use COCO-CFs are underlined when a one-tailed t-test indicates that their improvement over training only on MS-COCO is statistically significant ( $p \leq 0.05$ )
Training dataset	|Dtrain|	% CFs	CIFAR10	CIFAR100	Food101	Caltech101	Caltech256	ImageNet	Mean
None (pre-trained CLIP)	0	0%	88.8	64.17	84.17	90.32	83.43	59.25	78.36
MS-COCO	13,928	0%	89.210.3	63.890.4	82.670.2	92.770.1	85.050.1	59.550.2	78.850.2
MS-COCO + COCO-CFs	13,928	50%	89.450.3	66.670.4	83.130.2	92.630.1	85.210.1	59.660.2	79.460.2
MS-COCO + COCO-CFs	34,820	60%	89.160.3	66.880.4	82.120.2	92.870.1	84.950.2	59.220.3	79.200.2
MS-COCO + COCO-CFs	41,784	67%	88.510.5	65.970.5	82.060.2	92.770.1	84.590.2	58.880.2	78.800.2
Table 4: Zero-shot classification accuracy of pre-trained CLIP and CLIP models trained on varying amounts of data from MS-COCO and COCO-CFs datasets. All other settings are identical to Table 3.
that excluding these incorrectly-labeled COCO-Counterfactuals from training data augmentation had a negligible impact on performance (see Appendix A.4). This suggests that training data augmentation is robust to noise introduced by synthetic data, and that the  $26.82\%$  of incorrectly-labeled COCO-Counterfactuals do not pose an issue for data augmentation applications. While certain use cases which require a high degree of confidence in the accuracy of generated counterfactuals may benefit from the use of human validation, we believe that these results demonstrate how our approach can be used for fully automated training data augmentation without human annotation.
5.2 Image Recognition
Despite being trained for image-text retrieval, CLIP has exhibited impressive performance at zero-shot image recognition. Using the same approach as Radford et al. (2021) for the image recognition task (i.e., form a sentence "A photo of a  $\{c\}$ " for each class label  $c$  to obtain image-text matching scores), we evaluate whether CLIP models trained on COCO-Counterfactuals exhibit competitive OOD performance improvement to baselines' performance in this zero-shot classification setting.
Using the same CLIP models trained on varying amounts of MS-COCO and COCO-CFs, Table 4 reports their zero-shot classification accuracy on six image recognition datasets. We observe that training with an approximately 50-50 split of MS-COCO & COCO-CFs provides the best overall performance, offering improvements over pre-trained CLIP (without any additional training) on all datasets except Food101 and outperforming training with only MS-COCO on most datasets (see Appendix A.2 for additional analysis of performance differences).
5.3 Discussion
Recent work investigating the suitability of synthetic training data for image recognition tasks has found that synthetic image data is much less efficient than real data, requiring 5x more synthetic training samples to achieve similar performance as models trained on real data (He et al., 2022). In contrast, our results show that training data augmentation with COCO-Counterfactuals is at least as efficient (Table 3) and sometimes more efficient (Table 4) than data augmentation with an identical amount of real data  $(|D_{\mathrm{train}}| = 13,928)$ . This finding suggests that our approach could be particularly valuable in low-resource settings where paired image-text data is scarce.
Consistent with prior work on training data augmentation with NLP counterfactuals (Howard et al., 2022; Joshi and He, 2022), Tables 3 and 4 show that improvements in OOD performance with increasing amounts of counterfactual examples reaches a saturation point, beyond which additional data augmentation does not lead to further improvements. For image-text retrieval on Flickr30k (Table 3), this saturation point is reached with a  $40 / 60\%$  mixture of MS-COCO/COCO-Counterfactuals in the training dataset. In contrast, Table 4 shows that the saturation point for the OOD image recognition datasets is reached with a  $50 / 50\%$  split based on the mean of the six datasets. These results suggest that the optimal mixture of real examples and synthetically generated counterfactual examples may differ depending on the evaluation task and dataset.
While training data augmentation with COCO-Counterfactuals produces statistically significant performance improvements relative to training with only real data, the overall magnitude of these improvements is limited and varies by evaluation setting. COCO-Counterfactuals produce the largest improvements on zero-shot image recognition tasks, where its overall mean improvement over pre-trained CLIP is twice as large as that achieved by training on an equivalent amount of real data from MS-COCO. However, OOD generalization performance varies by dataset, which further analysis suggests, is related to domain gaps between altered subjects in COCO-Counterfactuals and the domain of the evaluation dataset (see Appendix A.2 for details).
We proposed an automated data generation methodology for creating counterfactual examples from image-text pairs to address the challenge of low-resource settings involving multimodal data. This approach was used to create COCO-Counterfactuals (COCO-CFs), a high-quality synthetic dataset of paired image-text counterfactuals derived from MS-COCO captions. COCO-CFs are challenging for existing pre-trained multimodal models and significantly increase the difficulty of the zero-shot image-text retrieval and image-text matching tasks. Our experiments demonstrate that augmenting training data with COCO-CFs improves OOD generalization on multiple downstream tasks.
In this work, we focused on the creation of task-agnostic counterfactual examples. A promising direction for future research is the adaptation of our approach to produce task-specific counterfactuals. For example, in the case of image recognition, the counterfactual changes could be limited to a targeted label distribution to produce counterfactual examples more tailored to the end task. Alternatively, task-specific model failures or spurious correlations could be diagnosed and used as a basis for determining which counterfactual changes to consider when creating the counterfactual captions. We believe that such approaches have the potential to produce counterfactuals which are more targeted for improving specific model deficiencies.
Another opportunity for future work is larger-scale automatic generation of counterfactual examples to enable full counterfactual pre-training of multimodal models. Additionally, we believe that extending our image-text counterfactuals to the video domain could be a promising path towards improving video transformers through counterfactual data augmentation.
Limitations & Ethical Concerns Motivated by the desire to produce minimal-edit counterfactuals, we only considered changes to nouns. This is a common strategy for NLP counterfactuals (see Appendix B.1.1 for discussion), but alternative generation strategies such as controlled text decoding (Howard et al., 2022) could be used to enable a larger range of counterfactual changes, in addition to alterations of adjectives or verbs. We leave investigation of these directions to future studies.
Due to a limited compute budget, we only explored generating COCO-Counterfactuals using Stable Diffusion. Additionally, our training data augmentation experiments were limited to a single model (CLIP). It is possible that other text-to-image generation models may exhibit better performance for generating counterfactual image-text data. Additionally, the benefits of counterfactual data augmentation may vary for different multimodal vision-language models.
Despite the impressive recent improvements in text-to-image generation capabilities, models such as Stable Diffusion have well-known limitations that should be considered when utilizing datasets which are derived from them (see Appendix C.5 for a detailed discussion). We do not foresee significant risks of security threats or human rights violations in our work. However, the automated nature of our image generation process may introduce the possibility of our COCO-Counterfactuals dataset containing images that some individuals may consider inappropriate or offensive.
A Additional Experiments and Analysis
A.1 COCO-Counterfactuals Improve Model Robustness to Counterfactual Changes
By design, COCO-Counterfactuals may offer greater improvements to the robustness of models to minimal or counterfactual changes in images. Such examples are unlikely to be present in the datasets used previously to evaluate OOD generalization. Therefore, we also evaluate the performance of models on a withheld test set of COCO-Counterfactuals to determine their image-text retrieval capabilities on in-domain counterfactual examples. Specifically, we withhold  $30\%$  of the original-counterfactual paired examples in COCO-Counterfactuals for testing and train the pre-trained CLIP, BridgeTower, and Flava models on the remainder, with  $56\%$  of the total dataset used for training and  $14\%$  used as a development set.
Table 5 compares the performances of CLIP, BridgeTower, and Flava models trained on COCO-Counterfactuals to those trained on an equivalent amount of real examples from MS-COCO and to their pre-trained versions $^{12}$ . We observe that training on COCO-Counterfactuals results in a mean improvement of 11.83, 21.55, and 11.47 relative to the pre-trained CLIP, BridgeTower, and Flava models, respectively. This represents an average relative improvement of  $24.3\%$  for each model over the performance of its pre-trained version. In addition, the CLIP, BridgeTower, and Flava models that were trained on COCO-Counterfactuals achieve a mean absolute improvement of 6.06, 10.08, and 5.28, respectively, relative to those that were trained on MS-COCO. The greater magnitude of these performance gains relative to our OOD image-text retrieval evaluations (Table 3) suggests that training on COCO-Counterfactuals improves model robustness to counterfactual changes, which are not present in our (non-counterfactual) OOD evaluation datasets.
Pre-trained Models	Training dataset	Text Retrieval	Image Retrieval	Mean
R@1	R@5	R@10	R@1	R@5	R@10
CLIP	None (pre-trained CLIP)	50.96	79.33	86.45	47.89	77.19	85.73	71.26
MS-COCO	57.17	84.23	90.66	55.45	84.00	90.65	77.03
COCO-CFs	65.03	90.26	94.99	64.09	89.52	94.62	83.09
BridgeTower	None (pre-trained BridgeTower)	35.26	65.31	76.73	28.77	56.63	68.46	55.19
MS-COCO	41.78	71.78	81.88	44.68	75.38	84.48	66.66
COCO-CFs	54.37	83.08	90.53	56.63	84.48	91.36	76.74
Flava	None (pre-trained Flava)	34.40	66.63	78.02	51.55	80.64	88.24	66.58
MS-COCO	46.70	76.36	85.68	52.55	81.08	88.43	71.80
COCO-CFs	54.39	83.35	90.27	57.97	85.11	91.38	77.08
Table 5: Image-text retrieval performance on a withheld COCO-CFs test set.
A.2 Analysis of Differences in OOD Generalization on Image Recognition Datasets
To better understand the differences in OOD generalization performance across datasets, we measured the frequency in which the altered subjects used to produce COCO-Counterfactuals overlapped with class labels. Specifically, we define the COCO-CFs Label Frequency for each image recognition dataset as the total number of COCO-Counterfactuals in which one or more of the dataset's labels matched one of the two altered subjects used to produce the counterfactual pair.
Table 6 provides the COCO-CFs Label Frequency for each image recognition dataset along with the change in OOD performance relative to pre-trained CLIP after training on various sizes of COCO-CFs (see Appendix B.3.1 for a definition of dataset sizes). We observe that datasets having a higher COCO-CFs Label Frequency generally achieve larger improvements in OOD generalization
<sup>12</sup>Note that the image-text retrieval performance of the three pre-trained models (CLIP, BridgeTower, and Flava) on the in-domain COCO-Counterfactuals test set in Table 5 are higher than the respective values on the entire COCO-Counterfactuals dataset provided in Tables 2 and 13. This is expected because the retrieval space of the in-domain COCO-Counterfactuals test set is only  $30\%$  of the entire COCO-Counterfactuals dataset.
IR Dataset	COCO-CFs Label Frequency	COCO-CFsbase Δ	COCO-CFsmedium Δ	COCO-CFsall Δ
CIFAR100	3446	2.50	2.63	1.80
Caltech101	354	2.31	2.55	2.45
Caltech256	744	1.78	1.52	1.16
CIFAR10	398	0.65	0.36	-0.29
ImageNet	887	0.41	-0.03	-0.37
Food101	28	-1.04	-2.05	-2.11
Table 6: Frequency of class label occurrence in COCO-CFs and absolute change  $(\Delta)$  in performance relative to pre-trained CLIP after training on various sizes of COCO-CFs
Error category	% present in sampled COCO-CFs
Failure to generate subject/object	27%
Failure to generate fine-grained details	23%
Hyponymy relationship between altered subjects	15%
Human annotation error	15%
Failure to accurately depict spatial relationships	7%
Failure to generate correct number of objects	6%
Both altered subjects are present in the image	4%
Failure to bind attribute	3%
Table 7: Image-text retrieval performance on the in-domain COCO-CFs test set.
performance. The Pearson correlation coefficient between COCO-CFs Label Frequency and the 18 performance change measurements in Table 6 is 0.522 with a p-value of 0.026, indicating statistically significant positive correlation.
These results suggest that a major contributor to the variation in OOD generalization performance across datasets is the overlap between the evaluation dataset domain and the set of subjects which are altered in COCO-Counterfactuals. Food101, the only dataset which saw no improvement in performance on our best-performing COCO-CFs training dataset, had only 28 cases of overlap between its label set and the subject alterations in COCO-CFs. In contrast, the greatest performance improvements were achieved on CIFAR100, for which 3446 COCO-CFs had subject alterations matching at least one label from the dataset. These findings point to the potential usefulness of targeting counterfactual changes for task-specific datasets.
A.3 Analysis of Errors in COCO-Counterfactuals Identified by Human Annotators
In this section, we analyze errors in COCO-Counterfactuals using the labels assigned by human annotators (Section 4.1). Specifically, we consider an error to be any image-text pair from the COCO-Counterfactuals dataset for which the human annotator did not select the correct caption for the corresponding image.
A.3.1 Manual Categorization of Errors
To investigate potential failure cases in our counterfactual generation approach, we randomly sampled and categorized 100 image-text pairs which were identified as errors by the human annotators. Table 7 provides the percentage of sampled COCO-Counterfactuals which were assigned to various error categories. Additionally, Tables 8 and 9 provide examples of counterfactual pairs which were assigned to the top-six most frequent error categories.
We found that  $66\%$  of the sampled errors can be attributed to known limitations of existing text-to-image diffusion models (Chefer et al., 2023; Samuel et al., 2023; Cho et al., 2022), which include the categories for failure to generate a subject or object (e.g., Table 8, row 1), failure to generate fine-grained details (e.g., Table 8, row 2), failure to accurately depict spatial relationships (e.g., Table 9, row 2), failure to generate the correct number of objects described in the prompt (e.g., Table 9, row 3), and failure to bind attributes such as color.
Original	Counterfactual
Failure to generate subject/object
A cat walking through a kitchen by a eating tray	A cat walking through a field by a eating tray.
Failure to generate fine-grained details
A man playing Wii in a dirty room	A kid playing Wii in a dirty room
Hyponymy relationship between altered subjects
Two kids in pink and purple jackets standing by a fence	Two girls in pink and purple jackets standing by a fence
Table 8: Examples of failure cases identified by manual error analysis
Original	Counterfactual
Human annotation error	Two people dressed in red skiing across a snowy landscape	Two people dressed in red race across a snowy landscape
Failure to accurately depict spatial relationships	A woman lies on the ground under a suitcase.	A man lies on the ground under a suitcase.
Failure to generate correct number of objects	A bathroom sink with two toothbrush holders on it	A bathroom sink with two cup holders on it
Table 9: Additional examples of failure cases identified by manual error analysis
Altered Subjects	Count	Altered Subjects	Count	Altered Subjects	Count
woman → girl	126	man → boy	125	people → men	116
person → man	93	person → woman	42	person → boy	37
couple → group	36	people → guy	35	people → kid	33
person → girl	33	girl → woman	32	man → woman	30
men → people	29	people → student	27	woman → man	24
man → person	24	building → house	23	men → boy	21
women → girl	21	boy → man	21