SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tiep Le*
tiep.le@intel.com
Phillip Howard*
phillip.r.howard@intel.com
Counterfactual examples have proven to be valuable in the field of natural language processing (NLP) for both evaluating and improving the robustness of language models to spurious correlations in datasets. Despite their demonstrated utility for NLP, multimodal counterfactual examples have been relatively unexplored due to the difficulty of creating paired image-text data with minimal counterfactual changes. To address this challenge, we introduce a scalable framework for automatic generation of counterfactual examples using text-to-image diffusion models. We use our framework to create COCO-Counterfactuals, a multimodal counterfactual dataset of paired image and text captions based on the MS-COCO dataset. We validate the quality of COCO-Counterfactuals through human evaluations and show that existing multimodal models are challenged by our counterfactual image-text pairs. Additionally, we demonstrate the usefulness of COCO-Counterfactuals for improving out-of-domain generalization of multimodal vision-language models via training data augmentation. We make our code<sup>2</sup> and the COCO-Counterfactuals dataset<sup>3</sup> publicly available.
While vision and language models have achieved remarkable performance improvements in recent years, out-of-domain (OOD) generalization remains a challenge for even the best models, which typically exhibit much lower performance in zero-shot evaluation settings than on withheld in-domain test sets. This has often been attributed to spurious correlations between non-causal features and labels in datasets which can be exploited during training as shortcuts to achieving artificially high in-domain performance (Geirhos et al., 2020). For example, image recognition models often learn to utilize spurious features in the backgrounds of images when trained for classification on datasets such as ImageNet (Singla and Feizi, 2021; Xiao et al., 2020).
Augmenting training datasets with counterfactuals, which study the impact on a response variable following a change to a causal feature, has been previously proposed as a strategy for countering this effect in NLP models (Levesque et al., 2012; Kaushik et al., 2019). Motivated by concepts in causal learning (Feder et al., 2022), these methods typically form counterfactual examples by making minimal edits to an input text such that a corresponding label or attribute of the text (e.g., sentiment)
$^{2}$ https://github.com/IntelLabs/multimodal_cognitive.ai/tree/main/COCO-Counterfactuals
<sup>3</sup>https://huggingface.co/datasets/Intel/COCO-Counterfactuals
"Two men work in the kitchen of a restaurant."
"Two men work in the kitchen of a house."
"A large black ball sitting next to a glass of milk."
"A large black ball sitting next to a glass of water."
"A white dog sitting on a table next to a paper."
"A white dog sitting on a table next to a book."
"A meal of sandwiches, potatoes, and Red Stripe beer."
"A meal of sandwiches, pizza, and Red Stripe beer."
Figure 1: Examples of COCO-Counterfactuals, our minimal-edit counterfactuals dataset for images with paired text captions.
is changed. Training models with counterfactual examples therefore provides a strong inductive bias against learning spurious correlations in datasets, leading to greater robustness and improved generalization on OOD data (Eisenstein, 2022; Vig et al., 2020) as well as enabling better domain adaptation in low resource settings (Calderon et al., 2022).
Despite its success in the realm of NLP, the application of counterfactual data augmentation to multimodal vision-language models has largely remained unexplored, mainly due to low-resource settings involving multimodal data and challenges associated with creating paired counterfactual examples spanning multiple modalities. For example, consider the task of creating counterfactual examples for a multimodal dataset containing images with associated text captions. Creating a counterfactual to a given image-text example requires not only minimally editing a casual feature in the text caption, but also making a corresponding minimal edit to the image which ideally modifies only the changed causal feature while preserving other spurious features from the original image.
Collecting such counterfactual examples from existing image datasets is infeasible due to the massive variation in natural images that can accurately depict even identical text captions. While manual creation of counterfactual examples by humans is an option that has been employed previously for NLP (Kaushik et al., 2019; Gardner et al., 2020), this approach suffers from a lack of scalability due to the high cost of human labor, which would be compounded even further for multimodal counterfactuals due to the need for both text and image editing skills. Given these challenges, how can paired image-text counterfactual examples be created at the scale needed for effective model evaluation and data augmentation?
We address this problem by introducing a novel data generation pipeline for automatically creating multimodal counterfactual examples using text-to-image diffusion models. Our approach minimally edits captions from an existing image-text dataset and then leverages Stable Diffusion (Rombach et al., 2021) with cross-attention control (Hertz et al., 2022) to generate pairs of images with minimal differences (i.e., isolated to the counterfactual change). We employ our data generation pipeline to create at scale COCO-Counterfactuals (Figure 1), a counterfactual variant of the MS-COCO dataset (Lin et al., 2014).
We validate the quality of COCO-Counterfactuals using human evaluations and conduct zero-shot experiments showing that state-of-the-art multimodal models are challenged by our generated counterfactual examples. Our additional experiments show that training CLIP (Radford et al., 2021)
on COCO-Counterfactuals improves its performance on multiple OOD datasets, including zero-shot tasks not seen during training. We make COCO-Counterfactuals and the code for our counterfactual data generation pipeline publicly available under the CC BY 4.0 License.
2.1 Counterfactual Examples for NLP
Counterfactual data augmentation has been shown to improve the robustness of models across a wide range of problem domains in NLP. Kaushik et al. (2019) demonstrated that human-authored counterfactuals pose a significant challenge for existing models and that augmenting training datasets with counterfactual examples improves sentiment analysis and Natural Language Inference classifiers. Gardner et al. (2020) similarly used human experts to author minimally-edited contrast example sets for 10 NLP datasets and showed that model performance evaluated on them drops substantially.
A number of approaches have been proposed to move beyond reliance on human authors towards automated methods for generating counterfactual examples. Wang and Culotta (2021) and Yang et al. (2021) automatically construct counterfactual examples by identifying and removing or replacing potentially causal words. Howard et al. (2022) introduce a framework for generating looser counterfactuals which allow larger edits of original examples, resulting in more natural and linguistic counterfactual examples. Other semi-automated methods have been proposed to generate counterfactual examples while still relying on human input or labeling (Wu et al., 2021). To the best of our knowledge, none of these existing approaches for automatic counterfactual generation have been extended to multimodal image-text datasets.
2.2 Image Benchmarks for Measuring Spurious Correlations
Several image datasets have been proposed as benchmarks for measuring the degree to which models have learned to rely on spurious correlations during training. CelebA Hair Color (Liu et al., 2015) is a binary image classification dataset that labels whether a person depicted has a blonde hair color, which is spuriously correlated with gender. Sagawa et al. (2019) constructed the Waterbirds dataset by cropping images of landbirds or seabirds onto land and sea backgrounds, resulting in a binary classification task for bird type (i.e., landbird or seabird) in the presence of spurious correlations with the background. Colored MNIST (Arjovsky et al., 2019) artificially imposes colors on the MNIST handwritten digits dataset, where the color is spuriously correlated with the class label. Lynch et al. (2023) uses text-to-image models to generate Spawrious, an image classification dataset of four dog breeds spuriously correlated with six background locations. Unlike COCO-Counterfactuals, these datasets are limited only to image classification over a small number of labels and are primarily suited for evaluating model robustness as opposed to training data augmentation.
Thrush et al. (2022) introduced Winoground, an image-text dataset aimed at measuring visio-linguistic compositionality. Given two images and two captions which have the same words but in different order, the task is to correctly match each caption to its corresponding image. While their paired image-text examples can be viewed as counterfactuals, they focus only on edits to word order and rely on humans to create a dataset aimed specifically at evaluating compositionality. In contrast, our method automatically generates counterfactual examples with word content changes while also preserving non-causal spurious features across paired counterfactual images.
FOIL-COCO (Shekhar et al., 2017) contains 'foil' captions with a single change to the original MS-COCO caption to invalidate it for the accompanying image. They show that vision and language models struggle to correctly classify captions, detect the edited word, and correct the foiled caption. Our image-text counterfactuals similarly create 'foil' captions to MS-COCO captions, but goes further by also creating paired images which differ only according to how the caption was edited.
2.3 Data Augmentation with Synthetic Images Generated from Text-to-image Models
Motivated by recent advances in text-to-image diffusion models (Nichol et al., 2021; Rombach et al., 2021; Sahara et al., 2022; Ramesh et al., 2022), data augmentation with synthetically-generated images has emerged as a growing topic of interest. He et al. (2022) showed that images generated by GLIDE (Nichol et al., 2021) for specific classes in image recognition datasets can be used for
training to improve performance on the corresponding image classification tasks. Trabucco et al. (2023) perform image-to-image transformations for data augmentation using text-to-image diffusion models, observing improvements in few-shot image classification performance. Vendrow et al. (2023) represent class labels from image recognition datasets as custom tokens in the vocabulary of a text-to-image diffusion model, enabling them to generate images of objects from the original dataset under different domain shifts. While our data generation pipeline also leverages text-to-image diffusion models, our approach differs from prior work in our focus on producing minimal changes to paired image-text data in both the vision and language modalities.
3 COCO-Counterfactuals
We detail our data generation methodology for creating COCO-Counterfactuals (COCO-CFs), a synthetic multimodal counterfactual dataset of paired image and text captions based on the MS-COCO dataset (Lin et al., 2014). While we showcase our methodology by generating and releasing the COCO-Counterfactuals dataset, our approach can be applied to automatically construct multimodal counterfactuals for any dataset containing image captions.
3.1 Creating Counterfactual Captions
Given an original image caption  $C_o$ , our first task is to create a corresponding counterfactual caption  $C_c$  which alters a subject of  $C_o$  while preserving most of its original details. The altered subject represents the changed causal feature in our counterfactual example while the remaining preserved details from the original caption can be viewed as potentially spurious correlated features.
To alter a subject of  $C_o$ , we first identify all nouns using NLTK (Bird et al., 2009) as candidate words for substitution<sup>5</sup> For each of the  $i \in \{1,..,n\}$  identified nouns, we create 10 candidate counterfactual captions by replacing only the  $i$ -th noun in  $C_o$  with the [MASK] token and retrieving the top-10 most probable replacements via masked language modeling (MLM)<sup>6</sup>. This produces a total of  $n \times 10$  candidate counterfactual captions, which we then filter to retain only those in which the substituted word is also a noun.
Our aim is to substitute nouns with alternative words that represent different subjects, and yet still maintain ontological similarity to the original noun. Hence, we use a pre-trained sentence similarity model<sup>7</sup> to measure the similarity between each candidate counterfactual caption and the original caption  $C_o$ , keeping only those candidates which have a sentence similarity within the range (0.8, 0.91). Finally, we use GPT-2 to score the perplexity of all candidates which remain after filtering and choose the candidate having the lowest perplexity as our counterfactual caption  $C_c$ .
3.2 Generating Counterfactual Images
After creating a counterfactual caption  $C_c$ , our next task is to generate synthetic images  $I_o^s$  and  $I_c^s$  from the original caption  $C_o$  and counterfactual caption  $C_c$  (respectively). Ideally we would like  $I_o^s$  and  $I_c^s$  to differ only in terms of the noun which was replaced in  $C_o$  to produce  $C_c$ , thereby enabling the changed causal feature to be learned in the presence of other potentially spurious correlated features (i.e., the unchanged details between  $C_o$  and  $C_c$ ). However, this is a challenge for existing text-to-image generation models as minor changes to a text prompt can produce significantly different images. For instance, prompting Stable Diffusion with the captions "A small child lounges with a remote in his hand" and "A small child lounges with a toy in his hand" may produce images that differ not only in the object that the child is holding, but also in other details such as his facial features, the manner in which he is laying, the color of his clothes, and the image background.
To address this issue, Hertz et al. (2022) proposed a methodology called Prompt-to-Prompt which injects cross-attention maps during the diffusion process to control the attention between certain pixels and tokens of the prompt during denoising steps. This enables separate generations from
4 Appendix B.1 details hyper-parameters and pre-trained models used to generate COCO-Counterfactuals.
In this work, we focus on counterfactual captions that are derived from altering a noun from original captions. We leave the investigation of altering words of other types such as verbs and adjectives for future work.
<sup>6</sup>We use RoBERTa-base for MLM
<sup>7</sup>https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2
Figure 2: Examples of COCO-Counterfactuals generated with Prompt-to-Prompt (left) and without (right). Prompt-to-prompt enables us to extend the principle of minimal-edit text counterfactuals to the visual domain, isolating image differences to only the changed causal feature.
text-to-image diffusion models to maintain many of the same image details while isolating their differences according to how the text prompts differ. An example of counterfactual image-text pairs  $(C_{o},I_{o}^{s})$  and  $(C_c,I_c^s)$  generated with and without prompt-to-prompt is shown in Figure 2, illustrating how Prompt-to-Prompt enables the principle of minimal text edits for NLP counterfactuals to be extended to image generation.
Brooks et al. (2023) noted that making different changes to images may require varying the parameter  $p$  in Prompt-to-Prompt, which controls the number of denoising steps with shared attention weights. For example, changes that require more substantial structural modifications to the image may necessitate less overall similarity between the resulting images and thus fewer shared attention weights. We therefore adopt their proposed approach of over-generating 100 image pairs with Prompt-to-Prompt by randomly sampling values of the parameter  $p \sim U(0.1, 0.9)^8$ . The resulting 100 image pairs are filtered using CLIP (Radford et al., 2021) to ensure a minimum cosine similarity of 0.2 between the encoding of each caption and its corresponding generated image, with the best image pair  $(I_o^s, I_c^s)$  chosen from those which remain according to the directional similarity in CLIP space (Gal et al., 2022):
$$
\mathrm {C L I P} _ {d i r} = \frac {\left(E _ {T} \left(C _ {c}\right) - E _ {T} \left(C _ {o}\right)\right) \cdot \left(E _ {I} \left(I _ {c} ^ {s}\right) - E _ {I} \left(I _ {o} ^ {s}\right)\right)}{| | E _ {T} \left(C _ {c}\right) - E _ {T} \left(C _ {o}\right) | | | | E _ {I} \left(I _ {c} ^ {s}\right) - E _ {I} \left(I _ {o} ^ {s}\right) | | |} \tag {1}
$$
where  $E_{T}$  and  $E_{I}$  are CLIP's text and image encoders (respectively). The  $\mathrm{CLIP}_{dir}$  metric measures the consistency in changes between the two images  $(I_o^s, I_c^s)$  and their corresponding captions  $(C_o, C_c)$ . Thus, selecting images with a higher  $\mathrm{CLIP}_{dir}$  improves the overall quality of our generated counterfactuals via greater consistency between the alterations made in both modalities.
3.3 Generating COCO-Counterfactuals from MS-COCO
We apply our counterfactual caption and image generation pipeline described above to create the COCO-Counterfactuals dataset. Specifically, we generate candidate counterfactual captions for 25,014 original MS-COCO captions $^{910}$ , keeping only the best candidate counterfactual for each original caption that meets our filtering criteria. This produced a total of 24,508 original & counterfactual caption pairs  $(C_{o}, C_{c})$  after filtering and selection. Our image over-generation pipeline produced 2.45 million candidate image pairs  $(I_{o}^{s}, I_{c}^{s})$  for these 24.5k caption pairs, of which 17,410 had at least one generated image pair which met our filtering criteria. After selection according to the CLIP_dir metric, a total of 34,820 image-caption pairs remain, comprising our COCO-Counterfactuals dataset $^{11}$ .
<sup>8</sup>We use the implementation from Instruct-Pix2Pix (Brooks et al., 2023).
<sup>9</sup>We use the 5K validation split of the 2017 dataset from https://cocodataset.org/#download.
10While we use MS-COCO in this study as the source of our original captions, one advantage of our counterfactual generation approach is that the input dataset itself does not require paired image and text data.
11While the MS-COCO 5K validation split has 25,014 captions, COCO-Counterfactuals includes only 17,410 of them due to our filtering criteria. Thus, for a fair comparison in our experiments, hereafter we refer to this subset of the 5K validation split including only those 17,410 captions and their paired original images as the MS-COCO dataset.
Image set	Correct	Incorrect	Neither	Both
Generated from original caption	79.10%	8.16%	10.18%	2.56%
Generated from counterfactual caption	67.27%	18.43%	10.74%	3.56%
All images	73.18%	13.30%	10.46%	3.06%
Table 1: Human evaluation results for COCO-Counterfactuals
4 COCO-Counterfactuals Analysis
This section aims to show that, in addressing the challenges associated with low-resource settings involving multimodal data (see Section 1), our proposed novel data generation pipeline can serve as an efficient and scalable framework to automatically create high quality multimodal counterfactual examples in COCO-Counterfactuals (COCO-CFs). Toward this goal, we first employ human evaluation to analyze COCO-CFs. We then show that COCO-CFs can be used as a challenging dataset for model evaluation on zero-shot image-text retrieval and image-text matching tasks.
4.1 Human Evaluation of COCO-Counterfactuals
We employ professional data annotators to conduct a human study on the quality of COCO-CFs. For each of the 34,820 images in COCO-CFs, we have at least one annotator choose whether the corresponding original or counterfactual caption best fits the image. Annotators can also choose "both" if both captions describe the image equally well, or "neither" if neither caption accurately describes the image.  $10\%$  of the images are labeled by 3 different individuals to estimate inter-annotator agreement, with the remaining images each labeled by a single annotator (see Appendix B.2 for additional details).
Table 1 provides the percentage of images from COCO-CFs which were matched to their correct caption by the human annotators. We also report the percentage of incorrect matches (i.e., the wrong caption was picked as best describing the image) as well as the percentage of "both" and "neither" labels. Overall,  $73\%$  of images were correctly matched to their corresponding caption (see Appendix A.3 for an analysis of incorrect matches). Images generated from the counterfactual caption had a  $10\%$  greater incidence of incorrect caption selections than those generated from the original caption. This could be due to the constraints imposed on the counterfactual image by Prompt-to-Prompt (i.e., shared attention weights with the original image), which increases the likelihood that the generated image lacks some of the details in its corresponding caption.
The Fleiss' kappa coefficient for the  $10\%$  of images labeled by three annotators was 0.74, indicating strong agreement among the annotators who participated in this study. Among those images which had label disagreement,  $47.4\%$  of the labels were correct,  $27.3\%$  were incorrect, and  $18.6\%$  selected "neither." This suggests that many of the disagreements are associated with images for which the correct caption choice is more ambiguous.
While we employed human annotators to validate the quality of COCO-Counterfactuals for this analysis, our automated counterfactual generation approach does not require the use of human annotators to produce a new dataset. Indeed, our experiments described subsequently in Section 5.1 show that COCO-Counterfactuals which were labeled as incorrect by humans have no negative impact on training data augmentation.
4.2 COCO-Counterfactuals for Model Evaluation
Motivated by prior work which has proposed using counterfactuals as challenging test sets in NLP (Kaushik et al., 2019; Gardner et al., 2020), we further investigate whether our COCO-CFs can serve a similar purpose for state-of-the-art multimodal vision-language models such as CLIP, Flava (Singh et al., 2022), BridgeTower (Xu et al., 2022) and ViLT (Kim et al., 2021) for the zero-shot image-text retrieval and image-text matching tasks. We employed the HuggingFace implementations of these model in our experiments (see Appendix A.6 for more detail).
HuggingFace Pre-trained Models	Evaluated Dataset	Text Retrieval	Image Retrieval
R@1	R@5	R@10	R@1	R@5	R@10
bridgetower-large-itm-mlm-itic	COCO-CFs	21.72 (-51%)	46.94 (-35%)	58.65 (-29%)	17.93 (-47%)	38.94 (-35%)	49.95 (-30%)
human-evaluated COCO-CFs	26.36 (-41%)	54.1 (-25%)	66.13 (-20%)	21.44 (-37%)	45 (-25%)	56.39 (-21%)
flava-full	COCO-CFs	21.28 (-57%)	46.64 (-41%)	58.87 (-34%)	37.76 (-16%)	66.15 (-12%)	75.83 (-10%)
human-evaluated COCO-CFs	26.1 (-47%)	54.23 (-31%)	66.83 (-25%)	43.4 (-3%)	72.35 (-3%)	81.44 (-4%)
Table 2: Image-text retrieval performance on COCO-CFs and human-evaluated COCO-CFs for BridgeTower and Flava models. Largest drops of performance against the baseline are in boldface.
Figure 3: ITM score differences computed for existing multimodal models on COCO-Counterfactuals dataset.
4.2.1 Zero-shot Image-text Retrieval
We evaluate the zero-shot image-text retrieval (ITR) performance of pre-trained Flava and BridgeTower models on COCO-CFs as well as human-evaluated COCO-CFs, which consists of only image-text pairs that were correctly matched in our human evaluation study (Section 4.1). Since pre-trained CLIP was employed in our counterfactual image generation process (see Section 3.2), it is not suitable for zero-shot ITR setting. Thus, we only report its evaluation in Appendix. A.6 for completeness. For baselines, we evaluate ITR performance of these models on the MS-COCO dataset.