SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 10: Frequency of altered subjects which appeared at least 20 times in errors identified by human annotators
In many cases, these failures do not negatively impact the depiction of the counterfactual change in the two images because the inaccuracies pertain to details other than the altered subjects. For example, the first row of Table 8 shows the counterfactual pair associated with an image which was categorized as a failure to generate a subject/object; in this case, the altered subjects (kitchen  $\rightarrow$  field) are depicted correctly, but both images lack the eating tray described in the prompt. Similarly, the counterfactual pair shown in the second row of Table 8 lacks fine-grained details in the prompt (e.g., dirty room), but still depicts the altered subjects correctly (man  $\rightarrow$  kid).
We found that  $15\%$  of the sampled errors could be attributed to a hyponymy relationship between the altered subjects which caused both captions to be equally valid for a given image. For example, the third row of Table 8 shows a counterfactual pair where the counterfactual image was incorrectly labeled by the human annotator because both captions were valid descriptions of the image (i.e., girls can also be referred to as kids). Nevertheless, this example is still a valid counterfactual pair considering that the counterfactual caption does not accurately describe the original image and is more descriptive of the counterfactual image than the original caption.
An additional  $15\%$  of the sampled errors appeared to be valid image-text pairs without any significant deficiencies. We therefore concluded that such cases were human annotation errors (see Table 9 row 1 for an example). Finally,  $4\%$  of the sampled images had equally valid caption choices because both of the altered subjects appeared in the image that was annotated.
The results of this error analysis suggest that the quality of counterfactuals produced by our approach may improve as the capabilities of text-to-image diffusion models advance. New models which overcome known limitations of existing models could be used as a substitute for Stable Diffusion in our approach to produce higher-quality counterfactuals. Additionally, errors associated with hyponymy relationships could be addressed in future work through a refinement of our subject alteration process. For example, ontologies could be used to avoid noun substitutions where it can be determined that a hyponymy relationship exists between the noun candidates. Finally, additional constraints on the image generation process could be explored to prevent both altered subjects from appearing in the same image.
A.3.2 Taxonomic Analysis of Errors
To better understand the relationship between the altered subjects in our counterfactuals and potential failure cases, we conducted a taxonomic analysis of the altered subjects which occurred most frequently among errors identified by human annotators. Table 10 provides the frequency of altered subject pairs which occurred at least 20 times in the error cases identified by human annotators. Interestingly, we observe that 19 of these 20 most frequent altered subject pairs belong to the human taxonomy.
We further analyzed this human taxonomy in COCO-Counterfactuals by constructing a list of human-related words, which consists of 'girl', 'boy', 'man', 'men', 'woman', 'guy', 'kid', 'person', 'people', 'child', 'children', 'couple', 'group', and 'lady'. An image-text pair is said to be related to this human taxonomy if the altered subject of its caption belong to this list. We find that there are 4117 image-text pairs in COCO-Counterfactuals that are related to the human taxonomy, among which 1864 were identified as errors by human annotators. The corresponding error rate for altered subjects related to the human taxonomy is  $44.3\%$ , which indicates that generating counterfactual pairs involving human altered subjects is more challenging for our approach. This suggests that a
Training dataset	|Dtrain|	|DFtrain|	Text Retrieval	Image Retrieval	Mean
R@1	R@5	R@10	R@1	R@5	R@10
MS-COCO + COCO-CFs	34,313	20,385	75.91	93.95	96.90	77.66	94.51	97.20	89.36
Table 11: Mean image-text retrieval performance on the OOD Flickr30k test set using only COCO-Counterfactuals which were correctly labeled by humans, measured across 25 different random seeds.
Training dataset	|Dtrain|	|DFtrain|	Text Retrieval	Image Retrieval	Mean
R@1	R@5	R@10	R@1	R@5	R@10
None (pre-trained CLIP)	0	0	50.12	75.04	83.6	30.73	56.28	67.18	60.49
MS-COCO	13,928	0	57.330.3	81.280.2	88.710.2	41.130.1	68.460.1	78.450.1	69.230.1
MS-COCO + COCO-CFs	13,928	6,939	56.910.3	80.700.2	87.820.2	39.920.1	67.010.1	77.150.1	68.250.1
MS-COCO + COCO-CFs	34,820	20,894	58.060.3	81.390.2	88.910.2	41.630.2	68.640.1	78.850.1	69.580.1
MS-COCO + COCO-CFs	41,784	27,853	58.020.3	81.390.2	88.780.2	41.820.1	68.790.1	78.890.1	69.620.1
Table 12: Image-text retrieval performance on the in-domain MS-COCO test set. All other settings are identical to Table 3.
promising direction for future work is the exploration of improvements to the generation of images involving human subjects.
A.4 Training Data Augmentation with Only Correctly-annotated COCO-Counterfactuals
We investigate the potential impact of COCO-Counterfactuals which were incorrectly labeled by humans on training data augmentation. Table 11 provides the OOD image-text retrieval performance in this setting, where COCO-Counterfactuals were filtered to only include those which were correctly labeled by the human annotators. Overall we find similar performance as our previous experiments using the full COCO-Counterfactuals dataset (Table 3), suggesting that filtering our synthetic data using human evaluations is not necessary for data augmentation applications.
A.5 COCO-Counterfactuals Improve In-domain Performance
We evaluate the same models trained with counterfactual data augmentation described in Section 5 on the MS-COCO test set. The results of this in-domain evaluation are provided in Table 12. Similar to the OOD image-text retrieval setting, we find that data augmentation with 20,892 COCO-Counterfactuals provides statistically significant performance improvements relative to training without counterfactual data augmentations. Notably, previous work has observed that counterfactual data augmentation can degrade performance on withheld in-domain test sets (Wang and Culotta, 2021; Howard et al., 2022), whereas data augmentation with our COCO-Counterfactuals actually increases in-domain performance on MS-COCO.
A.6 COCO-Counterfactuals for Model Evaluation Experiments
We further investigate whether our COCO-Counterfactuals (COCO-CFs) can serve as a challenging test set for state-of-the-art multimodal vision-language models such as CLIP, Flava (Singh et al., 2022), BridgeTower (Xu et al., 2022) and ViLT (Kim et al., 2021) for the zero-shot image-text retrieval and image-text matching tasks. We employed the following HuggingFace implementations of these models via the transformers library:
- CLIP: We used the pre-trained model clip-vit-base-patch32
- Flava: We used the pre-trained model flava-full
- BridgeTower: We used the pre-trained model bridgetower-large-item-mlm-ito
ViLT: We used the pre-trained model vilt-b32-finetuned-coco
Zero-shot Image-text Retrieval. In Section 4, we evaluated the zero-shot image-text retrieval (ITR) performance of pre-trained Flava and BridgeTower models on COCO-CFs and human-evaluated
HuggingFace Pre-trained Models	Evaluated Dataset	Text Retrieval	Image Retrieval
R@1	R@5	R@10	R@1	R@5	R@10
Clip	COCO-CFs	37.65 (-21%)	64.89 (-9%)	74.57 (-7%)	34.98 (+5%)	62.29 (+7%)	72.43 (+4%)
human-evaluated-COCO-CFs	43.25 (-9%)	70.4 (-2%)	79.37 (-1%)	40.14 (+21%)	67.86 (+16%)	77.66 (+11%)
Table 13: Image-text retrieval performance on COCO-CFs and human-evaluated COCO-CFs for CLIP model. Largest drops of performance against the baseline are in boldface.
COCO-CFs that consists of only image-text pairs that were correctly matched in human evaluation in Section 4.1. Since a pre-trained CLIP model was employed in our counterfactual image generation process (see Section 3.2), CLIP models are not suitable for the zero-shot ITR evaluation. Hence, we only report evaluation of pre-trained CLIP model for ITR task here for completeness.
Table 13 reports ITR performance (i.e., Recall at 1, 5, and 10) on COCO-CFs and human-evaluated-COCO-CFs for the pre-trained CLIP model. Similar to Table 2, the percentages enclosed within parentheses indicate the change in performance of the CLIP model on an evaluated dataset versus the performance of that model on MS-COCO (baseline).
We observe that on both COCO-CFs and human-evaluated-COCO-CFs datasets, while the performance of the pre-trained CLIP model degrades marginally on Text Retrieval task, its performance increases for Image Retrieval task. We attribute this to potential data contamination due to how we employed a pre-trained CLIP model in our counterfactual image generation process (see Section 3.2). As a result, COCO-Counterfactuals includes image-text pairs for which CLIP achieves high image-text retrieval performance.
B Dataset and Experiment Details
B.1 Hyper-parameter Selection and Models Used to Generate COCO-Counterfactuals
In this section, we will detail hyper-parameters and pre-trained models used to our generate COCO-Counterfactuals dataset.
B.1.1 Creating Counterfactual Captions
Given an original caption from the MS-COCO dataset, we use Natural Language Toolkit (NLTK) (Bird et al., 2009) modules:
- punkt for sentence tokenizer, and
- averaged_perceptron_tagger for part-of-speech (POS) tagger
to identify all nouns as candidate words for substitution.
For each of the identified nouns, we create 10 candidate counterfactual captions by replacing only one noun with the [MASK] token and retrieving the top-10 most probable replacements via masked language modeling (MLM). For MLM, we used the pre-trained model roberta-base (Liu et al., 2019) implemented in the library transformers (Wolf et al., 2019)
A motivation for our use of noun substitutions is the desire to produce minimal-edit counterfactuals. This is a common strategy for NLP counterfactuals (Kaushik et al., 2019; Wang and Culotta, 2021; Yang et al., 2021) because the high degree of similarity between the original and counterfactual text preserves spurious correlations that models might rely on for discernment. Furthermore, our use of perplexity filtering mitigates the potential for such word substitutions to produce unrealistic counterfactual captions.
In order to measure similarity between each candidate counterfactual caption and an original caption, we used the pre-trained model all-MiniLM-L6-v2, which is implemented within the library sentence-transformers (Reimers and Gurevych, 2019).
Among generated candidate counterfactual captions, we kept only those candidates which have a sentence similarity within the range (0.8, 0.91). We selected this similarity range heuristically, observing that it produced best results after extensive experimentation.
Finally, we employed the pre-trained model gpt2-large, a GPT-2 (Radford et al., 2018) model implemented in the transformers library, to score the perplexity and choose the candidate having the lowest perplexity as our counterfactual caption.
B.1.2 Generating Counterfactual Images
After creating a counterfactual caption, our next task is to generate synthetic images from the corresponding original caption and counterfactual caption, respectively. In order to do so, we have adopted an implementation from Instruct-Pix2Pix (Brooks et al., 2023) in which all hyper-parameters are set to their default values.
Specifically, we over-generate 100 image pairs with Prompt-to-Prompt by randomly sampling values of the parameter  $p \sim U(0.1, 0.9)$  (i.e., parameter  $p$  indicates the portion of denoising for which to fix self attention maps). The resulting 100 image pairs are filtered using CLIP (Radford et al., 2021) to ensure:
i. a minimum cosine similarity of 0.2 between the encoding of each caption and its corresponding generated image, and
ii. a minimum cosine similarity of 0.7 between the encoding of the two respective images in each generated image pair.
From remaining image pairs, the best image pair is chosen such that it has the highest directional similarity  $CLIP_{dir}$  score. Selecting images with the highest  $CLIP_{dir}$  improves the overall quality of our generated counterfactuals via greater consistency between the alterations made in both modalities.
B.2 Human Annotation Study
Professional annotation services for our human study were provided by Mindy Support. The total cost of this study was \(1068.59 for 218 annotation hours. The instructions provided to annotators are depicted in Figure 4. We are unable to provide the hourly wages paid to workers as this is considered proprietary information by Mindy Support. However, the following statement was provided by the vendor regarding compensation:
"We prioritize compliance with all standards of local and international legislation, ensuring fair treatment and equal opportunities for individuals of various backgrounds, ages, and other characteristics. We are committed to upholding the principles of fair wages, non-discrimination, and labor standards, including the prohibition of child labor. As an organization, we strictly adhere to legal requirements and strive to create an inclusive and ethical working environment for all. Rest assured that our compensation rates reflect market demands and provide fair remuneration for the work performed by our participants. We remain dedicated to abiding by all labor regulations and social and economic standards."
B.3 Training Data Augmentation Experiments
In this section, we detail how we constructed our training datasets and how we finetuned the pretrained CLIP model for experiments described in Section 5.
B.3.1 Training Dataset Preparation
Our training data augmentation experiments utilize various combinations of the MS-COCO validation set and our COCO-Counterfactuals dataset. For simplicity, a caption-image pair is referred to as a sample. We define a counterfactual sample as following. Given a sample  $(C,I)$  (i.e., caption  $C$  and image  $I$ ) from our COCO-Counterfactuals dataset, a sample  $(C',I')$  from COCO-Counterfactuals dataset is called a counterfactual sample of  $(C,I)$  iff  $C'$  and  $C$  are counterfactual captions of each other. By this definition, COCO-Counterfactuals dataset includes 34,820 samples that correspond to 17,410 paired counterfactual samples.
For experiments in Section 5, we have prepared the following 4 datasets:
Select the caption which best describes the image. In cases where both captions are valid for the image, please try to pick the one which is more descriptive or detailed. If both captions are valid and describe the image equally well, select "Both". If neither of the captions accurately describe the image, select "Neither".
Figure 4: Instructions provided to data annotators
A woman standing in a kitchen by a window
- A man standing in a kitchen by a window
Both
- Neither
(a.) MS-COCO dataset. This is a subset of the 5K validation split of the 2017 MS-COCO dataset $^{13}$ , achieved by filtering out all samples with captions which are not included in our COCO-Counterfactuals. This results in a dataset (referred to as the MS-COCO dataset used in experiments in Section 5) of 17,410 captions and their paired original images.
(b.) [MS-COCO + COCO-CFs]base dataset. This dataset is a combination of:
-  $50\%$  random sampling (i.e., 8,705 caption-image pairs) of the MS-COCO dataset constructed in  $(a)$ .
-  $25\%$  random sampling of paired counterfactual samples from our COCO-Counterfactuals dataset. This results in a total of 4,353 pairs of samples with their corresponding counterfactuals, for a total of 8,706 caption-image samples from our COCO-Counterfactuals dataset.
<sup>13</sup>https://cocodataset.org/#download
Overall, the [MS-COCO + COCO-CFs]  ${}_{\text{base }}$  dataset consists of 17,411 captions and their paired original images, which is approximately equal in size to the MS-COCO dataset constructed in (a.)
(c.) [MS-COCO + COCO-CFs]medium dataset. This dataset is a combination of:
- all samples (i.e., 17,410 caption-image pairs) from the MS-COCO dataset constructed in  $(a.)$ .
-  $75\%$  random sampling (i.e., 26,115 caption-image pairs) from our COCO-Counterfactuals dataset.
Overall, dataset [MS-COCO + COCO-CFs] medium consists of 43,525 captions and their paired original images.
(d.) [MS-COCO + COCO-CFs]all dataset. This dataset is a combination of:
- all samples (i.e., 17,410 caption-image pairs) from the MS-COCO dataset constructed in  $(a.)$ .
- all samples (i.e., 34,820 caption-image pairs) from our COCO-Counterfactuals dataset.
Overall, dataset [MS-COCO + COCO-CFs]  ${}_{\text{all }}$  consists of 52,230 captions and their paired original images.
Each of the datasets described above is split into a training set  $(80\%)$  and a validation set  $(20\%)$ . In each experiment, the validation set is used to pick the best model checkpoint at the conclusion of training. Tables 3, 4, and 12 report experimental results for models trained using the train split of these four datasets.  $|D_{train}|$  indicates the total number of samples (i.e., image-text pairs) included in the respective training set, while  $|D_{train}^{\mathrm{CF}}|$  indicates how many of those image-text pairs were sampled from the COCO-Counterfactuals dataset.
B.3.2 Finetuning CLIP with Data Augmentation
We use each of the four training sets constructed in Section B.3.1 to finetune the CLIP model clip-vit-base-patch32. We adopted a publicly-available finetuning script provided by HuggingFace $^{14}$ .
We repeat each of our training experiments with 25 different seeds and data_seed from the ranges [107, 131] and [108, 132], respectively. In each experiment, we use a learning rate to 5e-7, weight decay of 0.001, training batch size of 128, and evaluation batch size of 128.
B.4 Compute Infrastructure Used In this Study
Our experiments were conducted using an Intel AI supercomputing cluster comprised of Intel Xeon processors and 512 Intel Gaudi® AI accelerators, as well as an internal Slurm linux cluster with Nvidia RTX 3090 GPUs. Our dataset generation pipeline was parallelized across this compute infrastructure and took approximately 3 days to complete. Our training data augmentation experiments varied in running time depending upon the size of the dataset, ranging between 2 to 10 hours.
B.5 License Information of Assets Employed in This Study
- NLTK is open source software distributed under the terms of the Apache License Version 2.0.
- Transformers is released under the Apache License Version 2.0 and is available on GitHub at https://github.com/huggingface/transformers.
- Pre-trained model Roberta-base is released under the MIT License.
- Library sentence-transformers is licensed under the Apache License Version 2.0 and is available on GitHub at https://github.com/UKPLab/sentence-transformers.
- Pre-trained model all-MiniLM-L6-v2 is licensed under the Apache License Version 2.0.
- Pre-trained gpt2-large model is license under the MIT License.
14The finetuning script can be accessed at https://github.com/huggingface/transformers/blob/main/examples/pytorch/contrastive-image-text/run Clip.py.
- Instruct-Pix2Pix is licensed under the MIT License and is available on GitHub at https://github.com/timothybrooks/instruct-pix2pix.
- Instruct-Pix2Pix further employs stable-diffusion-v1-5 that is released under CreativeML-Open-RAIL-M License.
- For the MS-COCO dataset:
- The annotations in the dataset are released under the Creative Commons Attribution 4.0 License.
- The use of the images in the dataset must abide by the Flickr Terms of Use.
- Pre-trained model clip-vit-base-patch32 is licensed under the MIT License.
- Pre-trained model flava-full is licensed under the 3-Clause BSD License.
- Pre-trained model BridgeTower large-item-mlm-itc is released under the MIT License.
- Pre-trained vilt-b32-finetuned-coco model is license under the Apache License Version 2.0.
C Datasheet for Dataset
C.1 Motivation
For what purpose was this dataset created? This dataset was created for the purpose of exploring the relevancy of counterfactual examples for multimodal vision-language models. Specifically, our aim was to create a dataset which can serve both as a challenging evaluation dataset for existing models and as a resource for training data augmentation to improve multimodal models on downstream tasks. For additional discussion of our motivation and the intuition behind counterfactual examples, see Section 1.
Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? The dataset was created by the authors of this paper who are affiliated with Intel Labs, a research and development organization within Intel Corporation.
Who funded the creation of the dataset? The creation of this dataset was funded by Intel Corporation.
C.2 Composition
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? The instances represent synthetically-generated images and accompanying text captions. The images depict a variety of different everyday scenarios.
How many instances are there in total (of each type, if appropriate)? COCO-Counterfactuals contains a total of 34,820 image-caption pairs.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? Yes, it contains all possible instances per our filtering criteria.
What data does each instance consist of? Each instance consists of a synthetically-generated image and an accompanying text caption.
Is there a label or target associated with each instance? No
Is any information missing from individual instances? No
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? Yes, instances which correspond to a single counterfactual pair are annotated as such in our dataset. Otherwise, there are no other relationships between individual instances.
Are there recommended data splits (e.g., training, development/validation, testing)? No
Are there any errors, sources of noise, or redundancies in the dataset? The automated methodology used to generate COCO-Counterfactuals introduces the possibility of noise and errors in the dataset. See Section ?? for additional discussion.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? Yes
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)? No
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? Yes, the dataset may contain offensive material due to the manner in which it was automatically constructed. See Section ?? for additional discussion.
Does the dataset identify any subpopulations (e.g., by age, gender)? No
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? No
Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? No
C.3 Collection Process
How was the data associated with each instance acquired? The data associated with each instance was acquired via our data generation methodology (see Section 3 for a detailed description).
What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? Please see Section 3 for a complete description of our data generation methodology.