SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dataset. We evaluate LANCE on a subset of the ImageNet [2] validation set. Specifically, we study the 15 classes included in the Hard ImageNet benchmark [62]. Models trained on ImageNet have been shown to rely heavily on spurious features (e.g. context) to make predictions for these classes, which makes it an ideal testbed for our approach. We consider the original ImageNet validation sets for these 15 classes, with 50 images/class, as our base set. Further, testing our approach on ImageNet has a few other advantages: i) The dataset contains naturally occurring spurious correlations rather than manually generated ones (e.g. by introducing a dataset imbalance). ii) Testing our method on ImageNet images rather than constrained settings used in prior work e.g. celebrity faces in CelebA allows us to validate its effectiveness in more practical settings. iii) Finally, using ImageNet allows us to stress-test a wide range of pretrained models.
Metrics. We report the model's accuracy@k ( $k \in \{1, 5\}$ ) over the original test set  $\mathbf{T}$  and generated counterfactual test set  $\mathbf{T}'$ . For model  $\mathcal{M}$ , we are interested in understanding the drop in accuracy@k over the counterfactual test set, compared to the original test set. We define this metric as follows:
$$
\Delta \operatorname {a c c} @ k = \left[ \frac {1}{| \mathbf {T} ^ {\prime} |} \sum_ {\left(\mathbf {x} ^ {\prime}, y\right) \in \mathbf {T} ^ {\prime}} \operatorname {a c c} @ k (\mathcal {M} \left(\mathbf {x} ^ {\prime}\right), y) \right] - \left[ \frac {1}{| \mathbf {T} |} \sum_ {(\mathbf {x}, y) \in \mathbf {T}} \operatorname {a c c} @ k (\mathcal {M} (\mathbf {x}), y) \right] \tag {3}
$$
However, it is possible for an intervention to alter model confidence without leading to a change in its final prediction. As a more fine-grained measure, we also report the absolute difference in predicted model confidence for the ground truth class over the original and counterfactual images. For a single instance, we define this as:  $\Delta p(y_{GT}|\mathbf{x}) = |p(y_{GT}|\mathbf{x}) - p(y_{GT}|\mathbf{x}')|$
Finally, to evaluate realism we also report the FID [63] score of our generated image test sets and perplexity of the generated and perturbed captions under LLAMA-7B [53].
Table 2: Results. We evaluate diverse models pretrained on ImageNet-1K on four test sets: the original Hard ImageNet test set (reference), its reconstruction generated by our method (control), and the counterfactual test sets generated by a random perturbation baseline (Ours-Baseline), and our proposed structured perturbation strategy (Ours). Our methods generate progressively more challenging test sets as evidenced by the consistent performance drop compared to the original test set ( $\Delta$ acc@k, subscript).
Test Set	Type	acc@1 / acc@5 (↑)
ResNet-50 [1]	ViT-B [67]	ConvNext [68]	CLIP [28]
1	Original	Reference	79.86 / 95.83	85.42 / 97.74	85.42 / 98.26	56.25 / 86.98
2	Reconstructed	Control	79.86+0.0 / 96.18+0.4	85.42+0.0 / 97.57-0.2	85.24-0.2 / 98.09-0.2	55.90-0.4 / 87.67+0.7
3	LANCE-R	Ours - Baseline	78.81-1.0 / 92.25-3.6	81.91-3.5 / 95.61-2.1	85.79+0.4 / 94.83-3.4	50.13-6.1 / 83.46-3.5
4	LANCE	Ours	74.01-5.8 / 92.96-2.9	79.00-6.4 / 94.24-3.5	81.18-4.2 / 94.62-3.6	48.27-8.0 / 84.38-2.6
Figure 3: Sensitivity across perturbation types as measured by average  $\Delta p(y_{GT}|\mathbf{x})$ .
Figure 4: Evaluating quality of generated images (FID) and captions (perplexity).
Method	FID(↓)	Perplexity(↓)
Original	0	17.1
Reconstructed	13.6	17.5
LANCE-R	51.6	18.3
LANCE	55.4	18.6
Implementation Details. We use BLIP-2 (for image captioning), LLAMA-7B [53] (for structured caption perturbation) with LoRA finetuning [54], StableDiffusion [16] (for text-to-image generation), and pretrained models from TIMM [64]. We use PyTorch [65] for all experiments. For finetuning LLaMA-7B, we concatenate the query, key, and value matrices per layer and approximate the updates to the resulting weight matrix using a rank-8 decomposition. We finetune the model for 37.5k steps updating the weights every 32 steps. For more details see appendix.
Baselines. In the absence of prior empirical work for our experimental setting, we compare the performance of LANCE to the following:
i) Original (Reference): The original, unmodified ImageNet test set, as a point of reference.
i) Reconstructed (Control): The reconstruction of the original ImageNet test set, as a control set
iii) LANCE-R (Baseline, Ours): As a baseline, we design a simple random caption perturbation strategy that randomly masks out a word in the original caption and replaces it with a different word using a masked language model [66]. To ensure meaningful perturbations, we impose two additional constraints: the word being modified should not be a stop word (to minimize wasteful edits e.g. changing 'a' to 'the'), and that the CLIP [28] similarity between the generated image and the new word should exceed its similarity to the original word.
Across pretrained models, LANCE generates more challenging counterfactuals than baselines. Our goal is to discover a set of example images that are especially challenging for a given model. Therefore, Table 2 reports top-1 and top-5 performance of multiple fixed models across two baseline test sets and our own (LANCE). We find that every model studied has significantly lower performance on our proposed generated test set. We compare this both to the standard ImageNet test set as well as a baseline variant of our method that uses random caption perturbations (LANCE-R). Finally, we verify that performance loss is not due to the image generation process by demonstrating that performance on a reconstructed test set is nearly identical to the original test set.
We provide example LANCE perturbations and generated images in Figure 5. You can find an example image for each perturbation type. In some cases, such as the domain edit (changing from
Figure 5: Visualizing counterfactual images. We visualize the counterfactual images generated by LANCE for images from the HardImageNet dataset. Each row corresponds to a specific edit type. Above each image, we display its generated caption, highlighting the original and edited word(s). Below each image, we display the top-5 classes predicted by a ResNet-50 model and the associated model confidence, coloring the ground truth class in green.
a photo to a sculpture) significant predictive changes result – from correctly identifying a howler monkey to incorrectly predicting a chimpanzee. As another example, the subject edit (changing the dogs from husky to Schnauzer) prohibits the model from accurately predictingdogsled, conceivably because few Schnauzers appeared pulling a sled in the training data.
Perturbation sensitivity varies by perturbation type and model. Now that we have confirmed that our generated test set provides challenging examples to assess our models, we may further analyze the impact of different types of perturbations. Figure 3 reports the relative change in predictive performance in response to different perturbation types. We find that changes to the background have the highest impact, followed by domain. Further, some models are more sensitive to LANCE images, with ResNet-50 [1] having the most sensitivity.
LANCE can be used to derive class-level insights into model bias. To do so, we compute the L1 distance between CLIP features for the original and edited words, and run K-Means clustering. We then visualize the clusters with the highest  $\Delta p(y_{GT}|\mathbf{x})$ , corresponding to high sensitivity to a given change. Figure 6 illustrates examples. As seen, our method can also be used to surface class-level failure modes to inform downstream mitigation strategies. Importantly, we stress that while some failure modes are corroborated by other diagnostic datasets [62], (e.g. using color-based context-cues to predict "howler monkey"), others go beyond the capabilities of conventional IID test set-based diagnosis (e.g. models relying on dog breed to predict "dog sled"; see Figs. 1, 11).
Are generated images and perturbed captions realistic? As we are generating a new test set, it is important to verify that any performance changes are not due to an impact on the realism of our images. In Table 4 we report the FID scores of our generated images and the perplexity scores of our generated captions and observe both to be within a reasonable range. Further, we reiterate that using our pipeline to reconstruct the original image results in a test set where each model measures near identical performance (Table 2).
Figure 6: Deriving class-level insights. LANCE can be used to derive model bias at a per-class level by clustering text edits and visualizing clusters with the highest average  $\Delta p(y_{GT}|\mathbf{x})$ . Above we highlight insights for three classes with a ResNet-50 model: i) "howler monkey": high sensitivity to the background color ii) "sunglasses": high sensitivity to the data domain, and iii) seat belt: high sensitivity to the (barely perceptible) vehicle type.
Table 3: Evaluating caption perturber. LANCE performs successful edits without mutating labels.
Metric	Subject (%)	Object (%)	Adjective (%)	Domain (%)	Background (%)	Overall (%)
Edit Success (%)	92	84	82	98	92	89
Filter prec. (%) / recall (%)	99.0/97.0	100/96.0	92.9/98.9	100/100	94.1/100	97.2/98.4
LANCE's checks & balances are highly effective. Next, we evaluate the accuracy of the caption perturber at making the type of edit specified by the prompt (e.g.correctly changing only the domain for a domain edit). To do so, we manually validate a random subset of 500 captions and their perturbations (100 per type) generated by the LANCE. We observe an overall accuracy of  $89\%$ , with the object  $(84\%)$  and adjective  $(82\%)$  types achieving the lowest accuracies: we find that most failures for these types result from incorrect identification of the object or adjective in the sentence. We note that even in failure cases (e.gchanging an irrelevant word), the edits made are still reasonable.
Next, we evaluate the efficacy of our caption editing checks and balances. We first label the same subset of 500 examples by hand with the correct action (filter / no filter) and compare against LANCE. Across perturbation types, LANCE achieves both high precision and recall, with a low overall false positive  $(2.8\%)$  and false negative  $(1.6\%)$  rate. Of these, a majority of false negatives ( edits that alter the ground truth that we fail to catch) are for the subject and object types: we find these typically
Table 4: Human study. LANCE-generated images score high on realism, edit success, and fidelity.
Metric	Subject	Object	Adjective	Domain	Background	Overall
Image Realism (1-5)	4.2±0.5	4.4±0.5	4.1±0.4	3.9±0.8	4.2±0.6	4.2±0.5
Edit Success (1-5)	3.8±0.4	3.6±0.5	4.1±0.5	3.7±0.5	3.3±0.4	3.7±0.4
Image Fidelity (1-5)	4.7±0.4	4.5±0.5	4.6±0.5	4.7±0.2	4.4±0.5	4.5±0.4
Label Consistency (%)	97.0±4.0	94.0±6.0	99.0±0	93.0±9.0	93.0±9.0	95.0±4.0
Ethical issues (%)	3.0±7.0	1.0±3.0	5.0±8.0	0±0	0±0	2.0±2.6
change the ground truth inadvertently e.g. for a "keyboard space bar" image of a typewriter, the edit typewriter  $\rightarrow$  painting erroneously passes our filter (paintings do not usually have space bars).
Human evaluators validate the realism and efficacy of LANCE-generated edits. We conduct a human evaluation along 5 axes: i) Image Realism (1-5, 5=best): How easy is it to tell that the counterfactual image was generated by AI? ii) Edit success (1-5, 5=best): Does the generated image correctly incorporate the edit? iii) Image fidelity (1-5, 5=best): Are all the changes made relevant to accomplishing the edit? iv) Label consistency (Yes/No): Is the original image label still plausible for the generated image?, and v) Ethical issues (Text input): Is the generated image objectionable, or raise ethical concerns around consent, privacy, stereotypes, demographics, etc.?
We collect responses from 15 external respondents for a random subset of  $50 < \text{image}$ , ground truth, generated caption, perturbed caption, counterfactual image> tuples (10 per-perturbation type), and report mean and variance. We include a screenshot of our study interface in Fig. 13, and report results in Table 4. As seen, images generated by LANCE are rated to be high on realism (4.2/5 on average) and fidelity (4.5/5 on average), and slightly lower on edit quality (3.7/5 on average). Of these, we find that background edits score lowest on edit success, due to sometimes altering the wrong region of the image. Further, generated images have high label consistency (95%), verifying the efficacy of LANCE's checks and balances. Finally,  $\sim 2\%$  of images are found to raise ethical concerns, which we further analyze below.
Ethical Issues. On inspection of the images marked as objectionable in our human study, we find that most arise from bias encoded in the generative model: e.g. for the edit woman  $\rightarrow$  lifeguard, the generative model also alters the perceived gender and makes the individual muscular. Further, for people  $\rightarrow$  athletes, the model also alters the person's perceived race. The model also sometimes imbues stereotypical definitions of subjective characteristics such as "stylish" (e.g. by adding makeup). To address this, we manually inspect the entire dataset of 781 images for similar issues and exclude 8 additional images. Going forward, we hope to ameliorate this by excluding subjective text edits and using text-to-image models with improved fairness [69] and steerability [57].
Failure modes and limitations. In Fig 12, we visualize LANCE's 5 most frequent failure modes: a) Label-incompatible text edit: e.g.black howler monkey  $\rightarrow$ purple howler monkey (howler monkeys are almost always black or red). b) Poor quality image edit: Images with low realism, which are largely eliminated via our checks and balances. c) Image edit that inadvertently changes the ground truth: e.g.typewriter  $\rightarrow$ stapler for the class "space bar" (staplers do not contain space bars). d) Inherited bias: e.g.changing perceived gender based on stereotypes. e) Meaningless caption edit: e.g.boy in back-seat driver in the back seat. Such failures can confound an observed performance drop but this is largely mitigated by their low overall incidence.
LANCE also has a few intrinsic limitations. Firstly, it leverages several large-scale pretrained models, each of which may possess its own inherent biases and failure modes, which is highly challenging to control for. Secondly, intervening on language may exclude more abstract perturbations that are not easily expressible in words. However, we envision that LANCE will directly benefit from the rapid ongoing progress in the generative capabilities [70, 53] and steerability [57] of image and text foundation models, paving the road towards robust stress-testing in high-stakes AI applications.
Acknowledgements. This work was supported in part by funding from NSF #2144194, Cisco, Google, and DARPA LwLL. We thank Simar Kareer for help with figures, members of the Hoffman Lab for project feedback, and the volunteers who participated in our human study.
[63] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "Gans trained by a two time-scale update rule converge to a local nash equilibrium," Advances in neural information processing systems, vol. 30, 2017.
[64] R. Wightman, "Pytorch image models." https://github.com/rwrightman/pytorch-image-models, 2019.
[65] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., "Pytorch: An imperative style, high-performance deep learning library," Advances in neural information processing systems, vol. 32, 2019.
[66] V. Sanh, L. Debut, J. Chaumont, and T. Wolf, "Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter," ArXiv, vol. abs/1910.01108, 2019.
[67] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010.11929, 2020.
[68] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976–11986, 2022.
[69] C. Zhang, X. Chen, S. Chai, C. H. Wu, D. Lagun, T. Beeler, and F. De la Torre, "Iti-gen: Inclusive text-to-image generation," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3969-3980, 2023.
[70] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, "Sdxl: improving latent diffusion models for high-resolution image synthesis," arXiv preprint arXiv:2307.01952, 2023.
Assets and license. All source images belong to the ImageNet dataset [2], which is distributed under a BSD-3 license that permits research and commercial use.
Statistics. We now provide an overview of the counterfactual test set generated via our method. After adding the checks and balances described in the main paper to ensure caption and image quality after editing, we approximately double the size of the HardImagenet dataset (from 750 to 1531). We provide a per-class breakdown of the generated samples in Figure 7: we find that our method is able to generate a larger number of counterfactuals for certain categories (e.g. "swimming cap") than others (e.g. "hockey puck"), presumably because one of two possible reasons: i) the structured caption perturber is able to generate several more plausible variations for certain classes, or ii) the checks and balances in our image editing pipeline filters out many more examples of certain classes due to not being of sufficiently high quality. We note that while we create this dataset as a proof-of-concept for our method, scaling it up to include additional classes and concepts is very straightforward, and simply requires finetuning the perturber on (a small number of) additional caption editing examples, which are relatively easy to collect.
B Per-class Analysis
In the main paper, we quantified performance in aggregate and per perturbation-type. We now investigate per-class performance over generated counterfactuals in Figure 8. We find that the counterfactuals generated for certain categories (e.g. "ski" or "swimming cap") are significantly more challenging across models than others.
Do class-level insights generalize to the full dataset? Recall that in Figure 3 of the main paper we derived class-level insights into model bias, for example finding that the accuracy of ResNet-50 models at recognizing "sunglasses" drops significantly if the data domain is changed from "photo" to "painting". A natural question then is whether such insights generalize directly to other classes. In Figure 9, we apply this intervention to the entire HardImageNet dataset and report the per-class drop in top-1 accuracy when going from the original test set to the generated counterfactual test set. As seen, across all but one category ("baseball player"), performance drops, often significantly (e.g. "balance beam" and "hockey puck").
Figure 7: Label histogram of additional counterfactual images generated via LANCE.
Figure 8: Per-class top-1 accuracy of trained models on the counterfactual images generated by LANCE on the HardImageNet [62] dataset.
We provide additional qualitative examples generated via LANCE in Figure 10. We demonstrate 3 examples for each perturbation type: subject, object, background, domain, and adjective. Some of these examples are particularly interesting e.g. that the model's confidence of there being a "balance beam" present goes down significantly if a "coach" is pictured doing a handstand (top row, middle). Similarly, the probability of the image containing a "patio" under the model goes down considerably if benches are replaced by bicycles (row 2, left). Similarly, the model can no longer recognize a "dogsled" if the weather conditions become misty (row 3, left). Finally, the model is much less confident in recognizing a "seatbelt" if the color of its buckle changes from silver to gold (bottom row, middle). We note that these are only image-level insights and require further investigation across images of a given class to draw conclusions of systematic model bias.
D List of Prompts
In Table 5 we provide an exhaustive list of the prompts we use for both GPT-3.5 turbo [18] and for finetuning LLAMA [53]. As seen, despite their simplicity our prompts are able to elicit the desired behavior from both models. We find that including a couple of examples in the prompt greatly improves performance.
Figure 9: Per-class drop in top-1 accuracy of a trained ResNet-50 [1] model on the counterfactual images generated by LANCE for the photo  $\rightarrow$  painting intervention.
E Analyzing Failure Modes
Despite the considerable number of checks and balances that we insert into our pipeline, while running it end-to-end automatically and at scale, we find that a few poor counterfactuals still fall through the cracks. We provide qualitative examples of representative failures modes in Figure 12, that we group into five categories:
i) Inconsistent with GT. LANCE sometimes generates edits that, while being reasonable in isolation, generates an image that is inconsistent with the ground truth label. For example, "howler monkeys" are always black or blonde in color, and so a purple howler monkey is statistically unlikely.
ii) Meaningless caption edit. In some cases, the structured perturber makes an edit that is visually meaningless, e.g. changing photo to video, while our pipeline is constrained to image generation.
iii) Poor quality image edit. Though we find this to occur very rarely due to our image quality filters, a few poor quality image edits are still retained e.g. accidentally cropping the limb of a person.
iv) Changing GT. A few edits inadvertently change the ground truth. For example, for the category "space bar", changing a typewriter to a stapler also ends up removing the space bar.
v) Image generation bias. Finally, we find that the latent diffusion model's inherent bias also sometimes seeps through, e.g. when changing an image of a woman to a middle-aged woman (ground truth category: "miniskirt"), the diffusion model also subtly changes the woman's attire to presumably be closer to attires commonly worn by middle-aged women in its training data, rather than changing physical age markers alone: in this case, the image cannot strictly be considered a counterfactual as multiple attributes are being intervened on.
We run all experiments on a single NVIDIA A40 GPU. All the models that we evaluate are trained with supervised learning on ImageNet-1K [2]. We include additional implementation details for hyperparameters used by LANCE for caption and image editing in Table 6.
a photo of a young boy in a baseball uniform hat. 1000 field top of a baseball field
a photo of a young girl1 in a baseball game holding a bat on top of a baseball field
Predictions: Predictions: ballPlayer (75.83%) ballPlayer (72.80%) baseball (24.17%) baseball (27.18%) rifle (0.00%) rifle (0.00%) rocket (0.00%) tripod (0.00%) football helmet (0.00%) park bench (0.00%)
Edit subject
a photo of a photo ofwoman doing a a coach doing abalance hardworking in a gym with beam in a gym withsleepe watchingpeople watching
a photo of a coupleof women standingnext on the beach with snorkelson their heads
Predictions:  
balance beam (91.91%)  
parallel bars (4.42%)  
horizontal bar (3.36%)  
volleyball (3.63%)  
basketball (0.02%)  
balance beam (0.38%)  
basketball (0.26%)
Predictions:  
snorkel (99.95%)  
sea snake (0.02%)  
scuba diver (0.02%)  
stingray (0.00%)  
sea cucumber (0.00%)
a photo of a couple of men standing next to each other on a beach with small hands on their heads
Predictions: snorkel (99.55%) sea snake (0.19%) scuba diver (0.14%) sea cucumber (0.05%) stingray (0.02%)