SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Viraj Prabhu Sriram Yenamandra Prithvijit Chattopadhyay Judy Hoffman
{virajp, sriramy, prithvijit3, judy}@gatech.edu
We propose an automated algorithm to stress-test a trained visual model by generating language-guided counterfactual test images (LANCE). Our method leverages recent progress in large language modeling and text-based image editing to augment an IID test set with a suite of diverse, realistic, and challenging test images without altering model weights. We benchmark the performance of a diverse set of pretrained models on our generated data and observe significant and consistent performance drops. We further analyze model sensitivity across different types of edits, and demonstrate its applicability at surfacing previously unknown class-level model biases in ImageNet. Code: https://github.com/virajprabhu/lance.
Figure 1: The predominant paradigm in computer vision is to benchmark trained models on IID test sets using aggregate metrics such as accuracy, which does not adequately vet models for deployment, e.g. for the test image above, while a trained ResNet-50 model [1] accurately predicts the ground truth label ("dog sled"), it is in truth highly sensitive to the dog breed. We propose LANCE, an automated method to surface model vulnerabilities across a diverse range of interventions by generating such counterfactual images using language guidance.
As deep visual models become ubiquitous in high-stakes applications, robust stress-testing assumes paramount importance. However, the traditional paradigm of evaluating performance on large-scale [2-4] IID test sets does not adequately vet models for deployment in the wild. First, such models are typically evaluated via aggregate metrics such as accuracy, IoU, or average precision [5], which treat all test samples equivalently and do not distinguish between error types. Further, such test sets do not adequately capture the "long-tail" [6] of the data distribution: rare concepts, unseen concepts, and the (combinatorial explosion) of their compositions [7].
To address this, a considerable number of recent efforts have sought to develop realistic benchmarks for out-of-distribution (OOD) evaluation [8-13]. However, while useful, such OOD benchmarks are typically static across models and time, rather than being curated to probe a specific instance of a trained model, which diminishes their utility.
In this work, we eschew the traditional paradigm of static test sets and instead generate dynamic test suites. Specifically, we propose a technique that automatically generates a test suite of challenging but realistic counterfactual [14] examples to stress-test a given visual model. Our main insight is that while the scope of possible visual variation is vast, language can serve as a concise intermediate scaffold that captures salient visual features while abstracting away irrelevant detail. Further, while it is extremely challenging to run counterfactual queries on images directly, intervening on a discrete text representation is considerably easier. We call counterfactuals generated by our method Language-guided Counterfactual Images (LANCE).
Our method leverages pretrained foundation models for text-to-image synthesis [15-17] and large language modeling (LLM) [18], in combination with recent progress in text-based image editing [19, 20], to generate realistic counterfactual examples. It operates as follows: Given a test image and its ground truth label, we generate a caption conditioned on its label using an image captioning model [21]. We then use an LLM that has been fine-tuned for text editing [22] to generate realistic perturbations to a single concept at a time - caption subject, object, background, adjectives, or image domain (while leaving words corresponding to the ground truth category unchanged). Finally, we use a guided-diffusion model [19] with null-text inversion [20] to generate an edited counterfactual image conditioned on the original image and the perturbed prompt. We obtain a prediction from the trained model on both the original image and the generated counterfactual and compute the average drop in accuracy as a measure of the model's sensitivity to the given attribute.
We benchmark the performance of diverse models pretrained on ImageNet on our generated images, and find that performance drops more consistently and significantly than baseline approaches. Next, we demonstrate applications of our approach: in comparing the relative sensitivity of models to perturbations of different types, and in deriving actionable insights by isolating consistent class-level interventions that change model predictions.
Diagnosing deep visual models. Several recent works have focused on the discovery of systematic failure modes in trained neural networks. Early works propose human-in-the-loop approaches based on annotating spurious features learned by sparse linear layers [23], adversarially robust models [24], and discovering a hyperplane corresponding to potentially biased attribute [25]. Some prior work also proposes fully automated techniques: by learning decision trees over misclassified instance features from an adversarially robust model [26] or targeted data collection [27]. Several recent approaches to this problem leverage multi-modal CLIP [28] embeddings, via error-aware mixture modeling [29], identifying failure directions in latent space [30] or learning the task directly on top of CLIP embeddings [31]. However, these works rationalize failures but don't close the loop by evaluating the predicted rationale. Further, shortcut learning [32] of spurious correlations can also lead to success but for the wrong reasons, but prior work only studies failure cases. In contrast, we propose a method that generates visual counterfactuals that can surface a diverse range of model biases that bidirectionally influence performance.
Image Editing with generative models. Using generative models to edit images has seen considerable work. Early efforts focused on specialized editing such as style transfer [33] or translating from one domain to another [34]. More recent work has performed editing in latent space of a model like StyleGAN [35-39]. Recently, pretrained text-to-image diffusion models have become the tool of choice for image editing [19, 20, 17]. While superior to GAN's at image synthesis [40], targeted editing using such models that modifies only a specific visual attribute while keeping the rest unchanged is non-trivial. Recent work has presented techniques based on prompt-to-prompt tuning [19], by targeted insertion of attention maps from cross-attention layers during the diffusion process. Follow-up work generalizes this approach to real images, with additional null-text inversion [20] or delta denoising scoring functions [41], which enables instruction-based editing of images [42] and 3D scenes [43]. We leverage this technique of prompt-to-prompt tuning with null-text inversion but rather than generic image editing focus on generating challenging counterfactual examples.
Probing discriminative networks with generated counterfactuals. Several works have attempted to use generative models to obtain explanations from visual models. A popular line of work generates minimal image perturbations that flip a model's prediction [44-47] (e.g. changing lip curvature to alter the prediction of a "smiling" classifier). Our work shares a similar goal but with a crucial distinction:
Trained Model, Test Set
LANCE: Language-guided Counterfactual Image Generation
Figure 2: Overview. We propose a method to generate challenging counterfactual examples to stress-test a given visual model. Given a trained model and test set, LANCE generates a textual description (from a captioning model) and perturbed caption (using a large language model (LLM)), which is fed along with the original image to a text-to-image denoising diffusion probabilistic model (DDPM) to perform counterfactual editing. The process is repeated for multiple perturbations to generate a challenging test set. Finally, we ascertain model sensitivity to different factors of variation by reporting the change in the model accuracy over the corresponding counterfactual test set.
we seek to discover minimal perturbations to parts of the image not directly relevant to the task at hand (e.g. does changing hair color or perceived gender alter the smiling classifier's prediction?).
Closest to our work is Luo et al. [48], which determines model sensitivity to a set of user-defined text attributes by tuning a weighted combination of edit vectors in StyleGAN [35] latent space so as to flip the model prediction while maintaining global structure and semantics via additional losses. However, this method requires the user to enumerate all possible attributes of interest, may not generalize to more complex datasets, and requires optimizing several losses in conjunction. Our work also shares similar motivations as Li et al. [49], which uses diffusion models to generate ImageNet-E(diting), a robustness benchmark with varying backgrounds, object sizes, position, and directions. However, their benchmark is static across models, requires object masks per image to generate, and measures robustness to a constrained set of attribute changes. Our work is also related to Wiles et al. [50] which generates cluster-based error rationales, Vendrow et al. [51], which diagnoses failures by generating near-counterfactuals from human-generated text variations, and Dunlap et al. [52], which performs diffusion-based data augmentation. In contrast, we study bias discovery rather than mitigation, deriving causal insights into failures using a targeted editing algorithm, and support stress-testing across an unconstrained, automatically discovered set of attributes.
3 Language-guided Counterfactual Image Generation (LANCE)
Overview. We introduce LANCE, our algorithm for generating language-guided counterfactual images to stress-test a given visual model. Our key insight is to use language as a structured discrete scaffold to perform interventions on the model. LANCE does this by perturbing a text representation (caption) of an image to conditionally generate a counterfactual image (see Fig 2).
Perturbed Captions. We first use a pre-trained captioner (BLIP-2 [21]), to produce a text description for a given test image. We use beam search decoding with a minimum caption length of 20 words and a high repetition penalty to encourage descriptive captions with low redundancy (see Fig. 5 for examples). Our algorithm then edits the generated caption using a language model to generate variations that only change a single aspect at a time.
Edited Images. Next, we use a text-to-image latent diffusion model to generate a new image that reflects the text edit while remaining faithful to the original image in every other respect. We repeat this process for multiple perturbations to generate a challenging test set.
Sensitivity Analysis. Finally, we ascertain model sensitivity to different factors of variation by reporting the change in the model's accuracy over the corresponding counterfactual test set.
Next, we detail each step, beginning with describing the factors of visual variation that we stress-test against and the strategy we use to train a structured caption perturbed. Then, we describe our image
Table 1: Perturbation dataset statistics. We first programmatically collect a small dataset (0.6k samples) of 5 types of targeted caption perturbations for random MSCOCO [4] captions from GPT-3.5 turbo [22] (top). We then finetune LLAMA-7B [53] on this dataset to perform targeted perturbation, and use it to perturb image captions generated for ImageNet [2] (bottom).
Generator	Sample Input	Edit Type (Count)	Sample Output
GPT-3.5 turbo (0.6k samples)	A bicycle replica with a clock as the front wheel	Subject (0.1k)	bicycle→{scooter, unicycle, ..., shopping cart}
Object (0.1k)	clock→{basketball, pizza, ..., globe}
Adjective (0.1k)	bicycle→{vintage, steampunk, ..., rusty} bicycle
Domain (0.1k)	a {painting, sculpture, ..., sketch} of a ...
Background (0.2k)	... {on a stormy day, ..., parked outside a coffee shop}
Finetuned LLAMA-7B (35k samples)	a hockey puck sitting on top of a wooden table	Subject (6.2k)	hockey puck→{basketball, ..., lacrosse ball}
Object (4.8k)	wooden table→{marble table, ..., stone wall}
Adjective (7.1k)	hockey puck→wooden, large, rustic hockey puck
Domain (3.4k)	a {painting, ..., 3D model} of a hockey puck...
Background (14.1k)	... {in a busy hockey arena, ..., during a thunderstorm}
editing methodology and the various checks and balances we insert to ensure that our end-to-end approach yields challenging yet realistic test examples.
3.1 Training a Structured Caption Perturber
To perform meaningful interventions, we train a structured caption perturber that generates targeted caption edits. Motivated by prior work in finetuning instruction-following large language models [22](LLMs), we seek to train a structured perturber LLM that can generate diverse and realistic caption edits that capture the long-tail of visual variation while still representing realistic scenarios.
Selecting factors of variation. To train our perturber model, we select five factors of visual variation against which we will stress-test. While the space of potential factors of visual variation is vast, image captions can constrain this search space by capturing salient visual concepts while abstracting away irrelevant detail. Concretely, we measure resilience to the following factors of variation:
- Subject. Modifications to the subject of an image caption (e.g., a {man, dog, cat, ..., horse,}) can stress-test a model's ability to recognize the ground truth category when it co-occurs with other subjects, including subjects that co-occur rarely (or never) with the ground truth in the training data.
- Object. Similarly, modifications to the object of an image caption (e.g., a {table, chair, ..., bed}) can stress-test a model's resilience to novel or unseen co-occurring concepts.
- Background. Modifications to the context of an image caption (e.g., a {kitchen, bedroom, ..., living room}) can stress-test a model's ability to generalize to different scenes, including diverse backgrounds and weather conditions.
- Adjective. Modifications to the adjective of an image caption (e.g., a {red, blue, ..., green}) can stress-test a model's ability to generalize to visual variations captured by object attributes.
- Data domain. Modifications to the data domain of an image caption (e.g., a {painting, sketch, ..., sculpture}) can stress-test a model's ability to generalize to different data distributions.
While by no means exhaustive, these perturbation types capture a large and representative set of visual variations. Further, we note that our approach is agnostic to this choice and can easily be extended to accommodate additional factors (e.g., camera angle, lighting, etc.). Having defined our perturbation set, we proceed to collect a dataset to train our structured perturbation model.
Dataset collection. We use GPT-3.5 turbo [18] to programmatically collect a small dataset of 0.6k caption perturbations spanning the aforementioned types. For instance, to edit an "adjective", we prompt the model with: Generate all possible variations of the provided sentence by only adding or altering a single adjective or attribute. (all prompts in appendix). We find that even with zero-shot prompting, we are able to acquire caption perturbations that modify only the desired factor of variation. We follow this procedure to generate perturbations for randomly selected captions from the MSCOCO dataset [4]. Table. 1 (top) highlights example inputs and perturbations. As seen, the dataset contains both simple edits (e.g., a bicycle  $\rightarrow$  {red, blue, ..., green} bicycle) as well as more complex ones (e.g., a bicycle  $\rightarrow$  bicycle parked outside a coffee shop), without requiring any human supervision whatsoever.
Model training. Next, we finetune an LLM on the dataset described above to perform targeted caption editing: specifically, we use a LLAMA-7B [53] LLM and perform LoRA (Low-Rank Adaptation) finetuning [54]: the original LLAMA-7B model is kept frozen, while the change in weight matrices  $W \in \mathbb{R}^{d \times k}$  of the self-attention modules post-adaptation is restricted to have a low-rank decomposition  $W_{ft} = W_{pt} + \Delta W = W_{pt} + AB$ , where  $A \in \mathbb{R}^{d \times r}$  and  $B \in \mathbb{R}^{r \times k}$ . Here the rank  $r$  is kept low, resulting in very few trainable parameters in  $A$  and  $B$ .
We use this finetuned model to perturb generated captions from ImageNet [2]. Table. 1 (top) shows examples of perturbations generated by this model. As seen, it is able to generate sensible and diverse edits corresponding to the edit type without changing the remainder of the caption.
Caption editing: Checks and balances. An undesirable type of edit would modify a word corresponding to the ground truth category label of the image. For instance, if the ground truth label of an image is 'dog', we do not want the model to generate a perturbation that changes the word 'dog' to 'cat'. To address this, we simply filter our caption edits wherein the edit is semantically similar (measured using a sentence BERT model [55]) to the ground truth category. Similarly, we also filter our captions where the edit is semantically similar to the original word or phrase being modified.
3.2 Counterfactual Image Generation
Having generated image captions and their perturbations, we proceed to generate counterfactual images conditioned on the original image and edited caption using a text-to-image latent diffusion model, specifically Stable Diffusion [16]. However, despite its remarkable ability at generating high-quality text-conditioned images, targeted editing using such models has historically been challenging, as changing even a single word in the text prompt could dramatically change the generated output image. Naturally, this is undesirable for our task, as we want to generate counterfactual images that are as similar as possible to the original image, while still reflecting the caption edit.
To address this, we leverage the recently proposed prompt-to-prompt [19]  ${}^{1}$  image editing technique, which performs targeted injection of cross-attention maps that correspond to the caption edit for a subset of the denoising diffusion process. However, prompt-to-prompt is designed for generated images, and applying it to real images requires accurate image inversion to latent space.
We employ the recent null-text inversion technique proposed by Mokady et al. [20]. Concretely, for input image  $\mathbf{x}$  with caption  $c$ , we perform DDIM inversion by setting the initial latent vector  $z_0$  to the encoding of the image to be inverted  $\mathbf{x}$ , and run the diffusion process in the reverse direction [40] for  $K$  timesteps  $z_0 \to z_K$ . Further, latent diffusion models employ classifier-free [59] guidance for text-conditioned generation, wherein the diffusion process is run twice, with text conditioning and unconditionally using a null-text token. To encourage accurate reconstruction of the original image, we follow Mokady et al. [20] to use the initial noisy diffusion trajectory (generated with a small guidance scale) as a pivot and update  $\emptyset_k$ , the null-text embedding at timestep  $k$ , to minimize a reconstruction mean square error between the predicted latent code  $\hat{z}_k$  and the pivot  $z_k$ , using the default large guidance scale recommended for classifier-free guidance. This helps bring the backward diffusion trajectory close to the original image encoding  $z_0$ , and thus achieve faithful reconstruction. Let  $S_{k-1}(\hat{z}_k, \emptyset_k, c)$  denote one step of deterministic DDIM sampling [60]. We optimize:
$$
\min  _ {\emptyset_ {k}} \| z _ {k - 1} - S _ {k - 1} \left(\hat {z} _ {k}, \emptyset_ {k}, c\right) \| _ {2} ^ {2} \tag {1}
$$
Image editing: Checks and balances. We find that image editing using prompt-to-prompt with null-text inversion is highly sensitive to a specific hyperparameter which controls the fraction of diffusion steps for which self-attention maps for the original image are injected. Let  $f$  denote this fraction. The optimal value of  $f$  varies according to the degree of change, with larger changes (say editing the background or weather) requiring a small value. In service of making our algorithm fully automated, we follow prior work [43] to automatically tune this hyperparameter: we sweep over a range of values and threshold based on a CLIP [28] directional similarity metric [61], which measures the consistency in the change across images and captions in embedding space. Let  $E_{I}$  and  $E_{T}$  denote the CLIP image and text encoders. The CLIP directional similarity criterion  $\phi(\cdot)$  is given by:
$$
\phi (\mathbf {x}, \mathbf {x} ^ {\prime}, c, c ^ {\prime}) = 1 - \frac {\left(E _ {I} (\mathbf {x}) - E _ {I} \left(\mathbf {x} ^ {\prime}\right)\right) \cdot \left(E _ {T} (c) - E _ {T} \left(c ^ {\prime}\right)\right)}{\left| E _ {I} (\mathbf {x}) - E _ {I} \left(\mathbf {x} ^ {\prime}\right) \right| \cdot \left| E _ {T} (c) - E _ {T} \left(c ^ {\prime}\right) \right|} \tag {2}
$$
<sup>1</sup>Several editing techniques have been proposed recently [56, 57, 41, 58], which may be equally suitable.
Algorithm 1 Generating Language-guided Counterfactual Images
1: Input: Test set  $\mathbf{T} = \{(\mathrm{image~x},\mathrm{label~y})\}_{1}^{M}$ , model  $\mathcal{M}$ , image captioner  $\mathcal{C}$ , perturber LLM  $\mathcal{P}$ , latent diffusion model  $\mathcal{L}$ , maximum perturbations per image  $N$ , edit similarity threshold  $\epsilon$ , image quality threshold  $\tau$   
2: Output: Counterfactual test set  $\mathbf{T}'$   
3:  $\mathbf{T}' \gets \emptyset$  ▷ Initialize counterfactual test set  
4:  $\mathcal{P} \gets \{\text{Subject, Object, Background, Domain, Adjective}\}$  ▷ Define perturbation types  
5: for  $(\mathbf{x},y) \in \mathbf{T}$  do  
6:  $\mathbf{x}^{-1} \gets \mathcal{L}(\mathbf{x},\mathbf{c})$  ▷ Invert image to latent space using diffusion model  
7:  $c \gets \mathcal{C}(\mathbf{x})$  ▷ Generate caption for input image  
8: for  $i \in 1,\dots,N$  do  
9:  $p \sim \mathcal{P}$  ▷ Sample perturbation type  
10:  $c' \gets \mathcal{P}(c|p)$  ▷ Perform text edit  
11: if  $\operatorname{sim}(c',y) < \epsilon$  then ▷ Ensure ground truth is unchanged  
12:  $\mathbf{x}' \gets \mathcal{L}(c'| \mathbf{x}^{-1})$  ▷ Generate counterfactual image, Sec. 3.2  
13: if  $\phi(\mathbf{x},\mathbf{x}',c,c') > \tau$  then ▷ Ensure image quality, Eq. 2  
14:  $\mathbf{T}' \gets \mathbf{T}' \cup (\mathbf{x}',y)$  ▷ Add to test set  
15: return  $\mathbf{T}'$
Finally, we ensure that the generated image is more similar to the edited rather than the original caption in CLIP's [28] embedding space. Algo. 1 details our full approach.
In Section 4.1, we overview our experimental setup, describing the data, metrics, baselines, and implementation details used. Next, we present our results (Section 4.2), comparing the performance of a diverse set of pretrained models on the subset of the ImageNet test set, and on our generated counterfactual test sets. Additionally, we analyze the sensitivity of models to different types of edits and demonstrate the applicability of our method in deriving class-level insights into model bias.