LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images
Authors: Viraj Uday Prabhu
Sriram Yenamandra
Prithvijit Chattopadhyay
Judy Hoffman
Venue: NeurIPS 2023 poster (2023)
Keywords: image classification
robustness
guided diffusion models
counterfactuals
TL;DR: Developing challenging test-suites to stress-test object recognition models at scale by generating language-guided counterfactual images

Abstract: We propose an automated algorithm to stress-test a trained visual model by generating language-guided counterfactual test images (LANCE). Our method leverages recent progress in large language modeling and text-based image editing to augment an IID test set with a suite of diverse, realistic, and challenging test images without altering model weights. We benchmark the performance of a diverse set of pre-trained models on our generated data and observe significant and consistent performance drops. We further analyze model sensitivity across different types of edits, and demonstrate its applicability at surfacing previously unknown class-level model biases in ImageNet. Code is available at https://github.com/virajprabhu/lance.

LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images
Viraj Prabhu Sriram Yenamandra
Prithvijit Chattopadhyay Judy Hoffman
Georgia Institute of Technology
jgdteeg
Abstract
We propose automated algorithm stress-test trained visual model by generat- ing uaguugge-guide counterfactual test images (LANCE). Our method leverages recent progress in large language modeling and text-based image editing aug- ment an IID test set with a suite of diverse, realistic, and challenging test images without altering model weights. We benchmark the performance of diverse set of pretrained models on our generated data and observe significant and consistent performance drops. We further analyze model sensitivity across different types of edits, and demonstrate its applicability at surfacing previously unknown class-level model biases in ImageNet. Code: https://github.com com/ http:///ihubb.coom//vaaapaaaaaae viraippaabhu/llaccc
Introduction
Test image (Label: Dog Sled)
Generating Language-guided Counterfactuals (LANCE)
"Huskies pulling dog sled"
Husky-Golden
Husky Husky-Pameranian
Res Rest NettNeet..
82%
99.7%
53%
27%
10%
10%
20%
Figure 1: The predominant paradigm in computer vision is to benchmark trained models IID test sets using aggregate metrics such as accuracy, which does not adequately vet models for deployment, fog. the test image above, while trained ResNet-50 model []] accurately predicts the ground truth label ("dog sled"), it is in truth highly sensitive to the dog breed. We propose LANCE, an automated method surface model vulnerabilities across diverse range of interventions by generating such counterfactual images using language guidance.
As deep visual models become ubiquitous high-stakes applications, robust stress-testing assumes paramount importance. However, the traditional paradigm of evaluating performance on large- scale (2H4] IID test sets does not adequately vet models for deployment in the wild. First, such models typically evaluated via aggregate metries such as accuracy, IoU, or average precision (5), which treat all test samples equivalently and do not distinguish between error types. Further, such test sets do adequately capture the "long-tail" 6 of the data distribution: rare concepts, unseen concepts, and the (combinatorial explosion) of their compositions [
To address this, considerable number of recent efforts have sought develop realistic benchmarks for diitiiiittttiittionnn (OOD) evaluation [H However, while useful, such OOD benchmarks are typically static across models and time, rather than being curated to probe specific instance of trained model, which diminishes their utility.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this work, we eschew the traditional paradigm of static test sets and instead generate dynamic test suites. Specifically, we propose a technique that automatically generates a test suite of challenging but realistic counterfactual [14] examples to stress-test given visual model. Our main insight is that while the scope of possible visual variation is vast, language can serve as a concise intermediate scaffold that captures salient visual features while abstracting away irrelevant detail. Further, while it is extremely challenging to run counterfactual queries on images directly, intervening on a discrete text representation is considerably easier. We call counterfactuals generated by our method Language- guided Counterfactual Images (LANCE).
Our method leverages pretrained foundation models for rext-to-image synthesis [15 17 and large language modeling (LLM) [8] in combination with recent progress in text-based image editing [19 201. to generate realistic counterfactual examples. It operates as follows: Given test image and its ground truth label, we generate a caption conditioned on its label using an image captioning model [21]. We then use an LLM that has been fine-tuned for text editing [22] to generate realistic perturbations to a single concept at a time - caption subject, object, background, adjectives, or image domain (while leaving words corresponding
the ground truth category unchanged). Finally, we use a guided-diffusion model [9] with null-text inversion [20] to generate an edited counterfactual image conditioned on the original image and the perturbed prompt. We obtain a prediction from the trained model on both the original image and the generated counterfactual and compute the average drop in accuracy as measure of the model's s sensitivity to the given attribute.
We benchmark the performance of diverse models pretrained on ImageNet on our generated images. and find that performance drops more consistently and significantly than baseline approaches. Next, we demonstrate applications of our approach: in comparing the relative sensitivity of models to perturbations of different types, and in deriving actionable insights by isolating consistent class-level interventions that change model predictions.
2
Related Work
Diagnosing deep visual models. Several recent works have focused on the discovery of systematic failure modes in trained neural networks. Early works propose human-in -n-ti-----tt-oopp approaches based on annotating spurious features learned by sparse linear layers (23), adversarially robust models (44,
and discovering a hyperplane corresponding to potentially biased attribute [25]. Some prior work also proposes fully automated techniques: by learning decision trees over misclassified instance features from an adversarially robust model (26] or targeted data collection [27]. Several recent
approaches to this problem leverage multi- modal CLIP (28) embeddings. via error-aware mixture modeling
229] identifying failure directions latent space (30) or learning the task directly on top of CLIP embeddings [31]. However, these works rationalize failures but don't 't close the loop by evaluating the predicted rationale. Further, shortcut learning 32) of spurious correlations can also lead to success but for the wrong reasons, but prior work only studies failure cases. In contrast, propose a method that generates visual counterfactuals that can surface a diverse range of model biases that bidirectionally influence performance.
Image Editing with generative models. Using generative models to edit images has seen consid- erable work. Early efforts focused on specialized editing such as style transfer (33] or translating from one domain another [34. More recent work has performed editing in latent space of a model
like StyleGAN 35) Recently, pretrained text-to-image diffusion models have become the tool of choice for image editing [19
17]. While superior to GAN's at image synthesis [40), targeted editing using such models that modifies only specific visual attribute while keeping the rest un- changed is non-trivial Recent work has presented techniques based on coopp-to--rompp tuning [19]. by targeted insertion of attention maps from Follow-up work generalizes this approach to real images, with additional null-text inversion 20) or delta denoising scoring functions [II]. which enables instruction- saasdd editing of images 42 and
layers during the diffusion process.
3D scenes 43 We leverage this technique of promp-to-promp tuning with null-text inversion but rather than generic image editing focus on generating challenging counterfactual examples.
Probing diseriminative networks with generated counterfactuals, Several works have attempted to use generative models to obtain explanations from visual models. A popular line of work generates minimal image perturbations that flip a model's ''ss prediction [ changing lip curvature to alter the prediction of a "smiling" classifier). Our work shares similar goal but with crucial distinction:

---

Trained Model, Test Set
LANCE: Language-guided Counterfactual Image Generation
invert
Captioner
a black monkey sitting on top Lire with green leaves
Perturber (LLM)
a black monkey sitting on top tree with orange leaves
Editor (DDPM)
"howler
Model
monkey"
ApCya x = 1p - POerlk))
"gorilla" " (756 conficence)
Model
Frozen
99% confidence)
Learned
Figure 2: Overview. We propose a method to generate challenging counterfactual examples to stress-test given visual model. Given trained model and test set, LANCE generates textual description (from a captioning model) and perturbed caption (using a large language model (LLM)), which is fed along with the original image text-to-image denoising diffusion probabilistic model (DDPM) to perform counterfactual editing. The process is repeated for multiple perturbations to generate challenging test set. Finally, ascertain model sensitivity to different factors of variation by reporting the change in the model accuracy over the corresponding counterfactual test set.
seek to discover minimal perturbations to parts of the image not directly relevant to the task hand (e.g.does changing hair color or perceived gender alter the smiling classifier's prediction?).
Closest to our work is Luo el al. 48 which determines model sensitivity to set of user-defined text attributes by tuning weighted combination of edit vectors StyleGAN 35] latent space so as to flip the model prediction while maintaining global structure and semantics via additional losses. However, this method requires the user to enumerate all possible attributes of interest, may not generalize to more complex datasets, and requires optimizing several losses in conjunction. Our work also shares similar motivations as Li et al. [49]. which uses diffusion models to generate ImageNet-E(diting)) robustness benchmark with varying backgrounds, object sizes, position, and directions. However, their benchmark is static across models, requires object masks per image to generate, and measures robustness to constrained set of attribute changes. Our work is also related to Wiles et al. [50] which generates cluster-based error rationales, Vendrow el al. [51]. which diagnoses failures by generating near-counnerffcttuals from human-generated text variations, and Dunlap et al. [52], which performs diffusion-based data augmentation. In contrast, we study bias discovery rather than mitigation, deriving causal insights into fallures using targeted editing algorithm, and support stress-testing across an unconstrained, automatically discovered set of attributes.
3 Language-guided Counterfactual Image Generation (LANCE)
Overview. We introduce LANCE, algorithm for generating language-guides counterfactual images to stress-test a given visual model. Our key insight is to use language as structured discrete scaffold perform interventions the model. LANCE does this by perturbing text representation
(caption) of an image to conditionally generate a counterfactual image (see Fig (}.
Perturbed Captions. We first use a pre-trained captioner (BLIP-2 (21)), to produce text description for a given test image. We use beam search decoding with a minimum caption length of 20 words and a high repetition penalty to encourage descriptive captions with low redundancy (see Fig. 5 for examples). Our algorithm then edits the generated caption using a language model to generate variations that only change single aspect time.
Edited Images. Next, use a text-to-image latent diffusion model to generate a new image that reflects the text edit while remaining faithful to the original image in every other respect. We repeat this process for multiple perturbations generate a challenging test set.
Sensitivity Analysis. Finally, we ascertain model sensitivity to different factors of variation by reporting the change in the model's accuracy over the corresponding counterfactual test set.
Next, we detail each step, beginning with describing the factors visual variation that we stress-test against and the strategy use to train structured caption perturbed. Then, describe image
3

---

Table 1: Perturbation dataset statistics. We first programmatically collect small dataset (0.6k samples) of 5 types of targeted caption perturbations for random MSCÓCO [4] captions from GPT-3.5 turbo [2] (top). We then finetune LLAMA-7B [53] on this dataset to perform targeted perturbation, and use it to perturb image captions generated for ImageNet [2] (boltom).
Generator
Sample Input
Edit Tyye (Count)
Sample Output
Subject (0.13)
bicycle (scattee, unicycle.
shogping cart)
GPT-355 turbo (0.5k camples)
bicycle replica with a clock as the front wheel
Object (o,ect(o210) 1x)
Rijective (0.1Kk
bicyale Kvintage, Stearpunk, .... ruaty, Bleycla a (painciag, aculpture, .... akotak) of a fon stormy day, parzed outside coffee shop)
clock clock-<btskeeballl pizza, globe)
Danain 1k.
Background (0.
Subjact (6.24)
hockey pack <[bakethhlll,
lacronse ball) vooden sable-ilmrble table, srs vall] hockay pick->
Finetuned LLAMA-7B (35k saxples)
hockey pack aitting top
Object (4.8K) 8R)
Adjent (7.1k)
Tarpe, hackay puck, (paleting, .... 30 modaly
pick
wooden table
DonaTn (3.4K)
Background (14,18)
[in a busy hockey arena, during thunderstorn)
editing methodology and the various checks and balances we insert to ensure that our end-to-end approach yields challenging yet realistic test examples.
3.1 Training a Structured Caption Perturber
To perform meaningful interventions, we train structured caption perturber that generates targeted caption edits. Motivated by prior work in finetuning instruction-follovin large language mod- els (LLMMs, we seek to train a structured perturber L.LM that can generate diverse and realistic caption edits that capture the long-tail of visual variation while still representing realistic scenarios.
Selecting factors of variation. To train our perturber model, we select five factors of visual variation against which we will stress-test. While the space of potential factors of visual variation is vast, image captions can constrain this search space by capturing salient visual concepts while abstracting away irrelevant detail. Concretely, we measure resilience to the following factors of variation:
Subject. Modifications to the subject of an image caption (e.g., a [man, dog, cat, ... horse, /) can stress-test a model's ability to recognize the ground truth category when it co-occurs with other subjects, including subjects that co-occur rarely (or never) with the ground truth in the training data.
Object. Similarly, modifications the object of image caption (e.g.. a (table, chair, .... bedl) can stress-test model's resilience to novel or unseen co-occurring concepts.
Background. Modifications to the context of an image caption (c.g.. a (kitchen, bedroom,
living room)) can stress-test model's ability to generalize different scenes, including diverse backgrounds and weather conditions.
Adjective. Modifications to the adjective of an image caption (e.g.. a (red, blue. .... green/) can stress-test model's ability to generalize to visual variations captured by object attributes.
Data domain. Modifications to the data domain of an image caption (e.g.. a (painting, sketch, sculpture) can stress-test a model's ability to generalize to different data distributions.
While by no means exhaustive, these perturbation types capture a large and representative set of visual variations. Further, we note that our approach is agnostic to this choice and can easily be extended to accommodate additional factors (e.g., camera angle, lighting, etc.). Having defined our perturbation set, we proceed to collect a dataset to train our structured perturbation model.
Dataset collection. We use GPT-3.5 turbo 18] to programmatically collect a small dataset of 0.6k caption perturbations spanning the aforementioned types. For instance, to edit an "adjective", we prompt the model with: Generate all possible variations of the provided sentence by only adding or altering a single adjective or attribute. (all prompts in appendix). We find that even with zero- shot prompting, we are able to acquire caption perturbations that modify only the desired factor of variation. We follow this procedure to generate perturbations for randomly selected captions from the MSCOCO dataset 4.. Table. D (top) highlights example inputs and perturbations. As seen, the dataset contains both simple edits (e.g., bicycle -fred, blue, ... green] bicycle) as well as more complex ones (e.g., a bicycle -bicycle parked outside a coffee shop), without requiring any human supervision whatsoever.

---

Model training, Next. we finetune an LLM on the dataset described above to perform targeted caption editing: specifically, we use LLAMA-77 [53] LLM and perform LoRA (Low-Rank Adaptation) finetuning [54] the original LLAMA-7B model is kept frozen, while the change in weight matrices W E Tidok of the self-attention modules post-adaptation is restricted have a low-rank decomposition Wrr ft Wpr pt + AW = Pp + AB, where A E Ræ and B E R Here the rank is kept low, resulting in very few trainable parameters in A and B.
We use this finetuned model to perturb generated captions from ImageNet (2]. Table. (top) shows examples of perturbations generated by this model. As seen, is able to generate sensible and diverse edits corresponding to the edit type without changing the remainder of the caption.
Caption editing: Checks and balances. An undesirable type of edit would modify a word corre- sponding to the ground truth category label of the image. For instance, if the ground truth label of an image is 'dog', we do not want the model to generate a perturbation that changes the word 'dog' to 'cat'. To address this, we simply filter our caption edits wherein the edit is semantically similar (measured using a sentence BERT model (55)) the ground truth category. Similarly, we also filter our captions where the edit is semantically similar to the original word or phrase being modified.
3.2 Counterfactual Image Generation
Having generated image captions and their perturbations, we proceed to generate counterfactual images conditioned on the original image and edited caption using a text-to-image latent diffusion model, specifically Stable Diffusson [16]. However, despite its remarkable ability at generating high- quality text-conditioned images, targeted editing using such models has historically been challenging, as changing even single word in the text prompt could dramatically change the generated output image. Naturally, this is undesirable for our task, as we want to generate counterfactual images that similar as possible the original image, while still reflecting the caption edit.
To address this, we leverage the recently proposed prompt-to- ter--tto-pmpppp [9 iimage editing technique, which performs targeted injection of cross-attention maps that correspond to the caption edit for subset of the denoising diffusion process. However, prompt-to-promp is designed for generated images, and applying it to real images requires accurate image inversion to latent space.
We employ the recent null-text inversion technique proposed by Mokady er al. [20]. Concretely, for input image x with caption c, we perform DDIM inversion by setting the initial latent vector zo to the encoding of the image to be inverted x, and run the diffusion process in the reverse direction [0 for K timesteps žy + 2k. Further, latent diffusion models employ classifier-free [59] guidance for rext-conditioned generation, wherein the diffusion process run twice, with text conditioning and unconditionally using null-text token. To encourage accurate reconstruction of the original
image, we follow Mokady el al. (20 to use the initial noisy diffusion trajectory (generated with a small guidance scale) as a pivot and update 0r., the null-text embedding at timestep k; to minimize a reconstruction mean square error between the predicted latent code & and the pivot 21, using the default large guidance scale recommended for classifier-free free guidance. This helps bring the backward diffusion trajectory close to the original image encoding 2g. and thus achieve faithful reconstruction. Let Sa-1 (āx, 0R, c) denote one step of deterministic DDIM sampling [60]. We optimize:
min | - Sk-1 (2x, Dc, c) 2
Image editing: Checks and balances. We find that image editing using prompt-to-prompt with null-text inversion is highly sensitive to a specific hyperparameter which controls the fraction of diffusion steps for which self-attention maps for the original image are injected. Let f denote this fraction. The optimal value of f varies according the degree of change, with larger changes (say editing the background or weather) requiring a small value. In service of making our algorithm fully automated, we follow prior work [43] to automatically tune this hyperparameter: we sweep over range of values and threshold based on a CLIP (28) directional similarity metric [61], which measures the consistency in the change across images and captions in embedding space. Let E5 and Ey denote the CLIP image and text encoders. The CLIP directional similarity criterion Φ(.) is given by:
o(x,x' c, c') = 1 (E, (x))) ) (c) (r))))
(2)
Ex(x) [x) (e) E (c))
Several editing techniques have been proposed recently
[6 57 41 58 which may be equally suitable.
5

---

Algorithm Generating Language-guided Counterfactual Images
1: Input: Test T=((image x, label y 1 model M, image captioner C, perturber L.LM P.
latent diffusion model L. maximum perturbations per image N, edit similarity threshold 6, image
quality threshold
2: Output: Counterfactual test set T
3: T 9
Initialize counterfactual test set D Define perturbation types
4: P < {Subject, Object, Background, Domain, Adjective]
5: for (x. y) ET do
6:
x + f(x. c)
> Invert image to latent space using diffusion model > Generate caption for input image
7:
ct C(x)
8:
for i TEE,.... N do
9:
p P
D Sample perturbation type D Perform text edit
10:
d' - P(p(pp
11:
if ifsim'c' y)) < then
B Ensure ground truth unchanged Generate counterfactual image, Sec. 3.2 > Ensure image quality, Eq.
12:
x'+ (x, x', C, (') then T U
13:
14:
Add to test set
return T
Finally, we ensure that the generated image more similar to the edited rather than the original caption in CLIP's 28 embedding space. Algo. details our full approach.
4 Experiments
In Section we overview our experimental setup, describing the data, metrics, baselines, and implementation details used. Next, we present our results (Section 4.2p, comparing the performance of a diverse set of pretrained models on the subset of the ImageNet test set, and on our generated counterfactual test sets. Additionally, we analyze the sensitivity of models different types of edits and demonstrate the applicability of our method in deriving class-level insights into model bias.
4.1 Experimental Setup
Dataset. We evaluate LANCE subset of the ImageNet 2 validation Specifically, we study the 15 classes included in the Hard ImageNet benchmark (62]. Models trained on ImageNet have been shown to rely heavily on spurious features (e.g. context) to make predictions for these classes, which makes it an ideal testbed for our approach. We consider the original ImageNet validation sets for these 15 classes, with 50 images/class, as our base set. Further, testing our approach on ImageNet has few other advantages: i) The dataset contains naturally occurring spurious correlations rather than manually generated ones (e.g. by introducing dataset imbalance). ii) Testing our method on ImageNet images rather than constrained settings used in prior work e.g. celebrity faces in CelebA allows us validate its effectiveness in more practical settings. iii) Finally, using ImageNet allows us to stress-test wide range of pretrained models.
Metrics. We report the model's accuracy @k (A (1,55) over the original test T and generated counterfactual test set T', For model M, we are interested in understanding the drop in accuracy @k over the counterfactual test set, compared to the original test set. We define this metric follows:
Aace@k= =
Σ acc@k(M(x').)) ET
-
T
Σ (T.
aco@k(M(x).1)
(3)
However, it is possible for an intervention to alter model confidence without leading to a change in its final prediction. As more fine- ine-grained measure, also report the absolute difference in predicted model confidence for the ground truth class over the original and counterfactual images. For a single
instance, define this as: Ap(yoT] x) |p(μTT| ix)
p(yGT xx)]
Finally, to evaluate realism we also report the FID 66 score of our generated image test sets and
perplexity of the generated and perturbed captions under LLAMA-7B [53]

---

Table 2: Results. We evaluate diverse models pretrained ImageNet- IK four test sets: the original Hard ImageNet test set (reference), its reconstruction generated by our method (control), and the counterfactual test sets generated by a random perturbation baseline btus-Baselline, and our proposed structured perturbation strategy (Ours). Our methods generate progressively more challenging test sets evidenced by the consistent performance drop compared to the original test set (Aacc@k @kc k, subseript).
acc@1 / acc (1)
Test
Type
ResNet-50 [T
VIT-B
ConwNent 68
CLIP 28
1 Original
Reference
79.86 / 95.83
85.42 97.74
85.42/ 98.26
56.25 / 86.98
2 Reconstructed
Control
79.86-o.o / 16.18..o.a
85.42.0.o/
97.57
-.2
85.24-oa
98.09
-02
55.90 (87.67 40.7
LANCE-R
Ours Baseline
78.81 1.0 92.25
81.91
3.5 95.61
2.1 85.79
10.4 94.83
50.13
6.1 /83.46
4 LANCE
Ours
74.01
92.96
79.00
94.24
81.18
94.62
48.27
/84,38
40
Method
FID FID(L)
Perplexity
5 Ap(/ar()))
Original
17.1
Reconstructed
13.6
17.5
LANCE-R
51.6
18.3
LANCE
55.4
18.6
Background
Perturbation type
Figure 3: Sensitivity across perturbation types measured by average Ap(ycr(x).
Figure 4: Evaluating quality of generated images (FID) and captions (perplexity).
Implementation Details. We use BLIP-2 (for image captioning). LLAMA-7B 7B [53 (for structured caption perturation) with LoRA finetuning [54], StableDiffusion [16] (for text-to-image generation), and pretrained models from TIMM (64). We use PyTTrrch (65) for all experiments. For finetuning LLaMA-7B, we concatenate the query, key, and value matrices per layer and approximate the updates to the resulting weight matrix using a rank-8 decomposition. We finetune the model for 37.5k steps updating the weights every 32 steps. For more details see appendix.
Baselines. In the absence of prior empirical work for experimental setting, we compare the performance of LANCE to the following:
i) Original (Reference): The original, unmodified ImageNet test set, as point of reference.
i) Reconstructed (Control): The reconstruction of the original ImageNet test set, as control set
iii) LANCE-R (Baseline, Ours): a baseline, we design simple rundom caption perturbation strategy that randomly masks out word in the original caption and replaces it with different word using a masked language model [66]. To ensure meaningful perturbations, we impose two additional constraints: the word being modified should not be stop word (to minimize wasteful edits e.g. changing 'a' to 'the'), and that the CLIP (28) similarity between the generated image and the new word should exceed its similarity to the original word.
4.2 Results
Across pretrained models, LANCE generates more challenging counterfactuals than baselines. Our goal is to discover a set of example images that are especially challenging for given model. Therefore, Table reports top-1 and top-5 performance of multiple fixed models across two baseline test sets and our own (LANCE). We find that every model studied has significantly lower performance on our proposed generated test set. We compare this both to the standard ImageNet test set as well baseline variant of our method that uses random caption perturbations (LANCE-R) Finally, we verify that performance loss is not due to the image generation process by demonstrating that performance reconstructed test set is nearly identical the original test set.
provide example LANCE perturbations and generated images in Figure 5 You can find an example image for each perturbation type. In some cases, such as the domain edit (changing from
7

---

Edit subject
Edit object
Edit background
of husky dogs putlinng
pioco group dng sled across scinauzer iing covered with
photo couple of towler mankey siteing on nop
photo Couple of honler ronkey sicting tup other barbow forest
of boy hanging from metal bar on ansny! playground
boy hanging from metal bar on playoround
covered tield with people the background
in the background
other foreet
Eskieo (53.040
predictions1
1450 grant seinuuee 1480
scandard schnouzer (50. 3550
predi ctfons:
capuchín sp idder morkey (11,280)
howler noskey (53.6530
Predictiions
Predictions chtmpanzee (78. 675)
gorilla
horizontal bar (68. 1500
parallle] bars (31, 8400
Predictionss
polle (35, 2693
Prodictions:
2400
Stherian busky (1D, 1793
salarute (9.37))
Eng shoupdog (0.2
ribetan
schnauzer (35)
stasang (7. (7.920 (1.58)
howler monkey C6. 5880
stanarg (5.320
beaver (0.67K)
ballance (9.com)
swing 00. 01000
pole 00.000)
hort zonta] Bar 134.1380
parallel
shovel (9.99%)
Edit donain
Edit adjectíve
howler phone stering
sculpture of a
phoco of young
photo young
on top of cree trunk boxler ronkey sitting text to green forest next to green forest filled with trees
wninn and
in swferinn
a yellow seirennng cap
a. stlver setintnn can
Teans over the edge
teans over the enge of a
Prodictions:
Prodictions:
Predfict one
predictiins
baching (44 7250
bathing (36,
howler wonkey C55. E500
chi marnae (76 8310
ininiiigg trunks (2.020)
tub (18. BINO
tub (0. 8230
kiileer whole (7.238)
spider monkary (12. 2900
5800
stamang (2.115)
stamang (5.325)
Toot 2230
great white shark (5.475
ciet (3.945)
beaver (0.6/%)
Figure 5: Visualizing counterfactual images. We visualize the counterfactual images generated by LANCE for images from the HardImageNet dataset. Each row corresponds to specific edit type. Above each image, we display its generated caption, highlighting the original and edited word(s). Below each image, we display the top-5 classes predicted by a ResNet-50 model and the associated model confidence, coloring the ground truth class in green.
a photo to a sculpture) significant predictive changes result - from correctly identifying a howler monkey to incorrectly predicting chimpanzee. As another example, the subject edit (changing the dogs from husky to schnauzer) prohibits the model from accurately predicting dogsled, conceivably because few schnauzers appeared pulling a sled in the training data.
Perturbation sensitivity varies by perturbation type and model. Now that we have confirmed that our generated test set provides challenging examples to assess our models, we may further analyze the impact of different types of perturbations. Figure 3 reports the relative change in predictive performance in response different perturbation types. We find that changes to the background have the highest impact, followed by domain. Further, some models are more sensitive to LANCE images, with ResNet-50 [1] having the most sensitivity.
LANCE can be used derive class-level insights into model bias. To do so, we compute the L1 distance between CLIP features for the original and edited words, and run K-Means clustering. We then visualize the clusters with the highest Ap(ycT|x] x), corresponding to high sensitivity to a given change. Figure iillustrates examples. As seen, our method can also be used to surface class-level failure modes to inform downstream mitigation strategies. Importantly, we stress that while some failure modes are corroborated by other diagnostic datasets (62]. (e.g. g.... color-based context-cues to predict "howler monkey"), others go beyond the capabilities of conventional IID test set-based diagnosis (e.g. models relying dog breed to predict "dog sled"; see Figs.
Are generated images and perturbed captions realistic? As we are generating a new test set, is important to verify that any performance changes are not due to an impact on the realism of our images. Table 4 we report the FID scores of our generated images and the perplexity scores of our generated captions and observe both to be within reasonable range. Further, we reiterate that using our pipeline to reconstruct the original image results in a test set where each model measures near identical performance (Table 2)
8

---

class: howler monkey, Intervention: change color, Avg. Ap(sarl))
40%
seekey cireies on scekes stretes
photo of tewler photo honler
plotoo black
phate slack
eice leases and
buckgmasni
wi onarge lanees and
tackgrmund
1ush arean field
vies cress
a Tuth pick field with trees
of thee filled
next tree
green
Teaons PONT Tree
with
trae
hoiler modloy (00 250
muneyy
A.blac hear 300
herlen suckke 424, 2900
hewler rosker (3) 330
boster cantey (3), 1130
seiter roskee CD, A800
Tanger (L. 13.
stech bear C32 4no
ap apferr sontay (33 2010
gorilla 20 30.3
cortlla c3 4840
capuchin (0,
Sedrt (8. 2030
casachii 00.000
groenendzel (3.000
efet (5.428)
onargation (0 (0.170
trongaitnn CD (0N)
Class: sungl lasses, Intervention: photo painting, Avg. Ap 18)
18%
has with reffectter
martig supplasasa and
ause noman patesteg venan
cent
a alald saceet and
a stard Jacust and
glasses of hie face
glasess face
hinseiy in san esats with
her side
surplesses sides
Finger her
aigs her
lushh site with hen
Fredicsioss: cuecl (D-45g
cti owintooi (4T,
Predicttora: kannenica (64, siy
Fredictiions:
dki wit 270 18-129
Geelate (10. $750
homentra (66 Je0 seng
sungl
wikk E1 00,000
snorkel 27 1hg cang see (1.13))
swertshipe 5413
lass C3 kmO
sueglass 210 5130
commllace C1 2lgg
cecrbel G.780
crunch cost 2010
conte book (3. 6e10
bew sie 20, 10.0900
wark (0.733
class: seat belt, Intervention:
Cur bus, Avg. Ap(ycr 31%
photo of young sark cest bear strapaed bear strapeed belt
2 phore stang asst taddy
photo yeang bay sest of oar mas belt. end boodee
ereetig the back curolzenee, cast
rearing a seat
pact ceat Ear
veartng sear sele seat bas
belt
Predicti
Pradictions: sest (22 saray 720
prodictionn:
Predicttona:
Predicttona:
Jest wirrer 1E0
anst belt 077 2200
cast belt (32
forchirt (5,330)
seat sumglass (10, 2013
sunglassess 029 99.0
di21 teTepheee (2.224)
censoward excele cardigan LS.AR0)
comboy hus C0 2000
liding 5MO
sosee cleecting 1ns (2.300
sleecting bas (3.470
#ng (2.50N)
wineiaa 09 0110
chein (1.755)
Figure 6: Deriving class-level insights. LANCE can be used to derive model bias at a per-class level by clustering text edits and visualizing clusters with the highest average Ap(uGT) (x). Above highlight insights for three classes with a ResNer-50 model: i) "howler monkey": high sensitivity to the background color ii) "sunglasses": high sensitivity the data domain, and iii) seat belt: high sensitivity to the (barely perceptible) vehicle type.
Table Evaluating caption perturber. LANCE performs successful edits without mutating labels.
Metric
Subject (%)
Object (%) Adjective (%) Domain (%
Background (%) Overall ( (%)
Edit Success (%)
92
84
92
Filter prec. / (%)
99.0/97.0
100/96.0
92.9/98.9
100/100
9/110100
97.2/98.4
LANCE's checks & balances are highly effective. Next, we evaluate the accuracy of the caption perturber at making the type of edit specified by the prompt (e.g. correctly changing only the domain for a domain edit). To do so, we manually validate a random subset of 500 captions and their perturbations (100 per type) generated by the LANCE. We observe overall accuracy of 89%, with the object (84%) and adjective (82%) types achieving the lowest accuracies: we find that most failures for these types result from incorrect identification of the object or adjective in the sentence. We note that even in failure cases (e.g.changing an irrelevant word), the edits made are still reasonable.
Next, we evaluate the efficacy of our caption editing checks and balances. We first label the same subset 500 examples by hand with the correct action (filter no filter) and compare against LANCE: Across perturbation types, LANCE achieves both high precision and recall, with low overall false positive (2.8%) and false negative (1.6%) rate. Of these, majority of false negatives (edits that alter the ground truth that we fail to catch) are for the subject and object types: we find these typically

---

Table 4: Human study. LANCE- images score high realism, edit success, and fidelity.
Metric
Subject
Object
Adjective
Domain
Background
Overall
Image Realism (1-5)
4.2
0.4
4.1
3.9
0.2
0.5
Edit Success (1-5)
3.840.4
3.6=0.5
4.1
3.7+0.5
3.3+0.4
3.7
Image Fidelity (1-5)
4.7::0.4
4.5+0.5
4.6::0.5
4.7
4.4
4.5
Label Consistency (%)
97.0++4.0
94.0=6.0
99.0=0
93.0+9.0
93.0=9.0
95.0=4.0
Ethical issues (%)
3.0
1.0
5.0
0+:
0+:0
2.0+2.6
change the ground truth inadvertently for "keyboard space bar" image of typewriter, the edit typewriter-> painting erroneously passes our filter (paintings do not usually have space bars).
Human evaluators validate the realism and efficacy of LLANCE-generater edits. We conduct a human evaluation along 5 axes: i) Image Realism (1-5, 5=best): How casy is it to tell that the counterfactual image was generated by AI? ii) Edit success (1-5, 5=best): Does the generated image correctly incorporate the edit? iii) Image fidelity (1-5, 5=best): Are all the changes made relevant to accomplishing the edit? iv) Label consistency (Yes/No): Is the original image label still plausible for the generated image?, and v) Ethical issues (Text input): Is the generated image objectionable, or raise ethical concerns around consent, privacy, stereotypes, demographics, etc.?
We collect responses from 15 external respondents for a random subset of 50 <image, ground truth, generated caption, perturbed caption, counterfactual image> tuples (10 per-perturbation type), and report mean and variance. We include a screenshot of our study interface in Fig. 13 and report results in Table 4 As seen, images generated by LANCE are rated to be high realism (4.2/5 average) and fidelity (4.5/5 on average), and slightly lower on edit quality (3.7/5 on average). Of these, we find that background edits score lowest on edit success, due to sometimes altering the wrong region of the image. Further, generated images have high label consistency (95%), verifying the efficacy of LANCE's checks and balances. Finally, 22 of images are found to raise ethical concerns, which we further analyze below.
On inspection of the images marked as objectionable in our human study, we find that most arise from bias encoded in the generative model: e.g. for the edit woman -lifen-lifegaard, the generative model also alters the perceived gender and makes the individual muscular. Further, for people-rathletes, the model also alters the person's 's perceived race. The model also sometimes imbues stereotypical definitions of subjective characteristics such as "stylish" (e.g.by by adding makeup). To address this, we manually inspect the entire dataset of 781 images for similar issues and exclude additional images. Going forward, hope to ameliorate this by excluding subjective text edits and using text-to-image models with improved fairness [69] and steerability [57].
Ethical Issues.
we visualize LANCE'S 5 most frequent failure modes: a) Label-incompatible text edit: e.g. black howler monkey->purple howler monkey (howler monkeys are almost always black red). b) Poor quality image edit: Images with low realism, which are largely climinated via our checks and balances. <) Image edit that inadvertently changes the ground truth: for the class "space bar" (staplers do not contain space bars). d) Inherited bias: e.g.changing perceived gender based on stereotypes. e) Meaningless caption edit: e.g.boy in back-seat driver in the back seat. Such failures can confound observed performance drop but this is largely mitigated by their low overall incidence.
Failure modes and limitations. In Fig 12
g.typewriter->staee
LANCE also has few intrinsic limitations. Firstly, it leverages several large-scale pretrained models, each of which may possess its own inherent biases and failure modes, which is highly challenging to control for. Secondly, intervening on language may exclude more abstract perturbations that are not casily expressible in words. However, we envision that LANCE will directly benefit from the rapid ongoing progress in the generative capabilities 170] [53 and steerability [57] of image and text foundation models, paving the road towards robust stress-testing in high-stakes Al applications.
Acknowledgements. This work was supported in part by funding from NSF #2144194, Cisco, Google, and DARPA LwLL. We thank Simar Kareer for help with figures, members of the Hoffmann Lab for project feedback, and the volunteers who participated in our human study.
10

---

References
[1] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[2] O. Russakovsky, J. Deng. H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bemstein, et al. "Imagenet large scale visual recognition challenge." International journal of computer vision, vol. 115. pp. 211 211-252, 2015.
jourmaa
[3] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schicle, "The cityscapes dataset for semantic urhan scene understanding." in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213-3223, 2016.
[4] T.-Y. Lin, M. Maire, S. Belongie, Hays, Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft coco: Common objects context," Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part 13, pp. 440-755, Springer, 2014.
[5] D. Hoiem, S. K. Divvala, and J. H. Hays, "Pascal voc 2008 challenge," World Literature Today, vol. 24, 2009.
[6] Z. Liu, Z. Miao, X. Zhan, J. Wang. B. Gong, and S. X. Yu, "Large-scak long-taibe loigetailed recognition in an open world," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 2537-2546. 2019.
[7] I. Misra, A. Gupta, and M. Hebert, "From red wine red tomato: Composition with context," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1792-1801, 2017.
[8] C. Sakaridis, D. Dai, and L. Van Gool, "Semantic Togey scene understanding with synthetic data." Interna- rional Journal of Computer Vision, vol. 126, pp. 973 973-992, 2018
[9] X. Peng. Q Bai, X. Xia, Z. Huang, K. Saenko, and B. Wang. "Moment matching for multi-source domain adaptation." in Proceedings of the IEEF/CVF international conference on computer vision, pp. 1406-1415, 2019.
[10] D. Hendrycks and T. Dietterich, "Benchmarking neural network robustness to common corruptions and perturbations," arXiv preprint arXiv:1903. 12261, 2019.
[11] B. Recht, R. Roclofs, L. Schmidt, and V. Shankar, "Do imagenet classifiers generalize to imagenet?," in International conference machine learning, pp. 5389-5400, PMLR, 2019.
[12] P. Kattakinda, A. Levine, and S. Feizi, "Invariant learning via diffusion dreamed distribution shifts." arXiv preprint 2rXiv2221..03370 2022.
[13] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. L. Phillips, I. Gao, al., "Wilds: A benchmark of in-the-wild distribution shifts," in International Conference on Machine Learning. pp. 5637-5664, PMLR, 2021.
[14] J. Pearl, Causality. Cambridge university press, 2009,
[15] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, "Zero-shot text-to-image generation," in International Conference on Machine Learning, pp. 8821-8831, PMLR, 2021.
[16] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image synthesis with latent diffusion models." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10695, 2022.
[17] B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani, "Imagic: Text-based real image editing with diffusion models," in Proceedings of the IEEE conference computer vision and pattern recognition, 2023.
[18] T. Brown, B, Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neclakantan, P. Shyam, G. Sastry, A. Askell, et al., "Language models are few-shot learners"" Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[19] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, Prompttto-promp image editing with
attention control." in International Conference on Learning Representations, 2022.
[20] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Ore Or. "Null-text inversion for editing real images using guided diffusion models." arXiv preprint arX.9:2211.09794, 2022.
11

---

[21] J. Li. D. Li. S. Savarese, and S. Hoi, "Blip-2: Bootstrapping language- -mage pre-training with frozen image encoders and large language models," arXiv preprint arXiv:2301.1259) 12597, 2023.
[22] L. Ouyang. J. Wu, X. Jiang. D. Almeida, C. Wainwright, P. Mishkin, C. Zhang. S. Agarwal, K. Slama, A. Ray, et al., "Training language models follow instructions with human feedback," Advances in Neural Information Processing Systems, vol. 35, pp. 27730-2777 277444, 2022.
[23] E. Wong. S. Santurkar, and A. Madry, "Leveraging sparse lincar layers for debuggable deep networks," in Proceedings of the 38th International Conference on Machine Learning (M. Meila and T. Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research, pp. 11205 11216, PMLR, 18 24 Jul 2021.
[24] S. Singla and Feizi, "Salient imagenet: How discover spurious fcatures in deep learning?"" in International Conference on Learning Representations, 2022.
[25] Z. and Xu. "Discover the unknown biased attribute of an image classifier, in The IEEE International Conference on Computer Vision (ICCV), 2021.
[26] S. Singla, Nushi, Shah, Kamar, and Horvitz, "Understanding failures of deep networks via robust feature extraction," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12853-12862, 2021.
[27] S. Singla, A. M. Chegini, M. Moayeri, and Feiz, "Duta-centric debugging: mitigating model failures via targeted data collection," arXiv preprint S.ti::2211.09855: 2022.
[28] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., "Learning transferable visual models from natural language supervision," in International conference machine learning, pp. 8748-8763, PMLR. 2021.
[29] S. Eyuboglu, M. Varma, K. K. Saab, J.-B. Delbrouck, C. Lee-Messer, J. Dunmmon, J. Zou, and C. Re, "Domino: Discovering systematic errors with cross-modal embeddings." in International Conference on Learning Representations, 2022.
[30] S. Jain, H. Lawrence, A. Moitra, and A. Madry, "Distilling model failures as directions in latent space," in International Conference on Learning Representations, 2023.
[31] Y. Zhang, J. Z. HaoChen, S.-C. Huang, K.-C. Wang. J. Zou, and S. Yeung, "Diagnosing and rectifying vision models using language," in The Eleventh International Conference Learning Representations, 2023.
[32] R. Geirhos, J. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, "Shorteut learning in deep neural networks," Nature Machine Intelligence, vol. 2, no. 11. pp. 665 065.673,770 2020.
[33] L. A. Gatys, A. S. Ecker, and M. Bethge, "A neural algorithm of artistic style," arXiv preprint arXiv: 100.006666666 2015.
[34] L. A. Gatys, A. S. Ecker, and M. Bethge, "Image style transfer using convolutional neural networks." in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2414-2423, 2016.
[35] T. Karras, S. Laine, and T. Aila, "A style-based generator architecture for generative adversarial networks," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-440 -4110, 2019.
[36] R. Abdal, Y. Qin, and Wonka, "Image2stylegan: How embed images into the stylegan latent space?," in Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4432 4441, 2019.
[37] R. Ahdal, Y. Qin, and P. Wonka, "Immag2stylegan+++ How to edit the embedded images?." " in Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 8296-8305, 2020.
[38] K. Crowson, S. Biderman, D. Kornis. D. Stander, E. Hallahan, L. Castricato, and E. RafI, "Vggan-clip: Open domain image generation and editing with natural language guidance," in Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVII, pp. 88 105, Springer, 2022.
[39] A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Megrew, I. Sutskever, and M. Chen, "Glide: Towards photorealistic image generation and editing with text-guided diffiusion models," in International Conference on Machine Learning, pp. 16784-16804, PMLR, 2022.
[40] P. Dhariwal and A. Nichol, "Diffusion models beat gans on image synthesis." Advances in Neural Informa- tion Processing Systems, vol. 34, pp. 8780-8794 2021.
12

---

[41] A. Hertz, K. Aberman, and D. Cohen-Or, "Delta denoising score," arXiv preprint arXiv:2304. 2023.
[42] T. Brooks, A. Holynski, and A. A. Efros, "Instructpix2pix: Learning to follow image editing instructions," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2023.
[43] A. Haque, M. Tancik, A. A. Efros, A. Holynski, and A. Kanazawa, "Instruct-nerf2nert: Editing 3d scenes with instructions," arXiv preprint arXiv: 12789, 2023.
[44] A. Sauer and A. Geiger, "Counterfactual generative networks." in International Conference Learning Representations.
[45] G. Jeanneret, L. Simon, and F. Jurie, "Diffusion models for counterfactual explanations." in Proceedings of the Asian Conference on Computer Vision, pp. 858 876, 2022.
[46] G. Jeanneret, L., Simon, and F. Jurie, "Adversarial counterfactual visual explanations." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2023.
[47] M. Zemni, M. Chen, É. Zablocki, H. Ben- Youne,,,,,, P. Pérez, and M. Cord, "Octet: Object-aware counterfac- tual explanations." " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2023.
[48] Luo, Z. Wang, C. Wu, Huang, and F. De Torre, "Zero-shot model diagnosis," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2023.
[49] X. Li, Y. Chen, Zhu, S. Wang. R. Zhang, and H. Xue. "Imagenet-e: Benchmarking neural network robustness via attrihute editing," " Proceedings of the IEFE conference on computer vision and pattern recognition, 2023.
[50] O. Wiles, Albuquerque, and S. Gowal, "Discovering bugs in vision models using off-the off-the-shelf image generation and captioning," in NewrIPS ML Safery Workshop, 2022.
[51] J. Vendrow, S. Jain, L. Engstrom, and A. Madry, "Dataset interfaces: Diagnosing model failures using controllable counterfactual generation." " arXiv preprint arXiv: 07865, 2023.
[52] L. Dunlap, A. Umino, H. Zhang, J. Yang, J. E. Gonzalez, and T. Darrell, "Diversify your vision datasets with automatic cestusioo-bbsed augmentation," arXiv preprint arXiv:2305, 16289, 2023.
[53] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., "Llama: Open and efficient foundation language models," arXiv preprint 23X 2023.
[54] F. J. Hu, Y. Shen, P. Wallis, 7. Allen-Zhu, Y. Li, S. Wang, I. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," " arXiv preprint (.X.::2016009683. 2021.
[55] N. Reimers and Gurevych, "Sentence-berr Sentence embeddings using siamese bert-networks." arXiv preprint arXiv: 1908. 10084, 2019.
[56] B. Wallace, A. Gokul, S. Ermon, and N. Naik. "End-to-end diffusion latent optimization improves classifier guidance." in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7280-7290, October 2023.
[57] L. Zhang, Rao, and Agrawala, "Adding conditional control text-to-immn tomatt---meeaaee diffusion models," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023. 202..
[58] C. Wu and F. De la Torre, "A latent space of stochastic diffuuson models for zero-shot image editing and guidance," " in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7378-7387, 2023.
[59] J. Ho and T. Salimans, "Classifier--fre diffusion guidance," arXiv preprint 2rXiv:2207.125000 2022.
[60] J. Song, C. Meng, and S. Ermon, "Denoising diffusion implicit models." arXiv preprint 201.:2000. 2020.
[61] R. Gal, O. Patashnik, H. Maron, G. Chechik, and D. Cohen-Or, "Stylegan-nada: Clip-guided domain adaptation of image generators," arXiv preprint 100i 00946, 2021.
[62] M. Moayeri, S. Singla, and S. Feizi, "Hard imagenet: Segmentations for objects with strong spurious cues." in Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2022.
13

---

[63] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, "Gans trained by two time-scale update rule converge local nash equilibrium," Advances in neural information processing systems, vol. 30, 2017.
[64] R. Wightman, "Pytorch image models." httppe///ggthahh.ooo/////aaaaaaa 2019.
[65] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., "Pytorch: An imperative style, performmaee deep learning library." Advances in neural information processing systems, vol. 32, 2019.
[66] V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "Distilbert, distilled version of bert: smaller, faster, cheaper and lighter," ArXiv, vol. abs/1910.01 108, 2019.
[67] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn. X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. "An image is worth 16x16 words: Transformers for image recognition scale," " arXiv preprint arXiv:2010. 1929, 2020.
[68] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer. T. Darrell, and S. Xie, "A convnet for the 2020s." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976- 11886. 2022.
[69] C. Zhang, X. Chen, S. Chai, C. H. Wu, D. Lagun, T. Beeler, and F. De la Torre, "Iti-gen: Inclusive text-to-image generation." in Proceedings the IEEE/CVF International Conference Computer Vision, pp. 3969-3980, 2023.
[70] D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, "Sdxl: improving latent diffusion models for high-resolution image synthesis," arXiv preprint arXiv:2307.0195. 2023.
Appendix
A
Dataset Details
Assets and license. All source images belong the ImageNet dataset 2 which is distributed under a BSD-3 license that permits research and commercial use.
Statisties. We now provide an overview of the counterfactual test set generated via our method. After adding the checks and balances described in the main paper to ensure caption and image quality after editing, we approximately double the size of the HardImagenct dataset (from 750 1531). We provide per-class breakdown the generated samples Figure we find that our method able to generate larger number founterfactuals for certain categories (e.g. "swimming cap") than others (e.g. "hockey puck"), ) presumably because one of two possible reasons: i) the structured caption perturber is able to generate several more plausible variations for certain classes, or ii) the checks and balances in our image editing pipeline filters out many more examples of certain classes due not being of sufficiently high quality. We note that while create this dataset as proof-of oo-cccept for our method, scaling it up to include additional classes and concepts very straightforward, and simply requires finetuning the perturber on (a small number of) additional caption editing examples, which are relatively easy to collect.
B Per-class Analysis
In the main paper, we quantified performance in aggregate and per perturbation-tyl typee. We now investigate per-class performance over generated counterfactuals in Figure X We find that the counterfactuals generated for certain categories (e.g. "ski" or "swimming cap")) are significantly more challenging across models than others.
Do class-level insights generalize to the full dataset? Recall that in Figure 3 of the main paper we derived class-level insights into model bias, for example finding that the accuracy of ResNet-50 models at recognizing "sunglasses"" drops signficantly if the data domain is changed from "photo" to "painting". A natural question then is whether such insights generalize directly to other classes. Figure entire HardImageNet dataset and report the per-class drop in top-1 accuracy when going from the original test set to the generated counterfactual test set. As seen, across all but one category ("baseball player"), performance drops, often significantly (e.g. "balance beam" and "hockey puck").
9 we apply this intervention to the
14

---

125
120
100
92
Count
50
58
54
455
50
59
45
20
14
dogsled
seat belt
Ski
volleyball
hackey puck
keyboard spare bar
howler monkey
balance beam
patio
miniskirt
sunglasses
swimming cap snarkel
gymnastic horizontal bar baseball player
Figure 7: Label histogram of additional counterfactual images generated via LANCE.
100
LANCE top-1 acc.
ResNet-50
VIT-B/16
CLIP
ConvNext
miniskir
seat belt
degsled
hockey Ruck
volleyball
Layboard space bar
patio
howler monkey
Swimming cap
balance beam
baseball player
sunglasses
shorkel
gymnastic hortmmiall bar
Class
Figure 8: Per-class top-1 accuracy of trained models on the counterfactual images generated by
LANCE on the HardImageNet 662 dataset.
C Additional Qualitative Results
We provide additional qualitative examples generated LANCE Figure I0 demonstrate examples for each perturbation type: subject, object, background, domain, and adjective. Some of these examples are particularly interesting egg. that the model's 's confidence of there being a "balance heam" present goes down significantly if "coach" is pictured doing handstand (top middle). Similarly, the prohability of the image containing "patio" under the model goes down considerably if henches Similarly. the model can no longer recognize a "dogsled" the weather conditions become misty (row left). Finally, the model is much less confident recognizing a "seatbelt" if the color of its buckle changes from silver to gold (bottom row, middle). We note that these are only image-level insights and require further investigation across images of given class to draw conclusions of systematic model bias.
replaced hy bicycles (row left).
D List of Prompts
In Table 5 we provide an exhaustive list of the prompts we use for both GPT-3.5 turbo 18] and for finetuning LLAMA 133. As seen, despite their simplicity our prompts are able to elicit the desired behavior from both models. We find that including couple of examples in the prompt greatly improves performance.
15

---

Drop in top-1 accuracy (%6)
20,32
20
15
10
8.85
9.08
5.23
5.67
4.294.86
2.18
2.39
Sungisines
Dog Sked
Howler Monkey Seat Bett
Ski
Swinmring Capp
Balance Beam
Gymnastic Horsion ofl Hkr
Horkey Pack Menssaot
Keybbaard Spaie Balss
Basata Player
model on the counterfactual images generated by LANCE for the photo->painting intervention.
Figure 9: Per-class drop in top-1 accuracy of trained ResNet-50 [
E
Analyzing Failure Modes
Despite the considerable number of checks and balances that we insert into our pipeline, while running end-to-end automatically and at scale, we find that few poor counterfactuals still fall through the cracks. We provide qualitative examples of representative failures modes in Figure 12 that group into five categories:
i) Inconsistent with GT. LANCE sometimes generates edits that, while being reasonable in isolation, generates image that is inconsistent with the ground truth label. For example, "howler monkeys" always black purple howler monkey statistically unlikely.
blonde in color,
Meaningless caption edit. some cases, structured perturber makes edit that visually meaningless. e.g. changing photo to video, while our pipeline constrained to image generation.
iii) Poor quality image edit. Though we find this to occur very rarely due to our image quality filters, a few poor quality image edits are still retained e.g. accidentally cropping the limb of person.
iv) Changing GT. A few edits inadvertently change the ground truth. For example, for the category "space bar", changing typewriter stapler also ends up removing the space har.
Image generation bias. Finally, find that the latent diffusion model's inherent bias also sometimes seeps through, e.g. when changing an image of the diffusion model also subtly changes the woman's 's attire to presumably be closer to attires commonly worn by middle-aged women its training data, rather than changing physical age markers alone: in this case, the image cannot strictly be considered a counterfactual as multiple attributes are being intervened on.
middle-aged woman (ground truth category: "miniskirt"),
Implementation Details
We run all experiments on a single NVIDIA A40 GPU. All the models that we evaluate are trained with supervised learning ImageNet- KK @2 We include additional implementation details for hyperparameters used by LANCE for caption and image editing in Table
16

---

Edit subject
SIit of 2ntfor kollley E31
shuto of
SOTE doing vits
ptacs of couzle
chets with coter or teach
1oidieg a bat That
ous 2 woth
base is
bucch with snoreal
Proficstera bertzental har bet
Ailppiee (72 1I0
balance beus (81 5000
searial (90 dlver (tt))
PXEEE (0, 0301
crieed (0 2703
wilkyall 00220
520
astegne (9 P..00
Edit object
shoca Mieacle to bause cocrea
por the sonch in Front to
cheas monas
custre top conere foy
chalis tap marble Heer
parts 07 4000
mobl's hole (B
aradlcrrtae:
cotl C46 1810
#radictions:
die Pradections: cal GL
plosee fooc
piccee 220
hor buntes bur 185
seeiillll bur 4000
pacto resRUnnn (3 630
E27
mut
0080
betheua 120
Edit background
Hot Sucker 15 auiliing sonde
rat jecket pollitrg aiste sends
shore group
proce greas CATh toe
choto baldirg pooee but
holdingg bastell trg of link
preen Haeld
Pradictions:
Pradictions:
2010
Prodicttend
200
aello OK
Prediction::
carcee will (V
ralumm
bally (9-000)
sroen 00 10.0
scarebear GL400 aocll
feetie bret s70
plange (2.125
foosball galt
ailiceene 1813
digital dowat
crequnt call
rillie dhs Ord
ritlieg
staadirg sncx for: wod
poter staoling 1at teal
IN trep 1o riale
arth trea the
locces
Sumplsson
Prodectine:: CakT dog 10-2200
Pradicctioes all (1)-
pradicction: pals (0.880
Pradict sure (2-22)
Predlcttien:
Caklno 13.0
Light with
Cleter with C20)
horse care (0 00.0
Nore fiet bara 0000
ella 200
singlastet Co 05,000
mocen spoi CS S000
Edit adjective
CATTE meuetales www
suno lasss and
and soee
sunglasnss and
sing asees and
nourtalrt ti Jon
Predicttons: uueg congloss 900
Predictionc:
Pedicstons: Jeak boble 00.400 scabard calludar shloeees
(77.400
dieinal witc (00-A00
conglacom: 05.190 G2.20
(2.200 Hel (7.48)
baffaoles
busble
Figure 10: We visualize the counterfactual images generated by LANCE for images from the HardImageNet (62) dataset. Each row corresponds to specific edit type. Above each image, we display its generated caption, highlighting the original and edited word(s). Below each image, display the top-5 classes predicted by a ResNet-50 model and the associated model confidence. coloring the ground truth class in green.
17

---

Claxac Intervention: change dog bread lamute Samoyed
led-dog breada)
dan the bichhppe
de pitin
Pradiateen 20
BATINI O MN
00 100
Goliaa 01 200
Figure 11: Beyond conventional diagnosis. LANCE can uncover nuanced model bias, e.g. predictive confidence for "dog sled" drops significantly even if the dog breed is changed to other popular sled dog breeds.
Table 5: Prompts used for programmatically collecting perturbations of captions using GPT-3.5 turbo [18] We the prompts without examples instructions when instruction fine-tuning
LLAMA-7B 53 with LoRA 54
Perturbation type
Prompt
Edit subject
Generate all possible variations by changing only the subject of the provided sentence. For example: Change "A man walking a dog" "A woman walking dog". <caption>
Edit object
In English grammar, the subject is the person, thing, or idea that per- forms the action of the verb a sentence, while the object the person, thing, idea that receives the action of the verb. Generate all possible variations by changing only the object of the provided sentence. For example: Change "A man walking dog" "A man walking horse". <caption>
Edit background
Gennrratt all possible variations of the provided sentence by changing or adding background or location details without altering the foreground or its attributes. For example: Change "A man walking a dog" and "A man walking a dog with mountains in the background", on grass". Change "A man walking a dog on the road" to "A man walking dog with mountains by the beach". <caption> Generate all possible variations of the provided sentence by only changing the weather conditions, adding description of the weather if not already present. For example: change "A walking dog" "A walking dog in the rain", and change "A man walking a dog in the rain" to "A man walking a dog in the snow". <caption>
change "A man walking dog
Edit domain
Generate few variations by only changing the data domain of the provided sentence without changing the content. For example: Change "A photo of a man" to "A painting of man" or "A sketch of man". ". A photo of <caption>
Edit adjective
Generate all possible variations of the provided sentence by only adding
altering single adjective or altribute. For example: change "A man walking a brown dog" to "A tall man walking brown dog". "A man walking a black dog" "A man walking a cheerful brown dog" <caption>
18

---

Inconsistent wi th GT
Heaning less caption edit
poar quality
image editt photo of taenagen doinng
of black mcnkey
Ittelle two
photo of uuman deing trick hears blu
purple monkey
Hette From a setal bar in
stecting on top of
si stcring on too
fros netal bar
wren
playground area
a playgoound area
leaves tree
surfface with blue background
sur Faca with
tree
background
background
bluw background
Predictions:
Predicti ons:
Predictions ons:
Predictions:
predictions:
Predictionns oncectoon::
howler monkay (37 5080 howler monkey 5230
hor Leantal 9200
(56.2630
lalnnce besn
(22)848)
parailel hars (0.145)
horizontal 30%
gurilll (0.315)
langur (14.5300
(0. (0.01K)
sating (0.015)
paral Tel bars (0.2200
huri suntal ber (8.020)
ctei (0. (0.216)
copuchin (6,625)
polle (9.00N)
pol (0.00K)
watillz 00 00-000
parallel bars (1.425)
orangutan (0.1/K)
chinpanzes (6.345)
unce (2.00k)
00.030
bon (0.288))
changing GT
Image generation bias
photo top Stack books on top of books of
photo of
A photo of stean scanding front of her hips and purse in her hand
pioco of motee
a stapter strctig on
aged woman standing
her hands on her hips
wooden table
wouden table
and a purse in
typeariter Predictionn::
modes (18.508)
Predfcttins:
Predicttins:: wifiiskirt (39, 2210 comboy bosts (0. 00-3880 trench cosT 10, 1010
doomat (34, 7880
Predictions:
kiniskirss 8770
cosputer keybbaadd (11,
notebook (7. (7-928)
retabook (1 35%
prister (5. 99N)
Jean (S.745)
disc (b.128)
sandal (0. D400
suit (5.625)
Figure 12: We visualize the different modes failure of LANCE for genenerating counterfactuals on the HardImageNet dataset. Each row corresponds to specific edit type. Above each image, we display its generated caption, highlighting the original and edited word(s). Below each image, we display the top-5 classes predicted by ResNet-50 model and the associated model confidence, coloring the ground truth class in green.
pnoructioon one Plaase rest Trast) FANUS in srcower Falioaahil wiw io Ton ned magee aeaalle gesenaing dhe by CAPPR lezving
so by
sual espect given image idsac
unchanged
1) Devilime - liww wechiae the image pereeted by AN ANCS how mady in's that Imary was cressed by an morree")
auzocau
cdit reflected generated image" how
caption the genertted imagef)
stal conslowwayy le the only nol Image tebel aill plauchie for gennestec imaga? (or the groudd in tstel (Dvil will a wesccnble march in gessaned image?)
consideratioss gencrated image ectorecii
coocerrs around corpect, pliaay stereotycs denograstess way
Original
(ea)
Image Capticn
Ipen voad)
(peeredd
LANCS
qpen was Aciect
LAMKS Caption
Graand
Trush
Ficeiry
Consiitency
leans
(16 MSF regalie regaiveaeooiinn dtaoooos
Anting bock
foy riting the
Soul bult
seatiaal
piooo realle
a
back
Fer and
3861
tat
Figure 13: Human study, A screenshot of the interface we deploy to evaluate the quality of images generated by LANCE.
19

---

Table 6: Hyperparameter values used for caption (top leff), LLAMA finetuning (top right) and image editing (bottom).
config
value
config
value
learning rale
3e-4
cagtion length
effective
max. caption length
100
grad. accumulation steps
32
ropetition penalty
1.0
weight decay
0.5
sequence len
1024
decoding strategy
beam search
lorn rank (54
8
beam
54
clustering strategy
K- Means
lora dropout
54
0.05
number clusters
weights
qlv
(a) Caption Generation and Clustering
(b) LLAMA Finetuning
confi
value
Stable Diffusion
Li version
1.4
image resolution
512x512
LR scheduler
DDIM scheduler 60
beta_start 60
bata_end (6
0.012
beta_schedule
scaled linear
diffusion steps
19
sstentioo__repece_.ee
LM
czoes_replace__tegp
self self_replace_stepa
19
0..4, 0.7, 0.9)
guidance
42
0.2
img sim. threshold
sim. threshold
+
(c) Image Editing
20


Reviews:
[{"review_id": "36LrjHf1f4", "reviewer": "Reviewer_G3fV", "summary": "The paper proposes to use recent advances in text-to-image generation/editing to design evaluation by generating \"counterfactual\" data. The main idea of the method is to make semantically irrelevant (to the output class) edits to an image and observe its effects on the model predictions. A variety of models including CNNs, and ViT (trained classically as well as via language supervision) are tested, which show that such editing operation can indeed make a \"hard\" test set where most tested methods struggle to do well.", "questions": "Please see the weaknesses above", "limitations": "Please see the weaknesses above", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "[S1] Great Presentation Clarity: The paper is very well written and the details about what has gone into the final data-generation scheme are presented in sufficient detail. The experimental setup, the algorithm, and the implementation of the baseline models are clearly explained and should be easily reproducible independently.\n\n\n[S2] Good evaluation/results: The proposed model, based on the experiments presented, show a huge potential for using counterfactual generation of images to stress-test models. While the paper only makes a handful of edit types, there is a possibility to design hyper-specific probes (does change clothing color change models predictions about a person's profession?) that can be designed for specific applications.", "weaknesses": "[W1] Need more discussion of scope and their limitations: While the main idea is sound and the descriptions are clear, the evaluation/discussion leaves a lot to be desired. Most of the evaluation is done \"end-to-end\" which does not give much insight into the success of the various stages involved in the process. Some example concerns that I have:\n\n- The success of checks and balances (L155 and L185) is not verified/discussed. Do they always work? When do they fail? What are the effects of their failing?\n- What is the diagnostic prowess of this method? Besides showing that the edit-based perturbations does indeed make models flip their predictions (which has also been previously shown via adverserial noise injection), what insights can we draw from this type of stress testing? Does the results of this type of probe correlate with other tests (perhaps for bias/fairness and/or other diagnostic datasets? Do the edits represent (in the paper's view), the realm of possible naturally occurring variations in background/unrelated content in the picture? If not, can this still diagnose/predict the model's robustness in the real world? Or, is this just another form of discovering \"adversarial attack\" on the model predictions?\n\nThere is only a very short limitation section at the end of the conclusion section. I would have liked to see a more open discussion about exactly what (in view of the paper) can this method tell us about a model and what it cannot. \n\n\n[W2] No manual/external/independent validation: The quality of the edits, and the accuracy of the labels post-edit are not validated by any external/manual evaluation. We only see decreased overall accuracies for popular methods. While a simple baseline of random edit is tested, it is not clear whether a noise perturbation/other simpler perturbation techniques could also result in similar loss. Regardless of the loss in accuracy produced, it is also important to know whether the editing process leaves any unwanted artifacts. E.g., In the headline image, there is no actual sled visible, and the prediction of it likely comes from context. As such, it is unsurprising that changing the dog breed to pomeranian (an unlikely breed to be pulling sleds, also both \"husky\" and \"pomeranian\" are actual categories by themselves) would result in flipped predictions. To be clear, I am not claiming that this is \"OKAY\" behavior, just that perhaps the mechanism is \"label noise\" in Imagenet rather than \"context bias\" (Since there is nothing but the context that can help us make predictions). I am claiming, however, that this is a different category of error than other examples in the paper. Both are important, and the proposed method is likely to help uncover both, but at a lack of external validation about the accuracy of the labels before and after the edit operation. Some questions that I'd have liked validated externally:\n- Can the accuracy of the \"perturber\" be measured independently?\n- What is the accuracy of the ground truth labels post-edit? Are they truly unchanged? Are the methods presented in L185 and L155 (re. checks and balances) enough? Do they work? Have they avoided this issue successfully?\n- Since there is a pervasive error in Imagenet (https://arxiv.org/abs/2103.14749), does this method have any correlation with the noisy labels? In other words, is this intervention making \"easy to predict\" examples harder or primarily just the \"confusing\" ones?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review! In our general response, we evaluate each component of our method (including checks and balances), and include a human study to assess the overall realism, edit success, and fidelity of our generated dataset. \n\n> Can the accuracy of the ''perturber'' be measured independently?\n\nYes! We measure the accuracy of the perturber at following the provided instruction: specifically at making the type of edit specified while leaving the remainder of the caption unaltered. We observe an overall accuracy of 89% (see general response for details). \n\n> What is the accuracy of the ground truth labels post-edit?\n\nIn our human study, respondents confirm label validity for 96.8% of generated images, certifying that LANCE-generated images almost never mutate the ground truth label.\n\n> The success of checks and balances (L155 and L185) is not verified/discussed. Do they always work? When do they fail? \n\nImage editing: See previous response. \n\nCaption editing: We measure the robustness of our strategy to filter out caption edits that mutate the ground truth, and find that our method successfully filters out 98.4% of such edits (see general response for details).\n\n> I would have liked to see a more open discussion about exactly what can this method tell us about a model and what it cannot.\n\nSorry about that! We actually included a characterization of LANCE’s failure modes in supp. (Fig. 5), and in our general response above include an expanded discussion of the limitations of our method.\n\n>  What is the diagnostic prowess of this method?\n\n**Class-level insights**. In Section 4.2 (L261), we show how LANCE can provide class-level insights by clustering edits and visualizing clusters with high sensitivity ($\\Delta p(y_{GT}|\\mathbf{x})$, Eq. 4). Via this we discover, for instance, that for the ''sunglasses'' class, ResNet-50 models are highly sensitive to the image domain, and underperform on renditions of the same images as paintings. In Sec. 2 of supp. (L28), we extend this analysis to other classes and find that accuracy of a ResNet-50 model drops consistently for the photo$\\to$painting across classes.\n\n**Bias Diagnosis**. Some of LANCE’s discovered error modes are indeed corroborated by other diagnostic datasets, eg. models have been shown (by collecting segmentation masks) to rely heavily on class-specific spurious cues to make predictions on the 15 classes included in the HardImageNet benchmark. LANCE discovers several of these without additional labels: eg. discovering that for the ''howler monkey'', model predictions are sensitive to the color of the surrounding foliage (Figure 6, top row). \n\n**Beyond conventional diagnosis**. LANCE can surface nuanced error modes that conventional diagnostic datasets that are constrained to an IID test set cannot. Our teaser figure is a good example, wherein LANCE surfaces a model reliance on dog breeds to predict ''dog sled''. We agree with the reviewer that whether such reliance is acceptable may be breed-dependent. In the rebuttal PDF, we investigate if the model confidently predicts ''dog sled'' for two other popular sled-dog breeds (Malamutes and Samoyeds), and find that it frequently does not! This suggests that the model strongly associates the ''dog sled'' class with huskies.\n\n> While a simple baseline of random edit is tested, it is not clear whether a noise perturbation/other simpler perturbation techniques could also result in similar loss. Does the editing process leave any unwanted artifacts? \n\nTo clarify, our random edit baseline only picks the word to edit at random but uses a masked language model to make the actual edit. To ensure that the editing process does not leave unwanted artifacts, we: i) validate that accuracy over reconstructed images (with the original prompt) is preserved. ii) only retain edits with high CLIP directional similarity (Eq. 2). The high ratings for image realism obtained in our human study further attest to the generation quality.\n\n> Do the edits represent (in the paper's view), the realm of possible naturally occurring variations in background/unrelated content in the picture? If not, can this still diagnose/predict the model's robustness in the real world?\n\nOur dataset contains >160 unique weather patterns, spanning generic (eg. windy, snowy, rainy, sunny, misty, hazy, blizzard, thunderstorm, hailstorm, humid) and specific (eg. bitterly cold wind, sunny day in the springtime, cold winter morning, blustery winter solstice) types. Similarly, it contains >190 unique backgrounds, including generic (eg. park, lake, ocean, river, swimming pool, beach, scenic mountain range, city street) and specific ones (deserted stadium on a rainy day, a luxurious hotel, small oceanarium).\n\nFurther, we find that with recent LLMs (eg. LLAMA-2 13B chat), we can generate challenging background edits (eg. ''rare celestial events'') with _no additional finetuning_! This opens up the possibility of using LANCE to stress-test against custom variations beyond the ones described in this paper.\n\n> Is this just another form of discovering \"adversarial attack\" on the model predictions? \n\nNo, we do not consider LANCE as an adversarial attack on model predictions, since by design, it is not optimized end-to-end to fool a downstream classifier, and interventions are only made through language. We will clarify.\n\n> Does this method have any correlation with the noisy labels? \n\nGreat question! To investigate, we cross-reference labeling errors found in Northcutt _et al._ [A] for the ''howler monkey'' class in ImageNet. They find only a single mislabeled example (true label: ''white-headed capuchin''), whereas we discover a consistent failure mode (reliance on background color). Further, in Table 2 we also report accuracy@5, which is less sensitive to changes in predictive order when multiple labels are plausible, and find that LANCE-generated test sets also lead to larger drops in accuracy@5 (eg. 3% for ResNet-50). \n"}, {"review_id": "wrm8PrZaQb", "reviewer": "Reviewer_muqu", "summary": "This paper proposed a new testing protocol LANCE for existing vision models.\nIt adopts LLM-based caption perturbation and DDPM based image editor to generate counterfactual samples. \nSpecifically, for each image, LANCE first use a pre-trained captioner (BLIP-2) to generate the corresponding caption. \nThen it adopts an LLM to perturb the caption with different factors (subject, object, background, etc.).\nThe goal of perturbation is to generate a caption corresponding to a counterfactual image with semantics preserved.\nWith the perturbed caption, the author generates another image with counterfactual aspects and similar looking with the original image.\nThe authors observed a significant performance drop when testing image classification models on these generated images.\nFurthermore, the authors also present a detailed analysis based on the perturbation factors.", "questions": "See Weakness.", "limitations": "See Weakness.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors establish an interesting pipeline to generate counterfactual hard examples for testing vision models. It involve an LLM-based image caption perturbator and a ddpm-based image editor. Qualitative results show that this pipeline can generate some impressive counterfactual images. \n2. The authors present a detailed analysis on the performance degradation when testing on the generated counterfactual images. ", "weaknesses": "1. The authors should present more quantitative statistics about the counterfactual examples generation pipeline, for example:\n    1. What is the success rate of an LLM to generate a plausible perturbed caption (GPT-3.5 & Finetuned LLAMA-7B). Also, I noticed that you are using Nvidia A40 (48GB), thus a comparison between different LLAMA variants (+13B, e.g.) are also welcomed. \n    2. What is the success rate of DDPM-based image editing? I'm specially interested in that since I observe poor quality when using StableDiffusion to generate common images (compared to other advanced AIGC models).\n2. Are all newly generated images (781 images, according to the supp) verified manually by human? If so, how many worker-hours are required to generate these samples?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review!\n\n> What is the success rate of an LLM to generate a plausible perturbed caption\n\nPlease see our general response above for a quantitative evaluation of our caption perturbation strategy: to summarize, we find the perturber to be 89% accurate at following the prompt provided (i.e. _correctly_ editing _only_ the domain if instructed to do so). We note that even in failure cases (eg. additionally changing an irrelevant word), the perturbed captions are almost always _plausible_. Further, edits that mutate the ground truth are correctly filtered out 98.4% of the time. \n\n> What is the success rate of DDPM-based image editing? \n\nAs our general response shows, our generated dataset is adjudged to be high quality (via a human study) in realism, edit success, and fidelity. Respondents also confirm label validity for 96.8% of generated images. We note that generating high-quality edits for ImageNet-style images with Stable Diffusion requires several additional measures that we describe in Sec. 3.2, such as using prompt-to-prompt with null-text inversion and (automated) hyperparameter tuning. That said, we emphasize that LANCE is not tied to a specific text-to-image model and can easily be updated to include more recent generative models.\n\n> Are all newly generated images (781 images, according to the supp) verified manually by human? If so, how many worker-hours are required to generate these samples?\n\nYes. As detailed in our general response above, our generated images are adjudged via a human study (11 respondents, 15-20 minutes per response) to have a high image and edit quality and fidelity, while almost always preserving the ground truth label. Further, we also manually screen all 781 images for ethical and quality issues (which took approximately 1 hour), in the process flagging 8 images that we filter out. \n\n> A comparison between different LLAMA variants (+13B, e.g.) are also welcomed.\n\nWe use zero-shot prompting (using the same prompts as before, see Table 1 in supp.) with the recently released LLAMA-2-13B chat model. We find that even without any task-specific finetuning, it leads to high-quality text edits, for eg:\n\n**LLAMA-2 Prompt**: Generate all possible variations of the provided sentence by only changing the weather conditions to a _rare celestial event_: \"a photo of a red umbrella sitting on top of a lush green field next to a sidewalk next to a building\"\n\n**Response:**\n1. \"a photo of a red umbrella sitting on top of a lush green field next to a sidewalk next to a building under a lunar rainbow.\"\n2. \"a photo of a red umbrella sitting on top of a lush green field next to a sidewalk next to a building during a meteor shower.\"\n\n...\n\n10. \"a photo of a red umbrella sitting on top of a lush green field next to a sidewalk next to a building during a rare solar eclipse.\"\n\nBesides obviating additional finetuning, this also opens up the possibility of using LANCE to stress-test against custom variations beyond the ones described in this paper. "}, {"review_id": "H04V5POJoI", "reviewer": "Reviewer_9YtG", "summary": "The paper proposes a way to generate new data to evaluated the robustness of model. It first captions an image, then changes part of it, then generates an image from the modified caption. The model is then evaluated on these new images to see how robust they are to things like background changes.", "questions": "How can the quality of the generated dataset be evaluated? How do we know it is generating a good benchmark?", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is interesting and is an easy way to generate new evaluation data. The paper is well written and easy to follow. The approach would be a valuable tool to evaluate models.", "weaknesses": "Since it is an automated benchmark, it is unclear how good the quality is. It relies on a lot of parts working right (captioning model, caption changing model and image generation model).  It would be good to have some human evaluation in Table 2 to see how well humans do on this set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review, and for recognizing the potential of LANCE as a valuable diagnosis tool!\n\n> It would be good to have some human evaluation in Table 2 to see how well humans do on this set. How can the quality of the generated dataset be evaluated? How do we know it is generating a good benchmark?\n\nPlease see our general response above for results from a human study evaluation of the quality of the generated dataset, as well as quantitative evaluations of each component of our method. \nOur dataset is adjudged to be high quality (via a human study) in realism, edit success, and fidelity. Additionally, we manually inspect all images in the dataset and filter out images that are potentially unethical. \n\nWhile measuring human accuracy in Table 2 is a great suggestion, having humans perform 1000-way classification is challenging;  as a proxy, we instead measure the label _consistency_ of our generated images, to ensure that the original ground truth label is still valid: encouragingly, our respondents find this is the case for 96.8% of images on average.\n"}, {"review_id": "A4a5RsznFX", "reviewer": "Reviewer_Fgt7", "summary": "This paper presents a method for stress-testing visual classifiers. The key idea is to use language to generate counterfactual images. Their approach can be summarized into four steps: 1) take an image input; 2) get its caption from a captioning model (BLIP-2); 3) perturb the caption by changing some words using ChatGPT and fine-tuned LLAMA; 4) generate new images based on perturbed captions using text inversion of Stable Diffusion. On the ImageNet dataset, they showed that various models such as ResNet-50 and ViT-B consistently achieve much lower performance on their generated new ImageNet compared to the original ImageNet, and the generated ImageNet enables surfacing model biases.", "questions": "- Algorithm 1: \n  - Line 7 duplicate definition of \"c\" (previously used as the class, now used as the caption)\n  - Line 13: \">\" should be \"<\"\n\n- Equation 2: \"|\" in the denominator should be \"||\"\n\n- Line 282: what is the percentage for the generated counterfactuals to be semantically inconsistent with the ground truth label?", "limitations": "See weaknesses. I'm happy to increase my rating if these are well addressed, especially 1 and 2.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper targets a novel direction --- stress-test visual models, which is important in real-world vision applications.\n\n2. The proposed method is technical sound, and the generated images from their pipeline look good and realistic.\n\n3. The paper conducted experiments on the realistic ImageNet dataset instead of simple synthetic datasets, showing the practical values of their method in realistic scenarios.\n\n4. This paper is generally well-written and easy to understand.", "weaknesses": "1. There is no baseline to compare, and the numbers are hard to justify their method's effectiveness. The main quantitive results from this paper showed that various vision models achieve much lower performance on their generated ImageNet dataset. However, it is unclear whether the generated images are 100% correct. If the text perturbation happens to change the class or introduce new classes to the image, the model prediction is expected to change and the performance will drop. Therefore, a human study showing the label validity of generated images is particularly necessary.\n\n2. There are some existing works proposing very similar directions. For example, [1] is almost the same as this work, which uses textual reversion to convert an image into a token and compose this token with natural language and generate new images to test vision models. [2] uses off-the-shelf image captioning and generation models to discover model bugs. While [1] can be viewed as concurrent work, it would be helpful if the authors could discuss the similarity and differences between this work and [1-2].\n\n3. From Figure 6 last row, changing the caption from \"car\" to \"bus\" generates almost the same images, but model predictions change a lot. The change may be because generated images have specific patterns that are not perceptible to humans, which cannot reflect the actual errors using real images. \n\n4. The stress test will be limited by language itself, as language is hard to represent certain properties of objects, such as orientation and lighting. However, this is also pointed out by the authors and is minor considering the advantage of using language.\n\n[1] Dataset interfaces: Diagnosing model failures using controllable counterfactual generation. https://arxiv.org/pdf/2302.07865.\n\n[2] Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning. https://arxiv.org/pdf/2208.08831.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful feedback!\n\n> … a human study showing the label validity of generated images is particularly necessary.\n\nThanks for the suggestion! Please see our general response above for results from a human study evaluation of the quality of our dataset. Our generated dataset is adjudged to be high quality along several metrics, including image realism, edit success, fidelity, and ethics. Importantly, *respondents confirm label validity for 96.8% of generated images*. \n\nAlso to clarify, we do propose and benchmark two additional baselines: performance on the image reconstruction from the latent diffusion model (''Reconstructed'' in Table 2), in order to verify DDIM inversion success, and a masked language-modeling based baseline (''LANCE-R'' in Table 2), both of which significantly lag behind LANCE. \n\n> It would be helpful if the authors could discuss the similarity and differences between this work and [1-2].\n\nThanks for pointing these out! \n\nWhile our work and Wiles _et al._ [1] do share similarities (both use text-to-image and image-to-text models), they actually differ _significantly_ in both motivation and approach: Wiles _et al._ [1] uses the text description of a misclassified image as a cluster seed to generate additional images, and measures whether the cluster failure rate exceeds the baseline failure rate for that class. While this may discover failure modes with plausible rationales, these insights themselves might be less reliable: text-to-image models can generate a set of visually diverse images that satisfy a given prompt, which could each result in a misclassifcation for a different reason rather than due to a specific shared visual attribute.  \n\nIn contrast, LANCE generates counterfactuals via targeted editing (using prompt-to-prompt tuning with null-text inversion). By altering only a _specific_ visual attribute, we can be confident that a change in model confidence is indeed due to that change, and thus obtain more reliable insights. Further, by using each image from an IID test set as a potential probe rather than failure cases alone, LANCE can surface vulnerabilities even for classes with seemingly high performance (eg. for the ''sunglasses'' class, models performance is initially high but drops when the domain is changed to painting, see Fig. 6).\n\nWhile concurrent work by Vendrow _et al._ [2] claims to generate counterfactuals from user-defined (rather than automatically generated) variations, they adopt a more relaxed definition for counterfactual: for eg. consider the generated counterfactual examples highlighted in Fig. 5 of their paper, which differ from their original images in more ways than one. This is presumably due to learning a class-specific text token (via inversion) which is then used to generate counterfactuals. As described before, this can confound analysis, since images are altered in unpredictable ways. In contrast, our approach results in significantly better edits (eg. see Fig 5 in our paper and Fig 4 in supp.), and therefore yield more reliable insights into model vulnerabilities.\n\n[1] Wiles et al., Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning. https://arxiv.org/pdf/2208.08831.\n\n[2] Vendrow et al., Dataset interfaces: Diagnosing model failures using controllable counterfactual generation. https://arxiv.org/pdf/2302.07865.\n\n> The change may be because generated images have specific patterns that are not perceptible to humans, which cannot reflect the actual errors using real images.\n\nLANCE surfaces a diverse range of model failure modes, some of which are highly interpretable (eg. sensitivity to background color for the ''howler monkey'' class), while others may be less so (as correctly pointed out by the reviewer). We include examples of both to highlight the diversity of discovered failure modes; admittedly, not all lead to actionable insights. \n\nThank you for pointing out errors in the text, we will incorporate all suggestions!\n\n"}]

Author rebuttal:
We thank reviewers for their thoughtful feedback. We are delighted that they found our problem novel and broadly applicable (reviewers Fgt7, G3fV), our method valuable and technically sound (reviewers muqu, 9YtG), our experiments realistic and reproducible (reviewers muqu, Fgt7, G3fV), our results impressive (reviewers muqu, Fgt7), and our paper very well-written (reviewers G3fV, 9YtG, Fgt7). We also appreciate and will incorporate the feedback from our ethics reviewers. **We will make our dataset publicly available as requested**.

The primary concern shared by reviewers pertains to the limited evaluation of LANCE’s components and an inadequate discussion of its limitations and ethical implications. 

We take these concerns seriously, and describe below our efforts to address them.

1) **Evaluation**. We independently evaluate (**full tables in PDF**): 


    a) **Caption perturber**. In Fig. 4 of the paper, we verified caption realism by reporting the low average perplexity of the generated and perturbed captions. We now also evaluate the accuracy of the perturber at making the type of edit specified by the prompt (eg. at _correctly_ changing _only_ the domain for a domain edit). To do so, we manually validate a random subset of 500 captions and their perturbations (100 per-type) generated by LANCE. We observe an overall accuracy of 89%, with the _object_ (84%) and _adjective_ (82%) types achieving the lowest accuracies: we find that most failures for these types result from incorrect identification of the object or adjective in the sentence. We note that even in failure cases (eg. also changing an irrelevant word), the edits made are still reasonable.

    Next, we evaluate the efficacy of the checks and balances we insert to weed out caption edits that alter the label. We first label the same subset of 500 examples by hand with the correct action (filter / no filter), and compare against our filtering strategy. Across perturbation types, our filtering strategy achieves both high precision and recall, with a low overall false positive (2.8%) and false negative (1.6%) rate. Of these, a majority of false negatives (edits that alter the ground truth that we fail to catch) are for the _subject_ and _object_ types: we find these typically change the ground truth inadvertently. For example, for an image of a typewriter labeled ''keyboard space bar'', the edit typewriter$\to$painting erroneously passes our filter but mutates the ground truth, as paintings do not usually have space bars.


    b) **Image generator.** In Fig. 4., we verify the realism of the generated images by reporting FID scores. As recommended by reviewers, we now conduct a human study to measure performance along 5 axes: 

    * **Image Realism** (1-5, 5=best): How easy is it to tell that the counterfactual image was generated by AI?

    * **Edit success** (1-5, 5=best): Does the generated image correctly incorporate the edit?

    * **Image fidelity** (1-5, 5=best): Are all the changes made relevant towards accomplishing the edit?

    * **Label consistency** (Yes/No): Is the original image label still plausible for the generated image? 

    * **Ethical issues** (Yes/No + Text box): Is the generated image objectionable, or raise ethical concerns around consent, privacy, stereotypes, demographics, etc.?

    We collect responses from 11 external respondents for a random subset of 50 <image, ground truth, generated caption, perturbed caption, counterfactual image> tuples (10 per-type), and report mean and variance. See the rebuttal PDF for a screenshot of the interface and full tabulated results. 

    Images generated by LANCE are rated to be high on realism (4.3/5 on average)  and fidelity (4.6/5 on average), and slightly lower on edit quality (3.8/5 on average). Of these, we find that background edits score lowest on edit success, due to sometimes altering the wrong region of the image. Further, generated images have high label consistency (96.8%), indicating that our checks and balances are effective at preserving the ground truth label. Finally, <2% of images are found to raise ethical concerns: we discuss these in detail below.



2) **Ethics and limitations.** On inspection of the images marked as objectionable in our human study, we find that most arise from bias encoded in the generative model: eg. for the edit woman$\to$lifeguard, the generative model also alters the perceived gender and makes the individual muscular. Further, for people$\to$athlete, the model also alters the person’s race. The model also sometimes imbues stereotypical definitions of subjective characteristics such as ''stylish'' (eg. by adding makeup). To address this, we manually inspect the entire dataset of 781 images for similar issues and exclude 8 additional images. Going forward, we hope to address this in the generation pipeline itself, by excluding inherently subjective text edits, and by using recent text-to-image with improved fairness and steerability.

    **Limitations.** In Fig. 5 of supp., we categorized and visualized the most common failure modes of our approach (re-included in rebuttal PDF). We enumerate them below:

    a) Label-incompatible text edit: eg. black howler monkey$\to$purple howler monkey (howler monkeys are almost always black or red).

    b) Poor quality image edit: Images with low realism, which are largely eliminated via our checks and balances.

    c) Image edit that inadvertently changes the ground truth: eg. typewriter$\to$stapler for the class ``space bar’’ (staplers do not contain space bars).

    d) Inherited bias: eg. changing perceived gender based on stereotypes.

    e) Meaningless caption edit: eg. boy in back seat $\to$ driver in the back seat.

    Such failures can confound an observed performance drop over a LANCE-generated counterfactual image. However, we observe such failures rarely, and find that class-level and aggregate performance variations still lead yield reliable insights. 


Meta-review: After discussion with the authors, all reviewers recommend acceptance to varying degrees. The AC does not see sufficient grounds to overturn this consensus.

It is important to note that this outcome is in part the result of three human studies presented during the discussion period that support the efficacy of two submodules. These studies are relatively small scale (largest at 11 participants), were presumably performed in a short time period, and two of the three rely on authors as subjects. From the AC's perspective, these therefore represent promising preliminary results rather than rigorous evaluations. Given the other merits of the work, these are still useful for building trust about the methodology. Authors are encouraged to strengthen these studies if possible and be clear about their limitations in the camera ready.

The discussion of limitations presented in the overall response and the rebuttal are useful. The AC believes these should be elevated to the main paper if possible. Likewise, the discussion with reviewer G3fV of precisely what diagnostic scenarios LANCE is likely to enable is useful context for readers. Discussion of these and their potential interactions with the method's limitations should be included.
Decision: Accept (poster)