Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning
Authors: Olivia Wiles
Isabela Albuquerque
Sven Gowal
Venue: Submitted to ICLR 2023 (2023)
Keywords: robustness
failure discovery

Abstract: Automatically discovering failures in vision models under real-world settings remains an open challenge. This work shows how off-the-shelf, large-scale, image-to-text and text-to-image models, trained on vast amounts of data, can be leveraged to automatically find such failures. In essence, a conditional text-to-image generative model is used to generate large amounts of synthetic, yet realistic, inputs given a ground-truth label. A captioning model is used to describe misclassified inputs. Descriptions are used in turn to generate more inputs, thereby assessing whether specific descriptions induce more failures than expected. As failures are grounded to natural language, we automatically obtain a high-level, human-interpretable explanation of each failure. We use this pipeline to demonstrate that we can effectively interrogate classifiers trained on ImageNet to find specific failure cases and discover spurious correlations. We also show that we can scale the approach to generate adversarial datasets targeting specific classifier architectures. This work demonstrates the utility of large-scale generative models to automatically discover bugs in vision models in an open-ended manner. We also describe a number of limitations and pitfalls related to this approach.

Under review as a conference paper at ICLR 2023
DISCOVERING BUGS IN VISION MODELS USING OFF- THE-SHELF IMAGE GENERATION AND CAPTIONING
Anonymous authors Paper under double-blind review
ABSTRACT
Automatically discovering failures in vision models under real-world settings remains open challenge. This work shows how off-the-shelf, large-scale, image- to-text and text-to-image models, trained on vast amounts of data, can be leveraged automatically find such failures. In essence, a conditional text-to-image genera- tive model is used to generate large amounts of synthetic, yet realistic, inputs given ground-truth label. A captioning model is used describe misclassified inputs. Descriptions are used in turn to generate more inputs, thereby assessing whether specific descriptions induce more failures than expected. As failures are grounded to natural language, we automatically obtain high-level, human-interpretable explanation of each failure. We use this pipeline to demonstrate that we can effec- tively interrogate classifiers trained on IMAGENET to find specific failure cases and discover spurious correlations. We also show that we can scale the approach to generate adversarial datasets targeting specific classifier architectures. This work demonstrates the utility of large-scale generative models to automatically discover bugs in vision models in open-ended manner. We also describe number of limitations and pitfalls related to this approach.
INTRODUCTION
Deep learning has enabled breakthroughs in wide variety of fields (Goodfellow et al., 2016; Krizhevsky et al., 2012; Hinton et al., 2012), and deep neural networks are ubiquitous in many applications, including autonomous driving (Bojarski et al., 2016) and medical imaging (De Fauw al., 2018). Unfortunately, these models are known to exhibit numerous failures arising from using shortcuts and spurious correlations (Geirhos et al., 2020a; Arjovsky et al., 2019; Torralba et al., 2011; Kuehlkamp et al., 2017). As a result, they can fail catastrophically when training and deployment data differ (Buolamwini & Gebru, 2018). Hence, it is important to ensure that models are robust and generalize to new deployment settings.
Yet, only few tools exist automatically find failure cases unseen data. Some methods analyze the performance models by collecting new datasets (usually by scraping the web). These datasets must be large enough to obtain some indication of how models perform on particular subset of inputs (Hendrycks et al., 2019; 2020; Recht et al., 2019), Other methods rely on expertly crafted, synthetic (and often unrealistic) datasets that highlight particular shortcomings (Geirhos et al., 2022; Xiao et al., 2020).
In this work, we present a methodology to automatically find failure cases of image classifiers in an open-ended manner, without prior assumptions on the types of failures and how they arise. We leverage off-the-shelf, large-scale, text-to-image, generative models, such as DALL-E 2 (Ramesh et al., 2022), IMAGEN (Sabaria et al., 2022) or STABLE- --FFU-DOO (Rombach et al., 2022), to obtain realistic images that can be reliably manipulated using the text prompt. We also leverage captioning models, such as FLAMINGO (Alayrac et al., 2022) or LEMON (Hu et al., 2021), to retrieve human- interpretable descriptions of each failure case. This provides the following advantages: (i) generative models trained on web-scale datasets can be re- re-used and have broad non-domain-se -odddi--specffiie coverage; (ii) they demonstrate basic compositionality, can generate novel data and can faithfully capture the essence of (most) prompts, thereby allowing images be realistically manipulated; (ii)) textual

---

Under review as a conference paper at ICLR 2023
descriptions of failures can be easily interpreted (even by non-experts) and interrogated (e.g., by performing counterfactual analyses). Overall, our contributions are as follows:
We describe a methodology to discover failures of image classifiers trained on IMAGENET (Deng et al., 2009). To the contrary of prior work, we leverage off-the-shele generative genee--- models, thereby avoiding the need to collect new datasets to rely on manually crafted synthetic images.
Our approach surfaces failures that are uuman-interpretabbl by clustering and captioning inputs on which classifiers fail. These captions can be modified to produce alternative hypotheses of why failures occur allowing insights into the limitations of a given model.
We demonstrate the scalability of the approach by generating adversarial datasets (akin to IMAGENET-A; Hendrycks et al., 2019). In contrast to IMAGENET-A, our new generated datasets align more closely with the original training distribution from IMAGENET and generalize to multiple classifier architectures.
Importantly, while this work focuses on vision models trained on IMAGENET, it is neither limited to IMAGENET nor the visual domain. It serves as proof-of-concept that demonstrates how large-scale, generative models (Bommasani al., 2021) can be combined automate the discovery of bugs in machine learning models and produce compelling, interpretable descriptions of model failures. The approach is agnostic to the model architecture, which can be treated as a black box.
2 RELATED WORK
Model failures. Spurious correlations can entice models to learn unintended shortcuts that obtain high accuracy on the training set but fail to generalize (Lapuschkin et al., 2019; Geirhos et al., 2020a). Recht et al. (2019) show that the accuracy of IMAGENET models is impacted by changes in the data collection process, while Torralba et al. 1); Khosla et al. (2012); Choi et al. (2012) explore how contextual bias affects generalization. Mania et al. (2019) demonstrate that models trained on IMAGENET make consistent mistakes with one another and Geirhos et al. (2020b) show that these mistakes are not necessarily consistent with human judgment.
Evaluation datasets. Understanding how model failures arise and empirically analyzing their consequences often requires collecting and annotating new test datasets. Hendrycks et al. (2019) collected datasets of natural adversarial examples (IMAGENET-A and IMAGENET-O) to evaluate how model performance degrades when inputs have limited spurious cues. Hendrycks et al. (2020) collected four real-world datasets (including IMAGENET-R R) to understand how models behave under distribution shifts. In many cases, particular shortcomings can only be explored using synthetic datasets (Cimpoi et al., 2013). Hendrycks & Dietterich (2018) introduced IMAGENET-C, a synthetic set of common corruptions. Geirhos et al. (2018) propose to use images with texture-shape cue conflict to evaluate the propensity of models to over-emphasize texture cues. Xiao et al. (2020); Sagawa et al. (2020) investigate whether models are biased towards background cues by compositing foreground objects with various background images (IMAGENET-9, WATERBIRDS). In all cases, building such datasets is time-consuming and requires expert knowledge.
Automated failure discovery. In some instances, it is possible to distill rules or specifications that constrain the input space enough to enable the automated discovery of failures via optimization or frute-force search. In vision tasks, adversarial examples, which are constructed using 4 -p-morm bounded perturbations of the input, can cause neural networks to make incorrect predictions with high confidence (Carlini & Wagner, 2017;;b; Goodfellow et al., 2014; Kurakin et al., 2016; Szegedy et al., 2013). In language tasks, some efforts manually compose templates to generate test cases for specific failures (Jia & Liang, 2017; Garg et al., 2019; Ribeiro al., 2020). Such approaches rely on human creativity and are intrinsically difficult to scale. Several works (Baluja & Fischer, 2017; Song et al., 2018; Xiao et al., 2018; Qiu et al., 2019; Wong & Kolter, 2021; Laidlaw et al., 2020; Gowal et al., 2019) go beyond hard-coded rules by leveraging generative and perceptual models. However, such approaches are difficult to automate as it is unclear how to relate specific latent variables to isolated structures of the original input. Finally, we highlight concurrent work (Ge et al., 2022), which leverages captioning and text-to-image models to construct background images to evaluate (and improve) object detector. Their approach requires compositing the resulting images with foreground objects and is not open-ended, in the sense that it requires a dataset of background images. Perhaps, the work by Perez et al. (2022) on red-teaming language models is the most similar ours.
2

---

Under review as a conference paper at ICLR 2023
Perez et al. demonstrate how to prompt language model to automatically generate test cases to probe another language model for toxic and other unintended output.
Interpretability. Interpretability techniques aim to explain a rationale behind individual model predictions. They surface information that allows user to inspect how model behaves. Casper et al. (2022) demonstrate that feature-level attacks, which create adversarial patches, can help diagnose brittle feature associations. However, much like LIME (Ribeiro et al., 2016) or GRAD-CAM (Selvaraju et al., 2019), the results are difficult to understand. Other works (Abid et al., 2022; Jain et al., 2022a; Eyuboglu al., 2022) leverage auxiliary information in the form of attributes embeddings (e.g.. from CLIP; Radford et al., 2021) to provide explanations in natural language. However, these methods often rely additional dataset which limits their scope.
image-to-text
METHOD
Notation. We consider a classifier f x + Y. where X is the set of inputs (i.c., images) and Y the label set. We also assume that inputs distribution p(x[z, y) conditioned on latent representations. 2 E Z. In the context of this specific work, is a textual description of the image æ. We are interested in discovering captions corresponding to images æ ~ p(x|z./) z. 3) with label y that lead to significantly higher misclassification rates than generic images drawn from the marginal distribution p(xly) conditioned solely on the label. Formally, given a label y, we would like to find x with
X with label y E Y are drawn from an underlying
E
((xx)())
E
[f(z) + y
(1)
#~p(X z.
x>~p(p())
where [-] represents the Iverson bracket. We may also be interested in identifying specific misclassifi- cations towards a wrong (target) label ÿ F y. and would like that
E
[f(x)) = Ø
E
[fxx) Ø)
x k.y]
As we do not have access to the true underlying distributions p(x|z, y) and p(x(y), we leverage large-scale text-to-image model (such as DALL-E 2 or IMAGEN) to approximate them. Similarly, we approximate p(z æ. y) with a captioning model (e.g., FLAMINGO or LEMON). We denote approximations of these distributions with the symbol p.
For each of the following steps, we highlight additional implementation details and explain how we construct prompts for the text-to-image and image-to-text models. Some minute details are omitted as to not break anonymity and will be added later.
Generating failure cases. Our approach is described in Fig. 1. It consists of initially finding baseline failures for the underlying model by sampling inputs from p(xly). Given label of interest y, the output of this step is a set ID = az ~ ) (where N is the number of images we wish to generate), a set Dia = {x E f))(()) ) o y} and an estimate of the baseline failure rate [Drail/ /N (corresponding to the right-hand side of Eq. 1). In this work, we consider problematic misclassifications only so we restrict ourselves to failures where any of the top-3 predicted labels are not under the same parent as the true label in the WORDNET hierarchy (Miller, 1995).2
This step leverages a text-to-image generative model conditioned on y with prompts such as "a realistic photograph of a (insect)." ", which are automatically generated from the corresponding class and WORDNET hierarchy. As our domain of interest is composed of real images (much like the images in IMAGENET), the prompt is designed enforce the generation of photographs of real objects and animals rather than drawings or paintings. Further prompt engineering can be explored to find failures on wide variety of domains (such as sketches or medical imaging; Hendrycks et al., 2020; Kather et al., 2022).
'In practice, it is possible to steer the generation process to induce more frequent failures (e.g.. by optimizing latents conditioning via gradient ascent the cross-entropy loss; Wong & Kolter, 2021). However, as proof-of-concept, we assume only black-hox access to off-the--hhlf text-to-image model.
"Other options include considering only misclassification where the true label is not in the top-k predicted lahels where the confidence the true label is lower than predefined threshold.
3

---

Under review as a conference paper at ICLR 2023
1
2
3
Generaaiig fallure cases
Fallure case captioning
Meswaring fallure rates of hopotheace
Generate imagis from haseline prompt
Gonerate more images from failure prompt eg.. realude photograph of (insect). It a yellow flower. lackgmand gown."
*g. 'a reatbstk photogroph of fly (iseect)?
realisil photogmphh ofa (lnsect). It yellow flower: the bockground green."
((x) (p)
~ f( (at, y)
IOUR
Classify measure ralle
f(x|=,
Cumpare Gallure rales and surface chosters leading to higher failures rates
Clastif and measure fatkrre rate
MLLR
VILK
biet
TOLK
Cagtion refinsnncnt
malliic photograph of finsect). it
fv
boe
(e-s)))))) a)) (f(se) #
yollow flwwer. bockground grean."
Px [f(m)) * xl
"a reallsti photograph of By (insect. it on flower."
DCLB
ICLB
beer
Bee
Figure 1: Diagram of our method, The method starts by generating images containing a given class to measure the baseline failure rate of that class Eight-hand side of Eq. 1). We construct textual description for cach misclassified image. This description is used to produce new images and measure the failure rate on images corresponding that description (left-hand side of Eq. I). The final description can be edited (manually or automatically) to understand the source of the failures.
Optional clustering failure cases. Clustering is not necessary, but makes the search for the caption z leading to high failure rates more efficient and computationally manageable. First, we split D by predicted label, ie. Diai ( U U ( where fail f f = y) Then, for each subset ( we group inputs that have similar feature representations together (e.g.. using the cosine distance between intermediate activations of a pretrained model). Many other choices of clustering method exist. The goal of this step is to reduce the number of clusters to a minimum without amalgamating different causes of failure together."
Failure case captioning. For each cluster A C Diii we would like to find caption Z4 that describes it. Grounding failures in simple textual descriptions allows us to maintain the diversity of the generated images: the generated images resemble the images leading to the original failure without being exact copies. Ideally. we would like to find the caption z that maximizes the likelihood of sampling elements of A, i.e., & arg max II P(x|z. y). We may wish to impose constraints on Za, such as a maximum number of words or sentences." 4 Finding such a caption is computationally hard and measuring exact likelihoods P(x) z, y) can be challenging. Hence, we resort to sampling captions directly from captioning model p(z az, y) for each image of cluster A.5 Captions are split into sentences, resulting in a set of sentences S. Sentences are greedily combined (up to maximum number of sentences K) to maximize the likelihood of sampling the overall caption ZA = [s)..... sx] with s, = arg max IL P([s)...... Sj-1, s] [z, v) and 81 = arg max,eg ses ILEA P[[s]|æ, y). If no clustering performed, we can directly sample caption [[=] ~ p(z|a, from the captioning model for each element x in Daa Each caption serves as failure hypothesis.
A Wrongffll clustering may lead the next step to produce descriptions that fail to induce more failures. As a result, we may miss failures that we would have discovered without clustering, but the failures that we do discover remain valid.
*These constraints guarantee that captions remain simple and not overly descriptive.
Thiis formulation implicitly assumes that any caption is as likely another under given label which in general does not hold true, but serves reasonable approximation.
4

---

Under review as a conference paper at ICLR 2023
This step leverages an image-to-text captioning model and forces descriptive completions via few-shot prompting. In particular, we use three shots, with each shot being image and caption pair. Images are publicly available online and are manually described using between four to seven short sentences that describe the subject of the photograph and its physical position with respect to the camera, as well as the background or context in which the subject appears (see Appendix A for more details). To condition on y, we force the captioning model to only consider completions to the original baseline prompt, which guarantees that the final caption contains the true label. An example of resulting caption is "a realistic photograph of fly (insect). the background is blurred. the fly is in focus. it is on yellow flower. the background is green." Each caption serves as a failure hypothesis.
Measuring failure rates of hypotheses. For each failure hypothesis or caption 2Λ. we can measure its failure rate via sampling Ex-ppxl sa.a) (fx) # This step allows us surface captions 2" that satisfy Eq. I (or alternatively Eq. 2) by comparing the resulting failure rate with the baseline failure rate obtained initially.
Much like the first step on generating the baseline failure cases, this step uses the text-to-image model, which we now prompt with descriptive cluster captions such as "a realistic photograph of fly (insect). the background is blurred. the fly is in focus. it is on a yellow flower. the back is green."
Caption refinement and counterfactual analysis. Given a caption z", we would like to provide a shorter, self-contained caption that obtains a similar failure rate. For this step, we rely on simple rules and evaluate promising caption rewrites. Finally, as captions are human-readable, users can interact with the system and test alternative hypotheses.
In our implementation, we exploit two rules: (i) we evaluate all individual sentences (for our ongoing example, these would be "the background is blurred", "the fly is in focus", "it is on a yellow flower" " and "the background is green") in conjunction with the original prompt (e.g.. "a realistic photograph of a flyy (insect)") "); (ii) the most promising prompt is further refined by dropping adjectives (such as "yellow" in "il is on a yellow flower"). More sophisticated rules and rewrites are possible (Ribeiro et al., 2018). Alternatvely, could leverage a harge-langaage model via few-shot prompting to automate this process (Witteveen & Andrews, 2019).
Finally, we note that it is possible to use part of this pipeline to understand known failure cases (e.g.. on failures reported by external users of the model f).
4 RESULTS
In this section, we elaborate on two use-cases. In the first, we find failure cases of a Residual Network (ReSNET) (He et al., 2016) model with 50 layers (RESNET-50), trained on IMAGENET, in an open-ended manner. We focus three arbitrarily chosen labels and show that the failures obtain arise from consistent misclassification caused by spurious correlations. In the second use-case, we show that we can generate failures scale for various models. We leverage IMAGENET-A A as source of initial failures and create new variants of IMAGENET-A A that target specific model architectures.
4.1 OPEN-ENDED FAILURE SEARCH
Setup. We evaluate a RESNET-50 trained on IMAGENET and available on TU- ? We select three labels at random: Persian cat, fly and crayfish. For each label y. we manually select target labels ÿ (snow leopard, bee and chainlink fence respectively) and execute the protocol defined in Sec. 3. At each step, we sample images from the generative model until we gather 20 images that misclassified the target label j and compute failure rates that point. Fig. 2 and Table show these automatically discovered failures. More failures for additional true and
This may seem at odds with our claim on open-endedness. However, we note that this step is optional and its goal is simply to produce shorter failure deseriptions. https: / / fhub. dev/ goog/ e/ weaggenet/renetttteee 10//5///llla assi www.asaaaaaaaccccc fication/5
5

---

Under review as a conference paper at ICLR 2023
realatic chotegraph a
TaN a Photoomah Aesion Sirmoait aemall
rellittc shatrenchh ff
nelatt shuteeokk of
sholopraph of cresliin
malak phziagaan ofa
Proran
Pelinese
Pomaron an
Jon dog
snow locpard
Porsorsnian
ion
1ghr haatte
deaganiy
cicado
adlyuug
Eiger Beettin scorp Loc
buaf beuile
dalsy (irboot
eilly
contlodd
tncycte
ant
carachute
stick
Teneygomb
streen
cicada
Bouran
(peda
marmoset
Japanese
whiae wolf
Wccvil
morerch
Tano homod beetie
repessed
'iper beetle
100g horred best
der
St Sratand Shocpdog
colle
Molhese dog
komondor
iTammozet
slack
one
and gold
leathosper
monatch
comicc book
pinwhool
electricc Tann
armadilo
10%
10%
10%
1%
1095
10%
1%
156
10%
Normalized falure rates
Normal Fodd fallure rates
Normalized falure rates
Normal lzed fallure rates
Normalizod fature rates
Normaizedd faillune rates
(a) Persian cat
Snow leopard
(b) rly Bee
(c) Crayfish + Chainlink Sence
Figure 2: Distribution of failures of a RESNET-50 for the baseline and automatically discovered captions for three true and target label pairs. For each panel, failures resulting from the baseline caption are on the left and failures resulting from the discovered caption are on the right. We show the top-15 mistakes and three randomly sampled images for each caption. We highlight red the bar corresponding to the target label. Absolute failure rates are given in Table 1.
label
Target label
Caption
Failure (any)
Failure (lieget)
Pezsian cat
Snow leapard
realistic pbotograph of Persian car (domestic animal).
background is green.
0.11%
1x
0,000228
0.0032%
14x
Fly
Bea
realisic photograph of fy (insect).
it on a flower.
4.11%
0.48%
1x
9x
0,72%
0.001455
497x
Crayfish
Uhsinlink
realistic phorograph crayfish (crustncean)
0.00045%
in net.
6.31%
1.73%
3721x
Table 1: Absolute failure rates of RESNET-SO for three true and target label pairs. We show the total failure rate (i.e.. the model prediction is different from the true label) and the target failure rate (i.e., the model prediction is the target label). Captions are automatically discovered using the method detailed in Sec. 3.
target label pairs are in Sec. B.1 in the appendix. In Sec. B. we also evaluate other architectures (i.e., VITs) and demonstrate that the discovered captions yield failure rates that are statistically significant.
Discovered failures. Fig. 2(a) shows the distribution of failures for the baseline label Pers i an cat. We observe that the most frequent confusion, on images generated using the baseline caption "a realistic photograph of a Persian cat (domestic animal)." is with lynx. This mistake arises about 0.1% of the time and constitutes 87.3% of all failures. In comparison, the confusion with snow leopard is rather infrequent and arises only 0,00022% of the time. Our approach automatically discovers that adding "the background green." " to the caption results in large increase in failure rates. Failures are 5.72 more likely and the model is 14.3x more likely to predict snow leopard. We generally observe that mistakes with wild animals become more prevalent when the cat outdoors. Similarly, Fig. 2(b) and Fig. 2(c) show failures on images of flies and crayfish, respectively. Flies on flowers are significantly more likely to be confused for bees when they are on flowers (497x), while crayfish in nets are more frequently confused as chainlink fences (3721 x), honeycomb, window screens or spider webs. These highlight two shortcomings of the underlying classifier: (i) the over-reliance spurious cues (such as the flower); (ii) the inability to determine which object is the main subject of a photograph (e.g., which of the net or crayfish is important).
Generalizability of failure descriptions. To verify that the discovered failures are not specific to the text-to-image model used in this manuscript and do not result from artifacts in the image generation process, we generate 30 images using the baseline and discovered captions with DALL-E and STABLE-DIIFUSSON (samples are shown in Fig. 13 and Fig. 14 in the appendix). We evaluate
As a point of comparison, we can also evaluate the baseline failure rates on images from the IMAGENET test set. For Persian cat, fly and crayfish the baseline failure rates are 16%, 8% and 18% respectively (the target failure rate is 0% for all labels). These failure rates are higher than on images generated by the generative model, this is perhaps indicating that the generative model produces images that are more canonical and conservative. We also point out that we generate 9. IM samples to compute the first row of the table, 625K for the second and 1.4M, 2.8K, 4.3M, 1.2K for the subsequent rows.

---

Under review as a conference paper at ICLR 2023
mushroom
lunter
bee
stup dispensur
robin
oigarma
vullure
Bagpole
ICLR
ICLR
ICLR
ICLR
Figure 3: Examples of failures automatically found in IN-G-RN. The correct label is to the left in green. The incorrect prediction is the right in red. Images not watermarked upon classification.
the failure rates for the fly and crayfish labels (which exhibited higher failure rates). With DALL-E 2, for the 30 images generated with the prompt "a realistic photogruph of a fly (insect).", 18 are correctly classified as flies and none are classified as bees. When adding "it is on a flower." to the prompt, the overall failure rate increases (only images are correctly classified) and nine images now classified bees. Similarly, for "a realistic photograph of crayfish (crustacean)., 29 images are correctly classified as either crayfish, spiny lobster, American lobster, Dungeness crab or king crab, while none are classified as chainlink fence. When fence (with chainlink fence
adding "it is in a net.", four are classified as chainlimg appearing ten times in the top-3 predictions), while only 21 images are correctly classified. Results MBLE-DIFFUSSON images." Overall, we observe that discovered failures generalize across generative models. Finally, we also leverage Google Image Search 10 to find 30 images for each of the following queries: "fly". "fly on flower", "crayfish", "crayfish in ner" (images must have a resolution of at least 256x256 and should contain the true label). We classify all images and observe that the number of failures towards bee increases from zero to two and those towards cha inniinie ink fence increase from zero four. This illustrates again that discovered failures are general and extend to real photographs.
are similar for
4.2 ADVERSARIAL DATASET GENERATION AT SCALE
We demonstrate how to apply our automated pipeline to generate large datasets of failures. We seed our search by captioning images from IMAGENET-A. We show that the discovered failures generalize across initializations of a given model architecture and between models of different architectures.
Generating large-scale datasets. We assume that we have access to set of captions that describe potential failure cases." These captions are automatically extracted from the 7,500 images of IMAGENET-A using the captioning model, limiting its output to two sentences maximum. For each caption, we sample up to a thousand images keeping those leading misclassifications and limit the number of misclassified images kept per caption. We consider two models: a NESNET-50 (abbreviated hereafter by RN) and a Vision Transformer its B/16 variant (Dosovitskiy et al., 2020), abbreviated by VIT. Both models are trained solely IMAGENET and achieve 76% and 80% top-1 accuracy, respectively. This yields two separate datasets of failures which we refer to as IN-G-RN and IN-G- VIT that of size 12,332 and 9,536 respectively.
Visualizations of failure cases. Samples from IN- IN-GRR are shown in Fig. 3, with true and predicted labels. We can see that while the images clearly show the correct class, the model erroneously predicts different one. Additional samples for both IN-G-RN and IN-G-VIT are visible in Fig. 15 and Fig. 16 (in the appendix). Appendix C provides an analysis for some failure cases found in IN- and details how images from the IMAGENET training set may have misled the classifier.
Generalizability of failure cases. To investigate whether the discovered failures generalize across initializations, we train an additional RESNET (RN2) and VIT (Vrr2) on IMAGENET with the exact
For STABLE-DIFFUSINN the number of failures increases from one to two and from four to 16, for the fly and crayfish labels, respectively.
10 httpps: //1 mages. googl ee com
'This assumption is not necessary. However, it accelerates our search by generating images that are more likely to induce failures.
7

---

Under review as a conference paper at ICLR 2023
IN-G-RN2
39%
53%
46%
63%
69%
71%
100%
78%
59%
54%
IN-
50%
61%
55%
76%
79%
100%
77%
Dataset
FID
KID 4
ImageNet
5%
8%
9%
13%
14%
12%
11%
10%
6%
6%
IMAGENET-A
56.6
0.0460
IN-G-RN
48.3
0.0305
VIT-8/16
VIT-8/32
BIT-M (10143)
BiT-S (bod))
Inception V/
Res V/
RN1
RN2
vm
VIT2
IN-G-VIT
53.9
0.0330
IMAGENET (train)
2.3
0.0003
Figure 4: Failure rates (top-3) for different models on two generated datasets and IMAGENET. We report the failure rates of different models trained on IMAGENET.
Table 2: FID and KID scores. We report FID and KID scores in relation to IMAGENET (test).
same setup as our two original models but different random seeds. We also consider large set of additional models trained on IMAGENET and optionally pre-train pre-trained on larger datasets obtained from TE-HUB. Fig. 4 shows the failure rates induced by both datasets on models (failures are accounted when the top-3 predictions do not include the true label). First, we observe that failures transfer well between models of the same architecture. Indeed, 78% of the failures in IN-G-RN transfer to RN2, while the ones in IN-G-VIT transfer with 77% chance to VIT2. Second, we observe that failures for a given model architecture transfer to a large extent across architectures. Even when large-scale pretraining is used (with the BIT-M (101x1), VIT-B/16 and VrT-B/32 models pretrained on IMAGENET2IK) K), failures transfer at a rate of 39-53% for IN-G-RN and 50-61% % for IN-G- VIT. Further results for additional models are in Sec. B.2 in the appendix.
Distribution shift. Finally, we compare our generated datasets to IMAGENET and IMAGENET-A. The aim is to validate that we generate images that are similar to those in IMAGENET. We compute the Fréchet Inception Distance (FID; Heusel et al., 2017) and the Kernel Inception Distance (KID; Binkowski et al., 2018) between the generated images and the IMAGENET test set. Table also shows the FID and KID of the IMAGENET train set and IMAGENET-A. We find that our generated images are more similar to those from IMAGENET under both metrics than those from IMAGENET-A.
5 DISCUSSION AND LIMITATIONS
The motivation behind our work is to develop proof-of-concept demonstrating that today's large- scale text-to-image and image-to-text models can be leveraged to find numan-interpretatie failures in vision models. While we focus exclusively on IMAGENET, there are encouraging signs that these generative models could be used to probe models trained on specialized tasks such as medical imaging (Kather et al., 2022). Overall, there remain number of key challenges to address.
Coverage. While our approach can successfully be used to demonstrate the presence of failures, important understand that (just like scraping the web) it cannot prove their absence. In other words, there is no guarantee that it will discover all failures of a given model. 12 Moreover, the generative model is only an approximation of the distribution of interest and may lack coverage. For example, it might almost never generate "a lawnmower falling down from the sky" (an actual image from the IMAGENET training set; Jain et al., 2022b) when prompted with "a realistic photograph of lawnmower". While this can help ground failures to scenes that are likely to occur in the real-world, also means that rare failures are unlikely to be discovered (see Fig. 5(a)).
Bias. While we take the view here that off-the-shelf harge-scale generative models are trained on diverse and unbiased data, the reality is far from it: these models mirror the distribution of images and captions seen on the web. The generative model may over-sample particular regions of the image manifold and, as a result, our approach is more likely to discover failures in these high-density regions and miss failures pertaining to other regions (see Fig. 5(b)). Possible solutions to reduce bias include clever prompting (which introduces expert knowledge) or discovering failure prompts more actively by avoiding random sampling (c.g., through adversarial techniques).
Howevrr, if the generative model matches the true distribution of images, is possible provide meaningful prohabilistic guarantees.
8

---

Under review as a conference paper at ICLR 2023
ICLR
ICLR
ICLR
ICLR
ICLR
(a) Coverage (b) Bias
(c) Captioning issues (d) Generation issues (e) OOD sampling
Figure 5: Illustrative examples of various challenges. (a) Persian cats in snow (generated using "a realistic photograph of a Persian cat (domestic animal). it is walking in the snow:") are misclassified as snow lcopards at rate of 0.016%, which is significantly higher than the failure rate of 0.0032% induced by the automatically the background green. "): the total failure rate also increases twelve-fold to 8.15% (from 0.64%). (b) is estimated that only 10,000 crayfish turn blue. However, 9% of the images generated using "a realistic photograph of a crayfish (crustacean)." contain a blue crayfish (estimated by manually looking 100 samples). (c) This image of crayfish is misclassified as chainlink fence. The output of the captioning model for this particular image "a realistic photograph of crayfish. the crayfish is very detailed. the crayfish facing the camera. the crayfish is orange. it has two antennae. While caption describes the image, does not provide enough detail to reconstruct the image. (d) This image is generated from the caption "a realistic photograph of saltshaker (container). there is lemon slice on the side of the salt shaker." While the image contains lemon, the true class y photograph of a ladybug (insect). it a plastic bag.", ", this image illustrates that rext-to-image models can create image that are not from the intended distribution (i.e., of realistic photographs).
found caption ("
is not visible. (c) Generated with the caption "a realistic
Captioning issues. Using captions as latent representation allows approach produce human-inten intererattrree explanations and constrains our search to failures that can be explained in words. Not only is it possible for the captioning model miss important details or produce ungrounded captions, but some failures may simply be hard to describe (even by a human). As a result, newly generated images may look different from the set of images that induced the original failure. We note that efficiently enforcing consistency between the generated and original images (through a common caption) is an open problem since we would like to search over reasonable captions that are likely to produce images corresponding to the original failure. Fig. 5(c) shows an example. The figure shows a cray fish misclassified as a chainlink fence. While the reason for that failure is immediately obvious to us, it remains difficult to describe with succinct caption.
Image generation issues. While the text-to-image model may make occasional mistakes (such as generating the wrong object for unambiguous prompts). subtle errors can also arise from the interplay between the model and its prompt. The prompt may be ambiguous, such using words that have multiple meanings (e.g., a "walking stick" can be both cane or an insect), or may describe multiple objects with complex relationships that exacerbate mistakes (see Fig. 5(d)).
Out-of-distribution sampling. Ensuring that images sampled from an off-the-shelf generative model are part of the intended distribution (e.g., resembling IMAGENET) is difficult. We start our prompts with "a realistic photograph" in a bid to help steer the approximated distribution p(x y, x) away from artistic drawings and closer the true distribution p(x(y, 2). This approach is effective, but not always successful (see Fig. 5(e)). In some cases, finding a suitable prompt is not obvious (e.g.. to output images from a particular medical domain; Kather et al., 2022) and fine-tuning models on the dataset of interest may be necessary.
generating large amounts of data, it is important to consider the associated privacy risks. While these risks can be mitigated by using generative models trained on public, non-sensitive data, more research on private generative modelling is necessary (Harder et al., 2022).
Privacy. As
Despite these challenges, we foresee that large-scale generative models will increasingly be used as debugging tools. In this work, we introduced automated pipeline that discovers failure cases in vision models. It constitutes a prroof-of-concye that such a system allows for large-scale investigations of vision models in open-ended manner.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully debugging model mistakes using conceptual counterfactual explanations. In International Conference on Machine Learning. PMLR, 2022. 3
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. arXiv preprint UrXiv:2204.141900 2022. 1
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: met::::0000099900 2019.
Shumeet Baluja and lan Fischer. Adversarial transformation networks: Learning to generate adver- sarial examples. arXiv preprint arXiv:1703.09388 2017. 2
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Sındic, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion Attacks against Machine Learning at Test Time. arXiv preprint (oXiv:/708.06631, 2013. 25
Mikolaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representations, 2018.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp. Prasoon Goyal, Lawrence Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. arXiv preprint 1 2016.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li. Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie. Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitrion, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0723 2021.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. Conference on Fairness, Accountability and Transparency, 2018.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the ACM Workshop on Artificial Intelligence and Security 2017a. 2
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEFE Symposium Security and Privacy, 2017b. 2
10

---

Under review as a conference paper at ICLR 2023
Stephen Casper, Max Nadeau, Dylan Bodfield-Menelle and Gabriel Kreiman. Robust feature-level adversaries are interpretability tools. arXiv preprint arXiv:2110.0360,, 2022. 3
Myung Jin Choi, Antonio Torralba, and Alan S Willsky. Context models and out-of-contex( objects.
Patter Recognition Letters, 2012. 2
Mircea Cimpoi. Subhransu Maji. lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. arXiv preprint arXiv:1311.3618, 2013.
Jeffrey De Fauw, Joseph R Ledsam, Bernardino Romera-Paredes, Stanislav Nikolov, Nenad Tomasev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O' Donoghue, Daniel Visentin, George van den Driessche, Balaji Lakshminarayanan, Clemens Meyer, Faith Mackinder, Simon Bouton, Kareem Ayoub, Reena Chopra, Dominic King, Alan Karthikesalingam, Cían O Hughes, Rosalind Raine, Julian Hughes, Dawn A Sim, Catherine Egan, Adnan Tufail, Hugh Montgomery, Demis Hassabis, Geraint Rees, Trevor Back, Peng T Khaw, Mustafa Suleyman, Julien Cornebise, Pearse Keane, and Olaf Ronneberger. Clinically applicable deep learning for diagnosis and referral in retinal disease. In Nature Medicine, 2018. 1
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2009. 2
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. arXiv preprint arXiv:2010. 1929, 2020. 7, 18
Sabri Eyuboglu, Maya Varma, Khaled Saab, Jean-Benoii Delbrouck, Christopher Lee- Lee-Messer, Jared Dunnmon, James Zou, and Christopher Ré. Domino: Discovering systematic errors with cross- modal embeddings. arXiv preprint arXiv:2203. 14960, 2022.
Sahaj Garg, Vincent Perot, Nicole Limtiaco, Ankur Taly, Ed H. Chi, and Alex Beutel, Counterfactual fairness text classification through robustness. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2019.
Yunhao Ge, Jiashu Xu, Brian Nlong Zhao, Laurent Itti, and Vibhav Vineet. Dall-e for detection: Language-driven context image synthesis for object detection. arXiv preprint arXiv:2206.09550 095922, 2022. 2
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2018. 2
Robert Geirhos,
Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. In Nature Machine Intelligence, 2020a. 1,2
Robert Geirhos, Kristof Meding, and Felix A. Wichmann. Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. In Advances in Neural Information Processing Systems. 2020b. 2
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. Advances in Neural Information Processing Systems, 2021. 1. 19
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representationss 2022.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 2014. 2

---

Under review as a conference paper at ICLR 2023
Sven Gowal, Chongli Qin, Po-Sen Huang, Taylan Cemgil, Krishnamurthy Dvijotham, Timothy Mann, and Pushmeet Kohli. Achieving Robustness in the Wild via Adversarial Mixing with Disentangled Representations. arXiv preprint arXiv: 2019. 2
Fredrik Harder, Milad Jalali Asadabadi, Danica J. Sutherland, and Mijung Park. Differentially private data generation needs better features. arXiv preprint htt::2205./2900 2022. 9
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016. 5, 18
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In International Conference on Learning Representations, 2018.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint arXiv:1907.07174. 2019. 1,2
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The Many Faces of Robustness: A Critical Analysis of preprint arXiv:2006. 16241, 2020. 1, 2,
Generalization. arXiv
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems. 2017. 8
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, and others. Deep neural networks for acoustic modeling speech recognition: The shared views of four research groups. IEEE Signal processing magazine, (6), 2012.
Xiaowei Hu, Zhe Gan, Jianfeng Wang. Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision- lasion-anguuae pre-training for image captioning. arXiv preprint arXiv:2111.12233, 2021.
Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as directions in latent space. arXiv preprint arXiv:2206. 14754, 2022a. 3
Saachi Jain, Hadi Salman, Alaa Khaddaj, Eric Wong, Sung Min Park, and Aleksander Madry. A data-based perspective on transfer learning, arXiv preprint arXiv:2207.05779, 2022b. 8
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. arXiv preprint arXiv: 1XX7:1777.07328 2017. 2
Jakob Nikolas Kather, Narmin Ghaffari Laleh, Sebastian Foersch, and Daniel Truhn. Medical domain knowledge in domain-agnostic generative ai. In npj Digital Medicine, 2022. 3, 8,9 9
Aditya Khosla, Tinghui Zhou, Tomasz Malisiewicz, Alexei A Efros, and Antonio Torralba. Undoing the damage of dataset bias. In European Conference on Computer Vision, 2012.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision, 2020. 18
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, 2012.
Andrey Kuehlkamp, Benedict Becker, and Kevin Bowyer. Gender-from. or gender-from from-maccara?? In IEEE Winter Conference on Applications of Computer Vision, 2017.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv: 1607.0255 2016. 2
Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual Adversarial Robustness: Defense Against Unseen Threat Models. arXiv preprint arXiv:2006. /2655, 2020.
12

---

Under review as a conference paper at ICLR 2023
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. In Nature Communicationss 2019. 2
Horia Mania, John Miller, Ludwig Schmidt, Moritz Hardt, and Benjamin Recht. Model similarity mitigates test set overuse. In Advances in Neural Information Processing Systems, 2019,
Henry B Mann and Donald R Whitney. On test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pp. 50-60, 1947. 18
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995. 3
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint arXiv:2202.0322 2022. 2.
Haonan Qiu, Chaowei Xiao, Lei Yang, Xinchen Yan, Honglak Lee, and Bo Li. SemanticAdv: Generating Adversarial Examples via Atrributecconditional Image Editing. arXiv preprini arXiv:1966.07927, 2019. 2
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. OpenA/ Blog, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/15, 2022. 1
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet Classifiers
Generalize to ImageNet? arXiv preprint 1902. 2019. 1,2
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. arXiv preprint arXiv: 1602.0490, 2016. URL httph:tt///aate //axiv.
org/pdf/1600 34938.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2018.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. 2
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, 2022. 1
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint 19X1::9111.881
2020. 2
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho. David Fleet, and Mohammad Norouzi. Photorealistic rex--to-mmage diffusion models with deep language understanding, arXiv preprint arXivv2205.1 1447, 2022.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. arXiv preprint arXiv:1610.02391, 2019. 3
Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples with generative models. In Advances in Neural Information Processing Systems, 2018.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. Transactions Machine Learning Research, 2022. 18
13

---

Under review as a conference paper at ICLR 2023
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv:1312.6199 2013. 2,
Christian Szegedy, Wei Liu, Yangqing Jía, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2015. 18
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander Alemi. Inception-ve v4, inception- resnet and the impact of residual connections on learning. In Thirty-first AAAI conference artificial intelligence, 2017. 18
Antonio Torralba, Alexei A Efros, and others. Unbiased look at dataset bias. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2011. 1,2
Sam Witteveen and Martin Andrews. Paraphrasing with large language models. arXiv preprint arXiv:1911.0966. 2019. 5
Eric Wong and Zico Kolter. Learning perturbation sets for robust machine learning. In International Conference on Learning Representations, 2021. 3
Chaowei Xiao, Bo Li, Jun Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. Generating adversarial examples with adversarial networks. arXiv preprint arXiv:1801.026/00 2018. 2
Kai Xiao, Logan Engstrom, Andrew Iyas, and Aleksander Madry. Noise or Signal: The Role of Image Backgrounds in Object Recognition, arXiv preprint urXiv:2006.09994 2020. 1,2
14


Reviews:
[{"review_id": "ivUauQCGMs", "reviewer": "Reviewer_GhTH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes utilizing off-the-shelf text-to-image and image-to-text models to discover vision models' failures automatically. The experimental results are delighting and convincing to some extent. This paper is also inspiring and potentially useful to improve the generalizability of vision models.", "summary_of_the_review": "Currently, I lean to positive for its novel idea.", "strength_and_weaknesses": "Overall, the idea of exploiting off-the-shelf generative and captioning models to discover failures of image classifiers trained in ImageNet is interesting and novel. I am also impressed by the extensive experimental results (including visualization analysis and variants effect analysis). \n\nNevertheless, I have several small issues:\n- Although adequate experiments are conducted on ResNet-50 on ImageNet, I am curious to see experiments on a stronger network backbone such as ResNet-152. This lies in the concern that a stronger network may not be easily found failures using the proposed method. As a matter of fact, I’m mostly convinced by the provided results. But I think this experiments will make the conclusions more solid.\n- Do the discovered failures can further improve the object classification model’s performance? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RtCyhf_bnL9", "reviewer": "Reviewer_v5Fg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper contributes a method to automatically find failure cases of image classifiers in an  oepn-ended manner by leverageing off-the-shelf, large-scale generative models. ", "summary_of_the_review": "This paper proposes a method to automatically find failure cases of image classifiers. The applying of generative models is novel, but the limitations mentioned in the paper are difficult to solve and will affect the effectiveness of the work.", "strength_and_weaknesses": "Strength: Automatically discovering failures is an interesting problem; The idea of applying image-to-text generative models to discover failures and obtain human-interpretable explanation of failure is novel.\n\nWeaknesses:\nThe failure cases in this paper are all synthetic and there is still a large margin gap remaining between real-world data and data generated by generative models; Though a lot of well-trained generative models can be applied in the method, the costs of training them should not be ignored; The limitations mentioned in the paper are difficult to solve and will affect the effectiveness of the work. Due to the randomness in the process of selecting cases via generative models, it is difficult to prove that this framework can be reproduced and effective.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Mk5FrblIlh", "reviewer": "Reviewer_B78A", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes a pipeline that can be used to identify failures of image classifiers. Proposed approach utilizes SOTA generative models to enable interpretable failure discovery of a model by providing targeted realistic images. Experiments are well conducted to demonstrate the benefit of the approach. Discussions on its limitations and potential concerns are well considered in the paper as well.", "summary_of_the_review": "This paper clearly describes their proposed method which is well designed and backed up with proper analysis and experiments. Although I have some doubts of the claimed scalability and how practical it would be if we were to use them for model improvement, as a proof-of-concept I think it falls above the acceptance bar.", "strength_and_weaknesses": "Strength\n\n* Formulated the problem of finding failures of image classifier in a probabilistic framework and showed how one can use off-the-shelf image-to-text captioning model and text-to-image generative model to create an interpretable and easy-to-use debugging tool.\n* Proposed approach is more interpretable and naturalistic than existing approaches and shows promise in generalizability.\n\n\nWeakness / other comments\n\n* “In particular, we use three shots, with each shot being an image and caption pair. Images are publicly available online and are manually described using between four to seven short sentences that describe the subject of the photograph and its physical position with respect to the camera, as well as the background or context in which the subject appears” → This indicates that the method requires human input with some expert knowledge which raises a question on scalability and potential bias for this process.\n* How can identified failures help improving the actual model? Further discussion / analysis on how the discovered failures could be used would be useful in order to give a better idea to the practitioners who would want to use this system to improve their model.\n* “For each label y, we manually select target labels  ŷ (snow leopard, bee and chainlink fence respectively)” → Why was this set of ŷ selected? What is the impact of the choice ŷ on failure discovery?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ARSMNtw8bP_", "reviewer": "Reviewer_38rC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach to automatically discover the failure cases of vision models under real-world settings. Off-the-shelf image-to-text and text-to-image models are leveraged to find such failures. Firstly, a conditional text-to-image synthesis model generates synthetic data based on the ground-truth label. Then, a captioning model is used to describe misclassified inputs. Next, the descriptions generated by the captioning model are used to synthesize more images to test whether the specific description causes a higher failure rate. This pipeline can be used to find the failure modes with language explanations for the classification models.", "summary_of_the_review": "My major concern is the robustness of the approach. If the text-to-image models and the image captioning models make mistakes, the results will be misleading. This approach assumes that the text-to-image and image captioning models are perfect, or at least the error rate is much smaller than the image classification model. But I suspect this assumption is not always true. ", "strength_and_weaknesses": "Strengths:\n1. Incorporating large-scale text-to-image and image-to-text models for diagnosing classification models is an interesting topic. The authors provide some insightful explorations.\n2. The proposed method can be used to diagnose image classification models and provide language explanations for the failure modes.\n\nWeaknesses:\nThe major limitation is that the proposed approach assumes that the text-to-image and image-to-text models are perfect. However, if error happens in text-to-image synthesis or image captioning synthesis, then the results will not be reliable. For example, what if the text-to-image synthesis model generates an image of a bee with the caption \"a realistic photograph of a fly\"? The authors only discussed this issue in one paragraph in limitations, but I think this is an important issue that should be investigated deeper. How frequently do the text-to-image synthesis and image captioning models make mistakes, and how these mistakes will affect the results? How robust is the proposed approach?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject