Distilling Model Failures as Directions in Latent Space
Authors: Saachi Jain
Hannah Lawrence
Ankur Moitra
Aleksander Madry
Venue: ICLR 2023 notable top 25% (2023)
Keywords: datasets
biases
subpopulations

Abstract: Existing methods for isolating hard subpopulations and spurious correlations in datasets often require human intervention. This can make these methods labor-intensive and dataset-specific. To address these shortcomings, we present a scalable method for automatically distilling a model's failure modes. Specifically, we harness linear classifiers to identify consistent error patterns, and, in turn, induce a natural representation of these failure modes as directions within the feature space. We demonstrate that this framework allows us to discover and automatically caption challenging subpopulations within the training dataset. Moreover, by combining our framework with off-the-shelf diffusion models, we can generate images that are especially challenging for the analyzed model, and thus can be used to perform synthetic data augmentation that helps remedy the model's failure modes.

Published as a conference paper at ICLR 2023
DISTILLING MODEL FAILURES AS DIRECTIONS IN LATENT SPACE
Saachi Jain Hannah Lawrence*, Ankur Moitra & Aleksander Madry Massachusetts Institute of Technology. Cambridge, USA (saachi 3, hanl law moi tra,, madry) @mit. edu, (*Equal contribution)
ABSTRACT
Existing methods for isolating hard subpopulations and spurious correlations datasets often require human intervention. This make these methods labor- intensive and dataset-specific, To address these shortcomings, we present a scalable method for automatically distilling model's failure modes. Specifically, we harness linear classifiers to identify consistent eπrоr patterns, and, in turn, induce a natural representation of these failure modes directions within the feature space. We demonstrate that this framework allows us to discover and automatically caption challenging subpopulations within the training dataset. Moreover, by combining our framework with off-the-shelf diffusion models, we can generate images that are especially challenging for the analyzed model, and thus can be used perform synthetic data augmentation that helps remedy the model's 's failure modes.
INTRODUCTION
The composition of the training dataset has key implications for machine learning models' behavior (Feldman, 2019; Carlini et al., 2019; Koh & Liang, 2017; Ghorbani & Zou, 2019; Ilyas et al., 2022), especially the training environments often deviate from deployment conditions (Rabanser et al., 2019; Koh et al., 2020; Hendrycks et al., 2020). For example, a model might struggle on specific subpopulations in the data if that subpopulation was mislabeled (Northcutt et al., 2021; Stock & Cisse, 2018; Beyer et al., 2020; Vasudevan et al., 2022), underrepresented (Sagawa et al., 2020; Santurkar al., 2021), corrupted (Hendrycks & Dietterich, 2019; Hendrycks et al., 2020). More broadly, the training dataset might contain spurious correlations, encouraging the model to depend on prediction rules that do not generalize to deployment (Xiao et al., 2020; Geirhos et al., 2020; DeGrave et al., 2021). Moreover, identifying meaningful subpopulations within data allows for dataset refinement (such as filtering or relabeling) (Yang et al., 2019; Stock Cisse, 2018), and training more fair (Kim et al., 2019; Du et al., 2021) accurate (Jabbour et al., 2020; Srivastava al., 2020) models.
However, dominant approaches to such identification of biases and difficult subpopulations within datasets often require human intervention, which is typically labor intensive and thus not conducive to routine usage. For example, recent works (Tsipras et al., 2020; Vasudevan et al., 2022) need to resort to manual data exploration to identify label idiosyncrasies and failure modes in widely used datasets such as ImageNet. On the other hand, a different line of work (Sohoni et al., 2020; Nam et al., 2020; Kim et al., 2019; Liu al., 2021; Hashimoto et al., 2018) does present automatic methods for identifying and intervening on hard examples, but these methods are not designed to capture simple, human-urm human-understandde understandable patterns. For instance, Liu al. (2021) directly upweights inputs that were misclassified early in training, but these examples do not necessarily represent consistent failure mode. This motivates the question:
How
extract meaningful patterns of model errors large datasets?
One way to approach this question is through model interpretability methods. These include saliency maps (Adebayo et al., 2018 8; Simonyan et al., 2013), integrated gradients (Sundararajan et al., 2017), and LIME (Ribeiro et al., 2016b), and perform feature attribution for particular inputs. Specifically, they aim to highlight which parts of an input were most important for making a model prediction. and can thus hint at brittleness of that prediction. However, while feature attribution indeed help debug individual test examples, it does not provide a global understanding of the underlying biases of the dataset at least without manually examining many such individual attributions.

---

Published as a conference paper at ICLR 2023
OUR CONTRIBUTIONS
In this work, we build a scalable mechanism for globally understanding large datasets from the perspective of the model's prediction rules. Specifically, our goal is not only identify interpretable failure modes within the data, but also to inform actionable interventions to remedy these problems.
Our approach distills given model's failure modes as directions in certain feature space. In particular, we train linear classifiers on normalized feature embeddings within that space to identify consistent mistakes in the original model's predictions. The decision boundary of each such classifier then defines a "direction" of hard examples. By measuring each data-pon toiatp'is'ss ss alignment with this identified direction, we can understand how relevant that example is for the failure mode we intend to capture. We leverage this framework to:
Delection: Automatically detect and quantify reliability failures, such as brittleness to distribution shifts or performance degradation hard subpopulations (Section 2.1).
Interpretation: Understand and automatically assign meaningful captions to the error patterns identified by our method (Section 2.2).
Intervention: Intervene during training in order to improve model reliability along the identified axes of failure (Section 2.3). In particular, by leveraging our framework in conjunction with off-the-shelf diffusion models, we can perform synthetic data augmentation tailored to improve the analyzed model's 's mistakes.
Using our framework, datasets such as CIFAR-1O, ImageNet, and ChestX-ray14 14. Importantly, we do not require direct human intervention or pre-annotated subgroups. The resulting framework is thus scalable approach to identifying important subpopulations large datasets with respect to their downstream tasks.
automatically identify and intervene on hard subpopulations in image
2 CAPTURING FAILURE MODES AS DIRECTIONS WITHIN A LATENT SPACE
The presence of certain undesirable patterns, such as spurious correlations or underrepresented subpopulations, in training dataset can prevent learned model from properly generalizing during deployment. As a running example, consider the task of distinguishing "old" versus "young" faces, wherein the training dataset age is spuriously correlated with gender (such that the faces of older men and younger women are overrepresented). Such correlations occur in the CelebA dataset (Liu et al., 2015) (though here we construct a dataset that strengthens them)'. Thus, a model trained on such a dataset might rely too heavily on gender, and will struggle to predict the age of younger men or older women. How can we detect model failures on these subpopulations?
The guiding principle of our framework is to model such failure modes as directions within a certain latent space (Figure 1). In the above example, we would like to identify an axis such that the (easier) examples of "old men" and the (harder) examples of "old women" lie in opposite directions. We then can capture the role of an individual data point in the dataset by evaluating how closely its normalized embedding aligns with that extracted direction (axis). But how can we learn these directions?
Our method. The key idea of our approach is to find a hyperplane that best separates the correct examples from incorrect ones. In the presence of global failure modes such as spurious correlations, the original model will likely make consistent mistakes, and these mistakes will share features. Using held out validation set, we can therefore train linear support vector machine (SVM) for each class to predict the original model's mistakes based on these shared features. The SVM then establishes a decision boundary between the correct and incorrect examples, and the direction of the failure mode will be orthogonal to this decision boundary (i.c., the normal vector to the hyperplane). Intuitively, the more aligned an example is with the identified failure direction, the harder the example was for the original neural network. Details of our method can be found in Appendix A.
The choice of latent space: Leveraging shared vision/language embeddings. Naturally, the choice of embedding space for the SVM greatly impacts the types of failure modes it picks up. Which embedding space should we choose? One option is to use the latent space of the original neural network. However, especially if the model fits the training data perfectly, it has likely learned latent
We can also detect this failure mode in the original CelebA dataset (See Appendix B.1)
2

---

Published as a conference paper at ICLR 2023
Vision/Language Space
"e platio of an old verun wieh glazses"
Hard Exemplars
pboto
Incoreett
Base Model
bale"
Regative young woman: facc
caption:
photo of man who baz a bassd"
Correct
Easy Exemplars
ploco of an old mac. prtth grey haiv"
Most Positive caption: photo of old man who had no beard"
Per-olass Which "old" instances are predicted correctly by the base model?
Figure I: A summary of our approach, as applied to CelebA. Consider the task of classifying faces as "young" or "old". where the training set contains spurious correlation with gender. (Leftt) Given trained base model, we evaluate it on a held-out validation set. (Middle) For each class (here "old"), train linear SVM on shared vision/language latent space predict whether the base model would have classified an input from the validation set correctly. We then extract a direction (gray) for the captured failure mode the vector orthogonal to the learned hyperplane. Here, the SVM learns to use gender to separate the incorrect (red) vs. correct (green) examples. (Right) Images farthest from the SVM decision boundary exemplify the hardest ("old women") ") or the easiest ("old men") test examples. Furthermore, we can select captions that, when embedded in the shared latent space, are farthest from the decision boundary and thus capture the pattern of errors learned by the SVM.
representations which overfit to the training labels. our running example of CelebA, the few older women in the training set could be memorized, and so their latent embeddings would likely be very different from those of older women in the test set. As shown in Appendix B. B.1.6, these discontinuities reduce the efficacy of our method, the resulting featurizations are likely inconsistent.
To address this problem, we use an embedding that is agnostic to the specific dataset. In particular, we featurize our images using CLIP (Radford al., 2021), which embeds both images and language into a shared latent space of unit vectors. Using these embeddings will also let us automatically assign captions to the directions extracted from the SVM. We consider other latent spaces in Appendix B.
Having captured the model's failure modes as directions within that latent space, we can detect, interpret, and intervene on difficult subpopulations. We now describe implementing these primitives.
2.1 DETECTION
How can we tell whether the extracted direction actually encapsulates prevalent error pattern? To answer this question, we need a way to quantify the strength of the identified failure mode. Our framework provides natural metrie: the validation error of the trained SVMs. The more consistent the failure mode is, the more easily a simple linear classifier can separate the errors in the CLIP embedding space. Thus, the cross-validation as measure of the failure mode's s strength in the dataset. It can then be used to detect classes that have clear bias present, which will be useful for the following interpretation and intervention steps.
(i.e, the SVM's accuracy held-out data) serves
2.2 INTERPRETATION
Since the trained SVMs are constrained to capture simple (approximately linearly separable) patterns in embedding space, we can easily interpret the extracted directions to understand the failure modes of the dataset. We explore two approaches to extracting the subpopulations captured by framework.
Most aligned examples. The examples whose normalized embeddings are most aligned with the extracted direction represent the most prototypically correct incorrect inputs. Therefore, to get sense of what failure mode the direction is capturing. we can examine the most extreme examples according the SVM's decision value, which is proportional to the signed distance to the SVM's
3

---

Published as a conference paper at ICLR 2023
decision boundary. Returning to our running example, in Figure the images of the "old" class that are the "most correct" correspond to men, while the "most incorrect" ones correspond to women.
Automatic captioning. We can leverage the fact that the SVM is trained on shared vi- sion/language latent space (i.e., CLIP) to automatically assign captions to the captured failure mode. Just above we surfaced the most aligned images, can similarly surface the most aligned captions, i.e, captions whose normalized embedding best matches the extracted direction.
Specifically, assume that each class has reference caption which is generie phrase that could describe all examples of the class (e.g., "a photo of a person"). We then generate a candidate set of more specific captions c1,--. Cm that include additional attributes (e.g.. "a photo of person with ---
moustache"). Our goal is to pick the caption for which the additional information provided by beyond that which was already provided by r best captures the the extracted failure mode.
To do so, we score a caption c by how aligned the normalized embedding è c-r is with the Te--lla
direction captured by the SVM. This amounts to choosing the captions for which è has the most positive (casiest) or negative (hardest) SVM decision values. In contrast to previous works that choose the captions closest to the mean of selected group of hard examples (c.f., Eyuboglu et al. (2022)), our method avoids this proxy and directly assigns a caption to the captured failure mode itself.
Directly decoding the SVM direction with diffusion models. Some of the recently developed diffusion models (e.g., retrieval augmented diffusion models (Blattmann et al., 2022). DALL-E 2 (Ramesh al., 2022)) generate images directly from the shared vision/language (CLIP) space. In such cases, we can directly decode the extracted SVM direction into generated images. This enables us visually capture the direction itself, without needing generate a set of candidate captions.
Specifically, let be the normalized embedding of the reference caption, and w the normalized SVM direction. By rotating between and either or -w via spherical interpolation, we can generate harder or easier images, respectively. Here, we expect the degree of rotation a to determine the extent of difficulty?. As shown in Section 4, passing this rotated embedding the diffusion model indeed lets us directly generate images that encapsulate the extracted failure mode.
2.3 INTERVENTION
Now that we have identified some of the failure modes of the model, can we improve our model's s performance on the corresponding challenging subpopulations? It turns out that this is possible, via both real and synthetic data augmentation.
Filtering intervention. Given an external pool of examples that was not used to train the original model, we can select the best of these examples to improve performance on the hard subpopulations. Specifically, if the goal is to add only K examples per class to the training dataset (e.g.. due to com- putation constraints), Appendix 1.2, we discuss an alternative intervention scheme, upweighting hard training examples.
simply add the K images with the most negative SVM decision values. In
Synthetic data augmentation. In the absence of such an external pool of such data, can leverage our framework together with text-to-image diffusion models (e.g.. Stable Diffusion (Rombach et al., 2022), DALL-E 2 (Ramesh et al., 2022), and Imagen (Saharia al., 2022)) to generate synthetic images instead. After automatically captioning each failure mode, we simply input these captions into the diffusion model to generate images from the corresponding subpopulation. In Section we show that fine-tuning the model on such images improves model reliability on these subpopulations.
3 VALIDATING OUR FRAMEWORK IN THE PRESENCE OF KNOWN SUBPOPULATIONS
In Section 2, we presented an approach for distilling the failure modes as directions within a latent space. In this section, we validate our framework by evaluating its performance on datasets with known pathologies. In Section 4, we apply our framework to discover challenging subpopulations in datasets where the bias is not known beforehand. Experimental details can be found Appendix
"Our approach mirrors the "text-diff" technique in DALLE-22 (Ramesh al., 2022), which uses spherical interpolation to transform images according textual commands.
4

---

Published as a conference paper at ICLR 2023
SVM Decision Va
SVM Decision Value
LF [NCA+20]
[NCA+ Confs. (Earfly Stop)
Fraction Female
Confs. (Ea Stap) Confidences
0.8
Confidences
Base Population
Fraction
Population
0.6
0,4
1000
2000
3000 4000
5000
5000
10000
15000
Top K Flagged
Flagged
(a) Class Old
(b) Class Young
For each CelebA class, the fraction of test images that are of the minority gender when ordering the images by either their SVM decision value or model confidences. We include additional baselines: Domino (Eyuboglu et al., 2022), LfF (Nam al., 2020), and confidences after early stopping. Our framework more reliably captures the spurious correlation than all other baselines.
Figure 2:
Old
Young
Hard Exemplars
0.70
SVM Caption:
photo has cnal faee"
beard"
0.66
Easy Exemplars
Degree Spurious Age-Gender
Figure 4: Our SVM's cross val- idation score (corresponding its estimated ability to assess the strength of the extracted failure mode) compared to the strength the planted spurious correlation. The SVM scores are highly corre- lated with the strength of the shift.
Coption: photo who has anal faze"
no beard"
Figure 3: The images and captions for each class with the most extreme SVM decision values. Those scored as most incorrect in the minority subpopulations ("old women" and "young men"), while those scored as most correct are in the majority subpopulations ("old men" and "young women").
We focus on two settings: (1) detecting spurious correlation in a facial recognition dataset, and (2) isolating underrepresented subtypes in image classification. In Appendix B, we consider other settings, such Colored MNIST (Arjovsky al., 2019) and ImageNet-C (Hendrycks & Dietterich, 2019).
3.1 USING OUR FRAMEWORK TO ISOLATE AND LABBL SPURIOUS CORRELATIONS
We revisit our running example from Section 2, where a model is trained to predict age from the CelebA dataset. The training dataset contains a spurious correlation with gender (which enhance by selectively subsetting the original dataset). We use a validation set that is balanced across age and gender, but also explore unbalanced validation sets in Appendix B. and Section 4.
Capturing the spurious correlation. Does our framework identify gender as the key failure mode in this setting? It turns out that even though only 33% of the dataset comes from the challenging sub- populations "old women" and "young men", 89% of the images flagged as incorrect by our framework from these groups. Indeed, shown in Figure 2, the SVM more consistently selects those hard sub- populations, compared to using the original confidences. As the underlying linear classifier is forced use simple prediction rules, the corresponding SVM flags a more homogenous population than does using the original confidences. We find that our method further outperforms other baselines such as Domino (Fyuboglu et al., 2022), Learning from Failure (Nam et al., 2020), or using confidences after carly stopping (as in Liu et al. (2021)) (see Appendix B. for further experimental details).
Automatically interpreting the captured failure mode. In Figure 3, we surface the examples most aligned with the extracted direction using the SVM's decision values. As shown, examples with the most negative SVM decision values are indeed from the hard subpopulations. We also automatically caption each captured failure mode. To this end, consider candidate caption set which includes attributes such as age, gender, and facial descriptors. The captions that are most aligned with the extracted direction in CL.IP space capture the spurious correlation gender (Figure 3).
5

---

Published as a conference paper at ICLR 2023
Prwntte Nbootly Subriaes
0.6
0.52
0.54
SUM Dedaion Valle
confdorne
Random
0.50
Basi Popuittion
0.4
0.45
0.3
Accuracy
1000
200
300
400
500
25
50
100
Top K.Flgges
K.Added Images
Figure 5: For each CIFAR-100 superclass, the fraction of the top K flagged test images that be- long to the underrepresented subclass (averaged over classes). We compare using the SVM's deci- sion value and the model's confidences select the top K images; the SVM more consistently identifies the minority subpopulation.
Figure 6: For CIFAR-100, the accuracy on the minority subclasses after adding K images per superclass from the extra data (reported over five runs.) We compare using the SVM's decision value or the model's confidences to select images. Relying on the SVM's values provides the most improvement on the minority subclasses.
Quantifying the strength of the shift. As mentioned in Section 2, the cross-validation scores of the SVM quantify the strength of the spurious correlation. In Figure 4, we train base models on versions of CelebA with increasing degrees of the planted spurious correlation, and find that the SVM cross-validation scores strongly correlate with the strength of the planted shift.
3.2 IDENTIFYING UNDERREPRESENTED SUB-TYPES
Above, we considered setting where the model predicts incorrectly because it relies on a spurious feature (gender). What if model struggles on a subpopulation not due to a spurious feature, but simply because there are not enough examples of that image type?
To evaluate our approach in this setting, consider the task of predicting the superclass a hierarchical dataset when subclasses have been underrepresented. In particular, the CIFAR-100 (Krizhevsky, 2009) dataset contains twenty superclasses, each of which contains five equally represented subclasses. For each such superclass, data. For instance, for the superclass "aquatic mammals", we remove a large fraction of the subclass beavers from the training dataset. In Figure 5, find that framework correctly isolates the minority subelass as the most incorrect for each superclass more consistently than does confidences.
subsample subclass that it is underrepresented in the training
Validating automatic captioning. For each superclass, construct a caption set that corresponds to the possible subclasses (e.g., for the superclass "aquatic mammals," the caption set is "a photo of a beaver", "a photo of dolphin", etc.). As we find, the caption corresponding the minority subclass was the top negative caption for 80% of the superclasses, and in the top 2 for 95% of the superclasses.
Filtering intervention. Having isolated hard subpopulations, we now apply our framework down- stream to improve performance on them. We use the trained SVMs to choose a subset of images from a larger pool of data to add to the training set. Figure 6, we find that using our framework to select this extra data improves accuracy on the minority subclasses more than what using model confidences or choosing a random subset offers, while maintaining approximately the same overall accuracy.
DISCOVERING NEW SUBPOPULATIONS IN IMAGE DATASETS
Above, considered datasets with pathologies, such spurious correlations or underrepresented subtypes. that were known. In this section, we apply our framework to datasets without any pre- annotated difficult subpopulations. Our approach discovers new subpopulations representing key failure modes for the models. Specifically, we apply our framework to CIFAR-10 (Krizhevsky, 2009) and ImageNet (Deng et al., 2009), and defer ChestX-ray (Rajpurkar et al., 2017) to Appendix C.3. Experimental details can be found in Appendix C.
4.1 CIFAR-10
We begin with the CIFAR-10 (Krizhevsky, 2009) dataset. Since the accuracy of a R on CIFAR-10 10 is very high (93%), the original model does not make many errors. Thus, we instead

---

Published as a conference paper at ICLR 2023
Cat
Dog
0.85
Hard Exemplars
0.80
0.75
Capoon: protto amdo grasa
SVM Caption:" photo dog"
0.70
Foeibre
Negadve
Easy Exemplars
Ovenll Oass
1000
200
Test Caption
SVM Caption: "a photo of cat ruide"
SVM Caption:" photo of white dog on grant"
(b) Model accuracies on each identified subpopulation.
(a) Most extreme images and extracted SVM captions
Figure 7: (a) The images with the most extreme SVM decision values for the CIFAR-IC 10 classes cat and dog, along with the most positive/negative captions according the SVM. (See Appendix C.1.3 for more examples.) (b) For each class, the accuracy of the K test images closest in CLIP embedding space the most positiveenegative SVM captions for that class (averaged over classes). The images closest to the negative caption have lower accuracy than those closest to the positive caption.
Cat
Dog
Generated Hard Examples
0.6
Lard Syntheii Images Easy Syntheze Imapes Syntheii imagai
0.4
0.2
Generated
Easy
0.4
Examples
(b) Model accuracies on the gen- erated images.
(a) Examples of hard and easy generated images.
Figure 8: To directly decode the SVM direction into images, we spherically interpolate the extracted direction and the embedding of the reference caption before passing this vector to diffusion model. (a) Examples of synthetic hard and easy images of CIFAR-10 cats and dogs. The generated images match the trends found in Figure 7a. Further examples can be found in Appendix C.1.4. (b) Base model aceuracy on generated hard and easy images (100 images per CIFAR-10 class) over varying generated using the reference caption.
degrees of spherical interpolation (a). Neutral images The base model performs worst on the hard generated images and best on the casy ones.
consider weaker base model trained with 20% of the original CIFAR-10 10 dataset, where the other 20% used for validation and the last 60% is used as extra data for the subset intervention. (See Appendix framework the larger CIFAR- 10 dataset.)
C.I for additional experiments, including applying
Finding interpretable subpopulations within the CIFAR-10 dataset. Figure 7a displays exam- ples of the failure modes identified by our framework. We identify white cats grass and black dogs as hard, while classifying brown cats and white/brown dogs that are inside as easy.
Do these directions in latent space map real failure modes of the base model? Without manual annotations, we can no longer directly report the original model's s accuracy on these minority groups However, proxy, evaluate images that closest in cosine similarity to the captions chosen by the SVM. For example, to get white cats on the grass, we rank the CIFAR-10 cats by how close their embeddings are the caption "a photo of white on grass" in CLIP space". Using this proxy, in Figure 7b we confirm that the surfaced SVM captions represent real failure modes in the dataset.
Decoding the SVM direction to generate challenging images. Since the SVM operates in CLIP space, we can directly decode the extracted direction into an image using off-the-shelf diffusion model (Blattmann al., 2022) which also samples from this space. By spherically interpolating between the embedding of the reference caption (e.g., "a photo of a cat") and the extracted SVM direction, we can generate harder or easier images corresponding to the extracted failure mode (Figure 8a). For example, interpolating with the hard direction for CIFAR-10 10 cats generates white cats on grass, matching the extracted SVM caption in Figure 7a. Across classes, we verify that the original model performs worse the "hard" generated images and better the "casy" ones (Figure 8b), In the Appendix C.1, validate that this approach surfaces images that visually match the caption.
7

---

Published as a conference paper at ICLR 2023
Subpopuation Accuracy
0.75
0.85
Subpgrvition
Adding generic synemetic images Adding hard synthetic images Adding casy syntrctic images Original Accuracy
Adding generic syrenetic images Adding hard synthetic images Adding sasy synthctic images
0.80
Driginal Accuracy
0.65
0.75
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
Top Test Images by Caption
Top K Test Images Caption
(a) Evaluating images closest the hard subpopulation (b) Evaluating images closest to the easy subpopulation
Figure 9: We fine-tune the model using 100 synthetic images per class generated via stable diffusion based on the hard, easy, reference caption. We measure the accuracy of the K test images closest to the (a) hard or (b) easy SVM caption CLIP space. tine-tuning the model on the images generated from the hard captions boosts the accuracy of the model on these corresponding. real test images more than augmenting with generic synthetic images does. The analogous phenomenon does not hold, however, when we target the easy subpopulation.
Targeted synthetic data augmentation. By leveraging text-to-image generative models, we can generate images for targeted data augmentation. Using a off-the-shelf stable diffusion (Rombach et al., 2022) model, we generate 100 images per class using the corresponding negative SVM caption (e.g.. "a photo of a white cat on the grass") ") as the prompt. After adding these images to the training set, we retrain the last layer of the original model. Fine-tuning the model on these synthetic images improves accuracy on the hard subpopulation - defined according to similarity in CLIP space to the negative caption compared using generic images generated from the reference caption (Figure 9a).
turns out that this procedure for targeted synthetic data augmentation is particularly effective for the identified challenging subpopulation. Generating images using the easy caption (e.g.. "a photo of a cat inside") does not improve accuracy on images within that subpopulation any more than augmenting with generie synthetic images (Figure 9b).
It is also possible to augment the dataset with images generated directly from the SVM direction, as in Figure 8a, using a diffusion model that samples directly from CLIP space. In doing so, we could skip the intermediate captioning stage entirely. Currently, the diffusion models that operate in CLIP space are either not open-soures (DALL-E (Ramesh et al., 2022)) or trained with less data (Blattmann et al., 2022). However, with further progress in CLIP-space diffusion models, this may be a promising mechanism for tailoring data augmentation directly from the SVM direction.
4.2 IMAGENET
We also apply our framework on larger scale: to find challenging subpopulations ImageNet (Deng et al., 2009) (see Appendix C.2 for further experimental details). In Figure 10a, we display examples of the failure modes captured by our framework with their associated SVM captions. These biases include over-reliance color (c.g., the coat of red wolf) or sensitivity to co-occurring objects (c.g the presence of person holding the fish tench). We include more examples in Appendix C.2.1.
As in Section 4. validate that these directions correspond to real failure modes by evaluating the accuracy of the test images that are closest to the positive or negative SVM caption in CLIP space (Figure 10b). The identified subpopulations indeed have significant difference in overall accuracy.
RELATED WORK
Debiasing/fairnes under known biases. There are many works on optimizing group fairness with respect to known biases (Feldman et al., 2015; Kamishima et al., 2011; Hardt et al., 2016; Zafar et al., 2017; Kusner et al., 2017; Balashankar et al., 2019); see also (Mchrabi et al., 2021; Hashimoto et al., 2018) and the references therein. Outside the fairness literature, de-biasing approaches in machine learning have been proposed for mitigating the effect of known biases during training (Sagawa et al., 2020: Yang et al., 2019; He et al., 2019; Belinkov et al., 2019; Clark al., 2019).
Discovery and mitigation of hard subpopulations. One line of work uses manual exploration to identify pathologies, such as label ambiguity, in ImageNet (Stock & Cisse, 2018 8; Vasudevan et al.,
8

---

Published as a conference paper at ICLR 2023
Tench
Red Wolf
Hard Exemplars
0.55
Accrssy
SVM Caption: "a photo dose up ftar"
5VM Caption:" photo of white cantine outside"
0.50
Feeiiiv
Easy Exemplars
Regethe
Over Closs
Caption:
photo fish
SVM Caption: phatio of blue bockground"
Test Images by Caption
(b) Model accuracies on each identified subpopulation
(a) Most extreme images and extracted SVM captions.
Figure 10: (a) The most extreme images and captions by SVM decision value for ImageNet classes tench and red wolf. Our framework identifies clear biases on color (e.g red wolf) and co-occurrence of unrelated objects (e.g tench). (b) For each class, the accuracy of the top images that were closest in CLIP space to the most positive/negative SVM captions for that class. The identified hard (negative) subpopulation has a lower accuracy than the identified easy (positive) subpopulation.
2022: Tsipras et al., 2020). Another presents automatic methods for discovering and intervening on hard subpopulations. Sohoni et al. (2020) estimates subclass structure by clustering in the model's latent space, while Liu et al. (2021) directly upweights examples that the model misclassified early in training. Other works directly train a second model which "emphasizes" the examples on which base model struggles (Nam et al., 2020; Hashimoto al., 2018; Bao & Barzilay, 2022; Kim et al., 2019; Utama et al., 2020; Sanh et al., 2020). Unlike our framework, these methods focus on the overall test accuracy, rather than finding a consistent failure mode. Perhaps the closest work in spirit to is Eyuboglu et al. (2022), which presents automatic method for identifying challenging subsets by fitting a Gaussian mixture model in CLIP space. However, our SVM provides a stronger inductive bias towards simple subelasses, and furthermore imposes a useful geometry on the latent space for scoring data points, intervening, and selecting captions in a more principled manner.
Model interpretability. Model interpretability methods seek identify the relevant features for a model's prediction. They include gradient- (Simonyan et al., 2013; Dabkowski & Gal, 2017: Sundararajan et al., 2017) and perturbation-based (Ribeiro al., 201 Goyal al., 2019; Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Zintgraf et al., 2017; Dhurandhar et al., 2018: Chang al., 2019: Hendricks et al., 2018: Singla al., 2021) methods. Other works analyze the model's activations (Bau et al., 2017; Kim et al., 2018; Ych et al., 2020; Wong et al., 2021). Li & Xu (2021) use generative model to find biased attributes that impact model predictions. Concurrent work by Abid et al. (2022) uses predefined concept activation vectors to discover latent space perturbations that correct a model's mistake. Finally, several papers investigate the vulnerability of models to specific biases such as sensitivity to texture (Geirhos et al., 2019), background (Xiao al., 2020), spatial perturbations (Engstrom et al., 2019), common corruptions (Hendrycks & Dietterich, 2019), and non-robust features (Szegedy al., 2014; Goodfellow et al., 2015; Madry et al., 2018).
CONCLUSION
In this paper, we introduce a framework for automatically isolating, interpreting, and intervening on hard but interpretable subpopulations of a dataset. In particular, by distilling the model's failure modes as directions in latent space, our framework provides mechanism for studying data-pooie in the context of the model's 's prediction rules. We thus can leverage our framework to automatically caption the captured failure mode, quantify the strength of distribution shifts, and intervene to improve the model's 's performance on minority subgroups. Notably, our method enables the effective, automated application of text-to-image models for synthetic data augmentation. We find that technique succeeds not only on datasets with planted hard subpopulations, but also on natural and widely-used datasets; such, we hope will serve useful tool for scalable dataset exploration and debiasing.
Several interesting directions remain for further study. One of them is developing sophisticated methods for caption candidate generation. Moreover, a given class may have multiple important failure modes. Therefore, extending our method to disentangle these distinct failure modes- -rateer than identifying only the most prominent one-is another promising direction for future work.

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGEMENTS
Work supported in part by the NSF grants CCF-1553428 and CNS-1815221 and Open Philanthropy. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR00I 22000015. HL is supported by the Fannie and John Hertz Foundation and the National Science Foundation Graduate Research Fellowship under Grant No. 1745302. SJ is supported by the Two Sigma Diversity Fellowship.
Research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19 1111-99---1100 2-000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully debugging model mistakes using conceptual counterfactual explanations. 2022.
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Neural Information Processing Systems (NeurIPS), 2018.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint wtXiv::9070028880 2019.
Ananth Balashankar, Alyssa Whitlock Lees, Chris Welty, and Lakshminarayanan Subramanian. Pareto-efficient fairness for skewed subgroup data. In AISG, 2019.
Yujia Bao and Regina Barzilay. Learning to split for automatic bias detection. 2022.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. Computer Vision and Puttern Recognition (CVPR), 2017.
2017.
Yonatan Belinkov, Adam Poliak, Stuart Shieber, Benjamin Van Durme, and Alexander Rush. Don't t take the premise for granted: Mitigating artifacts in natural language inference. In Association for Computational Linguistics, 2019.
Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? In arXiv preprint arXiv:2006.07155, 2020.
Andreas Blattmann, Robin Rombach, Kaan Oktay, and Bjöm Ommer. Retrieval- diffusion models. arXiv preprini arXiv:2204. 1824, 2022.
Nicholas Carlini, Chang Liu, Jernej Kos, Úlfar Erlingsson, and Dawn Song. The secret sharer: Measuring unintended neural network memorization & extracting secrets. In USENIX Security Symposium, 2019.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In International Conference on Learning Representations (ICLR), 2019.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) 2019.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Neural Information Processing Systems (NeurIPS), 2017.
Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610-619) 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR), 2009.
Amit Dhurandhar, Pin- Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. arXiv preprint arXiv: 87220762., 2018.
Mengnan Du, Fan Yang, Na Zou, and Xia Hu. Fairness in deep learning: computational perspective. IEEE Intelligent Systems, 2021.
Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. International Conference on Machine Learning (ICML), 2019.
Sabri Eyuboglu. Maya Varma, Khaled Saab, Jean-Bende Delbrouck, Christopher fee-Messe,, Jared Dunnmon, James Zou, and Christopher Ré. Domino: Discovering systematic errors with cross- modal embeddings. arXiv preprint arXiv:2203. 14960, 2022.

---

Published as a conference paper at ICLR 2023
Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2015.
Vitaly Feldman. Does learning require memorization? short tale about long tail. In Symposium on Theory of Computing (STOC), 2019.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In International Conference on Computer Vision (ICCV), 2017.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR), 2019.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. In Nature Machine Intelligence, 2020.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning (ICML), 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. arXiv preprint 100iv:190..04451, 2019.
Moritz Hardt, Eric Price, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems, 2016.
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness without demographies in repeated loss minimization. International Conference on Machine Learning (ICML), 2018
He He, Sheng Zha, and Haohan Wang. Unlearn dataset bias in natural language inference by fitting the residual. In EMNLP-IJCNLP, 2019.
Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, and Zeynep Akata. Grounding visual expla- nations. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 264-279, 2018.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and surface variations. In International Conference on Learning Representations (ICLR), 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song. Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of -utto--distribution generalization, 2020.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Data- models: Predicting predictions from training data. In International Conference on Machine Learning (ICML), 2022.
Sarah Jabbour, David Fouhey, Ella Kazerooni. Michael W. Sjoding. and Jenna Wiens. Deep learning applied to chest x-rays: Exploiting and preventing shortcuts. In Proceedings of the Machine Learning for Healthcare Conference, 2020.
Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma. Fairnes-aaware learning through regulariza- tion approach. In IEEE International Conference Data Mining Workshops, 2011.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning (ICMI.), 2018
12

---

Published as a conference paper at ICLR 2023
Michael P. Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black- post-processing for fairness in classification. In AAAI/ACM Conference on Al, Ethics, and Sociery, 2019.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Sara Beery, et al. Wilds: A benchmark of in-the-wild distribution shifts. arXiv preprint urXiv:2012.0744) 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. In Technical report, 2009.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in Neural Information Processing Systems, 2017.
Guillaume Leclere, Andrew Ilyas, Logan Engstrom, Sung Min Park, Hadi Salman, and Aleksander Madry. ffev. https: /github......... com/ http///tfff//eeteeeeeeeeee ffcv/. 2022.
Zhiheng Li and Chenliang Xu. Discover the unknown biased attribute of an image classifier. The IEEE International Conference Computer Vision (ICCV), 2021.
Evan Z. Liu, Behzad Haghgoo, Annie Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. International Conference on Machine Learning, 2021.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision (ICCV), 2015.
Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXiv preprint cs/0205028, 2002.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. A survey bias and fairness in machine learning. 2021.
George A Miller. Wordnet: a lexical database for english. Communications the
the ACM, 1995.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier. In Neural Information Processing Systems (NeurIPS), 2020.
Curtis Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors test sets destabilize machine learning benchmarks. arXiv preprint arXiv:2 103. 14749, 2021.
Stephan Rabanser, Stephan Günnemann, and Zachary Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. Advances in Neural Information Processing Systems, 32, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In arXiv preprint 2rXiv:2103.0000 000000 2021.
Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225, 2017.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 2022.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should trust you?" explaining the predictions of any classifier. In International Conference on Knowledge Discovery and Data Mining (KDD), 2016a.
13

---

Published as a conference paper at ICLR 2023
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?". Explaining the predictions of any classifier. In International Conference on Knowledge Discovery and Data Mining (KDD), 2016b.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2020.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorcalistic -xtaee--mmae diffusion models with deep language understanding. arXiv preprint vrXiiv2205.11488 2022.
Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M. Rush. Learning from others' mistakes: Avoiding dataset biases without modeling them. In ArXiv preprint arXiv:2012.01300 2020.
Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpopulation shift. In International Conference on Learning Representations (ICLR), 2021.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: 13/2.6644 2013.
Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12853 12862,, 2021.
Nimit Sohoni, Jared A. Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Re. No subclass left behind: Fine-grained robustness in coarse coarse-grained classification problems. In Advances Neural Information Processing Systems, 2020.
Megha Srivastava, Tatsunori Hashimoto, and Percy Liang. Robustness to spurious correlations via human annotations. In International Conference Machine Learning (ICML), 2020.
Pierre Stock and Moustapha Cisse. Convnets and imagenet beyond accuracy: Understanding mistakes and uncovering biases. In European Conference on Computer Vision (ECCV), 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference Machine Learning (ICML), 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference Learning Representations (ICLR), 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. Computer Vision and Puttern Recognition (CVPR), 2016.
Amin Tafti and Doug W Byerly. X-ray radiographic patient positioning. StatPearls [Internet]. StatPearls Publishing, 2020.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. From imagenet to image classification: Contextualizing progress benchmarks. In International Conference on Machine Learning (ICML), 2020.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. Towards debiasing NLU models from unknown biases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
14

---

Published as a conference paper at ICLR 2023
Vijay Vasudevan, Benjamin Caine, Raphael Gontijo-Lopes Sara Fridovich- Keil, and Rebecca Roelofs. When does dough become a bagel? analyzing the remaining mistakes on imagenet. 2022.
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In International Conferencee on Machine Learning (ICML), 2021.
Kai Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. arXiv preprint arXiv:2006. 09994, 2020.
Kaiyu Yang, Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky. Towards fairer datasets: Filtering and balancing the distribution of the people subtree the imagenet hicrarchy. http:
// /image-net. org/ /pppttteeeeeee sep-17-2011 9, 2019. Accessed: 2020-10-01.
Chin-Kuan Yeh, Been Kim, Sercan Arik, Chun-L.iang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-awarc concept-ber bascept-based explanations in deep neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna Gummadi. Fair- ness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In International World Wide Web Conference (WWW), 2017.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint ar Stt..0770.049555 2017.
15

---

Published as a conference paper at ICLR 2023
A
SETUP DETAILS
A.1
CODE
Our code is available httpp/////attaaoocoo///////////a
A.2 FORMAL DESCRIPTION OF METHOD
Preliminaries We consider standard image classification problem with image space X' and label space V/ training data (x, y) E Dtrain: held-out validation set Dett, and test set Drast- Using the training data, we have trained base model (such as neural network) fun X + 3/ to predict label given the input. For convenience, we define the "correctness" function c X' + +11, where c(x) = 1 if fon (z) correctly outputs the label associated to z, and c(x) -1 otherwise.
linear support vector machine (SVM) is a simple linear classifier, which finds a hyperplane separating the data with the maximum margin. Specifically, given d-dimensional embeddings { with labels {y) E (- linear SVM computes coefficients E IRd and intercept b E IR which optimize the objective:
1 + CCY
max 1 - (w +))
This optimization problem seeks maximize the margin, while penalizing misclassifications using a soft hinge loss. The parameter C, learned via cross-validation, trades off between these two objectives.
Once the parameters w and b have been learned, for a given embedding x, we can calculate the decision value:
h(x) = w x b
This represents the confidence of the SVM, and is proportional to the (signed) distance of x to the hyperplane determined by w. b. The prediction of the SVM on is simply then the sign of f(x)), and w is the "direction" of the failure mode.
Diagnose We evaluate the strength of the most interpretable error pattern within class y by quanti- fying the ability of per-class SVM h to predict validation data errors on that class. Recall that each SVM is trained by volssavali..ion. We use the average balanced accuracy, across cross validation splits to measure the strength of the underlying spurious correlation. Letting ny denote the number of instances of class y in given subset of data, the balanced accuracy is defined by
a =
Σ- VEY ny =:(.y))
Σ
((cx))11)
Intuitively, the more interpretable and widespread the error pattern is, the better it can be captured by linear predictor in latent space. To demonstrate this, on CelebA, we subselect the data to underrepresent the hard demographics ("old" "female" and "young" "male") in the training data to lesser or greater degrees. As was shown in Figure 4, ā increases as the strength of the spurious correlation increases (i.e. as the proportion of hard demographics decreases). This measure can be used determine which classes have clear bias present, for use in the following interpretation and intervention steps.
Interpret Once the SVMs have captured any difficult subpopulations, we interpret them by surfacing the most extreme examples in the test set, as well as automatically generating captions for the identified failure modes. These interpretability techniques are only possible due to two key properties of SVMs: (1) they are constrained to capture simple (approximately linearly separable) patterns in the embedding space, and (2) their decision boundary is easily accessible, therefore providing notion of direction and distance in the embedding space.
Most Extreme Examples Each SVM h2 provides natural measure of how hard an input is via its distance to the decision boundary, which was validated in Figure 2. Thus, we subs- elect from Dteat the images within class y. {., } 1e which minimize Σ, h" (clip(=;)) )), and likewise the & images {a } which maximize E, h2 (clip(=;)) )). By definition, these
16

---

Published as a conference paper at ICLR 2023
images exemplify the decision rules learned by h2 Therefore, {. provide con- cise visual summary of "hard" examples, and similarly {x, ke of "easy" examples. As was shown in Figure 3, for class y ="old", {., { indeed consists of all women, while {2ī Vi=1 consists of all men.
Captioning To generate caption for the predicted error-prone subpopulation of a class y. we leverage the shared language CLIP embedding space on which the SVMs were trained. For class let the base caption 3 be "a photo of y and its embedding r = ((). Given set of candidate captions C, we aim to select the caption that h classifies most strongly, in terms of distance from the decision hyperplane, as an error. Consider a candidate caption y E and its embedding c = clip(7). Since directions have been recently shown to be meaningful in CLIP space (Radford et al., 2021), associate each caption with a direction in the CLIP embedding space via de c-r The chosen caption for errors class y is then argmax, h (de)-
Intervene After identifying and interpreting the hard per-class trends, we use them to improve accuracy Jnn by retraining. Let = {xr,)) Diraii s.t. (c(clip(x)) < 0} denote the set of training examples predicted to be hard. We consider various methods for intervening on H during retraining.
Upweighting Suppose fm originally trained to minimize objective function
((Drriin)
Σ
fnn (x))
(a.g))EDenasn
We modify this objective by upweighting the flagged examples by some factor h:
C'(Derain)
Σ
(Mi(x E H) + 1(x H)) E(y, [nn()))
(x.g)
Subsetting If there exists an external pool of examles Dext that were not used to originally train fans Hezt ((x, y) E EDrt..... (cllp((()) < 0} and let Dirain Dirain + Hext- Thus, we add to the training set the most useful new examples from Dent. (In practice we simulate this setting by dividing the original training set into Dirain and Dexts prior to training frn the first time.)
A.3 MODEL TRAINING DETAILS
We train ResNer18 models and perform hyperparameter search using the held out validation set. Dataset specific experimental details can be found in Sections B and C. For all datasets except ChestX-ray 14, we use standard softmax classification with cross-entropy loss.
Our training uses SGD with momentum and weight decay. We further use triangular eyclic learning rate, where the learning rate linearly increases for pre-specified number of epochs ("peak epochs") until hitting the base learning rate ("peak LR"), and then linearly decays for the rest of the epochs until reaching zero at the end of training.
For ChestX-ray14 14, we predict for each condition binary output with binary cross-entropy loss; the threshold is then chosen to maximize the F1 score on the held out validation set.
We train our models using NVIDIA V100 gpus. We use the FFCV framework (Leclere al., 2022) to further speed up training.
A.4 SVM TRAINING DETAILS
We use the pre-trained CLIP (Radford et al., 2021) model, equivalent to a ViT-B/32. We embed the images using this encoder. We whiten the data (using the mean/standard deviation of the embeddings computed from the training images) and then normalize the embeddings to be unit vectors.
We train linear SVM on the embeddings of the held-out validation set with balanced class weighting to predict whether the model got the image correct or incorrect. The regularization constant is chosen using 2-fold cross validation by selecting the constant with the best balanced accuracy (i.e the average
17

---

Published as a conference paper at ICLR 2023
of the recall obtained for each class). These cross validation scores are also exported to quantify the strength of the captured shift, as described in Section
A.5 FILTERING DETAILS
After holding a pool of data that was not used train the original model, we select those images with the most negative SVM decision values to add to the training set; we also compare to alternative selection methods (by confidence value, and random). We applied this method CIFAR- -IFAR-100, and reported the accuracies on the hard subpopulation in Figure 6. Below, we report the overall test accuracies before and after intervention when adding 100 images per superclass.
Original
Random
Confidences
SVM
Overall accuracy
0.696
0.712
0.716
0.720
A.6 CAPTION GENERATION
In Section 2, we described how we can assign a caption to each captured failure mode by choosing the caption that is most aligned with the positive or negative direction extracted by the SVM. To perform caption assignment, we need to create a set of relevant candidate captions that we expect, roughly, to lie the same space as the images that the SVM was trained on. particular, completely nonsensical captions may have undefined CLIP embeddings. Similarly, completely irrelevant captions may be out of distribution for the SVM, which was trained on images of a specific class.
To generate a sensible candidate set, we programmatically generate captions of the form "A photo of <adjective>> <noun> <prepositional phrase>", where the adjective and prepositional phrase are optional. We describe the list of adjectives, nouns, and prepositional phrases for each dataset in experimental details section for that task. We chose this approach order to guarantee the descriptiveness of the candidate captions set. Automatically curating this candidate set, for example by generating reasonable sentences using a language model, is an interesting avenue for future work.
A.7 DIRECTLY DECODING SVM DIRECTIONS WITH DIFFUSION MODELS
In Section 2, described how use diffusion models that sample from CLIP space directly decode our extracted SVM direction. We provide further details here.
Several models, including retrieval augmented diffusion models (Blattmann al., 2022) and DALL-E 2 (Ramesh et al., 2022) sample from CLIP's shared vision/language space. In contrast, Stable Diffusion (Rombach et al., 2022) uses the hidden states of the CLIP's text encoder (before it is projected into the common image/language space). In our work, we use the pre-trained 768 x 768 cerieval-auugennt diffusion model (RDM) (Blattmann et al., 2022), but only use text prompting (no retrieval step). The checkpoint can be found here.
We fit the SVM on the same CLIP space that the RDM uses (CLIP ViT-L/14) with &a normalization but not whitening of mean/std. Let w be the normalized SVM direction. For a reference caption with normalized embedding т, we then spherically interpolate r and either w or -w (e.g.. z = slerp(r, a)) to generate hard or easy images respectively. The degree of rotation specifies the level of difficulty (we generally choose a in the range [0, 0.3).. Unless otherwise stated, we use = 0.1 to generate the images. We then pass the spherically interpolated vector in lieu of the text-conditioning vector to the RDM.
18

---

Published as a conference paper at ICLR 2023
A.8 TAILORED DATA AUGMENTATION USING TEXT-TO GENERATIVE MODELS
In Section 2, we described how we use text-to-image generative models to perform stable diffusion. Specifically, input the captions extracted by SVM (easy, hard, and reference) into a pre-trained Stable Diffusion (Rombach et al., 2022) model and generate 100 synthetic images per class. The checkpoint ("sd-v1-4. ("sd-v1-4.ckpt") can found here. We then fine-tune the original model (retraining only the last layer) on the combination of the new images and the original training dataset. Below we report the overall test accuracies after fine-tuning: overall accuracies are maintained up 1.3%.
Original
Easy
Hard
Neutral
Overall accuracy
|
0.791
0.785
0.778
0.784
19

---

Published as a conference paper at ICLR 2023
B DATASETS WITH KNOWN CHALLENGING SUBPOPULATIONS.
B.I CELEBA
In Section 3, we applied our framework to the task of predicting age on CelebA (Liu et al., 2015) with a planted correlation with gender. Here, in this section, we discuss the training details and additional experiments on that task.
B.1.1 EXPERIMENTAL DETAILS
Dataset construction We construct training dataset with the following demographic breakdown: 6203 each "old" "female" and "young" "male", and 24812 of "old" "male" and "young" "female". There are thus four times many faces of old men old women (resp. young women young men). The held-out validation set contains 1795 examples for each of the demographic categories, and the original test split of CelebA. for our final test set. We consider the full CelebA dataset and setting where the validation distribution matches the training distribution in Appendix Sections 1.1.7 and 1.1.4. We consider images with resolution of 75x75
Hyperparameters
We use the following hyperparameters.
Parameter
Value
Batch Size
512
Epochs
30
Peak LR
0.02
Momentum
0.9
Weight Decay
5 x 10-4
Peak Epoch 2
Caption generation We reference caption "A photo of a person." We generate prepositional phrases using the attributes provided by the CelebA dataset. The adjectives, nouns, and prepositional phrases are as follows:
Adjectives: [none, 'old', 'young']
Nouns: ['person', 'man', 'woman']
Prepositions: none, 'who has stubble', 'who has arched eyebrows', 'who has bags under their eyes', "who has bangs', "who has big lips', "who has a big nose", 'who has black hair', 'who has blond hair', 'who has brown hair', 'who has bushy eyebrows', "who has a double chin', 'who is wearing eyeglasses', 'who has a goatee', ; 'who has gray hair', "who has heavy makeup on', 'who has high checkbones', 'who has mouth that slightly open', 'who has a mustache', "who has narrow eyes', "who has no beard', "who has an oval face', 'who has pointy nose', 'who has receding hairline", 'who has rosy cheeks", 'who has sideburns'. who has smile', "who has straight hair", 'who has wavy hair", 'who has earrings', 'who is wearing a hat', 'who has a lipstick on', "who is wearing a necktie"
We thus generate a caption for every combination of adjective, noun, and prepositional phrase.
B.1.2 UPWEIGHTING INTERVENTION
Can we use the linear classifiers from our framework to reduce the model's reliance on gender by intervening during training? To examine this, after flagging examples in the training set as members of the minority population, we upweight these examples during the model retraining. As shown in Table 1, this intervention increases performance on the minority subpopulations, and is even comparable in its impact to the "oracle" approach of upweighting all "young men" and "old women". Note that. since the base model has nearly zero training error, it is confident on all training examples. suitable for guiding analogous intervention.
Thus, confidences
20

---

Published as a conference paper at ICLR 2023
Test Subgroup Accurucy
Minority Subclasses
Majority Subclasses
Overall Accuracy
Young Male
Old Female
Young Female
Old Male
Base Model
0.568 0.006
0.542 + 0.010
0.915 + 0.002
0.904 + 0.002
0.795 + 0.002
Oracle Upweight
0.600 0.007
0.569 = 0.010
0.896 + 0.004
0.872 + 0.004
0.797 + 0.002
SVM Upweight
0.590 0.011
0.568 + 0.004
0.903 + 0.003
0.901 + 0.008
0.796 + 0.003
Table The performance (mean/std over five runs) of intervening during training by upweighting examples in the CelebA training set. SVM Upweight doubles the loss weight for all training examples that were predicted as incorrect by our framework. Oracle Upweight doubles the loss weight of all minority examples using the dataset's annotations. We find that intervening using the SVM increases the accuracy of the minority subclasses to almost the same extent as the (optimal) oracle intervention.
B.1.3 GRADATIONS SPURIOUSNESS
In Section applied our framework subset of the CelebA dataset, chosen such that the spurious correlation between age and gender was intensified. Specifically, we vary the intensity of that spurious correlation, and demonstrate that the test error of the per-class SVMs reflects the degree of the spurious correlation. In particular, for from 1 to 8, the train dataset is filtered such that there are n-times as many old male instances as young male instances, and n-times as many young female instances old female instances.
As shown Figure 11, the test accuracy of the base model decreases as the spurious correlation grows stronger, because the base model has relied more and more the spurious correlation. This complements Figure 4 in the main text, which demonstrated that the SVMs fit the test data better with increasing spurious correlation.
Test Acc. on Spurious
0.7
0.6
Degree of Spurious Age-Gender Correlation
Figure 11: Test accuracy on spurious instances of CelebA, as a function the spuriousness n of the training dataset.
21

---

Published as a conference paper at ICLR 2023
B.1.4 CELEBA EXPERIMENTS WHEN THE VALIDATION MATCHES THE TRAINING DISTRIBUTION.
In Section 3, we applied our framework to the CelebA dataset and used a validation set that was balanced across age and gender. Here, we instead consider the case where the validation matches the train distribution, and thus also contains the strong spurious correlation with gender: specifically, the validation set contains 810 old female and young male faces, and 3240 old male and young female faces.
In this setting, there are fewer minority examples for the SVM to learn from, which thus makes the task of selecting errors harder. However, we find that our framework still captures the gender correlation (Figure 12). Moreover, our method able to isolate this hard subpopulation better than using model confidences (Figure 13).
CelebA: Old
CelebA: Young
FVP Gores 4,335
200
Sin Gooe
2225
SAM Scar
Hard Exemplars
checkbones"
Easy Exemplars
SVM Caption: photo of old man who chewks"
SVM Caption: "a photo of woman who Ovail feca"
Figure 12: The images and captions for each class with the most extreme SVM decision values
1.0
Order
1.0
Order SVM Decision Value Confidence Base Population
Fraction Female
SVM Decision Value
Fraction Male
0.8
Confidence
0.8
Base Population
0.6
0.4
0.4
1000
2000
3000
4000
5000
5000
10000 Top K Flagged
15000
Top Flagged
Class: Old. Minority Gender: Female
(b) Class Young. Minority Gender: Male
We repeat our CelebA experiments from Section 3, using a held out validation set that matches the training distribution. We display for each class, the fraction of the top images that are of the minority gender when ordering the images by either their SVM decision value or by the model's s confidences.
Figure 13:
B.1.5 FURTHER EXPERIMENTAL DETAILS ON BASELINES
In Figure 2 of from Section B of the main paper, we compare the efficacy of using our SVM (versus several baselines) for selecting the minority gender in the CelebA setting (replicated again below). We found that method was able more consistently select the hard subpopulation than the other baselines. In this section, we discuss the training details for this experiment.
Domino Domino (Eyuboglu et al., 2022) is automatic method for identifying challenging subsets of the data ("slices") by fitting a Gaussian mixture model CLIP embedding space. They then choose captions generated from large language model that are closes the mean of each identified cluster. Our use of an SVM provides a stronger inductive bias towards simple subclasses. Furthermore by encapsulating the failure modes direction in the latent space, more easily score, intervene,
22

---

Published as a conference paper at ICLR 2023
SVM Decision
SVM Decision Value
1.0
Damiina [EVS+22]
1.0
Damino [EVS+22]
Fraction Female
LfF [NCA+2D]
LfF [NCA+20]
Confs. (Early Stop) Confidences
Fraction Male
0.8
Confs. (Early Stop) Confidences Base Population
Base Population
0.6
0.6
0.4
0.4
0
1000
2000
3000 4000
5000
5000
10000
15000
Top K Flagged
Top K Flagged
(a) Class 0: Old
(b) Class I: Young
Figure 14: Repeated again for convenience: Figure from Section B of the main paper. For each class in CelebA, the fraction of test images that are of the minority gender when ordering the images by either their SVM decision value, model's 's confidences, or a variety of other baselines. Our framework more reliably captures the spurious correlation than all other baselines.
and caption the failure modes. In particular, we are able to caption the extracted failure mode directly, rather than using the proxy of cluster of hard examples. On the other hand, Domino can more easily distinguish between multiple failure modes for the same class because it identifies multiple hard clusters in the latent space.
We use the same parameters as suggested in the paper with log likelihood weight=10, hat log likelihood weight=10, num slices=2.
Captioning:
We first surface the top 3 identified negative captions from our framework and from Domino:
Our Framework:
Old: "a photo young woman who has an oval face", "a photo young woman who has heavy makeup on", "a photo of young woman who has brown hair"
Young: "a photo of a man who has no beard", "a photo of a man who has a mouth that slightly open", "a photo of who has smile"
Domino
Slice 0: "a photo of her debut album.", "a photo of carole lombard.", "a photo of tina turner."
Slice 1: "a photo of compelling man.", "a photo of satisfied man.", "a photo of hungry man."
Both methods capture gender the primary failure mode. However, Domino tends toward more specific and thus sometimes more niche captions (i.e "a photo of carole lombard") than overall trends. This is because Domino picks the caption that happens to be closest to the mean of the slice.
Learning From Failure Learning from Failure (LfF) (Nam et al., 2020), uses a biased classifier trained with Generalized Cross Entropy (GCE) loss to select hard examples. Specifically, if p(x) is the softmax probability vector assigned by the model for input æ, the GCE loss for example (I. y) is
1 (z)
GCE(p(z),
For this baseline, train model with GCE loss using the default hyperparameter q 0.7 as from the original LfF paper.
Early Stopping In Just Train Twice (JTT) (Liu et al., 2021), classifier trained only few epochs is used to identify low confidence examples. For this baseline, we stop at epoch=10 (out of 30) and evaluate model confidences.
23

---

Published as a conference paper at ICLR 2023
B.1.6 USING OTHER LATENT SPACES
In our paper, we distill failure modes using the CLIP latent space. However, our method is not specific to CLIP (we can use any embedding that can capture important features and is agnostic to the original training dataset). In Figure 15, we explore using two other latent spaces for our method:
Inception: We take features from Inception V3 model (Szegedy al., 2016) trained on ImageNet.
Original Latent Space: We take features from the model's original latent space (i.e. its penultimate layer).
We find that using our method with Inception embeddings is only slightly worse than using CLIP. Since CLIP embeds both language and images the same latent space, using CLIP further enables us to automatically caption the extracted directions.
However, since the model's latent space tied closely to the training dataset, training SVM on the original latent space closely mirrors model confidences. Moreover, since the original model fit the training data perfectly, the SVM extracted by our method predicts all training examples as correct even images that belong to the minority subpopulation and thus cannot be used for upweighting interventions. Thus, our method works best when using embeddings that are agnostic to the specific dataset.
1.0
1.0
Inception
Inception
Fraction Female
Original Latent Space
Original Latent Space
Confidences
Fraction Male
0.8
Base Population
Base Population
0.6
0.4
1000
2000
3000
4000
5000
0
5000
10000
15000
Top K Flagged
Top K Flagged
(a) Class O: Old
(b) Class 1: Young
Figure 15:
We repeat our CelebA experiments from Section 3, and compare using the CLIP latent space to using Inception features or the model's 's original latent space.
24

---

Published as a conference paper at ICLR 2023
B.1.7 FULL CELEBA DATASET
In Section 3, we subselected images from the CelebA dataset to intensify the spurious correlation between age and gender. Here, we train base model on the unaltered CelebA dataset instead. Table 2 summarizes the presence of this spurious correlation in the dataset. The per-class SVMs pick up on the spurious correlation, and to a greater extent than do confidences, as shown in Figures 16a and 16b. The validation set on which the SVMs were trained is the standard CelebA validation set.
Class (Age)
% Minority Gender Overall
Old
Young
39.5%
31.6%
Incorrect 55.0%
57.1%
Table First row: the percentage of each age group consisting of that age group's minority gender (female for old, male for young), Second row: the percentage of the base model's 's mistakes on each age group which are of the minority gender. Together, these capture the degree of spurious correlation present, as the base model makes a disproportionately high number of mistakes on the minority gender for each age group.
Class: Old
Class: Young
SVM Decision Value
1.0
SVM Decision Value
Fraction Corrupted
Confidence
Base Population
Fraction Corrupted
0.8
Confidence
Base Population
0.8
0.6
0.6
0.4
0.5
0.2
0.0
1000
2000
3000
4000
5000
2000
4000
6000
8000
10000 12000 14000
Top
Top
(a) Class: Old. Minority Gender: Female
(b) Class Young. Minority Gender: Male
Figure 16: For each class in the original CelebA dataset, the fraction of the top K images that are of the minority gender when ordering the images by either their SVM decision value by the model's confidences.
25

---

Published as a conference paper at ICLR 2023
B.2 COLORED MNIST
In the Colored MNIST dataset (Arjovsky et al., 2019), a spurious correlation is introduced between tint and label for the handwritten MNIST dataset. The learning task labels y binary, and generated as follows. We begin with labels y. which are 0 for digits less than 5 and 1 for digits greater than equal to To obtain the labels y actually used in the classification task, the labels are flipped with probability 0.25. In other words, 75% of the time, the class label in the learning task is associated with the digit, and this feature generalizes
test set.
However, we use color to plant spurious correlation between the class label and the tinr of the add red/green tint that correlates perfectly with y (y/ corresponding
digit. In particular, to red, and = 1 corresponding to color green), to 90% of the training set, but only 50% of the validation and set tests. As a result, there a spurious correlation between the tint and the label y in the train set, which does not generalize to the validation and test sets.
Thus on training data, the color the digit is a better predictor of its label y than the digit's s true shape, but the digit's true shape is better predictor than its color the test and validation data.
Hyperparameters. We use the following hyperparameters for training.
Parameter
Value
Batch Size
512
Epochs
15
Peak LR
0.1
Momentum
0.9
Weight Decay
5e-04
Peak Epoch
As shown in Figure 17, the fit of the SVM on the test data improves as the spurious correlation grows stronger. For each value of x, we also intervene by doubling the weight of the examples flagged as "hard" by the SVM. Figure 18 demonstrates that while the test accuracy decreases with increasing spurious correlation, intervention effective (and increasingly so with v)) at improving test accuracy,
SVM Cross Validation Score
1.0
1.0
0.9
Test Accuracy
0.8
0.8
0.7
0.6
Before Intervention After Intervention
0.6
0.5
0.6
0.7
0.8 0.9
0.5
0.6
0.7
0.8
0.9
Degree of Spurious Digit-Color Correlation
Degree of Spurious Digit-Color Correlation
Cross-validation score on the test as function of the spuriousness, i.e. the tinted fraction x of the training set, for Colored MNIST.
Figure 17:
Figure 18:
Test accuracy of Colored MNIST before and after the upweighting intervention, as function of the spuriousness training set.
26

---

Published as a conference paper at ICLR 2023
B.3 IMAGENET-C
B.3.1 EXPERIMENTAL DETAILS
ImageNet-C is a dataset created by Hendrycks & Dietterich (2019) to benchmark neural nets' robustness common perturbations. Using the associated code, we create corrupted ImageNet dataset in which 10% of the training set is corrupted, and 50% of the validation and test sets are corrupted. For simplicity, we restrict corruptions to Gaussian pixel-wise noise, with variance equal to 0.26. We train a ResNet18 the base architecture, with standard ImageNet normalization. The resultant validation and test accuracies are 51.1% and 48.9%, respectively. but drop to 45.5% and 43.0% if restricted to corrupted images (validating that the corrupted images indeed harder).
Hyperparameters. Hyperparameters were chosen to match those of the ordinary ImageNet dataset:
Parameter
Value
Batch Size
1024
Epochs
16
Peak LR
0.5
Momentum
0.9
Weight Decay
5e-4
Peak Epoch
B.3.2 RESULTS
Capturing common corruptions. We find that the per-class SVMs pick up the Gaussian corruption, as shown in Figure 19. Since the corruption is light, this is non-trivial: in particular, notice that the images chosen "hardest" not necessarily corrupted, as shown in Figure 20.
0.650
Fraction Corrupted
0.625
0.600
0.575
0.550
SVM Decision value
0.525
Confidence
Base Population
10
20
40
50
Figure 19;
The fraction of ImageNet-C test images that corrupted with Gaussian noise when ordering the images by either their SVM decision value or by the model's 's confidences, averaged over all classes. Our framework more reliably captures the corruption than using confidences.
27

---

Published as a conference paper at ICLR 2023
Intervention. In this setting, does not make sense intervene based the captured corruption: since blurred images represent neither a spurious correlation nor an underrepresented subpopulation, but instead simply "hard" class, expect neither upweighting nor subsetting to improve perfor- mance. The purpose of this dataset is to exhibit that our method can capture the hard subelass of though (as shown in Figure 20) there many ways image can be hard, because it is a shared pattern across many of the errors.
corrupted images,
Sur 201 200
Iht Sover
CAR
GAB 2.11
CAM
OAe Gan Geren 211
Ove
200 Tan 2113 Owedd
IN 0.11
Figure 20: The images with the most positive (upper) and negative (lower) decision values for the ImageNet-C class "tench". While more of the bottom images are blurred, the "hardest" images may be tenches in strange positions.
28

---

Published as a conference paper at ICLR 2023
B.4 CIFAR-100
In this section, we explain the experimental details of the experiment on CIFAR-100 in Section 3. Specifically, we consider the task of predicted the super-classes of CIFAR-100 dataset, which contains 20 super-classes (each of which contain 5 subclasses).
Dataset construction We split the original training split into 20% validation set, 40% training set, and 40% extra data (used in the subset intervention). The original test split serves our test set. In the training dataset, we choose one subelass to under-represent by removing 75% that subclass from the data. We choose this subclass by picking the worst- porst-performing subclass for each superclass based on a model trained on the original training split. Specifically, we drop the classes:
faquatic mammals: beaver, fish: shark, flowers: orchid, food containers: bowl, fruit and vegetables: mushroom, household electrical devices: lamp, household furniture: table, insects: caterpillar, large carnivores: bear, large outdoor man-made things: bridge, large outdoor natural scenes: forest, large omnivores and herbivores: kangaroo, medium sized mammals: possum, non-insect invertebrates: lobster, people: baby, reptiles; lizard, small mammals: squirrel, trees:
willow tree, vehicles bus, vehicles 2: streetcar)
Hyperparameters.
We use the following hyperparameters train our models.
Parameter
Value
Batch Size
512
Epochs
35
Peak LR
0.5
Momentum
0.9
Weight Decay
5.0 10-4
Peak Epoch
5
B.4.1 RESULTS ON 80% TRAIN, 20% VALIDATION
In this section, discuss results on CIFAR-I00 setup with 80% of the original train split allocated for the train dataset and 20% allocated for the validation dataset (and no extra data). As above, in the training set, we underrepresent one subelass per superclass (by removing 75% of that subclass from the dataset). We then will explore the downstream implications of the CV score in this setting.
Efficacy in isolating the minority subpopulation In Figure 21, we replicate the experiment in Figure 5 for this new split. The results are similar: using the SVM's decision value more consistently identifies the top K images over using model confidences.
Fraction Minority Subclass
0.7
SVM Decision Value
Confidence
0.6
Population
0.5
0.4
0.3
0.2
0
100
200
300
400
500
Flagged
We replicate the experiment in Figure 5 for a scenario where 80% of the training dataset train and 20% is validation. For each superclass, the fraction of the top K flagged test images that belong to the underrepresented subclass (averaged over classes) is shown. We compare using the SVM's 's decision value and the model's 's confidences to select the top K images; the SVM more consistently identifies the minority subpopulation.
Figure 21:
29

---

Published as a conference paper at ICLR 2023
A deep-dive into the cross-validation (CV) score Recall from Section 2 that the cross validation (CV) score of the SVM serves as a measure of the extracted failure mode's ' strength in the dataset. In Figure 4 of the main paper, we found that (in the CclebA setting) the CV score of the SVM was highly correlated with the degree of the planted shift.
In this section, analyze the downstream implications of the CV score in the CIFAR- setup. Specifically, we evaluate our method on CIFAR-100 where the minority subpopulation underrepre- sented different degrees (e.g., removing 10%, 20%, of the minority subclass).
As in the CelebA case, the mean CV score across all classes increases as we remove more of the minority subclass (Figure 22a). The CV score is negatively correlated with the downstream model's accuracy on the minority subpopulation (Figure 22b). Thus, the CV score can help measure the degree of the underlying failure mode and its impact on downstream performance.
0.71
the minorty suspopulation ation
0.70
0.70
Model accuracy on
0.69
mean CV
0.60
0.55
0.67
0.50
0.0
0.2
0.4
0.6
0.8
0.67
0.68
0.69
0,70
0.71
Fraction minority population removed
mean CV score
(b)
Figure 22: CV score correlated with the strength the failure mode. (a) CV score (averaged over classes) when different fractions of the minority population have been removed from the training data. CV score increases as the degree of underrepresentation becomes more severe. (b) The CV score negatively correlates with the model's final accuracy on the minority subpopulation.
Since the CV score correlated with the strength of the failure mode, also reflects the membership of the error set (predicted and actual). With higher CV scores, more of the predicted errors actually part of the planted subpopulation (Figure 23).
of actual emors that were of the minority subclass
% of predicted errors that were subciass
0.450
0.425
0.400
0.40
0.375
0.35
0.350
0.67
0.68
0.69
0.70
0.71
0.67
0.68
0.69
0.71
mean CV score
mean CV score
(a)
Figure 23: As shown in Figure 22a, the degree of the underrepresennaaiio goes up, the CV score also increases. The CV score is thus correlated with the proportion of the error set - (a) predicted by the SVM and (b) actual that is actually part of the minority subpopulation.
Inter-class CV Scores We now fix the degree of underrepresenttatin (removing 75% of the minority examples). In this case, the size of the minority subclass is the same for each of the 20 superclasses. However, the impact of underrepresentation is not equal between classes. For example, one might need fewer examples to distinguish beaver vs. bear over two different types of trees. Recall that we train an SVM per class: thus, each class has its own CV score. What do the differences in the class CV score signify?
30

---

Published as a conference paper at ICLR 2023
For each of the 20 superclasses, plot the CV score of that class's SVM against the fraction of errors (predicted and actual) which are the minority subpopulation (Figure 24). Indeed, classes with higher CV score seem to indicate stronger shift, in that a greater proportion of error set belongs to the minority subpopulation.
r= 0.616
r= 0.675
Fraction Predicted Errors that are Minority Subclass
Fraction of Actual Errors are Minority Subciass
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0.65
0.70 CV Score
0.75
0.65
0.70
0.75
Score
(b)
Figure 24: For each of the 20 superclasses, we plot the CV score for that class against the fraction of the error set for that class - (a) predicted by the SVM and (b) actual that belongs to the minority subpopulation. We further display the Pearson correlation.
31

---

Published as a conference paper at ICLR 2023
C NATURAL DATASETS
C.I
CIFAR-10
In this section, describe the experimental details and additional results for applying our framework on the CIFAR-10 dataset.
C.I.1 EXPERIMENTAL DETAILS
Dataset. Since the original CIFAR- 10 model has very high (93%) accuracy, we do the majority of our experiments using subset of CIFAR-10. Specifically, we train the model with 20% of the original training split, and reserve 20% for the validation set and 60% as extra data for the subset intervention. We discuss results on the full CIFAR-I0 10 dataset in Appendix Section C.1.7.
Hyperparameters
We use the following hyperparameters.
Parameter
Value
Batch Size
512
Epochs
35
Peak LR
0.5
Momentum
0.9
Weight Decay
5.0 10-4
Peak Epoch
5
For fine-tuning, we use the same parameters, except with peak I.R of 0.1 and for epochs.
C.1.2 CAPTION GENERATION DETAILS
For our CIFAR-10 experiments, we consider two caption sets: CIFAR-Simple, and CIFAR-Extended The two caption sets only differ in their sets of possible nouns (extended has larger vocabulary): they otherwise have the same list of adjectives and prepositions. In the main paper, and unless otherwise noted, we use CIFAR-Simple this is because the diffusion models struggle with extended vocabularies. In the appendix, we include results with -xttedeeded where noted.
For each CIFAR- 10 class, we use the reference caption "a photo of a <class>." " We then generate candidate caption set of the form "a photo of <adjective>> <noun> <preposition>". The adjectives, nouns, and prepositional phrases are selected as follows:
Adjectives: [none, 'white", 'blue', 'red', 'green', 'black", 'yellow', 'orange", 'brown'), ].
Prepositions: We choose a set of prepositions for each class. Specifically, all classes have the following potential prepositions: [None, 'outside', 'inside", 'on black background', "on a white background', 'on a green background', "on a blue background") Then each class has the following class-specific prepositions:
'dgg: I'on the grass', 'in house", 'in the snow", 'in the forest'), ], cat': I'on the grass', 'in house', 'in the snow', 'in the forest'], 'bird': ['flying', 'perched', 'in the air', 'on the ground'].
'horse': ['in field'. 'in the grass'],
"airplane": ['flying', , 'in the air', 'on the tarmac', 'on road'), 'truck': ('on the road', 'parked'].
"automobile:: I'on the road', 'parked'),
'deer': I'in a field', 'in the grass', 'in the snow", 'in a forest'],
'ship:: I'in the ocean', 'docked", 'in the water', 'on the horizon'). 'rro':: ['in the grass', 'in pond'],
do not vary the noun (i.e., captions of class cat will use the noun we find the corresponding synset in the WordNet (Miller, 1995) hicrarchy. We then choose as possible nouns all the common names of the synsets that descendents of the
Nouns: For CIFAR-Simple
cat). For
32

---

Published as a conference paper at ICLR 2023
CIFAR synset. For example, for the class cat, possible noun is "Persian cat." We exclude nouns that contain words that are not in the NLTK (Loper Bird, 2002) words corpus.
For both sets, generate the captions by taking every combination of adjective, noun, verb.
33

---

Published as a conference paper at ICLR 2023
C.1.3 MOST EXTREME EXAMPLES AND CAPTIONS
In Figure 25 we show the most extreme examples and captions by SVM decision value for all 10 classes.
Cat
Automobile
Hard Exemplars
Simple:
grass"
Simple: photo Extended: photo of ambulance"
Extended: la photo of white mouser on the gross"
Easy Exemplars
Simple: 'a photo inside"
Simple: photo blue automobile black background Extended: photo of black seder"
Estended:" la photo a brown domestic cat invider"
Bird
Airplane
Hard Exemplars
Simple:: "a photo of bird Tlying"
Simple: "a photo of red airplane inside" Extended bomber"
Extended: white Inside"
Easy Exemplars
Simplec photo of bleck sind the ground" Extended: shota black
Simple: photo black sirplane"
Extended:" black whise background"
Horse
Dog
Hard Exemplars
Simple: "a photo of blue horse Inside" Ertended: "a photo of protohippus noide"
Simple: photo of black dog"
Extended:: 'a photo of black fest""
Easy Exemplars
Simple: photo a brown hone outside"
Simple: photo of white dag on the grass"
Extended: "a photo of black stesplechaser outside"
Extended: phoso of white saint bernard inside"
Frog
Deer
Hard Exemplars
Extended: photo of aqua"
Simple "3 photo of's deer
Simple: photo of red frog outside"
Extended: "a photo of bob incid""
Easy Exemplars
Simple: "a photo of yelliw frrg" Extended: photo of chamelson tree Irag"
Simple: "e photo of grewn deer in firee"" Extended: photo of mapiliin fores""
Figure 25: Most extreme images with captions from CIFAR-Siminn ---mpl or
34

---

Published as a conference paper at ICLR 2023
Ship
Truck
Hard Exemplars
inside" photo racing de
Simple: truck
Easy Exemplars
Simple cloched Extended: photo brown bugioat outside"
Simple: papooo passenger
Figure 25: Most extreme images with captions from CIFAR-Simple or CIFAR-Extended
C.1.4 MORE EXAMPLES DIRECTLY DECODING SVM DIRECTION
In Figure 26, we include more examples of using spherical interpolation directly decode the SVM direction for CIFAR-10 (as in Figure 8a).
Airplane
Automobile
Generated Hard Examples
Generated Easy Examples
Deer
Horse
Generated
Hard Examples
Generated
Examples
Figure 26: More images generated by directly decoding the SVM direction.
35

---

Published as a conference paper at ICLR 2023
C.1.5 VALIDATING THE PROXY OF EVALUATING IMAGES CLOSEST THE CAPTIONS
In Figure 7b from the main paper, we used the K closest images the positive negative SVM caption as a proxy for the subpopulation that corresponded to that caption. Here, we validate that the images which are closest in cosine distance to given caption indeed closely match the description of the caption itself. We use the CIFAR CIFAR-Extendee Extended caption set.
Cord 0.433
Coall & 231
Coalt 9033
Cord 0.396
Card DJ/46
Cord CJN0
Caal 9
Coal 9.200
Cast
(a) Caption "a photo of red bomber"
Cont 8.321
Cret 8.317
Cent 0:36
Cont 0.300
Cand 0:304
Cont C.00
Cent 303
Cent 0.300
(b) Caption: "a photo of a black jet white background"
Gast
Cala
SWilScore 2.310
SVM Scane 2017
SUN
SVM 0.208
SWM Soowe 0,0ce
SVW
(c) "a photo of white mouser on the grass"
SWore 8,303
SWM Scaec 313
2133
0.33
SVM Soore 0.333
SVR Scur 8,310
SUM Scanc 307
8:27
SuN Scaa: 2017
SUM Scamc 2030
SVM Score 0.206
SUM Scure 0,304
SUM Score 0.304
Sue Scae- 3355
(d) "a photo brown domestic cat inside"
Figure 27: The images closest in cosine distance to the given CLIP caption.
36

---

Published as a conference paper at ICLR 2023
C.1.6 FILTERING INTERVENTION FOR CIFAR-10
Here, we perform the filtering intervention for CIFAR-IO. For each class, take the top 100 examples according to our SVM decision values, the model's base confidence, or random baseline, and then add those examples use the proxy of evaluating the images closest to the extracted negative SVM captions (here from We find that our framework improves the performance on the minority subgroups to larger extent than other methods.
the training dataset (Figure 28). Without annotations, we again
Subpopulation Accuracy
0.75
0.70
Order
0.65
Decision Value
Confidence
Random
0.60
Base Model
25
125 150
Top K Test Images by Caption
Figure 28: For each CIFAR-10 class, the accuracy of the K examples closest to the negative SVM caption after adding 100 images from the extra data (and retraining) either random, based on the SVM decision values, or based on the model confidences. Choosing these 100 images by using the SVM decision value best improves the accuracy population described by the negative caption. Results are averaged over classes and reported over 10 runs.
37

---

Published as a conference paper at ICLR 2023
C.1.7 RESULTS FOR FULL CIFAR-ID 10 DATASET
Here show the results of applying our framework the full CIFAR-IO dataset. We therefore use the entire training split, except for 20% used for validation. We find that similar failure modes are captured (Figure 29), and the identified negative subpopulations have a lower accuracy than the identified positive subpopulations (Figure 30).
Cat
Dog
PA POT CAR1
FIM CONE CAA
CAN JOTE 2214
RAM SOPE 100
SIM SaaE L0112 SAN TIS
Hard Exemplars
Caption:
grass"
photo
One D0C
aL3 Coe'1.000
Cwn 3
Ownnt 100
Suaes 1600
Vee Cart 303
Oren die
Easy Exemplars
SVM Caption: photo of domestic cot"
SVM Caption: photo of tbbetan tereer"
Figure 29: Most extreme images/captions surfaced by our framework for the full CIFAR-10 dataset
Accuraty
0.92
0.90
0.88
Order
Positive
0.86
Negative
Top K RRstt Images by Caption
Figure 30: Averaging over 10 classes, we evaluate the accuracy of the top K images closest in cosine distance to the positive or negative caption on the test set.
38

---

Published as a conference paper at ICLR 2023
C.2 IMAGENET
We discuss the experimental details for the ImageNet dataset.
Dataset. We consider 40% of the original dataset training data, 40% data, and the last 20% as the validation set. We perform training and evaluation at a resolution of x24.
Hyperparameters We the following hyperparameters, which taken from the recom- mended FFCV configuration:
Parameter
Value
Batch Size
1024
Epochs
16
Peak LR
0.5
Momentum
0.9
Weight Decay
5 x 10-4
Peak Epoch
2
Caption generation We again generate the captions programmatically using the caption "a photo of <adjective> <noun> <prepositional phrase>." We use the following adjectives: [None, 'white", 'blue', 'red', "green', 'black', 'elllow,, 'orange', 'brown', "group of", "close-up', 'blurry', 'far away')
Since there are 1000 ImageNet classes, we need way group classes together to more scalably assign prepositional phrases that makes sense (for example, the sentence purple lawn-mower that is flying in the air" is likely out of distribution for the CLIP encoder.) We consider set of ancestor nodes the WordNet hierarchy, and assign each ImageNet class to closest ancestor the set. The set of ancestors and the number of ImageNet classes associated to each is below.
('fish': 16, 'bird': 59, 'amphibian': 9, "reptile": 36, 'invertebrate": 61, 'mammal': 30,
"marsupial": 3, 'aquatic mammal': 4, 'canine": 130, 'feline': 13, 'rodent': 6, 'swine":
'bovid': 9, 'primate': 20, 'device": 124, 'entity": : 59, 'vehicle': 7, ''airccrat': 4, 'structure':
57, 'wheeled vehicle': 40, 'container': 32, "equipment': 37, 'implement': 36, 'covering':
43, 'funiitre': 21, 'vessel': 32, 'fabric': 6, 'train': 1, 'instrumentality:: 12, 'appliance': 12.
'bus': 3, 'food': 38, 'fruit': 16, 'geological formation': 9, 'person': 3, 'flower': 2, 'fungus": 7
We then use the following set of prepositions for each class. Specifically, all classes use the set of prepositions [None, "outside", 'inside', ; 'on black background', 'on white background', 'on green background', 'on a blue background', "on a brown background") Furthermore, depending on their assigned ancestor class, they use the following class specifie prepositions
'reptile', 'amphibian': ['in tank', 'on
'on the ground', 'on rock', 'in the grass"]
"marsupial', 'swine', 'bovid', 'feline', 'canine', 'rodent': ('in the grass', 'in a house', 'in the forest', 'on the ground", 'with a person']
fuuguu': ['on l'on the ground', 'in the grass']
food': ['on plate', 'one the ground', 'on a table', 'with a person')
geological formation', 'train', 'mammal', 'structure', 'entity':
wheeledd vehicle', 'bus', 'vehicle: I'on the road', 'parked']
'ruiit, 'fruit' 'flower': ['on table', 'on the ground', 'on tree', 'with person', 'in the grass']
"iihh, 'aquatic mammal': ['in a tank', 'with a person', 'underwater']
personn: ['in house", "on field']
'equipment" "instrumentality'. 'appliance', 'container', 'fabric', 'covering", 'device', 'imple- ment: I'on a table', "with a person', 'with a hand'] "rimatee" ['in a tree", 'on the ground', 'in the grass']
innertebratee: I' the grass', 'on the ground', 'in house']
39

---

Published as a conference paper at ICLR 2023
'bidd' ['in the air', 'on the ground', 'in cage', 'in the grass', 'flying', 'perched') ]
aireraft': ['in the air', 'on the ground']
furniture': ["in house"]
vvessll: ['in the water', "docked', 'in the ocean']
Finally, the ImageNet class names themselves can include quite niche words (which may not be well-represented by CLIP). We thus use as the noun the ancestor corresponding to the ImageNet class. As reference caption, we "A photo of <ancestor>".
40

---

Published as a conference paper at ICLR 2023
C.2.1 FURTHER IMAGENET EXAMPLES
In Figure 31 we show additional results for Imagenet classes cauliflower and agama.
Cauliflower
Agama
SAM Saae 3.12
SieeSoeee CLICE
DAM Staee 340
Sel Srar
SV Steee 2.06
The Sae 100
2318
Hard Exemplars
SVM Caption: photo ofa green food with
SVM Caption:
photo of blurry reptile white birkground
SeMdooe 1004
SVM Sooee u33
Fal Scoe 1211
SW 200 Orel 2i9
Sua 2019 Dxall
DVV 1,43
5M Soen cone
Easy Exemplars
fond
Figure 31: Most extreme images/captions surfaced by our framework for ImageNet classes cauliflower and Agama
C.2.2 IMAGENET AND CIFAR-10 NORMALIZED PLOTS
Since ImageNet and CIFAR-10 10 have different test set sizes (50 examples and 1000 examples per
class respectively). here we plot equivalents Figures and 10b. In Figure 32, instead of plotting the top K images considered on the x-axis, we plot the fraction of the test set considered (from to 0% 100% the test set). On the right-hand side of each plot, when this fraction 1, we consider the entire test set (thus, the easy and hard subpopulations both have accuracy equal to the overall test accuracy).
0.60
Accuracy
0.80
Accunacy
0.75
a.50
0.70
Pos
Postive
Nopel
Negative
0.05
Orcral Class
Overal Cass
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Fraction of Top K Test images by Caption
Fraction Top Test Images by Caption
(a) CIFAR-10 10
(b) ImageNet
Figure 32: For CIFAR-10 and ImageNet, we plot the accuracy of the K closest images to the positive or the negative caption for each class averaged over classes. On the x-axis, we plot the fraction of the test considered.
41

---

Published as a conference paper at ICLR 2023
C.2.3 DO FAILURE DIRECTIONS GENERALIZE TO OTHER ARCHITECTURES?
In this section, we perform our method on second independent run of ResNet18, as well as on a ResNet50. In Figure 33, we then plot the accuracies of the ImageNet test images that are closest the extracted positive and negative captions. We find that our method behaves similarly on these models as in Figure 10b.
0.60
0.65
0.55
Accuracy
Accuracy
0.60
0.50
Order
0.55
Order
Positive
Positive
0.45
Negative
Negative
Class
10
20
30
40
50
10
20
30
40
50
Top Test Images by Caption
Top K Test Images by Caption
ResNet18 (independent run)
ResNet50
Figure 33: We run method on (a) an independent run of a ResNet18 and (b) ResNet50. As in Figure 10b, we plot the accuracy of the top K images closest to the positive and negative captions. The identified hard subpopulation has lower accuracy than the identified easy subpopulation.
In Figure 34, we find that few example classes, the failure modes extracted by our method for ResNet50 are similar to those extracted by the ResNetl (see Figures 10a and 31).
Tench
Red Wolf
Hard Exempiars
SVM Caption: photo of group of fist"
SVM Caption:" "a photo of white canine with person"
Easy Exemplars
SVM Caption:" "a photo a orange fish with person"
SVM Caption: "a photo of
orange canine in house"
Cauliflower
Agama
Exemplars
SVM Caption: "a photo of blurry perso""
SVM Caption: "A photo of green reptlle
Easy Exemplars
SVM Caption: photo of yellow food on the ground"
SVM Caption: photo of blue reptlle on the ground"
Figure 34: We display the most extreme examples and SVM captions when running our method for ResNet50. We find that the failure directions very similar to those extracted for ResNet18.
Do the failure directions that extracted using the ResNet18 also capture failure modes for other architectures? To answer this question, we measure the subpopulation accuracies of each these models (as well as a pre-trained ViT-B) when using the ResNet18 positive/negative SVM captions to define our subpopulations interest (Figure 35). We find that these ResNet18 captions perform almost well the SVM captions extracted for the specific underlying model, and define clear
42

---

Published as a conference paper at ICLR 2023
"easy" and "hard" subpopulations. Thus, the directions extracted for the ResNet18 also represent real failure modes for other architectures.
0.50
0.82
Actuney
0.60
0.80
0.50
0.78
Megative Rechir11
0.50
Megafive Reskirt
Resbaell
20 30
40
50
40
50
10
20
Top K Test images Caption
Top K Test images by Caption
Top Tost Images by Caption
(a) ResNet18 (independent run)
(b) ResNet50
(c) VIT-B
Figure 35: We display the accuracies for various architectures of the test images closest to the SVM captions extracted from a ResNer18. The directions extracted for the ResNet18 represent real failure modes for other architectures.
C.2.4 RESULTS ON FULL IMAGENET SPLIT
In Figure 36, additionally method ImageNet split of 80% train and 20% validation (without any extra data). We find that our method behaves similarly to Figure 10b.
0.60
Accuracy
0.55
Order
0.50
Positive
Negative
Overall Class
10
20
30
40
50
Top Images by Caption
(a) ResNet18 (independent run)
Figure 36: run our model on an 80% train, 20% validation split ImageNet, and display accuracies of the test images closest to the positive and negative caption.
43

---

Published as a conference paper at ICLR 2023
SA Sar
SAM Seae
SVM Hard Exemplars
Lowest Confidence
WAP Nuve
IM Sure
Sraw CADA
VAP Suer
WM Sare CEE
SVM Easy Exemplars
Highest Confidence
Figure 37: The most extreme examples by SVM decision value and by confidence for the "no effusion" class. Notably, the SVM distills a failure mode that is not easily seen in the low confidence examples.
% Overall that is
AP
38.18%
0.5
Cent conce Bave Pipudriin
Dedisin
Incorrect
12.62%
Flagged
28.11% %
0.4
% Incorrect that is AP
55.351 %
Flagged that is AP
59.826
0.0
Hlggel
The fraction of examples from the "no effusion" class that are taken the AP position. Notably, the base model disproportionately strug- gles on AP images, and the majority of images flagged by our framework AP.
Figure 38:
Figure 39: The fraction the top K examples from "no effusion" class that are AP. We compare using the SVM's decision value and the model's confidences select these images.
C.3 CHESTX-RAY14
We apply our framework the ChestXray-14 14 (Rajpurkar et al., 2017) dataset. This dataset contains frontal X-ray images labeled with 14 different conditions. ChestXray- 14 is multi-label tagging task (i.e., single image can have more than one condition), so we treat each condition as its own binary problem. In this section, we focus the condition Effusion. Results on other conditions can be found in Appendix C.3.
The trained SVM identifies visually distinguishable failure mode directions in latent space. As shown in Figure 37, the representative images flagged by this SVM as most incorrect are blurrier and less bright. Moreover, this trend is not reflected by the least confident images, indicating that our framework isolating a different trend than the one corresponding to ordering the images by base model confidence.
In fact, we find that the SVM may be picking up on the position which the exam was conducted. While the majority of the X-rays Posterior-Anterior (PA) radiographs, little over third Interior-Posteree
(AP). PA radiographs are usually clearer, but require the patient to be well enough stand (Tafti Byerly, 2020). Examples of AP and PA radiographs from the dataset can be found later in this section.
As shown in Table 38, the SVM for the class "no effusion" flags a large number of the AP radiographs as incorrect. This indicates that the model might indeed rely on the position in which the radiograph was taken to predict whether the patient was healthy. Moreover, the SVM selects the AP examples more consistently than ordering the radiographs by the base model's 's confidence (Figure 39).
Finally, the rest of this appendix, we discuss the experimental details and additional results for applying our framework to the ChestX-ray 14 dataset.
C.3.1 EXPERIMENTAL DETAILS
We use the given train, validation, and test splits for the ChestX-ray -ayt dataset. We use a resolution of 224 x 224, and consider each of the 14 conditions separately.
44

---

Published as a conference paper at ICLR 2023
Hyperparameters
We use the following hyperparameters, which are the same as for ImageNet.
Parameter
Value
Batch Size
1024
Epochs
16
Peak LR
0.5
Momentum
0.9
Weight Decay
5 x 10-4
Peak Epoch
2
C.3.2 EXAMPLES OF AP AND PA IMAGES
The ChestX-rayl -ray4 14 Images have different positions in which the exam could have been conducted. While the majority of the X-rays are Posterior-Anterior (PA) radiographs, little over third are Anterior-Posterice (AP).
PA radiographs are usually clearer, and the position the scapula obstructs less the lung; however, they require the patient be well enough stand (Tafti & Byerly, 2020). Examples of AP and PA radiographs from the dataset for the same patient and same resolution can be found in Figure 40. There are often also visible markers on the radiograph (i.e the words AP or portable) which distinguish the two.
(a) Patient I: AP
(b) Patient 1: PA
(c) Patient 2: AP
(d) Patient 2: PA
Figure 40: Examples of AP vs PA radiographs for the same patient.
45

---

Published as a conference paper at ICLR 2023
C.3.3 RESULTS FOR ADDITIONAL CONDITIONS
Effusion We begin with the condition "effusion". Figure 41 displays the most extreme figures by SVM decision value and confidence for each class. We additionally plot, for the top K values ordered by most negative decision value confidence, the fraction of images that were AP radiographs. We find that, to higher degree than confidences, the model associates AP images as hard for the healthy class and easy for the non-healthy class.
SVM Hard Exemplars
Ceter in
SVM Easy Exemplars
MOTTR
Lowest Confidence
2302 5203 7500 10000 12300 15000 17300 20205
lop K.agggo
Highast Confidence
(a) Class: No Effusion
SVM Hard
Exemplars
0.4
Bredlin
0.2
Scl Decison Wine
danos
Confidence
0.1
Highest Confidence
(b) Class: Effusion
Figure 41: Effusion. Left: Most extreme images according to SVM decision value and confidences. Right: The fraction of the top K images that were AP. We order images by the most negative SVM decision value or by the lowest model confidence.
46

---

Published as a conference paper at ICLR 2023
Mass We now consider the condition Mass (Figure 42). We again find that the SVM picks up distinct visual patterns that were not surfaced by confidences. Moreover, we interestingly find that framework surfaces AP images hard for the non non-healthy class, opposed to the healthy class in the case of Effusion.
Fritag 2at
Exemplars
0.4
Exemplars
0.1
0.2
Ducuio Kais
Duell (eel-feeee
Confidence
5000
10000
15000
20000
KThggedd
Highest Confidence
(a) Class: No Mass
SVM Hard Exemplars
1.0
SVM Easy
Cenl pesis
becisin
0.8
0.6
0.4
200
400
600
800
1000
Confidence
(b) Class: Mass
Figure 42: Mass. Left: Most extreme images according SVM decision value and confidences. Right: The fraction of the top K images that were AP. We order images by the most negative SVM decision value or by the lowest model confidence.
47

---

Published as a conference paper at ICLR 2023
Infiltration Finally, consider the condition Infiltration (Figure 43). As in the case of Effusion, the framework associates AP with easier images for the non-healthy class and, to a lesser extent, hard images for the healthy class.
SVM Hard Exemplars
1.0
SVM Easy Exemplars
Fal Decioon Wir
0.8
Confidence
0.4
Tap K.llgged
Confidence
(a) Class: No Infiltration
SVM Hard Exemplars
Exemplars
0.2
5VM Oecitin 1a
Confidence
Confidence
(b) Class: Infiltration
Figure 43: Infiltration. Left: Most extreme images according to SVM decision value and confidences. Right: The fraction of the top K images that were AP. We order images by the most negative SVM decision value or by the lowest model confidence.
48


Reviews:
[{"review_id": "x20tWAkXjtS", "reviewer": "Reviewer_ySph", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an automatic method to look for hard subpopulations and spurious correlations in datasets,  by training SVMs in CLIP space to predict whether the original model will predict the data point correctly. Experiments show their method identified hard examples.\n", "summary_of_the_review": "The writing prevents readers from understanding the paper and the novelty is somewhat limited. \n\n----------------------------------------------\nPost rebuttal comment:\n\nAfter reading the other reviews and the rebuttal. We raise the score to 6.", "strength_and_weaknesses": "Strengths:\nThe paper proposes an automatic method to look for hard subpopulations and spurious correlations in datasets.\nThey performed extensive experiments to show their method identified the hard subpopulations and spurious correlations.\n\nWeaknesses:\nThe captions and the images in result figures do not seem to match well:\nIn figure 1 easy exemplars, the man in the third image has a beard but the caption is \"a photo of a old man who has no beard\" \nIn figure 7 dog easy exemplars, the caption is \"a photo of a white dog on the grass” but the images do not match\nIf the CLIP model is pretrained, based on the observation above, linear SVM seems not to be strong enough to separate the “easy” and “hard” examples for the original model\nThe writing of this paper prevents readers from understanding the paper:\nThe motivation is unclear, whether their ultimate goal is to look for error patterns or to improve the original model is unknown\nThe terms appearing in the abstract, such as “isolating hard subpopulations”, “spurious correlations ”, “distilling a model’s failure modes ”, ”directions within the feature space” …should be clearly defined early on.\nIn the main paper how to generate the captions is not mentioned. Although it appeared in the supplement but at least a brief description should be included in the main paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hA84PaLBDM", "reviewer": "Reviewer_ZiBp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple, yet effective method for identifying hard subpopulations in datasets. In addition, by utilizing the pre-trained CLIP and generative models, the underlying error patterns for identifying hard subpopulations become human-understandable. Based on such interpretations, appropriate data augmentations are applied to hard subpopulations, enhancing the performance of downstream tasks. Finally, the effects of the proposed method are validated for both cases where spurious correlation and the underrepresentation problem exist in the datasets.", "summary_of_the_review": "The proposed method is novel and interesting. In addition, the experimental results are impressive overall. I hope that the authors will address two concerns during the rebuttal period.", "strength_and_weaknesses": "Strength\n1. The proposed method is novel. It is highly impressive 1) to utilize CLIP embeddings to make the proposed method interpretable and 2) to demonstrate the effect of data augmentation for the downstream tasks using pre-trained generative models based on the results of the proposed method.\n\n2. The paper is well-written and easy to follow. Also, the appendix and figures help to understand the proposed method and experimental results easily. Especially, Figures 2 and 7-(b) enhance the reliability of the proposed method.\n\nWeakness\n1. My main concern is about the scalability of the proposed method. I think the experimental settings of Section 3 and Section 4.1 seem to be limited. Moreover, based on the results for ImageNet in Section 4.2 and Figure 27 in the appendix, it seems that the proposed method does not perform well with this dataset. Specifically, I think that the results of Figure 10-(a) and Figure 27 do not match well with the SVM caption. Also, in Figure 10-(b), the gap between the positive and negative samples decreases too quickly, unlike Figure 7-(b). Could the authors provide more evident results for the claim that the proposed methods are scalable?\n\n2. In the Detection part of Section 2.1, the authors claim that the cross-validation score can be used as a measure of the failure model's strength in datasets. However, it seems that there is no experiment to validate its practicality. Could the authors verify the practicality of this score rather than the result of the correlation in Figure 4?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "MyyYZ_K_4s", "reviewer": "Reviewer_ZdLg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an automated way to determine a model's error modes (hard subpopulations and spurious correlations). They do so by using linear classifiers (SVM) on a feature space (CLIP) such that these failure modes are a direction within the feature space (orthogonal to the hyperplane created by the SVM). Given the feature space used, diffusion models can allow for the generation of additional samples to address the failure modes of the model. The authors use the CelebA dataset (which has a spurious correlation between gender and age) and the CIFAR-10 dataset (which has underrepresented subtypes).", "summary_of_the_review": "A thorough a well written paper showing how SVM and CLIP feature space can be used to automatically identify hard subpopulations and spurious correlations. Would like the two weaknesses to be clarified. The technique proposed could provide an additional tool for model understanding and post-hoc improvement.", "strength_and_weaknesses": "Strengths.\n\n- The use of the SVM to create the boundary between the model's correct and incorrect outputs and having an image/caption style model's feature space provides a lot of capabilities to this approach. Captions can be provided to understand what the easiest versus hardest examples are, and the model allows hard and easy examples to be generated.\n- Excellent observation that the SVM does better than using the model confidences to determine hard/easy exemplars.\n\n\nWeakness\n- I worried that the paper was too CLIP limited; would the approach have worked as well with a different feature space representation.\n- In \"Targeted synthetic data augmentation\", the authors only retrained the last layer of the model but gave no reason as to why.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FMlcJh2_Gk", "reviewer": "Reviewer_HQAk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Deep learning models often exhibit consistent error patterns, where these errors often correspond to hard subpopulations in the data they are deployed on. This paper introduces a framework for automatic distillation and surfacing of a model’s error patterns. In detail, this framework uses SVM to predict the error of the original model in the CLIP’s embedding space, where the vector orthogonal to the decision boundary of SVM represents the direction of the failure mode of the original model. In addition, the authors also propose actionable interventions to remedy the failure mode.\n\nThe evaluation is done on hard subpopulations in image datasets such as CIFAR-10, ImageNet, and ChestX-ray14. Notably, the proposed method does not require direct\nhuman intervention or pre-annotated subgroups and is therefore a scalable approach.\n", "summary_of_the_review": "I have a few comments around modeling designs but I think the comments are addressable.", "strength_and_weaknesses": "The proposed method is novel and opens potential for new line of research in identification of biases and difficult subpopulations. I personally find the submission to be clear and well-written. Overall, I am positive and would like to give a \"accept\".\nHowever, I have few more comments that I hope to help improve the paper. Several of them are around the modeling choices and some of them are to clarify my understanding of the practical usefulness of the proposed method.\n1.\tThe framework uses a linear support vector machine (SVM) to separate correct from incorrect examples and claims that the direction of the failure mode will be orthogonal to this decision boundary. My concern is whether the correct and incorrect examples are linearly separable. Can the authors provide data to support this claim? If it is not completely linearly separable, I suggest that the framework can be extended to use the simplest nonlinear classifiers.\n2.\tI find that the SVM will reduce the accuracy of the major subclasses (Table 1 in the Supplementary material). I suggest that the author discuss this phenomenon in the main text.\n3.\tI am concerned that the effectiveness of the proposed method depends on the data distribution of the validation set. My first request is for the authors to provide more experiments to verify the influence of long-tailed classes on the decision boundary of SVM. Second, I would like the authors to report the results with different proportions of dataset partitioning (e.g. 80% for training and 20% for validation).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%