Spuriosity Rankings: Sorting Data to Measure and Mitigate Biases
Authors: Mazda Moayeri
Wenxiao Wang
Sahil Singla
Soheil Feizi
Venue: NeurIPS 2023 spotlight (2023)
Keywords: spurious correlations
interpretability
bias
distributional robustness
TL;DR: Sorting data using activations of an interpretable model enables efficient discovery, measurement, and mitigation of bias caused by spurious correlations.

Abstract: We present a simple but effective method to measure and mitigate model biases caused by reliance on spurious cues. Instead of requiring costly changes to one's data or model training, our method better utilizes the data one already has by sorting them. Specifically, we rank images within their classes based on spuriosity (the degree to which common spurious cues are present), proxied via deep neural features of an interpretable network. With spuriosity rankings, it is easy to identify minority subpopulations (i.e. low spuriosity images) and assess model bias as the gap in accuracy between high and low spuriosity images. One can even efficiently remove a model's bias at little cost to accuracy by finetuning its classification head on low spuriosity images, resulting in fairer treatment of samples regardless of spuriosity. We demonstrate our method on ImageNet, annotating $5000$ class-feature dependencies ($630$ of which we find to be spurious) and generating a dataset of $325k$ soft segmentations for these features along the way. Having computed spuriosity rankings via the identified spurious neural features, we assess biases for $89$ diverse models and find that class-wise biases are highly correlated across models. Our results suggest that model bias due to spurious feature reliance is influenced far more by what the model is trained on than how it is trained.

Spuriosity Rankings:
Sorting Data to Measure and Mitigate Biases
Mazda Moayeri'
Wenxiao Wang'
Sahil 2
Soheil Feizi!
University of Maryland
2 Google
mmoayeri, wwx, sfeizi @umd. edu,
sas sasingla@roogle.cm
Low Spuriosity
Class: Otter
High Spuriosity
Class: Rhinoceros Beetle
Class:
Lighter
Figure 1: Spuriosity rankings sort images based on the presence of spurious cues. The rankings reveal hidden spurious features that models rely on, like rippling water, human hands, and flame in the dark (right; top to bottom), and uncover minority subpopulations where spurious cues are absent (left).
Abstract
We present a simple but effective method to measure and mitigate model biases caused by reliance on spurious cues. Instead of requiring costly changes to one's data or model training, our method better utilizes the data one already has by sorting them. Specifically, we rank images within their classes based on spuriosity (the degree to which common spurious cues are present), proxied via deep neural features of an interpretable network. With spuriosity rankings, minority subpopulations (i.e. low spuriosity images) and assess model bias as the gap accuracy between high and low spuriosity images. One can even efficiently remove model's 's bias at little cost to accuracy by finetuning its classification head low spuriosity images, resulting in fairer treatment of samples regardless of spuriosity. We demonstrate our method ImageNet, annotating 5000 class- feature dependencies (630 of which we find to be spurious) and generating a dataset of 325k soft segmentations for these features along the way. Having computed spuriosity rankings via the identified spurious neural features, we assess biases for 89 diverse models and find that class- biases are highly correlated across models. Our results suggest that model bias due to spurious feature reliance influenced far more by what the model is trained on than how it is trained.
easy to identify
Work carried out while at the University of Maryland.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(1) Train or Fit head for Interpretable Model
(2) Identify Spurious Features with minimal human supervision
(3) Sort Images by Spuricsity to Measure and Mirigate Blases
Complementary visualizaionnseeee neural feature focus Top Image Neural Activation Map Attack
Spariesity prosied by activation
Semontic Gradients
Spuricsity
High Spuriosity
Unorganized Data
the atterciom.
O00
The Pecture for. - it's
Quter lngit
Naural Highly encade the salient potterns.
The water, it's spurious
Measure blas spurkosityy gop (drop socuracy high and
images!
Mitigatee blas by existing classihcation
hwad spuriosity images
Figure? 2: Overview of our framework. Per class, we sort images by spuriosity, which we proxy via the activation of neural features (i.e. nodes in the penultimate layer of trained classifier) that focus on spurious cues. After sorting, measuring model bias to the discovered spurious features is as simple as computing the drop in accuracy across high and low spuriosity images. This bias can be mitigated efficiently by tuning the classification head on low spuriosity images.
1 Introduction
Deep models often exhibit bias, underperforming on certain minority subpopulations 6 10 One cause of bias is the tendency of deep models to rely on spurious features [45] 39 1. as samples that retain spurious correlations are predicted more accurately than rarer samples where spurious correlations are broken. For example, classifier may learn to associate water with the otter class if many training images contain water, which in turn can lead to reduced accuracy on images of otters out of water. Spurious feature reliance also hinders robust generalization (e.g. medical systems trained in one hospital may fail when deployed to others (5 12 potentially posing major reliability risks.
Recent trends suggest that scaling up models and datasets may resolve many existing limitations (50). However, in addition being costly, collecting more data would not alter the long-tail data distribution that leads to spurious feature reliance, as the spurious cues will remain just as correlated, if not more, with class labels. Instead of seeking more data, we suggest to better utilize the data one already has. Specifically, we aim to rank images within their classes by spuriosity, which can be understood informally as the degree to which relevant spurious cues are present. Figure [ shows examples of our rankings. For the otter class, high spuriosity images prominently contain the common spurious feature of water, while low spuriosity images do not, instead displaying the rarer case of otters on land.
Spuriosity, while intuitive, is rather challenging to measure scalably: for each class, one must (i) discover relevant cues (i.e. those that a model will rely upon), (ii) identify the spurious ones, and (iii obtain a function that scores the level to which these spurious cues are present in a given image. To this end, we propose extract spurious concept detectors from within an interpretable model trained on the given data. Namely, we automatically select neural features (i.c. nodes in the penultimate layer) that contribute highly to a class logit, as they encode concepts that are both salient to that class and relied upon by the model. Then, we interpret the concept each neural feature detects, with limited hu- man supervision, so keep the ones activated by spurious cues. Finally, we average the activation on these neural features to yield scalar for each image that proxies the presence of relevant spurious cues (i.e. spuriosity), thus enabling sorting. Once data is ranked by spuriosity, multiple significant benefits emerge immediately: First, low spuriosity images reveal minority subpopulations, where common spurious correlations are broken. Second, we can casily quantify model bias as the drop in accuracy between high and low spuriosity images; call this spurious gap. Third, we can mitigate bias towards high spuriosity images by simply finetuning the classification head on low spuriosity images.
Figure 2 diagrams our approach. The interpretable model we use to detect relevant neural features and extract spurious ones (steps 1 and respectively) is adversarially trained, in (46). However, we show for the first time that we can extend this framework without adversarial training. Namely, to discover patterns salient for a new task, it suffices to merely fit a linear classification head on the new data over fixed adversarially pre-trained neural features. This dramatically increases the efficiency of the method, and expands use cases to low-data regimes where adversarial training is less feasible. We carry out steps and 2 at an unprecedented scale, resulting in 5000 annotated class-neura neural feature

---

dependencies over all of the ubiquitous ImageNet, including 630 spurious features discovered over 357 classes. For these classes, we compute spuriosity rankings (step 3), which, along with a web-UI" -f for easy viewing and 325k soft segmentations of salient visual features in ImageNet, we present as dataset to shed insight on how deep models perform ImageNet classification.
Using spuriosity rankings, we compute spurious gaps (drop in accuracy from high to low spuriosity images) for 89 models. observing that all models underperform on low spuriosity images. Models exhibiting the most and least bias to spurious features are CLIP and adversarially trained models re- spectively, corroborating prior work [28 37 and thus further validating our rankings. We also observe diverse models to have highly correlated class-wise spurious gaps. This implies that models absorb the same biases, suggesting that bias is inftuenced far less by how a model is trained, and far more by what it is trained on. Thus, frameworks like ours that demystify large datasets can be instrumental in understanding and mitigating bias. Indeed, we show that finetuning existing classification heads on low spuriosity images is an efficient and effective model-agnostic way to close spurious gaps at little cost of validation accuracy, resulting in fairer, more stable treatment of inputs, regardless of spuriosity.
We also observe surprising instances of negative spurious gaps, i.c. classes where models perform worse when spurious cues are present. These cases are due spurious feature collision, where one class' spurious feature is also correlated more strongly with another class. Existing benchmarks [23] lack the scale of ours reveal this atypical yet harmful consequence of spurious feature reliance. Closer inspection reveals high spuriosity images for classes with negative spurious gaps are often mislabeled or contain multiple objects. Thus, spuriosity rankings can help in flagging label noise, and in some cases (via cropping to isolate regions activating core neural features), resolving it.
In summary, we present simple, extensible method for (i) discovering spurious features (ii) measur- ing and (iii) mitigating model biases caused by these spurious features. Centering on the fact that spurious correlations are determined by data, our solution focuses on better understanding and using the data one already has, in contrast to data-agnostic approaches which focus on altering model train- ing 2 41. Arguably, our method provides more practical robustness, as we first interpret potential biases a model trained on the given data is likely to suffer, and then devise solutions specifically tailored to these biases. By shifting focus from model training to data understanding, spuriosity rankings allow for efficient and interpretable robustness interventions, towards more reliable AL.
2 Review of Literature
Spurious correlation benchmarks typically consider a small set of predefined spurious attributes fixed across classes (55] 42 (25), and real-world benchmarks tend to be very specific to narrow task (23 In both cases, the fixed spurious attributes are annotated manually. Similarly, many have proposed variants of ImageNet's 's validation set, collecting new data (e.g. with altered renditions, as sketches, in household settings and odd poses, etc [6 49 3) and inspecting the drop in accuracy com- pared to the original validation set. In contrast, our framework provides general method to uncover and automatically annotate (i.e. per sample) the presence of spurious features relevant to any task given its training data, as well as measure and mitigate model bias caused by these spurious features. Since bias arises from data, we believe our step of first interpreting the correlations within a dataset, in- stead of simply choosing spurious features of interest apriori, is paramount in order to build practical robustness (i.c. to distribution shifts that are naturally occurring and potentially harmful, since they break spurious correlations model trained on the given data likely uses). Also, we account for the fact that whether or not a feature is spurious depends crucially on its class: e.g. while the water feature is spurious for the otter class, is not spurious for a class like lakeshore (also, see figure 3. Most bench- marks overlook this, defining spurious features tass-agnost,cally. Other approaches to benchmark spurious feature reliance require synthetic data 2 54 [3] or corrupting spurious regions, which unde- sirably takes models out of distribution 29 30 Instead, our benchmark operates only on natural im- ages one already has, using spuriosity to curate subsets with spurious cues strongly present and absent.
Natural image-based interpretability consists of explaining inferences using examples from the corpus 88 [ or generated examples intended visually reflect traversal between concepts (15) Similarly, our rankings traverse from strong spuriosity to minority examples with spurious cues absent, though we do not require generative models. Other related approaches are influence functions 22
and data models [19) though they are computationally expensive and the former can be fragile 4
welienntimaggenn es. und. edu
3

---

Feature 595 is core for the class Daisy
Feature 595 is spurious for the class Monarch Butterfly
Top Activating Image Neural Activation Map
Feature Attack
Top Activating Image Neural Activation Map
Feature Attack
Figure 3: Examples of the visualizations we use to identify the focus of robust neural feature. Here, feature 595 focuses on flowers, making it core for Daisy and spurious for Monarch Butterfly.
Mitigating spurious correlation reliance often involves altering the training objective to learn more
invariant features 2 (8] or directly optimize worst group accuracy 41
40
58 48) Most methods
require extra supervision with respect to non-class- attributes [3 Retraining approaches either
focus correcting enrors 224
[59 32 using balanced data (i.e. dominant spurious cues balanced
with minority samples) 220; our rankings would directly assist in constructing such balanced subsets.
We more directly compare error-correcting approaches to our method in Appendix F
Neural features have been studied to interpret models [35 17. Notably, adversarially robust neural features have enhanced interpretability [47] [2]. Singla and Feizi [46] introduced framework that uses robust neural features to discover core and spurious data patterns with minimal human supervision. We apply this framework to scalably compute spuriosity rankings. In the next section, explain this framework and our novel contributions to it, including how to use the framework without requiring its most prohibitive step of adversarial training, thus dramatically increasing its use cases.
3 Discovering Spurious Features in ImageNet and Beyond
key component of our framework for interpreting and improving a model's robustness to spurious correlations is that we first discover relevant spurious features based on the training data, as opposed to data-agnostic approaches. Specifically, we leverage the feature discovery method of [46], performing it an unprecedented scale (i.e. across all of ImageNet). We now provide brief overview of the method, details on our expansion, and novel extension of the method with significant impacts. For complete details on feature discovery and our human studies, we refer readers Appendix
3.1 5000 Reasons Deep Models Use to Perform ImageNet Classification
Despite its ubiquity, ImageNet (and any other large-scale dataset) is opaque in the sense that human cannot anticipate the patterns a model trained on it will associate with each class. The reason for this is simple: humans cannot process million (or even 1000) images once. Moreover, the patterns a human may use will not necessarily align with those model will use [4]. Nonetheless, understanding the features (especially the spurious ones) that model will rely upon instrumental in anticipating and mitigating the biases model will suffer from.
To understand the features any general model may rely upon, we inspect the neural features of a single, interpretable model; namely, an adversarially trained one. Adversarial training leads to perceptually aligned gradients [.]. which greatly improve the utility of gradient-based interpretations. Specifically, using the gradient of a neural feature's activation w.r.t. the input image, one can reliably generate a heatmap highlighting the input regions activating the neural feature, and even perturb the input image to visually amplify the cue that the feature detects; the latter method is called a feature attack. Thus, given a robust neural feature, human can annotate its function as core or spurious for a class by in- specting the images within the class that activate the feature most (we use top 5), along with heatmaps and feature attacks for those images (see Figure 3) Note that given class, one can automatically se- lect important neural features based on the average contribution of the feature to the class logit, which can easily be computed by inspecting feature activations and linear classification head weights. These steps make up the feature discovery framework of [46]: to summarize, (i) adversarially train model, (ii) automatically select important neural features per class, and (iii) use complementary visualization techniques to annotate neural feature core or spurious with minimal human supervision.

---

(a) Background bias in Waterbird vs. Landbird Classifi- (b) Racial bias in Celeb-A Blond vs. Brown Hair Clas- cation; the focus should be on the bird, not background. sification; the focus should be on the hair, not skin.
Figure 4: With only linear layer retraining, we detect designated spurious attributes like background in the Waterbirds benchmark (left), as well as new spurious dependencies such as racial bias in Celeb-A Hair Classification (right). Presented above are highest activating images (top row), corresponding heatmaps (middle), and lowest activating images (bottom) for spurious features observed for the classes waterbird, landbird, blond hair, and brown hair (left to right).
Singla and Feizi 46 annotated the 5 most relevant neural features for 232 classes of ImageNet. Through a large-scale human study (details in Appendix [. we expand the analysis to all 1000 classes, resulting in 5000 annotated class-feature pairs, of which 630 are spurious over 357 classes. We host a web-Ul to view all 5000 pairs, offering direct visual look into the patterns a neural network sees and uses across ImageNet. We also verify that heatmaps for annotated features localize the same cue in images whose activation on the feature is in the top 20th percentile for the class, successfully validating 95.3% of annotated class-feature pairs (see appendix for details). Thus, we generate 325, 000 feature soft segmentations across ImageNet as a bonus. Crucially, the validation confirms that sorting by feature activation is effective in gathering instances where a feature is present.
While we use the feature discovery method of [46] directly, we make a number of impactful contribu- tions atop it. Namely, we expand its use cases significantly by removing the requirement of adversarial training. Also, we overhaul the original procedure for assessing model reliance on spurious features, making it far more efficient, less biased, and more stable (Appendix (I4). Key the use of lowest activating images as natural counterfactuals to better interpret spurious features and measure their effects, enabling new cross-class and cross-model analyses. Lowest activating images (never used in (46]) are also crucial for computing and closing spurious gaps (Sections 4.2 and 4.3)
improvements is
3.2 Spurious Feature Discovery without Adversarial Training
Adversarially robust neural features are at the heart of our framework, though adversarial training can be challenging, particularly when data is limited. We propose to simply fit linear layer for new task atop fixed features from an adversarially robust feature encoder pretrained on ImageNet; prior work shows transfer learning on robust features is very effecttvee 43 We then perform the usual process of identifying important robust neural features per class and annotating them as core or spurious.
We demonstrate this light- extension of the feature discovery framework of [46] on two spurious correlation benchmarks: Waterbirds, and Celeb-A Hair Color classification. Figure 4 showws some of the discovered cues. For Waterbirds, we find neural features that focus entirely on backgrounds (the intended spurious feature for this benchmark). The lowest-ranked images within a class for these features contain the opposite background (land instead of water), thus revealing the minority subgroup (waterbirds on land). For Celeb-A, we observe features that spuriously focus on faces (instead of hair). Further, race appears to be homogeneous amongst the top-ranked images, and opposite in the bottom-ranker images. Thus, a model may spuriously associate brown skin with the brown hair class, likely leading to failures for Mond-haired to uncover this racial bias in the Celeb-A benchmark, whose intended spurious feature was gender.
people. To our knowledge, we are the first
To showcase the ease-of-ue saee-ofuuee use of our method and the wide-reaching nature of the spurious correlation problem, we apply the lightweight feature discovery framework to a non-spurious correlation bench- mark, finding numerous spurious cues. We take UTKFaace (60) dataset of face images, and focus on the tasks of gender and race classification (details in Appendix E Figure visualizes three detected spurious features: suit and tie for the male class, the color pink for the female class, and glasses
5

---

Class: Male, Spurious Cue: Suit & Tie Class: Female, Spuricus Cue: Piik (calor)
Class: Indian, Spurious Cue: Glasses
ABEKL
Figure 5: Our lightweight extension of [46] also reveals spurious features in non-spurious correlation benchmarks, such as UTKFaae gender and race classification. Spurious cues are absent in the lowest activating images (bottom row). Thus, sorting by feature activation uncovers minority subpopulations.
for the Indian class. Thus, using robust neural features to discover spurious features effective. even when the neural features were not trained on the data of interest; simply fitting a linear layer (which can be done in minutes) over these fixed features sufficed in revealing spurious features. The spurious features are prominently displayed in the top activating images, highlighted in the heatmaps, and notably absent in the least activating images, indicating that sorting images based on robust neural feature activations can reveal minority subpopulations where spurious correlations are broken. These low spuriosity images prove to be instrumental in measuring and mitigating biases (Section
3.3 Is a Human in the Loop Necessary?
Our framework involves a human in the loop, who is given (i) concise insight to the cues model trained on the given data may rely upon and (ii) agency to decide which of these cues model performance should be invariant to. We believe this increased transparency fosters greater trust with practitioner. Further, the human involvement is limited to viewing a handful of images per class, making our framework tractable (as we demonstrate by performing on the 1000 class ImageNet dataset). Nonetheless, in cases where minimizing costs is preferred, the human in the loop can be removed, either by automating the feature interpretation step, or by using directly embed concepts into representation space from text descriptions
models to [37] we describe these variations to our framework in Appendix G The key idea of our work is to leverage the interpretability of certain existing models to uncover vectors in representation space corresponding to spurious cues, with which we can scalably sort data by computing similarity to these vectors. The method we present uses axis axis-aligned vectors of adversarially trained network (i.e. robust neural features), though our framework is not restricted to this instance. We now show how sorting enables for greater utilization of the data one already has, towards resolving the biases caused by relying on spurious correlations.
Measuring and Mitigating Biases with Spuriosity Rankings
4.1 Spuriosity Rankings: Organizing Image Data via Robust Neural Features
Equipped with method to identify robust neural features that detect relevant spurious cues, we now utilize these features to scalably quantify spuriosity (i.e. how strongly spurious cues are present in an image). Letting r. (z) denote the activation of image x on the 1t robust neural feature, and S(c) denote the set of neural features annotated as spurious for class c, we compute spuriosity as follows:
The Spuriosity of an image x for class c, with Pio and die denoting the mean and standard deviation of activations on feature i over class c, is approximated as [St)) Eieste) r:(x)- -usc
dic
Essentially, we use the activation of robust neural features as spurious concept detectors to proxy the degree to which relevant spurious cues are present each image of a dataset efficiently; that is, with only a single forward pass of the dataset through the adversarially trained network. Given a class, averaging the distribution-adjuuteed activations of relevant spurious neural features yields a single scalar value for each image, with which images can be ranked within their classes.

---

Sap VITu
1.0
Accuracy, Spurious Cuss Absant
0.8
CLIP Fineuued
Done Cache
Average Spurious Gaps
CUI mie Zou-5
Trained
CLIP Foedueed
0.7
0.6
0.0
0.5
-0.5
0.1
Frcar
0:75 0.80
Accuraty, Spurious Cues Present
Per Class Per Model
Figure 6: All models are biased: accuracy is 10 to 30% lower when spurious cues are absent compared when they are present (left). However, bias varies far more significantly across classes than across models (right), emphasizing that bias due to relying on spurious cues is crucially dles-deeendent.
Note that we define the spuriosity of an image for specific class, since whether a neural feature is relevant or spurious depends on what class it is being used for: figure Souree3shee shows an example feature (flowers) that is core for one class (Daisy) and spurious for another (Monarch Butterfty). Accounting for the crucial class-dependence of the spurious correlation problem is unique our framework.
4.2 Measuring Bias: Computing Spurious Gaps
Relying spurious cues can bias a model, as its performance may degrade when spurious correlations are broken. Measuring this bias can be expensive, as it requires collecting new data where spurious cues are absent. With spuriosity rankings, we instead can easily select such subset from one' own data. Indeed, high spuriosity images prominently display spurious cues, while low spuriosity images do not (Figure Thus, natural metric enabled by spuriosity rankings is spurious gap, defined as the drop in accuracy between the top-k highest and lowest spuriosity validation images per class. We evaluate spurious gaps over the 357 ImageNet classes with at least one spurious feature discovered.
Figure shows accuracy on the 10 lowest (y-axis) vs. highest (æ-axis) spuriosity images for 89 diverse models pretrained on ImageNet (details in appendix). We observe all models to be biased, suffering lower accuracy when spurious cues are absent. As in (27]. we observe a strong correlation between the two accuracy measures. However, some models stray from this linear trend, exhibiting 'ef- fective robustness' by performing significantly better on images with spurious cues absent than the lin- ear trend predicts (e.g. zero-shot CLIP) or the opposite (e.g. adversarially trained models). Prior work on distributional robustness corroborates our findings on the effective robustness (and lack thereof) of CLIP and adversarially trained models respectively [37 28 thus validating our metric. Essentially, spuriosity rankings organize our data so that we can simulate distribution shifts caused by break- ing spurious correlations, to assess model's robustness without needing to collect new data.
We now turn our attention to class-wine spurious gaps (i.e. we average spurious gaps per class over all models, instead of averaging over all classes per model, as done in Figure left). We find that spurious gap varies significantly more across classes than across models (Figure 6 right). Specifically, the sample variance is 3.1 larger for spurious gaps across classes than across models, suggesting that the spurious correlation problem pertains more the data model is trained on than to the specifics of the model itself. That is, argue that class wisee spurious gaps vary much because each class has its own set of spurious cues that are correlated with the class label to different degrees. In contrast, despite differences in training procedure and architecture, all models learn from the same data, and thus, absorb the same biases. To investigate this claim, for each pair of models our study, we compute the correlation between class-wise spurious gaps. On average, observe high Pearson's 's of 0.69.
Figure 7 visualizes this strong correlation for five diverse models compared to the interpretable model used to discover spurious features in Section 3 We highlight this case as it justifies our approach in using a single model (namely, an &g adversarially ly trained ResNet50) to reveal biases for any general
"exeept for the CLIP models, which are trained more data, and also, perhaps not surprisingly. happen to be the most exceptional in their robustness.
7

---

0.63
---0.51
Sin LR ResNecti Spuriouz Gaps
7-
ts AT Nec50 Gags
0.88
Moto (S) Spuriouu Gaps
ResNee50 Spuricus Gapa
Dekt Spurious
0.30
-0.25
-0.30
6.5
1.
65
a
65
05 on 45
05
60 0.5
0.5
8.0
0.5
AT ResNett0 Gaps
le AT ReaNet50 Gaps
4g AT ResNet50 Gaps
6g AT ResNcctO Gaps
AT ResNett0 Gaps
Figure 7: Class-wise spurious gaps are strongly correlated across models. While spuriosity rankings are empirically computed with the help of a single & adversarially trained (AT) ResNet, diverse models show similar sensitivities to the detected spurious features.
Tunes Handom Data
Turtd Low Spunosity Uata
RosNee50
<-400
SimCLR
Spurious Gap
20
Moca VIT-5
ResNet50
Accuracy
0:25
0.00
(5)
ResNet50
Deit
Moco
(Smoothed)
SinCLR
Low Spuriosity
Tuned
AT RedNet50
Randomly Tuned
-4.0
Orginal
0.96
(No Tuning]
0.30
ImageNet Accuracy
Spuriosity
Rank
Spuriosity
Rank
Figure 8: (Left) Finetuning on low spuriosity images closes reduces spurious gap to below 5% at little cost of validation accuracy. (Right) Closing spurious gap removes bias images with high spurious ranks; that is, regardless of spuriosity, the model performs with roughly the same accuracy.
model. Because the spurious correlations a model learns depends far more on what is trained on (i.e. the data) than how it is trained, we can generalize the spurious features we discover with one model to others. The results of our large-scale empirical analysis support this claim, as models display bias to the spurious cues discovered with our interpretable model, and furthermore, the degree to which these models are biased per-class are strongly correlated. While is not surprising that data determines the spurious features a model learns to rely methods for spurious correlation robustness are data-agnostic. In contrast, our method is data-centric. Namely, we first discover the spurious features models may rely upon based on the data, via an interpretable (essentially, a surrogate) model trained or fit on the data. Then, with spuriosity rankings, organize data allow for efficient measurement of biases per class. We now show how wisely selecting data, via our rankings, for further tuning can mitigate model bias caused by spurious features.
stress this point, many existing
4.3 Mitigating Bias: Closing Spurious Gaps
Having identified a bias across Ietttanetrr models, where images that lack spurious cues common their class are predicted with considerably lower accuracy, we now present a very simple and efficient manner to mitigate this bias at little cost of validation accuracy. In general, to improve some pretrained models on more specific downstream data distribution, one simply finetunes the model on new data from the target distribution. With spuriosity rankings, we remove the need to acquire new data, as we instead simply select from the training data we already have. In this case, we seek to improve the performance of models on images with spurious cues absent; this is precisely the low spuriasity images. Applying our rankings to training data, we can extract small subset that captures minority subpopulations under-represented in the overall training distribution. Then, we tune the existing classification head on this small subset so to recalibrate the model towards reduced reliance on spurious features and fairer treatment of samples, regardless of spuriosity.

---

High Spuriosity Images for Classes with Negative Spurious Caps
Core Feature Soft Segmentations (via NAMs)
Car Weed
Class Label Computer Mouse
Core Cropped Images
Spider Web
Figure 9: (Left) Training images with highest spuriosity for the classes with the most negative spurious gaps. These images often contain multiple objects, at times more prominently showing an object from a class different than the image's 's label (e.g. car, monitor or keyboard, spider). (Right) Using neural activation maps for neural features annotated core (i.e. not spurious) for the class may serve as an automated way to crop out the conflicting objects.
We choose to only modify the classification head, as prior work suggests deep models already learn features that are sufficiently informative to correctly predict minority subpopulations [20). Also, this way, we minimally change the existing model, leading to better retention validation accuracy, and extremely fast bias mitigation, a small fraction of the entire dataset. Specifically, we tune the classification heads of five models spanning diverse architectures (ResNet and ViT) and training procedures (supervised, sulf-supervised, adversarial) on the 100 lowest spuriosity images for each of the 357 classes for which we discover one spurious feature. This tuning occurs in minutes on a single GPU, especially since the data passes through the entire model only once; after caching features, all computation is limited to the linear classification head. Also, we employ early stopping. halting tuning once spurious gap drops below 5%, so to avoid overfitting to the minority subpopulations at the cost of overall accuracy.
only optimize a tiny fraction of the total model parameters using
Figure 8 visualizes the results of our bias mitigation, plotting the spurious gap (over all 357 classes) vs. validation accuracy before and after our tuning. We also include as a baseline the results for tuning subset consisting of 100 randomly selected images from each of the same 357 classes. We find that low spuriosity tuning mitigates bias at a minimal cost to validation accuracy, while the baseline does not. Namely, low spuriosity tuning closes spurious gaps, reducing it by 10 to 20%, while only sacrificing to 3% validation accuracy. In the subplot on the right, we see how low spuriosity tuning effectively removes bias to high spuriosity images. While the randomly tuned baseline (like the original model) has far better accuracy on images with high spuriosity ranks, the low spuriosity tuned model has far more stable treatment of all samples, evidenced by a much flatter curve when plotting accuracy vs. spuriosity rank. In appendix FF we consider a second baseline where we tune on samples misclassified by the original model, inspired by existing spurious correlation mitigation techniques (24] 59 32). Unlike low-spuriy tuming, this baseline also fails to close spurious gaps, and actually results in a more substantial drop overall validation accuracy.
5 Additional Application: Flagging and Fixing Labeling Errors
We now demonstrate an additional manner in which spuriosity rankings can help manage and improve the data one already has. Modern datasets, like models, are opaque in the sense that it can be challenging to answer simple yet significant questions about them, largely due to their ever-increasing size. One such question is what spurious features a model will learn to rely upon, and to what degree, after training on a given dataset; our spuriosity rankings framework is designed to answer this question. A related question is how accurate the labels in dataset are. Even in some of the most widely used vision datasets like ImageNet, label noise has been observed to be pervasive [3], and when is resolved, models trained on the refined data experience improved accuracy and robustness (56).
common culprit for label noise in ImageNet the presence of multiple objects in the same image. Similarly, co-occurring objects are amongst the most common types of spurious cues observe

---

models to rely upon. For example, car wheels almost always co-occur with a car, and so, model may associate the metallic body of car with the car wheel class, Indeed, we discover this spurious feature in our analysis. However, while we typically expect model to underperform when spurious cue is absent, we observe the opposite: the car wheel class has a negative spurious gap, with models on average having 54.2% lower accuracy on high spuriosity car wheel images than low spuriosity ones. While uncommon, negative spurious gaps are still prevalent, occurring in 15.7% of classes we study. This highlights second, often overlooked, harm incurred by spurious feature reliance. Namely, since spurious features are not essential to class, they can be correlated with multiple classes once, leading to spurious feature collision. Indeed, a spurious feature for class may also be core for another, like the car body being spurious for the car wheel class and core for any of the numerous car classes. This is actually the case for majority of the spurious features we discover, with a staggering 63.8% of them also being core for a different class. these instances, if a feature is correlated more with class ei than c2, high spuriosity images from 22 may be misclassified to c1-
Taking a closer look at classes with severely negative spurious gaps reveals instances of mislaheled training data. Figure | shows high spuriosity images for classes with the most negative spurious gaps. These images often more prominently display co-occurring objects different from the class label (such as a car, monitor, or spider) which themselves pertain to another class. Thus, one can automatically flag potential instances of label noise by inspecting high spuriosity images for classes with negative spurious gaps. To quantitatively validate this claim, we leverage ImageNet ReaL labels [3]. which indicate for each ImageNet validation image whether objects from multiple classes are present. ReaL labels show 14% of validation images contain multiple objects. We find that for classes with the 5 most negative spurious gaps (averaged over our model suite), high spuriosity (i.e. ranked in top 10% of spuriosity) validation images contain multiple objects in 80% of cases. In contrast, low (bottom 10%) spuriosity images from the same classes contain multiple objects in only 8% of cases, indicating that the label noise is likely a result of the hypothesized spurious feature collision. Similarly, for classes who's 's spurious gap is less than - 20%, ReaL labels reveal 42% of the high spuriosity images to contain multiple objects, many times higher than the average rate of label noise.
Taking label-noise flagging a step further, in some cases, we can utilize the neural activation maps of core robust neural features for these classes to automatically refine the mislabeled high spuriosity images. Namely, we can crop the image based on the region that activates core features for the class, to focus on the actual object of interest and remove the spurious co-occurring one (details in Appendix H). Alternatively, segmentation models may be used to refine flagged samples.
6 Conclusion
In this work, we tackle the problem of spurious correlations in image classifiers by shifting attention from training algorithms to data. As datasets continue to grow, it can be challenging to answer even the simplest questions about what resides within them. We propose to leverage interpretability tools to harness machine perception towards scalably organizing data. Namely, we sort data within each class based on spuriosity, as measured by neural features in an interpretable network. With spuriosity rankings, we can easily retrieve natural examples and counterfactuals of relevant spurious cues from large datasets, enabling efficient measurement and mitigation of spurious feature reliance. We demonstrate the feasibility of our approach on a vast set of models on ImageNet a far larger and more realistic setting than most prior spurious correlation benchmarks. Further, we observe highly similar biases across diverse model set, suggesting that biases may be determined more so by what a model sees than how it is trained. We hope our work spurs further exploration into how to best understand and utilize the data we already have, towards more robust and interpretable models.
7
Acknowledgements
This project was supported in part by a grant from an NSF CAREER AWARD 1942230, ONR YIP award N00014-22-1-2771, ARO's Early Career Program Award 310902-00001, Meta grant 23010098, HR001 12090132 (DARPA/RED), HR001 I 19S00226 (DARPA/GARD), Army Grant No. W91 INF2 20076. the NSF award CCF22 12458, an Amazon Research Award and an award from Capital One. MM also receives support from the ARCS foundation.
10

---

References
[1] Michael A. Alcom, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh M Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4840-4849, 2019.
[2] Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk mini- mization, 2020.
[3] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Joshua B. Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. NeurIPS, 2019.
[4] Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. ArXiv, abs/2006. 14651, 2021.
[5] Lucas Beyer, Olivier J. H' enaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? ArXiv, 00w/2006.071553 2020.
[6] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In FAT, 2018.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[8] Chaofan Chen, Oscar Li, Alina Jade Barnett, Jonathan Su, and Cynthia Rudin. This looks like that: deep learning for interpretable image recognition. ArXiv, abs/1 10574, 2019.
[9] Xinlei Chen Saining Xie* and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint 2rXivv:2/44.022 2021.
[10] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 5 1--153663,,, 2017.
[11] Jonathan Crabbe, Zhaozhi Qian, Fergus Imric, and Mihaela der Schaar. Explaining latent representations with a corpus of examples. In NeurlPS, 2021.
[12] Alex DeGrave, Joseph D. Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7): 619, 2021.
[13] Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, Sylvan Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucic. On robustness and transferability of convolutional neural networks. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16453-16463, 2021.
[14] Thomas Fel, Ivan Felipe, Drew Linsley, and Thomas Serre. Harmonizing the object recognition strategies of deep neural networks with humans. Advances in Neural Information Processing Systems (NeurIPS), 2022.
(15) Asma Ghandeharioun, Been Kim, Chun-L.ingg Li, Brendan Jou, Brian Eoff, and Rosalind W. Picard. Dissect: Disentangled simultaneous explanations via concept traversals. ArXiv, abs/2105.15164, 2022.
[16] Dan Hendrycks, Steven Basart, Norman MU, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Lixuan Zhu, Samyak Parajuli, Mike Guo, Dawn Xiaodong Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of- distribution generalization. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8320-8329, 2021.
[17] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. ArXiv, abs/2201.11114, 2022.
11

---

[18] Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust super- vised learning give robust classifiers? In ICML, 2018.
[19] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Predicting predictions from training data. ArXiv, abs/2202.00622, 2022.
[20] P. Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re-training is sufficient for robustness to spurious correlations. ArXiv, abs/2204.029377, 2022.
[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02264; 2023.
[22] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, ICML'17, page 1885-1894 1894. JMLR.org, 2017.
[23] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard I. Phillips, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang, Wilds: A benchmark of wn-the-wild distribution shifts. In ICML, 2021.
[24] Evan Zheran Liu, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In ICMI., 2021.
[25] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[26] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
[27] John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlatoo between out-of-distribution and in-distribution generalization. ArXiv, 0tbs/2107.04649, 2021.
[28] Mazda Moayeri, Kiarash Banihashem, and Soheil Feizi. Explicit tradeoffs between adversar- ial and natural distributional robustness, 2022. URL
07592]
[29] Mazda Moayeri, Phillip E. Pope, Yogesh Balaji, and Soheil Feizi. comprehensive study of image classification model sensitivity to foregrounds, backgrounds, and visual attributes. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19065-19075, 2022.
[30] Mazda Moayeri, Sahil Singla, and Soheil Feizi Hard imagenet: Segmentations for objects with strong spurious cues. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. WtRLhhtts:// /ttp::://///eeeeeeeeee eww..wiiwwwww...............e..e net. forum?id
76w7bsdvi2E
[31] Mazda Moayeri, Keivan Rezaei, Maziar Sanjabi, and Soheil Feizi. Text-to-concept (and back) via cross-model alignment. In International Conference on Machine Learning, 2023. URL
https://app wwwanttmesssee commanticccholar. www.:aaar.rrrcoopuu.....::.
[32] Jun Hyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: Training debiased classifier from biased classifier. ArXiv, /bb/2207.02261.. 2020. URL
https: //ppi. semant in ceschol ar. .rg. http//orpuss.... II:22D:332999
[33] Jun Hyun Nam, Jachyung Kim, Jacho Lee, and Jinwoo Shin. Spread spurious attribute: Improving worst-group accuracy with spurious attribute estimation. ArXiv, abs/2204.02070. 2022.
[34] Muzammal Naseer, Kanchana Ranasinghe, Salman Hameed Khan, Munawar Hayat, Fahad Shah- baz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. In Neural Information Processing Systems, 2021.
12

---

[35] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: (0.23915/distill000000
[36] Yao Qin, Chiyuan Zhang, Ting Chen, Balaji Lakshminarayanan, Alex Beutel, and Xuezhi Wang. Understanding and improving robustness of vision transformers through patch-based negative augmentation. ArXiv, abs/21 10.07858, 2021.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 2021.
[38] Alexander Robey, George J. Pappas, and Hamed Hassani. Model-based domain generalization. In NeurlPS, 2021.
[39] Amir Rosenfeld, Richard S. Zemel, and John K. Tsotsos. The elephant in the room. ArXiv, hts/1808.03300 2018.
[40] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ArXiv, abs/1911.08733 2019.
[41] Shiori Sagawa*, Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL
https://openrev//
net wtte://ouuunniirrruurrrrr.r.
[42] Shiori Sagawa Pang Wei Koh*, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learming Representations, 2020. URL
https://openrevier net http:/ooommmmiiiyuurrrrrr
[43] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In ArXiv preprint arXiv:2007.0844 200700889, 2020.
[44] Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Iyyas, Logan Engstrom, and Aleksander Madry, Image synthesis with a single (robust) classifier, 2019.
[45] Rakshith Shetty, Bernt Schiele, and Mario Fritz. Not using the car to see the sidewalk quantifying and controlling the effects of context in classification and segmentation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8210-8218, 2019.
[46] Sahil Singla and Soheil Feizi. Salient imagenet: How to discover spurious features in deep learning? In International Conference on Leurning Representations, 2022. [
//openreviee eew. n wttp//orrmmmddd/imesssee
[47] Sahil Singla, Besmira Nushi, Shah, Ece Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages
2021.
[48] Nimit Sharad Sohoni, Jared A. Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in coarse-grained classification problems. ArXiv, abs/201 12945, 2020.
[49] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global represen- tations by penalizing local predictive power. In Advances Neural Information Processing Systems, pages 10506-10518, 2019.
[50] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud. Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Huai hsin Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, ArXiv, 22./2206.076822 2022.
[51] Ross Wightman. Pytorch image models. 2019.
htttp::/gitthb..om wttp:/tthhhh.ccc.........omt wwt.aa/atwagaaaaa/
pytorch- -
13

---

[52] Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. ArXiv, 0bbs22055048857 2021.
[53] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Korbblith, Rebecca Roclofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, and Ludwig Schmidt. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1959-7971, June 2022.
[54] Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. In International Conference on Learning Reprexentations, 2021. URL httpttss//oppeeeiiiiieeeeeeeeeee http://ppnneeeeeeeeeeeeooooooinn
[55] Nanyang Ye, Kaican Li, Lanqing Hong, Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li. Ood-bench: Benchmarking and understanding out-of- -iit----istiittionn generalization datasets and algorithms. ArXiv, abs/2106.03721, 2021.
[56] Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. Re-labeling imagenet: from single to multi-labels, from global to localized labels. 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2340-2350, 2021.
[57] John R. Zech, Marcus A. Badgeley, Manway Liu, Anthony Beardsworth Costa, Joseph J. Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: secssoaectional study. PLoS Medicine, 15, 2018.
[58] Zhang, Aditya Krishna Menon, Andreas Veit, Srinadh Bhojanapalli, Sanjiv Kumar, and Suvrit Sra. Coping with label shift via distributionally robust optimisation. ArXiv, abs/2010.12230, 2021.
[59] Michael Zhang, Nimit Sharad Sohoni, Hongyang R. Zhang, Chelsea Finn, and Christopher R'e. Correct-n-contrast:: A contrastive approach for improving robustness to spurious correlations. In ICMI., 2022.
[60] Zhifci Zhang, Yang Song, and Hairong Qi. Age progression/regresssonn by conditional adversar- ial autoencoder. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26. 2017, pages 1352-4366 IEEE Computer Society, 2017. doi: 10.1 1011009V...200....0.
URL httph:tps::.. //oii org/ 10. 1109/CVPR. 2017. 463
[61] B. Zhou, A. Khosla, Lapedriza. A., A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
14

---

Appendix
A Impact and Limitations
We hope work will allow for better understanding of how spurious feature reliance affects deep models, leading to improved handling of algorithmic fairness and model robustness, which are both related the spurious correlation problem. We open source all code, and modify key framework of our method to vastly increase its usability for people with fewer computational or data resources. We also create a website for easier viewing of our analysis, making our entire annotation process completely transparent. We obtain verbal approval Serroval[fron from our institution's 's IRB for our study, and pay workers above minimum wage. Key limitations features (only a subset), and uses a single model to detect and measure them, though we argue that the features detect are very relevant and likely relied upon by all models; we elaborate on this point in Section L.3
that our method does not detect all spurious
B Characterizing Feature Sensitivities
Using visualizations from our robust neural feature annotation, we obtain soft segmentations for features in input space that activate specific neural feature. Namely, the Neural Activation Map for a neural feature can serve as soft segmentation for the feature. Using Mechanical Turk human study, validate that neural activation maps for images in the top 20th percentile within its class for activation of the neural feature segment the same input pattern (see Appendix [.2.2]
European Green Lizard
Desktop Computer
BBookcas
Agama
Monitor
Tobacco Shop
(a) Gray (Color)
(b) Blur (Texture)
(c) Patch Rotate (Shape)
Figure 10: Examples of features that are particularly sensitive color, texture, and shape corruptions. Original images (top) and corrupted images (bottom) shown, along with model predictions. In one case, corrupting the spurious feature of keyboard on desk for class monitor actually improves prediction.
We then use targeted corruptions to gain insight on the information within an input region that models rely most. The core premise of argument is that degradation in model performance due to corruption of some information can be proxy of the importance of the corrupted information. Indeed, similar arguments employed to assess core/spurious or wwngrrounddbcckgroune sensitivity respectively. However, in our
made in 46 29]. where adding Gaussian noise specific regions was
'We were informed that formal approval is not needed because we do not obtain information about our crowd worker, and thus study does not constitute suujnn-sujject research.
15

---

Pearson's r- 0.724
0.8
DeiT Relative Accuracy Drop
0.6
0.4
0.2
0.0
-0.2
-0.4
Gray
Blur
-0.6
Patch Rotate
-0.5
0.0
0.5
1.0
ResNet Relative Accuracy Drop
Figure 11: Sensitivities of each class-feature pair to specific corruptions for ResNet-50 and DeiT- Small. The two distinct models rely on features in the same way.
analysis, we use carefully chosen corruptions to remove one aspect of primitive information (i.e. color, shape, or texture) without disrupting the others. Namely, our corruptions are graying out image regions, blurring out image regions, or patching an image region and randomly rotating each patch by multiple of 90 degrees. Graying out an image removes color, blurring out destroys local (i.e. textural) details, and random patch rotation breaks longer edges, hence disrupting shape information.
We apply these targeted corruptions segmented feature regions for all 5000 annotated class-feature pairs. Specifically, we focus on the 65 images per class where feature is activated most highly (i.c. roughly top 5th percentile). since the feature is most prominent these images. Also, for spurious features, we include a filtering step to avoid corrupting core regions. We do this by removing any over- lap of the spurious feature segmentation with the consolidated core mask 46), which is pixel-wise maximum of soft segmentations for any relevant core features. We track model accuracy samples with and without each corruption applied, using change in accuracy as measure of sensitivity.
Figure 10 shows examples of features that are the most sensitive (i.e. highest change in accuracy) each of the three corruption types, visualizing the corruption and its affect model prediction. In the left panel, we see that the color information in the core feature of lizard body is crucial for successful prediction of the class European Green Lizard (matching intuition). In the middle panel, we see case where the spurious feature of keyboard on desk hurts classification. Namely, blurring the keyboard and desk leads to more accurate prediction while the shape and color information is retained, it appears that the local details in the texture of the keyboard and desk contribute more to confusing the model and leading misclassification. Lastly, for the spurious feature of books on bookcase, we observe that the shape of the books is crucial, as rotating patches leads an incorrect prediction of tobacco shop.
the desktop computer. We note that
We conduct this analysis using two pretrained models roughly equal size: ResNet50 and small DeiT. Figure TT shows the relative accuracy drops incurred by performing each of the three corruptions for all 5000 annotated class-feature pairs on both models. We find that the two models have strongly correlated sensitivities (r 0.724). The least correlated sensitivities appear be for the patch- rotate corruption, where the transformer has nearly zero accuracy drop for many cases where the ResNet experiences greater accuracy drop. This most likely due to the unique robustness of vision transformers to various patch transformations like permutation, rotation, and ablation [34] 36) Despite this, the strong correlation between feature sensitivities for two diverse models suggests that model behavior is determined much more by the data it operates over than the specific decisions made during training. In other words, the features that models rely on and the ways in which they rely on them may be far more a function of data than a function of the model. If this conjecture holds, it would warrant greater inspection of the role of data in various open problems in modern deep learning (robust generalization, efficient learning, etc).
16

---

Paropss
Pattpii
junco
Junco
Limptee
Limpkin
Velcano
Volcano
Greenbouse
Creenbouse
(a) Training Split
(b) Validation Split
Figure 12: Images with highest and lowest spuriosity for five randomly selected classes (top/bottom row in each chunk corresponds to images with highest/lowest spuriosity respectively). The spurious features depicted are lake water, branches, water, sky with clouds or lava, and plants. The spurious features easier to in the training split than in the validation split, because there roughly 25x more samples to choose from. Nonetheless, the validation images still seem to be organized so that the top-10 10 images contain the spurious feature much more than the bottom- 10, indicating that our spuriosity ranks generalize to the validation split.
C More Examples of Spuriosity Rankings
To further validate our spuriosity rankings, now present more examples of images ranked by spuriosity. While this validation is qualitative, we argue that it is sufficient since we are attempting to proxy fundamentally qualitative notion. Namely, in figure 12] we present images organized by spuriosity ranking for five randomly selected classes. Here, we show images both ranked in the training and validation split to demonstrate that even though trends are clearer when ranking training images (because there are ~ 25x more them to choose from), the underlying semantic concepts by which images ranked still generalize to the validation split.
D Full Pretrained Model Evaluation Results
We now provide greater detail for the pretrained model evaluation presented in Section 4.2 We
evaluate 89 models spanning diverse architectures and training procedures. Namely, the models we consider fall into the following categories, with category nicknames parenthesis: supervised vision transformers (Sup VīTs), self-supervise vision transformers (SS VITS), CLIP vision transformers (CLIP ViTs, both Zero-Shot and Finetuned), CLIP ResNets (CLIP RNs), self- or semi-supervised ResNets (SS ResNets), supervised ResNets (Sup ResNets), mixer and ResMLP models (MLP-Basee various other convolutional networks (Other CNNs), and adversarially trained ResNets (Robust
ResNets). The vast of majority of pretrained weights are obtained from the timm library [51 while
others come directly from their original respective repositories 9 43 Figure shows
effective robustness for each model, as well as the average effective robustness per category.
17

---

0.15
Effective Robustries
0.10
0.05
-0.05
-0.10
Average Effective Rabustress
0.10
0.00
-0.05
Sup VIT.
55 Vm
CL 0T..
55 NusANE
9/ RoNdee
ODtar CNN
Figure 13: Effective robustness for all 89 models considered (top), along with effective robustness averaged model category (bottom).
We measure effective robustness by first fitting a line to predict accuracy on the 10 validation images per class with the least spuriosity function of accuracy the 10 validation images per class with the most spuriosity. Let acCup and acCbo denote accuracy on images with the most and least spuriosity respectively (i.e. top and bottom with respect spuriosity rankings). We refer to the line of best as 3 R + R. Effective robustness for model with accuracies on top and bottom spuriosity ranked images acCip-: acCнot respectively is simply acCыε 3(acc In other words, measure how high above the line of best fit a model's marker falls in figure 6 (left).
We note that this differs slightly from the originally presented notion of effective robustness in (27] where log-linear fit is performed. Morcover, instead of predicting -utto--distribution accuracy based on in-distribution accuracy, data. We argue that we capture subsets where subpopulation shift has occurred, specifically with respect common spurious cues for each class.
compare accuracy two carefully chosen subsets of in-distribution
Our findings show that some models are more effectively robust than others. Namely, zero-shot CLIP models have much higher accuracy images with lowest spuriosity than their performance on the images with highest spuriosity would predict. After finetuning linear head using ImageNet on top of the fixed CLIP image encoder, we see effective robustness drops significantly. We note that both these results were observed using independent distributional robustness measures in [53]. At the other extreme, adversarially trained ResNets have the lowest effective robustness, result carefully studied in 228]
Aside from these two extremes, most trends are muted. We believe the stronger signal exists in pla-slass spurious gaps, as shown ( suggesting that while models generally behave similarly on the same inputs, their behavior (i.e. with respect to spurious feature dependence) varies dramatically across inputs. Indeed, we find that per-class spurious gaps correlate strongly between pairs of models in our evaluation suite. Figure 14 visualizes these correlations, with average correlation of = 0.69 being observed. We thus recommend closer attention be paid to the individual data classes and potential spurious features per class model is to be deployed over.
18

---

Correlation Spurious Gaps
Oraa
Fask 20 D
2010 154
Mil Pooteds
In moneer
Rotual Seatect
cur Nemaeto start
CUR Aetleto setured
Cas
21d Terle500
Figure 14: Correlation of per-class spurious gaps for all pairs of models. Numbers shown are Pearson's correlation coefficients times 100. On average, pair of diverse models have strongly correlated class-wise spurious gaps, with 0.69. Notably, zero-shot CLIP models have the worst correlation with other models, suggesting that their perception fundamentally different.
Details on Feature Discovery Beyond ImageNet
F.1
General Training Details
Recall that we introduce lightweight extension of spurious feature/concept discovery without requiring adversarial training by simply fitting a linear layer on new data over a fixed adversarially trained feature encoder (Section 3.2). Specifically, in all cases, we use a ResNet50 feature encoder (i.c. all layers except for the final linear classification head) adversarially trained for ImageNet classification with projected gradient descent using an l, norm attack of budget 3.0, downloaded from 43). Note that this feature encoder comes from the same network used to discover core and spurious features in ImageNet. However, we train new linear heads over fixed features for four tasks over three datasets: Waterbirds [42], Celeb-A 255 and UTKFace [60] In all cases, we use an Adam optimizer with learning rate 0. and weight decay of 0.003. We train for 20 epochs. While model we use may not be as accurate as model trained from scratch, they still achieve relatively high accuracies given the ease of their training. Notably, for Celeb-A hair task (2-way), UTCe gender task (2-way), and UTKFace Race task (5-way), our linear layer fitting robust neural features yields
19

---

(a) Waterbird core feature. Focus elongated and curved necks.
(b) Landbird core feature. Fncus is feathers with bright colors.
Figure 15: Core features discovered for the Waterbirds waterbird vs. landbird classification task. We identify features that occur in specific subpopulations of the broader classes. Low activating images reveal subpopulations where the feature is absent. The rows show (from top to bottom): highly activating images, corresponding heatmaps and feature attacks, and lowly activating images.
(a) Blond curly hair.
(b) Blond straight hair.
(c) Brown curly hair.
(d) Brown wavy hair.
Figure 16: Core features discovered for the Celeb-A blond vs. brown hair classification task. We identify features corresponding to different types of hair within the same broader class of hair color. The rows show (from top to bottom): highly activating images, corresponding heatmaps and feature attacks, and lowly activating images. The captions reflect the focus of the robust neural feature.
accuracies 95.3%, 86.5%, and 66.7% respectively. Acceuracy on the Waterbirds validation is lower (77.6%), though this is typical for Waterbirds, as the validation shift features a significant subpopulation shift (by design, due to breaking the background correlation). More importantly, in all cases, important neural features are observed to correspond to data patterns (i.e. human concepts) salient to the task at hand, including spurious ones.
E.2 Additional Discovered Features for Waterbirds and Celeb-A
We now present additional features identified via our lightweight extension of the feature discovery framework of [46]. Specifically, we show core features, which are relied upon more frequently than spurious features, which we show in Figure 4 We observe robust neural features to respond to specific patterns shared among subpopulations within the broader classes. Surprisingly, these robust neural features were not at trained on images from the downstream tasks. Yet, the patterns they respond to are semantically meaningful for the downstream task. We attribute this to the diversity in ImageNet, which leads to learning many general visual patterns that are manifested in more specific contexts for vast number of downstream tasks.
20

---

AT ResNet50
25
Esp 6 =4.0
Moco ViT-$
Spurious Gap
ResNet50
201
Deit (S)
Low Spuriosity
15
Tuned
Randomly Tuned
Tuned on Errors
10
Original
(No Tuning)
ImageNet Accuracy
Figure 17: In addition to tuning on low-spuriosity images and on random samples, we include results from tuning on misclassified samples in white. Error tuning minimally closes spurious gaps, and actually leads to substantial drops in overall validation accuracy.
E.3 UTKFaae Experiments
UTKFace (60 is attributed dataset originally intended for the task of synthetically aging face image using conditional generative models. The dataset consists over 20, 000 images with age, ethnicity, and gender annotations. We consider the task of gender and age classification. We use 90% of the data for training, and hold out the remaining 10%. We inspect the 15 most important features for each class per task. For all classes and tasks, are able identify least on spurious feature with our method. Notably, these tasks were not designed to contain spurious features, like the other two benchmarks analyzed. This goes show that spurious correlations may arise in any setting. and our method can reveal such spurious features with minimal computation. Having discovered robust neural features that detect spurious concepts, one can sort data by spuriosity to measure and mitigate the resultant bias.
F Alternate Baseline: Error Tuning Does Not Close Spurious Gaps
A common approach within recent spurious correlation literature is to automatically select samples to conduct additional training on by simply inspecting whether they were classified correctly originally. For example, Liu et al. (24) propose to 'just train twice', upweighting етгог samples in the second round of training: Nam al. [32] 'learn from failure' by training one classifier that is intentionally biased, and using the hard (i.e. misclassified) samples from the biased classifier to obtain an unbiased classifier; Zhang et [59 use error samples hard positives in an additional round of contrastive training. Inspired by these approaches, we include an additional baseline to the experiment from section 4.3 where we tune existing classification heads on samples originally misclassified. Specifically, we train misclassified samples for all classes. All three methods train 100 samples per class. When the number errors in a class less than 100, we add randomly selected correctly classified class instances. When the number of errors is more than 100, we randomly select 100 of them. Results shown are averaged over three trials.
We observe that while error tuning closes spurious gaps more than tuning on random samples, the reduction spurious gaps is far less than that from low spuriosity tuning. More importantly, overall validation accuracy drops substantial when tuning on errors. We conjecture this occurs because errors can be caused by many things aside from the absence of spurious cues, such as label noise. Thus, tuning on misclassified samples could lead to overfitting on unreliable data. In contrast, low spuriosity images are designed to only differ from typical samples in that they lack particular cues that have already been deemed spurious by human. Arguably, low spuriosity images can at times offer more reliable learning signal, as they do contain distracting shorteuts (e.g. the low spuriosity lighters in figure are far casier to than the high spuriosity ones). Also, while the error-centric
21

---

approaches were shown to be effective on existing spurious correlation benchmarks, we hypothesize that the biases in those benchmarks are overly simplistic: they typically only include one spurious correlation and at most handful of classes. Therefore, these methods may struggle in more realistic settings, as demonstrated by their ineffectiveness on this ImageNet scale experiment.
Thus, we claim that low spuriosity tuning offers unique advantages compared to prior work. While most existing methods focus on altering the training algorithm, our method acknowledges that data may have a larger (and often overlooked) role in determining the biases a model absorbs, and focuses on finding the right data to tune. The experiments in this section show that compared to another line of data-centric approaches, our low spuriosity tuning may be more effective, in both retaining overall accuracy and reducing biases.
G On Automating Spuriosity Rankings
We believe that human involvement is strength of our framework, as it increases transparency. Namely, the human in the loop is given a concise inside look on the cues a model trained on the given data is likely to rely upon. Moreover, the human is given agency to decide which cues model performance should be invariant to. Also, the level of human involvement is relatively low. Given the complexity of the task of sorting thousands of images within class, only requiring human to inspect a handful of images is relatively efficient. With an appropriate UI, we can confirm that this takes no longer than about a 30-40 seconds per class.
However, in cases where minimizing cost is preferred, Spuriosity Rankings can be automated. One way to do so is to automate the annotation of neural features core or spurious. Specifically, one could automatically segment the class object with vocabbacyyy segmentation models [21], and then compute the amount of saliency placed the object by the neural activation map. If most of the salient pixels for a neural fcature lie outside of the image region containing the class object, one could automatically flag such feature spurious.
Another way to automate Spuriosity Rankings is to leverage laaiunn models (VLMs) like CLIP (37. With VLMs, we can compute the similarity of an image to any concept, encoded directly from text. After sorting images by similarity to spurious concepts, we can inspect the Spurious gap to measure bias, and train low spuriosity images to mitigate bias. Thus, one must simply enumerate (in the form of text) potential relevant spurious cues; since encoding the cue and sorting along is free, one can discard the cues which do not result in an accuracy gap between the most and least similar images, as the model of interest apparently would not be biased to the presence of that cue. We note that while VLM training is expensive, pre-trained model would suffice, and moreover, recent work shows that VLM abilities can be extended to arbitrary vision-only models cheaply as well 31.
Note that the underlying mechanism of Spuriosity Rankings preserved in both of these automateed variants: we utilize the representation space of an interpretable model to quantify the presence of relevant cues, and rank images by spuriosity (proxied by similarity/activationn of concept directions in representation space) to enable interpretation of spurious cues in context and measure model bias caused by them.
H Details on Core-Cropping
To perform core-cropping, we first obtain a soft segmentation of core regions, done by averaging the neural activation maps for core neural features. Next, we threshold the averaged mask to only retain highly activated pixels. We use a threshold of 0.9. Then, we obtain a bounding box localize a rectangular region encompassing the pixels that most highly activate the core region. Since we use high threshold, we expand our bounding box by 20% along both height and width dimensions. Finally, we convert our rectangular region to square one by simply extending the shorter side to match the length of the larger side. We note that this method only as effective as the core soft segmentation. Note that core neural activation maps are generally reliable for images that reasonably activate the core features. Namely, we validate the quality of these segmentations for images in the top thth percentile of activation for a class (see Appendix 1.2.2). However, for the remaining 80% of images, these soft segmentations may not be perfectly reliable. Nonetheless, as segmentation models improve, core-cropping may also improve, as we can simply replace core soft segmentations obtained
22

---

via neural activation maps with actual segmentations, conditioned on the class object. Thus, flagging potentially mislabeled samples
I Robust Neural Feature Annotation Details
Our work uses the feature discovery framework introduced in [46) also known as Salient ImageNet. We now i. introduce taxonomy of relevant terms, ii. review the framework, iii. discuss its merits and potential limitations, and iv. detail our expansion/modiiicaionn
L.1 Taxonomy of Relevant Terms
Below enumerate and explain some terms refer to throughout the paper.
Core feature part of the class object, like fur for an otter.
visual pattern that is essential to the class object; i.e., the pattern is
Spurious feature or cue: a visual pattern that is not essential to the class object; i.e., the presence of the spurious cue is not required for the image to belong the class, and the spurious cue can exist in images from other classes.
Neural Feature: node in the penultimate layer of a classifier parameterized by deep neural network. Importantly, each logit of the classifier is linear function of neural features.
Robust neural feature: neural feature from adversarially trained network. We use 8 PGD-trained 26 network with e 3.
Neural activation map: an extension of class activation map 6 for neural features, used to highlight the region of image responsible for activating neural feature.
Feature attack: a visualization technique to amplify the visual cues present in an image that activate a neural feature.
1.2 Salient ImageNet Framework
The steps of the Salient ImageNet Framework are:
1. Adversarially train model for your classification task.
2. Identify important neural features per-class based on feature importance, which simply the product of the average feature activation and the weight of linear head connecting feature to class logit. Feature importance explicitly computes the average contribution of a feature to class logit for samples in the class.
3. Select the top-k (i.e. k 5) activating instances of class for selected feature and additionally generate heatmaps (via overlaying the Neural Activation Map on the original image) and feature attacks (via adversarially attacking the original image using gradients from the adversarially robust model so to maximize activation on the feature).
4. Have humans inspect the 3 x 15 visualizations to determine the focus of the neural feature the main object (i.e. pertaining the class and hence core) or on the background or a separate object (hence making the feature spurious).
including an We elaborate on the feature discovery and annotation procedure in subsection I.2.1
example of the MTurk form. In [46], the annotated robust neural features were primarily used to softly segment image regions containing core and spurious features at scale. Thus, in addition to the Mechanical Turk study performed annotate neural features. second study was conducted to confirm that neural activation maps for annotated features softly segment the same input features across the top 5th percentile of samples within the relevant class. With this confirmation, they computed neural activation maps for all of these images and used these as feature segmentation maps. To assess reliance on core and spurious features, they would inspect drop accuracy due to adding small amounts of Gaussian noise to these regions. We repeat modified and expanded version of this second study, so to validate larger number of feature segmentations. Details on this human study are in subsection 12.2
23

---

1.2.1 Mechanical Turk study for discovering spurious features
The design for the Mechanical Turk study is shown in Figure 18 The left panel visualizing the neuron is shown in Figure [8a] The right panel describing the object class is shown in Figure [8b] The questionnaire is shown Figure [8c] We ask the workers to determine whether they think the visual attribute (given on the left) is a part of the main object (given on the right), some separate object or the background of the main object. We also ask the workers to provide reasons for their answers and rate their confidence on a likert scale from 1 to 5. The visualizations for which majority of workers selected either background or separate object as the answer were deemed be spurious. Workers were paid $0.1 per HIT, with an average salary of $8 per hour. In total, we had 137 unique workers, each completing 140.15 tasks (on average).
We conduct this study for the five neural features with highest contribution to each of the 768 ImageNet classes not analyzed in [46), resulting in 3840 newly annotated class-feature dependencies, including 468 spurious ones (3.3x and 2.9x increases respectively).
1.2.2 Mechanical Turk study for validating heatmaps
We now detail the second MTuurk study we carry out to validate the feature soft segmentations generated by Neural Activation Maps of annotated robust neural features. For each annotated class- feature dependency. we first obtain the training images from the class who's activation on the feature is within the top 20th percentile for the class. This results in a set of 260 (20% of 1300 training images per class) images per class-feature pair, as opposed to only 65 in [46]. From this set of 260 images, we selected 5 images with the lowest activations on the neural feature and randomly selected 10 images from the remaining set (excluding the already selected images). We show the workers three panels: images and heatmaps with (i) the highest 5 activations, (ii) the next 5 highest activations. and (ii) the lowest 5 activations. For each panel, workers were asked to determine if the highlighted attribute looked different from at least 3 other heatmaps in the same panel. Next, they were asked to determine if the heatmaps in the 3 different panels focused on the same visual attribute, different attributes or if the visualization in any of the panels was unclear. Thus, we validate that both within each panel and across panels, the focus of the neural feature is consistent.
The design for the Mechanical Turk study is shown in Figure 19 The three panels showing heatmaps for different images from a class are shown in Figure 19a The questionnaire is shown in Figure 196 For each heatmap, workers were asked to determine if the highlighted attribute looked different from at least 3 other heatmaps in the same panel. We also ask the workers to determine whether they think the focus of the heatmap on the same object (in the three panels), different objects or whether they think the visualization in any of the panels is unclear. Same as in the previous study (Section I.2.T]. we ask the workers to provide reasons for their answers and rate their confidence on a likert scale from 1 to 5. The visualizations for which at least 4 workers selected same as the answer and for which at least workers did not select "different" as the answer for all 15 heatmaps were deemed to be validated i.e for this subset of 260 images, we assume that the robust neural feature's ' focus is on the same visual attribute. Showing three panels as opposed to two is novel to our work and increases the standard for validation.
For all (class, feature) pairs (1000 We successfully validate 4763 pairs (i.c. 95.26%). demonstrating the effectiveness of using neural feature supervision automatically (softly) segment an input pattern in a large number of images. We utilize these feature segmentations in Appendix B
5000 total), we obtained answers from 5 workers each.
I.3
On Feature Discovery Using Only One Model
The key idea of our framework (expanded from [46]) was to use neural features of an adversarially trained network as automatic 'concept' detectors, and further to use their activation maps as soft seg- mentations. One may argue then that the features discovered are biased to the underlying model used. However, we argue that the primary patterns that models rely on are generally the same. Note that the existence of patterns predictive of a class is determined by the data distribution and not the model. While models may learn to rely on different patterns and may also vary the degree to which they rely on the same patterns, we believe that the most predictive patterns will be detected and relied upon by nearly all models. In our experiments, we find that the class-wise spurious gaps are highly correlated for any two pairs of models, with average correlation of 2 0.69 (see figure 14 We also observe
24

---

Visual attribute
Highiigttd
Ampliieed
Main object
category namejjj
Otject suporastegery: the terbratte
Otject defisition any brighity colored buttrilies
images present theee
possibly abpecte
you are ussure about the object meaning, you may consult papeottntiiss/eee
(a) Visual attribute
(b) Main object
Please the questions below
Which of the hallowing best describes focus the region Highiighteed attributee?
Amplrted suaa attribute be usetu scee cases when red region overtaps over muttple otgects such thise overtap bebwaan cougar and windss ovennip between growm and Eowerss
ignoro
attibutee
region does overlup over mutiiple objects.
The main object helonge the fol flllowing supercategon gory: veni clee
Focuo the background of main object.
Examples: greenery the racecmuns crasshaszes, natter back tnokground altatines tower background bee,
Focus some other object (not of the main object).
Examples: babe is nota cart bassthel ttatura bash locs ans nota mail idness dancant, branch attadd.
on some part
object.
Examples:
How confidant about
rightly en-hheed
Eewes mentdee
Peity metbde
Comeistro awidert
Q3. Pieaso justlyy your anower to Ofl by dosoribing the hightighted sual attibude? This quostion is mandanary for completing this HIT.
which as bous it fooone bas.ground
Examppe
(c) Questionnaire
Figure 18: Mechanical Turk study for discovering spurious features. This figure is from Singla and Feizi 466 and included here for completeness.
that the accuracy drops experienced due to targeted corruption of input regions where certain features reside for ResNet and vision transformer are highly correlated too (r 0.724) (see figure 11) These results suggest that models are sensitive to the same data patterns and even rely on the different pieces of information (i.e. color, shape, texture) captured in the data patterns in the same ways.
25

---

Section
Section B
Section C
Heatmaps
dfleeet
dltwent
diheent
diferet
A4
dffrrtt
04.
diforrnt
A5.
difereet
USS
citfeent
CC
diferent
(a) Heatmaps highlighting the visual attributes
Please answer the questions below
Q1. Which
Sections and
attribute.
Sections A, and focus on difforent visual attributes.
The focus of region unclear elther Section C...
02. How confidant are about answers above??
corfdert
siigtly conficert
Samewhal conticent
conndert
Compless confident
solected same
Q1,
not,
Example answers: cas locasi backs, dag draas.
(b) Questionnaire
Figure 19: Mechanical Turk study for validating heatmaps
To be perfectly clear, do not claim have captured all spurious (or core) dependencies, mainly because we only inspect the 5 most important robust neural features per class. We acknowledge that another reason for certain dependencies be overlooked because we use single model detect and measure the presence of concepts. However, this only occurs for patterns that other models are sensitive to but adversarially trained model not. The main patterns that fit this description are adversarial ones, which contain semantic meaning and thus arguably should be overlooked for
26

---

our purposes. In other work, adversarialy trained models were found to have increased reliance on spurious features [28]; this result was also corroborated our spurious gap analysis. Thus, anything, it seems like using an adversarially trained model may overlook core features, favoring spurious ones. Seeing as the goal of our analysis is to detect and assess spurious feature dependencies, we believe using robust neural features from an adversarially trained model is warranted, especially given the enhanced interpretability of this model, which allows for reliable heatmaps and feature attacks.
1.4 Our Contributions
We now detail the novel contributions we make to the Salient ImageNet framework and analysis from [4]_ The most crucial contribution is the notion that to interpret and improve spurious correlation robustness, one can seek to simply reorganize the data they already have, as opposed collecting more samples or additional instance-wise supervision (i.e. segmentations, manual annotations beyond class-label) that would require fundamental changes the training strategy. Using spuriosity rankings, we gain new and surprising insight into the nuanced ways that models rely on spurious features. Namely, we find that contrary to common belief, models can perform worse when spurious features are present than when they are absent (i.e. because of spurious feature collision, as some spurious features are depended on for multiple classes). Further, our spuriosity rankings naturally lends itself to simple, effective, and efficient manner to both measure the biases caused by spurious feature reliance, and mitigate theses biases for any model. Finally, we make a crucial adjustment to the feature discovery framework of [46] that removes its most costly and at times prohibitive step (adversarial training). Observing that the framework can be applied by simply fitting a linear layer atop a frozen adversarially robust feature encoder vastly increases the usability of this framework, cutting the time and data required to perform feature discovery by huge margins.
We now take a closer look at our method for measuring bias caused by spurious feature reliance compared the one proposed in [46), as we believe our method corrects crucial shortcomings of the prior work. Recall that the prior evaluation framework consisted of corrupting core or spurious regions by adding Gaussian noise multiplied by soft segmentations of core/spurious regions obtained via Neural Activation Maps of annotated robust neural features. The drop in accuracy due to these regional corruptions was used to measure the importance of the region. We enumerate the shortcomings of this prior framework, and explain how our method evades these concerns, below.
Introduction of a synthetic distribution shift. The previous framework required taking images out of distribution by synthetically corrupting core and spurious regions with Gaus- sian noise. Model behavior on these non-natural images may not truly reflect how the model would behave on the more realistic distribution shift caused by breaking spurious correlations.
Instability. Adding noise introduced hyperparameters, like the amplitude and norm of the noise, which we observe the metric to be undesirably sensitive to. In our method, the only hyperparameter is the number of images we include in the sets of highest and lowest spuriosity images. We verify our results our stable with respect to this hyperparameter.
Inability to compare across diverse models. Because models have varying robustness to Gaussian noise (e.g. some more recent models, especially vision transformers which utilize heavier regularization to compensate for lacking inductive biases of convolutional networks, use noising as data augmentation during training), drop in accuracy due to Gaussian noise cannot be reasonably compared as a means to measure how models use the corrupted image region. Our metric utilizes accuracy, which is standard and established for model comparisons. Moreover, we can appeal to the notion of effective robustness (27 to further control for varying accuracies of the models we consider.
Other contributions that are less novel but still impactful include massively expanding the scale of the feature annotation analysis and creating web interface to view the discovered dependencies. Specifically, we annotate over 4x more classes, yielding interpretation of feature dependencies for all 1000 classes in ImageNet. Given that this benchmark is ubiquitous, we believe deep dissection of what one model perceives as important patterns in ImageNet can be extremely informative in better understanding how machine perception differs from human perception. We also perform the MTurk study to demonstrate the generalizability of our feature annotations for the top 20th percentile of images based feature activation as opposed to the top 5th percentile, leading to reliable feature
27

---

Class: 350 (Alpine lbex), Feature: 389
This featurm spartsse clase Abshne Abos it the thind most important destuure for class Appll thes
Below diiptry Aiptin tiex troaper that menth activotte Reatime, Noural Activot Map thestmappi and Fosture Attack
additionally responses other parna naraa sarn featural).
armosated
ANNOTATIONS
now responses crowdworkers annotate
feature
pair conwor muuribur Warkers demane focae image is the main objjcct (corc) or soporati or backgr our object (spurious), They also mark their confidence ona scale of to provider then irration for their annctation..
Macu Confidence
Explararian
cogre
Badkgrourd
object
background.
Background
Focus mountain bad ligoo coprra Red focus on Ibev.
Main Object
Separate
Object
Focus on mountain which not port bbx
Votes for core: Votes Feature annotated countoust
RELATED CLASS FEATURE PAIRS
IMPORITANT FEATURES
There features annotated corc features as
Core featuree inc 462, 1207, 1526. Spurious stures es 020.
CLASSES Wi IERE FEATURE amp AMONG MOST IMPORTAT TDP-4)
Featuure 389 covel for the fo lowine classen Mountain, Valley, Volcamo.
Feature 38P sparlousfor che fowlingg classes Marmot. Alpine Ibos. Sti
Figure 20: Example of landing page for one class-feature dependency, We present visualizations shown to MTurk workers to annotate the feature as cor spurious, well as the annotations and explanations given by the MTurk workers. Further, we provide links to pages for other features relevant to the class, and other classes that also rely on the feature.
soft segmentations for many more images. Lastly, we create website, which, while simply user-interface, will greatly increase ease of viewing and transparency into our analysis, the visualizations used to perform our core/spurious neural feature annotations. The website can be found at / /ttl:::///aeenttt cinntt-imggenet. es umd. edu We present screencaps of pages from the website in Figure and and Figure 21] As web-interface, our work is now accessible to people who do not code or lack the resources to generate our visualizations on their own devices.
show all
28

---

COMPLETE LIST OF CLASS-FEATURE PAIRS
We catalog all 5000 annotated features below. Click any table cell to navigate to page designated to the corresponding class-feature pair.
Classes are sorted by number spurious features. Sparious features are in cells with red backgrounds.
Alternatively, use ctri-f to find class directly.
Class
Feature
2nd Feature
3rd Feature
4th Feature
Sth Festure
Index
Dog
Howler Monkey
Seat Belt
785
Ski
795
Sunglasses
B37
Swimming
433
Balance
Gymnastic
Horizontal Bar
Hockey Puck
746
Figure 21: Screencap of main table from our web-interface for accessing all 5000 class-feature dependencies. One can simply search for class by its name or ImageNet class index to see the five features deemed most important for (i.e. contributing the most to the logit of) the class. Each image then links an individual landing page designated for the example).
pair (see Figure 20 for an
29


Reviews:
[{"review_id": "HWL7Stsj3e", "reviewer": "Reviewer_4unL", "summary": "- This work proposed Spuriosity, a quantity for determining the spuriosity ranking of data. The framework builds upon [42], which identifies spurious and core neural features by analyzing the neural activation map of an adversarially trained model. Spuriosity is defined based on these spurious neural features. The author also proposed the concept of ‘spurious gap,’ which measures the accuracy drop between the top-k highest and lowest validation images. By re-training the classifier head from a subsampled dataset using spuriosity, the authors demonstrate a reduction of around 10-20% in the spuriosity gap at the cost of 1-3% reduction in validation accuracy.\n- While I acknowledge the challenging nature of the addressed problem and the novelty of this work, I believe that the experiment in this paper primarily focuses on the use case of the proposed quantities rather than proving comprehensive validation of them.", "questions": "- [Figure 8] How about comparing the reduction in the spurious gap with a subsampled dataset constructed using misclassified data from an auxiliary model?\n- Is there any idea for establishing the quantitative metrics and baselines of Spuriosity ranking and Spurious gap?", "limitations": "- Their method heavily relies on interpretation methods and human judgment, which is costly.\n- The utilized neural activation map cannot distinguish certain features that are not discernible based on spatial information alone, such as the color and texture of the objects.\n- Most of the framework originates from previous work [42], which diminishes the contribution of this paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This paper addresses the challenging problem: figuring out the spuriosity of large-scale datasets and investigates its impact on model training. I highly appreciate these contributions, as the field often focuses on improving the performance of simplistic benchmark datasets such as Waterbirds.\n- The paper introduced a novel approach that leverages the neural features to distinguish minority samples, which is algorithmically distinct from the previous works such as [1, 2, 3], which rely on classification results of auxiliary models.\n- Their method can be effectively combined with balanced re-training methods, such as DFR.\n\n\n[1] Liu, Evan Z., et al. \"Just train twice: Improving group robustness without training group information.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Kim, Nayeong, et al. \"Learning debiased classifier with biased committee.\" Advances in Neural Information Processing Systems 35 (2022): 18403-18415.\n\n[3] Hendrycks, Dan, et al. \"Natural adversarial examples.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "weaknesses": "- Regarding the Spuriosity ranking, I have concerns about the lack of quantitative metrics and comparison with baselines, making it hard to estimate how reliable the suggested quantity is and how much better compared to the other methods. I suggest comparing the performance of detecting minority samples with [1, 2] on the group annotated benchmark datasets such as WaterBrids and CelebA.\n- Another concern is the lack of a baseline and metrics of Spurious Gap. Say, someone proposed a Spurious Gap ver.2, then how to know which one is better? \n- The proposed method for detecting spurious correlation relies on human effort.\n- Minor correction for misinterpreted terminologies\n    - Interpretable model: Consider using an ‘adversarially trained model,’ as normally trained models are also interpretable.\n    - [L239] How it is trained: This expression is too broad. It seems to encompass algorithmic approaches like GroupDRO, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for taking the time to read our paper and provide insightful comments. We address them below.\n\n**Comparison to existing baselines**\nWe thank the reviewer for noting that our method tackles the “challenging problem [of] figuring out spuriosity on large scale datasets”, in contrast to most work which “focuses … on simplistic benchmark datasets”. We also thank the reviewer for their suggested follow up experiment, in which we finetune model classification heads on misclassified samples, a la JTT or LfF. To provide additional quantitative demonstration of the advantages of our approach, we perform this experiment, and find that finetuning on misclassified training samples does not close spurious gaps. Also, we find that tuning on misclassified samples leads to a larger reduction in validation accuracy (see table attached to global rebuttal). We believe the reason for this is that misclassifications can be caused by many things aside from the absence of spurious cues, including label noise. Thus, tuning on misclassified samples could lead to overfitting on unreliable data. In contrast, low spuriosity images are designed to only differ from typical samples in that they lack particular cues that have already been deemed spurious by a human. Arguably, low spuriosity images can at times offer a more reliable learning signal, as they do not contain distracting shortcuts (e.g. the low spuriosity lighters in figure 1 are far easier to see than the high spuriosity ones). \n\n\n**On evaluating ranking quality, with a potential automated solution**\nWe thank the reviewer for their important question. We note that given the complexity of sorting a large number of images, validating/evaluating the quality rankings is non-trivial, especially since a single ground-truth ranking often does not exist. In our work, we primarily use human studies to validate several key properties of our rankings; we refer the reviewer to our rebuttal to Reviewer ck2C for more details. \n\nWhile we believe our validation is convincing, we agree that in the future, an automated quantitative procedure would be valuable to the community. In response to the reviewer’s question, we hypothesize the following scheme: one could utilize an open-vocabulary object detector or image tagger to annotate the objects / concepts that are present in highest and lowest spuriosity images (based on some proposed ranking we wish to evaluate). Then, one could assess ranking quality as the difference in presence of some detected spurious tag between the most and least spuriosity images. Options for quantitative metrics could be the difference in (a) number of times a spurious concept occurs in the the top vs. bottom spuriosity images, (b) similarity of top vs. bottom spuriosity images to the text embedding of a spurious concept (embedded using a vision-language model like CLIP). We find this an interesting avenue of future research, but opted for human validation in this paper, so as to avoid inheriting biases from the auxiliary model used in validation. We agree with the reviewer that such an automatic quantitative evaluation method would be of value, and hope to pursue this in follow up work.\n\n**Minor corrections**. We thank the reviewer for the detailed comments. We will use softer language in L239. We note that we intentionally use the term ‘interpretable’ model, as our overall framework can certainly extend to non-adversarially trained models. So long as one can interpret internal nodes or, more broadly, directions in the representation space of a deep model, one could use the model to extract spurious concept detectors and proxy spuriosity quantitatively. \n\n**Use of human judgment**. We refer to the global rebuttal, where we detail how human involvement can be removed if desirable. \n\n**Beyond spatial interpretation of neural feature focus**. While NAMs only highlight a particular image region, feature attacks change the image to amplify the true concept causing activation on the neural feature of interest. For example, in Figure 3 (right), while the NAM may leave question as to whether the butterfly or flower causes activation, the feature attack actually replaces the butterfly with more flowers, making the focus of the feature crystal clear. \n\n**Contributions over prior work**. We note that the key idea of our work (i.e. sorting data at scale towards interpreting, measuring, and mitigating bias) is novel relative to the last work. While we utilize the spurious feature discovery method of [42], we believe it is just an implementation for one step of the pipeline (extracting concept detectors from trained networks). We refer to Appendix F.3 for extensive discussion of the novel contributions of our work compared to [42]."}, {"review_id": "0Zckb1ps8b", "reviewer": "Reviewer_ck2C", "summary": "The paper proposes a method to rank images in a given dataset in order of their \"spuriosity\". The proposal uses methodology from previous work [42] which examines most-active neural features of a trained model at per-class level, and hand-labels some sample images w.r.t. whether those features are “core” or “spurious”.  \n\nThe authors 1) scale out the labeling to imagenet-1k, 2) use these sample labels to rank instances according to \"spuriosity\", 3) present a range of qualitative and quantitative results arguing that they can, e.g., reveal minority subpopulations, measure model bias & mitigate it, enable study of model stability by distributional perturbations, identify noisy labels, etc. ", "questions": "1. Overall, if many of the qualitative / exemplar applications could be quantified with more rigorous comparisons, this would strengthen the paper. E.g., recovery of “known” rankings, human-rating or other quantification of ranking quality, comparison against at least some simple bias mitigation methods,  quantitative success at identifying label noise, etc.\n2. Are the rankings / human labeling data to be released publicly? Are the authors claiming a “dataset contribution” as part of their submission?\n", "limitations": "The process of identifying spurious correlations depends on a) neural activations of trained models, and b) human supervision–this can limit the applicability of the approach, both in terms of only considering or covering some aspects of bias, and in terms of labeling costs which are higher than typical category labeling (although, of course, the entire dataset does not need to be labeled).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The authors give a data centric view to the problem of spurious correlation – i.e., instance rankings instead of feature / attribute labels or descriptions. This paper extends the study of [42] for all 1000 classes of imagenet.\n2. The authors present some useful analyses e.g., showing ubiquity and correlation of bias across models and training methods, demonstrating “auto-”discovery of novel dimensions of bias using the new supervision, and  the class-specificity of certain biased features. In particular, it makes sense, as the authors argue, to consider the “spuriosity” of a feature at class granularity rather than dataset granularity. \n3. The proposal of data rankings opens up the possibility of trying out many mitigation methods and metrics to evaluate bias. The authors demonstrate one simple mechanism, and also suggest that their rankings can help flag mislabelled examples.\n", "weaknesses": "1. There is no quantified measure to evaluate the quality of rankings proposed. Is it possible to look at synthetic or small data to get a quantitative sense of ranking quality?\n2. The bias mitigation approach is not compared with the very wide existing literature on bias / spurious feature mitigation – this limits a thorough assessment of  the value of the proposed rankings in mitigation.\n3. The claim about identifying incorrect labels is largely qualitative – some quantification would strengthen the case.\n4. The process of identifying spurious correlations requires human supervision–this can limit the applicability of the approach, both in terms of only considering or covering some aspects of bias, and in terms of labeling costs which are higher than typical category labeling.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to truly engage with our paper and provide insightful comments. We now address main concerns.\n\n**Validating Rankings**. Given the complexity of sorting thousands of images, evaluating the quality rankings is non-trivial, especially since a single ground-truth ranking often does not exist. Nonetheless, we validate several key properties of our rankings. First, we claim our rankings organize data along interpretable notions. Prior work shows that robust neural features are interpretable [42], and we similarly confirm this for all robust neural features used in our study (Appendix F.1.1). Next, we claim that our rankings identify potential biases caused by spurious correlations. In a large-scale evaluation of 89 models, we find that all models underperform on low spuriosity images, thus validating that our rankings capture biases. Further, we claim that high spuriosity images contain the spurious concept of interest. We validate this with a human study, finding that images in the top 20th percentile of spuriosity contain the intended spurious cues in nearly 90% of cases (see Appendix F.1.2). To provide further validation, we conduct another validation in our rebuttal. To confirm that the low spuriosity images do not contain the relevant spurious cues, we inspect the ten lowest spuriosity images for 20 randomly selected classes. We find that the relevant spurious cues are absent in 97% of inspected training images and 93% of inspected validation images. See Appendix C for a visualization of 5 randomly selected classes.\n\nWhile the specific rankings obtained from our proposed framework do not have a straightforward evaluation procedure, this does not take away from the value of the framework as a whole. We believe the core idea of scalably sorting data along interpretable directions by leveraging existing models is novel, significant, and effective, as demonstrated by our experiments. We thank the reviewer for recognizing that our proposal ‘opens up the possibility of … many mitigation methods and metrics.’\n\nA Potential Automated Quantitative Scheme to Validate Rankings\nDespite the challenging nature of validating rankings, we believe an automated method can be constructed. Namely, one could utilize an open-vocabulary object detector or image tagger to annotate the objects / concepts that are present in highest and lowest spuriosity images (based on some proposed ranking we wish to evaluate). Then, one could assess ranking quality as the difference in presence of some detected spurious tag between the most and least spuriosity images. We find this an interesting avenue of future research, but opted for human validation in this paper, so as to avoid inheriting biases from the auxiliary model used in validation. We agree with the reviewer that such an automatic quantitative evaluation method would be of value, and hope to pursue this in follow up work.\n\n**Comparing to existing bias mitigation techniques**. We prioritized conveying our approach over comparing to other methods, many of which differ in fundamental ways (e.g. data centric vs agnostic) from our idea. Nonetheless, we agree with the reviewer that more comparison would be insightful. We'll include more discussion of other methods in the revised draft. \n\nFurther, we perform a new experiment in the rebuttal period, in which we finetune model classification heads on misclassified samples (a common approach). We find that finetuning on misclassified training samples does not close spurious gaps. Also, tuning on errors leads to a larger reduction in validation accuracy (see figure in attached pdf). We conjecture this occurs because errors can be caused by many things aside from the absence of spurious cues, such as label noise. Thus, tuning on misclassified samples could lead to overfitting on unreliable data. In contrast, low spuriosity images are designed to only differ from typical samples in that they lack particular cues that have already been deemed spurious by a human. Arguably, low spuriosity images can at times offer more reliable learning signal, as they do not contain distracting shortcuts (e.g. the low spuriosity lighters in figure 1 are far easier to see than the high spuriosity ones). \n\n**Validating label-noise flagging**. We thank the reviewer for reading our paper to the end! In section 5, we present qualitative evidence that spuriosity rankings can aid in flagging mislabeled samples, as high spuriosity images with severe negative gaps appear to be mislabeled. To strengthen this claim, we conduct a new experiment leveraging ImageNet ReaL labels [1], which indicate for each ImageNet validation image whether objects from multiple classes are present. ReaL labels show 14% of validation images contain multiple objects. We find that for classes with the 5 most negative spurious gaps (averaged over our model suite), high spuriosity (i.e. ranked in top 10% of spuriosity) validation images contain multiple objects in **80%** of cases. In contrast, low (bottom 10%) spuriosity images from the same classes contain multiple objects in only 8% of cases, indicating that the label noise is likely a result of the hypothesized spurious feature collision. Similarly, for classes who’s spurious gap is less than -20%, ReaL labels reveal 42% of the high spuriosity images to contain multiple objects, many times higher than the average rate of label noise. We'll include discussion of this quantitative validation in the updated draft and thank the reviewer for the suggestion. \n\n**Release**. We cannot recall if a dataset contribution was claimed, though regardless, we intend on taking every step to keep our work reproducible and accessible to future researchers. We have already built a web UI for easy viewing and access of our annotations. \n\n**Human cost**. See global rebuttal for ideas on how Spuriosity Rankings can be fully automated. \n\n[1] Are we done with ImageNet? Beyer et al, 2020"}, {"review_id": "sgNeLO6SBh", "reviewer": "Reviewer_9RXj", "summary": "This paper propsed a framework to measure model biases by ranking images within their class based on the strength of spurious cues and evaluate the gap in accuracy on the highest and lowest ranked images.\nThe analysis is comprehensive for a very large number of models. \n", "questions": "The definition of spuriosity in Sec 4.1 appears to implicitly assume the normality with respect to r.\nFor example, if the distribution of r is long-tailed, the variance would be too large and this definition is undesirable. \nThis may not be a problem if the features are after batch normalization, but for some models, this calculation may b undesirable, and would that affect the measurements?", "limitations": "The validity of the interpretation method for deep learning models itself is directly related to the validity of this method.\nThe significance and meaningfulness of the proposed method is entirely dependent on the  interpretation model.\nBecause several papers have shown that explanations using heat maps are sometimes fragile, the usefulness of this technique is influenced by the goodness of other techniques.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper propsed a simple method to measure model biases by ranking images within their class based on the strength of spurious cues.\nThe proposed method is simple but novel and unprecedented and seems reasonable.\nIn particular, the most strong point is that the analysis is comprehensive for a very large number of models. \n", "weaknesses": "It seems more compatible with a computer vision conference than a machine learning conference.\nThe definition of spuriosity in Sec 4.1 is a bit ad hoc.\nThe ending without discussion or conclusion is a bit of a dead end in terms of the paper structure.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their kind words. We now address each concern.\n\n**On the modality-agnostic potential of Spuriosity Rankings**. While we demonstrate our method on a fundamental task (classification) in a fundamental modality (vision), we believe our framework and the lessons obtained by them are relevant for all in the ML community. \n\nWe demonstrate that the tendency of ML models to specialize the function of deep neural nodes can be leveraged to scalably sort data, specifically by finding such nodes that focus on spurious features. We also show the power of sorting data, to various ends, including interpreting spurious features, measuring bias caused by them, and fixing those same biases. We hope our work will inspire people from all subfields to consider sorting their data, namely by re-using existing models.\n\nOur large-scale empirical study also sheds important insight to the spurious correlation problem, which is pervasive in all ML subfields. Namely, we learn that since diverse models seem to inherit the same biases, data-centric approaches are needed. Further, we detail the crucial class-dependence in defining and handling spurious correlations. Since most existing research around spurious correlations (across fields) is model/algorithm-centric and class-agnostic, we hope our findings will be novel and insightful to a wide audience.   \n\nWhile implementation details (i.e. how to interpret and utilize spurious concept detectors inside existing models) will vary, we believe that Spuriosity Rankings at its core is modality-agnostic, and thus potentially impactful to anyone studying spurious correlations. \n\n**Conclusion**. We will add a conclusion in the updated draft (where we will have an extra page) so to tie together the whole work, with emphasis on higher level takeaways that apply to wider audiences, like the ones mentioned above.  \n\n**Spuriosity approximation in Sec 4.1**. We note that we offer just one way to approximate spuriosity, which is in many ways qualitative. We provide more detailed explanations of the notion of spuriosity and its implications in the introduction (L30-37). We approximate spuriosity quantitatively by averaging normalized activations of robust neural features annotated as detecting spurious features. We normalize these activations so that one neural feature does not dominate others in the case that feature activation distributions vary significantly (i.e. the long-tailed case the reviewer describes). Moreover, since we ultimately wish to rank images, we aim to average the ‘percentile’ an image falls w.r.t. activation on any of the spurious neural features, which is proxied by the normalized activation we employ. At the suggestion of the reviewer, we inspect activations of neural features in a diverse model set, finding that they all generally follow a normal distribution on their right tail. Further, within each model, activation variances are similar across neural features. Thus, while we thank the reviewer for drawing attention to a potential shortcoming, we feel comfortable in the reliability of our method. Nonetheless, spuriosity can be approximated in different ways, and we are eager to see how others in the community will do so in future work. \n\n**Quality of employed interpretability methods**. We employ an interpretability method with significant precedent [42, 40 in original references], and further conduct extensive human validation of our interpretations (detailed in Appendix F). We agree with the reviewer that the success of our approach relates to the quality of interpretability methods employed, and we argue that this is a strength, as our method will continue to improve as models become more interpretable, which due to increased regulation, we believe is a certainty in the future. "}, {"review_id": "EjSZStRMbS", "reviewer": "Reviewer_y13p", "summary": "Shorcut learning has received increasing attention from the community recently. The paper proposes measuring and ranking data by \"spuriosity,\" or the degree to which relevant spurious cues are present, as a way to detect and mitigate biases in deep models that arise from their tendency to rely on spurious correlations in the data. The authors use an interpretable model to identify neural features relevant to a class and then select the spurious ones based on limited human supervision. Ranking data by the activation of these spurious features yields many benefits, like revealing less biased subsets, quantifying model bias via \"spurious gaps\" in accuracy between high and low spuriosity data, and finetuning models on less spurious data to obtain robust performance. Analyzing spurious gaps for 89 models reveals that all underperform on less spurious data, indicating that bias stems more from what data models are trained on rather than how they are trained, and that spuriosity rankings provide an efficient and interpretable method to complement model-centric bias mitigation approaches.\n", "questions": "None", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper works on a timely and important problem. The paper is overall well written (except its dependence on a related work [42]) and I enjoy reading it.\n2. It provides a scalable method to discover and rank data by how much spurious features they contain, i.e., spuriosity, which can help detect and mitigate biases in models.\n3. This paper proposes to finetune models on low spuriosity data to improve performance on less biased instances while maintaining overall accuracy, resulting in more robust and more stable models.\n4. Experiments results are insightful. The analysis of spurious gaps for many models indicates that biases stem more from the data than how models are trained, suggesting that data-centric approaches like this paper's can complement standard model-centric bias mitigation techniques.\n", "weaknesses": "1. One major concern is that this work is heavily based on a related work [42], which seems like from the same team of this submission. This makes the writing of this paper less friendly for readers not familiar with [42]. The authors are encouraged to revise the writing to make it more self-contained.\n2. In line 176-177, the authors claim that “we are the first to uncover this racial bias in the Celeb-A benchmark”. Actually, this racial bias issue in Celeb-A has been well studied in various literature.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and comments. Also we appreciate the reviewer for saying the problem we tackle is ‘timely and important’, our ‘results are insightful’, and ‘the paper is well written’. We are very happy to hear you ‘enjoyed reading it’.\n\n**Clarity**\nWe thank the reviewer for suggesting how to further improve our presentation, specifically w.r.t. explaining the mentioned prior work, from which we utilize their spurious feature discovery method. We note that we made multiple intentional efforts to make our work self-contained; we draw attention to these now. In the main text, we explain the method from prior work in several spots, each time providing more details: first at a high level in the introduction (L37-41), then in more depth in the related literature (L113-118), then in extensive detail in Section 3.1 (L132-148), where we also later explain our novel contributions atop the prior work (L155-161). Second, we design multiple visualizations to explain the framework, including images demonstrating specific methodologies from [42] (figures 2 and 3). Lastly, we note that we discuss the mentioned prior work in great detail in Appendix F, which includes a subsection (F.1) intended precisely to review the prior work so as to keep our paper self-contained.\n\nTo further aide the extensive explanations we detail above, we will add the following to the revised draft:\nMore explicit mentions to Appendix F to guide readers who desire more details.\nAn additional appendix section that serves as a taxonomy for terms like ‘robust neural feature’, ‘feature attack’, ‘neural activation map’, etc. \nWe hope these amendments will improve the presentation of the paper in the eyes of the reviewer, and if they do, we humbly ask that the reviewer increase their score for ‘presentation’. \n\nLastly, we highlight that while we utilize a method from the prior work, we make several novel and impactful contributions atop it, detailed in L155-161 and at length in Appendix F.3. \n\n**Racial Bias in Celeb-A Hair Classification**\nWe are unaware of any mentions of racial bias in the Celeb-A hair classification task in prior work. We conducted an additional review of literature, and were unable to find such work. While [1] mentions that Celeb-A likely has a bias toward images of white people, we note that this bias is different than the one we claim to discover. We refer to how “a model may spuriously associate brown skin with the brown hair class, 176 likely leading to failures for blond-haired brown-skinned people” (L175-176), and only discuss Celeb-A in the context of the hair classification task, a known benchmark in spurious correlation literature [2]. If the reviewer has a different citation in mind, we would be more than happy to include the citation and walk back the claim. Furthermore, we will change ‘Celeb-A benchmark’ to ‘Celeb-A hair classification benchmark’ to make this even clearer.  \n\n[1] FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation, Karkainen et al, WACV 2021 \n\n[2] WILDS: A Benchmark of In-the-wild Distribution Shifts, Koh et al, ICML 2021"}]

Author rebuttal:
We sincerely thank all reviewers for their time and insightful comments. A couple reviewers wonder if the involvement of a human in our framework is a limitation. We provide extensive discussion below, where we argue that having a human in the loop is a strength, though it is not necessary, as automated alternatives are completely feasible within our proposed Spuriosity Rankings framework. 

We would also like to draw attention to the four small additional small experiments/validations conducted during the rebuttal period. 
i. We add a new baseline inspired by a common family of approaches in bias mitigation, where a model upweights error samples. We find that this tuning does not close spurious gaps and reduces validation accuracy more than our proposed method.
ii. We perform a small scale human study to validate that low spuriosity images do not contain the relevant spurious cues detected by robust neural features annotated as spurious.
iii. We leverage ImageNet ReaL labels to quantitatively verify that the rate of label noise is significantly higher amongst high spuriosity images from classes with strong negative spurious gaps, as mentioned in Section 5.
iv. We demonstrate strong preliminary signal that an automated version of Spuriosity Rankings leveraging recent VLMs and LLMs can very efficiently extract interesting model biases (details below). 

We are grateful that the reviewers see the merits of our simple yet powerful idea to sort existing data using existing models so that we can better utilize them, towards more reliable/less biased ML models. We hope our rebuttals have addressed most of the reviewer comments. Thank you. 

**On human involvement**
We believe that human involvement is a strength of our framework, as it increases transparency. Namely, the human in the loop is given a concise inside look on the cues a model trained on the given data is likely to rely upon. Moreover, the human is given agency to decide which cues model performance should be invariant to. Next, we note that the level of human involvement is relatively low. Given the complexity of the task of sorting thousands of images within a class, we believe that only requiring a human to inspect a handful of images is quite efficient. With an appropriate UI, we can confirm that this takes no longer than about a 30-40 seconds per class. 

However, in cases where minimizing cost is preferred, Spuriosity Rankings can absolutely be automated. One way to do so is to automate the annotation of neural features as core or spurious. Specifically, one could automatically segment the class object with open-vocabulary segmentation models [1], and then compute the amount of saliency placed on the object by the neural activation map. If most of the salient pixels for a neural feature lie outside of the image region containing the class object, one could automatically flag such a feature as spurious. Indeed, a similar idea is explored with promising early signs in this recent workshop paper [2].

Another way to automate Spuriosity Rankings is to leverage vision-language models (VLMs) like CLIP [3]. With VLMs, we can compute the similarity of an image to any concept, encoded directly from text. After sorting images by similarity to spurious concepts, we can inspect the Spurious gap to measure bias, and train on low spuriosity images to mitigate bias. Further, one can utilize an LLM to automatically generate spurious concepts that a model may rely upon. To demonstrate the feasibility of this extension, we perform it to discover spurious correlations for CLIP. Specifically, we ask Vicuna-13b ‘List 16 different locations in which a {classname} may appear in an image’ and 'In an image of a {classname}, list 16 other objects that may also appear' for all classes in ImageNet, so to extract common backgrounds and co-occurring objects (i.e. potential relevant spurious cues). We then rank images within their class by spuriosity (measure by similarity of image embedding to the text embedding of ‘a photo of a {LLM inferred potential spurious concept}’), compute spuriosity gaps, and inspect class-concept pairs with the highest gaps. Indeed, the framework detects numerous interesting biases, such as ‘clay’ for the class potter’s wheel and ‘credit cards’ for the class wallet. Moreover, inspection of the images with most and least spuriosity confirm that Spuriosity Rankings computed in this manner also can extract from one’s own data natural examples and counterfactuals of the spurious correlation (e.g. potter's wheel images with and without clay). See the attached pdf to inspect these images. 

Note that the underlying mechanism of Spuriosity Rankings is preserved: we utilize the representation space of an interpretable model to quantify the presence of relevant cues, and rank images by spuriosity (proxied by similarity/activation of concept directions in representation space) to enable interpretation of spurious cues in context and measure model bias caused by them. While we believe the human in the loop of our presented framework offers some advantages, we appreciate the reviewers' questions regarding automated alternatives, are confident that they exist, and will include some discussion of them in the revised draft. 

[1] Segment Anything, Kirillov et al, 2023
[2] Identifying and Disentangling Spurious Features in Pretrained Image Representations, Darbinyan et al, 2023
[3] Learning Transferable Vision Models from Natural Language Supervision, Radford et al, 2020


Meta-review: The paper suggest a new method for measuring spuriosity of data examples, namely ranking data points by the spurious features that they contain. To do this, the authors propose to use a method combining adversarial training, utilizing deep features indicating spurious correlations and manual labeling. This results in a method for sorting the data examples from low to high spuriosity. The authors show that this can be used to mitigate problems that arise from training on data with spurious correlations, identify minority subpopulations and combat class-wise bias.

It seems that after discussion with the authors, all the reviewers are supporting the paper for acceptance. Some of the reviewers were concerned about the requirement of using human feedback in the spuriosity evaluation framework, but to my understanding the response of the authors answered most of the concerns.

I will therefore recommend that the paper is accepted, and urge the authors to improve the paper based on the comments raised by the reviewers.
Decision: Accept (spotlight)