Salient ImageNet: How to discover spurious features in Deep Learning?
Authors: Sahil Singla
Soheil Feizi
Venue: ICLR 2022 Poster (2022)
Keywords: interpretability
failure explanation
debugging
robustness

Abstract: Deep neural networks can be unreliable in the real world especially when they heavily use {\it spurious} features for their predictions. Focusing on image classifications, we define {\it core features} as the set of visual features that are always a part of the object definition while {\it spurious features} are the ones that are likely to {\it co-occur} with the object but not a part of it (e.g., attribute ``fingers" for class ``band aid"). Traditional methods for discovering spurious features either require extensive human annotations (thus, not scalable), or are useful on specific models. In this work, we introduce a {\it general} framework to discover a subset of spurious and core visual features used in inferences of a general model and localize them on a large number of images with minimal human supervision. Our methodology is based on this key idea: to identify spurious or core \textit{visual features} used in model predictions, we identify spurious or core \textit{neural features} (penultimate layer neurons of a robust model) via limited human supervision (e.g., using top 5 activating images per feature). We then show that these neural feature annotations {\it generalize} extremely well to many more images {\it without} any human supervision. We use the activation maps for these neural features as the soft masks to highlight spurious or core visual features. Using this methodology, we introduce the {\it Salient Imagenet} dataset containing core and spurious masks for a large set of samples from Imagenet. Using this dataset, we show that several popular Imagenet models rely heavily on various spurious features in their predictions, indicating the standard accuracy alone is not sufficient to fully assess model' performance specially in safety-critical applications. Code is available at \url{https://github.com/singlasahil14/salient_imagenet}.

Published as a conference paper at ICLR 2022
SALIENT IMAGENET: How TO DISCOVER SPURIOUS FEATURES IN DEEP LEARNING?
Sahil Singla & Soheil Feizi University of Maryland, College Park (ssingla, sfei @mdd edu
(a) class: band aid, (b) class: space bar, (c) class: plate,
(d) class:
butterfly, (e) class: potter's feature: wheel, spurious fea- ture: vase, -21.54% %
spurious feature: spurious feature: spurious feature: spurious
fingers, -41.54%
keys, -46.15%
food, -32.31 %
flowers, 22..5%
Figure 1: Examples of spurious features discovered using our framework for Standard Resnet-50 model. In the top row, red color highlights regions with spurious features. The bottom row shows images generated by visually amplifying these features. Adding small amount of gaussian noise (with a 0.25) to spurious regions (red regions) significantly reduces the model accuracy (shown dark red captions) for subset of 65 images with the shown class labels.
ABSTRACT
Deep neural networks can be unreliable in the real world especially when they heavily use spurious features for their predictions. Focusing on image classifi- cations, we define core featurey as the set of visual features that are always a part of the object definition while spurious features are the ones that are likely to co-occur with the object but not part of it (c.g., attribute "fingers" for class "band aid"). Traditional methods for discovering spurious features either require extensive human annotations (thus, not scalable), or are useful on specific models. In this work, introduce general framework to discover subset of spurious and core visual features used in inferences of a general model and localize them a large number of images with minimal human supervision. Our methodol- ogy is based on this key idea: to identify spurious or core visual features used in model predictions, we identify spurious core neural features (penultimate layer neurons of robust model) via limited human supervision (c.g., using top 5 activating images per feature). We then show that these neural feature anno- tations generalize extremely well to many more images without any human su- pervision. We use the activation maps for these neural features as the soft masks to highlight spurious core visual features. Using this methodology, we intro- duce the Salient Imagenet dataset containing core and spurious masks for a large set of samples from Imagenet. Using this dataset, show that several popular Imagenet models rely heavily on various spurious fcatures in their predictions, in- dicating the standard accuracy alone is not sufficient fully assess model perfor-

---

Published as a conference paper at ICLR 2022
mance. Code and dataset for reproducing all experiments in the paper is available at
http:/aaaieimmmmmeeeeettte imagenet]
INTRODUCTION
The growing use of deep learning systems in sensitive applications such as medicine, autonomous driving, law enforcement and finance raises concerns about their trustworthiness and reliability in the real world. A key reason for the lack of reliability of deep models is their reliance on spurious input features (i.e., features that are not essential to the true label) in their inferences. For example, convolutional neural network (CNN) trained to classify camels from cows associates green pastures with cows and fails to classify pictures of cows in sandy beaches correctly since most of the images of cows had green pastures in the training set (Beery et al. 2018: Arjovsky et al. (020). Similarly,
Bissoto al. (2020) discovered that skin-lesion detectors use spurious visual artifacts for making predictions. The list of such examples goes on (de Haan et al. 2019 Singla et al. 2021).
Most of the prior work on discovering spurious features (Nushi et al. 2018 Zhang et al., 2018 Chung et al., 2019) requires humans to first come up with a possible list of spurious features. This often followed by an expensive ouman-guided, labeling of visual attributes which is not scalable for datasets with large number of classes and samples such as ImageNet. In some cases, this may even require the construction of new datasets with unusual visual attributes to validate the hypothesis (such as cows in sandy beaches discussed previously). To address these limitations, recent works (Singla et al. 2021: Wong et al. 202I) use the neurons of robust models as visual attribute detectors thereby circumventing the need for manual annotations. However, their applicability is either limited to robust models which achieve low clean accuracy or to models that achieve low accuracy on the training dataset. We discuss limitations of these methods in more detail in Section 2
In this work, we introduce a general methodology to discover a subset of spurious and core (non- spurious) visual attributes used in model inferences and localize them on large number of images with minimal human supervision. Our work builds upon the prior observations (Engstrom et al.] 2019) that for robust model, neurons in the penultimate layer (called neural features) often corre- spond to human-interpretabl visual attributes. These visual attributes can be inferred by visualizing the heatmaps (e.g. via CAM, Zhou al. (2016)) that highlight the importance of image pixels for those neural features (top row in Figure 1). Alternatively, one can amplify these visual attributes in images by maximizing the corresponding neural feature values via a procedure called the feature attack (Engstrom et al. (2019); (Singla et al. (2021), bottom row in Figure D.
Our framework (shown in Figure 2 is based on this key idea: to identify spurious or core visual attributes used for predicting the class i, we identify spurious or core neural features via limited human supervision. We define core visual attribute as an attribute for class that is always a part of the object definition and spurious otherwise. To annotate neural feature as core or spurious, we show only the top-5 images (with a predicted label of i) that maximally activate that neural feature to Mechanical Turk workers. We then show that these neural feature annotations generalize extremely well to top-k (with & >> 5) images with label that maximally activate that neural feature. For example, Imagenet, in Section 3.4 we show a generalization accuracy of 95% for A = 65 (i.e., a 13 fold generalization). Thus, by using neural features and their corresponding neural activation maps (heatmaps), with limited human supervision, we can identify a large subset of samples that contain the relevant spurious or core visual attributes. The neural activation maps for these images can then be used as soft segmentation masks for these visual attributes. We emphasize that the usual method of obtaining such a set would require the manual detection and segmentation of a visual attribute across all images in the training set and therefore not scalable.
We apply our proposed methodology to the Imagenet dataset: study using 232 classes of Imagenet and 5 neural features per class. For each neural feature, we obtained its annotation as either core (non-spurious) or spurious for the label. Out of the 232 x 5
we conducted a Mechanical Turk
1, 160 neural features, 160 were deemed to be spurious by the workers. For 93 classes, at least one spurious neural feature was discovered. Next, for each (class=i, feature=j)) pair, we obtained 65 images with class showing highest activations of the neural feature j and computed their neural activation maps. The union of these images is what we call the Salient Imagenet dataset. The dataset contains 52, 521 images with around 226 images per class on average. Each instance in the dataset is of the form (x, y, M MMM,,, MM) where y is the ground truth class and M M represent the set of
2

---

Published as a conference paper at ICLR 2022
aage
penultimate layer neurons (neural features)
limited human annotations
highly activating neuron
Robust Resnet-50
spurious per class
core
spurious
automatic annotation of spurious visual attributes
Figure 2: Our framework for constructing the Salient Imagenet dataset. We annotate neural features core (non-spurious) spurious using minimal human supervision. We next show that these annotations generalize to many more new samples. We highlight core and spurious regions on the input samples using activation maps of these neural features.
spurious and core masks, respectively (obtained from neural activation maps of neural features). This dataset can be used to test the sensitivity of any pretrained model to different image features by corrupting images using their relevant masks and observing the drop in the accuracy.
Idcally, we expect trained models to show a low drop in accuracy for coruuptions in spurious regions and a high drop for that the core regions. However, using a standard Resnet-50 model, we discover multiple classes for which the model shows trend contradictory nificantly higher spurious drop compared to the core drop. As an example, for the class drake (Fig. 5). discover that corrupting any of spurious features results in significantly higher drop (> 40%) compared to corrupting any of core features (< 5%). We show many more examples in Appendix For various standard models (Resnet-50,, Wide-Resnet--Se evaluate their accuracy drops due to cortuptions in spurious or core regions at varying degrees of corruption noise and find no significant differences between their spurious or core drops suggesting that none of the models significantly differentiates between core and spurious visual attributes in their predictions (Fig. (). In summary, our methodology has the following steps:
expectations: sig-
Efficientnet-ble Efficientnet-b7), we
Step 1:
We select neural features as penultimate-layer neurons of pretrained robust network and visualize them using a few highly activating samples (Section B.2).
Step 2: Using Mechanical Turk, we annotate neural features as core or spurious (Section 3.3
Step 3: Using neural feature annotations, we automatically annotate core and spurious visual
attributes on many more samples without any human supervision (Section 3.3).
Step 4: Using another Mechanical Turk study, we show that our methodology generalizes ex- tremely well identify core/spurious visual attributes
samples (.44))on[3.4)
Step 5: Applying steps 1-4 to Imagenet, we develop a new dataset called Salient Imagenet whose
samples, in addition to the class label, are annotated by core and/or spurious masks (Section
Step 6: Using such richly annotated dataset, assess core/spurious accuracy of standard models by adding small noise to spurious/core visual attributes, respectively (Section 5)
To the best of our knowledge, this is the first work that introduces general methodology discover core and spurious features with limited human supervision by leveraging neurons in the penultimate layer of robust model as interpretable visual feature detectors. Also, this is the first work that releases a version of Imagenet with annotated spurious and core image features. We believe that this work opens future research directions leverage richly annotated datasets to develop deep models that mainly rely on core and meaningful features in their inferences.
2 RELATED WORK
Robustness and Interpretabilit:: In recent years, there have been many efforts towards post-hoc interpretability techniques for trained neural networks. Most of these efforts have focused local
3

---

Published as a conference paper at ICLR 2022
explanations where decisions about single images are inspected ZZiierr & Fergus 2014: Yosinski
eet al.
2016; Dosovitskiy & Brox, 2016 Mahendran & Vedaldi 2016 Nguyen et al. 2016 Zhou
et al.
2018 Olah et 2018]Adebayy et al. 2018
et al. 20.9111991 [a199Carteree et al. |019 Ych et al.
2019
Sturmfels et al. "Shaugghhhsssssy et al. 2019, et al. 2020). These include saliency
al.] (017] (Singla et al. class activation maps (Zhou et al. [016]] Selvaraju et al. 2019, Bau et al. 22.20 Ismail et al.
maps
(Simonyan al. |2144] Smilkov et al. 2017] SSnddararajan
2019
2019
2020), surrogate models to interpret local decision boundaries such LIME (Ribeiro et al.
2016)
methods to maximize neural activation values by optimizing input images (N Nguyyen et al. 2015
Mahendran & Vedaldi 2013) and finding influential inputs (Koh Liang/ 2017). However, recent work suggests that class activation or saliency maps may not be sufficient to narrow down visual attributes and methods for maximizing neural activations often produce noisy visualizations (Olah) et a. 2017] (2188) with changes imperceptible to humans. To address these limitations, recent works (sipras et al.|aa0..]_1 [2018] EEgstrrom et al., 2019) show that for robust (Madry et al. (018)) or adversarially trained models (in contrast to standard models), activation maps are qualitatively more interpretable and optimizing an image directly to maximize a certain neuron produces human perceptible changes in the images which can be useful for visualizing the learned features.
Failure explanation: Most of the prior works on failure explanation either operate on tabular data where interpretable features are readily available (Zhang al., 2018 Chung et al. 2019)), language data where domain-specific textual queries can be easily generated (Wu et al. 2019: Ribeiro et al., 2020 images where visual attributes can be collected using crowdsoureing (Nushi al., 2018 Xiao et al., 2021 Plumb et al., (021) or photorealistic simulation (Leclerc et al. [021]. However, collecting visual attributes via crowdsoureing can be expensive for large datasets. Moreover, these methods require humans to hypothesize about the possible failures and one could miss critical failure modes when the visual attributes used by model are different from the ones hypothesized by humans.
To address these limitations, recent works (Wong et al. 202 Singla et al. (202T) use the neurons of robust models as the visual attribute detectors for discovering failure modes, thus avoiding the need for crowdsourcing. However, Wong et al. (202T) cannot be used to analyze the failures of standard (non-robust) models which achieve significantly higher accuracy and are more widely deployed in practice. Moreover, to discover the spurious feature, they ask MTurk workers to identify the com- mon visual attribute across multiple images without highlighting any region of interest. However, different images may have multiple common attributes even if they come from different classes and this approach may not be useful in such cases. Barlow (Singla et al.) (02I) learns decision trees using misclassified instances to discover leaf nodes with high failure concentration. However, the instances could be correctly classified due to some spurious feature (e.g., background) and Barlow will discover no failure modes in such cases. In contrast, our approach circumvents these limitations and discovers the spurious features for standard models even when they achieve high accuracy.
3 A GENERAL FRAMEWORK FOR DISCOVERING SPURIOUS FEATURES
Consider an image classification problem X - y where the goal is to predict the ground truth label y y' for inputs E X. For each class i V, we want to identify and localize a set of core or spurious visual altributes in the training set that can be used by neural networks to predict i:
Core Attributes are the set of visual features that are always part of the object definition.
Spurious Attributes are the ones that are likely to co-occur with the object, but not a part of
(such as food, vase, flowers in Figure
more formal discussion of these abstract definitions can be found in Appendix G Note that local- izing these visual attributes on a large number of images requires significant human annotations and can be very expensive. In this paper, we propose a general methodology to tackle this issue.
3.1 NOTATION AND DEFINITIONS
For trained neural network, the set of activations in the penultimate layer (adjacent to logits) is what we call the neural feature vector. Each element of this vector is called a neural feature. For an input image and neural feature,
obtain the Neural Activation Map or NAM, similar to CAM (Zouu et al. 2016), that provides a soft segmentation mask for highly activating pixels (for the neural feature) in the image (details in Appendix A.T. The corresponding heatmap can then be
4

---

Published as a conference paper at ICLR 2022
Image
Heatmap
Feature Attack
Image
Heatmap
Feature Attack
Figure 3: On the left, the heatmap suffices to explain that the focus on ground. But on the right, the heatmap covers both cougar and wires. Feature attack clarifies that the focus is on wires.
obtained by overlaying the NAM top of the image that the red region highlights the highly activating pixels (Appendix (A.)). The feature attack (Singla et al.] (201] ) is generated by optimizing
the input image increase the values of the desired neural feature (Appendix G
3.2 EXTRACTING, VISUALIZING AND SELECTING NEURAL FEATURES
Prior works EEnssrrm al. 2019 Singla al. 2021 provide evidence that neural features of robust model encode human interpretable visual attributes. Building on these works, to identify spurious core visual attributes used for predicting class these attributes. For each image in the Imagenet training set (Deng et al. 2009), we extract the neural feature vector using robust Resnet-50 model. The robust model ad adversarially trained using the l2 threat model of radius 3. Each neural feature is visualized using the following techniques:
identify neural features encoding
Most activating images: By selecting 5 images that maximally activate the neural feature, the common visual pattern in these images can be interpreted the visual attribute encoded in it.
Heatmaps (NAMs): When there are multiple common visual attributes, heatmaps highlight the region of interest in images and can be useful to disambiguate; e.g., in Figure 3 (left panels), heatmap disambiguates that the focus of the neural feature is on the ground (not water or sky).
Feature attack: In some cases, heatmaps may highlight multiple visual attributes. In such cases, the feature attack can be useful to disambiguate; e.g., in Figure (right panels), the feature attack indicates that the focus of the neural feature on the wires (not cougar).
However, the neural feature vector can have large size (2048 for the Resnet-50 used in this work) and visualizing all of these features per class to determine whether they are core or spurious can be difficult. Thus, and annotate them as core or spurious for í. Prior work (Singla et al. 2021) selects this subset by considering the top-k neural features with the highest mutual information (MT) between the neural feature and the model failure (a binary variable that is if the model misclassifies x and otherwise). However, the model could be classifying all the images correctly using some spurious features. In this case, the MI value between the (spurious) neural feature and the model failure will be small and thus this approach will fail to discover such critical failure modes.
select small subset of these features that are highly predictive of class i
In this work, we select subset of neural features without using the model failure variable. To do so, first select subset of images (from the training set) on which the robust model predicts the class i. We compute the mean of neural feature vectors across all images in this subset denoted by r(i). From the weight matrix of the last linear layer of the robust model, we extract the ith row w,,: that maps the neural feature vector to the logit for the class i. Next, compute the hadamard product F(i) O Wi.. Intuitively, the jth element of this vector (F(1) @ wi.)); captures the mean contribution of neural feature j for predicting the class i. This procedure leads to the following definition:
Definition 1 The Neural Feature Importance of feature j. for class i is defined TV (F(I) @ w..)) For class i, the neural feature with 1.th highest IV is said to have the feature rank k.
We thus select the neural features with the highest-5 importance values (defined above) per class.
3.3 MECHANICAL TURK STUDY FOR DISCOVERING SPURIOUS AND CORE FEATURES
To determine whether a neural feature j is core or spurious for class i, we conducted a crowd study using Mechanical Turk (MTurk). We show the MTurk workers two panels: One panel visualizes the
5

---

Published as a conference paper at ICLR 2022
feature j and the other describes the class i. To visualize j. we show the 5 most activating images (with predicted class i), their heatmaps highlighting the important pixels for j. and the feature attack visualizations (Engstrom et al. 2019). For the class i, we show the object names (Miller 1995), object supercategory (from siipaas et al. (2020), object definition and the wikipedia links. We also show 3 images of the object from the Imagenet validation set. We then ask the workers determine whether the visual attribute is a part of the main object, some separate objects or the background. They were also required to provide reasons for their answers and rate their confidence on a likert scale from 1 to 5. The design for this study is shown in Figure 9 in Appendix [ Each of these tasks (also called Human Intelligence Tasks or HITs) were evaluated by workers. The HITs for which the majority of workers selected either separate objects or background as their responses were deemed to be spurious and the ones with main object as the majority answer were deemed to be core. We note that our proposed approach might not be effective for discovering non-spatial spurious signals (such as gender/race) that are not visualizable using heatmaps/feature attacks.
Because conducting study for all 1000 Imagenet classes can be expensive, we selected smaller subset of 232 classes (denoted by 7) by taking a union of 50 classes with the highest and 50 with the lowest accuracy across multiple models (details in Appendix [B) For each class, we selected 5 neural features with the highest feature importance values resulting in 232 x 5 = 1160 HITs. For 160 HITs, workers deemed the feature be spurious for the label. Thus, for each class E T, we obtain a set of core and spurious neural features denoted by C(t) and S(i), respectively.
3.4 GENERALIZATION OF VISUAL ATTRIBUTES ENCODED IN NEURAL FEATURES
We show that the visual attribute inferred for some neural feature j (via visualizing the top-5 images with predicted class i as discussed in the previous section) generalizes extremely well to top-k (with >> 5) images with the ground truth label that maximally activate feature j. That is, we can obtain a large set of images containing the visual attribute (encoded in neural feature j) by only inspecting a small set of images. To obtain the desired set, denoted by j), we first select training images with label i (Imagenet contains 1300 images per label). Next, select h 65 images (5% of 1300) from this set with the highest activations of the neural feature j. We use the Neural Activation Maps (NAMs) for these images as soft segmentation masks to highlight the desired visual attributes.
To validate that the NAMs focus on the desired visual attributes, we conducted another Mechanical Turk study. From the set D(i.jj), we selected images with the highest and 5 with the lowest activa- tions of feature j. We show the workers two panels: The first panel shows images with the highest 5 activations (along with their heatmaps) and the second shows images with the lowest 5 activations (and heatmaps). We then ask them to determine if the focus of heatmaps in both visualizations was on the same visual attribute, different attributes or if either of the two visualizations was unclear. For each spurious feature (160 total), we obtained answers from 5 workers each. For 152 visualizations (95%), majority of the workers selected same as the answer, thereby validating that the NAMs focus on the same visual attribute. The design for the study is shown in Figure 10 in Appendix D
We emphasize that although using k = 65 significantly increases the size of the set (i.e., a 13 fold increase over the manually annotated set), the value of k can be changed depending the number of training images with the desired properties. One can quickly search over various possible values of k by visualizing the 5 images (and their heatmaps) with the highest and lowest activations of the relevant neural feature (in the set of & images) until they both focus on the desired visual attribute.
4 THE SALIENT IMAGENET DATASET
Recall that for each class i E T, we obtain set of core and spurious neural features denoted by C(I) and Si),, respectively (Section 3.3p. Then, using the generalization property (Section (.4), (.44, we obtain the set Di,,j) for each i E 7 and E C(i) U S(t). Each (ii,j) contains 65 images along with the NAM (per image) acting as the soft segmentation mask for the visual attribute encoded in feature j. We note these soft segmentation masks overlap with the desired visual attribute and may not cover it completely. However, since we corrupt these regions using Gaussian noise which pre- serves the content of the original images (discussed in Section 55) they can be useful for evaluating the model sensitivity to various spurious/core visual attributes. If j E C(i), these are called core masks, otherwise spurious masks. The union of all these datasets D(i,j) )) is what we call the Salient

---

Published as a conference paper at ICLR 2022
(a) Number of images for different classes with at least 1 spurious feature
Number df
23
ner
Numbber of spurious foetures
Reatare rink
(b) Number of classes vs. (c) Number of classes with sp- ber of spurious features
(d) Number common spurious neural features between classes
urious feature at various ranks
Figure 4: In the top row, we show number of images for different classes with at least spurious feature in the Salient Imagenet dataset. Bottom row shows various plots using the dataset.
Imagenet dataset. Each instance in the dataset of the form (x. y, M (M)) where y is the ground truth label while M /M represent the set of spurious/core masks for the image x. The dataset and anonymized Mechanical Turk study results are also available at the associated github repository,
The dataset contains 52, 521 images with around 226 images per class on average. From the MTurk
study (Section 3.3), 160 features were discovered to be spurious and 1, 000 to core for relevant classes. We visualize several spurious features (59 total) in Appendix J Examples of background and foreground spurious features are in Appendix I.I and I.2 respectively. For 93 classes, we discover at least one spurious feature (the number of images per class shown in Fig. 4a
We show
a histogram of the number of classes vs. the number of spurious features Fig. 4b For classes (namely space bar, miniskirt and seatbelt) all 5 features were found to be spurious. This finding
suggests that in safety critical applications, it can be unsafe to just use the test accuracy to assess the model performance because the model may be highly accurate on the benchmark for all the wrong reasons. We plot the histogram of the number of classes with spurious feature at rank (from 1 to 5) in Fig. [4c] We discover that across all feature ranks, significant number of spurious features are discovered (minimum 19) and larger number of spurious features are discovered at higher feature ranks (4 and 5). This suggests that inspecting larger number of features per class (i.e., > 5) may be necessary for a more thorough analysis of spurious features. In Fig. 4d we show the number of shared spurious features between different classes (diagonal zeroed out for brevity). By visualizing common neural features, we diagnose that tree branches are a confusing visual attribute between titi and junco (Appendix K.T); food between plate and icecream (Appendix
EVALUATING DEEP MODELS USING THE SALIENT IMAGENET DATASET
In this section, we use the Salient Imagenet dataset evaluate the performance of several pretrained models on Imagenet. In particular, each set Dii.j) can be used to test the sensitivity of any trained model to the visual attribute encoded with feature j for predicting the class i. We can either blackout (fill with black color) or corrupt (using some noise model) the region containing the attribute (using the soft segmentation masks) for all images in this set and evaluate the accuracy of the model on these corrupted images. However, because the existing Imagenet pretrained models have not explicitly been trained on images with black regions, these images can be characterized as out of distribution due to the
of the training set. Thus, it may be difficult to ascertain whether changes in accuracy distribution shift or the removal of spurious features. Instead, we choose to corrupt the region of interest using Gaussian noise to preserve the content of the original images (which can be useful
7

---

Published as a conference paper at ICLR 2022
(a) Core, -1.5%
(b) Core, -4.6 %
(c) Core, %
(d) Spurious,441.57 (e) Spurious,-46.1. 1%
Figure 5: Each image denotes the heatmap for different neural features for the class drake. Using standard Resnet-50 model, (clean accuracy of 95.4%), observe drop of at least 41.5% by adding gaussian noise with o = 0.25 to spurious masks while a drop at most 4.6% for core masks. This shows that the model heavily relies on spurious features in its predictions for this class.
when the soft segmentation masks are inaccurate). Such Gaussian noise also known occur in the real world due to sensor electronic circuit noise. Since the segmentation masks we derive are soft (i.e., their values lie between and 1, not binary), we use these masks to control the degree of the Gaussian noise corruption across the original image. That is, given image and mask m, add noise z ~ NO,, I) using the equation below (examples in Appendix
(1) where a is hyperparameter that can be used to control the degree of corruption in the image. The above equation ensures that regions where the mask m has values close to are corrupted by noise with the desired standard deviation a and the ones with values close to 0 suffer little changes.
x + a ④ m),
5.1 CORR AND SPURIOUS ACCURACY
We now introduce concepts core and spurious accuracy to evaluate deep models. Informally, the core accuracy is the accuracy of the model only due to the core features of the images and the spurious accuracy is due to the spurious regions. To compute the core accuracy for class i, we first obtain the sets Dii,j). . For some image x E D(i, j). let m be the mask obtained for some feature Next, we take the union of all these sets for E S(i) denoted by DS(i). Note that for some images in the union, we may have multiple spurious masks but to evaluate the core accuracy, we want to obtain single mask per image that covers all the spurious attributes. Thus, for the image x, the (p. g)-th element of the single spurious mask s (or, the single core mask c), is computed by taking the element-wise over all the relevant masks:
P.g
JES(I), XED(.J)
max
(i) m
p.qa
Cp.4
JEC(),, XED().)) P
max
I (7)
(2)
Definition (Core Accuracy) We define the Core Accuracy or (c) as follows:
(C)
Σ ()) NET
acc ())))
Σ
1(b(x+
[T|
[DS(1)
xEDS(()
We acknowledge that definition of core accuracy is incomplete because the set of spurious vi- sual attributes discovered using our framework may not cover all spurious attributes in the dataset. Finally, note that the defined accuracy is function of the noise parameter (a) used to cor- rupt spurious regions. The Spurious Accuracy or acc(s) DS(1) with DC(i) = (ivec)D((i,jj and s with e in equation?? 2 Observe that DS(i) A DC(i) in general because D(i.j) may contain different sets of images for different j's. Thus, the standard accuracy of the model (i.e. without adding any noise) on the two sets pf DS(i) and DC(i) can be different. We want our trained models to show a low degradation in performance when noise is added to spurious regions (i.e. high core accuracy) and high degradation when corrupting core regions (i.c. low spurious accuracy).
(i) can be computed similarly by replacing
5.2 RESULTS FROM EVALUATING DEEP MODELS ON THE SALIENT IMAGENET DATASET
In this section, we test the sensitivity of several models various spurious/core features.
8

---

Published as a conference paper at ICLR 2022
ioor: Strrcari) Accuncy
daeo Standend Acorasy Sourieas Standard Acoracy
(Cee: Standinn ACOUKY
Kan Standati Acuucy
Beeutous Starent Accurcyy
(Seuroas Standard AcCuRCC
22
Soumios Standard Acor
100
-00
2-3
0.0
7.0
0.0
0.3
Io U-
deunn be
Unurshel 253
slets bcorsdee
eleck houendee
Unanshul PSt
(a) Resnet-50
(b) Wide-Reseie Ridle-Resnet50--
(c) Efficienttec-be
(d) Efficientnet-bb
Figure 6: In the top row, we observe similar drop in accuracy irrespective of whether noise is added to spurious or core regions suggesting that trained models do not differentiate between the spurious and core regions for predicting the desired class. In the bottom row, we show core accuracy vs. noise parameter a for several classes. In particular, for both classes "triumphal arch" and "ostrich", the model has a standard accuracy of 100% (at o 0). However, the core accuracy for triumphal arch is high even at a = 2.0 (>= 40%) while that for ostrich is almost 0%.
Testing model sensitivity to spurious features. We test the sensitivity of standard Resnet-50 model to different spurious and core features (one feature at a time) by corrupting images from the datasets (i..j) and evaluating the drop in model accuracy. We use = 0.25 (equation I because we observe that it preserves the content of the images (Examples in Appendix D so we would expect the model prediction to remain unchanged. In Figure [ we show multiple examples of spurious features discovered using this procedure. In Figure 5 we show a class (namely drake) which the model has a high clean accuracy (95.4% 49). However, it exhibits a large drop in performance when any of the spurious regions (e.g. water) are corrupted while small drop is observed when any of the core regions (e.g. bird's head) are corrupted. This highlights that the model relies on mostly spurious features in its predictions for this class. Out of the 93 classes with at least one spurious feature, we discovered 34 classes such that the accuracy drop due to some spurious feature was at least 20% higher than due to some core feature. In Appendix L we show visualizations of core and spurious features for 15 of these classes. On 11 of these 15 classes, the model has >> 95% accuracy. Since the la norm of noise perturbations can be different for different features (e.g. due to different l2 norms of their activation maps), we also evaluate accuracy drops due to corrupting spurious/core features when the mean 12 perturbations are similar and find similar trends (details in Appendix [.)
Comparing core and spurious accuracy of trained models. Next, we compute the core and spuri- ous accuracy of four standard (i.e. not adversarially trained) pretrained models namely, Resnet- 50 (He et al.] 2015), Wide Resnet-50-2 (Zagoruyko & Komodakis 2016). Efficientnet-b4 and Efficientnet- (Tan& & Le 2019). In Figure (top row), we plot the drop in accuracy when noise is added to spurious regions (i.e., core accuracy standard accuracy) and also core regions (i.e., spurious accuracy standard accuracy). We observe a similar drop in performance for all values of and for all trained models. This suggests that trained models do not differentiate between spurious and core regions of the images for predicting an object class, In some sense, this is expected because in the standard Empirical Risk Minimization (ERM) paradigm, models have never explicitly been trained to make such differentiation, suggesting that providing additional supervision during train- ing (for example, segmenting core/spurious regions as provided in Salient Imagenet or diversifying the training dataset) may be essential to train models that achieve high core accuracy. In Figure (bottom row), we plot the core accuracy for different classes as the noise level a increases. In particular, consider the two classes "triumphal arch" and "ostrich" that both have standard accu- racy of 100% (at a = 0). However, the core accuracy for triumphal arch is high even at a = 2.0 (<= 40%) while that for ostrich is almost 0%. This provides further evidence that the standard accuracy alone is not a reliable measure for model performance in the real world because the core accuracy for two classes with the same standard accuracy can be very different. We believe having richly annotated training datasets such as Salient Imagenet can lead to training reliable deep models that mainly rely on core and informative features in their predictions.
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
This project was supported in part by NSF CAREER AWARD 1942230, a grant from NIST 60NANB20D134, HR0011 1950026.GRDD-F--02. ONR YIP award N00014-22-1-2271, Army Grant W91 INF211220076 and AWS Machine Learning Research Award. We would like to thank Besmira Nushi, Eric Horvitz and Shital Shah for helpful discussions.
REPRODUCIBILITY
We provide the design details of our Mechanical Turk studies in Appendix Sections C and D The code for user studies, experiments and the dataset is available at https::pss //tp://ghhttb com/ singlasahi 114 ieeeeeennniinneeeeeeeeee
ETHICS STATEMENT
This paper introduces scalable framework for discovering spurious features in predictions made by deep neural networks. Our approach can be useful for various downstream tasks such as model debugging and developing improved models that rely on core features their predictions. We intro- duce a new dataset called Salient ImageNet that includes annotations of various core and spurious visual attributes for a subset of images in the ImageNet training set. We do collect any new samples for constructing our dataset. As this work paves the way to develop reliable models, we do see any foreseeable negative consequences associated with our work.
REFERENCES
Julius Adebayo. J. Gilmer, M. Muelly, Ian J. Goodfellow, M. Hardt, and Been Kim. Sanity checks for saliency maps. In NeurIPS, 2018.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization, 2020.
David Bau, Jun- Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Tor- ralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 117748)330711330078 2020. ISSN 0027-8424. doi:
N0.1073/ppaas.1073351117 URLhttps: / /www pnas org/ wwwt:///ttteteeceeooo/eee 1atptt//11/////00111
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. CoRR, 0bs//187.044 2018. URL
wttp://www..rga///////////// 04975
Alceu Bissoto, E. Valle, and S. Avila. Debiasing skin lesion datasets and models? not so fast. 2020 IEFF/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp- 3192-3201, 2020.
Shan Carter, Zan Armstrong. Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. Distill, 2019. doi: 10.23915/diittil.000055
http://illlllllt/////////ac////e atlas.
Chun-Har Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image clas- sifiers by counterfactual generation. In International Conference on Learning Representations, 2019. /RLLAttts///ppezevviee
Yeounoh Chung, Tim Kraska, Neoklis Polyzotis, Ki Hyun Tae, and Steven Euijong Whang. Slice finder: Automated data slicing for model validation. In 2019 IEEE 35th International Conference on Data Engineering (ICDE), pp. 1550-1553. IEEE, 2019.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. CoRR, abs/ Abs//905.1119 11979, 2019, URL ht arxiv. org/ abs 1905. 11979
J. Deng. Wei Dong, R. Socher, L. Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248- 255, 2009.
10

---

Published as a conference paper at ICLR 2022
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness prior for learned representations, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. The IEEE Conference on Computer Vision and Patern Recognition (CVPR), pp. 770-778, 2015.
Aya Abdelsalam Ismail, Mohamed K. Gunady, Luiz Pessoa, Héctor Corrada Bravo, and Soheil Feizi. Input-cell attention reduces vanishing saliency of recurrent neural networks, In NewrIPS, 2019.
Aya Abdelsalam Ismail, Mohamed K. Gunady, Héctor Corrada Bravo, and Soheil Feizi. Bench- marking deep learning interpretability time series predictions. In NeurlPS, 2020.
Fereshte Khani and Percy Liang. Removing spurious features can hurt accuracy and affect groups disproportionately. In Proceedings of the 2021 ACM Conference Fairness, Accountability, and Transparency, FAceT 221, pp. 196-205, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 450383097.
doi: 10. 10.114557442188834458888
URL https://doi.
oxg/10.1145 3442188.... 24 3445883
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, pp. 1885-1894. JoLR.org, 2017.
Guillaume Leclerc, Hadi Salman, Andrew Ilyas, Sai Vemprala, Logan Engstrom, Vibhav Vineet, Kai Xiao, Pengchuan Zhang, Shibani Santurkar, Greg Yang. Ashish Kapoor, and Aleksander Madry. 3db: A framework for debugging computer vision models. In Arxiv preprint arXiv:2106.03805, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
Aravindh Mahendran and A. Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal of Computer Vision, 120:233-255 255, 2016.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In 2015 IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pp. 5188- 5196, 2015. doi: 10.1109/CVPR22055.22999
George A. Miller. Wordnet: A lexical database for english. COMMUNICATIONS OF THE ACM, 38:39 41, 1995.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 427-436, 2015, doi: 10. 10.1109/CVPR2205552222888
Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. In ICML Workshop on Visualization for Deep Learning. 2016
Besmira Nushi, Ece Kamar, and Eric Horvitz. Towards accountable AI: hybrid human- human-machine machine
analyses for characterizing system failure. In Yiling Chen and Gabriella Kazai (eds.), Pro- ceedings of the Sixth AAA/ Conference on Human Computation and Crowdsoureing, HCOMP, pp. 135.135.10 AAAI Press, 2018. URL https: httt:s///aaiiooggooc////eetteee seccs/index..m php/HCOMP
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 10.23911/ddisiill00000
Chris Olah, Arvind Satyanarayan. Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and
Alexander Mordvintsev. The building blocks of interpretability. Distill, 2018. doi: 10,23915/ distill.00010. https://distill..atee pub/201 8/hulldigg-block

---

Published as a conference paper at ICLR 2022
Matthew O' Shhaughnsssy, Gregory Canal, Marissa Connor, Mark Davenport, and Christopher Rozell. Generative causal explanations of black-box classifiers. In NeurIPS, 2019.
Gregory Plumb, Marco Túlio Ribeiro, and Ameet Talwalkar. Finding and fixing spurious patterns with explanations. CoRR, abs/2106.02112 12, 2021. URL httpp////aaxxlv.o///bbb/////////
02112
Marco Tulio Ribeiro. Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '16, pp. 1135-1144, New York, NY. USA, 2016. Association for Computing Machinery. ISBN 9781450342322. doi: 10.1145/293967.. 2939778. URL httphttss://doi.orrr
1......1199991550 2939772
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Be- havioral testing of nlp models with checklist. In Association for Computational Linguistics (ACL), 2020.
R. R. Selvaraju, Abhishek Das, Ramakrishma Vedantam, Michael Cogswell, D. Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. Inter- national Journal of Computer Vision, 128:336-359, 2019.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In Workshop at International Confer- ence on Learning Reprexentations, 2014.
Sahil Singla, Eric Wallace, Shi Feng, and Soheil Feizi. Understanding impacts of high-order loss approximations and features in deep learning interpretation. In ICML, 2019.
Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. Understanding failures of deep networks via robust feature extraction. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. In ICML Workshop on Visualization for Deep Learning, 2017.
Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 2020. doi: 10.23915/distill.0000 http:///iistlllul22220ttttttttrt basslines.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017.
Mingxing Tan and Quoc Le. EfficientNer: Rethinking model scaling for convolutional neural net- works. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th ternational Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114. PMLR, 09-15 Jun 2019. URL btttps:/pprcceediie .omiinggsmml.. mmrr.
preas/v97/t.s
html
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, 2018
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. From imagenet to image classification: Contextualizing progress on benchmarks. In ArXiv preprint arXiv:2005.11295. 2020.
Sahil Verma, John Dickerson, and Keegan Hines. Counterfactual explanations for machine learning: A review, 2020.
Eric Wong. Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debug- gable deep networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Inter- national Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 101122551120 1116.. PMLR, 18-24 Jul 2021. URL attps: pectps://proceee http:a/:///ccceeingg.ittt mmr.
preas/v139/wone htm.lmll
12

---

Published as a conference paper at ICLR 2022
Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. Errudite: Scalable, repro- ducible, and testable error analysis. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 747-763, 2019.
Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise signal: The role of image backgrounds in object recognition. In International Conference on Learning Rep- resentations, 2021. URL htwps: /ttttpss///ppereeieeee net faret/forummaa hastt/foruumm1ggg333332W
Chih-Kuan Yeh, Cheng- Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep D. Ravikumar. On the (in)fidelity and sensitivity of explanations. In NeurIPS, 2019.
Jason Yosinski, Jeff Clune, Anh Mai Nguyen, Thomas J. Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. ICML Deep Learning Workshop, 2016.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference (BMVC). pp. 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. 30.87. URL https::/ doi ww..orgg////ooo 5244/C.30.2..
doi: 10.5244/C.
Matthew D. Zeiler and Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
Jiawei Zhang, Yang Wang, Piero Molino, Lezhi Li, and David S Ebert. Manifold: A modeel-ggnstic framework for interpretation and diagnosis of machine learning models. IEEE transactions on visualization and computer graphics, 25(1):364-37, 2018.
B. Zhou, A. Khosla, Lapedriza. A., A. Oliva, and A. Torralba. Learning Deep Features for Dis- criminative Localization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representations via network dissection. IEEE transactions pattern analysis and machine intelligence, 41(9): 2131-2145 2018
13

---

Published as a conference paper at ICLR 2022
Appendix
A VISUALIZING THE NEURAL FEATURES OF A ROBUST MODEL
A.1 NEURAL ACTIVATION MAP (NAM)
global average pooling
image
conv layers
penultimate layer (neural features)
activations
neural activation map
feature map of interest
neural feature of interest
normalize to (0,1)
resize to image
Figure 7: Neural activation map generation
Figure 7
describes the Neural Activation Map generation procedure. To obtain the neural activation map for feature j. we select the feature map from the output of the tensor of the previous layer (i.e the layer before the global average pooling operation). Next, we simply normalize the feature map between 0 and and resize the feature map to match the image size, giving the neural activation map.
A.2 HEATMAP
Heatmap can be generated by first converting the neural activation map (which grayscale) to an RGB image (using the jet colormap). This is followed by overlaying the jet colormap on top of the original image using the following few lines of code:
import cv2
def compute heatmaahaapppppp (img fam):
hm cv2 applyColorMap (np. uint8 (255 nam)
cv2. COLORMAPJET)
hm np. float32 (hm) 255
hm hm img hm hm np max (hm)
return hm
A.3 FEATURE ATTACK
In Figure we illustrate the procedure for the feature attack. We select the feature we are inter- ested in and simply optimize the image maximize its value generate the visualization. p is hyperparameter used to control the amount of change allowed in the image. For optimization, we gradient ascent with step size 40, number of iterations 25 and 500.
14

---

Published as a conference paper at ICLR 2022
global average pooling
penultimate layer (neural features)
image (x)
conv layers + activations
feature attack (xx)
fi(x*)
arg
max ||x x 21 Sp
neural feature of interest (i)
P is large that
7 to a human
Figure 8: Feature attack generation
B SELECTING THE CLASSES FOR DISCOVERING SPURIOUS FEATURES
Because conducting a Mechanical Turk (MTurk) study for discovering the spurious features for all 1000 classes of Imagenet can be expensive, we selected smaller subset of classes as follows. Using some pretrained neural network h, for each class i in Imagenet, we obtain groups of images with the label (called label grouping) and prediction (i.e h predicts i, called prediction grouping) giving 2000 groups. For each group, we compute their accuracy using the network h. For each grouping (label/predietion). we selected classes with the highest and 50 with the lowest accuracy giving 100 classes per grouping and take the union. We used two pretrained neural networks: standard and robust Resnet-50 resulting in total 232 classes.
C MECHANICAL TURK STUDY FOR DISCOVERING SPURIOUS FEATURES
The design for the Mechanical Turk study is shown in Figure We showed the workers two panels side by side. The left panel visualizes a neuron (from a robust resnet-50 model trained on Imagenet) and the right panel describes object class from Imagenet.
The left panel visualizing the neuron is shown in Figure 9a To visualize the neuron, we first select the subset of images for which the robust model predicts the object class (given the right). We show three sections: Top five images (five images with highest activation values of the neuron in the subset), Highlighted visual attributes (heatmaps computed using the neural activation maps as described in Section A.2) and Amplified visual attributes (feature attack computed by optimizing the top five images to maximize the neuron as described in Section A.3)
The right panel describing the object class is shown in Figure 9b. We show the object category name (also called synset) from the wordnet heirarchy (Miller 1995), object supercategory (from [Tipras et al. (2020)) object definition (also called gloss) and the relevant wikipedia links describing the object. We also show images of the object from the Imagenet validation set.
The questionnaire is shown in Figure 9c visual attribute (given on the left) is part of the main object (given on the right), some separate object the background of the main object. We also ask the workers to provide reasons for their answers and rate their confidence on a likert scale from 1 to 5. The visualizations for which majority of workers selected either separate object background as the answer deemed to be spurious. In total, we had 205 unique workers, each completing 5.66 tasks (on average). Workers were paid $0.1 per HIT, with average salary of $8 per hour.
We ask the workers to determine whether they think the
15

---

Published as a conference paper at ICLR 2022
Visual attribute
Top images
Highlighted visual attributes
Amplifled
Main object
Object category namets]i odm
Object superoateggry: invoterrrrtts
Object definitioee
butterfiis
Below you can find some images containing the object, Note that other possibly larger objects may also be present those images:
If you are still unsure about the object meaning, you may also consult the following Wikipedia pagebie telpac/in.wk wilk pod www.g///lllllaaloo Mil/iimmett
(b) Main object
(a) Visual attribute
Q1. Which of the following best describes the focus of the red region in the Highlighted visual attributes?
Amplfied visual attributes useful
groam the
multiple objects such these examples region overlap over multiple objects.
cougar
can linore Ampllfed attributes
The main object has the following namods): admiral
The main object balongs the following supercatogory: invrtterrrtts
background the main Examples: greenary
albatross, background
plich background player.
Focus other object (not the main object)
Examples:
pari bassinet Ibaby's bed), less
mailio (dress dancers) branch part bind. not cougar.
graffiti
freight flowors
Focus is on some part of the main object
Examples: nase of cougar, Wilakers of a sat. loog of Inbater, stnuctura of steel hridoe.
Q2. How confident are you about answer Q1?
Bi
90 Fairly
siignty
Compltely
Q8. Please roasonts) your
completing HIT. Detalled
(c) Questionnaire
Figure 9: Mechanical Turk study for discovering spurious features
16

---

Published as a conference paper at ICLR 2022
C.I QUALITY CONTROL
Only allowed workers with at least 95% approval rates and minimum of 1000 HIT' were allowed to complete our tasks. Additionally, short and generic answers Q3 were rejected.
D MECHANICAL TURK STUDY FOR VALIDATING HEATMAPS
The design for the Mechanical Turk study is shown in Figure TO Given subset of the training (constructed as descrihed in Section 3.4D, our goal is to validate whether the heatmaps focus on the same visual attribute for all images the subset.
Recall that this subset was constructed for each spurious feature discovered using the previous crowd study (Section C.. For each spurious feature, we already know both the object class and the neuron index of the robust model. We construct the subset of training set by first selecting the subset of images for which the label the object class. Note that this is different from the previous crowd study where the prediction of the robust model is the object class. Next, we select the top-65 images from this subset where the desired neuron has the highest activation values.
We showed the workers two panels side by side, each panel visualizing the same neuron. The left panel visualizing the neuron is shown in Figure [0a The visualization shows the images with the highest-5 neural activations in the subset and corresponding heatmaps. The right panel visualizing the neuron is shown in Figure TOb.. That shows the images with the lowest-5 neural activations in the subset and corresponding heatmaps.
The guestionnaire shown in Figure [0c] We ask the workers to determine whether they think the focus of the heatmap is on the same object (in both the left and right panels), different objects whether they think the visualization either of the sections is unclear... They were given the option to choose from these four options: same, different, Section A is unclear and Section B is unclear. Same as in the previous study (Section C), we ask the workers to provide reasons for their answers and rate their confidence
likert scale from 1 to 5.
The visualizations for which majority of workers selected same as the answer were deemed to be validated ie for this subset of 65 images, we assume that the neural activation maps focus on the same visual attribute.
17

---

Published as a conference paper at ICLR 2022
Highlighted visual attributes
Highlighted visual attributes
Images
Images
U
U
H
N
N
H
H
(a) Section A: Images with the highest activation
(b) Section B: Images with the lowest activation values for the neuron of interest in the subset.
values for the neuron of interest in the subset.
Q1. following describse
region Highlighted vieual attributes?
In both Section A and Section B, focus on the same visual attribute.
In Swation focus is on different visual attribute compared Section
The focus rod ragi unclear in Section A.
focus regi unclear Section
Q2. How confident are you about your answer to Q11?
(1)
80 Siighly
DI
(4)
Q3. Describe the visual attributes Section and Section B for your Q1? This question mandatory completing this HIT. Detalled answers will
(c) Questionnaire.
Figure 10: Mechanical Turk study for validating heatmaps
18

---

Published as a conference paper at ICLR 2022
E DISCOVERING SPURIOUS VISUAL ATTRIBUTES IN OTHER IMAGE DATASETS USING OUR FRAMEWORK
In this work, consider the problem of classification Imagenet and introduce methodology discover core/spurious features. Our framework can be applied to discover the spurious/core visual attributes of other image classification datasets using the following steps:
ii Train robust model using the l2 norm of some large radii (we use 3 in this work).
(ii) To discover spurious visual attributes for some class i in the dataset, select the subset of images from the training dataset on which the robust model predicts
(iii) Select neural features from the penultimate layer of the robust model with the highest importance values (as defined in Section D. We use h = 5 in this work. We note that other layers of the network may also provide informative neural features in some applications.
(iv) Visualize each neural feature using highly activating images, their heatmaps and feature attack (Section 3.2). We use m = this work. Note that both and m may be changed depending on the problem hand.
(v) Create a Mechanical Turk HIT per (class=i, neural feature=j) pair such that the HIT contains both the visualization for feature j and also the details about the class i (similar to Figure 9 and obtain worker annotations.
The HITs for which the majority of the workers annotate the neural feature to be spurious for class are deemed to the spurious visual attributes. Then, simply one can use any Heatmap methods such as CAM to automatically highlight core and spurious masks many other samples in the dataset. At this point, we recommend another MTurk study to validate the quality of highlighted samples.
core and spurious masks
F ANALYZING THE REASONS PROVIDED BY MECHANICAL TURK WORKERS
Understanding the reasoning behind the answers of MTurk workers is general and challenging problem in all crowd studies. To understand the reasons behind the answers provided by our MTurk workers, have taken the following steps: (i) workers were required give reasons for their answers to complete each task and (ii) they were explicitly instructed to use heatmaps (called high- lighted visual attributes in the study) for their answers and feature attack visualization (called amplified visual attributes the study) for their answers when the heatmaps are unclear. For the crowd study for discovering spurious features (Section 3.3) we have observed more than 1200 men- tions of the word "highlighted" and only 29 mentions of the word "amplified" suggesting that the workers use highlighted visual attributes (heatmaps) significantly more than the amplified visual tributes (feature attack). Some example descriptions provided by workers are given below in Figures and 22
19

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 11: Visualization of feature 625 for class barn spider (class index: 73).
Reasons for the answers provided by workers (all workers answered main object): (i) The red region in the Highlighted visual attributes focuses on the leg of the barn spider that is the main object, (ii) Legs of spider are mainly focused. (iii) Focus spider, (iv) The red is the legs of the barn spider, (v) Focus on another part of the main object, such as the barn spider's legs.
images
heatmaps
feature attack
Figure 12: Visualization of feature 957 for class ruffed grouse (class index: 82).
Reasons for the answers provided by workers (all workers answered main object): (i) The red region the Highlighted visual attributes focuses on the feathers of the ruffed grouse that the main object, (ii) focus is highlighting the bird's wing, (ii) The focus is on the feathers of the bird, (iv) Focus is on the bird, (v) All of the red parts are within the object
20

---

Published as a conference paper at ICLR 2022
G FORMAL DEFINITIONS OF CORE AND SPURIOUS VISUAL ATTRIBUTES
Core visual attributes
Spurious visual
Label
attributes
Figure 13: A simple graphical model relationship between core, spurious and label variables.
In the main text, have explained definitions of core and spurious visual attributes used in studies as following:
Core Attributes are the set of visual features that are always a part of the object definition.
Spurious Attributes the ones that are likely co-occur with the object, but part of (such as food, vase, flowers in Figure T.
In fact, defining these abstract concepts for Mturkers was quite challenging: For example, in our first user studies Mechanical Turk, have observed that defining spurious visual attributes as attributes that could be removed without changing the label of the object resulted in workers labeling attributes such "limbs" to be spurious for the class "dog" (because dog still be recognized when the limbs are removed). Thus, for the purpose of our user studies, we defined the core visual attributes the attributes that part of the relevant object definition. With this definition, "limbs" would be considered as core feature for the class "dog". Similarly, we have defined spurious visual attributes the attributes that are a part of the object definition.
Although not necessary in our empirical Mechanical Turk studies, one can define core and spurious visual attributes formally through simple graphical model depicted in Figure [3] (as explained in Khani & Liang (202I)) Let us consider L, C' and S as label, core and spurious random variables, respectively. This graphical model indicates that given C. and S are conditionally independent. That is, spurious variables are not essential in the object definition. This graphical model can be viewed through the lens of the Reichenbach's 's Common Cause Principle which indicates that since there is correlation (and not causation) relationship between S and L, they have common cause that renders them conditionally independent. We emphasize that even this more formal definition of core/spurious features is incomplete since in practice there may be other confounding factors ignored in the representation Figure
21

---

Published as a conference paper at ICLR 2022
H ADDITIONAL RESULTS FROM EVALUATING DEEP MODELS ON THE SALIENT IMAGENET DATASET
Resnet-18
Mobilenet-cc-large
Resnet-50
Pesnext30-3zkkd
Core Accuracy le 0.255
Vgg-19-bn
Moo_rresnet50-2
70
Inception-vB gongle
Mmassntl--
Googlenet
tffisientnet-bo
Shufflanetv2-xt
#fficientntt
Mobilenet-v2
uffisientnet-b/
30
40
80
Usual Accuracy
Figure 14: Comparing between the accuracy (using 0.25) and usual accuracy of different
Imagenet trained models.
100
100
Cara Accuracy
20
Cara Accuracy
80
80
60
0.0
0.5
1.3
1.5
2.0
0.
04
1.
1.5
20
0
0
(a) Resnet-50
(b) Wide-Resnet-500
100
100
Care Accuracy
Care Accuracy
1.3
(c) Efficientnet-be
(d) Efficientnet b7
Imagenet classes
triumphal arch ostrich
electric locomotive
coucal
little blue heron limpkin
jellyfish
missile
(e) Legend
We plot the core accuracy for different classes as noise level ar increases. For some classes, core accuracy high even large a, but for most it decreases rapidly (e.g., triumphal arch, jellyfish on Efficientnet-b7).
Figure 15:
22

---

Published as a conference paper at ICLR 2022
I EXAMPLES OF IMAGES FROM THE SALIENT IMAGENET DATASET CORRUPTED BY ADDING GAUSSIAN NOISE TO THE SPURIOUS REGIONS
images
0= -= 0.25
0=
Figure 16: Images randomly sampled from the set D( where i 810 (class index) and 325
(feature index). Class name: space bar.
images
0 -= 0.25
O= 0.5
Figure 17: Images randomly sampled from the set D) where i 626 (class index) and j 1986
(feature index). Class name: lighter.
23

---

Published as a conference paper at ICLR 2022
images
0= a= 0.25
0= 0.5
Figure 18: Images randomly sampled from the set D(i,) where 975 (class index) and j 516
(feature index). Class name: lakeside, lakeshore.
images
a= 0.25
0=5
Figure 19: Images randomly sampled from the set Di,,) where 809 (class index) and j = 895
(feature index). Class name: soup bowl.
24

---

Published as a conference paper at ICLR 2022
images
0= a= 0.25
-= 0.5
Figure 20: Images randomly sampled from the set D(i,) where -557 (class index) and j 961
(feature index). Class name: missile.
images
a= Q=0A 0025
0.5
0
Figure 21: Images randomly sampled from the set D(i,)) where 536 (class index) and j = 76
(feature index). Class name: dock.
25

---

Published as a conference paper at ICLR 2022
J EXAMPLES OF SPURIOUS FEATURES
J.1 BACKGROUND SPURIOUS FEATURES
images
heatmaps
feature attack
Figure 22:
Visualization of feature 230 for class sorrel (class index: 339). Train accuracy using Standard Resnet-50: 99.385%
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -12.308%.
images
heatmaps
feature attack
Figure 23: Visualization of feature 925 for class black swan (class index: 100).
Train accuracy using Standard Resnet-50: 98.769%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -4.115%
26

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 24: Visualization of feature 223 for class geyser (class index: 974).
Train accuracy using Standard Resnet-50: 97.923%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -20.0%.
images
heatmaps
feature attack
Figure 25: Visualization of feature 1468 for class oystercatcher (class index: 143).
Train accuracy using Standard Resnet-50: 97.769%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -21.539%
27

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 26: Visualization of feature 341 for class red breasted merganser (class index: 98).
Train accuracy using Standard Resnet-50: 97.195%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -12.308%
images
heatmaps
feature attack
Figure 27: Visualization of feature 1697 for class albatross (class index: 146).
Train accuracy using Standard Resnet-50: 96.615%
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -20.0%.
28

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 28: Visualization of feature 981 for class rock beauty (class index: 392).
Train accuracy using Standard Resnet-50: 96.285%
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -32.308%.
images
heatmaps
feature attack
Figure 29: Visualization of feature 820 for class water ouzel (class index: 20).
Train accuracy using Standard Resnet-50: 96.231%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -15.385%
29

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 30: Visualization of feature 535 for class water ouzel (class index: 20).
Train accuracy using Standard Resnet-50: 96.231%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -16.923%
images
heatmaps
feature attack
Figure 31: Visualization of feature 1475 for class coral fungus (class index: 991).
Train accuracy using Standard Resnet-50: 93.308%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -9.231%.
30

---

Published as a conference paper at ICLR 2022
J.2
FOREGROUND SPURIOUS FEATURES
images
heatmaps
feature attack
Figure 32: Visualization of feature 1556 for class jacamar (class index: 95).
Train accuracy using Standard Resnet-50: 97.000%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -21.538%.
images
heatmaps
feature attack
Figure 33: Visualization of feature 189 for class potter's wheel (class index: 739). Train accuracy using Standard Resnet-50: 95.615%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -21.538%
31

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 34: Visualization of feature 1832 for class desk (class index: 526).
Train accuracy using Standard Resnet-50: 69.769%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -7.692%.
images
heatmaps
feature attack
Figure 35: Visualization of feature 895 for class wok (class index: 909).
Train accuracy using Standard Resnet-50: 71.077%
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -46.154%
32

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 36: Visualization of feature 43 for class plate (class index: 923).
Train accuracy using Standard Resnet-50: 64.538%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -32.308%.
33

---

Published as a conference paper at ICLR 2022
J.3
SPURIOUS FEATURES FEATURE RANKS SPURIOUS FEATURES AT FEATURE RANK
J.3.1
images
heatmaps
feature attack
Figure 37: Visualization of feature 56 for class bullet train (class index: 466).
Train accuracy using Standard Resnet-50: 98.077%.
Drop in accuracy when gaussian noise is added the highlighted (red) regions: -1.338%
images
usted Mayarss
heatmaps
Mapanis
feature attack
Figure 38: Visualization of feature 925 for class red breasted merganser (class index: 98).
Train accuracy using Standard Resnet-50: 97.195%.
Drop in accuracy when gaussian noise is added the highlighted (red) regions: -3.077%.
34

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 39: Visualization of feature 1406 for class jellyfish (class index: 107).
Train accuracy using Standard Resnet-50: 98.077%
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -4.615%
images
heatmaps
feature attack
Figure 40: Visualization of feature 925 for class hippopotamus (class index: 344).
Train accuracy using Standard Resnet-50: 97.846%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -27.693%
35

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 41: Visualization of feature 123 for class band aid (class index: 419).
Train accuracy using Standard Resnet-50: 81.000%
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -41.538%
images
heatmaps
feature attack
Figure 42:
Visualization of feature 325 for class space bar (class index: 810).
Train accuracy using Standard Resnet-50:: 59.043%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -46.154%
36

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 43: Visualization of feature 1536 for class dock (class index: 536).
Train accuracy using Standard Resnet-50: 71.098%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -35.385%.
images
heatmaps
feature attack
Figure 44: Visualization of feature 421 for class promontory (class index: 976).
Train accuracy using Standard Resnet-50: 82.462%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -15.384%
37

---

Published as a conference paper at ICLR 2022
J.3.2 SPURIOUS FEATURES AT FEATURE RANK 2
images
heatmaps
feature attack
Figure 45: Visualization of feature 1772 for class gondola (class index: 576).
Train accuracy using Standard Resnet-50: 98.308%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -1.538%
images
heatmaps
feature attack
Figure 46: Visualization of feature 925 for class american coot (class index: 137).
Train accuracy using Standard Resnet-50: 97.538%.
Drop in accuracy when gaussian noise added to the highlighted (red) regions: -3.077%.
38

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 47: Visualization of feature 1753 for class rock beauty (class index: 392).
Train accuracy using Standard Resnet-50: 96.285%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -12.308%
images
heatmaps
feature attack
Figure 48: Visualization of feature 1986 for class lighter (class index: 626).
Train accuracy using Standard Resnet-50: 86.154%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -35.385%
39

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 49: Visualization of feature 895 for class soup bowl (class index: 809).
Train accuracy using Standard Resnet-50: 79.462%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -46.154%
images
heatmaps
feature attack
Figure 50: Visualization of feature 798 for class spider monkey (class index: 381).
Train accuracy using Standard Resnet-50: 74.154%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -26.154%.
40

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 51: Visualization of feature 516 for class dock (class index: 536).
Train accuracy using Standard Resnet-50: 71.098%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -15.384%
images
heatmaps
feature attack
Figure 52: Visualization of feature 516 for class lakeside (class index: 975).
Train accuracy using Standard Resnet-50: 61.769%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -32.308%.
41

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 53: Visualization of feature 1832 for class space bar (class index: 810).
Train accuracy using Standard Resnet-50: 59.043%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -36.923%
42

---

Published as a conference paper at ICLR 2022
J.3.3
SPURIOUS FEATURES AT FEATURE RANK 3
images
heatmaps
feature attack
Figure 54: Visualization of feature 1797 for class sulphur butterfly (class index: 325). Train accuracy using Standard Resnet-50: 95.769%.
Drop in accuracy when gaussian noise added to the highlighted (red) regions: -18.462%
images
heatmaps
feature attack
Figure 55: Visualization of feature 481 for class gondola (class index: 576).
Train accuracy using Standard Resnet-50: 98.308%.
Drop in accuracy when gaussian noise added to the highlighted (red) regions: -6.154%
43

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 56: Visualization of feature 432 for class potter's wheel (class index: 739).
Train accuracy using Standard Resnet-50: 95.615%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -20.0%.
images
heatmaps
feature attack
Figure 57: Visualization of feature 1287 for class lighter (class index: 626).
Train accuracy using Standard Resnet-50: 86.154%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -10.77%
44

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 58: Visualization of feature 840 for class soup bowl (class index: 809).
Train accuracy using Standard Resnet-50: 79.462%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -33.846%
images
heatmaps
feature attack
Figure 59: Visualization of feature 2000 for class wok (class index: 909).
Train accuracy using Standard Resnet-50: 71.077%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -38.462%
45

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 60: Visualization of feature 961 for class missile (class index: 657).
Train accuracy using Standard Resnet-50: 67.846%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -13.846%
images
heatmaps
feature attack
Figure 61: Visualization of feature 2025 for class plate (class index: 923).
Train accuracy using Standard Resnet-50: 64.538%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -36.923%
46

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 62: Visualization of feature 387 for class space bar (class index: 810).
Train accuracy using Standard Resnet-50: 59.043%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -29.231%.
images
heatmaps
feature attack
Figure 63: Visualization of feature 1120 for class titi (class index: 380).
Train accuracy using Standard Resnet-50: 58.692%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -38.462%
47

---

Published as a conference paper at ICLR 2022
J.3.4 SPURIOUS FEATURES AT FEATURE RANK 4
images
heatmaps
feature attack
Figure 64: Visualization of feature 1797 for class admiral (class index: 321).
Train accuracy using Standard Resnet-50: 99.846%
Drop in accuracy when gaussian noise added to the highlighted (red) regions: -3.077%
images
heatmaps
feature attack
Figure 65: Visualization of feature 595 for class sulphur butterfly (class index: 325). Train accuracy using Standard Resnet-50:: 95.769%.
Drop in accuracy when gaussian noise added to the highlighted (red) regions: -21.539%.
48

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 66: Visualization of feature 1642 for class lighter (class index: 626).
Train accuracy using Standard Resnet-50: 86.154%
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -27.693%.
images
CocaCo
heatmaps
Coca
feature attack
Figure 67: Visualization of feature 447 for class band aid (class index: 419).
Train accuracy using Standard Resnet-50: 81.000%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -43.077%
49

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 68: Visualization of feature 1296 for class soup bowl (class index: 809).
Train accuracy using Standard Resnet-50: 79.462%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -49.23%.
images
heatmaps
feature attack
Figure 69: Visualization of feature 76 for class dock (class index: 536).
Train accuracy using Standard Resnet-50: 71.098%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -24.616%.
50

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 70: Visualization of feature 2 for class desk (class index: 526).
Train accuracy using Standard Resnet-50: 69.769%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -16.923%
images
heatmaps
feature attack
Figure 71: Visualization of feature 1469 for class space bar (class index: 810).
Train accuracy using Standard Resnet-50: 59.043%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -15.384%
51

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 72: Visualization of feature 961 for class projectile (class index: 744).
Train accuracy using Standard Resnet-50: 53.538%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -23.077%
52

---

Published as a conference paper at ICLR 2022
J.3.5
SPURIOUS FEATURES AT FEATURE RANK 5
images
heatmaps
feature attack
Figure 73: Visualization of feature 36 for class stupa (class index: 832).
Train accuracy using Standard Resnet-50: 98.385%. Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -9.231% %.
images
heatmaps
feature attack
Figure 74: Visualization of feature 371 for class house finch (class index: 12).
Train accuracy using Standard Resnet-50:: 98.308%.
Drop in accuracy when gaussian noise added to the highlighted (red) regions: -24.615%
53

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 75: Visualization of feature 1556 for class bittern (class index: 133).
Train accuracy using Standard Resnet-50: 97.077%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -10.769%
images
heatmaps
feature attack
Figure 76: Visualization of feature 1541 for class junco (class index: 13).
Train accuracy using Standard Resnet-50: 97.000%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -9.231%.
54

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 77: Visualization of feature 96 for class pineapple (class index: 953).
Train accuracy using Standard Resnet-50: 94.538%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -1.539%
images
heatmaps
feature attack
Figure 78: Visualization of feature 961 for class sandbar (class index: 977).
Train accuracy using Standard Resnet-50: 71.923%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -60.0%.
55

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 79: Visualization of feature 510 for class space bar (class index: 810).
Train accuracy using Standard Resnet-50: 59.043%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -24.615%
images
heatmaps
feature attack
Figure 80: Visualization of feature 1541 for class titi (class index: 380).
Train accuracy using Standard Resnet-50: 58.692%.
Drop in accuracy when gaussian noise is added to the highlighted (red) regions: -30.769%
56

---

Published as a conference paper at ICLR 2022
K
DIAGNOSING THE CONFUSING VISUAL ATTRIBUTE BETWEEN DIFFERENT CLASSES
K.1
FEATURE INDEX: 1541
images
heatmaps
feature attack
Figure 81: Visualization for class junco (class index: 13).
images
heatmaps
feature attack
Figure 82: Visualization for class titi (class index: 380).
57

---

Published as a conference paper at ICLR 2022
K.2
FEATURE INDEX: 2025
images
heatmaps
feature attack
Figure 83: Visualization for class plate (class index: 923).
images
heatmaps
feature attack
RIN
Figure 84: Visualization for class icecream (class index: 928)
58

---

Published as a conference paper at ICLR 2022
L COMPARING BETWEEN THE ACCURACY DROPS DUE TO SPURIOUS AND CORE NEURAL FEATURES
For some class i, want to test the sensitivity of trained model the visual attribute encoded in the feature j for predicting the class i. To this end, we add Gaussian noise to highly activating regions for images in the dataset D(i, j) and evaluate the drop in the accuracy of the model being inspected for failure. In particular, we show the drop in the accuracy for different classes by perturbing both core and spurious neural features using Gaussian noise with o 0.25.
However, for some of the classes, given a value, the mean of noise perturbation magnitudes im- posed on images the dataset ji..j) (referred to "mean & perturbation") can be different for different features. This is because the activation maps of different neural features can have differ- ent /2 norms. Thus, for such classes, we also compute accuracy drops using multiple values of a: [0.30, 0.35. .... 0.60). Then, for the features whose mean (2 perturbations were smaller, choose the smallest value of a (referred to as o,) that results in a mean l2 perturbation similar to (or even slightly higher than) that of the other neural feature (that initially had higher mean l2 perturbation).
Thus, for each neural feature, we show the accuracy drops using both a = 0.25 and 0g in the figure captions in this section.
59

---

Published as a conference paper at ICLR 2022
CLASS NAME: OSTRICH, TRAIN ACCURACY (STANDARD RESNET-50): 98.615% L.1
images
heatmaps
feature attack
Figure 85: Visualization of feature 1964 for class ostrich (class index: 9).
Feature label annotated by Mechanical Turk workers:
Using a = 0.25, drop in accuracy: 0.0%, mean la perturbation: 35.782. Using 0.35, drop in accuracy: -6.154, mean perturbation: 47.815.
images
heatmaps
feature attack
Figure 86: Visualization of feature 63 for class ostrich (class index: 9). Feature label as annotated by Mechanical Turk workers: spurious. Using 0.25, drop in accuracy: -20.0%, mean l2 perturbation: 47.653. Using σ = 0.35, drop accuracy: -32.308% mean l perturbation: 63.911.
60

---

Published as a conference paper at ICLR 2022
L.2
CLASS NAME: BRAMBLING, TRAIN ACCURACY (STANDARD RESNET- 95.615%
images
heatmaps
feature attack
Figure 87: Visualization of feature 457 for class brambling (class index: 10).
Feature label annotated by Mechanical Turk workers: core.
Using a 0.25, drop in accuracy: -1.538%, mean la perturbation: 25.337.
Using 0.6, drop in accuracy: -21.538% mean la perturbation: 51.371.
images
heatmaps
feature attack
Figure 88: Visualization of feature 371 for class brambling (class index: 10).
Feature label as annotated by Mechanical Turk workers: spurious.
Using 0.25, drop in accuracy: -33.846%, mean la perturbation: 50.046.
Using σ = 0.6, drop in accuracy: -93.846%, mean la perturbation: 100.689.
61

---

Published as a conference paper at ICLR 2022
L.3
CLASS NAME: HOUSE FINCH, TRAIN ACCURACY (STANDARD RESNET-50): 98.308%
images
heatmaps
feature attack
Figure 89: Visualization of feature 1667 for class house finch (class index: 12).
Feature label annotated by Mechanical Turk workers: core.
Using a 0.25, drop in accuracy: -3.077% mean la perturbation: 33.214.
Using 0.4, drop in accuracy: -60.0%, mean la perturbation: 49.5.
images
heatmaps
feature attack
Figure 90: Visualization of feature 371 for class house finch (class index: 12).
Feature label as annotated by Mechanical Turk workers: spurious.
Using 0.25, drop in accuracy: -24.615% mean la perturbation: 49.474.
Using σ = 0.4, drop in accuracy: -60.0%, mean la perturbation: 73.833.
62

---

Published as a conference paper at ICLR 2022
L.4
CLASS NAME: BULBUL, TRAIN ACCURACY (STANDARD RESNET-50): 98.308%
images
heatmaps
feature attack
Figure 91: Visualization of feature 762 for class bulbul (class index: 16). Feature label annotated by Mechanical Turk workers: core.
Using a 0.25, drop in accuracy: -4.615%, mean la perturbation: 29.186.
Using 0.45, drop in accuracy: -29.231% %, mean l2 perturbation: 48.193.
images
heatmaps
feature attack
Figure 92: Visualization of feature 660 for class bulbul (class index: 16).
Feature label as annotated by Mechanical Turk workers: spurious. Using 0.25, drop in accuracy: -27.692%. mean la perturbation: 48.715.
Using σ = 0.4, drop in accuracy: -84.11%, mean la perturbation: 72.98.
63

---

Published as a conference paper at ICLR 2022
L.5
CLASS NAME: COUCAL, TRAIN ACCURACY (STANDARD RESNET-50) 97.308%
images
heatmaps
feature attack
Figure 93: Visualization of feature 170 for class coucal (class index: 91). Feature label annotated by Mechanical Turk workers: core.
Using a = 0.25, drop in accuracy: 0.0%, mean la perturbation: 30.588.
Using 0.5, drop in accuracy: -41.538% mean perturbation: 54.295.
images
heatmaps
feature attack
Figure 94: Visualization of feature 614 for class coucal (class index: 91). Feature label as annotated by Mechanical Turk workers: spurious. Using 0.25, drop in accuracy: -27.693% mean la perturbation: 54.854.
Using σ = 0.5, drop in accuracy: -92.308%, mean la perturbation: 96.12.
64

---

Published as a conference paper at ICLR 2022
L.6 CLASS NAME: JACAMAR, TRAIN ACCURACY (STANDARD RESNET-50); 97.0%
images
heatmaps
feature attack
Figure 95: Visualization of feature 1339 for class jacamar (class index: 95).
Feature label annotated by Mechanical Turk workers: core.
Using a = 0.25, drop in accuracy: 0.0%, mean la perturbation: 33.526.
Using 0.45, drop in accuracy: -38.462% mean l2 perturbation: 54.749.
images
heatmaps
feature attack
Figure 96: Visualization of feature 1556 for class jacamar (class index: 95).
Feature label as annotated by Mechanical Turk workers: spurious.
Using 0.25, drop in accuracy: -21.538%, mean la perturbation: 53.542.
Using σ = 0.45, drop in accuracy: -76.923% mean l2 perturbation: 87.339.
65

---

Published as a conference paper at ICLR 2022
L.7 CLASS NAME: DRAKE, TRAIN ACCURACY (STANDARD RESNET-50): 95.462%
images
heatmaps
feature attack
Figure 97: Visualization of feature 1113 for class drake (class index: 97).
Feature label annotated by Mechanical Turk workers: core.
Using a = 0.25, drop in accuracy: -1.538% mean la perturbation: 36.558.
Using 0.45, drop in accuracy: -16.923% mean l2 perturbation: 60.362.
images
heatmaps
feature attack
Figure 98: Visualization of feature 1339 for class drake (class index: 97).
Feature label as annotated by Mechanical Turk workers: core.
Using 0.25, drop in accuracy: -4.615% mean la perturbation: 32.946. Using σ = 0.45, drop in accuracy: -16.923% mean l2 perturbation: 54.731.
66

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 99: Visualization of feature 925 for class drake (class index: 97).
Feature label as annotated by Mechanical Turk workers: spurious. Using σ 0.25, drop in accuracy: -41.539%, mean le perturbation: 47.465. Using a = 0.45, drop in accuracy: -72.308% mean l2 perturbation: 78.273.
images
heatmaps
feature attack
Figure 100: Visualization of feature 341 for class drake (class index: 97). Feature label as annotated by Mechanical Turk workers: spurious. Using 0.25, drop in accuracy: -4.1154. mean la perturbation: 55.205.
Using a = 0.45, drop in accuracy: -86. 154%, mean la perturbation: 89.644.
67

---

Published as a conference paper at ICLR 2022
images
heatmaps
feature attack
Figure 101: Visualization feature 736 for class drake (class index: 97).
Feature label as annotated by Mechanical Turk workers: core.
Using a 0.25, drop accuracy: 0.0%, mean la perturbation: 28.878.
Using 0.45, drop in accuracy: -6.154% mean la perturbation: 47.661.
Using σ = 0.55, drop in accuracy: -7.692%, mean la perturbation: 55.554.
68

---

Published as a conference paper at ICLR 2022
L.8 CLASS NAME: OYSTERCATCHER, TRAIN ACCURACY (STANDARD RESNET-50): 97.769%
images
heatmaps
feature attack
Figure 102:
Visualization feature 618 for class oystercatcher (class index: 143).
Feature label annotated by Mechanical Turk workers: core.
Using a = 0.25, drop in accuracy: -1.538% mean la perturbation: 28.932.
Using 0.5, drop in accuracy: -12.308%, mean la perturbation: 52.0.
images
heatmaps
feature attack
Figure 103: Visualization of feature 1468 for class oystercatcher (class index: 143).
Feature label as annotated by Mechanical Turk workers: spurious.
Using 0.25, drop in accuracy: -21.539%. mean la perturbation: 51.153.
Using a = 0.5, drop in accuracy: -70.77%, mean /2 perturbation: 90.219.
69

---

Published as a conference paper at ICLR 2022
L.9 CLASS NAME: RINGLET, TRAIN ACCURACY (STANDARD RESNET-50); 95.692%
images
heatmaps
feature attack
Figure 104:
Visualization feature 1305 for class ringlet (class index: 322).
Feature label annotated by Mechanical Turk workers: core.
Using a = 0.25, drop in accuracy: 0.0%, mean la perturbation: 41.931.
Using 0.35, drop in accuracy: -7.692%, mean perturbation: 55.774.
images
heatmaps
feature attack
Figure 105: Visualization of feature 1556 for class ringlet (class index: 322).
Feature label as annotated by Mechanical Turk workers: spurious.
Using 0.25, drop in accuracy: -36.924% mean la perturbation: 53.174.
Using σ = 0.35, drop in accuracy: -70.77%, mean la perturbation: 71.192.
70

---

Published as a conference paper at ICLR 2022
L.10 CLASS NAME: LYCAENID, TRAIN ACCURACY (STANDARD RESNET-50), 97.846%
images
heatmaps
feature attack
Figure 106:
Visualization feature 1041 for class lycaenid (class index: 326).
Feature label annotated by Mechanical Turk workers: core.
Using a = 0.25, drop in accuracy: -3.077%, mean la perturbation: 36.564.
Using 0.4, drop in accuracy: -16.923% mean la perturbation: 53.997.
images
heatmaps
feature attack
Figure 107: Visualization of feature 1390 for class lycaenid (class index: 326).
Feature label as annotated by Mechanical Turk workers: spurious.
Using 0.25, drop in accuracy: -43.077% mean la perturbation: 51.468.
Using σ = 0.4, drop in accuracy: -95.385% mean la perturbation: 76.594.
71

---

Published as a conference paper at ICLR 2022
L.11 CLASS NAME: PROBOSCIS MONKEY, TRAIN ACCURACY (STANDARD RESNET-50): 98.231%
images
heatmaps
feature attack
Figure 108:
Visualization feature 334 for class proboscis monkey (class index: 376).
Feature label annotated by Mechanical Turk workers: core.
Using a = 0.25, drop in accuracy: -6.154%, mean la perturbation: 37.884.
Using 0.35, drop accuracy: -15.385% mean le perturbation: 50.698.
images
heatmaps
feature attack
Figure 109: Visualization of feature 1120 for class proboscis monkey (class index: 376).
Feature label as annotated by Mechanical Turk workers: spurious.
Using 0.25, drop in accuracy: -32.308%. mean la perturbation: 51.138.
Using a = 0.35, drop in accuracy: -55.385%, mean la perturbation: 68.414.
72

---

Published as a conference paper at ICLR 2022
L.12 CLASS NAME: TITI, TRAIN ACCURACY (STANDARD RESNET-50): 58.692%
images
heatmaps
feature attack
Figure 110:
Visualization feature 1003 for class titi (class index: 380).
Feature label annotated by Mechanical Turk workers: core.
Using a = 0.25, drop in accuracy: -1.539%, mean le perturbation: 34.703.
Using 0.4, drop in accuracy: +12.307%, mean la perturbation: 51.98.
images
heatmaps
feature attack
Figure 111: Visualization of feature 1120 for class titi (class index: 380). Feature label as annotated by Mechanical Turk workers: spurious.
Using 0.25, drop in accuracy: -38.462%. mean la perturbation: 50.591.
Using σ = 0.4, drop in accuracy: -53.846%, mean la perturbation: 75.361.
73

---

Published as a conference paper at ICLR 2022
L.13
CLASS NAME: BAND AID, TRAIN ACCURACY (STANDARD RESNET- 81.0%
PAD
BAN
BAND-AID
ADD
BANDAD
AID
images
piastic strips
BAND-AID
D-AI
plas strip
Care Bears
BAND
BAND-AID
SANDR
AID
heatmaps
DICHROME PADD
BAN
plastic stops
BAND-AID
D-AI
Care Bears
plas
strip
BANDAD
feature attack
BANEANI
BAN
A&O Vca
Figure 112: Visualization feature 1164 for class band aid (class index: 419). Feature label annotated by Mechanical Turk workers: core. Using a = 0.25, drop in accuracy: -6.154%, mean la perturbation: 42.589.
images
GoCa
heatmaps
feature attack
Figure 113: Visualization feature 447 for class band aid (class index: 419).
Feature label annotated by Mechanical Turk workers: spurious.
Using σ = 0.25, drop accuracy: -43.077%, mean la perturbation: 41.225.
74

---

Published as a conference paper at ICLR 2022
L.14
CLASS NAME: PROJECTILE, TRAIN ACCURACY (STANDARD RESNET-50): 53.538%
images
heatmaps
feature attack
Figure 114:
Visualization feature 1606 for class projectile (class index: 744).
Feature label annotated by Mechanical Turk workers: core.
Using a = 0.25, drop in accuracy: +9.23%, mean /2 perturbation: 34.188.
Using 0.45, drop in accuracy: +6.153% mean la perturbation: 56.571.
images
heatmaps
feature attack
Figure 115: Visualization of feature 961 for class projectile (class index: 744).
Feature label as annotated by Mechanical Turk workers: spurious.
Using 0.25, drop in accuracy: -23.077%, mean la perturbation: 56.179.
Using σ = 0.45, drop in accuracy: -30.769%, mean l2 perturbation: 91.036.
75

---

Published as a conference paper at ICLR 2022
L.15
CLASS NAME: WOK, TRAIN ACCURACY (STANDARD RESNET-501, 71.077%
OO
images
heatmaps
feature attack
Figure 116:
Visualization feature 628 for class wok (class index: 909).
Feature label annotated by Mechanical Turk workers: core.
Using a = 0.25, drop in accuracy: -1.539%, mean la perturbation: 34.231.
Using 0.4, drop in accuracy: -3.077%, mean l2 perturbation: 51.388.
images
heatmaps
feature attack
Figure 117: Visualization of feature 895 for class wok (class index: 909). Feature label as annotated by Mechanical Turk workers: spurious.
Using 0.25, drop in accuracy: 146.154% mean la perturbation: 48.968.
Using σ = 0.4, drop in accuracy: -43.077%, mean la perturbation: 73.24.
76


Reviews:
[{"review_id": "jBBwwaxeO9n", "reviewer": "Reviewer_iQgx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. The paper is well-motivated and well-written, making its readers easier to follow the core idea delivered in the paper.\nThis will potentially make a good impact on the community.\n\n2. The Visualization is very intuitive, this also helps me to quickly pinpoint the spurious vs causal features mentioned in the paper.\n\n3. The scalability contribution is well-justified by the paper.\n\nCons:\n\n1. Despite the acknowledgement of using less rigorous causal terminology, the issue remains. Take 'spurious' as an example, the author used descriptive language such as 'co-occur', it would be better later in the main content to add formal discussion related to Reichenbach’s common cause principle (Peters et al. (2017)).  \n\n2. What does causal formally mean in this paper?  \n\nThe above two questions actually relate to my thought inspired Fig 5 d and e. In Fig 5 d and e, the mask highlights the suspicious attributes which could be the reflected shadow of the bird (maybe I am wrong.) But must this feature be spurious? What if the river is very quiet without any ripple and it is literally a mirror? I think a more precise discussion about causal vs spurious in vision scenario could thus be very useful.\n\n3.  I am a little bit unclear about the concrete usage of this method. Could the authors enlighten me a little more on how we could potentially utilize this model to further improve either the interpretability or robustness of NNs.\n", "summary_of_the_paper": "In the paper, the authors presented a novel scalable method to differentiate the so-called causal or non-causal attributes. \nBy utilizing the neural features activations of in the penultimate layer, i.e. the one adjacent to logits, the proposed method can allow efficiently annotate plausible masks indicating whether the decision-making process of an NN relies on the correct features. \n", "summary_of_the_review": "Based on the pros and cons discussed above, I believe this paper bears the good potential to bring useful insights to the community, though a more thorough discussion from the theoretical side should be improved. \n\nAfter reading the response from the authors, I am pleased with the paper revision and decide to raise my score. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T88zDM-nfvM", "reviewer": "Reviewer_DwGM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tackles an important problem and provides a convincing demonstration that current SOTA models are susceptible to reliance on spurious correlation in the training set. I detail key strengths, weaknesses, and additional questions below. \n\n###  Strengthens\n- The focus on the ImageNet dataset is impressive and crucial to demonstrating that the reliance on spurious correlation issue is not isolated to toy datasets, but prevalent for one of the currently most important datasets in machine learning.\n- The use of crowd source annotations is also helpful and interesting. \n- While the pieces of the entire scheme presented in this work are not new, the combination is interesting and extends recent work that is focused on detecting whether a model relies on spurious correlation.\n\n\n### Weaknesses, Concerns,  & Additional Questions\n- 'Causality': This paper uses the term 'causal' inappropriately in my opinion. Nothing about the new dataset is 'causal' in any sense of the word. At this point, causality refers to a specific set of techniques that capture the data generating process. The Causal ImageNet here is a dataset that indicates which are 'core' features and which are spurious features. I would encourage the authors to completely discard the notions of causality that they refer to repeatedly across the paper. Further, the definition of 'causal' features in this work is quite vague, but acceptable. I would strongly suggest that the authors call those features 'core' features or something of that sort. \n- Non spatial spurious signals: The scheme presented here would only be effective for spatial/dimension dependent features. If the DNN learns say Gender/Race or some other concept that is not visualizable by a heatmap then the approach described here will not work. I would suggest that the authors make this clear in the paper. This is not a limitation in my view, but it allows the reader to understand the scope of the work presented. \n- Clarify the entire scheme and specific assumptions: As it stands, it is difficult to understand the entire scheme here. Part of the difficulty is that Section 3 should be written better to delineate each component of the entire scheme, i.e., if someone else wanted to apply this technique to a different dataset and model, what is needed? State the inputs, outputs, assumptions, etc necessary to apply the scheme discussed here to another setting. Since the authors say they are presenting a scalable framework for discovering spurious features, then it is important to completely state all the necessary requirements to apply the method.\n- What prevents this method from being applied to a non-robust model? Will the approach be less effective? I don't understand whether the models used in Figure 6 are robust or standard models. I think the models for Fig. 6 are standard, i.e. non-adversarially trained, and if yes, why would annotations be collected for a robust model and used to test a non-robust one?\n- Why CAM? Could any feature attribution method have worked here?\n- The user study is interesting, but it is unclear what one can learn from how the crowd workers determined which feature is spurious or not. I say this because the study had the top activating images, the heat maps, the feature attack, wikipedia link, object names etc. To me, it is unclear which of these sources of information helped the crowd workers to determine that a feature is spurious or not. This is quite important information if this approach is to be replicated else where.\n- Are spurious features only learned at the top layer? One assumption in this work is that the important features learned by a DNN are at the penultimate layer? What is the justification for this assumption? I am fine with this assumption, but curious if there is any past work that has made this clear. Could it be possible that a model could concentrate a spurious signal to a filter at the intermediate layer? \n- Given that one can detect reliance on spurious signals, could as straightforward fix be to fine tune the model and include a penalty on the heat maps to force the network to be invariant to the spurious features? This kind of approach has been proposed in prior work.\n\n### Minor Feedback\n- The change in accuracy numbers in Figure 1 were confusing on first read. Perhaps the authors can provide additional statements in the caption to explain what the numbers mean. After reading the paper it makes sense, but it was confusing at first. \n- In the introduction and the related work, the authors claim that their approach is more 'scalable' than current alternatives. I don't agree. To use the approach presented here, one needs to run a user study to collect more features, and another study to confirm that the spurious features detected will generalize. I think the spurious feature detection issue is so important that the paper doesn't need the scalability argument. Any method that requires one to run a study per dataset is not necessarily scalable, but that is also fine!\n- Here is recent work that this paper should also acknowledge: https://arxiv.org/abs/2106.02112\n\n### Post Rebuttal Updates\nThe author response directly addresses most of the concerns that I had. I maintain my accept rating for this paper.", "summary_of_the_paper": "This work proposes a scheme to identify 'causal' and 'spurious' features with human supervision. More specifically, the method presented decomposes the activations that map to the logits of a supervised DNN into neural features. The neural features are then ranked and visualized with the CAM feature attribution method (called neural activation map in this work). The paper then shows the top ranked features and the corresponding heatmaps to crowd workers, who then ascertain whether a feature is spurious or not. \n\nGiven a ranking of causal and spurious features for different output classes, the work then maps these features across an entire 'audit'/validation set. Given this audit set, one can then mask out the regions in the heatmap that correspond to the casual features and observe the corresponding change in model accuracy. Similar sensitivity measurements can also be made for the spurious features as well. Overall, this paper produces an annotated version of the ImageNet dataset with causal and spurious features. This is then used to assess the performance of current state of the art models showing that these models show significant susceptibility to spurious training signals.\n\nOverall, this paper shows how to use additional supervision for detecting a model's reliance on spurious features. The 'Causal Imagenet' dataset should be useful for future work as a stress test for models.", "summary_of_the_review": "I recommend an accept at this time because most of the weaknesses I identified with this paper have to do with the strength of the claims made, which can be addressed in a revision. I particularly like that the paper demonstrates significant weaknesses, reliance on spurious features, of current models on the ImageNet dataset. On the strength of that contribution alone, I think this work is an important one. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KuUrFT5kmIZ", "reviewer": "Reviewer_1LYa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrong (+) and weak (-) points\n\n- (+) The need for annotations of causal/spurious features is important to support research on robust models/domains generalization/etc.\n- (+) The method to propagate annotations from a small set of human labels is simple, effective, and based on observations from prior work that features of the last layer of a CNN are semantically aligned with human concepts.\n- (-) There is however no theoretical basis/guarantee that this is the case.\n- (+) But the authors performed a validation (Sec. 3.4) by crowd-sourcing that seems to support their assumptions.\n- (-) The dataset is large but not huge (50k images, 232 classes).\n- (+) The composition of the dataset (e.g. selection of the subset of 232 classes) seems to have been well thought.\n- (+) The overall design of the data collection procedure generally inspires confidence that the crowd-sourced annotations are of good quality.\n- (+) The authors use the new dataset to evaluate existing models and provide new observations that high accuracy on some classes is achieved by relying on spurious features.\n\n\n\nAdditional advice to improve the paper\n\n- In the intro, the mention of cows/pastures is cited with Arjovsky et al. I'm not sure this was actually done as an experiment as part of this paper. My recollection is that it was more of a thought experiment (to be verified) and/or a citation of the Terra incognita paper of Beery et al.\n- Typo: casual instead of causal\n- The message in the last section that the standard accuracy metric does not tell the whole story is a very important one, IMO. It may benefit from being highlighted more prominently, possibly in the abstract, or maybe in bold in the last section as a form on conclusion/takeaway.", "summary_of_the_paper": "- The paper describes a new benchmark. It uses images from ImageNet, which now have annotations of regions containing features spuriously correlated with the ground truth labels.\n- The key contribution is a method to generate these annotations semi-automatically, by propagating a small set of human annotations to a large number of images.\n- The dataset is used to evaluate existing models, showing they often rely on regions marked as spurious.", "summary_of_the_review": "Preliminary recommendation: accept\n\nKey reasons\n- Importance of the topic/addresses a real need of the research community.\n- Simple method, no technical flaw, underlying assumption are verified empirically.\n- Experiments with the new dataset already provides new insights about existing models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles the important problem of spurious feature detection in deep neural networks. Specifically, it proposes a framework to identify core and spurious features by investigating the activation maps with human supervision. Then, it produces an annotated version of the ImageNet dataset with core and spurious features, called Salient ImageNet, which is then used to empirically assess the robustness of the method against spurious training signals in comparison with current SOTA models.

As pointed out by the reviewers, this work is not about causality and the definitions of causal and spurious features were originally vague and inaccurate. During the revision and discussion,  the authors changed the terms "causal" features/accuracy to "core" features/accuracy. They also called the provided dataset "Salient Imagenet", instead of "Causal Imagenet", and changed the title to "Salient ImageNet: How to Discover Spurious Features in Deep Learning?". Following the prior discussion, we strongly recommend the authors discard any discussion about causality in the camera-ready version of the paper to avoid confusion. Further, we encourage the authors to consider the reviewers’ thoughts and comments in preparing the camera-ready version of their manuscript.
Decision: Accept (Poster)