Causal ImageNet: How to discover spurious features in Deep Learning?

1 INTRODUCTION . The growing use of deep learning systems in sensitive applications such as medicine , autonomous driving , law enforcement and finance raises concerns about their trustworthiness and reliability in the real world . A root cause for the lack of reliability of deep models is their reliance on spurious input features ( i.e. , features that are not essential to the true label ) in their inferences . For example , a convolutional neural network ( CNN ) trained to classify camels from cows associates green pastures with cows and fails to classify pictures of cows in sandy beaches correctly since most of the images of cows had green pastures in the training set ( Beery et al. , 2018 ; Arjovsky et al. , 2020 ) . Similarly , Bissoto et al . ( 2020 ) discovered that skin-lesion detectors use spurious visual artifacts for making predictions . The list of such examples goes on ( de Haan et al. , 2019 ; Singla et al. , 2021 ) . Most of the prior work on discovering spurious features in image classification ( Nushi et al. , 2018 ; Zhang et al. , 2018 ; Chung et al. , 2019 ) requires humans to first come up with a possible list of spurious features . This is often followed by an expensive human-guided labeling of visual attributes which is not scalable for datasets with a large number of classes and samples such as ImageNet . In some cases , this may even require the construction of new datasets with unusual visual attributes to validate the hypothesis ( such as the cows in sandy beaches example discussed previously ) . To address these limitations , some recent works use the neurons of robust models as visual attribute detectors thereby circumventing the need for manual annotations . However , their applicability is either limited to robust models which are less widely deployed due to their low clean accuracy ( Wong et al. , 2021 ) or to identifying background spurious features ( Xiao et al. , 2021 ) , ignoring foreground ones such as food in Figure 1 or to models that achieve low accuracy on the training dataset ( Singla et al. , 2021 ) . We discuss limitations of these methods in more detail in Section 2 . In this work , we introduce a general methodology to discover a subset of spurious and core ( nonspurious ) visual attributes used in model inferences and localize them on a large number of images with minimal human supervision . Our work builds upon the prior observations ( Engstrom et al. , 2019 ) that for a robust model , neurons in the penultimate layer ( called neural features ) often correspond to human-interpretable visual attributes . These visual attributes can be inferred by visualizing the heatmaps ( e.g . via CAM , Zhou et al . ( 2016 ) ) that highlight the importance of image pixels for those neural features ( top row in Figure 1 ) . Alternatively , one can amplify these visual attributes in images by maximizing the corresponding neural feature values via a procedure called the feature attack ( Engstrom et al . ( 2019 ) ; Singla et al . ( 2021 ) , bottom row in Figure 1 ) . Our framework ( shown in Figure 2 ) is based on this key idea : to identify spurious or core visual attributes used for predicting the class i , we identify spurious or core neural features via limited human supervision . We define a core visual attribute as an attribute for class i that is always a part of the object definition and spurious otherwise . To annotate a neural feature as core or spurious , we show only the top-5 images ( with a predicted label of i ) that maximally activate that neural feature to Mechanical Turk workers . We then show that these neural feature annotations generalize extremely well to top-k ( with k 5 ) images with label i that maximally activate that neural feature . For example , on Imagenet , in Section 3.4 , we show a generalization accuracy of 95 % for k = 65 ( i.e. , a 13 fold generalization ) . Thus , by using neural features and their corresponding neural activation maps ( heatmaps ) , with limited human supervision , we can identify a large subset of samples that contain the relevant spurious or core visual attributes . The neural activation maps for these images can then be used as soft segmentation masks for these visual attributes . We emphasize that the usual method of obtaining such a set would require the manual detection and segmentation of a visual attribute across all images in the training set and is therefore not scalable . We apply our proposed methodology to the Imagenet dataset : we conducted a Mechanical Turk study using 232 classes of Imagenet and 5 neural features per class . For each neural feature , we obtained its annotation as either core ( non-spurious ) or spurious for the label . Out of the 232× 5 = 1 , 160 neural features , 160 were deemed to be spurious by the workers . For 93 classes , at least one spurious neural feature was discovered . Next , for each ( class=i , feature=j ) pair , we obtained 65 images with class i showing highest activations of the neural feature j and computed their neural activation maps . The union of these images is what we call the Salient Imagenet dataset . The dataset contains 52 , 521 images with around 226 images per class on average . Each instance in the dataset is of the form ( x , y , Mc , Ms ) where y is the ground truth class andMs , Mc represent the set of spurious and core masks , respectively ( obtained from neural activation maps of neural features ) . This dataset can be used to test the sensitivity of any pretrained model to different image features by corrupting images using their relevant masks and observing the drop in the accuracy . Ideally , we expect our trained models to show a low drop in accuracy for corruptions in spurious regions and a high drop for that of the core regions . However , using a standard Resnet-50 model , we discover multiple classes for which the model shows a trend contradictory to our expectations : a significantly higher spurious drop compared to the core drop . As an example , for the class drake ( Fig . 5 ) , we discover that corrupting any of spurious features results in a significantly higher drop ( > 40 % ) compared to corrupting any of core features ( < 5 % ) . We show many more examples in Appendix L. For various standard models ( Resnet-50 , Wide-Resnet-50-2 , Efficientnet-b4 , Efficientnet-b7 ) , we evaluate their accuracy drops due to corruptions in spurious or core regions at varying degrees of corruption noise and find no significant differences between their spurious or core drops suggesting that none of the models significantly differentiates between core and spurious visual attributes in their predictions ( Fig . 6 ) . In summary , our methodology has the following steps : • Step 1 : We select neural features as penultimate-layer neurons of a pretrained robust network and visualize them using a few highly activating samples ( Section 3.2 ) . • Step 2 : Using Mechanical Turk , we annotate neural features as core or spurious ( Section 3.3 ) . • Step 3 : Using neural feature annotations , we automatically annotate core and spurious visual attributes on many more samples without any human supervision ( Section 3.3 ) . • Step 4 : Using another Mechanical Turk study , we show that our methodology generalizes ex- tremely well to identify core/spurious visual attributes on new samples ( Section 3.4 ) . • Step 5 : Applying steps 1-4 to Imagenet , we develop a new dataset called Salient Imagenet whose samples , in addition to the class label , are annotated by core and/or spurious masks ( Section 4 ) . • Step 6 : Using such a richly annotated dataset , we assess core/spurious accuracy of standard models by adding small noise to spurious/core visual attributes , respectively ( Section 5 ) . To the best of our knowledge , this is the first work that introduces a general methodology to discover core and spurious features with limited human supervision by leveraging neurons in the penultimate layer of a robust model as interpretable visual feature detectors . Also , this is the first work that releases a version of Imagenet with annotated spurious and core image features . We believe that this work opens future research directions to leverage richly annotated datasets to develop deep models that mainly rely on core and meaningful features in their inferences . 2 RELATED WORK . Robustness and Interpretability : In recent years , there have been many efforts towards post-hoc interpretability techniques for trained neural networks . Most of these efforts have focused on local explanations where decisions about single images are inspected ( Zeiler & Fergus , 2014 ; Yosinski et al. , 2016 ; Dosovitskiy & Brox , 2016 ; Mahendran & Vedaldi , 2016 ; Nguyen et al. , 2016 ; Zhou et al. , 2018 ; Olah et al. , 2018 ; Adebayo et al. , 2018 ; Chang et al. , 2019 ; Carter et al. , 2019 ; Yeh et al. , 2019 ; Sturmfels et al. , 2020 ; O ’ Shaughnessy et al. , 2019 ; Verma et al. , 2020 ) . These include saliency maps ( Simonyan et al. , 2014 ; Smilkov et al. , 2017 ; Sundararajan et al. , 2017 ; Singla et al. , 2019 ) , class activation maps ( Zhou et al. , 2016 ; Selvaraju et al. , 2019 ; Bau et al. , 2020 ; Ismail et al. , 2019 ; 2020 ) , surrogate models to interpret local decision boundaries such as LIME ( Ribeiro et al. , 2016 ) , methods to maximize neural activation values by optimizing input images ( Nguyen et al. , 2015 ; Mahendran & Vedaldi , 2015 ) and finding influential inputs ( Koh & Liang , 2017 ) . However , recent work suggests that class activation or saliency maps may not be sufficient to narrow down visual attributes and methods for maximizing neural activations often produce noisy visualizations ( Olah et al. , 2017 ; 2018 ) with changes imperceptible to humans . To address these limitations , recent works ( Tsipras et al. , 2018 ; Engstrom et al. , 2019 ) show that for robust ( Madry et al. , 2018 ) or adversarially trained models ( in contrast to standard models ) , activation maps are qualitatively more interpretable and optimizing an image directly to maximize a certain neuron produces human perceptible changes in the images which can be useful for visualizing the learned features . Failure explanation : Most of the prior works on failure explanation either operate on tabular data where interpretable features are readily available ( Zhang et al. , 2018 ; Chung et al. , 2019 ) , language data where domain-specific textual queries can be easily generated ( Wu et al. , 2019 ; Ribeiro et al. , 2020 ) , images where visual attributes can be collected using crowdsourcing ( Nushi et al. , 2018 ; Xiao et al. , 2021 ; Plumb et al. , 2021 ) or photorealistic simulation ( Leclerc et al. , 2021 ) . However , collecting visual attributes via crowdsourcing can be expensive for large datasets . Moreover , these methods require humans to hypothesize about the possible failures and one could miss critical failure modes when the visual attributes used by model are different from the ones hypothesized by humans . To address these limitations , recent works ( Wong et al. , 2021 ; Singla et al. , 2021 ) use the neurons of robust models as the visual attribute detectors for discovering failure modes , thus avoiding the need for crowdsourcing . However , Wong et al . ( 2021 ) can not be used to analyze the failures of standard ( non-robust ) models which achieve significantly higher accuracy and are more widely deployed in practice . Moreover , to discover the spurious feature , they ask MTurk workers to identify the common visual attribute across multiple images without highlighting any region of interest . However , different images may have multiple common attributes even if they come from different classes and this approach may not be useful in such cases . Barlow ( Singla et al. , 2021 ) learns decision trees using misclassified instances to discover leaf nodes with high failure concentration . However , the instances could be correctly classified due to some spurious feature ( e.g. , background ) and Barlow will discover no failure modes in such cases . In contrast , our approach circumvents these limitations and discovers the spurious features for standard models even when they achieve high accuracy .

Summary: In the paper, the authors presented a novel scalable method to differentiate the so-called causal or non-causal attributes.  By utilizing the neural features activations of in the penultimate layer, i.e. the one adjacent to logits, the proposed method can allow efficiently annotate plausible masks indicating whether the decision-making process of an NN relies on the correct features.