Overinterpretation reveals image classification model pathologies
Authors: Brandon Carter
Siddhartha Jain
Jonas Mueller
David Gifford
Venue: NeurIPS 2021 Poster (2021)
Keywords: computer vision
benchmarks
datasets
convolutional neural networks
interpretability
robustness
overinterpretation

Abstract: Image classifiers are typically scored on their test set accuracy, but high accuracy can mask a subtle type of model failure. We find that high scoring convolutional neural networks (CNNs) on popular benchmarks exhibit troubling pathologies that allow them to display high accuracy even in the absence of semantically salient features. When a model provides a high-confidence decision without salient supporting input features, we say the classifier has overinterpreted its input, finding too much class-evidence in patterns that appear nonsensical to humans. Here, we demonstrate that neural networks trained on CIFAR-10 and ImageNet suffer from overinterpretation, and we find models on CIFAR-10 make confident predictions even when 95% of input images are masked and humans cannot discern salient features in the remaining pixel-subsets. We introduce Batched Gradient SIS, a new method for discovering sufficient input subsets for complex datasets, and use this method to show the sufficiency of border pixels in ImageNet for training and testing. Although these patterns portend potential model fragility in real-world deployment, they are in fact valid statistical patterns of the benchmark that alone suffice to attain high test accuracy. Unlike adversarial examples, overinterpretation relies upon unmodified image pixels.  We find ensembling and input dropout can each help mitigate overinterpretation.

Overinterpretation reveals image classification model pathologies
Brandon Carter MIT CSAIL
Siddhartha Jain MIT CSAIL
Jonas Mueller Amazon Web Services
David Gifford MIT CSAIL gi ifffordmii..cmm
bcarter@csail mit.
Abstract
Image classifiers typically scored their test accuracy, but high accuracy can mask a subtle type of model failure. We find that high scoring convolutional neural networks (CNNs) on popular benchmarks exhibit troubling pathologies that allow them to display high accuracy even in the absence of semantically salient features. When model provides high-confidence decision without salient supporting input features, we say the classifier has overinterpreted its input, finding too much class-evidence in patterns that appear nonsensical to humans. Here, we demonstrate that neural networks trained on CIFAR- 10 and ImageNet suffer from overinterpretation, and we find models on CIFAR 10 make confident predictions even when 95% of input images are masked and humans cannot discern salient features the remaining pixel-subsets. We introduce Batched Gradient SIS, a new method for discovering sufficient input subsets for complex datasets, and use this method to show the sufficiency of border pixels in ImageNet for training and testing, Although these patterns portend potential model fragility in real-world deployment, they are fact valid statistical patterns of the benchmark that alone suffice attain high test accuracy. Unlike adversarial examples, overinterprettaton relies upon unmodified image pixels. We find ensembling and input dropout can each help mitigate overinterpretation.
Introduction
Well-founded decisions by machine learning (ML) systems critical for high-stakes applications such as autonomous vehicles and medical diagnosis. Pathologies models and their respective training datasets result in unintended behavior during deployment if the systems confronted with novel situations. For example, a medical image classifier for cancer detection attained high accuracy in benchmark test data, but was found base decisions upon presence of rulers in image (present when dermatologists already suspected cancer) [ We define model overinterpretation to occur when classifier finds strong class-evidence regions of image that contain no semantically salient features. Overinterpretation is related to overfitting, but overfitting can be diagnosed via reduced test accuracy. Overinterpretation can stem from true statistical signals in the underlying dataset distribution that happen to arise from particular properties of the data source (e.g., derma- tologists' rulers). Thus, overinterpretation can be harder diagnose as admits decisions that are made by statistically valid criteria, and models that use such criteria can excel at benchmarks. We demonstrate overinterpretation occurs with unmodified subsets of the original images. In contrast to adversarial examples that modify images with extra information, overinterpretation is based on real patterns already present in the training data that also generalize to the test distribution. Hidden statistical signals of benchmark datasets can result in models that overinterpret or do not generalize to new data from a different distribution. Computer vision (CV) research relies on datasets like CIFAR- 10 2 and ImageNet K to provide standardized performance benchmarks. Here, we analyze the overinterpretation of popular CNN architectures on these benchmarks to characterize pathologies.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Revealing overinterpretation requires a systematic way to identify which features are used by a model reach its decision. Feature attribution is addressed by a large number of interpretability methods, although they propose differing explanations for the decisions of a model. One natural explanation for image classification lies in the set of pixels that is sufficient for the model to make a confident prediction, even in the absence of information about the rest of the image. In the example of the medical image classifier for cancer detection, one might identify the pathological behavior by finding pixels depicting the ruler alone suffice for the model to confidently output the same classifications. This idea of Sufficient Input Subsets (SIS) has been proposed to help humans interpret the decisions of black-box models [ An SIS subset is minimal subset of features (e.g.. pixels) that suffices to yield a class probability above a certain threshold with all other features masked.
We demonstrate that classifiers trained on CIFAR-10 10 and ImageNet can base their decisions on SIS subsets that contain few pixels and lack human understandable semantic content. Nevertheless, these SIS subsets contain statistical signals that generalize across the benchmark data distribution, and we are able to train classifiers on CIFAR-10 10 images missing 95% of their pixels and ImageNet images missing 90% of their pixels with minimal loss of test accuracy. Thus, these benchmarks contain inherent statistical shortcuts that classifiers optimized for accuracy can learn to exploit, instead of learning more complex semantic relationships between the image pixels and the assigned class label. While recent work suggests adversarially robust models base their predictions on more semantically meaningful features [3. find these models suffer from overinterpretation As we subsequently show, overinterpretation is not only a conceptual issue, but can actually harm overall classifier performance in practice. We find model ensembling and input dropout partially mitigate overinterpretation, increasing the semantic content of the resulting SIS subsets. However, this mitigation is not substitute for better training data, and find that overinterpretation is statistical property of common benchmarks. Intriguingly, the number of pixels in the SIS rationale behind a particular classification is often indicative of whether the image is correctly classified.
well.
may seem unnatural to use an interpretability method that produces feature attributions that look uninterpretable. However, we do not want to bias extracted rationales towards human visual priors when analyzing a model's ' pathologies, but rather faithfully report the features used by a model. To our knowledge, this is the first analysis showing one extract nonsensical features from CIFAR-10 and ImageNet that intuitively should be insufficient or irrelevant for a confident prediction, yet are alone sufficient to train classifiers with minimal loss of performance. Our contributions include:
We discover the pathology of overinterpretatton and find it is common failure mode of MI. models, which latch onto non-salient but statistically valid signals in datasets (Section 4.1
We introduce Batched Gradient SIS, a new masking algorithm to scale SIS to high- dimensional inputs and apply it to characterize overinterpretation ImageNet (Section
We provide pipeline for detecting overinterpretation by masking over 90% of each image, demonstrating minimal loss of test accuracy, and establish lack of saliency in these patterns through human accuracy evaluations (Sections
We show misclassifications often rely on smaller and more spurious feature subsets suggest- ing overinterpretation is serious practical issue (Section 4.4).
We demonstrate that overinterpretation caused by spurious statistical signals training data, and thus training data must be carefully curated to eliminate overinterpretation artifacts.
We identify two strategies for mitigating overinterpretation (Section 4.5)
Code for this paper is available at:
2
Related Work
While existing work has demonstrated numerous distinct flaws in deep image classifiers our paper demonstrates a new distinct flaw, overinterpretation, previously undocumented in the literature. There has been substantial research on understanding dataset bias in CV [6 7 and the fragility of image classifiers deployed outside benchmark settings. We extend previous work on sufficient input subsets (SIS) 4 with the Batched Gradient SIS method, and use this method to show that ImageNet sufficient input subset pixels for training and testing often exist at image borders. Many alternative interpretability methods also aim to understand models by extracting rationales (pixel-subsets) that

---

provide positive evidence for a class &II. and we adopt SIS throughout this work as particularly straightforward method for producing such rationales. This prior work (including SIS [+I] limited to understanding models and does not use the enhanced understanding of models to identify the
overinterpretation flaw discovered in this paper. We contrast the issue of overinterpretation against other previously known model flaws below:
Image classifiers have been shown to be fragile when objects from one image are transplanted in another image [12]. and can be biased by object context [3 [4]. In contrast, overinterpretation differs because we demonstrate that highly sparse, unmodified subsets of pixels in images suffice for image classifiers to make the same predictions on the full images.
Lapuschkin et al. 1S] demonstrate that DNNs can learn to rely on spurious signals in datasets, including source tags and artificial padding, but which are still human-interpretabbe In contrast, the patterns we identify are minimal collections of pixels in images that are semantically meaningless to humans (they do not comprise human- iinnaannnterrretable parts of images). We demonstrate such patterns generalize to the test distribution suggesting they arise from degenerate signals in popular benchmarks, and thus models trained these datasets may fail to generalize to real-world data.
CNNs in particular have been conjectured to pick up on localized features like texture instead of more global features like object shape [6 [7.. Brendel and Bethge [8] show CNNs trained on natural ImageNet images may rely on local features and, unlike humans, are able to classify texturized images, suggesting ImageNet alone is insufficient to force DNNs to rely on more causal representations. Our work demonstrates another source of degeneracy of popular image datasets, where sparse, unmodified subsets of training images that are meaningless humans can enable model to generalize to test data. We provide one explanation for why ImageNet-trained models may struggle to generalize to out-o--disributtoo data.
Geirhos et al. [9 discover that DNNs trained on distorted images fail to generalize well as human observers when trained under image distortions. In contrast, overinterpretation reveals a different failure mode of DNNs, whereby models latch onto spurious but statistically valid sets of features undistorted images. This phenomenon can limit the ability of a DNN to generalize to real-world data even when trained natural images.
Other work has shown deep image classifiers can make confident predictions on nonsensical patterns (20 and the susceptibility of DNNs adversarial examples or synthetic images has been widely studied (5 [1122] However, these adversarial examples synthesize artificial images or modify real images with auxiliary information. contrast, we demonstrate overinterpretation of unmodified subsets of actual training images, indicating the patterns are already present in the original dataset. We further demonstrate that such signals in training data actually generalize to the test distribution and that adversarially robust models also suffer from overinterpretation.
Hooker et al. [4 found sparse pixel subsets suffice to attain high classification accuracy on popular image classification datasets, but evaluate interpretability methods rather than demonstrate spurious features or discover overinterpretation.
Ghorbani al. 25 introduce principles and methods for uumman-underttaddable concept-based explanations of MI. models. In contrast, overinterpretation differs because the features we identify are semantically meaningless to humans, stem from single images, and are not aggregated into interpretable concepts. The existence of such subsets stemming from unmodified subsets of images suggests degeneracies the underlying benchmark datasets and failures of modern CNN models to rely on more robust and interpretable signals in training datasets.
Geirhos et al. 26 discuss the general problem of "shortcut learning" but do not recognize that 5% (CIFAR-10) 10% (ImageNet) spurious pixel-subsets are statistically valid signals in these datasets, nor characterize pixels that provide sufficient support and lead to overinterpretation.
natural language processing (NLP), Feng et al. 27 explored model pathologies using a similar technique, but did not analyze whether the semantically spurious patterns relied on are a statistical property the dataset. Other work has demonstrated the presence of various spurious statistical shortcuts in major NLP benchmarks, showing this problem is not unique to CV 28
3

---

3 Methods
3.1 Datasets and Models
CIFAR-10 [ and ImageNet R have become two of the most popular image classification bench- marks. Most image classifiers are evaluated by the CV community based on their accuracy in one of these benchmarks. We also use the CIFAR-IO-C dataset [29 to evaluate the extent to which our CIFAR-10 models can generalize to out-of-distribution (OOD) data. CIFAR-IO-C contains variants of CIFAR-I0 test images altered by various corruptions (e.g., Gaussian noise, motion blur). Where computing sufficient input subsets on CIFAR-10-C images, we use a uniform random sample of 2000 images across the entire CIFAR-10-C set. Additional results on CIFAR-10.1 v6 30 are presented in Table $4
We use the ILSVRC2012 ImageNet dataset 3
For CI we explore three common CNN architectures: deep residual network with depth 20 (ResNet20)
5 v2 deep residual network with depth 18 (ResNet18) 22 and VGG16
[633
We train these networks using cross-entropy loss optimized via SGD with Nesterov momentum RE
and employ standard data augmentation strategies 32 (Section S2) After training many CIFAR-10
networks individually, we construct four different ensemble classifiers by grouping various networks together. Each ensemble outputs the average prediction over its member networks (specifically, the arithmetic mean of their logits). For each of three architectures, we create corresponding homogeneous ensemble by individually training five networks of that architecture. Each network has different random initialization, which suffices to produce substantially different models despite having been trained on the same data (35. Our fourth ensemble is heterogeneous, containing all 15 networks (5 replicates of each of 3 distinct CNN architectures).
For ImageNet, we use pre-trained Inception v3 model 36 that achieves 22.55% and 6.44% top-1 and top-5 error 7 Additional results from an ImageNet ResNet50 are presented in Section S6
3.2 Discovering Sufficient Features
CIFAR-10. We interpret the feature patterns learned by CIFAR-10 CNNs using the Sufficient Input Subsets (SIS) procedure 4. which produces rationales (SIS subsets) of black-box model's decision-making. SIS subsets are minimal subsets of input features (pixels) whose values alone suffice for the model to make the same decision as on the original input. Let fe(x:) denote the probability that an image v belongs to class c. An SIS subset S is a minimal subset of pixels of x such that fe (xs)) 2 T. where is prespecified confidence threshold and zg is modified input in which all information about values outside S are masked. We mask pixels by replacement with the mean value over all images (equal to zero when images have been normalized), which is presumably least informative to a trained classifier 4. SIS subsets are found via a local backward selection algorithm applied to the function giving the confidence of the predicted (most likely) class.
ImageNet. We scale the SIS backward selection procedure to ImageNet with the introduction of Batched Gradient SIS, a crudient-based method to find sufficient input subsets on high-din nigh-dimenssonal inputs. The sufficient input subsets discovered by Batched Gradient SIS are guaranteed to be sufficient, but may be larger than those discovered by the original exhaustive SIS algorithm. Here we find small SIS subsets with Batched Gradient SIS (Figure S15). Rather than separately masking every remaining pixel at each iteration to find the pixel whose masking least reduces f. we use the gradient of f with respect to the input pixels x and mask M, V. fx @ (1 M)), to order pixels (via single backward pass). Instead of masking only one pixel per iteration, we mask larger subsets of k pixels per iteration. Given p input features, our Batched Gradient FindSIS procedure finds each SIS subset in O(E) () evaluations of V (as opposed to O(p²) evaluations of ʃ in FindSIS 4)).
The complete Batched Gradient SIS algorithm is presented in Section ST
3.3 Detecting Overinterpretation
We produce sparse variants of all train and test set images retaining 5% (CIFAR-10) or 10% (Im- ageNet) of pixels each image. Our goal is to identify sparse pixel-subsets that contain feature patterns the model identifies as strong class-evidence as it classifies an image. We identify pixels to retain based on sorting by SIS BackSelect 4 (CIFAR-10) or our Batched Gradient BackSelect procedure (ImageNet). These backward selection (BS) pixel-subset images contain the final pixels

---

airplane
automobile
cat
deer
dog
frog
horse
ship
truck
Adv. Robust
Figure 1: Sufficient input subsets (SIS) for a sample of CIFAR-10 10 test images (top). Each SIS image shown below is classified by the respective model with 2 99% confidence.
(with their same RGB values in the original images) while all other pixels' values replaced with zero. Note that we apply backward selection to the function giving the confidence of the predicted class from the original model prevent adding information about the true class for misclassified images, and we use the true labels for training/evaluating models on pixel-subsets. As backward selection is applied locally on each image, the specifie pixels retained differ across images.
We train new classifiers on solely these pixcl-subsets of training images and evaluate accuracy on corresponding pixel-subsets of test images determine whether such pixcl-subsets are statistically valid for generalization
the benchmark. We use the same training setup and hyperparameters (Section 3.J without data augmentation of training images (results with data augmentation in Table STp. p. We consider model to overinterpret its input when these signals can generalize to test data but lack semantic meaning (Section
3.4 Human Classification Benchmark
To evaluate whether sparse pixel-subsets of images can accurately classified by humans, we asked four participants to classify images containing various degrees of masking. We randomly sampled 100 images from the CIFAR-10 10 test set (10 images per class) that were correctly and confidently (z 99% confidence) classified by our models, and for each image, kept only 5%, 30%, or 50% of pixels as ranked by backward selection (all other pixels masked). Backward selection image subsets are sampled across our three models. Since larger subsets of pixels are by construction supersets of smaller subsets identified by the same model, we presented each batch of 100 images in order of increasing subset size and shuffled the order of images within each batch. Users were asked to classify each of the 300 images as one of the 10 classes in CIFAR-10 10 and were not provided training images. The same task was given to each user (and is shown in Section S5
4 Results
4.1 CNNs Classify Images Using Spurious Features
CIFAR-10. Figure shows example SIS subsets (threshold 0.99) from CIFAR-10 test images (additional examples in Section $3). These SIS subset images are confidently and correctly classified by each model with > 99% confidence toward the predicted class. We observe these SIS subsets are highly sparse and the average SIS size at this threshold is < 5% of each image (Figure 2 suggesting these CNNs confidently classify images that appear nonsensical to humans (Section leading to
5

---

ResNee10
0.150
515 S.00 trrecsonn er imager
VGG16
Ensemb Reshet15
Ersembb Reshie22
0.100
VSG16
0.000
0.800
Maae
T9a
hots
ship
Predicted Cliss
Figure Distribution of SIS size per predicted class by CIFAR- models computed on all CIFAR-10 test set images classified with 2 99% confidence (SIS confidence threshold 0.99).
ResNet20
ResNet18
VGG16
Freq.
Inception
0.30
0.18
0.12
64
Mean Conf. 0.998
Mean Conf. 0.993
Mean Conf. 0.996
Image
(a) CIFAR-10
(b) ImageNet
Figure Heatmaps pixel locations comprising pixel-subsets. Frequency indicates fraction of subsets containing each pixel. (a) 5% pixel-subsets across CIFAR-10 test set for each model. Mean confidence indicates confidence on 5% pixcl-subsets. (b) Sufficient input subsets (confidence threshold 0.9) across ImageNet validation images from Inception v3.
concern about their robustness and generalizability. We also find that SIS size can differ significantly by predicted class (Figure
We retain 5% of pixels in each image using local backward selection and mask the remaining 95% with zeros (Section 3.3 and find models trained on full images classify these pixel-subsets as accurately as full images (Table D. Figure 3a the pixel locations and confidence of these 5% pixel-subsets across all CIFAR- test images. We found the concentration of pixels on bottom border for ResNet20 is result of tie-breaking during SIS backward selection (Section S4 Morcover, the CNNs are more confident on these pixels subsets than on full images: the mean drop in confidence for the predicted class between original images and these 5% subsets is -0.035 (std dev. 0.107), -0.016 (0.094), and 0.012 (0.074) computed over all CIFAR- 10 test images for our ResNet20, ResNet18, and VGG16 models, respectively, suggesting severe overinterpretation (negative values imply greater confidence on the 5% subsets). We find pixel-subsets chosen via backward selection significantly more predictive than equally large pixel-subsets chosen uniformly random from each image (Table M).
shows
We also find SIS subsets confidently classified by one model do not transfer to other models. For instance, 5% pixel-subsets derived from CIFAR-10 test images using one ResNet18 model (which classifies them with 94.8% accuracy) only classified with 25.8%, 29.2%, and 27.5% accuracy by another ResNet18 replicate, ResNet20, and VGG16 models, respectively, suggesting there exist many different statistical patterns that flexible model might learn to rely on, and thus CIFAR-10 image classification remains a highly underdetermined problem. Training classifiers that make predictions for the right reasons may require clever regularization strategies and architecture design to ensure models favor salient features over spurious pixel subsets.
While recent work has suggested semantics can be better captured by models that are robust to adversarial inputs that fool standard neural networks human-imperceptibble modifications to images A 48), explore wide residual network that is adversarially robust for CIFAR-10

---

Table 1: Accuracy of CIFAR-10 classifiers trained and evaluated full images, 5% backward selection (BS) pixel-subsets, and 5% random pixel-subsets. Where possible, accuracy is reported as mean + standard deviation (%) five runs. For training BS subsets, BS on all images for a single model of each type and average over five models trained on these subsets. Additional results CIFAR-10.1 presented in Table S4
Model
Train On
Evaluate On
CIFAR-10 Test Acc.
CIFAR-I0-C Acc.
Full Images
92.52 = 0.09
69.44 = 0.52
Full Images
5% BS Subscts
92.48
70.65
5% Random
9.98 + 0.03
10.02 + 0.01
ResNet20
5% BS Subsets
5% BS Subscts
92.49 = 0.02
70.58 = 0.03
5% Random
5% Random
50.25 + 0.19
44.04 = 0.33
Input Dropout (Full)
Input Dropout (Full)
91.02 + 0.25
75.46 + 0.74
Full Images
95.17 = 94.76
75.08 = 75.15
Full Images
5% BS Subsets
5% Random
10.08 = 0.15
10.08 = 0.07
ResNet18
5% BS Subsets
5% BS Subsets
94.96 = 0.04
75.25 = 0.05
5% Random
Random
51.27 = 0.82
45.24 + 0.45
Input Dropout (Full)
Input Dropout (Full)
94.15 = 0.26
80.35 = 0.39
Full Images
93.69 + 0.12
74.14 1 0.45
Full Images
BS Subsets
93.27
73.95
5% Random
10.02 + 0.18
9.97 + 0.18
VGG16
5% BS Subsets
BS Subsets
92.60 + 0.08
73.27 + 0.18
5% Random
Random
53.66 = 1.96
46.88 =
Input Dropout (Full)
Input Dropout (Full)
91.09 + 0.15
80.43 + 0.24
Ensemble (ResNet1 8)
Full Images
96.07
77.00
Full Images
5% Random
9.98
10.01
2 and find evidence of overinterpretation (Figure D. This finding suggests adversarial robustness alone does prevent models from overinterpreting spurious signals in CIFAR-10.
classification
We also ran Batched Gradient SIS on CIFAR-10 10 and found edge-heavy sufficient input subsets for CIFAR-10 (Section S4. These heatmap differences are result of the different valid equivalent sufficient input subsets found by the two SIS discovery algorithms. However, since all sufficient input subsets are validated with model and guaranteed to be sufficient for classification at the specified threshold, the heatmaps are accurate depictions of what is sufficient for the model to classify images the threshold. Overinterpretation is independent of the SIS algorithm used because both algorithms produce human- unintrr-nnnterreeaablll sufficient subsets as shown in the examples.
ImageNet. We find models trained on ImageNet images suffer from severe overinterpretation. Figure 4 shows example SIS subsets (threshold 0.9) found via Batched Gradient SIS images confidently classified by the pre-trained Inception v3 (additional examples in Figures SS12 ( These SIS subsets appear visually nonsensical, yet the network classifies them with > 90% confidence. We find SIS pixels are concentrated outside of the actual object that determines the class label. For example, in the "pizza" image, the SIS is concentrated on the shape of the plate and the background table, rather than the pizza itself, suggesting the model could generalize poorly on images containing different circular items likely appeared in the collection of ImageNet photos for this class. In the "traffic light" and "street sign" images, the SIS consists of pixels in sky, suggesting that autonomous vehicle systems that may depend on these models should be carefully evaluated for overinterpretation pathologies.
table. In the "giant panda" image, the SIS contains bamboo, which
Figure 3b shows SIS pixel locations from random sample of 1000 ImageNet validation images. We find concentration along image borders, suggesting the model relies heavily on image backgrounds and suffers from severe overinterpretation. This is serious problem objects determining ImageNet
7

---

parachute
pizza
catamaran
giant panda
golden retriever
traffic light
street sign
Figure 4: Sufficient input subsets (threshold 0.9) for example ImageNet validation images. The bottom row shows the corresponding images with all pixels outside of each SIS subset masked but are still classified by the Inception v3 model with 2 90% confidence.
classes are often located near image centers, and thus this network fails to focus on salient features. We found the mean fraction of an image required for classification with > 90% confidence is only 0.0914, and mean SIS size differs significantly by predicted class (Figure S16
4.2 Sparse Subsets are Real Statistical Patterns
The overconfidence of CNNs for image classification 139 may lead one wonder whether the observed overconfidence on semantically meaningless SIS subsets is an artifact of calibration rather than true statistical signals in the dataset. We train models 5% pixel-subsets of CIFAR-10 training images found via backward selection (Section 3.3). We find models trained solely on these pixel- subsets can classify corresponding test image pixel-subsets with minimal accuracy loss compared to models trained on full images (Table D and thus these 5% pixel-subsets are valid statistical signals in training images that generalize to the test distribution. As a baseline to the 5% pixel-subsets identified by backward selection, we create variants of all images where the 5% pixel-subsets are selected at random from each image (rather than by backward selection) and use the same random pixel-subsets for training each new model. Models trained on random subsets have significantly lower test accuracy compared models trained on 5% pixel-subsets from backward selection (Table D We observe, however, that random 5% subsets of images still capture enough signal to predict roughly 5 times better than blind guessing, but do not capture nearly enough information for models to make accurate predictions.
We found that the 5% backward selection pixel-subsets did not contain model-specific features, and thus reflected valid predictive signals regardless of the model architecture employed for subset discovery. Our hypothesis was that 5% pixel-subsets discovered with one architecture would provide robust performance when used to train and evaluate a second architecture. We found this hypothesis supported for all six pairs of subset discovery and train-test architectures evaluated (Table (2)). These results demonstrate that the highly sparse subsets found via backward selection offer valid predictive signal in the CIFAR-10 10 benchmark exploited by models to attain high test accuracy.
We observe similar results on ImageNet. Inception v3 trained on 10% pixel-subsets of ImageNet training images achieves 71.4% top-1 accuracy (mean over 5 runs) on the corresponding pixel-subset ImageNet validation set (Table S7 Additional ImageNet results for Inception v3 and ResNet50, including training and evaluation on random pixel-subsets and pixel-subsets of different architectures, are provided in Table S7
4.3 Humans Struggle to Classify Sparse Subsets
We find strong correlation between the fraction of unmasked pixels in each image and human classification accuracy (R² = 0.94, Figure S1 Human accuracy on 5% pixel-subsets of CIFAR-10 images (mean 19.2%. std dev 4.8%, Table S6 is significantly lower than original, unmasked images (roughly 94% 20). though greater than random guessing, presumably due to correlations between labels and features such color (e.g.. blue sky suggests airplane, ship, or bird).
However, CNNs (even when trained on full images and achieve accuracy on par with human accuracy on full images) classify these sparse image subsets with very high accuracy (Table [p. indicating
8

---

Moan sis Sioe hnreaae for Correct Pre3s.
ResNet10
0.045
25-
ResNet20
Moan 56 Stes struction of imrort
0.0440
20-
irput Ranheets 9420
rrpt Drepout Neshe222 10 9130
15
0.035
rput Drepout VoGLé 0n 01.512)
Creambie All 657)
10
0.030
Crasnite RecAact 10.50 10.8111
orecmibi Reshee20 9661
Grezmble VGGLe (8.95)
2.025
-5
0.5
0.6
0.7
0.8
0.9
1.0
Confidence of Prodicted Class
Figure 5: Percentage increase in mean SIS size of correctly classified compared to misclassified CIFAR-10 10 test images. Positive values indicate larger mean SIS size for correctly classified im- ages. Error bars indicate 95% confidence interval for the difference in means.
Figure 6: Mean SIS size on CIFAR- 10 test images as SIS threshold varies. SIS size indicates fraction of pixels necessary for model to make the same prediction at each confidence threshold. Model accuracies are shown in the legend. 95% confi- dence intervals are shaded around each mean.
benchmark images contain statistical signals that are not salient to humans. Models solely trained to minimize prediction error may thus latch onto these signals while still accurately generalizing to test data, but may behave counterintuitively when fed images from a different source that does not share these exact statistics. The strong correlation between the size of CIFAR-10 pixel-subsets and the corresponding human classification accuracy suggests larger subsets contain more semantically salient content. Thus, a model whose decisions have larger corresponding SIS subsets presumably exhibits less overinterpretation than one with smaller SIS subsets, as we investigate Section 4.4
4.4 SIS Size is Related to Model Accuracy
Given that smaller SIS contain fewer salient features according human classifiers, models that justify their classifications based on sparse SIS subsets may be limited in terms of attainable accuracy, particularly in -itt---diittibution settings. Here, we investigate the relationship between a single model's 's predictive accuracy and the size of the SIS subsets in which it identifies class- cluss-evidence. We draw no conclusions between models as they are uncalibrated (additional results of SIS from calibrated models are presented in Section S4p. For each of our three classifiers, we compute the average SIS size increase for correctly classified images as compared to incorrectly classified images (expressed as a percentage). We find SIS subsets of correctly classified images are consistently significantly larger than those of misclassified images at all SIS confidence thresholds for both CIFAR-10 10 test images (Figure 5) and CIFAR-10-C OOD images (Figure S3). This is especially striking given model confidence is uniformly lower on the misclassified inputs (Figure (S4). Lower confidence would normally imply a larger SIS subset at given confidence level, fewer pixels can be masked before the model's confidence drops below the SIS threshold. Thus, we rule out overall model confidence This result suggests the sparse SIS subsets highlighted this paper are not just a curiosity, but may be leading to poor generalization real images.
expects
explanation of the smaller SIS of misclassified images.
4.5 Mitigating Overinterpretation
As found pixel-subset size to be strongly correlated with human pixel-subset classification accuracy (Section 4.3) metric for measuring how much ensembling may alleviate overinterpretation is the increase in SIS subset size. We find ensembling uniformly increases test accuracy as expected but also increases the SIS size (Figure 6 hence mitigating overinterpretation.
Ensembling. Model ensembling is known improve classification performance 41 42
We conjecture cause both the increase the accuracy and SIS size for ensembles is the same. We observe that SIS subsets are generally not transferable from one model to another i.e., an SIS for one model is rarely an SIS for another (Section 4.Tp. Thus, different models rely on different independent signals to arrive at the same prediction. An ensemble bases its prediction on multiple

---

such signals, increasing predictive accuracy and SIS subset size by requiring simultaneous activation of multiple independently trained feature detectors. We find SIS subsets of the ensemble are larger than the SIS of its individual members (examples in Figure
+ to both train and test images. We retain each input pixel with probability = 0.8 and set the values of dropped pixels to zero. We find a small decrease CIFAR-10 test accuracy for models regularized with input dropout though find significant (~~ 6%) increase in OOD test accuracy on CIFAR-IO-C images (Table I Figure SS Figure 3 shows corresponding increase in SIS subset size for these models, suggesting input dropout applied at train and test time helps mitigate overinterpretation. We conjecture that random dropout of input pixels disrupts spurious signals that lead to overinterpretation.
Input Dropout. We apply input dropout
Discussion
We find that modern image classifiers overinterpret small nonsensical patterns present in popular benchmark datasets, identifying strong class evidence in the pixel-subsets that constitute these patterns. We introduced the Batched Gradient SIS method for the efficient discovery of such patterns. Despite their lack of salient features, these sparse pixel-subsets are underlying statistical signals that suffice accurately generalize from the benchmark training data the benchmark test data. We found that different models rationalize their predictions based on different sufficient input subsets, suggesting optimal image classification rules remain highly underdetermined by the training data. In high-stakes applications, we recommend ensembles of networks or regularization via input dropout.
Our results call into question model interpretabiliity methods whose outputs encouraged to align with prior human beliefs of proper classifier operating behavior [44]. Given the existence of non- salient pixel-subsets that alone suffice for correct classification, a model may solely rely such patterns. In this case, an interpretability method that faithfully describes the model should output these nonsensical rationales, whereas interpretability methods that bias rationales toward human priors may produce results that mislead users to think their models behave as intended.
Mitigating overinterpretation and the broader task of ensuring classifiers are accurate for the right reasons remain significant challenges for ML. While we identify strategies for partially mitigating overinterpretation, additional research needs to develop ML methods that rely exclusively well- formed interpretable inputs, and methods for creating training data that do not contain spurious signals. One alternative is regularize CNNs by constraining the pixel attributions generated via Unfortunately, such methods require a human annotator to highlight the auxiliary supervision signal. Saliency maps have also been shown provide unreliable insights into model operating behavior and must be interpreted as approximations [4S] In contrast, our SIS subsets constitute actual pathological examples that have been misconstrued by the model. An important application of our methods is the evaluation of training datasets to ensure decisions are made on interpretable rather than spurious signals. We found popular image datasets contain such spurious signals, and the resulting overinterpretation may be difficult to overcome with ML methods alone.
a saliency map 45 47
correct pixels
Acknowledgments and Disclosure of Funding
This work was supported by Schmidt Futures and the National Institutes of Health [R01CA218094].
Author Contributions
All authors contributed to conceptualization, methodology, formal analysis, and writing. BC led execution of the experiments.
10

---

References
[1] Neel V. Patel.
Why
Doctors
Aren't
Afraid of Better. More
Efficient
AI Diagnosing Cancer. why- aeeet--reett-oae -reett-fraidd-aaeaeeaaaaa--aaaaa of beeteer-more -tw.oreeeiiiii--aii----i--i--eee -iagnossin cancer Accessed
2017.
URL https://www thedailybeast.
September 27, 2020.
[2] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[3] Olga Russakovsky. Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (ICCV), 115(3):211-255 2015. doi: 10.1007/s1 -11-11..
[4] Brandon Carter, Jonas Mueller, Siddhartha Jain, and David Gifford. What made you this? Understanding black-hox decisions with sufficient input subsets. In The 22nd International Conference on Artificial Intelligence and Staristics, pages 567-576, 2019.
[5] Andrew Ilyas, Shibani Sanurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, 2019.
[6] Antonio Torralba and Alexei Efros. Unbiased look at datasct bias. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1521-1528. IEEE, 2011.
[7] Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. A deeper look at dataset bias. In Domain Adaptation in Computer Vision Applications, pages 37-55. Springer, 2017.
[8] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2950- 2958, 2019.
[9] Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sehastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization of what deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems, 28(1 (8(11):2660-663, 2016.
[10] Chirag Agarwal and Anh Nguyen. Explaining image classifiers by removing input features using generative models. In Proceedings of the Asian Conference on Computer Vision, 2020.
[11] Amit Dhurandhar, Pin-Yo Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shanmugam, and Payel Das. Explanations based on the missing: towards contrastive explanations with pertinent negatives. In Advances in Neural Information Processing Systems, 2018.
[12] Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant in the room. arXiv preprint arXiv:1808.0 03300, 2018.
[13] Rakshith Shetty. Bernt Schiele, and Mario Fritz. Not using the to see the sidewalk quantifying and controlling the effects of context in classification and segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8218-8226, 2019.
[14] Krishna Kumar Singh, Dhruv Mahajan, Kristen Grauman, Yong Jac Lec, Matt Feiszli, and Deepti Ghadi- yaram. Don't 't judge an object by its context: Learning to overcome contextual bias. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 1070-11078, 2020.
[15] Sebastian Lapuschkin. Stephan Wäldchen. Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn Nature Commanications, 10(1):1-8, 2019.
[16] Leon A Gatys, Alexander Ecker, and Matthias Bethge. Texture and art with deep neural networks. Current Opinion in Neurobiology, 46:178-186, 2017.
[17] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. ImageNNe-traineed CNNs are biased towards texture: increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
[18] Wieland Brendel and Matthiss Bethge. Approximating CNNs with Bag-of-local Featurres models works surprisingly well ImageNet. In International Conference Learning Representations, 2019.

---

[19] Robert Geirhos, Carlos R Medina Temme, Jonas Rauber, Heiko H Schütt, Mathhis Bethge, and Felix A Wichmann. Generalisation in humans and deep neural networks. In Advances in Neural Information Processing Systems, 2018
[20] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pages 427-436, 2015.
[21] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
[22] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are casily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 427-436, 20155
[23] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
[24] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems, 2019.
[25] Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems, 2019.
[26] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 2(11):655-673 2020.
[27] Shi Feng, Eric Wallace, Alvin Grissom II, Mohit lyyer, Pedro Rodriguez, and Jordan Boyd-Graber. Pathologics of neural models make interpretations difficult. In Empirical Methods in Natural Language P Processing, 2018.
[28] Timothy Niven and Hung- Yu Kao. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 4658 4664, 2019.
[29] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[30] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do CIFAR-10 classifiers generalize to CIFAR-107 arXiv preprint arXiv: 180401110000,11, 2018.
[31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision. Springer, 2016.
[33] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, 2015.
[34] Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning, 2013.
[35] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, 2016.
[36] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Partern Recognition, pages 2218-222,2222 2016.
[37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-perfornance deep learning library. In Advances in Neural Information Processing Systems, 2019.
[38] Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with a single (robust) classifier. In Advances in Neural Information P Processing Systems, 2019.
12

---

[39] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference Machine Learning, 2017.
[40] Andrej Karpathy. Lessons learned from manually classifying CIFAR-10. http:/karpathy, github. io/201 htt11////////aaalaaaaaaaaaaaa/// -aaaall--ccaassifieeeir1... 2011.
Published online at
[41] King-Shy Goh, Edward Chang, and Swang-Ting Cheng. SVM binary classifier ensembles for image classification. In Proceedings the Tenth International Conference on Information and Knowledge Management, pages 395 402. ACM, 2001.
[42] Cheng Ju, Aurélien Bibaut, and Mark der Laan. The relative performance of ensemble methods with deep convolutional neural networks image classification. Journal of Applied Statistics, 45(15): 2800-2818, 2018.
[43] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15: 1929-1958 2014.
[44] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, 2018.
[45] Andrew Slavin Ross, Michael C Hughes, and Finale Doshi Velez. Right for the right reasons: training differentiable models by constraining their explanations. In Proceedings of the 26th International Joint Conference Artificial Intelligence, pages 2662-26777 2017.
[46] Becks Simpson, Francis Dutil, Yoshua Bengio, and Joseph Paul Cohen. GradMask: Reduce overfitting by regularizing saliency. arXiv preprint arXiv:1904.07475. 2019.
[47] Joseph D Viviano, Becks Simpson, Francis Dutil, Yoshua Bengio, and Joseph Paul Cohen. Saliency is possible red herring when diagnosing poor generalization. In International Conference Learning Representations, 2021.
[48] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schüitt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability saliency methods. In Explainable Al: Interpreting. Explaining and Visualizing Deep Learning, pages 267-280. Springer, 2019.
[49] Antonio Torralba, Rob Fergus, and William Freeman. 80 million tiny images: large data set for nomparametric object and scene recognition. IEEE Transactions on Puttern Analysis and Machine Intelligence, 30(1 1978- 2008.
[50] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolu- tional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4700 4708, 2017.
[51] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
[52] Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. arXiv preprint mrxxv:2210. 10497, 2021.
13


Reviews:
[{"review_id": "sdFc6VIV64k", "reviewer": "Reviewer_dNig", "summary": "The paper presents an interesting finding that CNNs overinterpret a few non-meaningful pixels in the image to make accurate predictions. The authors use SIS and extend it to Batched Gradient SIS to discover the limited number of pixels such CNNs rely on. The authors conduct extensive experiments on two major image datasets to verify the finding. Besides, ensembling and input dropout are explored to mitigate the overinterpretation problem.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: One of my concerns is the novelty of the method part. The proposed Batched Gradient SIS method seems to be a straightforward extension to SIS for better efficiency.\n\n**Quality**: The paper contains extensive experiments on two image datasets to verify the overinterpretation problem (e.g., detailed analysis in Sec. 4.1 - 4.4 provides). One minor concern is that SIS pixels’ patterns are different between Fig. 1 and Fig. 3. Is it due to different datasets or due to different SIS algorithms (vanilla SIS vs. Batched Gradient SIS). Adding an experiment to show the SIS pixels from Batched Gradient SIS method on the CIFAR-10 dataset may explain that difference.\n\n**Clarity**: The paper is well-written and easy to follow.\n\n**Significance**: The finding of the paper, i.e., overinterpretation, is significant, which can facilitate future research to solve the overinterpretation problem for achieving more accurate and robust (not relying on spurious pixels) image classification models.\n\nIn summary, although the paper’s method is not novel, I believe that the paper’s main focus, i.e., revealing and analyzing the overinterpretation problem, does make a good contribution to the research community. Thus my rating is “Marginally above the acceptance threshold” to this paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although the ensembling and input dropout methods do not completely mitigate the overinterpretation issue (L471-474), the paper’s major focus is revealing and analyzing the overinterpretation problem. It is acceptable to leave it for future work. Therefore, the paper’s limitation is well-addressed. In terms of the potential negative societal impact, I do not think “training datasets may become more complex” is negative. In fact, removing such spurious signals in the dataset can make the model more robust, leading to a positive impact. Therefore, I do not think the paper has potential negative impacts because the paper tries to reduce them.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "c_HWE_Kfc4", "reviewer": "Reviewer_eccV", "summary": "Authors define \"overinterpretation\" as an undesired behavior in deep neural networks where the networks find strong class evidence in semantically unrelated image regions. The behaviour is quantified by identifying the minimal number of pixels sufficient to predict a given class in an input image while masking all the other features/pixels. In this regard, protocols are proposed to evaluate overinterpretation and the models trained on CIFAR and ImageNet are shown to suffer from this problem. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n+ A batched gradient approach is developed to discover sufficient subsets in the input images. \n+ Semantically irrelevant features contribute towards the prediction of true class in an image, this behavior is analyzed systematically. An interesting result is that masking ratios of up to 90-95% still lead to comparable test accuracies. To the best of my knowledge, this has not been shown earlier in this context. \n+ Approaches to mitigate overinterpretation are also analyzed with some useful recommendations such as model ensembling. \n+ Good number of experiments are performed to study the overinterpretation problem. The analysis also shows that the SIS size is related to the model performance on clean images. \n+ Paper is clearly written with nice illustrations and visualizations. \n\nCons/Questions:\n- The term SIS (sufficient input subsets) [4] is not defined before its use in Line 11.\n- ResNet and VGG family models are explored to study overinterpretation. Some recent works (e.g., [A]) in the literature study randomly masking inputs and found ViTs to have much better robustness compared to CNNs.  It will be interesting to see how the robustness holds against masks learned in a white-box setting, similar to the setting studied in this work. \n- It is important to understand here if the SIS patterns are specific to the input data distribution/image or the target model. There is one experiment on this point in the supplementary material for CIFAR10 dataset where different architectures are tried out with the same SIS. Due to the significance of this expeirment, the authors can improve it further with an analysis on a large dataset like ImageNet or across an altogether different family e.g., ViTs (as an extension to the above comment).\n- The human study is performed with 4 participants. Were they shown sufficient examples of 5% SIS images to learn about the 10 CIFAR classes. Without such a precursor learning task, this experiment seems to be biased towards the model since SIS are found using the model in a white-box setting and therefore would make sense to the model but not for humans. A revserse eperiment has also been done in the literature [13] where humans can easily classifly textureless objects while DNN models suffer. \n\n[A] Naseer et al., \"Intriguing Properties of Vision Transformers\", Arxiv 2021. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impact is appropriately discussed in the paper. ", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "BEfK5nXlK9", "reviewer": "Reviewer_LoQm", "summary": "The paper identifies sparse sets of pixels which can be used with deep neural nets to classify images with high confidence. The authors demonstrate results with CIFAR-10, an augmentation corrupted version of it and with ImageNet. They show a number of experiments to obtain insights in why these spurious subsets are classified with high confidence.\n\nFor example, they show that the sparse subsets can be used to train a different model to predict well on CIFAR-10 and on Imagenet.\nFurthermore they show that the SIS sizes are larger on correctly classified images compared to misclassified images, which implies that the sets are indeed used to properly predict on images.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "clarity: it is an easy read, well written.\n\noriginality: The novelty of batchgradientSIS is limited. It is a natural iterative extension of the brute force SIS optimization. The paper would be more straightforward if one stressed the understanding aspect rather than the novelty of batchgradientSIS.\n\nThe existence of sparse subsets has not been analyzed so far, though shape-based background-artefacts in Imagenet have been observed in Finding and Removing Clever Hans: Using Explanation Methods to Debug and Improve Deep Models  (see some later figures in this paper).\nThe paper has a novelty in discussing pathologies of neural nets.\n\nquality:\nThe experiments are well designed, to analyze various hypotheses, e.g. using sparse subsets discovered by one model to train another model and obtain similar accuracy.\n\nas criticism points on quality:\n\nmissing citations:\n\n-Regarding optimizing masks to retain minimal examples: Fong, Patrick, Vedaldi, Understanding Deep Networks via Extremal Perturbations and Smooth Masks ICCV2019\n-By computing attributions and using attribution-score to guide iterative masking it belongs to a class of approaches which were used for CNNs in Samek et al, Evaluating the visualization of what a deep neural network has learned, TNNLS 2017\n-Occlusion methods - e.g. Agarwal, Nguyen ACCV 2020\n-Pertinent positives, Dhurandhar et al. 2018. While this method may have some degrees of freedom and ambiguities, it can be still mentioned.\n\nsignificance: this paper adds a new angle to the discussion on deep neural network pathologies.\n\nDespite limited novelty in methods, the reviewer finds the paper acceptable because it discusses a statistical defect of NNs from a new angle, and thus increases the understanding of existing algorithms rather than proposing a novel algorithm.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nSocietal Impact:\n\nThis paper shows that CIFAR-10 is similar to MNIST a dataset suffering from simplicity, and results on it should be taken with care when it comes to transferability to other more complex datasets. It can be assumed that not every researcher will like to hear this kind of news. So it is worth to bring this in NeurIPS.\n\nLimitations:\n\nOne should give a justification why batchgradSIS is chosen over e.g. optimizing masks and occlusion methods (or integrated gradients?) . Reviewer guesses likely due to fast runtime, or maybe the authors were interested in a greedy optimization which does not have strong connectedness priors on the remaining unmasked regions for the sake of mining unconnected sets?\n\nLikely the noisyness of gradients (Balduzzi, The Shattered Gradients Problem: If resnets are the answer, then what is the question? , 2017) probably also helps in finding disconnected subsets.\n\n\nThe work can be put in a wider context of adversarial optimization, although it does not aim at small changes in the input but rather semantically meaningless examples as the original \"neural networks are easily fooled\" paper by AM Nguyen.\n\nsee also the missing citations in the box before.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "A7v4FRDbLz4", "reviewer": "Reviewer_cYV5", "summary": "The paper evaluates pathologies of modern neural networks on CIFAR-10 and ImageNet. Specifically, the authors show that these networks exhibit *overinterpretation*, which is a phenomenon where a model can use sparse subsets of an image which are semantically meaningless to humans in order to confidently and accurately classify the image. The authors utilize the Sufficient Input Subsets (SIS) algorithm from the interpretability literature to identify these subsets, but they also propose modifications to this algorithm which allow it to scale to ImageNet. Surprisingly, the authors find that models evaluated on a small subset of SIS pixels from each image (5% on CIFAR-10 and 10% on ImageNet) exhibit only a small drop in accuracy. The authors explore two techniques for mitigating overinterpretation: ensembling and input dropout.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality** Though the SIS algorithm is not new, the authors use it to make important and novel observations about neural network behaviors. They also modify the algorithm to be applicable to ImageNet.\n\n**Quality**:  Overall, the submission is technically sound and the experimental results are convincing. However, I have some comments I hope the authors can address:\n\n**Major comments**\n\n1. *Calibration of models*. Since neural networks are known to be poorly calibrated and over confident, thresholding the model confidences based on a fixed parameter may lead to inconsistent results across uncalibrated models. Are the models calibrated before applying SIS?\n2. *Use of confidence instead of accuracy in SIS algorithm*.  Would it be possible to set a threshold on the model accuracy instead of the model confidence in the SIS algorithm?  The size of the subset needed to reach a certain accuracy threshold on both in-distribution and out-of-distribution datasets might be a more meaningful way to compare the performance of different models.\n3. *Evaluating on full images when training on subsets* In Table 1, it would be interesting to compare the model accuracy on the full images when trained on either the BS or random subsets in order to understand how much training on the subsets hurts model performance.\n4. *Size of the subset versus model accuracy* . The authors claim that accuracy and SIS size are correlated for ensembles, but not for models trained with input dropout. For regular models, it’s not clear what the trend is. It would be interesting to see model accuracy on both in-distribution and out-of-distribution data versus SIS size.  \n5. *Evaluation on CIFAR-10.1*  CIFAR-10C is a synthetic distribution shift whereas CIFAR-10.1 (Recht et. al. 2018) is a natural distribution shift. Natural distribution shifts are more realistic than synthetic shifts. Moreover, techniques that work well on synthetic shifts do not necessarily transfer to natural distribution shifts (Taori et. al. 2020). The authors should evaluate out-of-distribution robustness on CIFAR-10.1 in addition to CIFAR-10C.\n\n**Minor comments**\n\n6. *Explanation for heatmaps* Can the authors comment on why the ResNet20 heat map is so different from the ResNet18 and VGG16 heat map? Also, why are the ImageNet subsets concentrated on the border of the images, but the same is not true for CIFAR?\n7. *SIS size per class* Does the size of the SIS subset vary across ImageNet classes?\n8. *Conflicting observations* The observations on lines 267-273 seem to directly conflict with the observations on lines 226-233. Can the authors clarify?\n\n**Significance** The paper shows a way to detect spurious features that a model over-relies on when making predictions.  This has significant implications for robustness research which aims to prevent such over-reliance. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss limitations briefly, but do not address negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work argues that deep neural networks overinterpret the input, i.e. they rely on too small portions of the image to make the decision. This is novel insight and I am enthusiastic that it will help the community understand better deep networks. All reviewers agree that the paper should be accepted, and I am glad to support this decision. Please remember to address all reviewers' comments in the camera-ready version.
Decision: Accept (Poster)