Overinterpretation reveals image classification model pathologies
Authors: Brandon Carter
Siddhartha Jain
Jonas Mueller
David Gifford
Venue:  (2021)
Keywords: computer vision
benchmarks
datasets
convolutional neural networks
interpretability
robustness
overinterpretation

Abstract: Image classifiers are typically scored on their test set accuracy, but high accuracy can mask a subtle type of model failure. We find that high scoring convolutional neural networks (CNNs) on popular benchmarks exhibit troubling pathologies that allow them to display high accuracy even in the absence of semantically salient features. When a model provides a high-confidence decision without salient supporting input features, we say the classifier has overinterpreted its input, finding too much class-evidence in patterns that appear nonsensical to humans. Here, we demonstrate that neural networks trained on CIFAR-10 and ImageNet suffer from overinterpretation, and we find models on CIFAR-10 make confident predictions even when 95% of input images are masked and humans cannot discern salient features in the remaining pixel-subsets. Although these patterns portend potential model fragility in real-world deployment, they are in fact valid statistical patterns of the benchmark that alone suffice to attain high test accuracy. Unlike adversarial examples, overinterpretation relies upon unmodified image pixels.  We find ensembling and input dropout can each help mitigate overinterpretation.

Under review as a conference paper at ICLR 2021
OVERINTERPRETATION REVEALS IMAGE CLASSIFICA- TION MODEL PATHOLOGIES
Anonymous authors Paper under double-bling review
ABSTRACT
Image classifiers are typically scored on their test set accuracy, but high accuracy can mask a subtle type of model failure. We find that high scoring convolutional neural networks (CNNs) on popular benchmarks exhibit troubling pathologies that allow them to display high accuracy even in the absence of semantically salient features. When model provides a high-confidence decision without salient supporting input features, we say the classifier has overinterpreted its input, finding too much class-evidence in patterns that appear nonsensical to humans. Here, we demonstrate that neural networks trained on CIFAR-I0 and ImageNet suffer from overinterpretatinn, and we find models on CIFAR-10 10 make confident predictions even when 95% of input images are masked and humans cannot discern salient features in the remaining pixel-subsets. Although these patterns portend potential model fragility in real-world deployment, they are in fact valid statistical patterns of the benchmark that alone suffice to attain high test accuracy. Unlike adversarial examples, overinterpretation relies upon unmodified image pixels. We find ensembling and input dropout can each help mitigate overinterpretation.
INTRODUCTION
Well-founded decisions by machine learning (ML) systems are critical for high-stakes applications such as autonomous vehicles and medical diagnosis. Pathologies in models and their respective training datasets can result in unintended behavior during deployment if the systems are confronted with novel situations. For example, medical image classifier for cancer detection attained high accuracy benchmark test data, but was found to base decisions upon presence of rulers in an image (present when dermatologists already suspected cancer) (Patel) 2017). We define model overinterpretation to occur when a classifier finds strong class-evidence in regions of an image that contain semantically salient features. Overinterpretation is related to overfitting, but overfitting can be diagnosed via reduced test accuracy. Overinterpretation can stem from true statistical signals in the underlying dataset distribution that happen to arise from particular properties of the data source (e.g., dermatologists' rulers). Thus, overinterpretation can be harder to diagnose as it admits decisions that are made by statistically valid criteria, and models that use such criteria excel at benchmarks. We demonstrate overinterpretation occurs with unmodified subsets of the original images; in contrast adversarial examples that modify images with extra information, overinterpretation is based on real patterns already present in the training data that also generalize to the test distribution.
Hidden statistical signals of benchmark datasets can result in models that overinterpret do not generalize to new data from a different distribution. Computer vision (CV) research relies on datasets like CIFAR-10 (Krizhevsky 2009) and ImageNet (Russakovsky et al.) 2015) to provide standardized performance benchmarks. Here, we analyze the overinterpretation of popular CNN architectures on these benchmarks to characterize pathologies.
Revealing overinterpretation requires a systematic way to identify which features are used by a model to reach its decision. Feature attribution is addressed by a large number of interpretability methods, although they propose differing explanations for the decisions of model. One natural explanation for image classification lies in the set pixels that is sufficient for the model to make a confident prediction,
in the absence of information about the rest of the image. In the example of the medical image classifier for cancer detection, one might identify the pathological behavior by finding pixels depicting the ruler alone suffice for the model to confidently output the same classifications.

---

Under review as a conference paper at ICLR 2021
This idea of Sufficient Input Subsets (SIS) has been proposed to help humans interpret the decisions of black-box models (Carter et al. 2019). An SIS subset is a minimal subset of features (e.g., pixels) that suffices to yield a class probability above certain threshold with all other features masked.
We demonstrate that classifiers trained on CIFAR- 10 and ImageNet can base their decisions on SIS subsets that contain few pixels and lack human understandable semantic content. Nevertheless, these SIS subsets contain statistical signals that generalize across the benchmark data distribution, and we are able to train classifiers on CIFAR-I0 images missing 95% of their pixels and ImageNet images missing 90% of their pixels with minimal loss of test accuracy. Thus, these benchmarks contain inherent statistical shortcuts that classifiers optimized for accuracy can learn to exploit, instead of learning more complex semantic relationships between the image pixels and the assigned class label. While recent work suggests adversarially robust models base their predictions on more semantically meaningful features (llyas et al. (019), we find these models suffer from overinterpretation As we subsequently show, overinterpretation is not only a conceptual issue, but can actually harm overall classifier performance in practice. We find model ensembling and input dropout partially mitigate overinterpretation, increasing the semantic content of the resulting SIS subsets. However, this mitigation is not substitute for better training data, and find that overinterpretation is statistical property of common benchmarks. Intriguingly, the number of pixels in the SIS rationale behind particular classification is often indicative of whether the image will be correctly classified.
well.
may seem unnatural to use an interpretability method that produces feature attributions that look uninterpretable. However, we do not want to bias extracted rationales towards human visual priors when analyzing model's pathologies, but rather faithfully report the features used by model. To our knowledge, this is the first analysis showing one can extract nonsensical features from CIFAR-10 and ImageNet that intuitively should be insufficient or irrelevant for a confident prediction, yet are alone sufficient to train classifiers with minimal loss of performance. Our contributions include:
We discover the pathology of overinterpretation and find it is common failure mode of ML models, which latch onto non non-salient but statistically valid signals in datasets (Section 4.1)
We introduce Batched Gradient SIS, a new masking algorithm to scale SIS to high- dimensional inputs and apply it to characterize overinterpretation on ImageNet (Section 3.2)
We provide a pipeline for detecting overinterpretation by masking over 90% of each image, demonstrating minimal loss of test accuracy, and establish lack saliency in these patterns through human accuracy evaluations (Sections 3.3
We show misclassifications often rely on smaller and more spurious feature subsets suggest- ing overinterpretaaton is serious practical issue (Section[4.4)
We identify two strategies for mitigating overinterpretation (Section 4.5) We demonstrate that overinterpretation is caused by spurious statistical signals in training data, and thus training data must be carefully curated to eliminate overinterpretation artifacts.
2 RELATED WORK
There has been substantial research understanding dataset bias in CV (Torralba & Eiros 2011 Tommasi et al., 2017) and the fragility of image classifiers deployed outside of the benchmark setting (Rosenfeld features like texture instead of more global features like object shape (Gatys et al. 2017 Brendel & Bethge 2019). Other work has shown deep image classifiers can make confident predictions on nonsensical patterns (Nguyen et al. 2015: Iyas et al.) 2019), though these adversarial examples synthesize artificial images or modify real images with auxiliary information. In contrast, we demonstrate overinterpretation of unmodified subsets of actual training images, indicating the patterns are already present in the original dataset. We further demonstrate that such signals in training data actually generalize to the test distribution. Hooker et al. (2019) found sparse pixel subsets suffice to attain high classification accuracy on popular image classification datasets, but evaluate interpretability methods rather than demonstrate spurious features or discover overinterpretation. In natural language processing (NLP), Feng et al. (2018) explored model pathologies using a similar technique, but did a statistical property of
al. 2018). CNNs in particular have been conjectured to pick up on localized
not analyze whether the semantically spurious patterns the models rely the dataset. Other research has demonstrated the presence of various spurious statistical shortcuts in major NL. benchmarks, showing this problem not unique to CV (Niven & Kao! 2019).
2

---

Under review as a conference paper at ICLR 2021
3 METHODS
3.1 DATASETS AND MODELS
CIFAR-10 10 KKrizhevsky 2009) and ImageNet (Russakovsky et al. 2013) have become two of the most popular image classification benchmarks. Most image classifiers are evaluated by the CV community based on their accuracy in one of these benchmarks. We also use the CIFAR-10-C dataset (Hendrycks & Dietterich 2019) to evaluate the extent to which our CIFAR-10 models can generalize to out-of-distributee altered by various corruptions (such as Gaussian noise, motion blur, and snow). Where computing sufficient input subsets on CIFAR-10 10-C images, we use uniform random sample of 2000 images across the entire CIFAR- 10-C set. We use the ILSVRC2012 ImageNet dataset.
(OOD) data. CIFAR-1O-C contains variants of C1FAR-10 test images
For CIFAR-10, we explore three common CNN architectures: deep residual network with depth 20 (ResNet20) (He et al., 2016a), a v2 deep residual network with depth 18 (ResNet18) (He et al. 2016b), and VGGI6 (Simonyan & Zisserman 2014). We train these networks using cross-entrope loss optimized via SGD with Nesterov momentum (Sutskever et al.) 2013) and employ standard data augmentation strategies (He al. 2016b) (additional details in Section (ST). After training many CIFAR-10 networks individually, we construct four different ensemble classifiers by grouping various networks together. Each ensemble outputs the average prediction over its member networks (specifically, the arithmetic mean of their logits). For each of three architectures, we create a corresponding homogeneous ensemble by individually training five networks of that architecture. Each network has a different random initialization, which suffices to produce substantially different models despite having been trained on the same data (Osband et al., 2016). Our fourth ensemble heterogeneous, containing all 15 networks (5 replicates of each of 3 distinct CNN architectures).
For ImageNet, we use a pre-trained Inception v3 model (Szegedy et al. and 6.44% top-1 and top-5 error (Paszke et al. 2019).
2016) that achieves 22.55%
3.2 DISCOVERING SUFFICIENT FEATURES
CIFAR-10. We interpret the feature patterns learned by CIFAR- 10 CNNs using the Sufficient Input Subsets (SIS) procedure (Carter et al. 2019). which produces rationales (SIS subsets) of a black-box model's 's decision- maasing. SIS subsets are minimal subsets of input features (pixels) whose values alone suffice for the model to make the same decision as on the original input. Let fe(2x) denote the probability that an image æ belongs to class c. An SIS subset is a minimal subset of pixels of x such that fx(ss) 2 where is prespecified confidence threshold and zg is modified input in which all information about values outside S are masked. We mask pixels by replacement with the mean value over all images (equal zero when images have been normalized), which is presumably least informative to a trained classifier (Carter et al., 2019). SIS subsets are found via a local backward selection algorithm applied the function giving the confidence of the predicted (most likely) class.
ImageNet. present, is computationally infeasible to scale the original SIS backward selection method to find inputs. Rather than separately masking every remaining
procedure to ImageNet. We introduce Batched Gradient SIS, a gradient- sufficient input subsets on high-dimension pixel at each iteration to find the pixel whose masking least reduces f. we use the gradient of with respect to the input pixels x and mask M, V M [x @ (1 M)), to order pixels (via a single backward pass). Instead of masking only one pixel per iteration, we mask larger subsets of k > 1 pixels per iteration. Given p imput features, our Batched Gradient FindSIS procedure finds each SIS subset in O ()) evaluations of f (as opposed Op²)) evaluations of f in FindSIS (Carter et al. 2019). The complete Batched Gradient SIS algorithm is presented in Section S5
3.3 DETECTING OVERINTERPRETATION
We produce sparse variants of all train and test set images retaining 5% (CIFAR-10) or 10% (Im- ageNet) of pixels in each image. Our goal is to identify sparse pixel-subsets that contain feature patterns the model identifies as strong class-evidence as it classifies image. We identify pixels to retain based on sorting by SIS BackSelect (Carter et al. 2019) (CIFAR-10) or our Batched Gradient BackSelect procedure (ImageNet). These backward selection (BS) pixel-subset images contain the
3

---

Under review as a conference paper at ICLR 2021
final pixels (with their same RGB values as in the original images) while all other pixels' values replaced with zero. Note that we apply backward selection to the function giving the confidence of the predicted class from the original model to prevent adding information about the true class for misclassified images, and we use the true labels for training/evaluating models on pixel-subsets. As backward selection is applied locally on each image, the specific pixels retained differ across images.
We train new classifiers on solely these pixel-subsets of training images and evaluate accuracy on corresponding pixel-subsets of test images determine whether such pixel-subsets are statistically valid for generalization in the benchmark. We use the same training setup and hyperparameters (Section 3.1) without data augmentation of training images (results with data augmentation in Table(SIp We consider model to overinterpret its input when these signals can generalize to test data but lack semantic meaning (Section
3.4 HUMAN CLASSIFICATION BENCHMARK
To evaluate whether sparse pixel-subsets of images can accurately classified by humans, we asked four participants to classify images containing various degrees of masking. We randomly sampled 100 images from the CIFAR-IO test set (10 images per class) that were correctly and confidently (z 99% confidence) classified by our models, and for each image, kept only 5%, 30%, or 50% of pixels as ranked by backward selection (all other pixels masked). Backward selection image subsets sampled across three models. Since larger subsets of pixels are by construction supersets of smaller subsets identified by the same model, we presented each batch of 100 images in order of increasing subset size and shuffled the order of images within each batch. Users were asked to classify each of the 300 images as one of the 10 classes in CIFAR-10 10 and were not provided training images. The same task was given each user (and is shown in Section S4
4 RESULTS
4.1 CNNS CLASSIFY IMAGES USING SPURIOUS FEATURES
CIFAR-10. Figure T shows example SIS subsets (threshold 0.99) from CIFAR-10 test images (additional examples in Section S2). These SIS subset images are confidently and correctly classified by each model with 2 99% confidence toward the predicted class. We observe these SIS subsets are highly sparse and the average SIS size at this threshold is < 5% of each image (see Figure 5). suggesting these CNNs confidently classify images that appear nonsensical to humans (Section leading to concern about their robustness and generalizability.
We retain 5% of pixels in each image using local backward selection and mask the remaining 95% with zeros (Section 3.3 and find models trained on full images classify these pixel-subsets as accurately as full images (Table T. Figure Saauree2aaaa 2a] shows the pixel locations and confidence of these 5% pixel-subsets across all CIFAR-TO test images. Moreover, the CNNs are more confident on these pixels subsets than on full images: the mean drop in confidence for the predicted class between original images and these 5% subsets is -0.035 (std dev. 0. -0.016 (0.094), and -0.012 (0.074) computed over all CIFAR-IC 10 test images for our ResNet20, ResNet18, and VGG16 models, respectively, suggesting severe overinterpretation (negative values imply greater confidence on the 5% subsets). We find pixel-subsets chosen via backward selection are significantly more predictive than equally large pixel-subsets chosen uniformly at random from each image (Table
We also find SIS subsets confidently classified by one model do not transfer to other models. For instance, 5% pixel-subsets derived from CIFAR-10 test images using one ResNet18 model (which classifies them with 94.8% accuracy) are only classified with 25.8%, 29.2%, and 27.5% accuracy by another ResNet18 replicate, ResNer20, and VGG16 models, respectively, suggesting there exist many different statistical patterns that a flexible model might learn to rely on, and thus CIFAR-10 image classification remains a highly underdetermined problem. Producing high-capacity classifiers that make predictions for the right reasons may require clever regularization strategies and architecture design to ensure models favor salient features over such sparse pixel subsets.
While recent work has suggested semantics can be better captured by models that robust to adversarial inputs that fool standard neural networks via uuman-imperceptibble modifications to images (Madry et al. 2017 Santurkar et al. 2019), we explore wide residual network that is
4

---

Under review as a conference paper at ICLR 2021
airpiane
automobile
bird
cat
deer
dog
frog
horse
ship
truck
VGG16 ResNei20
Figure I: Sufficient input subsets (SIS) for sample of CIFAR-10 test images (top). Each SIS image shown below is classified by the respective model with 2 99% confidence. The "Adv. Robust" (pre- trained adversarially robust) model we use is from Madry et al. (017) and robust to loo perturbations.
ResNel20
ResNet18
VGG16
Freq.
Inception-v3
0.30
0.24
0.18
0.12
04
0.06
Mean Conf. 0,998
Mean Conf. 0.993
Mean Conf. 0.996
(a) CIFAR-10
(b) ImageNet
Figure 2: Heatmaps of pixel locations comprising pixel-subsets. Frequency indicates fraction of subsets containing each pixel. (a) 5% pixel-subsets across CIFAR-10 test set for each model. Mean confidence indicates confidence on 5% pixel-subsets. (b) Sufficient input subsets (threshold 0.9) across ImageNet validation images from Inception v3.
adversarially robust for CIFAR-10 10 classification (Madry et al. 2017) and find evidence of overinter- pretation (Figure This finding suggests adversarial robustness alone does not prevent models from overinterpreting spurious signals in CIFAR-10.
parachute
pizza
catamaran
giant panda
golden retriever
traffic light
street sign
Figure 3: Sufficient input subsets (threshold 0.9) for example ImageNet validation images. The bottom row shows the corresponding images with all pixels outside of each SIS subset masked but still classified by the Inception v3 model with > 90% confidence.

---

Under review as a conference paper at ICLR 2021
Table 1: Accuracy of CIFAR-I0 classifiers trained and evaluated full images, 5% backward selection (BS) pixel-subsets, and 5% random pixel-subsets. Where possible, we report accuracy as mean standard deviation (%) over five runs. For training on BS subsets, we run BS on all images for single model of each type and average over five models trained on these subsets.
Model
Train On
Evaluate On
CIFAR-10 Test Acc.
CIFAR-IO-C Acc.
Full Images
92.52 = 0.09
69.44 = 0.52
Full Images
BS Subsets
92.48
70.65
Random
9.98 = 0.03
10.02 = 0.01
ResNer20
5% BS Subsets
5% BS Subscts
92.49 = 0.02
70.58 = 0.03
5% Random
5% Random
50.25 = 0.19
44.04 = 0.33
Input Dropout (Full)
Input Dropout (Full)
91.02 = 0.25
75.46 = 0.74
Full Images
95.17 + 0.21
75.08 + 0.20
Full Images
5% BS Subsets
94.76
75.15
5% Random
10.08 = 0.15
10.08 = 0.07
ResNet18
5% BS Subsets
5% BS Subsets
94.96 = 0.04
73.25 = 0.05
5% Random
5% Random
51.27 + 0.82
45.24 = 0.45
Input Dropout (Full)
Input Dropout (Full)
94.15 + 0.26
80.35 + 0.39
Full Images
93.69 = 0.12
74.14 = 0.45
Full Images
5% BS Subsets
93.27
73.95
5% Random
10.02 + 0.18
9.97 + 0.18
VGG16
5% Subsets
BS Subsets
92.60 + 0.08
73.27 + 0.18
5% Random
5% Random
53.66 + 1.96
46.88 + 1.27
Input Dropout (Full)
Input Dropout (Full)
91.09 + 0.15
80.43 + 0.24
Ensemble (ResNet18)
Full Images
96.07
77.00
Full Images
5% Random
9.98
10.01
ImageNet. We also find models trained on ImageNet images suffer from severe overinterpretation. Figure 3 shows example SIS subsets (threshold 0.9) found via Batched Gradient SIS images confidently classified by the pre-trained Inception v3 (additional examples in Figures S8 and [9). These SIS subsets appear visually nonsensical, yet the network classifies them with > 90% confidence. We find SIS pixels are concentrated outside of the actual object that determines the class label. For example, in the "pizza" image, the SIS is concentrated on the shape of the plate and the background table, rather than the pizza itself, suggesting the model could generalize poorly on images containing different circular items on a table. the "giant panda" image, the SIS contains bamboo, which likely appeared in the collection of ImageNet photos for this class. In the "traffic light" and "street sign" images, the SIS consists of pixels in sky, suggesting that autonomous vehicle systems that may depend on these models should be carefully evaluated for overinterpretation pathologies.
Figure 2b shows SIS pixel locations from random sample of 1000 ImageNet validation images. We find concentration along image borders, suggesting the model relies heavily image backgrounds and suffers from severe overinterpretation. This is serious problem as objects determining ImageNet classes are often located near image centers, and thus this network fails to focus salient features.
4.2 SPARSE SUBSETS ARE REAL STATISTICAL PATTERNS
The overconfidence of CNNs for image classification (Guo et al. 2017) may lead one to wonder whether the observed overconfidence on semantically meaningless SIS subsets an artifact of calibration rather than true statistical signals in the dataset. We train models 5% pixel-subsets of CIFAR-10 training images found via backward selection (Section 3.3 We find models trained solely on these pixel-subsets can classify corresponding test image subsess with minimal accuracy loss compared to models trained on full images (Table [T] As baseline to the 5% pixel-subset identified by backward selection, create variants of all images where the 5% pixel-subsets selected at

---

Under review as a conference paper at ICLR 2021
Mcan 1 Si8 Site increaze for Correct Preds.
ResNat18
0.345
RcsNctaa (0352)
7%
ResNet20
VG016
Mesn 915 Stee (rrcction af
0.040
ResNetis (0.521
Redhet23 (0913))
15
0.235
Irput Drppur yoGze (0.022)
Ensemtle All (0.157)
10
0.020
Encemble Redtetla 934011
Ensemble Redteb2 (2.66))
Ensemtle vouie 10.351
0.025
0.020
-5-
0.015
0.5
0.5
0.7
0.8
0.9
1.0
0.7
0.8
Confidence Predicted Class
Confidence Predicted
Figure 4: Percentage increase in mean SIS size of correctly classified compared misclassified CIFAR- 10 test images. Positive values indicate larger mean SIS size for correctly classified im- ages. Error bars indicate 95% confidence interval for the difference in means.
Figure 5: Mean SIS size on CIFAR-I0 test images as SIS threshold varies. SIS size indicates fraction of pixels necessary for model to make the same prediction at each confidence threshold. Model accuracies are shown in the legend. 95% confi- dence intervals are shaded around each mean,
random from each image (rather than by backward selection) and use the same random pixel-subsets for training each new model. Models trained on random subsets have significantly lower test accuracy (Table D compared to models trained on 5% pixcl-subsets from backward selection. This result suggests that the highly sparse subsets found via backward selection offer valid predictive signal in the CIFAR-10 10 benchmark exploited by models to attain high test accuracy. We observe similar results on ImageNet. Inception v3 trained on 10% pixel-subsets ImageNet training images achieves 71.4% accuracy (mean over 5 runs) the corresponding pixel-subset ImageNet validation set (Table S4h.
4.3 HUMANS STRUGGLE TO CLASSIFY SPARSE SUBSETS
We find a strong correlation between the fraction of unmasked pixels in each image and human classification accuracy (R2 0.94, Figure S7 Human accuracy on 5% pixel-subsets of CIFAR-10 10 images (mean = 19.2%% 19.2%, std dev 4.8%. Table S3 is significantly lower than on original, unmasked images (roughly 94% (Karpathy| 2011)), though greater than random guessing, presumably due to correlations between labels and features such as color (e.g., blue sky suggests airplane, ship, or bird).
However, CNNs (even when trained on full images and achieve accuracy on par with human accuracy on full images) classify these sparse image subsets with very high accuracy (Table D]. indicating benchmark images contain statistical signals that are not salient to humans. Models solely trained to minimize prediction error may thus latch onto these signals while still accurately generalizing to test data, but may behave counterintuitively when fed images from a different source that does not share these exact statistics. The strong correlation between the size of CIFAR-10 10 pixel-subsets and the corresponding human classification accuracy suggests larger subsets contain more semantically salient content. Thus, model whose decisions have larger corresponding SIS subsets presumably exhibits less overinterpretation than one with smaller SIS subsets, as we investigate in Section 4.4
4.4
SIS SIZE IS PREDICTIVE OF MODEL ACCURACY
Given that smaller SIS contain fewer salient features according to human classifiers, models that justify their classifications based on sparse SIS subsets may be limited in terms of attainable accuracy, particularly out-of-distribution settings. Here, we investigate the relationship between a model's predictive accuracy and the size of the SIS subsets in which it identifies class-evidence. For each of our three classifiers, we compute the average SIS size increase for correctly classified images as compared to incorrectly classified images (expressed as percentage). We find SIS subsets of correctly classified images are consistently significantly larger than those misclassified images at varying SIS confidence thresholds for both CIFAR- 10 test images (Figure and CIFAR-10-C OOD images (Figure (S3). This is especially striking given model confidence i8 uniformly lower on the misclassified inputs (Figure 54). Lower confidence would normally imply larger SIS subset at
7

---

Under review as a conference paper at ICLR 2021
given confidence level, as expects fewer pixels can be masked before the model's confidence drops below the SIS threshold. Thus, we can rule out overall model confidence as an explanation of the smaller SIS of misclassified images. This result suggests the sparse SIS subsets highlighted in this paper are not just a curiosity, but may be leading to poor generalization on real images.
4.5 MITIGATING OVERINTERPRETATIOON
Ensembling, Model ensembling is known to improve classification performance (Goh et al. 2001 Ju et al. 2018). As we found pixel-subset size to be strongly correlated with human pixel-subset classification accuracy (Section 4.3 our metric for measuring how much ensembling may alleviate overinterpretation is the increase in SIS subset size. We find ensembling uniformly increases test accuracy as expected but also increases the SIS size (Figure 5 hence mitigating overinterpretation.
We conjecture the cause both the increase in the accuracy and SIS size for ensembles is the same. We observe that SIS subsets generally not transferable from one model another- i.e., SIS for one model is rarely an SIS for another (Section [.1].. Thus, different models rely on different independent signals arrive at the same prediction. An ensemble bases its prediction on multiple such signals, increasing predictive accuracy and SIS subset size by requiring simultancous activation of multiple independently trained feature detectors. We find SIS subsets of the ensemble are larger than the SIS of its individual members (examples in Figure
Input Dropout We apply input dropout (Srivastava et al. 2014) to both train and test images. We retain each input pixel with probability p = 0.8 and set the values of dropped pixels to zero. We find small decrease in CIFAR-10 test accuracy for models regularized with input dropout though find a significant (~ 6%) increase in OOD test accuracy on CIFAR- 10-C images (Table ! Figure (5). F a corresponding increase in SIS subset size for these models, suggesting input dropout applied at train and test time helps to mitigate overinterpretation. We conjecture that random dropout of input pixels disrupts spurious signals that lead to overinterpretation.
5 DISCUSSION
We find that modern image classifiers overinterpret small nonsensical patterns present in popular benchmark datasets, identifying strong class evidence in the pixel-subsets that constitute these pattems.. Despite their lack of salient features, these sparse pixel-subsets are underlying statistical signals that suffice to accurately generalize from the benchmark training data to the benchmark test data. We found that different models rationalize their predictions based on different sufficient input subsets, suggesting optimal image classification rules remain highly underdetermined by the training data. In high-stakes applications, we recommend ensembles of networks or regularization via input dropout.
Our results call into question model interpretability methods whose outputs are encouraged to align with prior human beliefs regarding proper classifier operating behavior (Adebayo et al.) 2018). Given the existence of non-salient pixel- pixel-subsets that alone suffice for correct classification, a model may solely rely on these patterns. In this case, interpretability method that faithfully describes the model should output these nonsensical rationales, whereas interpretability methods that bias rationales toward human priors may produce results that mislead users think their models behave as intended.
Mitigating overinterpretation and the broader task of ensuring classifiers are accurate for the right reasons remain significant challenges for ML. While we identify strategies for partially mitigating overinterpretation, additional research needs to develop ML methods that rely exclusively on well- formed interpretable inputs, and methods for creating training data that do not contain spurious signals. One alternative is to regularize CNNs by constraining the pixel attributions generated via a saliency map (Ross et al., 2017: Simpson et al.) 2019 Viviano et al. 2019). Unfortunately, such methods require a human annotator to highlight the correct pixels as an auxiliary supervision signal. Saliency maps have also been shown to provide unreliable insights into model operating behavior and must be interpreted as approximations (Kindermans et al. 2019). In contrast, our SIS subsets constitute actual pathological examples that have been misconstrued by the model. An important application of our methods is the evaluation of training datasets to ensure decisions are made on interpretable rather than spurious signals. We found popular image datasets contain such spurious signals, and the resulting overinterpretation may be difficult overcome with ML. methods alone.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Julius Adebayo. Justin Gilmer, Michael Muelly, Ian Goodfellow. Moritz Hardt, and Been Kim. Sanity checks for salieney maps. In Advances in Neural Information Processing Systems, pp. 9505-9515, 2018.
Wieland Brendel and Matthias Bethge. Approximating CNNs with Bag-of locallooool-------oo Features models works surprisingly well on ImageNet. Proceedings of the International Conference on Learning Representations, 2019.
Brandon Carter, Jonas Mueller, Siddhartha Jain, and David Gifford. What made you do this? Understanding black-box decisions with sufficient input subsets. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 567-576, 2019.
Shi Feng, Eric Wallace, II Grissom, Mohit lyyer, Pedro Rodriguez, Jordan Boyd-Graher, et al. Pathologies of neural models make interpretations difficult. arXiv preprint 1804,08880 2018.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Texture and art with deep neural networks. Current Opinion in Neurobiology, 1e11778116 2017.
King-Shy Goh, Edward Chang, and Kwang- Ting Cheng. SVM binary classifier ensembles for image classifica- tion. In Proceedings of the tenth international conference Information and knowledge management, pp. 395 495-402.44 ACM, 2001.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning- Voluue 70, pp. 1321-1330 JMLR. org, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016b.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. benchmark interpretability methods in deep neural networks. In Advances in Neural Information Processing Systems, pp. 9734-9745, 2019.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-136, 2019.
Cheng Ju, Aurélien Bibaut, and Mark van der Laan. The relative performancc of ensemble methods with deep convolutional neural networks for image classification. Journal of Applied Statistics, 45(15):2800-281 2818, 2018.
Andrej Karpathy. Lessons learned from manually classifying CIFAR-10. Published online at http://karpathy. github. io/201 /
2011.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. Springer, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report. University of Toronto, 2009.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, arXiv preprint arXiv:1706.06088 2017.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015.
Timothy Niven and Hung- Yu Kao. Probing neural network comprchension of natural language arguments. ACL, 2019.
Ian Oshand, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pp. 4026-4034, 2016.

---

Under review as a conference paper at ICLR 2021
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024 8035, 2019.
Neel V. Patel.
Why Doctors Aren't
Afraid URL
of Better,
More
Efficient AI Di-
agnosing Cancer,
2017.
https: theda ybeast. com/
why-doctorss woscooos--ooooooooooo.. ors cenectorr--eettt..ee
aerenttaffaan afraid
Accessed September 27, 2020.
Amir Rosenfeld, Richard Zemel, and John K Tsotsos. The elephant in the room. arXiv preprint 1rXIv:1808.03303
2018.
Andrew Slavin Ross, Michael C Hughes, and Finale Boshi-Velez, Right for the right reasons: training differentiable models by constraining their explanations, In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pp. 2662-2670, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-222 2015, doi: 10.1007/s1 163-015-
Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with single (robust) classifier. In Advances in Neural Information Processing Systems, pp- 1260-1271, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409. 1556, 2014.
Becks Simpson, Francis Dutil, Yoshua Bengio, and Joseph Paul Cohen, GradMask: Reduce overfitting by regularizing saliency. arXiv preprint urXiv: (944::9990077788 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15: 1929-1958, 2014.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139-1147, 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey Toffe, Jon Shlens, and 7bigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEFE conference computer vision and pattern recognition, pp. 2818-2826, 2016.
Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. A deeper look dataset bias. In Domain adaptation in computer vision applications, pp. 37-55. Springer, 2017.
Antonio Torralba and Alexei Efros. Unbiased look dataset bias. In CVPR 2011, pp. 1521-1528. IEEE, 2011.
Joseph D Viviano, Becks Simpson, Francis Dutil, Yoshua Bengio, and Joseph Paul Cohen. Underwhelming generalization improvements from controlling feature attribution. arXiv preprint arXiv: 1910.00/99, 2019.
10

---

Under review as a conference paper at ICLR 2021
Supplementary Information:
Overinterpretation reveals image classification model pathologies
S1
MODEL IMPLEMENTATION AND TRAINING DETAILS
CIFAR-10 10 MODELS
We first describe the implementation and training details for the CIFAR-10 models used in this paper (Section 3.1). The ResNet20 architecture (He al.) 2016a) has 16 initial filters and total of 0.27M parameters. ResNet18 (He et al.) 2016b) has 64 iitiill filters and contains 11.2M parameters. The VGGI6 architecture (Simonyan & Zisserman 2014) uses batch normalization and contains 14.7M parameters.
All models are trained for 200 epochs with batch size of 128. We minimize cross-entropy via SGD with Nesterov momentum (Sutskever et al. (2013) using momentum of 0,9 and weight decay of 5e-4. The learning rate initialized as 0.1 and reduced by factor of 5 after epochs 60, 120, and 160. Datasets are normalized using per-channel mean and standard deviation, and we use standard data augmentation strategies consisting of random crops and horizontal flips (He et 2016b).
The adversarially robust model we evaluated is the adv_ I tra trainei ined model of Madry et al. (011)) available on GitHut
To apply the SIS procedure to CIFAR-10 10 images, we use an implementation available on GitHut For confidently classified images on which we run SIS, we find one sufficient input subset per image using the FindSIS procedure. When masking pixels, we mask all channels of each pixel as single feature.
IMAGENET MODELS
For finding SIS, we pre-trained models (Inception v3 (Szegedy et al. 2016) and ResNet50 (He) ct al. 2016al) provided by PyTorch (Paszke et al. 2019) in the torchvision package (PyTorch version 1.4.0, torchvision version 0.5.0).
When training new ImageNet classifiers, we adopt model implementations and training scripts from PyTorch (Paszke et al. 2019), obtained from GitHut Models are trained for 90 epochs using batch size 256 (Inception-v3) or 512 (ResNet50). We minimize cross-entropy via SGD using momentum of 0.9 and weight decay of le-4. The learning rate is initialized 0. and reduced by factor of 10 every 30 epochs. Datasets are normalized using per-channel mean and standard deviation. For Inception v3, images are cropped to 299 x 299 pixels. For ResNet50, images are cropped to 224 224. When training Inception v3, we define the model using the aux aux_logi DoggissFaals argument. We do not use data augmentation when training models on pixel-subsets of images.
HARDWARE DETAILS
Each CIFAR-10 model is trained on 1 NVIDIA GeForce RTX 2080 Ti GPU. Once models are trained, SIS are computed across multiple GPUs (by parallelizing over individual images). Each SIS (for CIFAR-10 image) takes roughly 30-60 seconds to compute (depending on the model architecture).
ImageNet models are trained on 2 NVIDIA Titan RTX GPUs. For finding SIS from pre-trained ImageNet models, run Batched Gradient BackSelect for batches of 32 images across 10 NVIDIA GeForce RTX 2080 Ti GPUs, which takes roughly 1-2 minutes per batch (details in Scction S5).
su sa

---

Under review as a conference paper at ICLR 2021
S2 ADDITIONAL EXAMPLES OF CIFAR-10 SUFFICIENT INPUT SUBSETS
SIS OF INDIVIDUAL NETWORKS
Figure ST shows sample of SIS for each of our three architectures. These images were randomly sampled among all CIFAR-10 test images confidently (confidence 2 0.99) predicted to belong the class written on the left. SIS are computed under threshold of 0.99, so all images shown in this figure are classified with probability 2 99% confidence as belonging to the listed class.
2

---

Under review as a conference paper at ICLR 2021
airplane
automobile
Vi
bird
cat
24
deer
2
dog
22
frog
bp
20
horse
B
ship
truck
(a) ResNet20
airplane
automobile
cat
x
deer
dog
frog
horse
ship
truck
(b) ResNet18
airplane
automobile
cat
deer
dog
frog
horse
ship
truck
(c) VGG16
Figure S1: Examples SIS (threshold 0.99) on random sample of CIFAR-10 10 test images (15 per class, different random sample for each architecture). All images shown here are predicted to belong the listed class with > 99% confidence.

---

Under review as a conference paper at ICLR 2021
SIS OF ENSEMBLE
Figure S2 shows examples of SIS from one of our model ensembles (a homogeneous ensemble of ResNet18 networks, see Section 3.1), along with corresponding SIS for the same image from each of the five member networks in the ensemble. We use a SIS threshold of 0.99, so all images are classified with 2 99% confidence. These examples highlight how the ensemble SIS are larger and draw class-evidence from the individual members SIS.
Original Image
Ensemble
RedNet18 #1
ResNet18 ResNet18
ResNet18
ResNet1
Figure S2: Examples of SIS (threshold 0.99) from the ResNet18 homogeneous ensemble (Section 3.1 and its member models. Each row shows original CIFAR-10 image (left), followed by SIS from the ensemble (second column) and the SIS from each of its member networks (remaining columns). Each image shown is classified with > 99% confidence by its respective network.
4

---

Under review as a conference paper at ICLR 2021
S3 ADDITIONAL MODEL PERFORMANCE RESULTS
TRAINING ON PIXEL-SUBSETS WITH DATA AUGMENTATION
Table ST presents results similar to those in Section 4.2 and Table but where models are trained on
5% pixel -subets with data augmentation (as described Section ST We find training without data augmentation slightly improves accuracy when training classifiers on 5% pixel-subsets of CIFAR-10 10.
Table S1: Accuracy of CIFAR-10 classifiers trained and evaluated on full images, 5% backward selection (BS) pixel-subsets, and 5% random pixel-subsets with data augmentation (+). Accuracy reported mean = standard deviation (%) over five runs.
Model
Train
Evaluate On
CIFAR-I0 Test Acc.
CIFAR-10-C Acc.
5% BS Subsets (+)
5% BS Subsets
92.26 + 0.01
70.21 + 0.14
ResNer20
5% Random (+)
5% Random
48.87 + 0.41
42.66 + 0.15
5% Subsets (+) BS Subsets
94.51 + 0.38
74.91 + 0.41
ResNet18
5% Random (+)
5% Random
49.03 + 0.92
42.97 + 0.82
5% Subsets (+) BS Subsets
91.17 1 0.04
71.82 1
VGG16
5% Random (+)
5% Random
51.32 1 1.35
44.56 1 0.96
TRAINING ON PIXEL-SUBSETS WITH DIFFERENT ARCHITECTURES
Table S2 presents results of training and evaluating models on 5% pixel-subsets drawn from different architectures. Models were trained without data augmentation on subsets from one replicate each base architecture. We find accuracy from training and evaluating model on 5% pixel-subsets of images derived from a different architecture is commensurate with accuracy of training and evaluating model of the same type those subsets (Table T
Table S2: Accuracy of CIFAR-IO 10 classifiers trained and evaluated on 5% backward selection (BS) pixel-subsets from different architectures. Accuracy is reported as mean + standard deviation (%) over five runs
5% Subsets from Model
Model Trained
CIFAR-10 Test Acc.
CIFAR-10-C Acc.
ResNet18
92.53 + 0.02
70.56 0.04
ResNet20
VGG16
92.47 + 0.02
70.42 + 0.14
ResNer20
94.88 + 0.03
75.14 + 0.10
ResNet18
VGG16
94.88 + 0.05
75.13 + 0.09
ResNer20
92.05 + 0.14
73.01 0.08
VGG16
ResNet18
92.57 + 0.10
73.33 + 0.21
ADDITIONAL RESULTS FOR SIS SIZE AND MODEL ACCURACY
Figure $3 shows percentage increase in mean SIS size for correctly classified images compared to misclassified images from the CIFAR-10-C dataset.
Figure S4 shows the mean confidence of each group of correctly and incorrectly classified images that we consider at each confidence threshold (at each confidence threshold along the x-axis, we evaluate SIS size in Figure 4 on the of images that originally were classified with least that level of confidence). We find model confidence is uniformly lower on the misclassified inputs.
5

---

Under review as a conference paper at ICLR 2021
Mean % SIS Siza increaze for Corract Frede.
60
ResNet18
Rechat20
VGG16
20
10
-10
Confidence Llass
Figure S3: Percentage increase in mean SIS size of correctly classified images compared miselassi- fied images from random sample of CIFAR-10-4 test set. Positive values indicate larger mean SIS size for correctly classified images. Error bars indicate 95% confidence interval for the difference in means.
1.000
1.000
Mealt Larfoonnce of ofiginal Insage
0.975
0.975
0.950
0.850
0.825
0.925
0.900
0.900
0.875
0.875
0.850
ResNet20
ResNet20
VSG16
VGG16
0.825
Clacsified
Correctly Classified
Misclessil
Misclessieee
0.7
0.6
0.9
0.5
0.6
0.7
0.9
1.0
515 Trieshold [Canfidance of Presd cted Clasel
5i5 Trreshold (Confidence of Pred cted Class)
(a) CIFAR- 10 test set
(b) CIFAR- 10-C test set
Figure S4: Mean confidence correctly vs. incorrectly classified images for each corresponding SIS threshold we evaluate in Figure 4 across the (a) CIFAR-10 test set and (b) our random sample of the CIFAR- 10-C test set. Shaded region indicates 95% confidence interval.

---

Under review as a conference paper at ICLR 2021
ADDITIONAL RESULTS FOR INPUT DROPOUT
Figure S5 shows the accuracy improvement on each individual corruption of the CIFAR-IO-C out- of-distribution test set for models trained with input dropout (Section 4.5) compared to original models.
Original Input Dropout
0.8
Accuracy
0.6
0.4
0.2
brighthess
spatter
noise
shot Lolee
saturate
roa
contrast
elasticc trensform
frast
pircllte
gaussin blur
gaussian_ Looke
impulse noise
ipeg compressiion
motion blur
7oom,
defocus
glass
CIFAR-10-C C Corruption
(a) ResNet20
Original Input Dropout
0.8
Accuracy
0.4
0.2
alastic transtorm
brightness
fog
frost
giasssian, blur
gaussion .roiee
glass blur-
impuise roise
ipeg compression
picolata
metion blur
saturate
shot noise
snow
sp roise
zoom, blur
defocus
CIFAR-10-C Corruption
(b) ResNet18
Original Input Dropout
0.8
Accuracy
0.6
0.4
0.2
saturate
goussian noise-
impaiio, noise.
jpng compressinn
mation, blur
pinelate-
sholl noibe-
SnoW
spackle, noise
zoom blur
brightre ess-
contract
tog
elastic transform
defocus, blur
glass
gaussian
CIFAR-10-C Corruption
(c) VGG16
Figure S5: Accuracy on individual corruptions of CIFAR- 10-C out-of-distribution images for original models and models trained with input dropout (Section 4.5). Accuracy is given as mean + standard deviation over five replicate models.
7

---

Under review as a conference paper at ICLR 2021
S4 DETAILS OF HUMAN CLASSIFICATION BENCHMARK
Here we include additional details on our benchmark human classification accuracy of sparse pixel-subsets (Section 3.4). Figure S6 shows all images shown to users (100 images each for 5%, 30% and 50% pixel-subsets of CIFAR-10 test images). Each set of 100 images has pixel-subsets stemming from each of the three architectures roughly equally (35 ResNet20, 35 ResNet18, 30 VGG16) Figure S7 shows the correlation between human classification accuracy and pixel-subset size (accuracies shown in Table
Table S3: Human classification accuracy on sample of CIFAR-10 test image pixcl-subsets of varying sparsity (see Section 3.4. Accuracies given as mean + standard deviation.
Fraction of Images
Human Classification Accuracy (%)
5%
19.2 + 4.8
30%
40.0 + 2.5
50%
68.2 + 3.6
"The human classification benchmark was performed using pixcl-subsets computed from earlier implementa- tions of the three CNN architectures (in Keras rather than Py Porch) Figure SS5 shows all pixel-subsets derived in the human classification benchmark. ResNet20 was based
from these models that were shown Keras example using 16 initial filters and optimized with Adam for 200 epochs (batch size 32, initial learning rate 0.001, reduced after epochs 80, 120, 160, and 180 Ic-4, lc-5, le-6, and 5e-7, respectively). 8 based on GutHlub implementation using 64 initial filters, initial strides (1, 1). initial kernel size (3, 3). no initial pooling layer, weight decay 0.0005 and trained using SGD with Nesterov momentum 0.9 for 200 epochs (batch size 128, initial learning rate 0.1, reduced by factor of 5 after epochs 60. 120. and 160). VGG16 was based on a GitHub implementation trained with weight decay 0.0005 and SGD with Nesterov momentum 0.9 for 250 epochs (batch size 128, initial learning rate 0. decayed after each epoch as 0.1 . 0.5 lppec/2h ). We selected the final model checkpoint that maximized test accuracy. We found these models exhibited similar overinterpretation behavior to the final models.
necos: //weras.....//////// 10/
netpa: //g com kerss- een/ ceras con't wttp://///tttttteeeeeeeeeeeeee come b/app Tet tons/ reanet
petpas//5 //9tt///hth Tmang/ wgg/b
L0vgg.2
8

---

Under review as a conference paper at ICLR 2021
(a) Pixel-Subsets
(b) 30% Pixel-Subsets
(c) 50% Pixel-Subsets
Figure S6: Pixel-subsets of CIFAR-10 test images shown to participants in our human classification benchmark (Section 3.4).
Rd = 0.94
0.7
0.0
Accuracy
0.4
0.3
0.1
0.1
0.2
0.3
0.5
Fraction of Images
Figure S7: Human classification accuracy on a sample of CIFAR-10 test image pixel-subsets (see Section 3.4

---

Under review as a conference paper at ICLR 2021
S5 DETAILS OF BATCHED GRADIENT SIS ALGORITHM
computationally infeasible to scale the original backward selection procedure of SIS (Carter et al. 2019) to ImageNet. As each ImageNet image contains 299 x 299 89401 pixels, running backward selection to find one SIS for an image would require ~ 4 billion forward passes through the network. Here we introduce more efficient gradient-based upproximation
the original SIS procedure (via Batched Gradient SIScollection, Batched Gradient BackSelect, and Batched Gradient FindSIS) that allows
find SIS on larger ImageNet images in reasonable time. The Batched Gradient SIScollection procedure described below identifies a complete collection of disjoint masks for input x, where each mask M specifies a pixel-subset of the input X5 x O (1 M such that f(xs 2 r). Here f outputs the probability assigned by the network to its predicted class (i.e., its confidence).
The idea behind our approximation algorithm is two-fold: (1) Instead of separately masking every remaining pixel to find the least critical pixel (whose masking least reduces the confidence in the network's s prediction), we use the gradient with respect to the mask as a means of ordering. (2) Instead of masking just pixel per iteration, we mask larger subsets of 2 pixels per iteration. More formally, let x be an image of dimensions H W x C' where H is the height, W the width, and C the channel. Let fxx be the network's confidence on image x and T the target SIS confidence threshold. Recall that we only compute SIS for images where f(x) 2 T. Let M be the mask with dimensions H x W with 0 indicating an unmasked feature (pixel) and indicating masked feature. We initialize M as all Os (all features unmasked). At iteration i, compute the gradient of f with respect to the input pixels and mask VM = ff @ (1 M)). Here M is the current mask updated after each iteration. In each iteration, we find the block of A features mask, G", chosen in descending order by value of entries in V V. The mask is updated after each iteration by masking this block of & features until all features have been masked. Given p input features, our Batched Gradient S1Scollection procedure returns j sufficient input subsets in O j) evaluations of f (as opposed to Opp2j) evaluations of f in the original SIS procedure (Carter et al. 2019)).
We use A 100 in this paper, which allows us to find one SIS for each of 32 ImageNet images (i.e.. a mini-batchi in 1-1 minutes using Batched Gradient FindSIS. Note that while our algorithm is an approximate procedure, the pixel-subsets produced are real sufficient input subsets, that is they always satisfy [xs 2 r). For CIFAR-10 images (which are smaller in size), we use the original SIS procedure from Carterr et al. 2019). For both datasets, we treat channels of each pixel as a single feature.
Batched Gradient SIScollection(f. x. r, k)
M 0 for j 1,2, ... 1...... do
R = Batched Gradient BackSelect xf,x, k)
M, Batched Gradient FindSIS(f, (f,
M + M + Mj
if f(x о (1 - M)) < return Mi.....M--
end
Batched Gradient BackSelect(f, x, M, k)
R = empty stack while M + 1 do G Top,, (V (1 M)) M)) Update M + G* G*
Push G* onto top of
end return R
10

---

Under review as a conference paper at ICLR 2021
Batched Gradient FindSIS(f, x, R)
M =1
while f(x o (1 M)) do
Pop from top of
Update M M - G
end
f(x @ (1- - M)) 2 r: return M
else: return None
S6 ADDITIONAL RESULTS OF IMAGENET OVERINTERPRETATION
TRAINING CNNS ON IMAGENET PIXEL-SUBSENS
We extracted 10% pixel-subsets by applying Batched Gradient BackSelect to all ImageNet train and validation images using pre-trained Inception v3 and ResNet50 models from PyTorch (Paszke) al. ) We kept the top 10% of pixels and masked the remaining 90% with zeros. We trained new models of the same type these 10% pixel-subsets of ImageNet training set images (training details in Section (ST) and evaluated the resulting models on the corresponding 10% pixel-subsets of ImageNet validation images. Table S4 shows small loss in validation accuracy, suggesting these 10% pixel-subsets that are indiscernible by humans contain statistically valid signals that generalize to validation images. Models trained on 10% pixcl-subsets were trained without data augmentation. As with CIFAR-10 (Section S3) we found training models on pixel-subsets with standard data augmentation techniques (random crops and horizontal flips) resulted in worse validation accuracy.
Table S4: Accuracy of models on ImageNet validation images trained and evaluated full images and 10% BS pixel-subsets. Accuracy for training on 10% BS Subsets is reported as mean + standard deviation (%) over five training runs with different random initialization. For training/evaluation on BS pixel-subeess, we run backward selection on all ImageNet images using a single pre-trained model of each type, but average over five models trained these subsets.
Model
Train On
Evaluate On
Top I Acc.
Top Acc.
Full Images 10% BS Subsets
77.21
93.53
Full Images (pre-trained)
Inception v3
73.87
83.43
10% BS Subsets
10% BS Subsets
71.37
0.15
83.73
0.10
Full Images 10% BS Subsets
76.13
92.86
Full Images (pre-trained)
ResNet50
45.14
64.12
10% BS Subsets
10% BS Subsets
65.71
0.08
80.45 + 0.08
ADDITIONAL EXAMPLES OF SIS ON IMAGENET
Figure SS shows additional examples of SIS (threshold 0.9) on ImageNet validation images from the pre-trained Inception v3 found via Batched Gradient FindSIS.
11

---

Under review as a conference paper at ICLR 2021
Italian greyhound
Pembroke
miniature schnauzer Pomeranian
schipperke
Samoyed
Shetland sheepdog
great grey owl
espresso
banana
bullfrog
little blue heron
basketball
golf ball
bee
jacamar
folding chair
spaghetti squash
plaw
coral fungus
planetarium
Figure S8: Example SIS (threshold 0.9) from ImageNet validation images (top row of each block). The middle rows show the location of SIS pixels (red) and the bottom rows show images with all non-SIS pixels masked but are still classified by the Inception v3 model with > 90% confidence.
12

---

Under review as a conference paper at ICLR 2021
We also explored the relationship between pixel saliency and the order pixels removed by Batched Gradient BackSelect. Surprisingly, as shown in Tigure[S9fon Inception v3, we found that the most salient pixels were often eliminated first and thus unnecessary for maintaining high predicted confidence on the remaining pixel-subsets and subsequently for training on pixel-subsets. Figure STO shows the predicted confidence remaining pixels each step of the Batched Gradient BackSelect procedure for random sample of 32 ImageNet validation images by the Inception v3 model.
bobsled
paddlewheel
projector
eft
cardigan
vizsla
lifeboat
Figure S9: SIS subsets and ordering of pixels removed by Batched Gradient FindSIS a sample of ImageNet validation images that confidently (2 90%) and correctly classified by the Inception v3 model. The top row shows original images, second row shows the location SIS pixels (red), and third row shows images with all non-SIS pixels masked (and are still classified correctly with 90% confidence). The heatmaps in the bottom row depict the ordering of batches of pixels removed during backward selection (blue earliest, yellow latest).
1.0
Contidance masked zoward clss
0.8
20
40
60
100
Parcentage
(ramoving step!
Figure $10: Prediction history remaining (unmasked) pixels at each step of the Batched Gradient BackSelect procedure for random sample of 32 ImageNet validation images by the Inception v3 model. Black line depicts mean confidence at each step.
13


Reviews:
[{"review_id": "valWHRyFpMk", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper proposes \"overinterpretation\" which describes the phenomenon that CNNs could achieve high test accuracy while replying on features that lack semantic meaning. To demonstrate overinterpretation on CIFAR-10 and ImageNet, the authors use Batched Gradient SIS to select a small subset of pixels for each image and trained CNNs on the modified images. While humans can not make accurate predictions on those modified images, CNNs can still achieve high test accuracy. Lastly, the authors propose to use ensembling and input dropout to address overinterpretation. \n\n##########################################################################\n\nReasons for score:\n\nOverall, I vote for rejecting. The main reason is that this paper lacks novelty. While the authors pointed out a realistic issue of CNNs that they could rely on irrelevant features for predictions, this drawback has been largely explored in previous works. Furthermore, the solutions (simple ensembling & input dropout ) proposed by the authors have also been thoroughly studied in previous works.\n\n##########################################################################\n\nPros:\n\nThis paper points out a realistic pathology that is shared by mainstream CNN architectures.\n\nThe paper is well-organized.\n\n\n##########################################################################\n\nCons:\n\nThis paper lacks novelty. Both the phenomenon of \"overinterpretation\" and the proposed solutions have been thoroughly studied in previous works.\n\nWhile the authors show that CNNs could rely on irrelevant features, they did not investigate the cause of their behavior. Specifically, the paper does not investigate whether such pathological behavior is caused by the properties of the datasets or it's originated from the model architecture.\n\nPredictions generated by neural networks are uncalibrated. It's questionable to directly measure the confidence of the models by their raw prediction values as in Section 4.\n\n\n#########################################################################", "extra_scores": {}, "extra_text": {}}, {"review_id": "ca_bzR4gGsy", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work utilizes the SIS (a local feature-importance method) to empirically prove that on existing benchmark datasets, the trained convnets are capable of making decisions based on a very small subset of pixels that are meaningless to the human observer but are nonetheless strong signals. Interestingly, unlike common belief, the same phenomenon is observed in adversarially trained models. The main problem with the work is the discrepancy between claims and results. \nPros:\n* The observed phenomenon is interesting and the work is novel\n* The experimental setup is comprehensive.\n* The writing is decent.\n\nDrawbacks\n* The main drawback with the work is its claim: \n\"that the highly sparse subsets found via backward selection offer a valid predictive signal in the CIFAR-10 benchmark\". \n\"these sparse pixel-subsets are underlying statistical signals that suffice to accurately generalize from the benchmark training data to the benchmark test data\".\n\nI'm not so sure: \n\n1- First, the fact that the observed phenomenon is very model-dependent (other models get low accuracy). This is against the assumption that there generally there exist sparse features that correlate with the class label. It might simply be a nuance of each architecture. One important modification that seems necessary is to create the training set using one architecture and then test the hypothesis using a different one. Just think about a simple scenario: In each architecture, the location of the interpretation mask tends to correlate with the class labels (important features of boat appear at bottom and dog appear at the top). In this case, the observed phenomenon is totally expected but not surprising. It seems like all the experiments on the sparse features are trained \"and\" tested with the \"corresponding\" dataset. I personally cannot be sure that the observed results are not simply indicative of the correlation between class labels and the shape of the SIS masks. One simple way of answering this question is how good a model trained on the sparse features is on clean images. If the answer is yes, then one can claim that there is enough signal that the chosen sparse subsets highly correlate with the label. Note that the reverse experiment (high accuracy of models trained on clean data when predicting sparse images) is not enough as in this experiment, the model itself is used to generate the sparse subset of features (which again means that it's a model-specific mask).\n\n* The tone of the work suggests that this behavior means that existing models will be fragile for out of distribution data. While that might be true, this paper's observations as mentioned above, do not provide enough evidence. The work should either show such OOD samples or create a set of sparse images of CIFAR10 that are classified with high accuracy using any CNN architecture trained on clean images. \n\nQuestions and notes:\n* It seems like choosing 5% of images randomly captures a large amount of signal. This is a very interesting observation.\n* The interpretation being 5-10% of the images is not necessarily an indicator of poor behavior. Although the shown examples are indicative, they are hand-selected. I would like to see the average total variation of masks reported in order to have an idea about how scattered the important pixels are one average\n* The work seems very related to adversarial examples are features, not bugs work, the towards automatic concept-explanations work, and ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. It would be good to explain the relation of this work to each in detail.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Z8UEIjrPo4_", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work reports the problem of image classification datasets (CIFAR-10 and ImageNet) which contains statistical patterns present in both training and tests that can be leveraged by neural networks to achieve high accuracy, but would not be discerned as salient features by humans. Using Sufficient Input Subsets (SIS), they show that retaining the smallest SIS to keep a confidence of 99% leads to spare sets of about 5% of the original pixels and that these subsets of pixels are not salient features for humans. Most importantly they show that training NNs on these SIS from a previously trained network achieves similar results.\n\n===============================\n\nPros:\n\n1. The paper targets a very important subject. Benchmarks are a fundamental part of the progress in machine learning but overreliance on a single metric can be problematic.\n2. The results on models trained on tiny SIS but achieving nevertheless high accuracies are surprising and a sign that statistical artifacts (or sampling bias) are present in both training and test sets (the authors name it ‘valid statistical patterns’).\n3. The experiments with models ensembles and input dropout show that the issue of overinterpretation can be alleviated to some degree.\n\nCons:\n\n1. I am strongly confident that the pathology should not be blamed on the models but rather on the data. I am confident humans trained on the tiny SIS can learn to classify the examples with much greater accuracy than 20%. (More on this in Additional observations below)\n2. Related to the last point, there is no mention in the paper (or I missed it) that all datasets studied in the original paper where SIS is proposed (Carter et al. 2019) do not lead to overinterpretation issues.\n3. SIS and Batched SIS are not clearly presented and defined in the main text. \n\n===============================\n\nReasons for score:\n\nI would vote for a weak accept. The message of the paper has important implications, we cannot use interpretation tools if models use incomprehensible features that are statistical artifacts, or rather quoting the authors ‘interpretability method that faithfully describes the model should output these nonsensical rationales’. However, the presentation of the paper should be improved, for instance there lacks explanation of SIS and Batched SIS in the main paper to help the reader follow. Also, the experiments about methods to alleviate overinterpretation should be presented with results of human scores to better convey the suspected decrease of overinterpretation.\n\n===============================\n\nAdditional observations\n\nI believe the observations are mainly pathologies of the data rather than the models. If such statistical patterns exist in the data that allows generalization, then a model learning such features is not pathological but rather well adapted to its task. Methods to alleviate such issues by biasing models to rely on larger subsets of the input is to me only a way of alleviating the data’s sampling bias. Cormier et al 2019 do not indeed report such pathologies, most probably because the data they studied did not have this issue. \n\nHumans are not trained only on CIFAR-10 to know what a frog or a car is. As authors say in section 4.2, there are statistical artifacts in the dataset that the humans do not know. I am fairly confident a human could be trained on the sparse versions and classify well the test example afterwards. The labels should also be changed to meaningless ones (ex: A, B, C, D, …) so that humans are learning from scratch like the CNNs.\n\n\nI first thought the following to quotes to be contradictory:\n\n‘We also find SIS subsets confidently classified by one model do not transfer to other models. For instance, 5% pixel-subsets derived from CIFAR-10 test images using one ResNet18 model (which classifies them with 94.8% accuracy) are only classified with 25.8%, 29.2%, and 27.5% by another ResNet18 replicate, ResNet20, and VGG16 models , respectively, suggesting there exist many different statistical patterns that a flexible model might learn to rely on [...].’\n\n‘We find models trained solely on these pixel-subsets can classify corresponding test image pixel-subsets with minimal accuracy loss compared to models trained on full images. [...] This result suggests that the highly sparse subsets found via backward selection offer a valid predictive signal in the CIFAR-10 benchmark explointed by models to attain high test accuracy.’\n\nAfter rereading many times I realized the first quote was about using SIS for one replicate on test set and compute test accuracy of another replicate using the same SIS, while the second quote was about training another replicate on the SIS and evaluating it on them. I think this could be made more clear in the text.\n\n===============================\n\nQuestions\n\nWhat is the precise threshold for the SIS? Is it always 99% confidence? But not all examples are predicted with 99% confidence isn’t? Is it 99% of f(original x)?\n\nThere must be a drawback with the Batched Gradient SIS algorithm, like lesser accuracy. Do the authors discuss it in the appendix? I have not found any discussion on that matter. \n\n===============================\n\nTypos\n\nPage 3: [...] for the model to the same -> for the model to make the same\n             [...] a gradient-based to find -> a gradient-based method to find?\n\n===============================\n\nPost-Rebuttal\n\nI thank the authors for their detailed answers.  After reading the other reviews and the author's rebuttal, I maintain my rating of 6 for the paper. My concerns on the description of the SIS methods and results on the proposed mitigation are not addressed.\n\nI am not convinced as R1 and R4 that training on the SIS and testing on the full image is the correct way of testing if SIS is sufficient for the model's predictions. If we would present SIS images with unrelatable labels (A instead of Cat, B instead of Dog, C instead of Boat, etc) to humans and ask them to learn the mappings, I am confident they could achieve good results. As pointed out by other reviewers we can see some patterns in the SIS. Showing a full image afterwards and asking to predict (A, B, C, ...) would be quite difficult however. It's easy to infer the pattern from the full image, but the other way around is more difficult. To me the most important is that a given model architecture can be trained on the SIS of another trained model (with different random initializations) and still be able to learn and generalize. That alone shows in my opinion that the dataset contains undesirable statistical artifacts shared by training and test sets, and as the authors says in the paper '‘interpretability method that faithfully describes the model should output these nonsensical rationales’. \n\nI believe R4 makes a valuable point when saying '[...] I think it might be more useful to look at the mean SIS size of those that are wrongly classified by humans'. This seems to be a better way of gauging what threshold should be used for the size.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6KMqOeRRQtj", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studies “overinterpretation”(provides a high-confidence decision without\nsalient supporting input features) of deep learning models. It modifies previous method Sufficient Input Subsets (SIS) (Carter et al., 2019) to scale to high-dimensional inputs. It detects overinterpretation on both CIFAR10 and ImageNet. In particular, when masking 90% of original pixels, the model can still achieve high accuracy. It then proposes two strategies (ensembles and dropout) to mitigate such overinterpretation artifacts.\n\n################################################\n\nReasons for score:\nThe paper is overall well-written and presents some interesting findings on the so-called “overinterpretation” of DNNs classifiers. The scaled version of SIS and the usage of ensembles / dropout to mitigate overinterpretation is intuitive although not novel. However, I have a concern regarding its experimental results and consequences.\n\n################################################\n\nPros:\n\n+mostly well-written\n\n+important topic and interesting findings\n\n+extensive experiments\n\n+code is provided\n\nCons:\n\n-No results are shown on training on sparse pixels and evaluating on full images\n\nThis is my biggest concern. This experiment seems important since otherwise it is not convincing that the model only uses those remaining pixels to make the decision. I am thus not fully convinced by the statement “We show misclassifications often rely on smaller and more spurious feature subsets suggesting overinterpretation is a serious practical issue”.\n\n-The consequence of the findings needs a bit more discussion. How can one benefit from reducing the “overinterpretation” in practice?\n\n-Only empirical results are presented and not enough discussions on the theoretical side.\n\n-Another minor concern is about the usage of mean SIS size for measuring semantic meaning. It seems to be a reasonable proxy. However, I think it might be more useful to look at the mean SIS size of those that are wrongly classified by humans. If an image is already correctly classified by humans (there are indeed some figures that show the semantic features of the original objects despite only 5-10% pixels showing up, see e.g. Figure 1 bottom row the third to last image shows the contour of a horse), the increase of SIS size may not add much more semantic meaning.\n\n################################################\n\nMinor:\n\n-last paragraph 1st page “although they propose differing explanations for the decisions of a model”, “differing” should be “different”\n\n-sec 3.2 ImageNet, the mask “M” is not defined\n\n################################################\n\nQuestions:\n\n-I am really surprised by the performance of training on 5% random and evaluating on 5% random (around 50% as shown in table 1). What can be a possible explanation here?\n\n-cifar10-c adds visual effect to images. It might be more interesting to also show the performance on spatially transformed images (those should be heavily influenced when only sparse original pixels are used).\n\n\n\n################################################\n\nPost-Rebuttal:\n\nThanks the authors for their detailed response! After reading the responses, I decide to maintain my initial assessment.\n\nThe statement \"full images are highly out-of-distribution for a model trained on images with only 5% unmasked pixel-subsets and hence such a model cannot properly generalize to fully unmasked images\" makes sense. However, I still feel that the authors need an experiment of this flavor to support their claim of “We show misclassifications often rely on smaller and more spurious feature subsets suggesting overinterpretation is a serious practical issue” as mentioned in my initial review as well as pointed out by R1.\n\nBesides, I also find the point \"the observed phenomenon is very model-dependent\" raised by R1 is a valid major concern. In the authors response, they did not add extra experiments to address it. \" We indeed find that models trained on 5% pixel-subsets can generalize to the corresponding 5% pixel-subsets of test images. \" - the stated experiment trains and tests on the same model so it does not address the concern that the observed phenomenon is model-dependent. In order to address this concern, the authors need to add some experiments on transferring across architectures (e.g. train on SIS of ResNet and test on SIS of VGG). \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally feel that the phenomenon discovered in this paper is relevant and could be very important when considering interpretability. However, there are still a number of remaining concerns. The reviewers are not convinced by the human study - they feel there is structure in the SIS’s such that a human trained on these images with an abstract category (i.e., without being told their real-world label) could potentially successfully learn to classify them. There is also a concern that SIS is model-based, that is, the inductive biases of the model (shape, color, etc.) could be leaking information into the SIS image. Finally, there should be some stronger evidence that this represents a serious practical problem for the community. Are there instances where current interpretable approaches break down because of this phenomenon?

One suggestion to potentially strengthen the human experiment: you could try training a denoising autoencoder on the full images, removing 95% of the pixels at random. Then, given an SIS, use the denoising autoencoder to reconstruct the image and then provide that to a human subject. The question is: how much information about the image as a whole is preserved in the SIS (when combined with an appropriate inductive bias)?

Decision: Reject