Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper evaluates pathologies of modern neural networks on CIFAR-10 and ImageNet. Specifically, the authors show that these networks exhibit *overinterpretation*, which is a phenomenon where a model can use sparse subsets of an image which are semantically meaningless to humans in order to confidently and accurately classify the image. The authors utilize the Sufficient Input Subsets (SIS) algorithm from the interpretability literature to identify these subsets, but they also propose modifications to this algorithm which allow it to scale to ImageNet. Surprisingly, the authors find that models evaluated on a small subset of SIS pixels from each image (5% on CIFAR-10 and 10% on ImageNet) exhibit only a small drop in accuracy. The authors explore two techniques for mitigating overinterpretation: ensembling and input dropout.
main_review: **Originality** Though the SIS algorithm is not new, the authors use it to make important and novel observations about neural network behaviors. They also modify the algorithm to be applicable to ImageNet.

**Quality**:  Overall, the submission is technically sound and the experimental results are convincing. However, I have some comments I hope the authors can address:

**Major comments**

1. *Calibration of models*. Since neural networks are known to be poorly calibrated and over confident, thresholding the model confidences based on a fixed parameter may lead to inconsistent results across uncalibrated models. Are the models calibrated before applying SIS?
2. *Use of confidence instead of accuracy in SIS algorithm*.  Would it be possible to set a threshold on the model accuracy instead of the model confidence in the SIS algorithm?  The size of the subset needed to reach a certain accuracy threshold on both in-distribution and out-of-distribution datasets might be a more meaningful way to compare the performance of different models.
3. *Evaluating on full images when training on subsets* In Table 1, it would be interesting to compare the model accuracy on the full images when trained on either the BS or random subsets in order to understand how much training on the subsets hurts model performance.
4. *Size of the subset versus model accuracy* . The authors claim that accuracy and SIS size are correlated for ensembles, but not for models trained with input dropout. For regular models, it’s not clear what the trend is. It would be interesting to see model accuracy on both in-distribution and out-of-distribution data versus SIS size.  
5. *Evaluation on CIFAR-10.1*  CIFAR-10C is a synthetic distribution shift whereas CIFAR-10.1 (Recht et. al. 2018) is a natural distribution shift. Natural distribution shifts are more realistic than synthetic shifts. Moreover, techniques that work well on synthetic shifts do not necessarily transfer to natural distribution shifts (Taori et. al. 2020). The authors should evaluate out-of-distribution robustness on CIFAR-10.1 in addition to CIFAR-10C.

**Minor comments**

6. *Explanation for heatmaps* Can the authors comment on why the ResNet20 heat map is so different from the ResNet18 and VGG16 heat map? Also, why are the ImageNet subsets concentrated on the border of the images, but the same is not true for CIFAR?
7. *SIS size per class* Does the size of the SIS subset vary across ImageNet classes?
8. *Conflicting observations* The observations on lines 267-273 seem to directly conflict with the observations on lines 226-233. Can the authors clarify?

**Significance** The paper shows a way to detect spurious features that a model over-relies on when making predictions.  This has significant implications for robustness research which aims to prevent such over-reliance. 

limitations_and_societal_impact: The authors discuss limitations briefly, but do not address negative societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors define "overinterpretation" as an undesired behavior in deep neural networks where the networks find strong class evidence in semantically unrelated image regions. The behaviour is quantified by identifying the minimal number of pixels sufficient to predict a given class in an input image while masking all the other features/pixels. In this regard, protocols are proposed to evaluate overinterpretation and the models trained on CIFAR and ImageNet are shown to suffer from this problem. 
main_review: Pros: 
+ A batched gradient approach is developed to discover sufficient subsets in the input images. 
+ Semantically irrelevant features contribute towards the prediction of true class in an image, this behavior is analyzed systematically. An interesting result is that masking ratios of up to 90-95% still lead to comparable test accuracies. To the best of my knowledge, this has not been shown earlier in this context. 
+ Approaches to mitigate overinterpretation are also analyzed with some useful recommendations such as model ensembling. 
+ Good number of experiments are performed to study the overinterpretation problem. The analysis also shows that the SIS size is related to the model performance on clean images. 
+ Paper is clearly written with nice illustrations and visualizations. 

Cons/Questions:
- The term SIS (sufficient input subsets) [4] is not defined before its use in Line 11.
- ResNet and VGG family models are explored to study overinterpretation. Some recent works (e.g., [A]) in the literature study randomly masking inputs and found ViTs to have much better robustness compared to CNNs.  It will be interesting to see how the robustness holds against masks learned in a white-box setting, similar to the setting studied in this work. 
- It is important to understand here if the SIS patterns are specific to the input data distribution/image or the target model. There is one experiment on this point in the supplementary material for CIFAR10 dataset where different architectures are tried out with the same SIS. Due to the significance of this expeirment, the authors can improve it further with an analysis on a large dataset like ImageNet or across an altogether different family e.g., ViTs (as an extension to the above comment).
- The human study is performed with 4 participants. Were they shown sufficient examples of 5% SIS images to learn about the 10 CIFAR classes. Without such a precursor learning task, this experiment seems to be biased towards the model since SIS are found using the model in a white-box setting and therefore would make sense to the model but not for humans. A revserse eperiment has also been done in the literature [13] where humans can easily classifly textureless objects while DNN models suffer. 

[A] Naseer et al., "Intriguing Properties of Vision Transformers", Arxiv 2021. 
limitations_and_societal_impact: The limitations and societal impact is appropriately discussed in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents an interesting finding that CNNs overinterpret a few non-meaningful pixels in the image to make accurate predictions. The authors use SIS and extend it to Batched Gradient SIS to discover the limited number of pixels such CNNs rely on. The authors conduct extensive experiments on two major image datasets to verify the finding. Besides, ensembling and input dropout are explored to mitigate the overinterpretation problem.
main_review: **Originality**: One of my concerns is the novelty of the method part. The proposed Batched Gradient SIS method seems to be a straightforward extension to SIS for better efficiency.

**Quality**: The paper contains extensive experiments on two image datasets to verify the overinterpretation problem (e.g., detailed analysis in Sec. 4.1 - 4.4 provides). One minor concern is that SIS pixels’ patterns are different between Fig. 1 and Fig. 3. Is it due to different datasets or due to different SIS algorithms (vanilla SIS vs. Batched Gradient SIS). Adding an experiment to show the SIS pixels from Batched Gradient SIS method on the CIFAR-10 dataset may explain that difference.

**Clarity**: The paper is well-written and easy to follow.

**Significance**: The finding of the paper, i.e., overinterpretation, is significant, which can facilitate future research to solve the overinterpretation problem for achieving more accurate and robust (not relying on spurious pixels) image classification models.

In summary, although the paper’s method is not novel, I believe that the paper’s main focus, i.e., revealing and analyzing the overinterpretation problem, does make a good contribution to the research community. Thus my rating is “Marginally above the acceptance threshold” to this paper.
limitations_and_societal_impact: Although the ensembling and input dropout methods do not completely mitigate the overinterpretation issue (L471-474), the paper’s major focus is revealing and analyzing the overinterpretation problem. It is acceptable to leave it for future work. Therefore, the paper’s limitation is well-addressed. In terms of the potential negative societal impact, I do not think “training datasets may become more complex” is negative. In fact, removing such spurious signals in the dataset can make the model more robust, leading to a positive impact. Therefore, I do not think the paper has potential negative impacts because the paper tries to reduce them.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper identifies sparse sets of pixels which can be used with deep neural nets to classify images with high confidence. The authors demonstrate results with CIFAR-10, an augmentation corrupted version of it and with ImageNet. They show a number of experiments to obtain insights in why these spurious subsets are classified with high confidence.

For example, they show that the sparse subsets can be used to train a different model to predict well on CIFAR-10 and on Imagenet.
Furthermore they show that the SIS sizes are larger on correctly classified images compared to misclassified images, which implies that the sets are indeed used to properly predict on images.
main_review: clarity: it is an easy read, well written.

originality: The novelty of batchgradientSIS is limited. It is a natural iterative extension of the brute force SIS optimization. The paper would be more straightforward if one stressed the understanding aspect rather than the novelty of batchgradientSIS.

The existence of sparse subsets has not been analyzed so far, though shape-based background-artefacts in Imagenet have been observed in Finding and Removing Clever Hans: Using Explanation Methods to Debug and Improve Deep Models  (see some later figures in this paper).
The paper has a novelty in discussing pathologies of neural nets.

quality:
The experiments are well designed, to analyze various hypotheses, e.g. using sparse subsets discovered by one model to train another model and obtain similar accuracy.

as criticism points on quality:

missing citations:

-Regarding optimizing masks to retain minimal examples: Fong, Patrick, Vedaldi, Understanding Deep Networks via Extremal Perturbations and Smooth Masks ICCV2019
-By computing attributions and using attribution-score to guide iterative masking it belongs to a class of approaches which were used for CNNs in Samek et al, Evaluating the visualization of what a deep neural network has learned, TNNLS 2017
-Occlusion methods - e.g. Agarwal, Nguyen ACCV 2020
-Pertinent positives, Dhurandhar et al. 2018. While this method may have some degrees of freedom and ambiguities, it can be still mentioned.

significance: this paper adds a new angle to the discussion on deep neural network pathologies.

Despite limited novelty in methods, the reviewer finds the paper acceptable because it discusses a statistical defect of NNs from a new angle, and thus increases the understanding of existing algorithms rather than proposing a novel algorithm.

limitations_and_societal_impact: 
Societal Impact:

This paper shows that CIFAR-10 is similar to MNIST a dataset suffering from simplicity, and results on it should be taken with care when it comes to transferability to other more complex datasets. It can be assumed that not every researcher will like to hear this kind of news. So it is worth to bring this in NeurIPS.

Limitations:

One should give a justification why batchgradSIS is chosen over e.g. optimizing masks and occlusion methods (or integrated gradients?) . Reviewer guesses likely due to fast runtime, or maybe the authors were interested in a greedy optimization which does not have strong connectedness priors on the remaining unmasked regions for the sake of mining unconnected sets?

Likely the noisyness of gradients (Balduzzi, The Shattered Gradients Problem: If resnets are the answer, then what is the question? , 2017) probably also helps in finding disconnected subsets.


The work can be put in a wider context of adversarial optimization, although it does not aim at small changes in the input but rather semantically meaningless examples as the original "neural networks are easily fooled" paper by AM Nguyen.

see also the missing citations in the box before.
review_title: null

Meta-review: This work argues that deep neural networks overinterpret the input, i.e. they rely on too small portions of the image to make the decision. This is novel insight and I am enthusiastic that it will help the community understand better deep networks. All reviewers agree that the paper should be accepted, and I am glad to support this decision. Please remember to address all reviewers' comments in the camera-ready version.
Decision: Accept (Poster)