Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work reports the problem of image classification datasets (CIFAR-10 and ImageNet) which contains statistical patterns present in both training and tests that can be leveraged by neural networks to achieve high accuracy, but would not be discerned as salient features by humans. Using Sufficient Input Subsets (SIS), they show that retaining the smallest SIS to keep a confidence of 99% leads to spare sets of about 5% of the original pixels and that these subsets of pixels are not salient features for humans. Most importantly they show that training NNs on these SIS from a previously trained network achieves similar results.

===============================

Pros:

1. The paper targets a very important subject. Benchmarks are a fundamental part of the progress in machine learning but overreliance on a single metric can be problematic.
2. The results on models trained on tiny SIS but achieving nevertheless high accuracies are surprising and a sign that statistical artifacts (or sampling bias) are present in both training and test sets (the authors name it ‘valid statistical patterns’).
3. The experiments with models ensembles and input dropout show that the issue of overinterpretation can be alleviated to some degree.

Cons:

1. I am strongly confident that the pathology should not be blamed on the models but rather on the data. I am confident humans trained on the tiny SIS can learn to classify the examples with much greater accuracy than 20%. (More on this in Additional observations below)
2. Related to the last point, there is no mention in the paper (or I missed it) that all datasets studied in the original paper where SIS is proposed (Carter et al. 2019) do not lead to overinterpretation issues.
3. SIS and Batched SIS are not clearly presented and defined in the main text. 

===============================

Reasons for score:

I would vote for a weak accept. The message of the paper has important implications, we cannot use interpretation tools if models use incomprehensible features that are statistical artifacts, or rather quoting the authors ‘interpretability method that faithfully describes the model should output these nonsensical rationales’. However, the presentation of the paper should be improved, for instance there lacks explanation of SIS and Batched SIS in the main paper to help the reader follow. Also, the experiments about methods to alleviate overinterpretation should be presented with results of human scores to better convey the suspected decrease of overinterpretation.

===============================

Additional observations

I believe the observations are mainly pathologies of the data rather than the models. If such statistical patterns exist in the data that allows generalization, then a model learning such features is not pathological but rather well adapted to its task. Methods to alleviate such issues by biasing models to rely on larger subsets of the input is to me only a way of alleviating the data’s sampling bias. Cormier et al 2019 do not indeed report such pathologies, most probably because the data they studied did not have this issue. 

Humans are not trained only on CIFAR-10 to know what a frog or a car is. As authors say in section 4.2, there are statistical artifacts in the dataset that the humans do not know. I am fairly confident a human could be trained on the sparse versions and classify well the test example afterwards. The labels should also be changed to meaningless ones (ex: A, B, C, D, …) so that humans are learning from scratch like the CNNs.


I first thought the following to quotes to be contradictory:

‘We also find SIS subsets confidently classified by one model do not transfer to other models. For instance, 5% pixel-subsets derived from CIFAR-10 test images using one ResNet18 model (which classifies them with 94.8% accuracy) are only classified with 25.8%, 29.2%, and 27.5% by another ResNet18 replicate, ResNet20, and VGG16 models , respectively, suggesting there exist many different statistical patterns that a flexible model might learn to rely on [...].’

‘We find models trained solely on these pixel-subsets can classify corresponding test image pixel-subsets with minimal accuracy loss compared to models trained on full images. [...] This result suggests that the highly sparse subsets found via backward selection offer a valid predictive signal in the CIFAR-10 benchmark explointed by models to attain high test accuracy.’

After rereading many times I realized the first quote was about using SIS for one replicate on test set and compute test accuracy of another replicate using the same SIS, while the second quote was about training another replicate on the SIS and evaluating it on them. I think this could be made more clear in the text.

===============================

Questions

What is the precise threshold for the SIS? Is it always 99% confidence? But not all examples are predicted with 99% confidence isn’t? Is it 99% of f(original x)?

There must be a drawback with the Batched Gradient SIS algorithm, like lesser accuracy. Do the authors discuss it in the appendix? I have not found any discussion on that matter. 

===============================

Typos

Page 3: [...] for the model to the same -> for the model to make the same
             [...] a gradient-based to find -> a gradient-based method to find?

===============================

Post-Rebuttal

I thank the authors for their detailed answers.  After reading the other reviews and the author's rebuttal, I maintain my rating of 6 for the paper. My concerns on the description of the SIS methods and results on the proposed mitigation are not addressed.

I am not convinced as R1 and R4 that training on the SIS and testing on the full image is the correct way of testing if SIS is sufficient for the model's predictions. If we would present SIS images with unrelatable labels (A instead of Cat, B instead of Dog, C instead of Boat, etc) to humans and ask them to learn the mappings, I am confident they could achieve good results. As pointed out by other reviewers we can see some patterns in the SIS. Showing a full image afterwards and asking to predict (A, B, C, ...) would be quite difficult however. It's easy to infer the pattern from the full image, but the other way around is more difficult. To me the most important is that a given model architecture can be trained on the SIS of another trained model (with different random initializations) and still be able to learn and generalize. That alone shows in my opinion that the dataset contains undesirable statistical artifacts shared by training and test sets, and as the authors says in the paper '‘interpretability method that faithfully describes the model should output these nonsensical rationales’. 

I believe R4 makes a valuable point when saying '[...] I think it might be more useful to look at the mean SIS size of those that are wrongly classified by humans'. This seems to be a better way of gauging what threshold should be used for the size.
review_title: Interesting use of interpretability tools to highlight data biases/statistical artifacts.

final_score: rating: 3: Clear rejection
review_content: The work utilizes the SIS (a local feature-importance method) to empirically prove that on existing benchmark datasets, the trained convnets are capable of making decisions based on a very small subset of pixels that are meaningless to the human observer but are nonetheless strong signals. Interestingly, unlike common belief, the same phenomenon is observed in adversarially trained models. The main problem with the work is the discrepancy between claims and results. 
Pros:
* The observed phenomenon is interesting and the work is novel
* The experimental setup is comprehensive.
* The writing is decent.

Drawbacks
* The main drawback with the work is its claim: 
"that the highly sparse subsets found via backward selection offer a valid predictive signal in the CIFAR-10 benchmark". 
"these sparse pixel-subsets are underlying statistical signals that suffice to accurately generalize from the benchmark training data to the benchmark test data".

I'm not so sure: 

1- First, the fact that the observed phenomenon is very model-dependent (other models get low accuracy). This is against the assumption that there generally there exist sparse features that correlate with the class label. It might simply be a nuance of each architecture. One important modification that seems necessary is to create the training set using one architecture and then test the hypothesis using a different one. Just think about a simple scenario: In each architecture, the location of the interpretation mask tends to correlate with the class labels (important features of boat appear at bottom and dog appear at the top). In this case, the observed phenomenon is totally expected but not surprising. It seems like all the experiments on the sparse features are trained "and" tested with the "corresponding" dataset. I personally cannot be sure that the observed results are not simply indicative of the correlation between class labels and the shape of the SIS masks. One simple way of answering this question is how good a model trained on the sparse features is on clean images. If the answer is yes, then one can claim that there is enough signal that the chosen sparse subsets highly correlate with the label. Note that the reverse experiment (high accuracy of models trained on clean data when predicting sparse images) is not enough as in this experiment, the model itself is used to generate the sparse subset of features (which again means that it's a model-specific mask).

* The tone of the work suggests that this behavior means that existing models will be fragile for out of distribution data. While that might be true, this paper's observations as mentioned above, do not provide enough evidence. The work should either show such OOD samples or create a set of sparse images of CIFAR10 that are classified with high accuracy using any CNN architecture trained on clean images. 

Questions and notes:
* It seems like choosing 5% of images randomly captures a large amount of signal. This is a very interesting observation.
* The interpretation being 5-10% of the images is not necessarily an indicator of poor behavior. Although the shown examples are indicative, they are hand-selected. I would like to see the average total variation of masks reported in order to have an idea about how scattered the important pixels are one average
* The work seems very related to adversarial examples are features, not bugs work, the towards automatic concept-explanations work, and ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. It would be good to explain the relation of this work to each in detail.
review_title: Interesting experiments but unrelated claims 

final_score: rating: 2: Strong rejection
review_content: ##########################################################################

Summary:

This paper proposes "overinterpretation" which describes the phenomenon that CNNs could achieve high test accuracy while replying on features that lack semantic meaning. To demonstrate overinterpretation on CIFAR-10 and ImageNet, the authors use Batched Gradient SIS to select a small subset of pixels for each image and trained CNNs on the modified images. While humans can not make accurate predictions on those modified images, CNNs can still achieve high test accuracy. Lastly, the authors propose to use ensembling and input dropout to address overinterpretation. 

##########################################################################

Reasons for score:

Overall, I vote for rejecting. The main reason is that this paper lacks novelty. While the authors pointed out a realistic issue of CNNs that they could rely on irrelevant features for predictions, this drawback has been largely explored in previous works. Furthermore, the solutions (simple ensembling & input dropout ) proposed by the authors have also been thoroughly studied in previous works.

##########################################################################

Pros:

This paper points out a realistic pathology that is shared by mainstream CNN architectures.

The paper is well-organized.


##########################################################################

Cons:

This paper lacks novelty. Both the phenomenon of "overinterpretation" and the proposed solutions have been thoroughly studied in previous works.

While the authors show that CNNs could rely on irrelevant features, they did not investigate the cause of their behavior. Specifically, the paper does not investigate whether such pathological behavior is caused by the properties of the datasets or it's originated from the model architecture.

Predictions generated by neural networks are uncalibrated. It's questionable to directly measure the confidence of the models by their raw prediction values as in Section 4.


#########################################################################
review_title: A below-borderline work that lacks novelty.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
This paper studies “overinterpretation”(provides a high-confidence decision without
salient supporting input features) of deep learning models. It modifies previous method Sufficient Input Subsets (SIS) (Carter et al., 2019) to scale to high-dimensional inputs. It detects overinterpretation on both CIFAR10 and ImageNet. In particular, when masking 90% of original pixels, the model can still achieve high accuracy. It then proposes two strategies (ensembles and dropout) to mitigate such overinterpretation artifacts.

################################################

Reasons for score:
The paper is overall well-written and presents some interesting findings on the so-called “overinterpretation” of DNNs classifiers. The scaled version of SIS and the usage of ensembles / dropout to mitigate overinterpretation is intuitive although not novel. However, I have a concern regarding its experimental results and consequences.

################################################

Pros:

+mostly well-written

+important topic and interesting findings

+extensive experiments

+code is provided

Cons:

-No results are shown on training on sparse pixels and evaluating on full images

This is my biggest concern. This experiment seems important since otherwise it is not convincing that the model only uses those remaining pixels to make the decision. I am thus not fully convinced by the statement “We show misclassifications often rely on smaller and more spurious feature subsets suggesting overinterpretation is a serious practical issue”.

-The consequence of the findings needs a bit more discussion. How can one benefit from reducing the “overinterpretation” in practice?

-Only empirical results are presented and not enough discussions on the theoretical side.

-Another minor concern is about the usage of mean SIS size for measuring semantic meaning. It seems to be a reasonable proxy. However, I think it might be more useful to look at the mean SIS size of those that are wrongly classified by humans. If an image is already correctly classified by humans (there are indeed some figures that show the semantic features of the original objects despite only 5-10% pixels showing up, see e.g. Figure 1 bottom row the third to last image shows the contour of a horse), the increase of SIS size may not add much more semantic meaning.

################################################

Minor:

-last paragraph 1st page “although they propose differing explanations for the decisions of a model”, “differing” should be “different”

-sec 3.2 ImageNet, the mask “M” is not defined

################################################

Questions:

-I am really surprised by the performance of training on 5% random and evaluating on 5% random (around 50% as shown in table 1). What can be a possible explanation here?

-cifar10-c adds visual effect to images. It might be more interesting to also show the performance on spatially transformed images (those should be heavily influenced when only sparse original pixels are used).



################################################

Post-Rebuttal:

Thanks the authors for their detailed response! After reading the responses, I decide to maintain my initial assessment.

The statement "full images are highly out-of-distribution for a model trained on images with only 5% unmasked pixel-subsets and hence such a model cannot properly generalize to fully unmasked images" makes sense. However, I still feel that the authors need an experiment of this flavor to support their claim of “We show misclassifications often rely on smaller and more spurious feature subsets suggesting overinterpretation is a serious practical issue” as mentioned in my initial review as well as pointed out by R1.

Besides, I also find the point "the observed phenomenon is very model-dependent" raised by R1 is a valid major concern. In the authors response, they did not add extra experiments to address it. " We indeed find that models trained on 5% pixel-subsets can generalize to the corresponding 5% pixel-subsets of test images. " - the stated experiment trains and tests on the same model so it does not address the concern that the observed phenomenon is model-dependent. In order to address this concern, the authors need to add some experiments on transferring across architectures (e.g. train on SIS of ResNet and test on SIS of VGG). 

review_title: Interesting empirical results on “overinterpretation” of DNNs with a caveat

Meta-review: The reviewers generally feel that the phenomenon discovered in this paper is relevant and could be very important when considering interpretability. However, there are still a number of remaining concerns. The reviewers are not convinced by the human study - they feel there is structure in the SIS’s such that a human trained on these images with an abstract category (i.e., without being told their real-world label) could potentially successfully learn to classify them. There is also a concern that SIS is model-based, that is, the inductive biases of the model (shape, color, etc.) could be leaking information into the SIS image. Finally, there should be some stronger evidence that this represents a serious practical problem for the community. Are there instances where current interpretable approaches break down because of this phenomenon?

One suggestion to potentially strengthen the human experiment: you could try training a denoising autoencoder on the full images, removing 95% of the pixels at random. Then, given an SIS, use the denoising autoencoder to reconstruct the image and then provide that to a human subject. The question is: how much information about the image as a whole is preserved in the SIS (when combined with an appropriate inductive bias)?

Decision: Reject