Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors present a procedure to check the relative influence on model prediction between pixels with largest input gradient magnitudes versus pixels with lowest input gradient magnitudes. This is an extension of ROAR, which trains a model on data with certain pixels masked out. The authors evaluate DiffROAR on 4 image datasets and 1 synthetic dataset, showing that more robust models are more likely to have input gradients on relevant pixels than non-robust models. They also present a theoretical analysis showing that a standard MLP with one hidden layer can violate the assumption that higher input gradients mean more relevant pixels.
main_review: *** Originality & Significance ***: The authors do a thorough job with citing related works. The very closely-related prior work of ROAR differs in one main aspect: ROAR checks the relevance of pixels with high measured importance, and DiffROAR compares the importance of those with pixels with low measured importance. The goal of the paper seems limited, in that it seems two key claims have already been established in prior work - to quote from the abstract:

"Our results suggest that (i) input gradients of standard models (i.e., trained on original data) may grossly violate (A), whereas (ii) input gradients of adversarially robust models satisfy (A) reasonably well."

As supported in Line 112,  (i) is established in the related ROAR work: 

"...the ROAR framework show[s] that multiple attribution methods, including vanilla input gradients, are no better than model-independent random attributions that lack explanatory power"

And as supported in Line 120, (ii) is established in the work of Kim et. al. [cited as 34]:

"Additionally, Kim et al. [34] use the ROAR framework to conjecture that adversarial training “tilts” input gradients to better align with the data manifold."

The authors write that their goal here is to augment ROAR "to understand when and why input gradients violate assumption (A)". This is partly addressed in their section on experiments (Section 4), by showing input gradients are more often higher magnitude on relevant pixels with robust models than in standard models. However, [34] has already addressed this in much more detail (the boundary tilting theory).

On the other hand, the theory in Section 6 seemed more significant to me. I am not as well versed in the theory literature in this area as compared to feature attribution, and cannot speak to originality very well. But, the result was very interesting - it's possible for a max-margin classifier (of a one-hidden-layer MLP) to place equal input gradient over locations where relevant features have appeared in the data, and 0 gradient on locations holding only noise. It was interesting to see this happen in BlockMNIST as well.

I am recommending a score of 5, because the majority of the paper's goal seems to have been already considerably explored in past papers (i.e., addressed the exact question of the paper's title), as evidenced in the related works section. In addition, the authors' stated contribution of exploring "when and why input gradients violate assumption (A)" seems to have been addressed more thoroughly in the paper cited as [34].

*** Clarity ***: The paper is very well-written, with only minor typos (e.g. missing period on Line 363). I thought it was enjoyable to read, and well-thought-out in its organization. The theory section was confusing at first - I didn't understand the difference between task-relevant and instance-specific, yet also task-relevant, features (e.g., the sentence on Lines 306-308 was not clear at all). Yet, Figure 5 cleared it up.
limitations_and_societal_impact: The limitations were very briefly discussed at the end. It was surprising to me that the authors chose to solely focus on input gradients, when there are so many other existing feature attribution methods, as cited by the authors. They did mention expanding to these as future work.

There is no discussion on potential negative societal impact. One example societal connection: because the authors are pointing out a flaw in the connection between input gradients and true feature importance, input gradients should not be a trusted measure of pixel importance (so if the claim that model is using "fair" features is based on input gradients, that should be taken with skepticism).
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates whether input gradients (of robust and normally trained models) highlight discriminative features. As part of this investigation, it introduces DiffROAR (a modified version of ROAR for evaluating the accuracy of feature discriminativeness rankings), as well as the BlockMNIST dataset (plus a simplified version where they perform theoretical analysis). It introduces and discusses the idea of feature leakage as an explanation for why non-robust model gradients sometimes highlight non-discriminative features.
main_review: **Originality:** Overall, I think the paper meets the originality threshold. Although DiffROAR is only a slight modification of ROAR, the paper mainly uses it as an analytic tool to investigate an important question in a novel way. Additionally, the simplified and full versions of BlockMNIST are novel.

**Quality:** The submission seems technically sound, with extensive experiments and theoretical analysis. I appreciate how they use multiple distinct methods for testing whether features are truly discriminative (synthetic datasets with ground truth and real data with DiffROAR) to reach consistent and coherent conclusions.

**Clarity:** The submission is well-written; every component was easy to understand, and I feel confident that its results are reproducible to readers.

**Significance:** I think the results are reasonably significant, and help advance our current state of understanding on both input gradients and adversarial robustness (in a novel way). However, I do think the work can and should do more to provide a fuller understanding of its results, particular around _why_ robustness prevents feature leakage; I'll elaborate in the next section.
limitations_and_societal_impact: I have a few suggestions for improvements, that could increase the impact of the paper:

- The paper seems to take it as a given that input gradients _ought to_ highlight discriminative features (and certainly many have assumed that), but maybe it's worth stepping back from that even further. In the case of BlockMNIST, it makes total sense that a model might learn to independently detect the same pattern in both sides of the image, and be sensitive to changes that would create a digit in the null block (which is what input gradients actually communicate -- that the model is sensitive to counterfactual changes that would introduce a new digit, even if that's in an unexpected place). To me, these hypothetical features seem totally worthy of being called "discriminative"; the only real distinction is that adding them would take us off the data manifold.

- Relatedly, recent research (e.g. <https://arxiv.org/pdf/1906.00945.pdf> and <https://arxiv.org/pdf/2106.10151.pdf>) suggests that adversarial training encourages model gradients and nearby decision boundaries to lie within the data manifold. In the case of BlockMNIST, that seems like it would likely prevent feature leakage (by preventing model gradients from adding phantom second digits to the null block, and instead forcing them to make more realistic changes to the signal block), and it's even more clear in the simplified 10-feature case (anything with multiple +1s or -1s in the signal block is off the data manifold). I think it's worth discussing this idea because it provides a potential explanation for _why_ adversarial robustness prevents feature leakage (which the paper defers to future work, but doesn't have to). This would really help complete the story and improve the usefulness of the work.

-  While I agree with the authors that DiffROAR is a very useful tool for investigating assumption (A) in this particular context, it might be worth clarifying its limitations if others wish to reuse it in other contexts. For example, consider a redundant dataset where features are either all negative (in which case y=0) or all positive (in which case y=1). In such cases, no feature is more or less informative than any other, so no information can be gained by ranking or removing features. (It also might be worth including some results in the appendix comparing DiffROAR to vanilla ROAR, though it's not critical.)
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors test the current assumption for gradient-based explanation methods, i.e., a high magnitude of vanilla gradients highlight more discriminative task-relevant features and vice-versa. The authors theoretically and empirically test the above assumption using DiffROAR, a new evaluation framework, and BlockMNIST, a new semi-real dataset. Empirical analysis using DiffROAR concludes that gradients of standard models violate the above assumption. whereas gradients of adversarially trained models don't. The BlockMNIST dataset separates the signal and noise part of a given sample by encoding a priori knowledge of discriminative and non-discriminative features in the samples. The work tackles one of the existing problems of gradient-based explanation methods and suggests that the proposed frameworks and datasets can be used as sanity checks to audit instance-specific interpretability methods.
main_review: Strengths:
1. The paper is well-written with coherent details of the proposed DiffROAR evaluation metric and BlockMNIST dataset.

2. The proposed DiffROAR evaluation metric is an intuitive extension of the previously proposed ROAR metric.

3. The empirical findings that standard one-hidden-layer MLPs violate the gradient assumptions are supported by respective theoretical analyses.

Weaknesses:
1. In the BlockMNIST dataset design, the authors mention -- unlike the MNIST signal block that is fully predictive of the class, the non-discriminative null block contains no information about the class. The argument is not clear because CNNs (both standard and adversarially trained) learn low-level features, such as horizontal and vertical edges, in their earlier layers, and they will be identified even for the null blocks in the dataset. Hence, contributing to the actual predicting class. A sanity check would have been to remove the null blocks from the images and check whether the classification performance of the model remains unchanged.

2. In the feature leakage hypothesis, the authors mention that standard models violate the gradient assumption because the discriminative signal blocks are randomly placed in the images and propose the BlockMNIST-Top dataset variant that fixed the location of the signal block. Some open questions regarding this experiment are: a) was the feature leakage due to the backpropagated gradients taken from a batch of images? One can expect that using a larger batch size will help ameliorate the effect of placing signal blocks in random locations, b) was the two models trained on BlockMNIST and BlockMNIST-Top trained using the same weight initializations?, and c) the feature leakage hypothesis contradicts that CNNs are invariant to translation, i.e., if we translate the inputs the CNN will still be able to detect the class to which the input belongs.

3. Bansal et al. 2020 analyzed the differences between the vanilla gradients of standard and robust models with similar findings that input gradients of adversarially trained models are more stable than their standard counterparts. Chen et al. 2020 hypothesize that robust models learn smoother kernels as feature extractors, and hence the gradients are stable with respect to non-discriminatory signals.

4. Previous works have detailed the disadvantages of replacing masked features with heuristics such as gray color, blurring, or random noise. It would be great if the authors can comment about the importance of choosing the removing operator in their analysis.

[1] Chen et al., The shape and simplicity biases of adversarially robust ImageNet-trained CNNs, 2020.
[2] Bansal et al., SAM: The sensitivity of attribution methods to hyperparameters, 2020.
limitations_and_societal_impact: Yes, the authors briefly mention the works' limitations. One of the major drawbacks of this work is its scope. The authors exclusively focus on vanilla input gradients, whereas works like Bansal et al. 2020 have already studied improved gradient variants like SmoothGrad and showed their differences for standard and adversarially trained models. It would be great to show similar findings for other gradient-based explanation methods.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper studies the validity of a key assumption (i.e. Assumption A) made in popular post-hoc attribution methods empirically and theoretically. Comprehensive experiments are conducted with specifically designed BlockMNIST. Moreover, the authors constructs a simplified setting for theoretical analysis to demonstrate feature leakage.
main_review: This work provides a series of systematic probes like BlockMNIST into the effects of input gradients in feature attribution. With the extensive studies for evaluating the feature leakage hypothesis for violating the Assumption (A), It provides insights for several aspects like why input gradients of standard models tend to violate assumption (A) including standard and robust models. I'm taken by the way to challenge the common attribution assumptions in interpretability and it will potentially benefit the development of this line of work.
limitations_and_societal_impact: A few limitations can be the restructre of the introduction section: the paper is a bit hard to follow with terms from the interpretability area and the intro part is introducing related works and case studies without a clear storyline.

Moreover, the feature leakage hypothesis is not very well justified in the BlockMNIST setting.

More investigations about whether the probes in this work can be leveraged to shed light on the predictive features containing group-sensitive attributes (male vs female) should be done to improve the potential societal impact of this work.
review_title: null

Meta-review: Thank you for your submission to NeurIPS.  The reviewers and I are in agreement that the paper presents interesting new insights into the problem of understanding the significance of input gradients as they concern robust and nonrobust models.  The addition of the BlockMNIST data set seems particularly helpful to better understanding and furthering research in the area.  I'm happy to recommend that the paper be accepted.

Since there was a fair amount of response to reviewer feedback, I'd mainly highlight that many of the comments seemed to center around further (discussion-based, not quantitative) comparison to previous works (such as [34] and Ilyas et a, 2019 mentioned by the reviewer).  Overall I think that relating the results here more to potentially qualify phenomena observed in previous studies, at least to the extent possible without substantial revisions, would improve most upon the current paper.
Decision: Accept (Poster)