Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper examines gradient-based attribution methods that have been proposed in the explainability literature from a theoretical perspective motivated by a recent observation in energy-based generative models. 
First, the authors point out a general weakness of gradient-based attribution that derives from the fact that input-gradients do not provide well-defined explanations, since the shift-invariance of the softmax output makes them arbitrary.
The authors then propose that the reason for the success of gradient-based attribution models can be explained by the fact that discriminative models "contain an implicit" class-conditional density model (the mentioned recent observation about energy-based generative models).
They then go on to elaborate on this idea showing how aligning the implicit class-conditional generative model to the "true" generative model of the data would help provide relates to gradient-based attribution, how the alignment can be efficiently promoted with a novel implementation of score-matching, and how this mechanism can be practically realized as regularization costs.
The authors then carry out empirical studies that convincingly confirm the prediction of their theoretical ideas. First, they show that samples generated with score-matching and the proposed gradient-norm regularization are better in the sense of being less noisy and in terms of their discriminative accuracy via a trained discriminative model as proposed by the "GAN-test approach".
Finally, they show that the quality of gradient-based explanations are better according to a discriminative version of the pixel perturbation test, a method to evaluate gradient explanations by perturbing pixels ranked in increasing order of relevance.
In conclusion, this paper establishes very interesting fundamental theoretical connections between discriminative models, energy-based generative models, and gradient-based explanations, uses this theoretical framework to explain how gradient-based explanation are overcoming the softmax shift-invariance problem (also pointed out in this paper), and introduces practical training procedures to take advantage of the gained theoretical insights to generate better explanations, which are also empirically verified in simulations.
review_title: Insightful theory grounded dissection of gradient-based explainability methods

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: Summary\
The key message of this paper is that input-gradients (gradient of the logit wrt to input) or loss-gradients are/might be unrelated to the discriminative capabilities of a DNN. The input-gradient is a key primitive in several interpretability and visualization methods. Until now, it has been taken as a given that these gradients reveal 'why' or what parts of the inputs the model is sensitive to. However, this paper questions this reasoning and says that if the input-gradients can be easily manipulated without changing the generalization ability of the model, then does the input-gradient really contain discriminative signals?  

To test their hypothesis, the paper re-interprets the input-gradient as class-conditional generative model using the score matching view. The paper then develops a 'regularizer' that is called a taylor trace estimator that requires less backward passes than the hutchinson estimator, which when added to the model objective can modulate how 'generative' the model is. With this regularizer, the papers tests the hypothesis that improving the implicit density model also improves input-gradient interpretability. The paper tests this through experiments on an image dataset and finds that this is the case.

Significance\
This work has far reaching significance for the field of visual interpretability of DNNs. It suggests that reading into these input-gradients might be akin to reading tea leaves. The key insight in this work is simple and the demonstration is quite powerful in my opinion. The argument in this paper seems obvious in hindsight, but that is exactly why the paper is a significant one. I have several additional questions later in my review, but this work is important and suggests that insights based on input-gradients might be spurious. 

Clarity\
Overall, the paper is relatively clear and easy to read. Several of the key experiments are well-justified.


Originality\
The insight in this paper via the score-matching perspective is new in the interpretability domain. The claim that input-gradients can be easily manipulated is not new, but the general insight in this work is new and important. Overall, this paper opens up several questions about what input-gradients really convey. 


Taylor Trace Estimator\
I am confused about the derivation of this estimator and I am probably missing something, so can you walk me through this? Let's say the Taylor series expansion around a point x is: 

$f(y) = f(x) + \nabla f(x)^\top(y-x) + \frac{1}{2}(y-x)^\top\nabla^2f(x)(y-x) + \mathcal{O}(\left\Vert y-x\right\Vert^3)$
now if solve for $(y-x)^\top\nabla^2f(x)(y-x) $, we get: 
$\frac{1}{2}(y-x)^\top\nabla^2f(x)(y-x) = f(y) - f(x) -  \nabla f(x)^\top(y-x) -  \mathcal{O}(\left\Vert y-x\right\Vert^3)$
$v= y-x$ in your notation, and it is a zero mean gaussian, so we get: $\mathbb{E}[\nabla f(x)^\top v] = 0$, which leads to the approximation that you get. However, where did the $\frac{1}{\sigma^2}$ outside the expectation come from in the final form?

The point on Adversarial Training\
In section 4, this paper notes that recent work has shown that when a model is trained with explanation penalization, it results in more 'interpretable' gradients. This connection was made more formal in recent work (https://arxiv.org/abs/1810.06583.pdf)  that notes that training models while penalizing their integrated gradients explanations is equivalent (Thm 5.1 in that paper, for the right loss function and some other assumptions) to $\ell-\infty$ adversarial training. 

Other Feedback and Questions
-  The current section 4 is really a discussion/implications section. I suggest the authors call it that. It should also likely come after section 5 since that is where the experimental results are. From reading section 4, am I right to conclude that the paper is also suggesting that activation maximization, pixel perturbation, and the results of adversarial training say more about the implicit generative model than discriminative information for a DNN? That is, I should also not take the results of activation maximization as explaining to me what a neuron that learned?

- Do these results extend to methods that post-process or use the input-gradients as a primitive? For example, smoothgrad adds noise to the input and takes an average of the corresponding input gradients. Integrated gradients can be seen as as sum of interpolated input-gradients along an all-zeroes input to an input of interest. Should I also take it from these results that Smoothgrad and integrated gradients don't indicate discriminative behavior as well?

- For example, consider grad-cam, which looks at the output of a convolutional layer in computing a sensitivity map as opposed to the logits, does your analysis apply to that case too? I think it probably doesn't, unless one can also view the output of convolutional filters are implicit density models as well. 

- Does the analysis in section 2 apply to the probability output from softmax? I.e., can the 'probability-gradients' also be arbitrarily manipulated? 

Overall, this work raises several important questions like why trained models have implicit density models that are aligned with the inputs in the first place. This question seems key to tying up the remaining loose ends in this work. This said, this paper is still a thought-provoking one and a useful one for the literature on DNN interpretability. 
review_title: Insightful paper with far reaching implications for gradient-based visualization methods

final_score: rating: 7: Good paper, accept
review_content: In this work the authors explore the link between the explanatory power of input-gradients and the alignment between the "implicit density model" of the softmax-based deep model and the "ground truth" class-conditional data distribution. The authors propose using score-matching method to create models with varying degrees of alignment. The paper is full of interesting insights and ideas, such as soft-max shift invariance property and trivial input-gradient perturbation, connections between score-matching and adversarial training, and others. However, in the end, the paper's impact on how ML engineers will use interpretability tools is in my opinion limited. The authors successfully introduce some interesting heuristics to make the training of score-matching models more scalable and stable. However even with those heuristics ,gradient-norm regularized models are comparable if not superior in the 3 evaluations presented by the authors: GAN-test score, pixel perturbation results, and perceptual alignment of input-gradients. The authors provide enough evidence to validate the main hypothesis of the paper. There are no inconsistencies or errors that I can see in the paper to the best of my knowledge. The paper is clearly written and well structured. To improve the paper, the authors could add some comments expanding on the practical impact that this results will have on the work of ML engineers who use input-gradients as a tool to improve model accuracy.      
review_title: Paper provides interesting perspective on input-gradients and their explanatory power

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper investigates the utility of interpreting deep classification models using the gradients of the output logits w.r.t the inputs, a common practice that is also potentially misleading.

The hypothesis stated for the paper is "input-gradients are highly structured because this implicit density model is aligned with the ‘ground truth’ class-conditional data distribution?"

For the observation in section 2 (also maybe number blocks like that similar to lemmas and hypotheses to make it easier to refer to them) if g is 0 then this is trivially true and not saying anything. I think I know what you want to say but this formalism is not adding any clarity. I think adding some constraints to g or simply calling it a variable which takes on specific values. It is not clear why it needs to be conditioned on x either.

3.3 Stabilized score matching: It seems multiple published methods did not help you prevent collapse of the Hessian-trace, but your heuristic did. Is this a common trick or a novel contribution? It would be nice to indicate it one way or another, if it solves a real problem that previous score-matching approaches fail to solve. Furthermore, it would be important to know the sensitivity of your approach to the choice of this hyperparameter.

The importance of section 4.1 is not clear to me… it appears that the authors believe activity maximization is a biased explainability measure and therefore should not be used if one accepts their framework. Their intention for this paragraph should be more explicit.

Section 4.2 draws a tight parallel between the pixel perturbation test and their density ratio test, demonstrating that the pixel perturbation test captures the sensitivity of the implicit density model and not the discriminative model. They therefore suggest this test always be done while tracking changes in classification output, which is a nice takeaway.

In section 4.3, the authors draw a parallel between their score-matching objective and adversarial training, although they state that “score matching imposes a stronger constraint”. I am not an expert on these topics, but I think these kinds of more speculative observations should be moved to the discussion in general.

In section 5, the authors introduce their experimental setup. They used a baseline model ResNet-18 model with 78.01% accuracy, and compared it with their regularized model that only achieved 72.20% accuracy (5.8% drop). The authors weight the entire regularization term with a single lambda =1e-3. This raises a few important questions.
First, the strength of the stability regularizer and the score matching terms should likely be decoupled to achieve maximum classification performance which is crucial in practice. The difficulty of tuning these hyperparamaters are also extremely important for us to understand the utility of the Author’s proposed approach. It would be good for us to see the results of a proper hyperparamater search over the weighting of the score-matching and stability regularization term independently. Ideally, the performance drop observed can be reduced or eliminated. 

Second, presumably, if the score matching loss itself hurts classification performance, then intuitively the intuitions built early in the paper cannot be correct: if the aligned density functions p_theta(X) and p_data(X) do can not arise from logits that produce the optimal classifier, then the saliency maps produced by this method cannot be used to diagnose model correctness (as the practitioners who utilize saliency maps would hope). Since the score matching loss is only accurate up to some constant, perhaps this is the source of the issue, but we cannot conclude one way or the other from the data provided.

Furthermore, few details are given for the training setup: how long was each model trained for, was early stopping employed, were multiple seeds evaluated, are we convinced that all models converged? These are important for doing the relevant comparisons, and many of the later results are hard to interpret in the absence of hyperparamater tuning or these experimental details. 

Anti-score matching is a great baseline experiment, but the use of the threshold seems arbitrary. This is doublyThis doubly true because the lambda of the regularization is smaller (1e-4). Why this discrepancy? It makes it harder for us to compare the results. How does this model perform as that threshold is varied? If one does a hyperparamater search with a fixed threshold, one would find the best performing anti-score matched model, which would potentially be easier to interpret. Crucially, this approach seems to actually outperform the score-matching model and underperform the baseline, implying that either the lower lambda or the anti-score constraint improved the performance of the classifier, but we cannot know which.

A similar comment can be made about the gradient-norm model RE: hyperparameter searches.

The Density Ratio experiments are excellent, but the y axis is hard to read.

For the Sample Quality experiments: why would the gradient norm model outperform score matching? I am not convinced by the speculation given in the paper. It might be the case that the score-matched models were not converged, highlighting the importance of improving or simply explaining the experimental setup as I mentioned earlier.

Finally, the Gradient Visualization experiments are very unclear. Intuitively, it might be the case that a small portion of the image is enough to explain the class of the data, which is what saliency maps are typically used for. The gradient norm approach and your score matching approach appear to perform almost identically, and it isn’t clear how much better they perform in a practical sense. It would be nice to have a more convincing demonstration of examples where the model obviously classifieds an image using the appropriate information present in using your method but not the baseline. As it stands, your results appear to be largely due to the fact that the gradients are smoother when using score matching or gradient-norm regularization.

In summary, I really like the approach and theory presented in the paper, and tackles an important issue with broad relevance to the field, but the experimental results as they stand are not sufficient to convince me that this approach works in practice.

Typo Section 4: "show how these can interpreted from"

review_title: Needs to be reworked to have a clear message and support evidence

Meta-review: This paper studies why input gradients can give meaningful feature attributions even though they can be changed arbitrarily without affecting the prediction. The claim in this paper is that "the learned logits in fact represent class conditional probabilities and hence input gradients given meaningful feature attributions". The main concern is that this claim is verified very indirectly, by adding a regularization term that promotes logits learning class conditional probabilities and observing that input gradient quality also improves. Nevertheless, there are interesting insights in the paper and the questions it asks are very timely and important, and overall, it could have a significant impact on further research in this area.
Decision: Accept (Oral)