Rethinking the Role of Gradient-based Attribution Methods for Model Interpretability
Authors: Suraj Srinivas
Francois Fleuret
Venue: ICLR 2021 Oral (2021)
Keywords: Interpretability
saliency maps
score-matching

Abstract: Current methods for the interpretability of discriminative deep neural networks commonly rely on the model's input-gradients, i.e., the gradients of the output logits w.r.t. the inputs. The common assumption is that these input-gradients contain information regarding $p_{\theta} ( y\mid \mathbf{x} )$, the model's discriminative capabilities, thus justifying their use for interpretability. However, in this work, we show that these input-gradients can be arbitrarily manipulated as a consequence of the shift-invariance of softmax without changing the discriminative function. This leaves an open question: given that input-gradients can be arbitrary, why are they highly structured and explanatory in standard models?

In this work, we re-interpret the logits of standard softmax-based classifiers as unnormalized log-densities of the data distribution and show that input-gradients can be viewed as gradients of a class-conditional generative model $p_{\theta}(\mathbf{x} \mid y)$ implicit in the discriminative model. This leads us to hypothesize that the highly structured and explanatory nature of input-gradients may be due to the alignment of this class-conditional model $p_{\theta}(\mathbf{x} \mid y)$ with that of the ground truth data distribution $p_{\text{data}} (\mathbf{x} \mid y)$. We test this hypothesis by studying the effect of density alignment on gradient explanations. To achieve this density alignment, we use an algorithm called score-matching, and propose novel approximations to this algorithm to enable training large-scale models.

Our experiments show that improving the alignment of the implicit density model with the data distribution enhances gradient structure and explanatory power while reducing this alignment has the opposite effect. This also leads us to conjecture that unintended density alignment in standard neural network training may explain the highly structured nature of input-gradients observed in practice. Overall, our finding that input-gradients capture information regarding an implicit generative model implies that we need to re-think their use for interpreting discriminative models.

Published as a conference paper at ICLR 2021
RETHINKING THE ROLE OF GRADIENT-BASED ATTRI- BUTION METHODS FOR MODEL INTERPRETABILITY
Suraj Srinivas
François Fleuret University of Geneva francois is. fleuret@unige.ch
Idiap Research Institute & EPFL suraj. a grinivaa vas@ didiap..c.
ABSTRACT
Current methods for the interpretability of discriminative deep neural networks commonly rely on the model's input-gradients, i.e., the gradients of the output logits w.r.t the inputs. The common assumption is that these input- gradients contain information regarding Pa(y x), the model's discriminative capabilities, thus justifying their use for interpretability. However, in this work show that these gouttgraddents can be arbitrarily manipulated as consequence of the shift- invariance of softmax without changing the discriminative function. This leaves an open question: if input-gradien can be arbitrary, why are they highly structured and explanatory standard models?
We investigate this by re-interpreting the logits of standard softmax-based classifiers as unnormalized log-densities of the data distribution and show that input-gradient can be viewed gradients of class-conditiones density model p y) implicit within the discriminative model. This leads us to hypothesize that the highly structured and explanatory nature of input-gradients may be due the alignment of this dtass-conditionas model (x(x y) with that of the ground truth data distribution Paau (x v). We this hypothesis by studying the effect of density alignment gradient explanations. To achieve this density alignment, we use an algorithm called matchnna, and propose novel approximations to this algorithm enable training large-scale models.
Our experiments show that improving the alignment of the implicit density model with the data distribution enhances gradient structure and explanatory power while reducing this alignment has the opposite effect. This also leads us to conjecture that unintended density alignment in standard neural network training may explain the highly structured nature of input- input-gradients observed in practice. Overall, our finding that input-gres gradients capture information regarding an implicit generative model implies that we need to re-think their use for interpreting discriminative models.
INTRODUCTION
Input-gradients, or gradients of outputs w.r.t. inputs, are commonly used for the interpretation of deep neural networks (Simonyan al. 2013). For image classification tasks, an input pixel with larger graddegrd magnitude is attributed a higher 'importance' value, and the resulting maps are observed agree with human intuition regarding which input pixels are important for the task at hand (Adebayo et al. 2018). Quantitative studies (Samek et al. 2016 Shrikumar et al.) 2017) also show that these importance estimates are meaningful predicting model response to larger structured perturbations. These results suggest that input-gradients do indeed capture relevant information regarding the underlying model. However in this work, we show that input-gradients be arbitrarily manipulated using the shift-invan hnftt-invariance of softmax without changing the underlying discriminative model, which calls into question the reliability of gpput-gr based attribution methods for interpreting arbitrary blx models.
Given that input- gradientsss can be arbitrarily structured, the reason for their highly structured and explanatory nature in standard pre-trained models is puzzling. Why imput-gradients relatively well-

---

Published as a conference paper at ICLR 2021
behaved when they can just easily be arbitrarily structured, without affecting discriminative model performance? What factors influence imput-gradient structure in standard deep neural networks?
To answer these, we consider the connections made between boftmax-based discriminative classifiers and generative models (Bridle 1990, Grathwohl et al., 2020), made by viewing the logits of standard classifiers as un-normalized log-densities. This connection reveals an alternate interpretation of input-gradients, as representing the log-gradiens of a class-conditional density model which is implicit within standard softmax-based deep models, which we shall call the implicit density model. This connection compels us to consider the following hypothesis: perhaps input-gradients are highly structured because this implicit density model is aligned with the "ground truth' class-conditions data distribution? The core of this paper is dedicated to testing the validity of this hypothesis, whether or not -rut-ggad do become more structured and explanatory if this alignment increases and vice versa.
For the purpose of validating this hypothesis, we require mechanisms to increase or decrease the alignment between the implicit density model and the data distribution. To this end, we consider generative modelling approach called score-matching. which reduces the density modelling problem to that of local geometric regularization. Hence by using score-matching, we are able to view commonly used geometric regularizers in deep learning as density modelling methods. In practice, the score-matching objective is known for being computationally expensive and unstable to train (Song & Ermon 2019 Kingma & LeCun 2010). To this end, we also introduce approximations and regularizers which allow to use score- score-matching on practical large-scale discriminative models.
This work is broadly connected to the literature around unreliability of saliency methods. While most such works consider how the explanations for nearly identical images can be arbitrarily different (Dombrowski et al.) 2019; Subramanya et al. 2019 Zhang et al. 20200; Ghorbani er al.) 2019), our work considers how one may change the model itself to yield arbitrary explanations without affecting discriminative performance. This is similar to Heo et al. (2019) who show this experimentally, whereas we provide an analytical reason for why this happens relating to the shift-invariance of softmax.
The rest of the paper is organized as follows. We show in 2 that is trivial to manipulate input-gradient of standard classifiers using the shift-invariance of softmax without affecting the discriminative model. In $ we state our main hypothesis and describe the details of score-matching present a tractable approximation for the same that eliminates the need for expensive Hessian computations. $ revisstss other interpretability tools from a density modelling perspective. Finally, sp experimental evidence for the validity of the hypothesis that improved alignment between the implicit density model and the data distribution can improve the structure and explanatory nature of gradings
INPUT-GRADIENTS ARE NOT UNIQUE
In this section, we show that is trivial to manipulate input-gradients of discriminative deep networks, using the well-known thift-invariance property of softmax. Here we shall make distinction between are gradients
grggi-ggaseents and loss- poss-grudients. While of the pre-softmax output of a given class w.r.t. the input, loss-gradients are the gradients of the loss w.r.t. the input. In both cases, we only consider outputs of a single class, usually the target class.
two types of input-gradients:
Let R2D be data point, which is the input for neural network model f RD + RC intended for classification, which produces pre-softmax logits for C' classes. The cross-entropy loss function for some class 1 < i C, i E N corresponding to input x is given by (() E R,, which is shortened to (x) for convenience. Note that here the loss function subsumes the softmax function as well. The logit-gradients are given by vx E RD for class i, while loss-gradients (x)( E D... Let the softmax function be p(y = ilx) = (x))/ exp(f, (x)), which we denote as Pt for simplicity. Here, we make the observation that upon adding the same scalar function y to all logits, the logit-gradients can arbitrarily change but the loss values do not.
Observation. Assume an arbitrary function g RD. - R. Consider another neural network functiion given by 1a()) fa(-) g(), for 0 S C. for which we obtain VxJ(() VxJ((.) Vx9(-).
2

---

Published as a conference paper at ICLR 2021
For this, the corresponding loss values and loss coss-gradients unchanged, i.e.; (((-) Le(-) and V (xh(()= Vx((()) consequence of the shift-invariance of softmax.
This explains how the structure of Dogit-gradiens can be arbitrarily changed: one simply needs to add an arbitrary function g to all logits. This implies that individual logit-gradients Vxfi(x) and logits f((x) are meaningless on their own, and their structure may be uninformative regarding the underlying discriminative model. Despite this, a large fraction of work in interpretable deep learning
(Simonyan et al., 2013; (elvaraju et al., 2017; Smmikov et al.) 2017: Fong et al.) 2019 [Srinivas & Fleuret (219)) uses individual logits and segittgradiets for saliency map computation. We also
provide a similar illustration in the supplementary material for the case of loss- loss- gradients, where we
show that it is possible for loss-gradients to diverge significantly even when the loss values themselves
do not.
These simple observations leave open question: why imput-gradics highly structured and explanatory when they can just as easily be arbitrarily structured, without affecting discriminative model performance? Further, if input-gradients do not depend strongly the underlying discrimina- tive function, what aspect of the model do they depend on instead? In the section that follows, we shall consider generative modelling view of discriminative neural networks that offers insight into the information encoded by logit-gradients
3 IMPLICIT DENSITY MODELS WITHIN DISCRIMINATIVE CLASSIFIERS
Let us consider the following link between generative models and the softmax function. We first define the following joint density on the logits f, of classifiers: px(x, y = i) = 2(0) where (()) is the partition function. We shall henceforth suppress the dependence of f on @ for brevity. Upon using Bayes' rule to obtain (y x), we observe that recover the standard softmax function. Thus the logits of discriminative classifiers can alternately be viewed as un-normalized log-densities of the joint distribution. Assuming equiprobable classes, we have (o y = i) 2(0//C
exp(f((x)) which is the
quantity of interest for us. Thus while the logits represent un-normalized log-densities, logit-gradients represent the score function, i.e.; Vz log (x y i) = V fxx).... which avoids dependence on the partition function (() as is independent of x.
This viewpoint naturally leads to the following hypothesis, that perhaps the reason for the highly structured and explanatory nature of input- gradients that the implicit density model po(x y) is close to that of the ground truth class-condins conditionai data distribution Pama(x (x y)? We propose to test this hypothesis explicitly using score-matching as a density modelling tool.
Hypothesis. (Informal) Improved alignment of the implicit density model to the ground truth class- conditional density model improves inpul-gradient interpretability via both qualitative and quantita- tive measures, whereas deteriorating this alignment has the opposite effect.
3.1 SCORE-MATCHING
mcore-matching (Hy (Hyvärinen, 2005) is a generative modelling objective that focusses solely on the derivatives of the log density instead of the density itself, and thus does require access the partition function (0). Specifically, for our case we have Vx log (x i) f fx(), which are the logit-gradee gradients.
Given i.i.d. samples X' = {a, E " from latent data distribution Pdota (x). the objective of generative modelling is to recover this latent distribution using only samples X. This often done by training parameterized distribution (x) to align with the latent data distribution (x)ta( The score-matching objective instead aligns the gradients of log densities, as given below.
Paiato ((0) Epaceo I (o()) - Vx Pagaaa (x)13
= E Eppacs trace V2 log + 1 プ V log (ogg)(())13 const
(2)
3

---

Published as a conference paper at ICLR 2021
The above relationship is proved (Hyvärinen 2005) using integration by parts. This is consistent objective, f.e, J(Ø) =0 = Pdata = Pe. This approch is appealing also because this reduces the problem of generative modelling that of regularization of the local geometry of functions, i.e.; the resulting terms only depend on the point-wise gradients and Hessian-trace.
3.2 EFFICIENT ESTIMATION OF HESSIAN-TRACE
In general, equation intractable for sith-immnnsioall data due the Hessian trace term. To address this, we can use the Hutchinson's 's trace estimator (Hutchinson, 1990) to efficiently compute an estimate of the trace by using random projections, which is given by: trace VV log po (x)) Ep-.N(0,11 n² VX log (x) u. This estimator has been previously applied to score-matching (Song (ct al., (019)) and can be computed efficiently using Pearlmutter's s trick (Pearlmutter| [1994). However, this trick still requires two backward passes for single monte-carlo sample, which is compu- tationally expensive. To further improve computational efficiency, we introduce the following approximation to Hutchinson's estimator using a Taylor series expansion, which applies to small values of a E R.
2
Ee o V (x))
2
0² (v (logPo(x + v) - log - Va log (x)
2 a² Ev (logpo(x + v) - logoo
Note that equation involves difference of log probabilitiess which is independent of the partition function. For our case, log px(x + vly = i) - log (xly = f:(x + v) f(().. We have thus considerably simplified and speeded-up the computation of the Hessian trace term, which now can be approximated with no backward passes, but using only single additional forward pass. We present details regarding the variance of this estimator the supplementary material. concurrent approach (Pang et al., 2020) also presents a similar algorithm, however it is applied primarily to Noise Contrastive Score Networks (Song Ermon 2019) and Denoising Score Matching (Vincent 2011, whereas we apply it to vanilla score- ccore-matching on discriminative models.
3.3
STABILIZED SCORE-MATCHING
In practice, a naive application of score-matching objective is unstable, causing the Hessian-trace to collapse to negative infinity. This occurs because the finite-sample variant of equation causes the model to 'overfit' to a centurr-of-diraccee density, which places a dirac-delta distribution at every data point. Gradients of such a distribution are undefined, causing training collapse. To overcome this, regularized score-matching (Kingma & LeCun 2010) and noise conditional score networks (Song & Ermon 2019) propose add noise to inputs for score-matching However, this did not help for our case. Instead, we use heuristic where we add small penalty term proportional the square of the Hessian-trace This discourages the Hessian-tras becoming too large, and thus stabilizes training.
make the problem well-defined.
4 IMPLICATIONS OF THE DENSITY MODELLING VIEWPOINT
In the previous section we related input-gradient to the implicit density model, thus linking gradient interpretability to density modelling through our hypothesis. In this section, we consider two other interpretability tools: activity maximization and the pixel perturbation test, and show how these can interpreted from density modelling perspective. These perspectives also enable us draw parallels between score-matching and adversarial training.
4.1 ACTIVITY MAXIMIZATION AS SAMPLING FROM THE IMPLICIT DENSITY MODEL
The camonical method to obtain samples from score- generative models is via Langevin sampling (Welling & Teh 2011 Song & Ermon 2019). which involves performing gradient ascent on the density model with noise added to the gradients. Without this added noise, the algorithm recovers the modes of the density model.
4

---

Published as a conference paper at ICLR 2021
We observe that activity maximization algorithms used for neural network visualizations are remark- ably similar to this scheme. For instance, Simonyya et al. (2013) recover inputs which maximize the logits of neural networks, thus exactly recovering the modes of the implicit density model. Similarly. deep-dream-liks methods (Mahendran & Vedaldi 2016; Nguyen et al., 2016 Mordvintsev et al. 2015) extend this by using "image priors" to ensure that the resulting samples are closer to the distribution of natural images, and by adding structured noise to the gradients in the form of jitter, to obtain more visually pleasing samples. From the density modelling perspective, we can alternately view these visualization techniques as biased sampling methods for score- score-based density models trained on natural images. However, given the fact that they draw samples from the implicit density model, their utility in interpreting discriminative models may be limited.
4.2 PIXEL PERTURBATION AS A DENSITY RATIO TEST
A popular test for saliency map evaluation is based on pixel perturbation (Samek et al. 2016). This involves first selecting the least-relevant (or most-relevant) pixels according to a saliency map representation, 'deleting' those pixels and measuring the resulting change in output value. Here, deleting a pixel usually involves replacing the pixel with a non-informative value such as a random or a fixed constant value. A good saliency method identifies those pixels as less relevant whose deletion does not cause a large change in output value.
We observe that this change in outputs criterion is identical to the density ratio, i.e., log (pe(x + u = (xly = i))
fi(x f((x+ + u) fi(x). Thus when logits are used for evalu- ating the change in outputs (Samek et al., [016)] Ancona et al., (018)) the pixel perturbation test exactly measures the density ratio between the perturbed image and the original image. Thus if a perturbed image has a similar density to that of the original image under the implicit density model, then the saliency method that generated these perturbations is considered to be explanatory. Similarly, Fong et al. (2019) optimize over this criterion to identify pixels whose removal causes minimal change in logit activity, thus obtaining perturbed images with a high implicit density value similar to that of activity maximization. Overall, this test captures sensitivity of the implicit density model, and not the underlying discriminative model which we wish to interpret. We thus recommend that the pixel perturbation test always be used in conjunction with either the change in output probabilities, or the change in the accuracy of classification, rather than the change in logits.
4.3 CONNECTING SCORE-MATCHING TO ADVERSARIAL TRAINING
Recent works in adversarial machine learning (Etmann et al. 2019 Engstrom et al. 2019 Santurkar al. 2019 Kaur al. 2019 Ross & Doshi Velez, 2017 7) have observed that saliency map structure and samples from activation maximization are more perceptually aligned for adversarially trained models than for standard models. However it is unclear from these works why this occurs. Separate from this line of work, Chalasani et al. (2018) 8) also connect regularization of a variant of integrated gradients with adversarial training, suggesting close interplay between the two.
We notice that these properties are shared with score-matched models, or models trained such that the implicit density model aligned with the ground truth. Further, we note that both score-matching and adversarial training are often based on local geometric regularization, usually involving regularization of the gradient-norm (Ross Doshi Velez 2017 Jakubovitz Giryes 2018), and training both the discriminative model and the implicit density model (Grathwohl et al.) (0200 has been shown to improve adversarial robustness. From these results, we can conjecture that training the implicit density model via score-matching may have similar outcomes as adversarial training. We leave the verification and proof of this conjecture to future work.
EXPERIMENTS
In this section, we present experimental results to show the efficacy of score-matching and the validation of the hypothesis that density alignment influences the gradient explanation quality. For experiments, we shall consider the CIFAR 100 dataset. We present experiments with CIFAR1O in the supplementary section. Unless stated otherwise, the network structure we use shall be 18-layer ResNet that achieves 78.01 % accuracy on CIFAR100, and the optimizer used shall be SGD with momentum. All models use the softplus non-liny with 3 10, which is necessary ensure that
5

---

Published as a conference paper at ICLR 2021
the Hessian is non-zero for score- score- matching. Before proceeding with our experiments, we shall briefly introduce the score-men matching variants we shall be using for comparisons.
Score-Matching We propose
the score-matching objective regularizer in neural network training to increase the alignment of the implicit density model to the ground truth, as shown in equation 4 with the stability regularizer discussed in $3.3] For this,
regularization constant le -e 3. 3. This model achieves 72.20% accuracy on the test set, which is a drop of about 5.8% compared to the original model. In the supplementary material, perform thorough pyper-parameter sweep and show that it is possible to obtain better performing models.
h(x) - Eu (f((x + f((x))
Hessian-trace
gradient-norm
Creg ()(,))))) regularized loss
()(
M(x)
[x
122
cross-entrrpp
stability regularizer
score-matching
mnti-score-matching We would like to have a tool that can decrease the alignment between the implicit density model and the ground truth. To enable this, we propose to maximize the hessian- trace, in an objective we call serii--ooo--macciin on hessian-trace which ensures that its maximization stops after threshold reached. We use threshold of = 1000, and regularization constant X = le 4. This model achieves an accuracy of 74.87%.
For this, we shall use a the clamping function
Gradient-Norva regularization We propose to use gradient-norm regularized models as another baseline for comparison, using a regularization constant of le 3. This model achieves an accuracy of 76.60%.
5.1 EVALUATING THE EFFICACY OF SCORE- MATCHING AND ANTI-SCORE SOTI-SCORE-MMTTCHIG
Here we demonstrate that training with score-matching! anti-score-matching possible, and that such training improves deteriorates the quality of the implicit density models respectively as expected.
5.1.1 DENSITY RATIOS
One way to characterize the generative behaviour models to compute likelihoods on data points. However this is intractable for nigh-dimensional problems, especially for un-normalized models. We observe although that the densities p(x density ratios p(x + 7) y i)/p(x y = exp(fi(x 7) fx(x)) for random noise variable Thus, we propose to plot the graph of density ratios locally along random directions. These can be thought of as local cross-sed secsiosctions of the density sliced random directions. We plot these values for gaussian noise v for different standard deviations, which are averaged across points in the entire dataset.
i) themselves are intractable, we can easily compute
In Figure | we plot the density ratios upon training the CIFAR100 dataset. We observe that the baseline model assigns higher density values to noisy inputs than real inputs. With anti-score- matching, we observe that the density profile grows still steeper, assigning higher densities inputs with smaller noise. Gradient-norm regularized models and score-matched models improve on this behaviour, and are robust larger amounts of noise added. Thus able to obtain penalty terms that can both improve and deteriorate the density modelling behaviour within discriminative models.
5.1.2 SAMPLE QUALITY
We are interested in recovering modes of our density models while having access to only the gradients of the log density. For this purpose, we apply gradient ascent on the log probability log y = i) fx(),, similar activity maximization. Our results are shown in Figure 2 We observe that samples from the score-matched and gradient-norm norm regularized models are significantly less noisy than other models.

---

Published as a conference paper at ICLR 2021
Model
GAN-test (%)
Baseline ResNet
59.47
Anti-Score- Scoor-Matchhng
16.40
Gradient Norm-reet regularizzation
80.07
+ Score-Matchin
72.75
Table I: GAN-test scores (higher is better) of class-conditional samples generated from various ResNet-18 18 models (see 1.21. We observe that samples from gradient-norm regularized models and ccore-matchee models achieve much better accuracies than the baselines and umi-score-matcher models.
(a) Baseline ResNet
(b) With anti ecore-matching
(c) With Gradient- regularization
(d) With score-matching
Figure 2: Samples generated from various models by performing gradient ascent on random inputs (see $ 5.1.2). While none of the generated samples are realistic, samples obtained from score-matched and gradient-norm regularized models are smoother and less noisy.
We also propose to qualitatively measure the sample quality using the GAN-test approach (Shmelkov et al.) 2018). This test proposes to measure the discrimina- tive accuracy of generated samples via an indepen- dently trained discriminative model. In contrast with more popular metrics such as the inception-score, this captures sample quality rather than diversity, which what we are interested in. We show the results in table I which confirms the qualitative trend seen in samples above. Surprisingly, we find that gradient- norm regularized models perform better than score- matched models. This implies that such models are able implicitly perform density modelling without being explicitly trained to do so. We leave further investigation of this phenomenon to future work.
2.0
18
1.2
10
Norm regularization
Matching
0.6
Andi Score Merching
10
10
10
15a1
Standard deviation of
Figure 1: Plots of density ratios representing local density profiles across varying levels of noise added the input (lower is better). We observe that score-matched model is robust to larger range of noise values, while anti- score-matee motehin is very sensitive even to small amounts noise.
5.2 EVALUATING THE EFFECT OF DENSITY ALIGNMENT ON GRADIENT EXPLANATIONS
Here we shall evaluate the gradient explanations of various models. First, we shall look at quantitative results on discriminative variant of the pixel perturbation test. Second, we visualize the gradient maps to assess qualitative differences between them.
5.2.1 QUANTITATIVE RESULTS ON DISCRIMINATIVE PIXEL PERTURBATION
As noted in 4.2 is recommended to use the pixel perturbation test using accuracy changes, and we call this variant discriminative pixel perturbation. We select the least relevant pixels and replace them with the mean pixel value of the image, note down the accuracy of the model on the resulting samples. We note that this test is only used so far to compare different saliency methods for the same underlying model. However, we here seek to compare saliency methods across models. For this we consider experiments. First, perform the pixel perturbation experiment with each of the four
7

---

Published as a conference paper at ICLR 2021
Fredboo of iraggs Comest Chealied
Focdio of mrggcc Cemedt Glosifee
0.4
0.2
3.4-
Scare Pere--orrinn Art Sunt-Scoom--iie Natriing
Art--Sore---aaae Marchin
Masding
Masking Praction
(a) Models evaluated with their own gradients
(b) Baseline ResNet evaluated with gradients of different models
Figure 3: Discriminative pixel perturbation results (higher is better) on the CIFAR 100 dataset (see $.2.II. We see that mcore-matched and gradient-norm regularized models best explain model behaviour in both cases, while the anti-score- aati-score-matched model performs the worst. This agrees with the hypothesis (stated in 3) that alignment of implicit density models improves gradient explanations and vice versa.
trained models on their own poputggrddeets and plot the results in Figure 3al These results indicate that the input-gradients of score-matched and madieent-nom regularized models are better equipped to identify least relevant pixels this model. However, it is difficult to completely disentangle the robustness benefits of such score-matched models against improved identification of less relevant pixels through such a plot.
To this end, conduct second experiment in Figure 3b
where use input-gradients obtained from these four trained models to explain the same standard baseline ResNet model. This disentangles the robustness of different models as inputs the same model is perturbed in all cases. Here also we find that gradients from score-matched and gradient-nore norm regularized models explain behavior of standard baseline models better than the gradients of the baseline model itself. Together, these tests show that training with score-matching indeed produces input-gradient that quantitatively more explanatory than baseline models.
5.2.2 QUALITATIVE GRADIENT VISUALIZATIONS
We visualize the structure of logit- grgit-gradients different models in Figure 4 We observe that gradient- norm regularized model and score-matched model have highly perceptually aligned gradients, when compared to the baseline and anti-score-matched gradients, corroborating the quantitative results.
(a) Input Image (b) Baseline ResNet
(c) With Anti score-matching
(d) With Gradient-norm regularization
(c) With Score-matching
Figure 4: Visualization of input-gru gradients of different models. We observe that gradients of score- matched and gradient-none norm regularized models are more perceptually aligned than the others, with the gradients of the mnti-coore-matchee model being the noisiest. This qualitatively verifies the hypothesis stated in
8

---

Published as a conference paper at ICLR 2021
6 CONCLUSION
In this paper, we investigated the cause for the highly structured and explanatory nature of input- gradients in standard pre-trained models, and showed that alignment of the implicit density model with the ground truth data density is a possible cause. This density modelling interpretation enabled us view canonical approaches in interpretability such gradient-basee saliency methods, activity maximization and the pixel perturbation test through a density modelling perspective, showing that these capture information relating to the implicit density model, not the underlying discriminative model which we wish to interpret. This calls for a need to re-think the role of these tools in the interpretation of discriminative models. For practitioners, we believe is best to avoid usage of logit gradient-basse based tools, for interpretability. If unavoidable, it is recommended to use only gradient- -orm regularized or score-matched models, as grautegradeent of these models produce more reliable estimates of the gradient of the underlying distribution. As our experiments show, these may be useful tool even though they are not directly related to the discriminative model.
However, our work still does not answer the question of why pre-trained models may have their implicit density models aligned with ground truth in the first place. One possible reason could be the the presence of an implicit gradient norm regularizer in standard SGD, similar to that shown independently by Barrett& Dherin (2020). Another open question is to understand why gradient- norm regularized models are able to perform implicit density modelling as observed in experiments in 5.1.2. which lead to improved gradient explanations.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muclly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp. 9505-9515, 2018.
Marco Ancona, Enea Ceolini, Cengiz Oztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In 6th International Conference on Learning Representations (ICLR 2018). 2018.
Haim Avron and Sivan Toledo. Randomized algorithms for estimating the trace of an implicit
symmetric positive semi-definite matrix. Journal of the ACM (JACM), 58(2):1 334, 2011.
David GT Barrett and Benoit Dherin. Implicit gradient regularization.
arXiv preprint
arXiv:2009. 2020.
John S Bridle. Probabilistic interpretation of feedforward classification network outputs, with relationships to statistical pattern recognition. In Neurocomputing, pp. 227-236. Springer, 1990.
Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury, Somesh Jha, and Xi Wu. Concise explana- tions of neural networks using adversarial training. arXiv, pp. arXiv-1810, 2018.
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alccc Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 13589- 13600. Curran Associates, Inc., 2019. URL audd- 9511-explame pdff
can www---e--mpppptedddeee and
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Alek-
sander Madry. Adversarial robustness prior for learned representations. arXiv preprint arXiv:1906.0 00905. 2019.
Christian Etmann, Sebastian Lunz, Peter Maass, and Carola-Bibiane Schönlieb. On the connection between adversarial robustness and saliency map interpretability. arXiv preprint arXiv: 110550::005.0111 0441 2019.
Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal pertur- bations and smooth masks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2950-2958, 2019.

---

Published as a conference paper at ICLR 2021
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3681-3688, 2019.
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2020. URL https
//operrevi ew. net firetfforuma d= xette/ooorrrmmmmmooxxx
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversarial model manipulation. In Advances in Neural Information Processing Systems, pp. 2019.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in 1990.
and Computation, 19(2)4333-50)
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(Apr):695-709 2005.
Daniel Jakubovitz and Raja Giryes. Improving dnn robustness to adversarial attacks using jacobian regularization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 514-529, 2018.
Simran Kaur, Jeremy Cohen, and Zachary C Lipton. Are perceptually--ligned gradients a general property of robust classifiers? arXiv preprint 1910::1900.0 08640, 2019.
Durk P Kingma and Yann LeCun. Regularized estimation of image statistics by score matching, In Advances in neural information processing systems, pp. 1126-1134, 2010.
Aravindh Mahendran and Andrea Vedaldi. Visualizing deep convolutional neural networks using
natural pre-images. International Journal of Computer Vision, 120(3):233-255 255, 2016
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural
networks. 2015.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in neural information processing systems, pp. 3387-3395, 2016.
Tianyu Pang, Taufik Xu, Chongxuan Li, Yang Song, Stefano Ermon, and Jun Zhu. Efficient learning of generative models via finite-difference score matching. Advances in Neural Information Processing Systems, 33, 2020.
Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 6(1):147-160, 1994.
Andrew Slavin Ross and Finale Doshi-Veler. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. arXiv preprint 1eXiv:1711.0944 2017.
Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions on neural networks and learning systems, 28(11):2600-267) 2016.
Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with a single (robust) classifier. In Advances in Neural Information Processing Systems, pp. 1260-12711 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618-626 IEEE, 2017.
10

---

Published as a conference paper at ICLR 2021
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? In Proceedings of the European Conference on Computer Vision (ECCV). pp. 213-229, 2018.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning-Vood Volume 70, pp. 3145-3153. JMLR. org, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:13/2.60344 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: 1706v:706.0055 170600385, 2017.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11895-11907, 2019.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. arXiv preprint (TT5::9900000000 2019.
Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. In Advances in Neural Information Processing Systems, pp. 4126-4135 2019.
Akshayvarun Subramanya, Vipin Pillai, and Hamed Pirsiavash. Fooling network interpretation in image classification. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 23(7):1661-1674, 2011.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 681-688, 2011.
Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling Ji, Xiapu Luo, and Ting Wang. Interpretable deep learning under fire. In 29th {USENIX) Security Symposium (!USENIX) Security 20), 2020.

---

Published as a conference paper at ICLR 2021
Appendix
A FOOLING GRADIENTS IS SIMPLE
Observation. Assume arbitrary function g -R_ - R. Consider another neural network function given by fa(-) f((.) + g(-), for 0 i S C. for which we obtain vxf(()) Vxxf(-))+ Vxg(-). For this, the corresponding loss values and loss- gradients unchanged, i.e.; (((-) = (((-) and
xxl(()) V-4(-).
Proof. The following expressions relate the loss and neural network function outputs, for the case of cross-entropy loss and usage of the softmax function.
4(x) -ff((x) + log
Σ exp (xp(f(
(5)
Vxl(xx) -Vxf((x) + Σ PJ (x)
(6)
Upon replacing Sr with J +.+t.
the proof follows.
A.1 MANIPULATING DASS-GRADIENTS
Here, we show how we can also change loss- poss-gradients arbitrarily without significantly changing the loss values themselves. this case, the trick to add a high frequency low amplitude sine function to the loss.
Observation. Consider g(x) (mix) and (x) (x + for) for 2 R, and x RP. Then, it is easy see that 1x,(x) - 4(x) S and (x) V S mxxxD.
Thus two models with losses differing by some small can have gradients differing by m X x D. For m + oc and fixed 6, the gradients can diverge significantly. Thus, loss-gradients are also unreliable, two models with very similar loss landscapes and hence discriminative abilities, can have drastically different loss-gradients.
This simple illustration highlights the fact that gradients of high-dimensional black-box models are not well-behaved in general, and this depends on both the model smoothness and the high-dimensionality of the inputs. Further, loss values and coss-gradients for highly confident samples are close to zero. Thus any external noise added (due to stochastic training. for instance) can easily dominate the loss- loss-gradient terms even when smoothness conditions (small m) are enforced.
B SCORE-MATCHING APPROXIMATION
We consider the approximation derived for the estimator the Hessian trace, which is first derived from Hutchinson's 's trace estimator Hutchinson 1990). We replace log po()) terms used in the main text with fx) terms here for clarity. The Taylor series trick for approximating the Hessian-trace is given below.
1
E o' (')x
2
EenN(00.01) (f v) f(x) (x
02
12

---

Published as a conference paper at ICLR 2021
As expected, the approximation enror vanishes in the limit of small σ. Let us now consider the finite sample variants of this estimator, with N samples. We shall call this the Taylor Trace Estimator.
Taylor Trace Estimator (TTE)
2 No²
(()x++0,) - (x(x) v, ,2~,N(0,001 (8)
We shall henceforth suppress the dependence for brevity. For this estimator, we can compute its variance for quadratic functions f. where higher-order Taylor expansion terms are zero. We make the following observation.
Observation. For functions f. the variance of the Taylor Trace Estimator is greater than the variance of the Hutchinson estimator by an amount most equal 4o V
Proof.
N N
2
2
(T.....)
=
σ
Ep
N
N
E
2
(f(x - v) - -f(x))
1
Σ
2
σ²
N
N
i=1
1=1
2
0
( (x)
N
N
(f(x - 0) - -((x) 1 N
2
=
σ
E2
N Σ
v'v
-
E.
Thus we have decomposed the variance of the overall estimator into two terms: the first captures the variance of the Taylor approximation, and the second captures the variance of the Hutchinson estimator.
Considering only the first term, i.e.; the variance of the Taylor approximation, we have:
N
N Σ o (x))2(())))
N
En No
2
2
=
No
ENp
Σ Vx [x))
1=1
4
I Vx
Eullol1
46
[xf())]
The intermediate steps involve expanding the summation, noticing that pairwise terms cancel, and applying the Cauchy-Schwand inequality.
Thus we have a trade-off: large results in lower estimator variance but a large Taylor approximation еттоr, whereas the opposite is true for small a. However for functions with small gradient norm, both the estimator variance and Taylor approximation error small for small a. We note that when applied to score-matching! [Hyvärine]] (2005), the gradient norm of the function is also minimized. This implies that in practice, the gradient norm of the function is likely be low, thus resulting in small estimator variance even for small a. The variance of the Hutchinson estimator is given below for reference Hutchinson] (1990): Avron & Toledo (00TT)
Var(Hutchinson) = N
22 fx)))?
13

---

Published as a conference paper at ICLR 2021
C EVALUATING EFFECT OF SCORE-MATCHING ON GRADIENT EXPLANATIONS (ON CIFAR10)
We repeat the pixel perturbation experiments the CIFAR10 dataset and observe similar qualitative trends. In both cases, we observe that score-matched and gradient norm regularized models have more explanatory gradients, while anti-score-me motored model contains the least explanatory gradients. We also present visualization results of input-gradient of various models for reference.
Pracion Imagee Cornetty Clusaied
Facde Imuggo Contecte Claso
Sacelise Rerbie
Bawalin RecNet
With Gredktom regutarrratin
with Gae-Norm miguar cntion
wit Score Maiching
will Scare Marching
wilh Ant Score Maxching
with and Score Matciiti
10
10-1 Maxding
30 30
10-1
13
Madding Traction
Models evaluated with their gradients
(h) Baseline ResNet evaluated with gradients of different models
Figure 5: Discriminative pixel perturbation results (higher is better) on the CIFAR10 dataset (see $..2.I. We see that score-matched and gradient-norm regularized models best explain model behaviour in both cases, while the anti-score-matchel model performs the worst. This agrees with the hypothesis (stated in $ 3 that alignment of implicit density models improves gradient explanations and vice versa.
(a) Input Image
(b) Baseline ResNet
(c) score-matching
(d) With Gradient-norm regularization
(e) With Score-matching
of different models. We observe that gradients of score- matched and gradient-norm regularized models are more perceptually aligned than the others, with gradients of the anti-seore-matched model being the noisiest. This qualitatively verifies hypothesis stated in
Figure 6: Visualization of
D HYPER- HYPER-PARAMETER SWEEP ON SCORE-MATCHED TRAINING
We present results on hyper- parameee sweep on the À and μ parameters of score-matching, where we provide both test-set accuracy on CIFAR 100 and the corresponding GAN-test scores. We find upon performing
hyper-parameter sweep that À 1e and μ best, whereas in the main paper we present results for À le 3 and μ 1e 4. It is possible that changing the training schedule by increasing the number of epochs or learning rate may further improve these results, but we did not explore that here.
3 seems to perform the
14

---

Published as a conference paper at ICLR 2021
44/P 7
1
1c
1c 1 4
1c -
le - 2
48.68% 60%
64.57%
64.75% 47.46%%
9.08%
le 3
64.64%/56. 78%
71.37% 4..7%%
72.34% 77..9%%
72.34%
34.46% 34.46%/3.399 /3.3%
le 4
69.85% 1355%/44.30 41.30%
73.97%
75.65%/79.399
72.97% 7..77%/61.529
1e 5
73.29%
/.8.9% /68.94%
75.37% 911%6/85..999
76.40%
(6.40%/63.966% /6..96%
74.80%
11.00%/78.45 48%
le 5-6 6
75.43%/82.8119
75.90%/66.1199
76.77%
75.91% 75.91% 525.22%
Table Results of hyper-parameter sweep on and μ. The numbers presented are in the format (accuracy % / GAN-test %
15


Reviews:
[{"review_id": "xawDWIvKxJ7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\\\nThe key message of this paper is that input-gradients (gradient of the logit wrt to input) or loss-gradients are/might be unrelated to the discriminative capabilities of a DNN. The input-gradient is a key primitive in several interpretability and visualization methods. Until now, it has been taken as a given that these gradients reveal 'why' or what parts of the inputs the model is sensitive to. However, this paper questions this reasoning and says that if the input-gradients can be easily manipulated without changing the generalization ability of the model, then does the input-gradient really contain discriminative signals?  \n\nTo test their hypothesis, the paper re-interprets the input-gradient as class-conditional generative model using the score matching view. The paper then develops a 'regularizer' that is called a taylor trace estimator that requires less backward passes than the hutchinson estimator, which when added to the model objective can modulate how 'generative' the model is. With this regularizer, the papers tests the hypothesis that improving the implicit density model also improves input-gradient interpretability. The paper tests this through experiments on an image dataset and finds that this is the case.\n\nSignificance\\\nThis work has far reaching significance for the field of visual interpretability of DNNs. It suggests that reading into these input-gradients might be akin to reading tea leaves. The key insight in this work is simple and the demonstration is quite powerful in my opinion. The argument in this paper seems obvious in hindsight, but that is exactly why the paper is a significant one. I have several additional questions later in my review, but this work is important and suggests that insights based on input-gradients might be spurious. \n\nClarity\\\nOverall, the paper is relatively clear and easy to read. Several of the key experiments are well-justified.\n\n\nOriginality\\\nThe insight in this paper via the score-matching perspective is new in the interpretability domain. The claim that input-gradients can be easily manipulated is not new, but the general insight in this work is new and important. Overall, this paper opens up several questions about what input-gradients really convey. \n\n\nTaylor Trace Estimator\\\nI am confused about the derivation of this estimator and I am probably missing something, so can you walk me through this? Let's say the Taylor series expansion around a point x is: \n\n$f(y) = f(x) + \\nabla f(x)^\\top(y-x) + \\frac{1}{2}(y-x)^\\top\\nabla^2f(x)(y-x) + \\mathcal{O}(\\left\\Vert y-x\\right\\Vert^3)$\nnow if solve for $(y-x)^\\top\\nabla^2f(x)(y-x) $, we get: \n$\\frac{1}{2}(y-x)^\\top\\nabla^2f(x)(y-x) = f(y) - f(x) -  \\nabla f(x)^\\top(y-x) -  \\mathcal{O}(\\left\\Vert y-x\\right\\Vert^3)$\n$v= y-x$ in your notation, and it is a zero mean gaussian, so we get: $\\mathbb{E}[\\nabla f(x)^\\top v] = 0$, which leads to the approximation that you get. However, where did the $\\frac{1}{\\sigma^2}$ outside the expectation come from in the final form?\n\nThe point on Adversarial Training\\\nIn section 4, this paper notes that recent work has shown that when a model is trained with explanation penalization, it results in more 'interpretable' gradients. This connection was made more formal in recent work (https://arxiv.org/abs/1810.06583.pdf)  that notes that training models while penalizing their integrated gradients explanations is equivalent (Thm 5.1 in that paper, for the right loss function and some other assumptions) to $\\ell-\\infty$ adversarial training. \n\nOther Feedback and Questions\n-  The current section 4 is really a discussion/implications section. I suggest the authors call it that. It should also likely come after section 5 since that is where the experimental results are. From reading section 4, am I right to conclude that the paper is also suggesting that activation maximization, pixel perturbation, and the results of adversarial training say more about the implicit generative model than discriminative information for a DNN? That is, I should also not take the results of activation maximization as explaining to me what a neuron that learned?\n\n- Do these results extend to methods that post-process or use the input-gradients as a primitive? For example, smoothgrad adds noise to the input and takes an average of the corresponding input gradients. Integrated gradients can be seen as as sum of interpolated input-gradients along an all-zeroes input to an input of interest. Should I also take it from these results that Smoothgrad and integrated gradients don't indicate discriminative behavior as well?\n\n- For example, consider grad-cam, which looks at the output of a convolutional layer in computing a sensitivity map as opposed to the logits, does your analysis apply to that case too? I think it probably doesn't, unless one can also view the output of convolutional filters are implicit density models as well. \n\n- Does the analysis in section 2 apply to the probability output from softmax? I.e., can the 'probability-gradients' also be arbitrarily manipulated? \n\nOverall, this work raises several important questions like why trained models have implicit density models that are aligned with the inputs in the first place. This question seems key to tying up the remaining loose ends in this work. This said, this paper is still a thought-provoking one and a useful one for the literature on DNN interpretability. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "kuQNscVqnM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work the authors explore the link between the explanatory power of input-gradients and the alignment between the \"implicit density model\" of the softmax-based deep model and the \"ground truth\" class-conditional data distribution. The authors propose using score-matching method to create models with varying degrees of alignment. The paper is full of interesting insights and ideas, such as soft-max shift invariance property and trivial input-gradient perturbation, connections between score-matching and adversarial training, and others. However, in the end, the paper's impact on how ML engineers will use interpretability tools is in my opinion limited. The authors successfully introduce some interesting heuristics to make the training of score-matching models more scalable and stable. However even with those heuristics ,gradient-norm regularized models are comparable if not superior in the 3 evaluations presented by the authors: GAN-test score, pixel perturbation results, and perceptual alignment of input-gradients. The authors provide enough evidence to validate the main hypothesis of the paper. There are no inconsistencies or errors that I can see in the paper to the best of my knowledge. The paper is clearly written and well structured. To improve the paper, the authors could add some comments expanding on the practical impact that this results will have on the work of ML engineers who use input-gradients as a tool to improve model accuracy.      ", "extra_scores": {}, "extra_text": {}}, {"review_id": "YzK1EGqm6t", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper investigates the utility of interpreting deep classification models using the gradients of the output logits w.r.t the inputs, a common practice that is also potentially misleading.\n\nThe hypothesis stated for the paper is \"input-gradients are highly structured because this implicit density model is aligned with the ‘ground truth’ class-conditional data distribution?\"\n\nFor the observation in section 2 (also maybe number blocks like that similar to lemmas and hypotheses to make it easier to refer to them) if g is 0 then this is trivially true and not saying anything. I think I know what you want to say but this formalism is not adding any clarity. I think adding some constraints to g or simply calling it a variable which takes on specific values. It is not clear why it needs to be conditioned on x either.\n\n3.3 Stabilized score matching: It seems multiple published methods did not help you prevent collapse of the Hessian-trace, but your heuristic did. Is this a common trick or a novel contribution? It would be nice to indicate it one way or another, if it solves a real problem that previous score-matching approaches fail to solve. Furthermore, it would be important to know the sensitivity of your approach to the choice of this hyperparameter.\n\nThe importance of section 4.1 is not clear to me… it appears that the authors believe activity maximization is a biased explainability measure and therefore should not be used if one accepts their framework. Their intention for this paragraph should be more explicit.\n\nSection 4.2 draws a tight parallel between the pixel perturbation test and their density ratio test, demonstrating that the pixel perturbation test captures the sensitivity of the implicit density model and not the discriminative model. They therefore suggest this test always be done while tracking changes in classification output, which is a nice takeaway.\n\nIn section 4.3, the authors draw a parallel between their score-matching objective and adversarial training, although they state that “score matching imposes a stronger constraint”. I am not an expert on these topics, but I think these kinds of more speculative observations should be moved to the discussion in general.\n\nIn section 5, the authors introduce their experimental setup. They used a baseline model ResNet-18 model with 78.01% accuracy, and compared it with their regularized model that only achieved 72.20% accuracy (5.8% drop). The authors weight the entire regularization term with a single lambda =1e-3. This raises a few important questions.\nFirst, the strength of the stability regularizer and the score matching terms should likely be decoupled to achieve maximum classification performance which is crucial in practice. The difficulty of tuning these hyperparamaters are also extremely important for us to understand the utility of the Author’s proposed approach. It would be good for us to see the results of a proper hyperparamater search over the weighting of the score-matching and stability regularization term independently. Ideally, the performance drop observed can be reduced or eliminated. \n\nSecond, presumably, if the score matching loss itself hurts classification performance, then intuitively the intuitions built early in the paper cannot be correct: if the aligned density functions p_theta(X) and p_data(X) do can not arise from logits that produce the optimal classifier, then the saliency maps produced by this method cannot be used to diagnose model correctness (as the practitioners who utilize saliency maps would hope). Since the score matching loss is only accurate up to some constant, perhaps this is the source of the issue, but we cannot conclude one way or the other from the data provided.\n\nFurthermore, few details are given for the training setup: how long was each model trained for, was early stopping employed, were multiple seeds evaluated, are we convinced that all models converged? These are important for doing the relevant comparisons, and many of the later results are hard to interpret in the absence of hyperparamater tuning or these experimental details. \n\nAnti-score matching is a great baseline experiment, but the use of the threshold seems arbitrary. This is doublyThis doubly true because the lambda of the regularization is smaller (1e-4). Why this discrepancy? It makes it harder for us to compare the results. How does this model perform as that threshold is varied? If one does a hyperparamater search with a fixed threshold, one would find the best performing anti-score matched model, which would potentially be easier to interpret. Crucially, this approach seems to actually outperform the score-matching model and underperform the baseline, implying that either the lower lambda or the anti-score constraint improved the performance of the classifier, but we cannot know which.\n\nA similar comment can be made about the gradient-norm model RE: hyperparameter searches.\n\nThe Density Ratio experiments are excellent, but the y axis is hard to read.\n\nFor the Sample Quality experiments: why would the gradient norm model outperform score matching? I am not convinced by the speculation given in the paper. It might be the case that the score-matched models were not converged, highlighting the importance of improving or simply explaining the experimental setup as I mentioned earlier.\n\nFinally, the Gradient Visualization experiments are very unclear. Intuitively, it might be the case that a small portion of the image is enough to explain the class of the data, which is what saliency maps are typically used for. The gradient norm approach and your score matching approach appear to perform almost identically, and it isn’t clear how much better they perform in a practical sense. It would be nice to have a more convincing demonstration of examples where the model obviously classifieds an image using the appropriate information present in using your method but not the baseline. As it stands, your results appear to be largely due to the fact that the gradients are smoother when using score matching or gradient-norm regularization.\n\nIn summary, I really like the approach and theory presented in the paper, and tackles an important issue with broad relevance to the field, but the experimental results as they stand are not sufficient to convince me that this approach works in practice.\n\nTypo Section 4: \"show how these can interpreted from\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "J7UZKqL-w1Z", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper examines gradient-based attribution methods that have been proposed in the explainability literature from a theoretical perspective motivated by a recent observation in energy-based generative models. \nFirst, the authors point out a general weakness of gradient-based attribution that derives from the fact that input-gradients do not provide well-defined explanations, since the shift-invariance of the softmax output makes them arbitrary.\nThe authors then propose that the reason for the success of gradient-based attribution models can be explained by the fact that discriminative models \"contain an implicit\" class-conditional density model (the mentioned recent observation about energy-based generative models).\nThey then go on to elaborate on this idea showing how aligning the implicit class-conditional generative model to the \"true\" generative model of the data would help provide relates to gradient-based attribution, how the alignment can be efficiently promoted with a novel implementation of score-matching, and how this mechanism can be practically realized as regularization costs.\nThe authors then carry out empirical studies that convincingly confirm the prediction of their theoretical ideas. First, they show that samples generated with score-matching and the proposed gradient-norm regularization are better in the sense of being less noisy and in terms of their discriminative accuracy via a trained discriminative model as proposed by the \"GAN-test approach\".\nFinally, they show that the quality of gradient-based explanations are better according to a discriminative version of the pixel perturbation test, a method to evaluate gradient explanations by perturbing pixels ranked in increasing order of relevance.\nIn conclusion, this paper establishes very interesting fundamental theoretical connections between discriminative models, energy-based generative models, and gradient-based explanations, uses this theoretical framework to explain how gradient-based explanation are overcoming the softmax shift-invariance problem (also pointed out in this paper), and introduces practical training procedures to take advantage of the gained theoretical insights to generate better explanations, which are also empirically verified in simulations.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies why input gradients can give meaningful feature attributions even though they can be changed arbitrarily without affecting the prediction. The claim in this paper is that "the learned logits in fact represent class conditional probabilities and hence input gradients given meaningful feature attributions". The main concern is that this claim is verified very indirectly, by adding a regularization term that promotes logits learning class conditional probabilities and observing that input gradient quality also improves. Nevertheless, there are interesting insights in the paper and the questions it asks are very timely and important, and overall, it could have a significant impact on further research in this area.
Decision: Accept (Oral)