Do Input Gradients Highlight Discriminative Features? 
Authors: Harshay Shah
Prateek Jain
Praneeth Netrapalli
Venue: NeurIPS 2021 Poster (2021)
Keywords: instance-specific explanations
post-hoc interpretability
feature attributions
input gradients
adversarial robustness

Abstract: Post-hoc gradient-based interpretability methods [Simonyan et al., 2013, Smilkov et al., 2017] that provide instance-specific explanations of model predictions are often based on assumption (A): magnitude of input gradients—gradients of logits with respect to input—noisily highlight discriminative task-relevant features. In this work, we test the validity of assumption (A) using a three-pronged approach:

1. We develop an evaluation framework, DiffROAR, to test assumption (A) on four image classification benchmarks. Our results suggest that (i) input gradients of standard models (i.e., trained on original data) may grossly violate (A), whereas (ii) input gradients of adversarially robust models satisfy (A).

2. We then introduce BlockMNIST, an MNIST-based semi-real dataset, that by design encodes a priori knowledge of discriminative features. Our analysis on BlockMNIST leverages this information to validate as well as characterize differences between input gradient attributions of standard and robust models.

3. Finally, we theoretically prove that our empirical findings hold on a simplified version of the BlockMNIST dataset. Specifically, we prove that input gradients of standard one-hidden-layer MLPs trained on this dataset do not highlight instance-specific signal coordinates, thus grossly violating assumption (A).

Our findings motivate the need to formalize and test common assumptions in interpretability in a falsifiable manner [Leavitt and Morcos, 2020]. We believe that the DiffROAR evaluation framework and BlockMNIST-based datasets can serve as sanity checks to audit instance-specific interpretability methods; code and data available at https://github.com/harshays/inputgradients.

Do Input Gradients Highlight
Discriminative Features?
Harshay Shah' Microsoft Research India harshayggoogle. com
Prateek Jain" Microsoft Research India prajain aiggiingggogee..oo com
Praneeth Netrapalli Microsoft Research India
pnetrapalli@goglee. com
Abstract
Post-hoc gradient-based interpretability methods 1 that provide instance- specific explanations of model predictions are often based on assumption (A): magnitude of impul graddents-g of logits with respect to input-noisily highlight discriminative task-relevant features. In this work, we test the validity of assumption (A) using three-pronged approach:
1. We develop an evaluation framework, DiffROAR, to test assumption (A) on four image classification benchmarks. Our results suggest that (i) input gradients of standard models (i.e., trained on original data) may grossly violate (A), whereas (ii) input gradients of adversarially robust models satisfy (A) reasonably well.
2. We then introduce B1ockMNIST, an MNIST-based semi-real dataset, that by design encodes a priori knowledge of discriminative features. Our analysis on BlockMNIST leverages this information to validate as well as characterize differences between imput gradient attributions of standard and robust models.
3. Finally, theoretically prove that our empirical findings hold simplified version the BlockMNIST dataset. Specifically, we prove that input gradients of standard one-hidden-layer MLPs trained on this dataset do not highlight instance-specifie "signal" coordinates, thus grossly violating (A).
Our findings motivate the need to formalize and test common assumptions in inter- pretability in falsifiable manner (3]. We believe that the DiffROAR framework and BlockMNIST datasets serve sanity checks audit interpretability methods; code and data available at /http::/// titb. com/
Introduction
Interpretability methods that provide instance-specific explanations of model predictions are often used to identify biased predictions 4. debug trained models (5), and aid decision-makinn in high- stakes domains such as medical diagnosis 7] common approach for providing instance-specing explanations is feature attribution. Feature attribution methods rank score input coordinates, or features, in the order their purported importance model prediction; coordinates achieving the top-most rank score are considered most important for prediction, whereas those with the bottom-most rank or score are considered least important.
Input gradient attributions. Ranking input coordinates based on the magnitude of input gradients fundamental feature attribution technique [8 [ that undergirds well-known methods such as SmoothGrad (2) and Integrated Gradients [9]. Given instance and trained model 0 with prediction i on J. the input gradient attribution scheme (i) computes the input gradient V, (x. 0) of the logit of the predicted label ý and (ii) ranks the input coordinates in decreasing order of their input gradient magnitude. Below we explicitly characterize the underlying intuitive assumption behind input gradient attribution methods:
Part the work completed after joining Google Research India
In Appendix
we show that our results also hold for input gradients taken w.r.t. the loxs C
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(b) Standard Resnet18 gradients
(d) & Robust Resnet18 gradients
(a) BLockNNIST images
Top block
(c) Standard MLP gradients
(c) f Robust MLP gradients
Bottom block
1
Class 0
Class 1
Figure I: Experiments on BlockMNIST dataset. (a) Four representative images from class & class 1 in B1ockMNIST dataset; every image consists of signal and null block that are randomly placed as the top or bottom block. The signal block, containing the MNIST digit, determines the image class. The null block, containing the square patch, does not encode any information of the image class. For these four images, subplots (b-e) show the input gradients of standard Resnet18, standard MLP, l2 robust Resnet18 (e=2) and la robust MLP (e=4) respectively. The plots clearly show that input gradients of standard BlockMNIST models highlight the signal block and the rin-discrmminative null block, thereby violating (A). In contrast, input gradients of adversarially robust models exclusively highlight the signal block, suppress the null block, and satisfy (A). Please see Section 5 for details.
Assumption (A): Coordinates with larger imput gradient magnitude are more relevant for model prediction compared to coordinates with smaller input gradient magnitude.
Sanity-checking attribution methods. Several attribution methods [T] are based input gradients and explicitly or implicitly assume an appropriately modified version of (A). For example, Integrated
Gradients [9] aggregate input gradients of linearly interpolated points, SmoothGrad (2) averages input gradients of points perturbed using gaussian noise, and Guided Backprop [I]] modifies input gradients by zeroing out negative values at every layer during backpropagation. Surprisingly, unlike vanilla input gradients, popular methods that output attributions with better visual quality fail simple On the other hand, while vanilla input gradients pass simple sanity checks, Hooker et al. [14] suggest that they produce estimates of feature importance that are no better than random designation of feature importance.
sanity checks that are indeed expected out of any valid attribution method [2 33
Do input gradients satisfy assumption (A)? Since (A) is necessary for input gradients attributions to accurately reflect model behavior, we introduce evaluation framework, DiffROAR, analyze whether input gradient attributions satisfy assumption (A) on real-world datasets. While DiffROAR adopts the semove-and-retralnn (ROAR) methodology [4] DiffROAR is more appropriate for testing the validity of assumption (A) because it directly compares top-ranked features against bottom-ranked features. We apply DiffROAR to evaluate input gradient attributions of MLPs CNNs trained on multiple image classification datasets. Consistent with the message in Hooker et al. [4], our experiments indicate that input gradients of standard models (i.e., trained on original data) can grossly violate (A) (see Section 4. Furthermore, we also observe that unlike standard models, adversarially trained models [1]] that are robust to lg and loo perturbations satisfy (A) in consistent manner.
Probing input gradient attributions using BlockMNIST. Our empirical findings mentioned above strongly suggest that standard models grossly violate (A). However, without knowledge of ground- truth discriminative features learned by models trained on real data, conclusively testing (A) remains clusive. In fact, this is a key shortcoming of the remove-and- and-rrrainn (ROAR) framework. So, to further verify and better understand our empirical findings, we introduce an MNIST-based semi-rea dataset, Bl BlockMNIST, that by design encodes a priori knowledge of ground-truth discriminative features. BlockMNIST is based on the principle that for different inputs, discriminative and non non-discriminative features may occur in different parts of the input. For example, in an object classification task, the object of interest can occur in different parts of the image (e.g., top-left, center, bottom- etc.) for different images. As shown in Figure aa), BlockMNIST images consist of signal block and a null block that are randomly placed at the top or bottom. The signal block contains the MNIST digit that determines the class of the image, whereas the null block, contains a square patch with two diagonals that has information about the label. This priori knowledge of ground- ground-truth discriminative

---

features in BlockMNIST data allows us to (i) validate our empirical findings ais-a-vis input gradients of standard and robust models (see fig. I and (ii) identify feature leakage as a reason that potentially explains why input gradients violate (A) in practice. Here, feature leakage refers to the phenomenon wherein given an instance, its input gradients highlight the location of discriminative features in the given instance as well as in other instances that are present in the dataset. For example, consider the first BlockMNIST image in fig. I a), in which the signal is placed in the bottom block. For this image, as shown in fig. [b,c), input gradients of standard models incorrectly highlight the top block because there are other instances in the BlockMNIST dataset which have signal in the top block.
Rigorously demonstrating feature leakage. In order to concretely verify as well as understand feature leakage more thoroughly, we design a simplified version of BlocKMNIST that is amenable to theoretical analysis. On this dataset, we first rigorously demonstrate that input gradients of standard one-hidden-lan -laee MI.Ps exhibit feature leakage the infinite width limit and then discuss how feature leakage results in input gradient attributions that clearly violate assumption (A).
Section discusses related work and section 3 presents our evaluation frame- work, DiffROAR, to test assumption (A). Section F employs DiffROAR to evaluate input gradient attributions on four image classification datasets. Section PecctionSanalyy anaayzes BlockMNIST data differen- tially characterize input gradients of standard and robust models using feature leakage. Section 6 provides theoretical results on simplified version on BlockMNIST that shed light on how feature leakage results in input gradients that violate assumption (A). Our code, along with the proposed datasets, is publicly available at
Paper organization:
2 Related work
Due to space constraints, we only discuss directly related work and defer the rest to Appendix ^
Sanity checks for explanations. Several explanation methods that provide feature attributions are often primarily evaluated using inherently subjective visual assessments 21.. Unsurprisingly, recent "sanity checks" show that sole reliance on visual assessment is misleading, as attributions can lack fidelity and inaccurately reflect model behavior. Adebayo et al. [12 and Kindermans et al. [K3 show that unlike input gradients [8) other popular methods- gratdodd-guidddd backprop (16). gradient C input [77, integrated gradients output explanations which lack fidelity on image data, as they remain invariant to model and label randomization. Similarly. Yang and Kim [8] use custom image datasets to show that several explanation methods are more likely to produce false positive explanations than vanilla input gradients. Moreover, several explanation methods based on modified backpropagation do not pass basic sanity checks [9] 22, 21.. To summarize, well-known gradient- based attribution methods that seek to mitigate gradient saturation (9] 22), discontinuity [23]. and visual noise [16] surprisingly fare worse than vanilla input gradients on multiple sanity checks.
Evaluating explanation fidelity. The black-box nature of neural networks necessitates frameworks that evaluate the fidelity or "correctness" of post-hoc explanations without knowledge of ground -ruth features learned by trained models. Modification- basod evaluation frameworks [24] 25 26) gauge explanation fidelity by measuring the change in model performance after masking input coordinates that a given explanation method considers most (or least) important. However, due to distribution shifts induced by input modifications, one cannot conclusively attribute changes in model performance to the fidelity of mnstance-specife explanations [27]. The remove-any aeeeev--ndd-etraii (RDAR) framework [14] accounts for distribution shifts by evaluating the performance of models retrained on train data masked using post-hoc explanations. Surprisingly, contrary to findings obtained via sanity checks, experiments with the ROAR framework show that multiple attribution methods, including vanilla input gradients, are no better than model-independent random attributions that lack explanatory power [14]. Therefore, motivated by the central role of vanilla input gradients in attribution methods, we augment the ROAR framework to understand when and why input gradients violate assumption (A).
Effect of adversarial robustness. Adversarial training [13] not only leads to robustness to & adversarial attacks (28), but also leads to perceptually-alinae feature representations (2). and improved visual quality of input gradients [30]. Recent works hypothesize that adversarial training improves the visual quality of input gradients by suppressing irrelevant features [1] and promoting sparsity and stability (32] in explanations. Kim et al. (33 use the ROAR framework to conjecture that adversarial training "tilts" input gradients to better align with the data manifold. In this work, we use experiments on real-world data and theory on data with features known a priori in order to differentially characterize input gradients of standard and robust models ais-a-vis assumption (A).
3

---

DiffROAR evaluation framework
In this section, we introduce evaluation framework, DiffROAR, to probe the extent to which instance-specific explanations, or feature attributions, highlight discriminative features in practice. Specifically, our framework, DiffROAR, builds upon the remove-and-retrain (ROAR) methodology [[4] to test whether feature attribution methods satisfy assumption (A) on real-world datasets.
Setting.
We consider the standard classification setting; Each data point (1()), where instance 20(() IRd and label ( E y for some label set y, is drawn independently from a distribution DD on Rd x y. Given dataset {a!!0 ( where E [n|:= {1, ... ,nI,
.. denotes the j"h coordinate of (.) Note that we also refer the d coordinates of instance x(() features interchangeably.
Attribution schemes. feature attribution scheme A : R + fo a is a permutation of [d]} maps d-dimensional instance x to a permutation, ordering, A(x) [d] [d] of its coordinates. For example, the imput gradient attribution scheme takes as input instance x' & predicted label ğ and outputs ordering [d] that ranks coordinates in decreasing order of their input gradient magnitude. That is, coordinate j ranked ahead of coordinate if the magnitude of the jth coordinate of V.Logit (x, is larger than that of the k.th coordinate.
Unmasking schemes. Given instance æ and subset S S [d of coordinates, the unmasked instance x zeroes out all coordinates that are not in subset S: zj Ij if j S and 0 if $ S. An uumasking scheme A F + (S S S [d]} simply maps instance x to a subset A(x) [d] of coordinates that can be used obtain unmasked instance x ^<(r), Any attribution scheme A naturally induces top-k and bottom-k unmasking schemes, A and Abs which output & coordinates with the top-most and bottom-most attributions in A(x) respectively. In other words, given attribution scheme A and level k, the top-k and bottom-k unmasking schemes, A. and A can be defined as follows:
S
(x): (A(a), :ssk) k), Aht (1)) (A(x)) d ddd).
Original Image
Top-k unmasked Imago.
For example, Figure depicts image and its top-k un- masked variant _- (z) In this case, the attribution scheme A assigns higher rank to pixels in the foreground. So. the top- 25% unmasking operation, =x,)),((1)) highlights the monkey by retaining pixels with t0p-25% attribution ranks and zeroing out the remaining pixels that correspond the green background.
Figure 2: Pictorial example of top- 25% unmasked image.
Predictive power of unmasking schemes. The predictive power an unmasking scheme with respect model architecture M (e.g., resnet18) can be defined as the best classification accuracy that can be attained by training a model with architecture M on unmasked instances that are obtained via unmasking scheme A. More formally, it can defined as follows:
ED [1 [((AA w]]
PredPower m (A) sup
FEM,f.R*->>
Due to masking-indiin inducedd distribution shifts, models with architecture M that are trained using original data cannot be plugged in estimate PredPower (A). The ROAR framework [14] sidesteps this issue by retraining models on unmasked data, as similar model architectures tend to learn "similar" employ the ROAR framework to estimate PredPower M (A) in two steps. First, we use unmasking scheme to obtain unmasked train and test datasets that comprise data points the form Azx) y). Then, we retrain a new model with the same architecture M unmasked train data and evaluate its accuracy unmasked test data.
classifiers 34 35 36 37]. Therefore,
DiffROAR evaluation metric to assumption (A). Recall that an attribution scheme maps an instance to a permutation of its coordinates that reflects the order of estimated importance in model prediction. An attribution scheme that satisfies assumption (A) must place coordinates that are more important for model prediction higher up in the the attribution order. More formally, given attribution scheme A, architecture M and level k; we define DiffROAR as difference between the predictive power of top-k and bottom-k: unmasking schemes, A and Abol.
DiffROAR by (A, k) PredPouer (A PredPower (Abo))
(1)

---

Interpreting the DiffROAR metric. The sign of the DiffROAR metric indicates whether the given attribution scheme satisfies or violates assumption (A). For example, DiffROAR (A. implies that A violates assumption (A) as coordinates with higher attribution ranks have worse predictive power with respect to architecture M. Similarly, the magnitude of the DiffROAR metric quantifies the extent to which the ordering in attribution scheme A separates the most and least discriminative coordinates into two disjoint subsets. For example, a random attribution scheme A,, which outputs attributions (x) chosen uniformly at random from all permutations of [d]. neither highlights nor suppresses discriminative features; E|DiffROAR (Ar, k)] = 0 for any architecture M.
On testing assumption (A). To verify (A) for a given attribution scheme A, it is necessary to evaluate whether input coordinates with higher attribution rank are more important for model prediction than coordinates with lower rank. Consequently, the ROAR-basce metric in Hooker et al. [14]. which essentially computes the top-k predictive power, is not sufficient to test whether attribution methods satisfy assumption (A). Therefore, as discussed above, DiffROAR tests (A) by comparing the top-k predictive power, PredPower (4%)), ), to the bottom-k predictive power, PredPower, (A4bt), using multiple values of k.
Testing assumption (A) on image classification benchmarks
In this section, we use DiffROAR to evaluate whether input gradient attributions of standard and adversarially robust MLPs and CNNs trained on four image classification benchmarks satisfy as- sumption (A). We first summarize the experiment setup and then describe key empirical findings.
Datasets and models. We consider four benchmark image classification datasets: SVHN [38,
Fashion MNIST (39] CIFAR-10 40) and ImageNet-10 10 [4]] ImageNet-ID 10 is an open-sourced variant (https: grouped into 10 super-classes. ImageNet-10 10 enables to test assumption (A) on Imagenet without the computational overload of training models on the 1000-way ILSVRC classification task [42]. We evaluate input gradient attributions of standard and adversarially trained two-hidden-layer MLPs and Resnets 43 We obtain & and loo e-robust models with perturbation budget using PGD adversarial training 15 Unless mentioned otherwise, we train models using stochastic gradient descent (SGD), with momentum 0.9, batch size 256, lg regularization 0.0005 and initial learning rate 0.1 that decays by factor of 0.75 every 20 epochs. Additionally, we use standard data augmentation and train models for at most 500 epochs, stopping early if cross-entropy loss on training data goes below 0.001. Appendix C provides additional details about the datasets and trained models 3
of Imagenet 41 with 80, 000 images
Estimating DiffROAR on real data. We compute the evaluation metric, DiffROAR (A, k)), on real datasets in four steps, as follows. First, we train a standard or robust model with architecture M on the original dataset and obtain its input gradient attribution scheme A. Second, as outlined in Section we use attribution scheme A and level & (i.e., fraction of pixels be unmasked) to extract the top-k and bottom-k unmasking schemes: A and Aboot Third, we apply A and Abe on the original train& & test datasets to obtain top-k and bottom-k unmasked datasets respectively. Finally, to compute DiffROARM/ (A, kc) via eq. (I). estimate top-k and bottom-k predictive power, PredPower M (A and PredPower m (Aba), by retraining new models with architecture M on top- k and bottom-k unmasked datasets respectively. Also, note that we (a) average the DiffROAR metric over five runs for each model and unmasking fraction or level i and (b) unmask individual image pixels without grouping them channel-wise.
Experiment setup. Now, we analyze the DiffRDAR metric as a function of the unmasking fraction E (5, 10, 20, ....... 100) % in order to evaluate whether input gradient attributions of models trained on four image classification benchmarks satisfy assumption (A). In particular, shown in Figure B we use DiffROAR to analyze input gradients of standard and adversarially robust two wwo-hidden-layer MLPs on SVHN & Fashion MNIST, Resnet18 18 on ImageNet- 10, and Resnet50 on CIFAR-10. 10. In order to calibrate our findings, we compare input gradient attributions of these models to two natural baselines: model-agnostic random attributions and input-agnostic attributions of linear models.
Input gradients of standard models. Input gradient attributions of standard MLPs trained on SVHN satisfy assumption (A), as the DiffROAR metric in Figure 3a) is positive for all values of level k < 100%. However, in Figure b), the DiffROAR curves of standard MLPs trained on
*Code publicly available at
5

---

(a) Input gradients of MLPs on SVHN
[c] Input gradients of Resnet18 on ImageNet-10
DiffRoAR metric at level k
0..5p
standard, 0
Eu robust.
DiFFRDAR metric at level k
11.31
standard. D Co robust.
F's robust, - 0.10
linear model
Fxx robust, it lincar model
0:15-
0.3-
F2 robust, 0.25
Fx robust,
random attribution
0.30
robust, random attribution
0.2
0.66
4
0.00
0.33
-0.00
0.00
-0.00
153
1.4
1.01
Level in fraction of unmasked picels
Level A: fraction of unmasked pixels
(b) Input gradients of MLPs on Fashion MNIST
(d) Input gradients of Resnet50 on CIFAR-10 10
D1ETROAR metric level K
0.30
DiF metric at level &
standard, 0.00 rabust,
standard,
For robust,
0.15
Cv robust,
linear model
robust, 0.25 8 linear model
* Ey robust, random attribution
robust, 0.50 random attribution
0.10
robust, 0.30
0.30
0.15
0.05
11.00
-1.00
0.2 0.3 0.5 0.5
0.7
0.3
0.2
6.3
0.4
0.5
0.56
0.8
0.0
1.0
Level i: fraction of unmasked pixels
Level l:
fraction of unmasked pixels
Figure 3: DiffROAR plots for input gradient attributions of standard and adversarially robust two-
hidden-layer MLPs on (a) SVHN & (b) Fashion MNIST, (c) Resnet18 on ImageNet-10 and (d) Resnet50 on CIFAR-1O. Subplot (a) indicates that adversarially robust MLPs consistently and considerably outperform standard MLPs the DiffROAR metric for all l 100%. Subplot (b) shows that for most unmasking fractions k standard MLPs trained on Fashion MNIST, unlike robust MLPs, fare no better than model-agnostic random attributions and input-agnostic attributions of linear models. Subplots (c) and (d) show that when k < 40%, standard Resnet models trained on CIFAR- 10 and ImageNet-10 grossly violate (A), thereby implying that coordinates with top-most gradient attribution rank have worse predictive power than coordinates with bottom-most rank. In stark contrast, input gradients of Resnets that are robust to &g and for adversarial perturbations satisfy assumption (A) reasonably well. We observe that increasing the perturbation budget e during adversarial training amplifies the magnitude of Di DiffROAR for every Is across all four image classification benchmarks.
Fashion MNIST indicate that input gradient attributions, consistent with findings in Hooker et al. [14, can fare no better than model-agnostic random attributions and input-agnostic attributions of linear models vis-a-vis vis assumption (A). Furthermore, and rather surprisingly, the shaded area in Fc) and Figure d) shows that when level < 40%, DiffROAR curves of standard Resnets trained on CIFAR-10 and Imagenet-10 10 are consistently negutive and perform considerably worse than madel-agnostic and input-agnostic baseline attributions. These results strongly suggest that on CIFAR-10 10 and Imagenet-10 10, input gradients of standard Resnets grossly violate assumption (A) and suppress discriminative features. In other words, coordinates with larger gradient magnitude have worse predictive power than coordinates with smaller gradient magnitude.
Input gradients of robust models. Models that are c-robust to by and loo adversarial perturba- tions fare considerably better than standard models on the DiffROAR metric. For example, in Fig- ure a). when level k equals 10%, robust MLPs trained on SVHN outperform standard MLPs on the Di DiffROAR metric by roughly 10-30%. The DiffROAR curves of adversarially robust MLPs in Fig- ure 3 are positive at every level & 100%, which strongly suggests that input gradient attributions of robust MLPs satisfy assumption (A). Similarly, robust resnet50 models trained on CIFAR- 10 and ImageNet-10 satisfy assumption (A) reasonably well and, unlike standard resnet50 models, starkly highlight discriminative features. Furthermore,
observe that increasing the perturbation budget e in l2 or Eoe PGD adversarial training [15) amplifies the magnitude of DiffROAR across and for all four datasets. That is, the adversarial perturbation budget determines the extent to which input gradients differentiates the most and least discriminative coordinates into two disjoint subsets.
Additional results. In Appendix C we show that our DiffROAR results are robust to choice of model architecture & SGD hyperparameters during retraining and also hold for input gradients taken with respect to cross-entropy loss. Additionally, while DifrROAR without retraining gives qualitatively similar results, they are not as consistent across architectures as with retraining, particularly for small unmasking fraction & that induce non-trivial distribution shifts.

---

5 Analyzing input gradient attributions using BlockMNIST data
To verify whether input gradients satisfy assumption (A) more thoroughly, we introduce and perform experiments on B1ockMNIST. an MNIST-based dataset that by design encodes a priori knowledge of ground-truth discriminative features.
BlockMNIST dataset design: The design of the BlocKMNIST dataset is based on two intuitive properties of real-world object classification tasks: (i) for different images, the object of interest may appear in different parts of the image (e.g.. top-left, bottom-right); (ii) the object of interest and the rest of the image often share low-level patterns such as edges that are not informative of the label on their own. We replicate these aspects BlockMNIST instances, which are vertical concatenations of two 28 x 28 signal and null image blocks that are randomly placed at the top bottom with equal probability. The signal block is an MNIST image of digit or digit 1, corresponding to class 0 or 1 of the B1ocKMNIST image respectively. On the other hand, the mull block in every BlockMNIST image, independent of its class, contains a square patch made of two horizontal, vertical, and slanted lines, as shown in Figure [a). It is important note that unlike the MNIST signal block that is fully predictive of the class, the non-diseriminative null block contains no information about the class. Standard as well as adversarially robust models trained on BlockMNIST data attain 99.99% test accuracy. thereby implying that model predictions are indeed based solely on the signal block for any given instance. We further verify this by noting that the predictions of trained model remain unchanged on almost every instance even when all pixels in the null block are set to zero
Do standard and robust models satisfy (A)? As discussed above, unlike the null block that has no task-relevant information, the MNIST digit in the signal block entirely determines the class of any given B1ocKMNIST image. Therefore, in this setting, we can restate assumption (A) as follows: Do imput gradient attributions highlight the signal block over the null block? Surprisingly, as shown in Figure I b,c), input gradient attributions of standard MLP and Resnet18 models highlight the signal block as well as the non-discriminative null block. In stark contrast, subplots (d) and (e) in Figure [ show that input gradient attributions of la robust MLP and Resnet18 models exclusively highlight MNIST digits in the signal block and clearly suppress the square patch in the null block. These results validate our findings on real-world datasets by showing that unlike standard models, adversarially robust models satisfy (A) on BlocKMNIST data.
Feature leakage hypothesis: Recall that the discriminative signal block in BlocKMNIST images is randomly placed at the top or bottom with equal probability. Given our results in Figure [ we hypothesize that when discriminative features vary across instances (e.g., signal block at top vs. bottom), input gradients of standard models not only highlight instance-specific features but also leak discriminative features from other instances. We term this hypothesis feature leakage.
To test our hypothesis, we leverage the modular structure in BlockMNIST to construct a slightly modified version, BLoCKMNIST -Top. wherein the location of the MNIST signal block is fixed at the top for all instances (see fig. 4... In this setting, in contrast to results on BlockMNIST, input gradients of stan- dard Resnet18 and MLP models trained on BLocKMNIST-Top satisfy assump- tion (A). Specifically, when the signal block is fixed at the top, input gradi- ent attributions in Figure Lb, c) clearly highlight the signal block and suppress the null block, thereby supporting periments, we believe that understanding how adversarial robustness mitigates feature leakage is an interesting direction for future work.
(a) BlockMWSST Top (b) Standard Resnet18
(c) Standard MLP
Bottom Top
Class 0 Class 1 Figure 4: (a) In BlockMNIST-Top images, the signal & null blocks are fixed at the top & bottom respectively. contrast to results on BlockMNIST in fig. []] input gradients of standard (b) Resnet18 and (c) MLP trained on BLocKMNIST-TOR highlight discriminative features in the signal block, suppress the null block, and satisfy (A).
Class Class Class 0 Class 1
feature leakage hypothesis. Based on our BlockMNIST ex-
Additional results. In Appendix D.I we (i) visualize input gradients of several BlockMNIST and Bl BlockMNIST- images, (ii) introduce a quantitative proxy metric to compare feature leakage between standard and robust models, (iii) show that our findings are fairly robust to the choice and number of classes in BlockMNIST data, and (iv) evaluate feature leakage in five feature attribution methods. We also provide experiments that falsify hypotheses vis-a-vis input gradients and assumption (A) that considered in addition to feature leakage.
7

---

Feature leakage in input gradient attributions
To understand the extent of feature leakage more thoroughly, we introduce a simplified version of the BlockMNIST dataset that is amenable to theoretical analysis. We rigorously show that input gradients of standard one-hidden-layer MLPs do not differentiate instance-specifie features from other task-relevant features that are not pertinent to the given instance.
Given dimension of each block d, feature vector u' Rd with ||u*] = 1, number of blocks and noise parameter 17, we will construct input instances of dimension d d. More concretely, sample (æ, y) E add x } from the distribution is generated as follows:
Dataset:
y +1 with probability 0.5 and x: = [91..
192.
yu" + 795>
m9aa with j chosen at random from [1/2]
(2)
where each E Rd is drawn uniformly at random from the unit ball. For simplicity, we take d to be even so that d/2 is integer. We can think of each as concatenation of d d-dimensional blocks (.i...., .d}. The first 4/2 blocks, {l.... d/2}. are task-relevant, as every example (x; y) contains an instance-specific signal block 2; yu" + +9: for some [</2] that is informative of its label y. Given instance v, we use j (x) to denote the unique instance-specific signal block such that (x) = yu (x)- On the other hand, noise blocks {4/2 + 1, ...-.: d) do not contain task-relevant signal for any instance x. At a high level, the instance-specifie signal block j (xx and noise blocks {d/2- - 1, I,...,d d} in instance I correspond to the discriminative MNIST digit and the null square patch in B1ockMNIST images respectively. For example, each row in Figure 51a) illustrates an instance x where d 10, d 1, and u²
Model: We consider one-hidden layer MLPs with ReLU nonlinearity in the infinite-width limit. More concretely, for a given width m. the network is parameterized by R E Rmxdd b E R and w E R" Given an input instance (Ir, y) R.dd {=1}, the output score (or logit) and cross-entropy (CE) loss L are given by:
f((,,,,b) x): ò (Rx b)), ((W,RRb) (x, v)): log (1 exp (-y f((w, R. b), z))).
where @b{)= max(0, 1) denotes the ReLU function. A remarkable set of recent results 44
show that as m + oo, the training procedure is equivalent to gradient descent (GD) an infinite dimensional Wasserstein space. In the Wasserstein space, the network can be interpreted as a probability distribution y over R x did x IR with output score f and cross entropy loss L defined as:
fvv, :y:: (w.,;))) (wo (rr 2) + 6)], L(v, (z, y)): log (1 exp (- y f(v,z))) (3)
Theoretical analysis: Our approach leverages the recent result in Chizat and Bach 48 which shows
that if GD in the Wasserstein space W2 [R x ddd x R on Ep [C(n, (z, v))] converges, it does so to max-margin classifier given by:
:== arg max min [(, (EP(S (r..))~
(4)
where gdd+1 denotes the surface of the Fuclidean unit ball in Rdd-2 and P (suh+11 denotes the
space of probability distributions over gdă+1 Intuitively, our main result shows that any data point (x, ~ D, the input gradient magnitude of the max-margin classifier D² is equal over all task-relevan blocks {1...., d/2} and on the remaining noise blocks {4/2 + +1...
Theorem 1. Consider distribution D ② with < 10d" There exists a max-margin classifier v" for D in Wasserstein space (i.e., training both layers of FCN with m + 00) given by (4) such that for all & (x, ~ D: (i) (V.L(v" (z. v)))g) c > 0 for every j E [d/2] and (ii) (VoL(v" (a, (I.,))),, for every j E (d/2+1, ,d). where (VeL(v", (x.1))), v))), denotes the 7th block of the input gradient V.L(v",(,,,
Theorem guarantees the existence of max-margin classifier such that the input gradient magnitude for any given instance is (i) non-zero constant each of the first d/2 task-relevant blocks, and
8

---

(a) Input data
(b) Standard linear model
(c) Standard MLP
(d) 6se Robust MLP
Inpost gradient attribations
Inputt gradient attribu itiins
Input gradient attributtios
1.0
Input data points
04
0.5
10
8
10
Task-rolovant
Noise
Task-rdeeannt
Noise
Tack-relesant
Nosse
Task-roievant
Noise
Input coordinates
Input coordinates
Input coordinates
Input coordinates
Figure 5: Input gradients of linear models and standard & robust MLPs trained on data from eq. with d = 10, 1, = and u' = 1. (a) Each row in corresponds to an instance æ, and the highlighted coordinate denotes the signal block (=) & label y. (b) Linear models suppress noise coordinates but lack the expressive power to highlight instance-specific signal j (æ), as their input gradients in subplot (b) are identical across all examples.
(c) Despite the expressive power to highlight instance-specific signal coordinate j [z)) input gradients of standard MLPs exhibit feature leakage (see Theorem D and violate (A) as well. (d) In stark contrast, input gradients of adversarially trained MLPs suppress feature leakage and starkly highlight instance-specific signal coordinates j (x)).
(ii) equal to zero on the remaining d/2 noise blocks that do not contain any information about the label. However, input gradients fail at highlighting the unique instance-specific signal block over the remaining task-relevant blocks. This clearly demonstrates feature leakage, input gradients for any given instance also highlight task-relevant features that are, in fact, not specifie to the given instance. Therefore, input gradients of standard one- mee-hidden-lene layer MLPs do not highlight instance- discriminative features and grossly violate assumption (A). Appendix F we present additional results that demonstrate that adversarially trained one-hidden-layer MIPs can suppress feature leakage and satisfy assumption (A).
Now, we supplement our theoretical results by evaluating input gradients of linear models as well standard & robust one-hidden-layer ReLU MLPs with width 10000 the dataset shown in Figure Note that all models obtain 100% test accuracy on this linearly separable dataset, simplified version of BlockMNIST that is obtained via eq. with d = 10, d = 1, = O and u' 1. Due to insufficient expressive power, linear models have input-agnostic gradients that suppress all five noise coordinates, but do not differentiate the instance-specifie signal coordinate from the remaining task-relevant coordinates. Consistent with Theorem [ even standard MLPs, which are expressive enough to have input gradients that correctly highlight instance-specing coordinates, apply equal weight on all five task-relevant coordinates and violate (A) due to feature leakage. On the other hand, Figure c) shows that the same MLP architecture, if robust to lo adversarial perturbations with norm 0.35, satisfies (A) by clearly highlighting the seetacc--peccfic signal coordinate over all other noise and task-relevant coordinates
Empirical results:
7 Discussion and conclusion
In this work, we took three- three-pronged approach to investigate the validity of a key assumption made in several popular post-hoc attribution methods: (A) coordinates with larger input gradient magnitude are more relevant for model prediction compared to coordinates with smaller input gradient magnitude. Through (i) evaluation on real-world data using our DiffROAR framework, (ii) empirical analysis on Blo BlockMNIST data that encodes information of ground-truti discriminative features, and (iii) a rigorous theoretical study, we present strong evidence to suggest that standard models do not satisfy assumption (A). In contrast, adversarially robust models satisfy (A) in a consistent manner. Furthermore, our analysis Section and Section 6 indicates that feature leakage sheds light on why input gradients of standard models tend to violate (A). We provide additional discussion in Appendix B
This work exclusively focused on "vanilla" input gradients due to their fundamental significance in feature attribution. similarly thorough investigation that analyzes other commonly- attribution methods is an interesting avenue for future work. Another interesting avenue for further analyses is understand how adversarial training mitigates feature leakage in input gradient attributions.

---

References
[1] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv: arXiv:1312.603: 1312.6034, 2013.
[2] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 2017.
[3] Matthew L. Leavitt and Ari S. Morcos. Towards falsifiable interpretability research. ArXiv, abs/2010.12014 2020.
[4] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135-1144, 2016.
[5] Julius Adebayo, Michael Muclly, Ilaria Liccardi, and Been Kim. Debugging tests for model explanations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, edi- tors, Advances in Neural Information Processing Systems, volume 33, pages 700-712, Cur- Associates, Inc., 2020. htcp:ttttp//ccceeeeeee pdf
newrips cc/ pttpp//ppp/
[6] John R Zech, Marcus A Badgeley, Manway Liu, Anthony В Costa, Joseph J Titano, and Eric Karl Oermann. Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: cross-sectional study. PLoS medicine, 15(1 15(11)xe1002683 2018.
[7] Gregor Stiglic, Primoz Kocbek, Nino Fijacko, Marinka Zitnik, Katrien Verbert, and Leona Cilar. Interpretability of machine learning-bsae basee prediction models in healthcare. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(5);e1379, 2020.
[8] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 11:1803-1831, 2010.
[9] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference Machine Learning, pages 3319-3328. PMLR, 2017.
[10] W James Murdoch, Chandan Singh, Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Definitions, methods, and applications in interpretable machine learning. Proceedings of the National Academy of Sciences, 1 116(44):20071222) 2019.
[11] J Springenberg, Alexey Dosovitskiy, Thomas Brox, and M Riedmiller. Striving for simplicity: The all convolutional net. In ICLR (workshop track), 2015.
[12] Julius Adebayo, Justin Gilmer, Michael Muclly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in Neural Information Processing Systems, 31: 9505-9515, 2018.
[13] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pages 267-280. Springer, 2019.
[14] Sara Hooker, D. Erhan, Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In NeurIPS, 2019.
[15] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. Utt.btttss://opeereee /ttp:p/rr///peeeeeeeeeeee/ee//er
[16] Jost Tobias Springenherg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806 2014.
10

---

[17] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just a black box: Learning important features through propagating activation differences. arXiv preprint arXiv: 160.v:1605.01713 2016.
[18] Mengjiao Yang and Been Kim. Benchmarking attribution methods with relative feature impor- tance. arXiv preprini arXiv:/907. 1907.0970, 2019.
[19] Leon Sixt. Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified BP attributions fail. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1056-9057. PMLR, 13-18 Jul 2020. URLhttpp:
7/ /ppoceedings..o mlr wtt.ppese//////sxt2200. /
[20] Chih-Kuan Yeh, Cheng- Yu Hsieh, Arun Suggala, David Inouye, and Pradeep K Ravikumar. On the (in)fidelity and sensitivity of explanations. Advances in Neural Information Processing Systems, volume 32, pages 10967-10978, 2019.
[21] Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In International Conference on Learning Representations, 2018. URL http:///ppppeeeeeeeeeeeeeeeeeeee
[22] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Gradients of counterfactuals. arXiv preprint 1rXiv:1611.02633 2016.
[23] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3 3155-3153, PMLR, 2017.
[24] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Rober Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7): 0130140. URL ftttps://doi.orgee
2015. doi: 10.137//jorrmal.pon.. 1371/journal pone 0130044
[25] W. Samek, A. Binder, G. Montavon, S. Lapuschkin. and K. Müller. Evaluating the visualization of what deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems, 21266)26602 2017, doi: 10. 2.110//NNN
[26] Leila Arras, Ahmed Osman, Klaus-Robert Müller, and Wojciech Samek. Evaluating recur- rent neural network explanations. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 1 113-126, Florence, Italy, Au- gust 2019, Association for Computational Linguistics. doi: 10.19453/11/111-4111.. URL https://www. http::/wwwwwwwwwb..rrro.... wwwwww.wwa........ wwt.:orggaaaoooog//ttaeeooae
[27] Richard Tomsett, D. Harborne, S. Chakraborty, Prudhvi Gurram, and A. Preece. Sanity checks for saliency metrics. ArXiv, abs/1912.01451, 2020.
[28] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry, On adaptive attacks to adversarial example defenses. arXiv preprint arXiv:2002.083477 2020.
[29] Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew llyas, Logan Engstrom, and Aleksander Madry. Image synthesis with a single (robust) classifier. Advances in Neural Information Processing Systems, 32, 2019.
[30] A. Ross and Finale Doshi- Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In AAAI, 2018
[31] Beomsu Kim, Junghoon Seo, Seunghyun Jeon, Jamyoung Koo, J. Choe, and Taegyun Jeon. Why are saliency maps noisy? cause of and solution to noisy saliency maps. 2019 IEFF/CVF International Conference Computer Vision Workshop (ICCVW), pages 4149-4157, 2019.
[32] Chalasani, Chen, A. Chowdhury, S. Jha, and X. Wu. Concise explanations of neural networks using adversarial training. In ICML, 2020.

---

[33] Beomsu Kim, Junghoon Seo, and Taegyun Jeon. Bridging adversarial robustness and gradient interpretability. ArXiv, abs//903. 1626, 2019.
[34] Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agree to agree: Neural net- works share classification order on real datasets. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pages 3950-3960, PMLR, 13-18 Jul 2020. URL http://proceediiin pwwpppoincess.....o mlr. prwss/ v1 11111999hhhheeeee html
[35] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent learning: Do different neural networks learn the same representations? In Dmitry Storcheus, Afshin Rostamizadeh, and Sanjiv Kumar, editors, Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges af NIPS 2015, volume 44 of Proceedings of Machine Learning Research, pages 196 212, Montreal, Canada, 11 Dec 2015. PMLR. URL
http://proceediing mlr. wwwppprsssseeee http:////44/111cooeeeeee www.convergeet.....mmm html
[36]
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang. Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing com- plexity. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d'Alché-But E. Fox, and R. Garnett, editors, Advances in Neurul Information Processing Systems, volume 32, Cur- ran Associates, Inc., 2019,
httttpp://proceednngg seapeeeiinnnnnnuuiiipp nww.inuurp.... ceturrpps.ceppppe//001111 wttppappeee019//lleeee.. b432f SuS22eaccaatrcceeecooaaaeeeeeee
pdf
[37] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9573-9585 Curran Associates, Inc., 2020. URL httppss:::pppceedde/ttee neurips htt/pp/e0000///llfooooo//2222222 -a57
[38] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[39] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: (000:0070.774 2017.
[40] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[41] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC),, 115(3):211 2015. doi: 10. 10070/// 1263-015-0816-7
[43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint
2015.
[44] Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33): E7665-E7671, 2018.
[45] Lénaïc Chizat and Francis Bach. On the global convergence of gradient descent for over- parameterized models using optimal transport. In Proceedings of the 32nd International Conference Neural Information Processing Systems, pages 3040-3050, 2018.
[46] Grant M Rotskoff and Eric Vanden-Eijnden. Trainability and accuracy of neural networks: An interacting particle system approach. arXiv preprint arXiv:1805.00 00991, 2018.
[47] Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of deep neural networks. Mathematics of Operations Research, 2021.
12

---

[48] Lénaïc Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss, In Conference on Learning Theory, pages 1305-1338. PMLR, 2020.
[49] A. AAAI, 2019.
Ghorbani, Abubakar Abid, and James Y. Zou. Interpretation of neural networks is fragile, In
[50] Juyeon Heo, Sunghwan Joo, and T. Moon. Fooling neural network interpretations via adversarial model manipulation. In NeurIPS, 2019.
[51] Ann-Kathrin Dombrowski, Maximilian Alber, Christopher J. Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is
to blame. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'ché Buc, Emily B. Fox, and Roman Garnett, editors. NewrlPS, pages 13567-13578, 2019. URL http://dblp... uni-trier /w//b//coo//ii///niiiiiiiieeeeee http///iis//ininniiee wwwp:HHHobboosssiAAAAA.....
[52] Naman Bansal, Chirag Agarwal, and Anh M Nguyen. Sam: The sensitivity of attribution methods to hyperparameters. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages I 11-21, 2020.
[53] Mayank Singh, Nupur Kumari, P. Mangla, Abhishek Sinha, V. Balasubramanian, and Balaji Krishnamurthy. Attributional robustness training using input-gradient spatial alignment. In ECCV, 2020.
[54] H. Lakkaraju, Nino Arsov, and Osbert Bastani. Robust and stable black box explanations, In ICML, 2020.
[55] E. Chu, Deb Roy, and Jacob Andreas, Are visual explanations useful? a case study in model-in- the-loop prediction. ArXiv, abs/2007.122488 2020.
[56] Forough Poursabzi-Sangdeeh, D. Goldstein, J. Hofman, Jennifer Wortman Vaughan, and H. Wal- lach. Manipulating and measuring model interpretability. ArXiv, abs/ 802.07810, 2018.
[57] Danish Pruthi, Bhuwan Dhingra, Livio Baldini Soares, M. Collins, Zachary C. Lipton, Graham Neubig, and William W. Cohen. Evaluating explanations: How much do explanations from the teacher aid students? ArXiv, abs/2012.0089) 2020.
[58) Peijie Chen, Chirag Agarwal, and Anh Nguyen. The shape and simplicity biases of adversarially robust imagenet-trainee cnns. arXiv preprint arXiv:2006.09:3 2020.
[59] Osman Semih Kayhan and Jan C van Gemert. On translation invariance in cmns: Convolutional layers exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14274-14285, 2020.
[60] Thomas Tanay and Lewis Griffin. A boundary tilting persepective on the phenomenon of adversarial examples. arXiv preprint
2016.
[61] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. arXiv preprint arXiv:195.002175. 2019.
[62] Ali Shafahi, W Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adver- sarial examples inevitable? arXiv preprint 1rXiv:1809.001 0200, 2018
[63] Sébastien Bubeck, Yin Tat Lee, Eric Price, and Ilya Razenshteyn. Adversarial examples from computational constraints. In International Conference on Machine Learning, pages 831-840. PMLR, 2019.
[64] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? arXiv preprint arXiv:2007. 20X.:20007.08489, 2020.
[65] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry, Adversarial robustness as a prior for learned representations. arXiv preprint 10Xiv:1906.00945, 2019.
13

---

[66] Adi Shamir, Odelia Melamed, and Oriel BenShmuel. The dimpled manifold model of adversarial examples in machine learning, arXiv preprint arXiv:2106.1013 10151, 2021.
[67] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks (2013), arXiv preprint arXiv:/311.2901, 2013.
[68] Lénaic Chizat. Personal communication, 2021.
14


Reviews:
[{"review_id": "yiRo87Jiosq", "reviewer": "Reviewer_L2GP", "summary": "The authors test the current assumption for gradient-based explanation methods, i.e., a high magnitude of vanilla gradients highlight more discriminative task-relevant features and vice-versa. The authors theoretically and empirically test the above assumption using DiffROAR, a new evaluation framework, and BlockMNIST, a new semi-real dataset. Empirical analysis using DiffROAR concludes that gradients of standard models violate the above assumption. whereas gradients of adversarially trained models don't. The BlockMNIST dataset separates the signal and noise part of a given sample by encoding a priori knowledge of discriminative and non-discriminative features in the samples. The work tackles one of the existing problems of gradient-based explanation methods and suggests that the proposed frameworks and datasets can be used as sanity checks to audit instance-specific interpretability methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well-written with coherent details of the proposed DiffROAR evaluation metric and BlockMNIST dataset.\n\n2. The proposed DiffROAR evaluation metric is an intuitive extension of the previously proposed ROAR metric.\n\n3. The empirical findings that standard one-hidden-layer MLPs violate the gradient assumptions are supported by respective theoretical analyses.\n\nWeaknesses:\n1. In the BlockMNIST dataset design, the authors mention -- unlike the MNIST signal block that is fully predictive of the class, the non-discriminative null block contains no information about the class. The argument is not clear because CNNs (both standard and adversarially trained) learn low-level features, such as horizontal and vertical edges, in their earlier layers, and they will be identified even for the null blocks in the dataset. Hence, contributing to the actual predicting class. A sanity check would have been to remove the null blocks from the images and check whether the classification performance of the model remains unchanged.\n\n2. In the feature leakage hypothesis, the authors mention that standard models violate the gradient assumption because the discriminative signal blocks are randomly placed in the images and propose the BlockMNIST-Top dataset variant that fixed the location of the signal block. Some open questions regarding this experiment are: a) was the feature leakage due to the backpropagated gradients taken from a batch of images? One can expect that using a larger batch size will help ameliorate the effect of placing signal blocks in random locations, b) was the two models trained on BlockMNIST and BlockMNIST-Top trained using the same weight initializations?, and c) the feature leakage hypothesis contradicts that CNNs are invariant to translation, i.e., if we translate the inputs the CNN will still be able to detect the class to which the input belongs.\n\n3. Bansal et al. 2020 analyzed the differences between the vanilla gradients of standard and robust models with similar findings that input gradients of adversarially trained models are more stable than their standard counterparts. Chen et al. 2020 hypothesize that robust models learn smoother kernels as feature extractors, and hence the gradients are stable with respect to non-discriminatory signals.\n\n4. Previous works have detailed the disadvantages of replacing masked features with heuristics such as gray color, blurring, or random noise. It would be great if the authors can comment about the importance of choosing the removing operator in their analysis.\n\n[1] Chen et al., The shape and simplicity biases of adversarially robust ImageNet-trained CNNs, 2020.\n[2] Bansal et al., SAM: The sensitivity of attribution methods to hyperparameters, 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors briefly mention the works' limitations. One of the major drawbacks of this work is its scope. The authors exclusively focus on vanilla input gradients, whereas works like Bansal et al. 2020 have already studied improved gradient variants like SmoothGrad and showed their differences for standard and adversarially trained models. It would be great to show similar findings for other gradient-based explanation methods.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "bVQsiu1Rt5o", "reviewer": "Reviewer_8z9h", "summary": "This paper studies the validity of a key assumption (i.e. Assumption A) made in popular post-hoc attribution methods empirically and theoretically. Comprehensive experiments are conducted with specifically designed BlockMNIST. Moreover, the authors constructs a simplified setting for theoretical analysis to demonstrate feature leakage.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work provides a series of systematic probes like BlockMNIST into the effects of input gradients in feature attribution. With the extensive studies for evaluating the feature leakage hypothesis for violating the Assumption (A), It provides insights for several aspects like why input gradients of standard models tend to violate assumption (A) including standard and robust models. I'm taken by the way to challenge the common attribution assumptions in interpretability and it will potentially benefit the development of this line of work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A few limitations can be the restructre of the introduction section: the paper is a bit hard to follow with terms from the interpretability area and the intro part is introducing related works and case studies without a clear storyline.\n\nMoreover, the feature leakage hypothesis is not very well justified in the BlockMNIST setting.\n\nMore investigations about whether the probes in this work can be leveraged to shed light on the predictive features containing group-sensitive attributes (male vs female) should be done to improve the potential societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "ZLGfbJgMakY", "reviewer": "Reviewer_ENet", "summary": "This paper investigates whether input gradients (of robust and normally trained models) highlight discriminative features. As part of this investigation, it introduces DiffROAR (a modified version of ROAR for evaluating the accuracy of feature discriminativeness rankings), as well as the BlockMNIST dataset (plus a simplified version where they perform theoretical analysis). It introduces and discusses the idea of feature leakage as an explanation for why non-robust model gradients sometimes highlight non-discriminative features.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** Overall, I think the paper meets the originality threshold. Although DiffROAR is only a slight modification of ROAR, the paper mainly uses it as an analytic tool to investigate an important question in a novel way. Additionally, the simplified and full versions of BlockMNIST are novel.\n\n**Quality:** The submission seems technically sound, with extensive experiments and theoretical analysis. I appreciate how they use multiple distinct methods for testing whether features are truly discriminative (synthetic datasets with ground truth and real data with DiffROAR) to reach consistent and coherent conclusions.\n\n**Clarity:** The submission is well-written; every component was easy to understand, and I feel confident that its results are reproducible to readers.\n\n**Significance:** I think the results are reasonably significant, and help advance our current state of understanding on both input gradients and adversarial robustness (in a novel way). However, I do think the work can and should do more to provide a fuller understanding of its results, particular around _why_ robustness prevents feature leakage; I'll elaborate in the next section.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have a few suggestions for improvements, that could increase the impact of the paper:\n\n- The paper seems to take it as a given that input gradients _ought to_ highlight discriminative features (and certainly many have assumed that), but maybe it's worth stepping back from that even further. In the case of BlockMNIST, it makes total sense that a model might learn to independently detect the same pattern in both sides of the image, and be sensitive to changes that would create a digit in the null block (which is what input gradients actually communicate -- that the model is sensitive to counterfactual changes that would introduce a new digit, even if that's in an unexpected place). To me, these hypothetical features seem totally worthy of being called \"discriminative\"; the only real distinction is that adding them would take us off the data manifold.\n\n- Relatedly, recent research (e.g. <https://arxiv.org/pdf/1906.00945.pdf> and <https://arxiv.org/pdf/2106.10151.pdf>) suggests that adversarial training encourages model gradients and nearby decision boundaries to lie within the data manifold. In the case of BlockMNIST, that seems like it would likely prevent feature leakage (by preventing model gradients from adding phantom second digits to the null block, and instead forcing them to make more realistic changes to the signal block), and it's even more clear in the simplified 10-feature case (anything with multiple +1s or -1s in the signal block is off the data manifold). I think it's worth discussing this idea because it provides a potential explanation for _why_ adversarial robustness prevents feature leakage (which the paper defers to future work, but doesn't have to). This would really help complete the story and improve the usefulness of the work.\n\n-  While I agree with the authors that DiffROAR is a very useful tool for investigating assumption (A) in this particular context, it might be worth clarifying its limitations if others wish to reuse it in other contexts. For example, consider a redundant dataset where features are either all negative (in which case y=0) or all positive (in which case y=1). In such cases, no feature is more or less informative than any other, so no information can be gained by ranking or removing features. (It also might be worth including some results in the appendix comparing DiffROAR to vanilla ROAR, though it's not critical.)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "HmmDVbMC-jr", "reviewer": "Reviewer_TbxU", "summary": "The authors present a procedure to check the relative influence on model prediction between pixels with largest input gradient magnitudes versus pixels with lowest input gradient magnitudes. This is an extension of ROAR, which trains a model on data with certain pixels masked out. The authors evaluate DiffROAR on 4 image datasets and 1 synthetic dataset, showing that more robust models are more likely to have input gradients on relevant pixels than non-robust models. They also present a theoretical analysis showing that a standard MLP with one hidden layer can violate the assumption that higher input gradients mean more relevant pixels.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*** Originality & Significance ***: The authors do a thorough job with citing related works. The very closely-related prior work of ROAR differs in one main aspect: ROAR checks the relevance of pixels with high measured importance, and DiffROAR compares the importance of those with pixels with low measured importance. The goal of the paper seems limited, in that it seems two key claims have already been established in prior work - to quote from the abstract:\n\n\"Our results suggest that (i) input gradients of standard models (i.e., trained on original data) may grossly violate (A), whereas (ii) input gradients of adversarially robust models satisfy (A) reasonably well.\"\n\nAs supported in Line 112,  (i) is established in the related ROAR work: \n\n\"...the ROAR framework show[s] that multiple attribution methods, including vanilla input gradients, are no better than model-independent random attributions that lack explanatory power\"\n\nAnd as supported in Line 120, (ii) is established in the work of Kim et. al. [cited as 34]:\n\n\"Additionally, Kim et al. [34] use the ROAR framework to conjecture that adversarial training “tilts” input gradients to better align with the data manifold.\"\n\nThe authors write that their goal here is to augment ROAR \"to understand when and why input gradients violate assumption (A)\". This is partly addressed in their section on experiments (Section 4), by showing input gradients are more often higher magnitude on relevant pixels with robust models than in standard models. However, [34] has already addressed this in much more detail (the boundary tilting theory).\n\nOn the other hand, the theory in Section 6 seemed more significant to me. I am not as well versed in the theory literature in this area as compared to feature attribution, and cannot speak to originality very well. But, the result was very interesting - it's possible for a max-margin classifier (of a one-hidden-layer MLP) to place equal input gradient over locations where relevant features have appeared in the data, and 0 gradient on locations holding only noise. It was interesting to see this happen in BlockMNIST as well.\n\nI am recommending a score of 5, because the majority of the paper's goal seems to have been already considerably explored in past papers (i.e., addressed the exact question of the paper's title), as evidenced in the related works section. In addition, the authors' stated contribution of exploring \"when and why input gradients violate assumption (A)\" seems to have been addressed more thoroughly in the paper cited as [34].\n\n*** Clarity ***: The paper is very well-written, with only minor typos (e.g. missing period on Line 363). I thought it was enjoyable to read, and well-thought-out in its organization. The theory section was confusing at first - I didn't understand the difference between task-relevant and instance-specific, yet also task-relevant, features (e.g., the sentence on Lines 306-308 was not clear at all). Yet, Figure 5 cleared it up.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations were very briefly discussed at the end. It was surprising to me that the authors chose to solely focus on input gradients, when there are so many other existing feature attribution methods, as cited by the authors. They did mention expanding to these as future work.\n\nThere is no discussion on potential negative societal impact. One example societal connection: because the authors are pointing out a flaw in the connection between input gradients and true feature importance, input gradients should not be a trusted measure of pixel importance (so if the claim that model is using \"fair\" features is based on input gradients, that should be taken with skepticism).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for your submission to NeurIPS.  The reviewers and I are in agreement that the paper presents interesting new insights into the problem of understanding the significance of input gradients as they concern robust and nonrobust models.  The addition of the BlockMNIST data set seems particularly helpful to better understanding and furthering research in the area.  I'm happy to recommend that the paper be accepted.

Since there was a fair amount of response to reviewer feedback, I'd mainly highlight that many of the comments seemed to center around further (discussion-based, not quantitative) comparison to previous works (such as [34] and Ilyas et a, 2019 mentioned by the reviewer).  Overall I think that relating the results here more to potentially qualify phenomena observed in previous studies, at least to the extent possible without substantial revisions, would improve most upon the current paper.
Decision: Accept (Poster)