Do Input Gradients Highlight Discriminative Features? 

image classification benchmarks . Our results suggest that ( i ) input gradients of standard models ( i.e. , trained on original data ) may grossly violate ( A ) , whereas ( ii ) input gradients of adversarially robust models satisfy ( A ) reasonably well . 2 . We then introduce BlockMNIST , an MNIST-based semi-real dataset , that by design encodes a priori knowledge of discriminative features . Our analysis on BlockMNIST leverages this information to validate as well as characterize differences between input gradient attributions of standard and robust models . 3 . Finally , we theoretically prove that our empirical findings hold on a simplified version of the BlockMNIST dataset . Specifically , we prove that input gradients of standard one-hidden-layer MLPs trained on this dataset do not highlight instance-specific “ signal ” coordinates , thus grossly violating ( A ) . Our findings motivate the need to formalize and test common assumptions in interpretability in a falsifiable manner [ 3 ] . We believe that the DiffROAR framework and BlockMNIST datasets serve as sanity checks to audit interpretability methods ; code and data available at https : //github.com/harshays/inputgradients . 1 Introduction . Interpretability methods that provide instance-specific explanations of model predictions are often used to identify biased predictions [ 4 ] , debug trained models [ 5 ] , and aid decision-making in highstakes domains such as medical diagnosis [ 6 , 7 ] . A common approach for providing instance-specific explanations is feature attribution . Feature attribution methods rank or score input coordinates , or features , in the order of their purported importance in model prediction ; coordinates achieving the top-most rank or score are considered most important for prediction , whereas those with the bottom-most rank or score are considered least important . Input gradient attributions . Ranking input coordinates based on the magnitude of input gradients is a fundamental feature attribution technique [ 8 , 1 ] that undergirds well-known methods such as SmoothGrad [ 2 ] and Integrated Gradients [ 9 ] . Given instance x and a trained model θ with prediction ŷ on x , the input gradient attribution scheme ( i ) computes the input gradient∇xLogitθ ( x , ŷ ) of the logit 2 of the predicted label ŷ and ( ii ) ranks the input coordinates in decreasing order of their input gradient magnitude . Below we explicitly characterize the underlying intuitive assumption behind input gradient attribution methods : ∗Part of the work completed after joining Google Research India 2In Appendix C , we show that our results also hold for input gradients taken w.r.t . the loss 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Assumption ( A ) : Coordinates with larger input gradient magnitude are more relevant for model prediction compared to coordinates with smaller input gradient magnitude . Sanity-checking attribution methods . Several attribution methods [ 10 ] are based on input gradients and explicitly or implicitly assume an appropriately modified version of ( A ) . For example , Integrated Gradients [ 9 ] aggregate input gradients of linearly interpolated points , SmoothGrad [ 2 ] averages input gradients of points perturbed using gaussian noise , and Guided Backprop [ 11 ] modifies input gradients by zeroing out negative values at every layer during backpropagation . Surprisingly , unlike vanilla input gradients , popular methods that output attributions with better visual quality fail simple sanity checks that are indeed expected out of any valid attribution method [ 12 , 13 ] . On the other hand , while vanilla input gradients pass simple sanity checks , Hooker et al . [ 14 ] suggest that they produce estimates of feature importance that are no better than a random designation of feature importance . Do input gradients satisfy assumption ( A ) ? Since ( A ) is necessary for input gradients attributions to accurately reflect model behavior , we introduce an evaluation framework , DiffROAR , to analyze whether input gradient attributions satisfy assumption ( A ) on real-world datasets . While DiffROAR adopts the remove-and-retrain ( ROAR ) methodology [ 14 ] , DiffROAR is more appropriate for testing the validity of assumption ( A ) because it directly compares top-ranked features against bottom-ranked features . We apply DiffROAR to evaluate input gradient attributions of MLPs & CNNs trained on multiple image classification datasets . Consistent with the message in Hooker et al . [ 14 ] , our experiments indicate that input gradients of standard models ( i.e. , trained on original data ) can grossly violate ( A ) ( see Section 4 ) . Furthermore , we also observe that unlike standard models , adversarially trained models [ 15 ] that are robust to ` 2 and ` ∞ perturbations satisfy ( A ) in a consistent manner . Probing input gradient attributions using BlockMNIST . Our empirical findings mentioned above strongly suggest that standard models grossly violate ( A ) . However , without knowledge of groundtruth discriminative features learned by models trained on real data , conclusively testing ( A ) remains elusive . In fact , this is a key shortcoming of the remove-and-retrain ( ROAR ) framework . So , to further verify and better understand our empirical findings , we introduce an MNIST-based semi-real dataset , BlockMNIST , that by design encodes a priori knowledge of ground-truth discriminative features . BlockMNIST is based on the principle that for different inputs , discriminative and non-discriminative features may occur in different parts of the input . For example , in an object classification task , the object of interest can occur in different parts of the image ( e.g. , top-left , center , bottom-right etc . ) for different images . As shown in Figure 1 ( a ) , BlockMNIST images consist of a signal block and a null block that are randomly placed at the top or bottom . The signal block contains the MNIST digit that determines the class of the image , whereas the null block , contains a square patch with two diagonals that has no information about the label . This a priori knowledge of ground-truth discriminative features in BlockMNIST data allows us to ( i ) validate our empirical findings vis-a-vis input gradients of standard and robust models ( see fig . 1 ) and ( ii ) identify feature leakage as a reason that potentially explains why input gradients violate ( A ) in practice . Here , feature leakage refers to the phenomenon wherein given an instance , its input gradients highlight the location of discriminative features in the given instance as well as in other instances that are present in the dataset . For example , consider the first BlockMNIST image in fig . 1 ( a ) , in which the signal is placed in the bottom block . For this image , as shown in fig . 1 ( b , c ) , input gradients of standard models incorrectly highlight the top block because there are other instances in the BlockMNIST dataset which have signal in the top block . Rigorously demonstrating feature leakage . In order to concretely verify as well as understand feature leakage more thoroughly , we design a simplified version of BlockMNIST that is amenable to theoretical analysis . On this dataset , we first rigorously demonstrate that input gradients of standard one-hidden-layer MLPs exhibit feature leakage in the infinite-width limit and then discuss how feature leakage results in input gradient attributions that clearly violate assumption ( A ) . Paper organization : Section 2 discusses related work and section 3 presents our evaluation framework , DiffROAR , to test assumption ( A ) . Section 4 employs DiffROAR to evaluate input gradient attributions on four image classification datasets . Section 5 analyzes BlockMNIST data to differentially characterize input gradients of standard and robust models using feature leakage . Section 6 provides theoretical results on a simplified version on BlockMNIST that shed light on how feature leakage results in input gradients that violate assumption ( A ) . Our code , along with the proposed datasets , is publicly available at https : //github.com/harshays/inputgradients . 2 Related work . Due to space constraints , we only discuss directly related work and defer the rest to Appendix A . Sanity checks for explanations . Several explanation methods that provide feature attributions are often primarily evaluated using inherently subjective visual assessments [ 1 , 2 ] . Unsurprisingly , recent “ sanity checks ” show that sole reliance on visual assessment is misleading , as attributions can lack fidelity and inaccurately reflect model behavior . Adebayo et al . [ 12 ] and Kindermans et al . [ 13 ] show that unlike input gradients [ 8 ] , other popular methods—guided backprop [ 16 ] , gradient input [ 17 ] , integrated gradients [ 9 ] —output explanations which lack fidelity on image data , as they remain invariant to model and label randomization . Similarly , Yang and Kim [ 18 ] use custom image datasets to show that several explanation methods are more likely to produce false positive explanations than vanilla input gradients . Moreover , several explanation methods based on modified backpropagation do not pass basic sanity checks [ 19 , 20 , 21 ] . To summarize , well-known gradient-based attribution methods that seek to mitigate gradient saturation [ 9 , 22 ] , discontinuity [ 23 ] , and visual noise [ 16 ] surprisingly fare worse than vanilla input gradients on multiple sanity checks . Evaluating explanation fidelity . The black-box nature of neural networks necessitates frameworks that evaluate the fidelity or “ correctness ” of post-hoc explanations without knowledge of ground-truth features learned by trained models . Modification-based evaluation frameworks [ 24 , 25 , 26 ] gauge explanation fidelity by measuring the change in model performance after masking input coordinates that a given explanation method considers most ( or least ) important . However , due to distribution shifts induced by input modifications , one can not conclusively attribute changes in model performance to the fidelity of instance-specific explanations [ 27 ] . The remove-and-retrain ( ROAR ) framework [ 14 ] accounts for distribution shifts by evaluating the performance of models retrained on train data masked using post-hoc explanations . Surprisingly , contrary to findings obtained via sanity checks , experiments with the ROAR framework show that multiple attribution methods , including vanilla input gradients , are no better than model-independent random attributions that lack explanatory power [ 14 ] . Therefore , motivated by the central role of vanilla input gradients in attribution methods , we augment the ROAR framework to understand when and why input gradients violate assumption ( A ) . Effect of adversarial robustness . Adversarial training [ 15 ] not only leads to robustness to ` p adversarial attacks [ 28 ] , but also leads to perceptually-aligned feature representations [ 29 ] , and improved visual quality of input gradients [ 30 ] . Recent works hypothesize that adversarial training improves the visual quality of input gradients by suppressing irrelevant features [ 31 ] and promoting sparsity and stability [ 32 ] in explanations . Kim et al . [ 33 ] use the ROAR framework to conjecture that adversarial training “ tilts ” input gradients to better align with the data manifold . In this work , we use experiments on real-world data and theory on data with features known a priori in order to differentially characterize input gradients of standard and robust models vis-a-vis assumption ( A ) . 3 DiffROAR evaluation framework In this section , we introduce our evaluation framework , DiffROAR , to probe the extent to which instance-specific explanations , or feature attributions , highlight discriminative features in practice . Specifically , our framework , DiffROAR , builds upon the remove-and-retrain ( ROAR ) methodology [ 14 ] to test whether feature attribution methods satisfy assumption ( A ) on real-world datasets . Setting . We consider the standard classification setting ; Each data point ( x ( i ) , y ( i ) ) , where instance x ( i ) ∈ Rd and label y ( i ) ∈ Y for some label set Y , is drawn independently from a distribution D on Rd × Y . Given dataset { ( x ( i ) , y ( i ) ) } where i ∈ [ n ] : = { 1 , · · · , n } , x ( i ) j denotes the jth coordinate of x ( i ) . Note that we also refer to the d coordinates of instance x ( i ) as features interchangeably . Attribution schemes . A feature attribution scheme A : Rd → { σ : σ is a permutation of [ d ] } maps a d-dimensional instance x to a permutation , or ordering , A ( x ) : [ d ] → [ d ] of its coordinates . For example , the input gradient attribution scheme takes as input instance x & predicted label ŷ and outputs an ordering [ d ] that ranks coordinates in decreasing order of their input gradient magnitude . That is , coordinate j is ranked ahead of coordinate k if the magnitude of the jth coordinate of ∇xLogitθ ( x , ŷ ) is larger than that of the kth coordinate . Unmasking schemes . Given instance x and a subset S ⊆ [ d ] of coordinates , the unmasked instance xS zeroes out all coordinates that are not in subset S : xSj = xj if j ∈ S and 0 if j /∈ S. An unmasking scheme A : Rd → { S : S ⊆ [ d ] } simply maps instance x to a subset A ( x ) ⊆ [ d ] of coordinates that can be used to obtain unmasked instance xA ( x ) . Any attribution scheme A naturally induces top-k and bottom-k unmasking schemes , Atopk and Abotk , which output k coordinates with the top-most and bottom-most attributions in A ( x ) respectively . In other words , given attribution scheme A and level k , the top-k and bottom-k unmasking schemes , Atopk and Abotk , can be defined as follows : Atopk ( x ) : = { A ( x ) j : j ≤ k } , Abotk ( x ) : = { A ( x ) j : d− k < j ≤ d } . For example , Figure 2 depicts an image x and its top-k unmasked variant xA top k ( x ) . In this case , the attribution scheme A assigns higher rank to pixels in the foreground . So , the top25 % unmasking operation , xA top 25 % ( x ) , highlights the monkey by retaining pixels with top-25 % attribution ranks and zeroing out the remaining pixels that correspond to the green background . Predictive power of unmasking schemes . The predictive power of an unmasking scheme A with respect to model architecture M ( e.g. , resnet18 ) can be defined as the best classification accuracy that can be attained by training a model with architecture M on unmasked instances that are obtained via unmasking scheme A . More formally , it can defined as follows : PredPowerM ( A ) : = sup f∈M , f : Rd→Y ED [ 1 [ f ( xA ( x ) ) = y ] ] . Due to masking-induced distribution shifts , models with architectureM that are trained using original data can not be plugged in to estimate PredPowerM ( A ) . The ROAR framework [ 14 ] sidesteps this issue by retraining models on unmasked data , as similar model architectures tend to learn “ similar ” classifiers [ 34 , 35 , 36 , 37 ] . Therefore , we employ the ROAR framework to estimate PredPowerM ( A ) in two steps . First , we use unmasking scheme A to obtain unmasked train and test datasets that comprise data points of the form ( xA ( x ) , y ) . Then , we retrain a new model with the same architecture M on unmasked train data and evaluate its accuracy on unmasked test data . DiffROAR evaluation metric to test assumption ( A ) . Recall that an attribution scheme A maps an instance x to a permutation of its coordinates that reflects the order of estimated importance in model prediction . An attribution scheme that satisfies assumption ( A ) must place coordinates that are more important for model prediction higher up in the the attribution order . More formally , given attribution scheme A , architecture M and level k , we define DiffROAR as the difference between the predictive power of top-k and bottom-k unmasking schemes , Atopk and Abotk : DiffROARM ( A , k ) = PredPowerM ( Atopk ) − PredPowerM ( Abotk ) ( 1 ) Interpreting the DiffROAR metric . The sign of the DiffROAR metric indicates whether the given attribution scheme satisfies or violates assumption ( A ) . For example , DiffROARM ( A , · ) < 0 implies that A violates assumption ( A ) , as coordinates with higher attribution ranks have worse predictive power with respect to architecture M . Similarly , the magnitude of the DiffROAR metric quantifies the extent to which the ordering in attribution scheme A separates the most and least discriminative coordinates into two disjoint subsets . For example , a random attribution scheme Ar , which outputs attributions Ar ( x ) chosen uniformly at random from all permutations of [ d ] , neither highlights nor suppresses discriminative features ; E [ DiffROARM ( Ar , k ) ] = 0 for any architecture M . On testing assumption ( A ) . To verify ( A ) for a given attribution schemeA , it is necessary to evaluate whether input coordinates with higher attribution rank are more important for model prediction than coordinates with lower rank . Consequently , the ROAR-based metric in Hooker et al . [ 14 ] , which essentially computes the top-k predictive power , is not sufficient to test whether attribution methods satisfy assumption ( A ) . Therefore , as discussed above , DiffROAR tests ( A ) by comparing the top-k predictive power , PredPowerM ( Atopk ) , to the bottom-k predictive power , PredPowerM ( Abotk ) , using multiple values of k. 4 Testing assumption ( A ) on image classification benchmarks In this section , we use DiffROAR to evaluate whether input gradient attributions of standard and adversarially robust MLPs and CNNs trained on four image classification benchmarks satisfy assumption ( A ) . We first summarize the experiment setup and then describe key empirical findings . Datasets and models . We consider four benchmark image classification datasets : SVHN [ 38 ] , Fashion MNIST [ 39 ] , CIFAR-10 [ 40 ] and ImageNet-10 [ 41 ] . ImageNet-10 is an open-sourced variant ( https : //github.com/MadryLab/robustness/ ) of Imagenet [ 41 ] , with 80 , 000 images grouped into 10 super-classes . ImageNet-10 enables us to test assumption ( A ) on Imagenet without the computational overload of training models on the 1000-way ILSVRC classification task [ 42 ] . We evaluate input gradient attributions of standard and adversarially trained two-hidden-layer MLPs and Resnets [ 43 ] . We obtain ` 2 and ` ∞ -robust models with perturbation budget using PGD adversarial training [ 15 ] . Unless mentioned otherwise , we train models using stochastic gradient descent ( SGD ) , with momentum 0.9 , batch size 256 , ` 2 regularization 0.0005 and initial learning rate 0.1 that decays by a factor of 0.75 every 20 epochs . Additionally , we use standard data augmentation and train models for at most 500 epochs , stopping early if cross-entropy loss on training data goes below 0.001 . Appendix C.1 provides additional details about the datasets and trained models.3 Estimating DiffROAR on real data . We compute the evaluation metric , DiffROARM ( A , k ) , on real datasets in four steps , as follows . First , we train a standard or robust model with architecture M on the original dataset and obtain its input gradient attribution scheme A . Second , as outlined in Section 3 , we use attribution scheme A and level k ( i.e. , fraction of pixels to be unmasked ) to extract the top-k and bottom-k unmasking schemes : Atopk and Abotk . Third , we apply A top k and Abotk on the original train & test datasets to obtain top-k and bottom-k unmasked datasets respectively . Finally , to compute DiffROARM ( A , k ) via eq . ( 1 ) , we estimate top-k and bottom-k predictive power , PredPowerM ( Atopk ) and PredPowerM ( Abotk ) , by retraining new models with architecture M on topk and bottom-k unmasked datasets respectively . Also , note that we ( a ) average the DiffROAR metric over five runs for each model and unmasking fraction or level k and ( b ) unmask individual image pixels without grouping them channel-wise . Experiment setup . Now , we analyze the DiffROAR metric as a function of the unmasking fraction k ∈ { 5 , 10 , 20 , . . . , 100 } % in order to evaluate whether input gradient attributions of models trained on four image classification benchmarks satisfy assumption ( A ) . In particular , as shown in Figure 3 , we use DiffROAR to analyze input gradients of standard and adversarially robust two-hidden-layer MLPs on SVHN & Fashion MNIST , Resnet18 on ImageNet-10 , and Resnet50 on CIFAR-10 . In order to calibrate our findings , we compare input gradient attributions of these models to two natural baselines : model-agnostic random attributions and input-agnostic attributions of linear models . Input gradients of standard models . Input gradient attributions of standard MLPs trained on SVHN satisfy assumption ( A ) , as the DiffROAR metric in Figure 3 ( a ) is positive for all values of level k < 100 % . However , in Figure 3 ( b ) , the DiffROAR curves of standard MLPs trained on 3Code publicly available at https : //github.com/harshays/inputgradients Fashion MNIST indicate that input gradient attributions , consistent with findings in Hooker et al . [ 14 ] , can fare no better than model-agnostic random attributions and input-agnostic attributions of linear models vis-a-vis assumption ( A ) . Furthermore , and rather surprisingly , the shaded area in Figure 3 ( c ) and Figure 3 ( d ) shows that when level k < 40 % , DiffROAR curves of standard Resnets trained on CIFAR-10 and Imagenet-10 are consistently negative and perform considerably worse than model-agnostic and input-agnostic baseline attributions . These results strongly suggest that on CIFAR-10 and Imagenet-10 , input gradients of standard Resnets grossly violate assumption ( A ) and suppress discriminative features . In other words , coordinates with larger gradient magnitude have worse predictive power than coordinates with smaller gradient magnitude . Input gradients of robust models . Models that are -robust to ` 2 and ` ∞ adversarial perturbations fare considerably better than standard models on the DiffROAR metric . For example , in Figure 3 ( a ) , when level k equals 10 % , robust MLPs trained on SVHN outperform standard MLPs on the DiffROAR metric by roughly 10-30 % . The DiffROAR curves of adversarially robust MLPs in Figure 3 are positive at every level k < 100 % , which strongly suggests that input gradient attributions of robust MLPs satisfy assumption ( A ) . Similarly , robust resnet50 models trained on CIFAR-10 and ImageNet-10 satisfy assumption ( A ) reasonably well and , unlike standard resnet50 models , starkly highlight discriminative features . Furthermore , we observe that increasing the perturbation budget in ` 2 or ` ∞ PGD adversarial training [ 15 ] amplifies the magnitude of DiffROAR across k and for all four datasets . That is , the adversarial perturbation budget determines the extent to which input gradients differentiates the most and least discriminative coordinates into two disjoint subsets . Additional results . In Appendix C , we show that our DiffROAR results are robust to choice of model architecture & SGD hyperparameters during retraining and also hold for input gradients taken with respect to cross-entropy loss . Additionally , while DiffROAR without retraining gives qualitatively similar results , they are not as consistent across architectures as with retraining , particularly for small unmasking fraction k that induce non-trivial distribution shifts . 5 Analyzing input gradient attributions using BlockMNIST data To verify whether input gradients satisfy assumption ( A ) more thoroughly , we introduce and perform experiments on BlockMNIST , an MNIST-based dataset that by design encodes a priori knowledge of ground-truth discriminative features . BlockMNIST dataset design : The design of the BlockMNIST dataset is based on two intuitive properties of real-world object classification tasks : ( i ) for different images , the object of interest may appear in different parts of the image ( e.g. , top-left , bottom-right ) ; ( ii ) the object of interest and the rest of the image often share low-level patterns such as edges that are not informative of the label on their own . We replicate these aspects in BlockMNIST instances , which are vertical concatenations of two 28× 28 signal and null image blocks that are randomly placed at the top or bottom with equal probability . The signal block is an MNIST image of digit 0 or digit 1 , corresponding to class 0 or 1 of the BlockMNIST image respectively . On the other hand , the null block in every BlockMNIST image , independent of its class , contains a square patch made of two horizontal , vertical , and slanted lines , as shown in Figure 1 ( a ) . It is important to note that unlike the MNIST signal block that is fully predictive of the class , the non-discriminative null block contains no information about the class . Standard as well as adversarially robust models trained on BlockMNIST data attain 99.99 % test accuracy , thereby implying that model predictions are indeed based solely on the signal block for any given instance . We further verify this by noting that the predictions of trained model remain unchanged on almost every instance even when all pixels in the null block are set to zero . Do standard and robust models satisfy ( A ) ? As discussed above , unlike the null block that has no task-relevant information , the MNIST digit in the signal block entirely determines the class of any given BlockMNIST image . Therefore , in this setting , we can restate assumption ( A ) as follows : Do input gradient attributions highlight the signal block over the null block ? Surprisingly , as shown in Figure 1 ( b , c ) , input gradient attributions of standard MLP and Resnet18 models highlight the signal block as well as the non-discriminative null block . In stark contrast , subplots ( d ) and ( e ) in Figure 1 show that input gradient attributions of ` 2 robust MLP and Resnet18 models exclusively highlight MNIST digits in the signal block and clearly suppress the square patch in the null block . These results validate our findings on real-world datasets by showing that unlike standard models , adversarially robust models satisfy ( A ) on BlockMNIST data . Feature leakage hypothesis : Recall that the discriminative signal block in BlockMNIST images is randomly placed at the top or bottom with equal probability . Given our results in Figure 1 , we hypothesize that when discriminative features vary across instances ( e.g. , signal block at top vs. bottom ) , input gradients of standard models not only highlight instance-specific features but also leak discriminative features from other instances . We term this hypothesis feature leakage . To test our hypothesis , we leverage the modular structure in BlockMNIST to construct a slightly modified version , BlockMNIST-Top , wherein the location of the MNIST signal block is fixed at the top for all instances ( see fig . 4 ) . In this setting , in contrast to results on BlockMNIST , input gradients of standard Resnet18 and MLP models trained on BlockMNIST-Top satisfy assumption ( A ) . Specifically , when the signal block is fixed at the top , input gradient attributions in Figure 4 ( b , c ) clearly highlight the signal block and suppress the null block , thereby supporting our feature leakage hypothesis . Based on our BlockMNIST experiments , we believe that understanding how adversarial robustness mitigates feature leakage is an interesting direction for future work . Additional results . In Appendix D.1 , we ( i ) visualize input gradients of several BlockMNIST and BlockMNIST-Top images , ( ii ) introduce a quantitative proxy metric to compare feature leakage between standard and robust models , ( iii ) show that our findings are fairly robust to the choice and number of classes in BlockMNIST data , and ( iv ) evaluate feature leakage in five feature attribution methods . We also provide experiments that falsify hypotheses vis-a-vis input gradients and assumption ( A ) that we considered in addition to feature leakage .

Summary: The authors present a procedure to check the relative influence on model prediction between pixels with largest input gradient magnitudes versus pixels with lowest input gradient magnitudes. This is an extension of ROAR, which trains a model on data with certain pixels masked out. The authors evaluate DiffROAR on 4 image datasets and 1 synthetic dataset, showing that more robust models are more likely to have input gradients on relevant pixels than non-robust models. They also present a theoretical analysis showing that a standard MLP with one hidden layer can violate the assumption that higher input gradients mean more relevant pixels.