Do Input Gradients Highlight Discriminative Features? 

image classification benchmarks . Our results suggest that ( i ) input gradients of standard models ( i.e. , trained on original data ) may grossly violate ( A ) , whereas ( ii ) input gradients of adversarially robust models satisfy ( A ) reasonably well . 2 . We then introduce BlockMNIST , an MNIST-based semi-real dataset , that by design encodes a priori knowledge of discriminative features . Our analysis on BlockMNIST leverages this information to validate as well as characterize differences between input gradient attributions of standard and robust models . 3 . Finally , we theoretically prove that our empirical findings hold on a simplified version of the BlockMNIST dataset . Specifically , we prove that input gradients of standard one-hidden-layer MLPs trained on this dataset do not highlight instance-specific “ signal ” coordinates , thus grossly violating ( A ) . Our findings motivate the need to formalize and test common assumptions in interpretability in a falsifiable manner [ 3 ] . We believe that the DiffROAR framework and BlockMNIST datasets serve as sanity checks to audit interpretability methods ; code and data available at https : //github.com/harshays/inputgradients . 1 Introduction . Interpretability methods that provide instance-specific explanations of model predictions are often used to identify biased predictions [ 4 ] , debug trained models [ 5 ] , and aid decision-making in highstakes domains such as medical diagnosis [ 6 , 7 ] . A common approach for providing instance-specific explanations is feature attribution . Feature attribution methods rank or score input coordinates , or features , in the order of their purported importance in model prediction ; coordinates achieving the top-most rank or score are considered most important for prediction , whereas those with the bottom-most rank or score are considered least important . Input gradient attributions . Ranking input coordinates based on the magnitude of input gradients is a fundamental feature attribution technique [ 8 , 1 ] that undergirds well-known methods such as SmoothGrad [ 2 ] and Integrated Gradients [ 9 ] . Given instance x and a trained model θ with prediction ŷ on x , the input gradient attribution scheme ( i ) computes the input gradient∇xLogitθ ( x , ŷ ) of the logit 2 of the predicted label ŷ and ( ii ) ranks the input coordinates in decreasing order of their input gradient magnitude . Below we explicitly characterize the underlying intuitive assumption behind input gradient attribution methods : ∗Part of the work completed after joining Google Research India 2In Appendix C , we show that our results also hold for input gradients taken w.r.t . the loss 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Assumption ( A ) : Coordinates with larger input gradient magnitude are more relevant for model prediction compared to coordinates with smaller input gradient magnitude . Sanity-checking attribution methods . Several attribution methods [ 10 ] are based on input gradients and explicitly or implicitly assume an appropriately modified version of ( A ) . For example , Integrated Gradients [ 9 ] aggregate input gradients of linearly interpolated points , SmoothGrad [ 2 ] averages input gradients of points perturbed using gaussian noise , and Guided Backprop [ 11 ] modifies input gradients by zeroing out negative values at every layer during backpropagation . Surprisingly , unlike vanilla input gradients , popular methods that output attributions with better visual quality fail simple sanity checks that are indeed expected out of any valid attribution method [ 12 , 13 ] . On the other hand , while vanilla input gradients pass simple sanity checks , Hooker et al . [ 14 ] suggest that they produce estimates of feature importance that are no better than a random designation of feature importance . Do input gradients satisfy assumption ( A ) ? Since ( A ) is necessary for input gradients attributions to accurately reflect model behavior , we introduce an evaluation framework , DiffROAR , to analyze whether input gradient attributions satisfy assumption ( A ) on real-world datasets . While DiffROAR adopts the remove-and-retrain ( ROAR ) methodology [ 14 ] , DiffROAR is more appropriate for testing the validity of assumption ( A ) because it directly compares top-ranked features against bottom-ranked features . We apply DiffROAR to evaluate input gradient attributions of MLPs & CNNs trained on multiple image classification datasets . Consistent with the message in Hooker et al . [ 14 ] , our experiments indicate that input gradients of standard models ( i.e. , trained on original data ) can grossly violate ( A ) ( see Section 4 ) . Furthermore , we also observe that unlike standard models , adversarially trained models [ 15 ] that are robust to ` 2 and ` ∞ perturbations satisfy ( A ) in a consistent manner . Probing input gradient attributions using BlockMNIST . Our empirical findings mentioned above strongly suggest that standard models grossly violate ( A ) . However , without knowledge of groundtruth discriminative features learned by models trained on real data , conclusively testing ( A ) remains elusive . In fact , this is a key shortcoming of the remove-and-retrain ( ROAR ) framework . So , to further verify and better understand our empirical findings , we introduce an MNIST-based semi-real dataset , BlockMNIST , that by design encodes a priori knowledge of ground-truth discriminative features . BlockMNIST is based on the principle that for different inputs , discriminative and non-discriminative features may occur in different parts of the input . For example , in an object classification task , the object of interest can occur in different parts of the image ( e.g. , top-left , center , bottom-right etc . ) for different images . As shown in Figure 1 ( a ) , BlockMNIST images consist of a signal block and a null block that are randomly placed at the top or bottom . The signal block contains the MNIST digit that determines the class of the image , whereas the null block , contains a square patch with two diagonals that has no information about the label . This a priori knowledge of ground-truth discriminative features in BlockMNIST data allows us to ( i ) validate our empirical findings vis-a-vis input gradients of standard and robust models ( see fig . 1 ) and ( ii ) identify feature leakage as a reason that potentially explains why input gradients violate ( A ) in practice . Here , feature leakage refers to the phenomenon wherein given an instance , its input gradients highlight the location of discriminative features in the given instance as well as in other instances that are present in the dataset . For example , consider the first BlockMNIST image in fig . 1 ( a ) , in which the signal is placed in the bottom block . For this image , as shown in fig . 1 ( b , c ) , input gradients of standard models incorrectly highlight the top block because there are other instances in the BlockMNIST dataset which have signal in the top block . Rigorously demonstrating feature leakage . In order to concretely verify as well as understand feature leakage more thoroughly , we design a simplified version of BlockMNIST that is amenable to theoretical analysis . On this dataset , we first rigorously demonstrate that input gradients of standard one-hidden-layer MLPs exhibit feature leakage in the infinite-width limit and then discuss how feature leakage results in input gradient attributions that clearly violate assumption ( A ) . Paper organization : Section 2 discusses related work and section 3 presents our evaluation framework , DiffROAR , to test assumption ( A ) . Section 4 employs DiffROAR to evaluate input gradient attributions on four image classification datasets . Section 5 analyzes BlockMNIST data to differentially characterize input gradients of standard and robust models using feature leakage . Section 6 provides theoretical results on a simplified version on BlockMNIST that shed light on how feature leakage results in input gradients that violate assumption ( A ) . Our code , along with the proposed datasets , is publicly available at https : //github.com/harshays/inputgradients . 2 Related work . Due to space constraints , we only discuss directly related work and defer the rest to Appendix A . Sanity checks for explanations . Several explanation methods that provide feature attributions are often primarily evaluated using inherently subjective visual assessments [ 1 , 2 ] . Unsurprisingly , recent “ sanity checks ” show that sole reliance on visual assessment is misleading , as attributions can lack fidelity and inaccurately reflect model behavior . Adebayo et al . [ 12 ] and Kindermans et al . [ 13 ] show that unlike input gradients [ 8 ] , other popular methods—guided backprop [ 16 ] , gradient input [ 17 ] , integrated gradients [ 9 ] —output explanations which lack fidelity on image data , as they remain invariant to model and label randomization . Similarly , Yang and Kim [ 18 ] use custom image datasets to show that several explanation methods are more likely to produce false positive explanations than vanilla input gradients . Moreover , several explanation methods based on modified backpropagation do not pass basic sanity checks [ 19 , 20 , 21 ] . To summarize , well-known gradient-based attribution methods that seek to mitigate gradient saturation [ 9 , 22 ] , discontinuity [ 23 ] , and visual noise [ 16 ] surprisingly fare worse than vanilla input gradients on multiple sanity checks . Evaluating explanation fidelity . The black-box nature of neural networks necessitates frameworks that evaluate the fidelity or “ correctness ” of post-hoc explanations without knowledge of ground-truth features learned by trained models . Modification-based evaluation frameworks [ 24 , 25 , 26 ] gauge explanation fidelity by measuring the change in model performance after masking input coordinates that a given explanation method considers most ( or least ) important . However , due to distribution shifts induced by input modifications , one can not conclusively attribute changes in model performance to the fidelity of instance-specific explanations [ 27 ] . The remove-and-retrain ( ROAR ) framework [ 14 ] accounts for distribution shifts by evaluating the performance of models retrained on train data masked using post-hoc explanations . Surprisingly , contrary to findings obtained via sanity checks , experiments with the ROAR framework show that multiple attribution methods , including vanilla input gradients , are no better than model-independent random attributions that lack explanatory power [ 14 ] . Therefore , motivated by the central role of vanilla input gradients in attribution methods , we augment the ROAR framework to understand when and why input gradients violate assumption ( A ) . Effect of adversarial robustness . Adversarial training [ 15 ] not only leads to robustness to ` p adversarial attacks [ 28 ] , but also leads to perceptually-aligned feature representations [ 29 ] , and improved visual quality of input gradients [ 30 ] . Recent works hypothesize that adversarial training improves the visual quality of input gradients by suppressing irrelevant features [ 31 ] and promoting sparsity and stability [ 32 ] in explanations . Kim et al . [ 33 ] use the ROAR framework to conjecture that adversarial training “ tilts ” input gradients to better align with the data manifold . In this work , we use experiments on real-world data and theory on data with features known a priori in order to differentially characterize input gradients of standard and robust models vis-a-vis assumption ( A ) . 3 DiffROAR evaluation framework In this section , we introduce our evaluation framework , DiffROAR , to probe the extent to which instance-specific explanations , or feature attributions , highlight discriminative features in practice . Specifically , our framework , DiffROAR , builds upon the remove-and-retrain ( ROAR ) methodology [ 14 ] to test whether feature attribution methods satisfy assumption ( A ) on real-world datasets . Setting . We consider the standard classification setting ; Each data point ( x ( i ) , y ( i ) ) , where instance x ( i ) ∈ Rd and label y ( i ) ∈ Y for some label set Y , is drawn independently from a distribution D on Rd × Y . Given dataset { ( x ( i ) , y ( i ) ) } where i ∈ [ n ] : = { 1 , · · · , n } , x ( i ) j denotes the jth coordinate of x ( i ) . Note that we also refer to the d coordinates of instance x ( i ) as features interchangeably . Attribution schemes . A feature attribution scheme A : Rd → { σ : σ is a permutation of [ d ] } maps a d-dimensional instance x to a permutation , or ordering , A ( x ) : [ d ] → [ d ] of its coordinates . For example , the input gradient attribution scheme takes as input instance x & predicted label ŷ and outputs an ordering [ d ] that ranks coordinates in decreasing order of their input gradient magnitude . That is , coordinate j is ranked ahead of coordinate k if the magnitude of the jth coordinate of ∇xLogitθ ( x , ŷ ) is larger than that of the kth coordinate . Unmasking schemes . Given instance x and a subset S ⊆ [ d ] of coordinates , the unmasked instance xS zeroes out all coordinates that are not in subset S : xSj = xj if j ∈ S and 0 if j /∈ S. An unmasking scheme A : Rd → { S : S ⊆ [ d ] } simply maps instance x to a subset A ( x ) ⊆ [ d ] of coordinates that can be used to obtain unmasked instance xA ( x ) . Any attribution scheme A naturally induces top-k and bottom-k unmasking schemes , Atopk and Abotk , which output k coordinates with the top-most and bottom-most attributions in A ( x ) respectively . In other words , given attribution scheme A and level k , the top-k and bottom-k unmasking schemes , Atopk and Abotk , can be defined as follows : Atopk ( x ) : = { A ( x ) j : j ≤ k } , Abotk ( x ) : = { A ( x ) j : d− k < j ≤ d } . For example , Figure 2 depicts an image x and its top-k unmasked variant xA top k ( x ) . In this case , the attribution scheme A assigns higher rank to pixels in the foreground . So , the top25 % unmasking operation , xA top 25 % ( x ) , highlights the monkey by retaining pixels with top-25 % attribution ranks and zeroing out the remaining pixels that correspond to the green background . Predictive power of unmasking schemes . The predictive power of an unmasking scheme A with respect to model architecture M ( e.g. , resnet18 ) can be defined as the best classification accuracy that can be attained by training a model with architecture M on unmasked instances that are obtained via unmasking scheme A . More formally , it can defined as follows : PredPowerM ( A ) : = sup f∈M , f : Rd→Y ED [ 1 [ f ( xA ( x ) ) = y ] ] . Due to masking-induced distribution shifts , models with architectureM that are trained using original data can not be plugged in to estimate PredPowerM ( A ) . The ROAR framework [ 14 ] sidesteps this issue by retraining models on unmasked data , as similar model architectures tend to learn “ similar ” classifiers [ 34 , 35 , 36 , 37 ] . Therefore , we employ the ROAR framework to estimate PredPowerM ( A ) in two steps . First , we use unmasking scheme A to obtain unmasked train and test datasets that comprise data points of the form ( xA ( x ) , y ) . Then , we retrain a new model with the same architecture M on unmasked train data and evaluate its accuracy on unmasked test data . DiffROAR evaluation metric to test assumption ( A ) . Recall that an attribution scheme A maps an instance x to a permutation of its coordinates that reflects the order of estimated importance in model prediction . An attribution scheme that satisfies assumption ( A ) must place coordinates that are more important for model prediction higher up in the the attribution order . More formally , given attribution scheme A , architecture M and level k , we define DiffROAR as the difference between the predictive power of top-k and bottom-k unmasking schemes , Atopk and Abotk : DiffROARM ( A , k ) = PredPowerM ( Atopk ) − PredPowerM ( Abotk ) ( 1 ) Interpreting the DiffROAR metric . The sign of the DiffROAR metric indicates whether the given attribution scheme satisfies or violates assumption ( A ) . For example , DiffROARM ( A , · ) < 0 implies that A violates assumption ( A ) , as coordinates with higher attribution ranks have worse predictive power with respect to architecture M . Similarly , the magnitude of the DiffROAR metric quantifies the extent to which the ordering in attribution scheme A separates the most and least discriminative coordinates into two disjoint subsets . For example , a random attribution scheme Ar , which outputs attributions Ar ( x ) chosen uniformly at random from all permutations of [ d ] , neither highlights nor suppresses discriminative features ; E [ DiffROARM ( Ar , k ) ] = 0 for any architecture M . On testing assumption ( A ) . To verify ( A ) for a given attribution schemeA , it is necessary to evaluate whether input coordinates with higher attribution rank are more important for model prediction than coordinates with lower rank . Consequently , the ROAR-based metric in Hooker et al . [ 14 ] , which essentially computes the top-k predictive power , is not sufficient to test whether attribution methods satisfy assumption ( A ) . Therefore , as discussed above , DiffROAR tests ( A ) by comparing the top-k predictive power , PredPowerM ( Atopk ) , to the bottom-k predictive power , PredPowerM ( Abotk ) , using multiple values of k. 4 Testing assumption ( A ) on image classification benchmarks In this section , we use DiffROAR to evaluate whether input gradient attributions of standard and adversarially robust MLPs and CNNs trained on four image classification benchmarks satisfy assumption ( A ) . We first summarize the experiment setup and then describe key empirical findings . Datasets and models . We consider four benchmark image classification datasets : SVHN [ 38 ] , Fashion MNIST [ 39 ] , CIFAR-10 [ 40 ] and ImageNet-10 [ 41 ] . ImageNet-10 is an open-sourced variant ( https : //github.com/MadryLab/robustness/ ) of Imagenet [ 41 ] , with 80 , 000 images grouped into 10 super-classes . ImageNet-10 enables us to test assumption ( A ) on Imagenet without the computational overload of training models on the 1000-way ILSVRC classification task [ 42 ] . We evaluate input gradient attributions of standard and adversarially trained two-hidden-layer MLPs and Resnets [ 43 ] . We obtain ` 2 and ` ∞ -robust models with perturbation budget using PGD adversarial training [ 15 ] . Unless mentioned otherwise , we train models using stochastic gradient descent ( SGD ) , with momentum 0.9 , batch size 256 , ` 2 regularization 0.0005 and initial learning rate 0.1 that decays by a factor of 0.75 every 20 epochs . Additionally , we use standard data augmentation and train models for at most 500 epochs , stopping early if cross-entropy loss on training data goes below 0.001 . Appendix C.1 provides additional details about the datasets and trained models.3 Estimating DiffROAR on real data . We compute the evaluation metric , DiffROARM ( A , k ) , on real datasets in four steps , as follows . First , we train a standard or robust model with architecture M on the original dataset and obtain its input gradient attribution scheme A . Second , as outlined in Section 3 , we use attribution scheme A and level k ( i.e. , fraction of pixels to be unmasked ) to extract the top-k and bottom-k unmasking schemes : Atopk and Abotk . Third , we apply A top k and Abotk on the original train & test datasets to obtain top-k and bottom-k unmasked datasets respectively . Finally , to compute DiffROARM ( A , k ) via eq . ( 1 ) , we estimate top-k and bottom-k predictive power , PredPowerM ( Atopk ) and PredPowerM ( Abotk ) , by retraining new models with architecture M on topk and bottom-k unmasked datasets respectively . Also , note that we ( a ) average the DiffROAR metric over five runs for each model and unmasking fraction or level k and ( b ) unmask individual image pixels without grouping them channel-wise . Experiment setup . Now , we analyze the DiffROAR metric as a function of the unmasking fraction k ∈ { 5 , 10 , 20 , . . . , 100 } % in order to evaluate whether input gradient attributions of models trained on four image classification benchmarks satisfy assumption ( A ) . In particular , as shown in Figure 3 , we use DiffROAR to analyze input gradients of standard and adversarially robust two-hidden-layer MLPs on SVHN & Fashion MNIST , Resnet18 on ImageNet-10 , and Resnet50 on CIFAR-10 . In order to calibrate our findings , we compare input gradient attributions of these models to two natural baselines : model-agnostic random attributions and input-agnostic attributions of linear models . Input gradients of standard models . Input gradient attributions of standard MLPs trained on SVHN satisfy assumption ( A ) , as the DiffROAR metric in Figure 3 ( a ) is positive for all values of level k < 100 % . However , in Figure 3 ( b ) , the DiffROAR curves of standard MLPs trained on 3Code publicly available at https : //github.com/harshays/inputgradients Fashion MNIST indicate that input gradient attributions , consistent with findings in Hooker et al . [ 14 ] , can fare no better than model-agnostic random attributions and input-agnostic attributions of linear models vis-a-vis assumption ( A ) . Furthermore , and rather surprisingly , the shaded area in Figure 3 ( c ) and Figure 3 ( d ) shows that when level k < 40 % , DiffROAR curves of standard Resnets trained on CIFAR-10 and Imagenet-10 are consistently negative and perform considerably worse than model-agnostic and input-agnostic baseline attributions . These results strongly suggest that on CIFAR-10 and Imagenet-10 , input gradients of standard Resnets grossly violate assumption ( A ) and suppress discriminative features . In other words , coordinates with larger gradient magnitude have worse predictive power than coordinates with smaller gradient magnitude . Input gradients of robust models . Models that are -robust to ` 2 and ` ∞ adversarial perturbations fare considerably better than standard models on the DiffROAR metric . For example , in Figure 3 ( a ) , when level k equals 10 % , robust MLPs trained on SVHN outperform standard MLPs on the DiffROAR metric by roughly 10-30 % . The DiffROAR curves of adversarially robust MLPs in Figure 3 are positive at every level k < 100 % , which strongly suggests that input gradient attributions of robust MLPs satisfy assumption ( A ) . Similarly , robust resnet50 models trained on CIFAR-10 and ImageNet-10 satisfy assumption ( A ) reasonably well and , unlike standard resnet50 models , starkly highlight discriminative features . Furthermore , we observe that increasing the perturbation budget in ` 2 or ` ∞ PGD adversarial training [ 15 ] amplifies the magnitude of DiffROAR across k and for all four datasets . That is , the adversarial perturbation budget determines the extent to which input gradients differentiates the most and least discriminative coordinates into two disjoint subsets . Additional results . In Appendix C , we show that our DiffROAR results are robust to choice of model architecture & SGD hyperparameters during retraining and also hold for input gradients taken with respect to cross-entropy loss . Additionally , while DiffROAR without retraining gives qualitatively similar results , they are not as consistent across architectures as with retraining , particularly for small unmasking fraction k that induce non-trivial distribution shifts . 5 Analyzing input gradient attributions using BlockMNIST data To verify whether input gradients satisfy assumption ( A ) more thoroughly , we introduce and perform experiments on BlockMNIST , an MNIST-based dataset that by design encodes a priori knowledge of ground-truth discriminative features . BlockMNIST dataset design : The design of the BlockMNIST dataset is based on two intuitive properties of real-world object classification tasks : ( i ) for different images , the object of interest may appear in different parts of the image ( e.g. , top-left , bottom-right ) ; ( ii ) the object of interest and the rest of the image often share low-level patterns such as edges that are not informative of the label on their own . We replicate these aspects in BlockMNIST instances , which are vertical concatenations of two 28× 28 signal and null image blocks that are randomly placed at the top or bottom with equal probability . The signal block is an MNIST image of digit 0 or digit 1 , corresponding to class 0 or 1 of the BlockMNIST image respectively . On the other hand , the null block in every BlockMNIST image , independent of its class , contains a square patch made of two horizontal , vertical , and slanted lines , as shown in Figure 1 ( a ) . It is important to note that unlike the MNIST signal block that is fully predictive of the class , the non-discriminative null block contains no information about the class . Standard as well as adversarially robust models trained on BlockMNIST data attain 99.99 % test accuracy , thereby implying that model predictions are indeed based solely on the signal block for any given instance . We further verify this by noting that the predictions of trained model remain unchanged on almost every instance even when all pixels in the null block are set to zero . Do standard and robust models satisfy ( A ) ? As discussed above , unlike the null block that has no task-relevant information , the MNIST digit in the signal block entirely determines the class of any given BlockMNIST image . Therefore , in this setting , we can restate assumption ( A ) as follows : Do input gradient attributions highlight the signal block over the null block ? Surprisingly , as shown in Figure 1 ( b , c ) , input gradient attributions of standard MLP and Resnet18 models highlight the signal block as well as the non-discriminative null block . In stark contrast , subplots ( d ) and ( e ) in Figure 1 show that input gradient attributions of ` 2 robust MLP and Resnet18 models exclusively highlight MNIST digits in the signal block and clearly suppress the square patch in the null block . These results validate our findings on real-world datasets by showing that unlike standard models , adversarially robust models satisfy ( A ) on BlockMNIST data . Feature leakage hypothesis : Recall that the discriminative signal block in BlockMNIST images is randomly placed at the top or bottom with equal probability . Given our results in Figure 1 , we hypothesize that when discriminative features vary across instances ( e.g. , signal block at top vs. bottom ) , input gradients of standard models not only highlight instance-specific features but also leak discriminative features from other instances . We term this hypothesis feature leakage . To test our hypothesis , we leverage the modular structure in BlockMNIST to construct a slightly modified version , BlockMNIST-Top , wherein the location of the MNIST signal block is fixed at the top for all instances ( see fig . 4 ) . In this setting , in contrast to results on BlockMNIST , input gradients of standard Resnet18 and MLP models trained on BlockMNIST-Top satisfy assumption ( A ) . Specifically , when the signal block is fixed at the top , input gradient attributions in Figure 4 ( b , c ) clearly highlight the signal block and suppress the null block , thereby supporting our feature leakage hypothesis . Based on our BlockMNIST experiments , we believe that understanding how adversarial robustness mitigates feature leakage is an interesting direction for future work . Additional results . In Appendix D.1 , we ( i ) visualize input gradients of several BlockMNIST and BlockMNIST-Top images , ( ii ) introduce a quantitative proxy metric to compare feature leakage between standard and robust models , ( iii ) show that our findings are fairly robust to the choice and number of classes in BlockMNIST data , and ( iv ) evaluate feature leakage in five feature attribution methods . We also provide experiments that falsify hypotheses vis-a-vis input gradients and assumption ( A ) that we considered in addition to feature leakage .

Summary: This paper studies the validity of a key assumption (i.e. Assumption A) made in popular post-hoc attribution methods empirically and theoretically. Comprehensive experiments are conducted with specifically designed BlockMNIST. Moreover, the authors constructs a simplified setting for theoretical analysis to demonstrate feature leakage.