A Simple Technique to Enable Saliency Methods to Pass the Sanity Checks

1 INTRODUCTION . Saliency methods attempt to explain a deep net ’ s decision to humans by assigning a score to each feature/pixel in the input , often doing this credit-assignment via the gradient of the output with respect to input ( from now on refered to as just “ gradient ” ) . Here we are interested in tasks involving multiclass classification , and for simplicity the exposition will assume the input is an image . Then a saliency method assigns scores to input pixels , which are presented as a heat map . ( Extensions of these ideas to higher-level features of the net will not be discussed here . ) While gradient alone is often too noisy , it as well as related notions are the basis of other more successful methods . In Gradient Input ( Shrikumar et al. , 2017 ) the pixel score is the product of the corresponding coordinate of gradient vector with the pixel value . Layer-wise Relevance Propagation ( LRP ) ( Bach et al. , 2015 ) uses a back-propagation technique where every node in the deep net receives a share of the output which it distributes to nodes below it . This happens all the way to the input layer , whereby every pixel gets assigned a share of the output , which is its score . Another rule Deep-Lift ( Shrikumar et al. , 2016 ) does this in a different way and is related to Shapley values of cooperative game theory . DASP Ancona et al . ( 2019 ) is a state of the art method that performs an efficient approximation of the Shapley values . The perceived limitations of these methods in turn motivated a long list of new ones . Montavon et al . ( 2018 ) provides a survey of existing methods , and brief discussion is presented in Section 2 . The focus of the current paper is an evaluation of saliency methods called sanity checks in Adebayo et al . ( 2018 ) . This involves randomizing the model parameters or the data labels ( see Section 2 for details ) . The authors show that maps produced using corrupted parameters and data are often difficult to visually distinguish from those produced using the original parameters and data . The authors concluded that “ ... widely deployed saliency methods are independent of both the data the model was trained on , and the model parameters. ” The current paper shows how to pass sanity checks via a simple modification to existing methods : Competition for pixels . Section 3 motivates this idea by pointing out a significant issue with previous methods : they produce saliency maps for a chosen output ( label ) node using gradient information only for that node while ignoring the gradient information from the other ( non-chosen ) outputs . To incorporate information from non-chosen labels/outputs in the multiclass setting we rely on an axiom called completeness satisfied by many saliency methods , according to which the sum of pixel scores in a map is equal to the value of the chosen node ( see Section 3 ) . Existing methods design saliency maps for all outputs and the map for each label satisfies completeness . One can then view the various scores assigned to a single pixel as its “ votes ” for different labels . The competition idea is roughly to zero out any pixel whose vote for the chosen label was lower than for another ( nonchosen ) label . Section 4 develops theory to explain why this modification helps pass sanity checks in the multi-class setting , and yet produces maps not too different from existing saliency maps . It also introduces a notion called approximate completeness and suggests that it is both a reasonable alternative to completeness in practice , and also allows our analysis of the competition idea to go through . We the present an new empirical finding that saliency methods that were not designed to satisfy completeness in practice seem to satisfy approximate completeness anyway . This may be relevant for future research in this area . Section 5 reports experiments applying the competition idea to three well-regarded methods , Gradient Input , LRP , and DASP , and shows that they produce sensible saliency maps while also passing the sanity checks . List of testbeds and methods is largely borrowed from Adebayo et al . ( 2018 ) , except for inclusion of DASP , which draws inspiration from cooperative game theory . 2 RELATED WORK . Adebayo et al . ( 2018 ) and Montavon et al . ( 2018 ) provide surveys of saliency methods . Brief descriptions of some methods used in our experiments appear in Appendix Section 7.1 . Here we briefly discuss the issue most relevant to the current paper , which is the interplay between tests/evaluations of saliency methods and principled design of new methods . Controversies here often boil down to interpretations of the word “ saliency , ” which method designers have sought to codify via axioms . ( Completeness is a simple example . ) Evaluations then suggest that the axioms fail to ensure other desirable properties . To give an example , does the map change significantly if we blank out or modify a portion of the image that humans find insignificant , as depicted in Kim et al . ( 2019 ) ? But in such cases it is sometimes unclear if the failure is due to the method alone , or traces to other unsolved issues in deep learning related to distribution shift , domain adaptation , adversarial examples , etc . ROAR evaluation Hooker et al . ( 2018 ) greys out pixels/features found informative by a saliency method , and retrains the classifier on these modified inputs . The method is considered low quality if the accuracy drop is less than greying out the same fraction of randomly-chosen pixels . Many popular methods ( including some tested here ) do poorly on this , and a subset of ensemble methods outperform a random assignment of saliency scores . But clearly this evaluation is using a different definition of saliency than the axiomatic approaches . The axiomatic approaches seek to find a set of pixels that are sufficient to justify the output label . Since real-life images have high redundancy , multiple sets of pixels could be sufficient to justify the label . ROAR defines the goal as identifying every pixel that is potentially relevant , and then ensembling would clearly help . ( To see this , imagine taking any dataset and duplicating each feature k times . ) The current paper sidesteps some of these controversies by focusing solely on the sanity checks evaluation of Adebayo et al . ( 2018 ) , and their exact testbed and framework . At first sight the problems uncovered by the sanity checks appear quite serious . While the distribution-shift based evaluations were asking “ Does the map shift too much upon changes to the input that humans find insignificant ? , ” the sanity checks asks the simpler question “ Does the map fail to shift when the model/labels are dramatically changed to become nonsensical ? ” So it is surprising that most methods already failed the following sanity checks ( the authors suggest there could be others ) . The model parameter randomization test . According to the authors , this “ compares the output of a saliency method on a trained model with the output of the saliency method on a randomly initialized untrained network of the same architecture. ” The saliency method fails if the maps are similar for trained models and randomized models . The randomization can be done in a cascading or layerwise fashion . The data randomization test “ compares a given saliency method applied to a model trained on a labeled data set with the method applied to the same model architecture but trained on a copy of the data set in which we randomly permuted all labels. ” Clearly the model in the second case has learnt no useful relationship between the data and the labels and does not generalize . The saliency method fails if the maps are similar in the two cases on test data . To the best of our knowledge , no subsequent paper has designed reasonable saliency methods that pass the sanity checks . 3 ADDING COMPETITION . The idea of competition suggests itself naturally when one examines saliency maps produced using all possible labels/logits in a multiclass problem , rather than just the chosen label . Figure 1 shows some Gradient Input maps produced using AlexNet trained on MNIST , where the first layer was modified to accept one color channel instead of 3 . Notice : Many pixels found irrelevant by humans receive heat ( i.e. , positive value ) in all the maps , and many relevant pixels receive heat in more than one map . Our experiments showed similar phenomenon on more complicated datasets such as ImageNet . Figure 1 highlights an important point of Adebayo et al . ( 2018 ) , which is that many saliency maps pick up a lot of information about the input itself —e.g. , presence of sharp edges– that may be incidental to the final classification . Furthermore , these incidental features can survive during the various randomization checks , leading to failure in the sanity check . Thus it is a natural idea to create a saliency map by combining information from all labels , in the process filtering out or downgrading the importance of incidental features . Suppose the input is x and the net is solving a k-way classification . We assume a standard softmax output layer whose inputs are k logits , one per label . Let x be an input , y be its label and ` y denote the corresponding logit . Usually prior methods do not combine information from maps from different labels , but we wish to do so to design a simple competition among labels for pixels . A priori it can be unclear how to compare scores across labels , since this could end up being an “ apples vs oranges ” comparison due to potentially different scaling . However , prior work Sundararajan et al . ( 2017 ) has identified a property called completeness : this requires that the sum of the pixel scores is exactly the logit value . For example , Gradient Input satisfies completeness exactly for ReLU nets with zero bias1 . ( Recall that the ReLU function with bias a is ReLU ( z , a ) max { z − a , 0 } . ) Ensuring completeness for nonzero bias requires more computationally expensive operations . Enter competition . Completeness ( and also approximate completeness , a property explained below in Section 4.1 ) allows an apples-to-apples comparison of saliency scores from different labels , and to view them as “ votes ” for a label . Now consider the case where y is the label predicted by the net for input x . Suppose pixel i has a positive score for label y and an even more positive score for label y1 . This pixel contributes positively to both logit values . But remember that since label y1 was not predicted by the net as the label , the logit ` y1 is less than than logit ` y , so the contribution of pixel xi ’ s “ vote ” to ` y1 is proportionately even higher than its contribution to ` y . This perhaps should make us realize that this pixel may be less relevant or even irrelevant to label y since it is effectively siding with label y1 ( recall Figure 1 ) . We conclude that looking at saliency maps for 1If function f is computed by a ReLU net with zero bias at each node , then it satisfies f ( λx ) = λf ( x ) . Partial differentiation with respect to λ at λ = 1 shows x · ∇x ( f ) = f ( x ) . non-chosen labels should allow us to fine-tune our estimate of the relevance of a pixel to the chosen label . Now we formalize the competition idea . Note that positive and negative pixel scores should in general be interpreted differently ; the former should be viewed as supporting the chosen label , and the latter as opposing that label . Competitive version of saliency method Input : An image , I , an underlying saliency method , Su and some chosen label y Output : A saliency map S of same dimensions as the input , I For each pixel pi in the input 1 . Calculate the saliency scores assigned by each label to pixel pi by Su 2 . Zero out pixel pi in the saliency map S if either ( a ) pi has a positive saliency score for label y that is not maximal among its saliency scores for all labels ( b ) pi has a negative saliency score for label y that is not minimal among its saliency scores for all labels

Summary: The paper directly answers two sanity checks for saliency maps proposed by Adebayo et al (2018): 1. randomizing the weights of a model to prove that the input's resulting saliency map is different from a trained model' saliency map.  2. randomizing the inputs' labels to make the same proof. The authors propose a "competitive version of saliency method" which uses the saliency scores of every pixel for each labels and zero out: positive scored pixels that would not be maximal for the predicted class and negative scored pixels that are not minimal for the worst predicted label.