A Simple Technique to Enable Saliency Methods to Pass the Sanity Checks
Authors: Arushi Gupta
Sanjeev Arora
Venue:  (2020)
Keywords: saliency
attribution
interpretability
sanity checks

Abstract:  {\em Saliency methods} attempt to explain a deep net's decision by assigning a {\em score} to each feature/pixel in the input, often doing this credit-assignment via the gradient of the output with respect to input. 
Recently \citet{adebayosan} questioned the validity of many of these methods since they do not pass simple {\em sanity checks}, which test whether the scores shift/vanish when  layers of the trained net are randomized, or when the net is retrained using random labels for inputs. % for the inputs.   %Surprisingly, the tested methods did not pass these checks: the explanations were relatively unchanged. 

We propose a simple fix to existing saliency methods that helps them pass sanity checks, which we call {\em competition for pixels}. This involves computing saliency maps for all possible labels in the classification task, and using a simple competition among them to identify and remove less relevant pixels from the map. Some theoretical justification is provided for it  and its performance is empirically demonstrated on several popular methods.

Under review as a conference paper at ICLR 2020
A SIMPLE TECHNIQUE TO ENABLE SALIENCY METH- ODS TO PASS THE SANITY CHECKS
Anonymous authors Paper under double-blind review
ABSTRACT
Saliency methods attempt to explain deep net's decision by assigning a score each feature/pixel in the input, often doing this credim-assi via the gradient of the output with respect input. Recently Adebayo et al. (2018) questioned the validity of many of these methods since they do not pass simple sanity checks, which test whether the scores shift/vanish when layers of the trained net ran- domized, or when the net is retrained using random labels for inputs.
We propose a simple fix existing saliency methods that helps them pass sanity checks, which we call comperition for pixels. This involves computing saliency maps for all possible labels in the classification task, and using simple com- petition among them to identify and remove less relevant pixels from the map. Section B provides a theoretical justification support our method and indicates when it can be applied, and its performance is empirically demonstrated on several popular methods.
INTRODUCTION
Saliency methods attempt to explain a deep net's decision to humans by assigning score to each feature/pixel in the input, often doing this credit-assignment via the gradient of the output with respect to input (from now on refered to as just "gradient"). Here we are interested in tasks involving multiclass classification, and for simplicity the exposition will assume the input is an image. Then saliency method assigns scores to input pixels, which are presented these ideas higher-level features of the net will not be discussed here.)
heat map. (Extensions of
While gradient alone is often too noisy, it as well as related notions are the basis of other more successful methods. Gradient @ Input (Shrikumar et al.) 2017) the pixel score is the product of the corresponding coordinate of gradient vector with the pixel value. Layer-wise Relevance Propagation (LRP) (Bach et al., 2015) uses a back- propagation technique where every node in the deep net receives share of the output which it distributes nodes below it. This happens all the way to the input layer, whereby every pixel gets assigned a share of the output, which is its score. Another rule Deep-Lift (Shrikumar et al., 2016) does this in different way and is related to DAnooAncona
Shapley values of cooperative game theory. DASP Ancona et al. (2019) is a state of the art method that performs an efficient approximation of the Shapley values. The perceived limitations of these methods in turn motivated a long list of new ones. Montavon et al. (2018) provides a survey of existing methods, and brief discussion presented in Section 2
The focus of the current paper is an evaluation of saliency methods called sanity checks in Adebayo al.) (2018). This involves randomizing the model parameters or the data labels (see Section P for details). The authors show that maps produced using corrupted parameters and data are often difficult to visually distinguish from those produced using the original parameters and data. The authors concluded that "...widely deployed saliency methods independent of both the data the model was trained on, and the model parameters."
The current paper shows how to pass sanity checks via simple modification to existing methods: Competition for pixels. Section Bistivatte this idea by pointing out significant issue with previous methods: they produce saliency maps for chosen output (label) node using gradient information only for that node while ignoring the gradient information from the other (non-chosen) outputs. To incorporate information from non-chosen labels/outputs in the multiclass setting rely on an

---

Under review as a conference paper at ICLR 2020
axiom called completeness satisfied by many saliency methods, according to which the sum of pixel scores in a map is equal to the value of the chosen node (see Section 3). Existing methods design saliency maps for all outputs and the map for each label satisfies completeness. One can then view the various scores assigned to a single pixel as its "votes" for different labels. The competition idea is roughly to zero out any pixel whose vote for the chosen label was lower than for another (non- chosen) label. Section develops theory to explain why this modification helps pass sanity checks in the multi-class setting, and yet produces maps not too different from existing saliency maps. It also introduces notion called approximate completeness and suggests that it is both a reasonable alternative to completeness in practice, and also allows our analysis of the competition idea to go through. We the present new empirical finding that saliency methods that were not designed to satisfy completeness practice seem to satisfy approximate completeness anyway. This may be relevant for future research in this area.
Section reports experiments applying the competition idea to three well-regarded methods, Gra- dient Input, LRP, and DASP, and shows that they produce sensible saliency maps while also passing the sanity checks. List of testbeds and methods is largely borrowed from Adebayo et al. 2018 , except for inclusion of DASP, which draws inspiration from cooperative game theory.
RELATED WORK
Adebayo et al. (0188) and Montavon et al. (2018) provide surveys of saliency methods. Brief de- scriptions of some methods used in our experiments appear in Appendix Section Here we briefly discuss the issue most relevant to the current paper, which is the interplay between tests/evaluations of saliency methods and principled design of new methods.
Controversies here often boil down to interpretations of the word "saliency." which method designers have sought to codify via axioms. (Completeness is simple example.) Evaluations then suggest that the axioms fail to ensure other desirable properties. To give an example, does the map change significantly if we blank out or modify a portion of the image that humans find insignificant, as depicted in Kim et al. (019) ? But in such cases it is sometimes unclear if the failure is due to the method alone, or traces to other unsolved issues in deep learning related to distribution shift, domain adaptation, adversarial examples, etc.
ROAR evaluation Hooker et al. (2018) greys out pixels/features found informative by a saliency method, and retrains the classifier on these modified inputs. The method is considered low quality the accuracy drop is less than greying out the same fraction of randomly-chosen pixels. Many popular methods (including some tested here) do poorly this, and subset of ensemble methods outperform a random assignment of saliency scores. But clearly this evaluation is using a different definition of saliency than the axiomatic approaches. The axiomatic approaches seek to find a set of pixels that are sufficient to justify the output label. Since real-life images have high redundancy, multiple sets of pixels could be sufficient to justify the label. ROAR defines the goal as identifying every pixel that is potentially relevant, and then ensembling would clearly help. (To see this, imagine taking any dataset and duplicating each feature times.)
The current paper sidesteps some of these controversies by focusing solely on the sanity checks eval- uation of [Adebayo et al. (2018), and their exact testbed and framework. At first sight the problems uncovered by the samity checks appear quite serious. While the distribution-shiit based evaluations were asking "Does the map shift too much upon changes to the input that humans find insignificant?." the sanity checks asks the simpler question "Does the map fail to shift when the model/labels are dramatically changed to become nonsensical?" So it is surprising that most methods already failed the following sanity checks (the authors suggest there could be others).
The model parameter randomization test. According to the authors, this "compares the output of a saliency method on trained model with the output of the saliency method on a randomly initialized untrained network of the same architecture." " The saliency method fails if the maps are similar for trained models and randomized models. The randomization can be done in a cascading or layerwise fashion.
The data randomization test "compares a given saliency method applied model trained labeled data set with the method applied to the same model architecture but trained on copy of the data set in which we randomly permuted all labels." Clearly the model in the second case has learnt
2

---

Under review as a conference paper at ICLR 2020
no useful relationship between the data and the labels and does not generalize. The saliency method fails if the maps are similar in the two cases on test data.
To the best of our knowledge, no subsequent paper has designed reasonable saliency methods that pass the sanity checks.
3 ADDING COMPETITION
The idea of competition suggests itself naturally when one examines saliency maps produced using all possible labels/logits in multiclass problem, rather than just the chosen label. Figure I shows some Gradient Inputt maps produced using AlexNet trained on MNIST, where the first layer was modified to accept one color channel instead of 3. Notice: Many pixels found irrelevant by humans receive heat (i.e.. positive value) in all the maps, and many relevant pixels receive heat in more than one map. Our experiments showed similar phenomenon on more complicated datasets such as ImageNet. Figure I saliency maps pick up lot of information about the input itself e.g., presence of sharp edges- that may be incidental to the final classification. Furthermore, these incidental features can survive during the various randomization checks, leading to failure in the sanity check. Thus it is a natural idea to create saliency map by combining information from all labels, in the process filtering out or downgrading the importance of incidental features.
highlights an important point of Adebayo et al. (018), which is that many
Suppose the input is and the net is solving k-way classification. We assume standard softmax output layer whose inputs are k logits, one per label. Let x be an input, y be its label and by denote the corresponding logit.
Map for logit "3"
Map for logit '7"
Map for logit "S"
CGI Map
Figure I: Heatmap of Gradient C Input saliency maps produced by various logits of deep net trained on MNIST. Red denotes pixels with positive values and Blue denotes negative values. The input image is of the number 3, which is clearly visible in all maps. Note how maps computed using logits/labels "7" and " 5" assign red color (resp., blue color) to pixels that would have been expected to be present (resp., absent) in those digits. The last figure shows the map produced using our CGI method.
Usually prior methods do not combine information from maps from different labels, but we wish to do so to design a simple competition among labels for pixcls. A priori it can be unclear how to compare scores across labels, since this could end up being an "apples oranges" comparison due to potentially different scaling. However, prior work (Sundararajan et al. (017) 7 has identified property called completeness: this requires that the sum of the pixel scores is exactly the logit value. For example, Gradient @ Input satisfies completeness exactly for ReLU nets with zero bia (Recall that the ReLU function with bias is ReLU (z, a) max(= a, 0)- ) Ensuring completeness for nonzero bias requires more computationally expensive operations.
Enter competition. Completeness (and also approximate completeness, property explained be- low in Section 4.1) allows an apples-to-apples comparison of saliency scores from different labels, and to view them as "votes" for a label. Now consider the case where y is the label predicted by the net for imput Suppose pixel: has a positive score for label y and an more positive score for label y/1- This pixel contributes positively to both logit values. But remember that since label yı was not predicted by the net the label, the logit by, is less than than logit Ey. the contribution of pixel x; 's "vote" to ey is proportionately even higher than its contribution to ly. This perhaps should make realize that this pixel may be less relevant even irrelevant label y since it is effectively siding with label y1 (recall Figure l]. We conclude that looking at saliency maps for
Iff function f is computed by ReI.U net with zero bias each node, then it satisfies f()x) Af(z). Partial differentiation with respect to at X shows Vz(f)= = fzx)
3

---

Under review as a conference paper at ICLR 2020
non-chosen labels should allow us fine-tune our estimate of the relevance of pixel to the chosen label.
Now we formalize the competition idea. Note that positive and negative pixel scores should in general be interpreted differently; the former should be viewed as supporting the chosen label, and the latter as opposing that label.
Competitive version of saliency method
Input: An image, 7, an underlying saliency method, S,, and some chosen label y Output A saliency map S same dimensions as the input.
For each pixel p; in the input
1. Calculate the saliency scores assigned by each label to pixel ps by Su
2. Zero out pixel p; in the saliency map S if either
(a) Pe has a positive saliency score for label y that is not maximal among its saliency scores for all labels
(b) has negative saliency score for label that is not minimal among its saliency scores for all labels
WHY COMPETITION WORKS: SOME THEORY
Figure suggests that good idea to zero out some pixels in existing saliency maps, as the existing saliency maps are reflecting information incidental to classification, such as edges. Here we develop a more principled understanding of why adding competition (a) is aggressive enough to zero out enough pixels to help pass sanity checks randomized nets and (b) not too aggressive so as to retain a reasonable saliency map for properly trained nets.
Adebayo al. 2018) used linear models explain why methods like Gradiento Input fail their randomization tests. These tests turn the gradient into a random vector, and if & are random vectors, then æ @ & and &2 are visually quite similar when æ is an image. (See Figure 10 in their appendix.) Thus the saliency map retains strong sense of I after the randomization test, even though the gradient essentially random. Now it is immediately clear that with k-way competition among the labels, the saliency map would be expected to become almost blank in the randomization tests since each pixel is equally likely receive its highest score from each label, and thus in each saliency map the pixel becomes zero with probability competition enables the map to pass the sanity checks in multiclass settings, even for relatively small number of classes. Our experiments later show that the final map is indeed very sparse.
1/k. Thus we would expect that adding
But one cannot use the simple (Adebayo et al. 2018) model to understand why competition yields reasonable saliency map for properly trained nets. The reason being that the saliency map is not random and depends on the input -iput-since completeness property requires that the sum of pixel saliencies is the label logit. We give simple model for thinking about this dependence, which hope will motivate further theory on the best way to aggregate information from different labels. We rely upon the completeness property, although Section 4.1 shows that weaker property should suffice.
Gradient Input Scores
Mixture of two Gaussians
Figure 2: Histogram of Gradient c Input scores for VGG-19 on Imagenet versus scores produced by mixture of two gaussians.
4

---

Under review as a conference paper at ICLR 2020
Mental model for saliency maps. One should keep in mind that saliency map is the outcome of a stochastic process: training a deep net, with stochasticity coming from dataset choice, random initialization, and SGD. Let by and Mz.y be random variables denoting respectively the logit value for label y on input x and the corresponding saliency map.
For simplicity we assume the saliency method correlates with certain ground truth saliency as follows. There exists a subset S2,y of pixels (e.g., the pixels of x that a human may find salient for label y) such that for each pixel in Sa.y the corresponding coordinate of pz,y is distributed as [N,,, (f) and each pixel in Say distributed as (0, o3) where e c(z. y) and 01, 02 are constants.
This modeling assumption "noisy signal" mixed with "white noise"-maker intuitive sense and has some experimental support; see Figure where distribution of saliency scores does seem similar to mixture of two Gaussians (maybe three?), one centered zero and other at some positive value. One could consider variants of our model, e.g., allowing means and variances to differ across pixels so the maps can include certain image portions (e.g., sharp edges) with very high probability. These don't 't qualitatively affect the phenomena discussed below.
Now Pz,y I is the sum of pixel saliencies (where I the all-1's vector), which by completeness, is ly. By linearity of expectation Elyy] c 1S By measure concentration, with probability at least 1 - exp(-t? /4) we have
ly c. cS st (Iv Sa.yl Ø2v (5 25,.)
(1)
Since the sets are fairly large (say, reasonable fraction of all pixels), this concentration should be good?
After applying competition, the saliency map changes to Pz,y O Iz.g where Iz.y is the vector with in coordinates where label y wins the competition for pixels, and 0 elsewhere. Now we note various plausible conditions under which Ma,y @ Iz.y can be seen as a reasonable approximation to pa,ar For simplicity we consider how well competition preserves completeness, though obviously other properties can be analysed.
Theorem (Informal meta-theorem) For many plausible conditions on the Sz.9' 's there is scale factor y > 0 (depending on the conditions) such that the vector "f 3 Ty satisfies completeness up to small additive error.
Proof (sketch) One uses measure concentration. As an illustrative example suppose Sz,y 's are disjoint for different y's. For each coordinate in S2.y there is a certain probability p = p(c, 01 σ2) that the coordinate is nonzero in Ir.y. Thus the expected sum of coordinates in the map Maxyy O Ir.y plS and so rescaling coordinates by c/p makes them satisfy completeness, up to an additive error given by measure concentration. Qualitatively similar results are obtained when Sz.," 's are not disjoint but for Sz,y n Un S2.,, is small compared to Sa.y"
Remarks:(1) Obviously, the additive error will have smaller impact for the larger logit values than for smaller ones. (2) The above proof suggests that adding competition is akin to sampling a subset of the salient pixels. At least for image data, where pixel-level information has lot of redundancy, the essence of the original map survives, as seen in the experiments later.
4.1 APPROXIMATE COMPLETENESS
While some methods satisfy completeness by design (Bach et al., 2015), others don't. Can our theory apply to the latter class of methods? While exhaustive study was not performed, ran- domly sampling a few methods suggests that the saliency maps Ancona et al. (2019) (Shrikumar et al. 2017) in practice satisfy approximate completeness anyway (theory is of course lacking), in sense of the following definition. Figure depicts two saliency methods satisfying approximate completeness.
Clearly,, weaker but qualitatively similar results hold under changes to the assumptions, e.g. some non- Gaussian distribution for entries of pr,yi allowing them be pairwise independent instead of iid, and on.
5

---

Under review as a conference paper at ICLR 2020
Logits versus the Sum of Gradient Input Scores
I Lngits versus Sum of DASS Sceres
Gradient input summation
Summation
Figure 3: Approximate completeness property some saliency methods on VGG19, illustrated by near-linear fit of (a) Gradient @ Input on ReLU nets with nonzero bias (b) DASP
seittol mage
Figure 4: Comparison of CGI saliency maps with Gradient @ Input saliency maps. Original images are shown on the left.
Definition 2 (Approximate completeness) A saliency method satisfies (a, 3a-pppxxxxiimatee com- pleteness if sum of the pixel saliencies for label y lies in the interval (oxey 3, ad,, + 3).
Theorem (Modified meta theorem) Ifa saliency method satisfies (ox, approxxmmmee complete- ness, then under various plausible conditions on the Sz,y 's similar Theorem there are constants (a', (8)) such that after applying competition the maps satisfy (a,33 -approxximt completeness.
5 EXPERIMENTS
We consider saliency methods Gradient @ Input, LRP, and DASP, and adding competition to them get methods CGI, CLRP, and CDASP, respectively. CGI is tested using VGG-19 architecture Simonyan & Zisserman (2015) trained on (mageNer[Russakonn architecture with Imagenet: DASP using CNN model on MNIST. While these architectures are not the most powerful ones around, this is not a limitation for testing saliency methods, which are supposed work for all architectures.
et al. (2015) LRP using VGG-16
5.1 VISUAL QUALITY OF MAPS
Figure shows underlying saliency maps versus maps with our modifications for VGG-19 Im- agenet. Applying competition does not visibly degrade the map. Some more examples (labeled "original") (") also appear Figure
5.2
PARAMETER RANDOMIZATION TEST
The goal of these experiments is to determine whether modification may be applied to under- lying saliency method to pass the first sanity check in Adebayo et al, (2018). We conduct cascading and layerwise randomization described in Adebayo et al. (2018) from top bottom.

---

Under review as a conference paper at ICLR 2020
The top figure in Figure 5 shows the results of layerwise randomization Gradient @ Input. (Figure in the Appendix shows the full figure ).The text underneath each image represents which layer of the model was randomized, with the leftmost label of 'original' representing the original saliency map the fully trained model. The top panel shows the saliency maps produced by CGI, and the bottom panel the maps produces by Gradient @ Input. We find that the Gradient @ Input method displays the bird no matter which layer is randomized, and that our method immediately stops revealing the structure of the bird in the saliency maps soon as any layer is randomized. Figure i0 in the Appendix shows similar result but utilizing absolute value visualization. Notice that CGI's sensitivity to model parameters still holds.
The second figure in Sigwss the results for cascading randomization on Gradient @ Input. The rightmost column represents the original saliency map when all layer weights and biases are set to their fully trained values. The leftmost saliency map represents the map produced when only the softmax layer has been randomized. The image the right of that when everything up to and including conv5_4 has been randomized, and so on. Again we find that CGI is much more sensitive to parameter randomization than Gradient @ Input.
The bottom figure in Figure shows our results for cascading randomization on LRP. We find that competitive selection process (CLRP) benefits the LRP maps well. The LRP maps show the structure of the bird even after multiple blocks of randomization, while our maps greatly reduce the prevalence the bird structure the images.
Figure G shows our results for cascading randomization on DASP Ancona al. (2019), state of the art method that satisfies approximate completeness. DASP still shows the struc- ture of the digit even after randomization, while CDASP eliminates much of this structure.
Layerwise ise Randomization for CGI versus Gradient @ Input
Gr trput
Cascading Rendomization for CGI versus Gradient Input
CaI
Gradleet Inpua
Cascading randomization for CLRP versus LRP
CLRP
LRP
Figure 5: The first two figures depict layerwise and cascading randomization, respectively, for VGG19 model trained Imagenet for Gradient Input saliency map is almost blank when any weights are reinitialized. By contrast, we find that the origi- nal Gradient Input method displays the structure the bird, matter which layer randomized. The third figure depicts saliency map for cascading randomization on VGG -16 on Imagenet LRP versus CLRP. We notice that LRP shows the structure of the bird even after multiple blocks of randomization. CLRP eliminates much of the structure of the bird.
CGI. We find that in CGI, the
5.3 DATA RANDOMIZATION TEST
We run experiments to determine whether our saliency method is sensitive to model training. We use version of Alexnet Krizhevsky al. (012) adjusted to accept color channel instead of three

---

Under review as a conference paper at ICLR 2020
Cascading Randomization for CDASP vs DASP
original
layer corrupted
lype comupied
layer corrutted
layer com-upted layer comrubted
Figure 6: Saliency map cascading randomization on MNIST for CDASP versus DASP. We notice that DASP shows the structure of the digit even after randomization. CDASP eliminates the structure of the digit.
Data Randomization Test
3
Original
Gradient e
CDI
Figure 7: Second sanity check for Alexnet MNIST. On the middle image we find that using the original gradient times input method results in an image where the original structure of the number 3 is still visible. On the right hand side image we find that our modification removes the structure of the original input image, as we would expect for model that had been fitted on randomized data.
and train on MNIST. We randomly permute the labels in the training data set and train the model to greater than 98 % accuracy and examine the saliency maps. Figure 0 shows results. On the left hand side is the original image. In the middle is the map produced by Gradient O Input We find that the input structure, the number 3, still shows through with the Gradient Input method. On the other hand, CGI removes the underlying structure of the number.
CONCLUSION
Competition among labels is simple modification to existing saliency methods that produces saliency maps by combining information from maps from all labels, instead just the chosen label. Our modification keeps existing methods relevant for visual evaluation (as shown three well- known methods Gradient c Input, IRP, and DASP) while allowing them to pass sanity checks of Adebayo al. (2018), 8), which had called into question the validity of saliency methods. Possibly our modification even improves the quality of the map, by zero-ing out irrelevant features. We gave some theory in Section4to justify the competition idea for methods which satisfy approximate com- pleteness. Many methods satisfy completeness by design, and experimentally we find other methods satisfy approximate completeness.
We hope the simple analysis of Section 4 modeling the saliency map as "noisy signal" mixed with "white noise"-will inspire design of other saliency maps. We leave open the question of what is the optimum way to design saliency maps by combining information from all label When pixel values are spatially correlated it is natural to involve that designing the competition. This is left for future work.
The sanity checks AAeebay al. 2018) randomize the net in significant way, either by ran- domizing a layer or training on corrupted data. It is an interesting research problem to devise sanity checks that are less disruptive.
Onee idea that initially looked promising looking at gradients of outputs of the softmax layer instead of the logits did yield gond methods experiments.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp- 9505-9515, 2018.
Marco Ancona, Enea Ceolini, Cengiz Oztireli, and Markus Gross. Towards better understanding of gradient-based attribution methods for deep neural networks. In 6th International Conference on Learning Representations (ICLR 2018), 2018.
Marco Ancona, Cengiz Öztireli, and Markus Gross. Explaining deep neural networks with a poly- nomial time algorithm for shapley values approximation. In 36th International Conference Machine Learning (ICML 2019), 2019.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Kabus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer- layer-wise relevance propagation. PloS one, (0())=e0300400, 2015.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. Evaluating feature importance estimates. Third Annual Workshop Human Interpretability in Machine Learning, 2018.
Beomsu Kim, Junghoo Seo, SeungHyun Jeon, Jamyoung Koo, Jeongyeol Choe, and Taegyun Jeon. Why are saliency maps noisy? cause of and solution to noisy saliency maps. In Workshop on Interpreting and Explaining Visual Al Models, 2019.
Pieter-Jan Kindermans, Kristof Schütt, influence of noise and distractors on the interpretation of neural networks. Workshop on Inter- pretable Machine Learning in Complex Systems, 2016.
Müller, and Sven Dähne. Investigating the
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and un- derstanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1133)22112222, 2015.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. CoRR, abs/1605.00177 2016.
wttp://w..ii...o.oao.ttaaaa.aaa wwt.///////..... 1605. 0ttp://ww.....raaa......1..1..1. C1713
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Ma- chine Learning- Volume 70, pp. 3145-3153, JMLR. org, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. ICLR, 2015.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In 70, pp. 3319-
Proceedings of the 34th International Conference Machine Learning-Voll 3328. JMLR. org, 2017.
7 APPENDIX
7.1 SOME SALIENCY METHOOSS
MOVE TO APPENDIX Let S., denote the logit computed for the chosen output node of interest, y.
1. The Gradient C Input explanation: Gradient @ Input method Shrikumar et al. (2017)
computes 0s where @ is the elementwise product.

---

Under review as a conference paper at ICLR 2020
2. Integrated Gradients Integrated gradients Sundararajan al. (2017) also computes the gradient of the chosen class's ' logit. However, instead of evaluating this gradient at one fixed data point, integrated gradients consider the path integral of this value the input varies from a baseline, _. to the actual input, z: along straight line.
3. Layerwise Relevance Propagation [Bach al. (2013) proposed an approach propagat- ing importance scores called Layerwise Relevance Propagation (LRP). LRP decomposes the output the neural network into a sum of the relevances of coordinates of the input. Specifically, if neural network computes function f()) they attempt to find relevance scores R such that f(x) = Ep R
4. Taylor decomposition As stated Montavon et al. (2018) for special classes of piecewise linear functions that satisfy f(t.x) = fz() including ReLU networks with no biases, one can always find root point near the origin such that f(x) = E (x) where the relevance scores R2 (2) simplify to R2 (z)
de.
5. DeepLIFT explanation The DeepLIFT explanation Shrikumar al. (2016) calculates the importance of the input by comparing each neuron's activation to some 'reference' activa- tion. Each neuron assigned an attribution that represents the amount of difference from the baseline that that neuron is responsible for. Reference activations are determined by propagating some reference input, 7, through the neural network.
6. DASP The Deep Approximate Shapley Propagation Ancona al. (2019) state of the art method that computes the approximate Shapley values. By Proposition 2 in Anconal ct (2019) the Shapley values satisfy completeness.
Relationships between different methods Kindermans al. 2016) and Shrikumar al. 2017) showed that if modifications for numerical stability are not taken into account, the LRP rules are equivalent within scaling factor to Gradient @ Input. Ancona et al. (2018) showed that for ReLU networks (with zero baseline and no biases) the e-LRP and DeepLIFT IIF (Rescale) explanation methods are equivalent to the Gradient @ Input. DASP Ancona et al. (2019), like DeepLIFT relies on the Shapley values, but designs an efficient way to approximately compute these values.
Layerwise Randomization
Figure 8: Saliency map for layer-wise randomization of the learned weights. Diverging visualization where we plot the positive importances in red and the negative importances in blue. We find that with CGI, the saliency map almost blank when any layer is reinitialized. By contrast, we find that Gradient @ Input displays the structure of the bird, no matter which layer is randomized.
Cascading randomization for versus
versus LRP
uea Moteoed te batted Hestoned aeauond aad
Figure 9: Saliency map cascading randomization LRP versus CLRP.
10

---

Under review as a conference paper at ICLR 2020
2013
2002 2000
2000
300
050
020
1000
1233
Figure 10: Saliency map for layer-wise randomization of the learned weights. Absolute value visu- alization where we plot the absolute value of the saliency map. We find that using CGI, the saliency map is almost blank when any layer is reinitialized. By contrast, we find that Gradient @ Input displays the structure of the bird, no matter which layer is randomized.


Reviews:
[{"review_id": "SJx3NyisqS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a simple technique to address the problem introduced by Adebayo et al. that several saliency approaches do not pass sanity checks. The proposed approach computes the saliency maps for all the classes and removes the pixels that play a role in predicting several classes. \n\nStrengths:\n\n1. Simple and intuitive approach. \n2. Well written and easy to read paper.\n3. The introduced approach makes Grad.Input pass the sanity checks introduced by Adebayo et al.\n\nWeaknesses:\n\n1. For any interpretability technique, passing the sanity check is a must, but just because a saliency technique passes the sanity checks, it doesn’t mean that these maps explain the network’s decision well. \n2. Lack of any quantitative evaluation (such as localization or pointing experiment) of their approach. \n3. Failure to show if the resultant maps are class-discriminative. Show performance on images with multiple classes. \n4. In fig 1,  In Grad . Input, I see positive values or negative values even when the original pixels are not active. This doesn’t explain the presence of edges causing high values in the G.I map for such pixels, right?\n5. In figure 1, These maps only assign values to the pixels that need to be removed to make a certain classification decision. The regions that need to be active but are not present are not highlighted. \n6. In figure 1 the shown CGI Map is for which class?\n7. So, is the approach only applicable to such systems where the completeness is true? Can the authors provide a list of approach that satisfy completeness:\n8. Page 3 last paragraph: Consider the example in figure 1. Let's consider the maps for digit 3 and 5. For the top horizontal part of the digit, it plays a role in determining both 3 and 5. Assume that for one such pixel the value of h_5_i is greater thatn h_3_i (looking at the figure it is not unreasonable to expect that). Just because the g.input value of h_5_i is greater that h_3_i , are the authors saying that the top part is irrelevant?\n9. How does CGI look for the original 3 on standard model?\n10. Could the authors provide more intuition as the why the gradients of outputs from softmax layer doesn’t give good results? The proposed approach from https://arxiv.org/pdf/1908.04351.pdf suggests that computing gradients from last layer improves the class discriminative behaviour.  \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1enNy8TFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I Summary\nThe paper directly answers two sanity checks for saliency maps proposed by Adebayo et al (2018): 1. randomizing the weights of a model to prove that the input's resulting saliency map is different from a trained model' saliency map.  2. randomizing the inputs' labels to make the same proof. The authors propose a \"competitive version of saliency method\" which uses the saliency scores of every pixel for each labels and zero out: positive scored pixels that would not be maximal for the predicted class and negative scored pixels that are not minimal for the worst predicted label.\nOverall the method solves the aforementioned sanity checks, the authors claim it also generates more refined saliency maps.\n\nII Comments\n\n1. Content\nThe paper can be hard to read, due to multiple writing mistakes, abrupt phrasing, not well-articulated sentences. However, the idea is easy to understand and interesting but the contribution does not seem strong enough in its actual state. \nMy main concern is that the method seems to be designed only to answer the sanity checks: the resulting saliency maps can hardly be seen as more informative as other existing methods (eg figure 1). Quantitative measures (ROAR & KAR, Hooker et al. 2018) or surveys to show that the newly obtained saliency maps are more refined or help to best localize regions of interest would be a big bonus.\n\n2. Writing\nThe paper comports numerous typos, those do not impact the score of the review except if the sentence is not understandable. Please see the following points as support to improve the clarity of the paper.\n- Abstract last sentence: \"Some theoretical justification is provided\" -> \"Some\" is vague and makes your claim less credible -> \"theoretical justifications are given in the last paragraph to support our method...\"\n- Intro \n   paragraph 2 first sentence lack some words, l 2 product -> a product\n   \"See paper XX et al\" -> \"As in XX et al, we can see that\" or \"As stated in XX et al\", \"See\" is too familiar, formalizing the phrasing gives more credibility to your work\n- Related work\n  \"To give an example, does the map change a lot if we blank out or modify a portion of the image that humans find insignificant Kim et al. (2019)? \" This is not very well articulated, \"a lot\" is vague and a little familiar, \"significantly\" could be used here. Moreover, the citation is a little abrupt \"as we can see in XX\" would work better \nLittle typo on etc..\n\"fare best\" -> far better? The wording is still vague, it would help to add a quantitative measure\nthat's -> that is\n- Section 3\n\"This figure highlights\" -> which figure? (I think you just missed citing the fig here)\n- Section 4\nFirst sentence: Why is it a good idea? The claim is a little abrupt and could be detailed a little more\n\"destroy the saliency map\" -> destroy is a very strong word\n\"These random variables are complicated.\" -> This statement seems a little out of place and abrupt\n\"some constants\" -> \"constants\" (too vague otherwise as before)\n- Subsection 4.1\n\"randomly sampling a few such methods\" I believe there is a typo?\n\"See figure 3\" is abrupt as a sentence itself \"as you can see in figure 3 bla bla\"\nFigure 4 The image is small and hard to see on printed paper (same for the images in the appendix, they could be stacked over multiple lines instead of just one horizontal row)\nDefinition 2 punctuation at the end\n- Section 5\n\"The available code for these maps is slow, and computing even gradient for all 1000 ImageNet labels can be rather slow.\" What is the aim of this sentence?\n- subsection 5.3 \nlables -> labels\n\nIII Conclusion\nThe method itself is interesting, it would be interesting to see more qualitative results on the obtained saliency map itself: Does it produce more information? Is it more meaningful etc. Because as of now, it only seems to answer the two aforementioned sanity checks.  As for the writing, it is not always clear and can impede the understanding of the paper. I would be glad to change my review if those points are addressed.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BJg6iRPDYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a strategy for visualizing activation in networks that corresponds to features in the input layer. It addresses a problem posed for existing methods for characterizing saliency in activation subject to sanity checks which measure the degree to which the activation (saliency) map changes subject to different randomization tests.\nThe proposed solution involves a simple competition mechanism across saliency maps produced when different logits are considered such that small values are zeroed out in favor of larger values across the logits.\nOverall, I find this paper to be interesting and to address a problem worthy of further consideration. While the mechanism for competition is very simple, the resulting activation maps subject to randomization tests are reasonably convincing.\nIn the ideal case, I would have liked to see different strategies for eliciting competition explored to determine their relative merits. Nevertheless, I expect that such work will follow with this being an initial step in this direction.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes a method to pass sanity checks on saliency methods for model explainability that were proposed in a prior work.

Pros:
-The method is simple, intuitive and does indeed pass the proposed checks.

Cons:
-The proposed method aims to pass the sanity checks, but is not well-evaluated on whether it provides good explanations. Passing these checks can be considered as necessary but not sufficient.
-All reviewers agreed that the evaluation could be improved and most reviewers found the evaluation insufficient.

Given the shortcomings, AC agrees with the majority recommendation to reject.

Decision: Reject