New Definitions and Evaluations for Saliency Methods: Staying Intrinsic and Sound
Authors: Arushi Gupta
Nikunj Saunshi
Dingli Yu
Kaifeng Lyu
Sanjeev Arora
Venue: ICLR 2022 Submitted (2022)
Keywords: saliency
masking based methods

Abstract:   Saliency methods seek to provide human-interpretable explanations for the output of machine learning model on a given input. A plethora of saliency methods exist, as well as an extensive literature on their justifications/criticisms/evaluations. This paper focuses on heat maps based saliency methods that often provide explanations that look best to humans. It tries to introduce methods and evaluations for masked-based saliency methods that are {\em intrinsic} --- use just the training dataset and the trained net, and do not use separately trained nets, distractor distributions, human evaluations or annotations. Since a mask can be seen as a "certificate" justifying the net's answer, we introduce notions of {\em completeness} and {\em soundness} (the latter being the new contribution) motivated by logical proof systems. These notions allow a new evaluation of  saliency methods, that experimentally provides a novel and stronger justification for several heuristic tricks in the field (T.V. regularization, upscaling). 

Under review as a conference paper at ICLR 2022
NEW DEFINITIONS AND EVALUATIONS FOR SALIENCY METHODS: STAYING INTRINSIC AND SOUND
Anonymous authors Paper under double-blind review
ABSTRACT
Saliency methods seek to provide human- numan-interpretable explanations for the output of machine learning model on a given input. plethora of saliency methods ex- ist, as well an extensive literature
their snfffiaattonscriiiissssevaaatton This paper focuses on heat maps based saliency methods that often provide expla- nations that look best humans. It tries introduce methods and evaluations for masked- based saliency methods that are intrinsic - use just the training dataset and the trained net, and do not use separately trained nets, distractor distributions, human evaluations or annotations. Since a mask can be seen as a "certificate" justifying the net's answer, we introduce notions of completeness and soundness (the latter being the new contribution) motivated by logical proof systems. These notions allow a new evaluation of saliency methods, that experimentally provides a novel and stronger justification for several heuristic tricks in the field (T.V. reg- ularization, upscaling).
INTRODUCTION
Why did the deep net give a certain answer on a particular input, and can we trust the answer? Saliency methods try to provide explanations, and thus of great interest from viewpoint of human explainability, fairness, robustness, etc. This paper restricts attention to methods that return an importance score for each coordinate of the input- ussually visualized as a heat map- that captures its importance to the final decision!" | We refer the reader to Samek et al. (2019) for an extensive survey and Section for short account of such methods and controversies.
There are two important components in the research on saliency: saliency methods that produce such heat maps for explanations, and saliency evaluation metrics that aim test and compare saliency methods. Numerous saliency methods have been proposed to learn such heat maps. Some methods learn maps through "credit attribution" " to individual input coordinates by using methods reminiscent of backpropagation (Binder et al.) 2016 [Selvaraju et al., 2019),, while some are derived using careful axiomatization of credit assignment using the idea of Shapley values from cooperative game theory [undbbeg & Lee, 2017b Ych et al. 2020). Some recent methods train another deep net to produce heat maps (Dabkowski & Gal 2017 7 Phang et al. 2020). Given the proliferation of saliency methods, an ecosystem of evaluation metrics has emerged to evaluate the quality of explanations produced by saliency methods, either through human evaluations (Adebayo et al. 2018), comparison to certain ground truth explanations (Zhang et al. 2018) other evaluations that do not require any external annotation or supervision (Dabkowski & Gal 2017) Petsiuk et al. (2018).
In this paper we restrict attention to a class of evaluations we refer to as intrinsic, that aim to evaluate saliency maps based on whether they are good explanations for the model prediction. These only involve computations using the provided heat map and the net, and do not involve extrinsic faccorr such as human judgements or retraining. A popular idea in such evaluations (see Section for references) is to create a new composite input - or sequence of such inputs- using the heat map and the original input, and to evaluate the original net on this composite input. For example, if M is a binary vector with I's in the coordinates with the highest values the heat map, then O M (with @ denoting coordinate-wis, multiplication) can be viewed composite inpu ? (aka
Heaat maps suffice for recognition/clasifccat tasks; other tasks may require more complex explanations.
When is image the zeroes in this masked input are often replaced with gray values.

---

Under review as a conference paper at ICLR 2022
Original Image
Masked Image for Cat Masked image for Ship
Original Image
Masked Image for Frog Masked Image for Bird
Figure 1: Masked CIFAR-10 images generated by our procedure with λTV 0.01 shows the artifacts exist for masks generated for incorrect labels. More examples can be find Figure 7 in Appendix. The base classifier outputs the correct label on the original image (ship and bird resp.) with probability at least 0.99, and assigns probability at most 10 for the incorrect label (cat and frog resp.). With the generated masks, the AUC metric for the correct label remains high (around 0.94 and 0.90), which corresponds to completeness, but AUC metric for the incorrect label rises tremendously (around 0.18 for cat mask, 0.71 for frog mask.) This suggests violation of soundness.
"masked input") that can be fed into the original net. By varying & one obtains sequence of masks M. Although there is an issue of distribution shift due to the net never having trained on composite inputs, in practice, especially for image data, trained nets work fine on masked inputs. (This is also exploited in Shapley value based saliency methods.)
Inspired by logical reasoning, we identity key component missing in existing intrinsic evaluntions: soundness of the saliency method. In simple terms, checking soundness for mask-based explanations entails verifying that it impossible for the method produce rell-performing (in the composite- input evaluations) heat maps corresponding to the incorrect labels, i.e. labels not predicted as the most likely by the model. By contrast current masking evaluations focus on a property akin to completenes in logic: the heat map "justifies" the correct label, i.e. the label judged to be most likely by the model. We argue that soundness can a useful criterion in addition to completeness.
F
The concept of soundness is motivated by observing that the recent mask-based saliency methods are inherently different from the earlier gradient-based methods. They produce a heatmap that is interpreted as masked image that, when fed into the net must make it output the same label that received the top logit value on the full image. Our paper observes that is unclear at a logical level why this should be considered an "explanation" since to be convinced one must verify that a different mask could not induce the net to output a label corresponding to other logits. Figure [ shows that in fact this issue is real; masks do exist that can "justify" the wrong labels. We introduce soundness as a quantitative measure for this issue (definitions in Section [.1.. and propose that evaluations of mask-based saliency methods should account for soundness. Many past works allude to difficulty of the soundness concept, mentioning that computations to find the "best" mask end up finding artifacts that can even justify labels that are obviously incorrect. To avoid uncovering artifacts, the methods to find masks/heat maps use a suitable regularization (usually TV). However, Figure Jillustrats that TV regularization is not full solution to removing artifacts.
Another reason for incorporating soundness with completeness in metrics is to implicitly encourage saliency methods to also produce saliency maps for incorrect labels. When using the classifier in the wild-as opposed to on a dataset with centered images- we might encounter multiple salient objects in the image, and the net may not have high confidence in a single label. It would then be reasonable to have saliency methods find evidence for all the labels which the net outputs with reasonable confidence. We find in Figure 2 that our saliency method that produces masks for all la- bels can produce different and meaningful masks for different labels when there are multiple objects present in the image") labels that contains both objects (more this in Appendix B
Fl compared to prior sfate-of-the-ar methods, that output a single mask for all
'In the literature devoted to saliency methods based upon backpropagation-lik procedure, such GRA- DIENT @ INPUT or LRP, the term completeness is used a slightly different sense: the sum of the heat map values has to be equal to the logit value for the label.
*In this paper, multiple objects in one image is not formally considered, we assume single object clas- sification is the prior knowledge. Therefore, either zebra or elephant in Figure 2 is regarded as an incorrect label. A quick extension to the multi-object case is to define correct labels as the labels whose model proba- bility exceeds a certain threshold. Even without the extended definition of correct labels. Definition 3.2 with suitable a, 3 already allows explanations to exist for both elephant and zebra as long the model probability for elephant and zebra are high enough.
2

---

Under review as a conference paper at ICLR 2022
Dephere
New
Figure 2: Images containing both elephant(s) and zebra(s), and the corresponding masked ones generated by our method and the best-perfeer best-performing CA model in [hhan et al. (2020). The masks by Phang et al. (2020) are identical for different labels, and contains both elephant and zebra. In contrast, our method outputs descent masks for elephant and zebra accordingly.
Main contributions: In this paper make case for pushing saliency methods and metrics to incorporate soundness in addition to completeness. We propose metrics and a method and exper- imentally evaluate them in comparison findings. Our main contributions are as follows:
previous methods and metrics, leading to some novel
new single number metric named consistency score that evaluates the completeness and sound- ness simultaneously. It measures the probability that the saliency map for the model prediction has the best quality compared all other labels. See Section 3.2 for a formal definition.
A saliency method in Section that bears similarity prior works that learns masks through optimization, but with subtle differences. One of the differences is based on the logic of soundness: we learn a mask by trying maximize the probability for the given label, rather than matching the probability, and learn the best mask for all labels this way. We find in experiments (Section 6 that our method not only performs comparably to prior masked-based methods on earlier saliency metrics, but also does better than those methods on the proposed consistency score.
Previous works proposed "tricks" like TV penalty and upscaling of masks (Fong & Vedaldi 2017; Dabkowski & Gal. 2017) that are motivated by their ability avoid "artifacts" and make the saliency maps visually look better. These justifications, however, are not in the spirit of an intrinsic evaluation of saliency maps. In this paper, we provide novel intrinsic justification for these choices of TV penalty and upscaling, by showing that they improve the saliency maps by making them more sound, as empirically demonstrated in Section 6 We complement this finding with a theoretical result in Section that proves that TV regularization can help with soundness in simple linear classification setting.
Paper outline: Section 2 discusses prior works saliency methods and evaluations. Section delves further into the concept of completeness and soundness in the context of saliency evaluation, and describes our proposed metric of consistency score. Section our saliency method and how it differs from prior approaches. Finally Section descrrbes various supportive experiments.
PRIOR APPROACHES
We delegate a more thorough description of prior work Appendix D but mention some common saliency and saliency evaluation methods here. Saliency methods aim to explain a model's 's decision about an input. Saliency evaluation methods aim to evaluate the goodness of saliency method.
Saliency methods include backpropagation based approaches such as Gradient c Input (Shrikumar et al. 2017). LRP (Binder et al., 2016), GradCAM (Selvaraju et al.) 2019), Smooth-Grad (Smilkov et al. 2017). Another line of work is masking methods which include techniques based averaging over randomly sampled masks (Petsiuk et al. 2018). optimizing over meaningful mask perturbations (Fong & Vedaldi 2017), and real time image saliency using masking network (Dabkowski & Gal 2017). ). Pixels that have been removed from the image by the mask may be replaced by greying out, by Gaussian blurring in (Fong & Vedaldi 2017), or with infillers such CA-GAN (Yu et al.
3

---

Under review as a conference paper at ICLR 2022
2018) used in (Chang al. 2018 Phang al. 2020), or DFNet (Hong et al. 2019). De Cao et al.
(2220) find masks using differentiable masking. Boolean logic is another approach for saliency
methods (Ignatiev et al. 2019b a. Macdonald et al. 2019 Mu & Andreas 2020 Zhou et al. 2018).
Arguments about saliency. As saliency methods have arisen, discussion about them has occurred e.g. in (Seo et al., 2018, Fryer et al., 2021] Gu et al. [018 Sundararajan & Najmi 2020). Some
works reveal situations where Shapley axioms work against feature selection or where Shapley val-
ues may be calculated in conflicting ways (Fryer et al. [002]] Sundararajan & Najmil 2020). Others
question why noise adding saliency methods work (Seo et al. 2018). with some pointing out that
some explanations are not class discriminative (Gu et al. 2018
Saliency evaluation methods. Extrinsic evaluation metrics include the WSOL metric, and Pointing Game metric proposed by Zhang et al. (2018) and ROAR (Hooker al. 2019). Other more intrinsic methods include early saliency evaluation techniques like MorF and LerF (Samek et al.) 2
Insertion and Deletion game proposed by |Petsikk et al.) (2018), which involve either inserting pixels in order of most importance or deleting pixels in order of most importance. BAM (Yang)
& Kim 2019) creates saliency maps by pasting object pixels from MSCOCO (Lin et al., 2014)
The Saliency Metric proposed by Dabkowski & Gal (0177) thresholds saliency values above some a chosen on a holdout set, finds the smallest bounding box containing these pixels, upsamples and measures the ratio of bounding box area to model accuracy on the cropped image, s(a. p) log( ngg(ma((a, (a. 0.05)) - log(p) where a is the area of the bounding box and p is the class probability of the upsampled image.
Controversies. Brunke et al. 2020 show that perturbation methods are sensitive to baseline and Petsiuk et al. (018)) point out that human centric explanations (based on bounding boxes) may not reveal why the model made a certain decision. Our notion of intrinsic saliency method aligns with the latter idea, and we introduce a new criteria for evaluating saliency maps: soundness, which captures the notion that maps should not produce explanations for low probability labels.
3 MASKING EXPLANATIONS AND COMPLETENESS/SOUNDNEN
A running theme in saliency methods/eealluatioss is that the salient pixels should be sufficient to convince us about the model's output, regardless of contents of the other pixels. For instance, grey- ing out (or setting them to average pixel value) non-salient pixels should have very little effect on the output. This idea appears in many saliency methods (Dabkowski & Gal, 2017| Phang et al. 2020) (and evaluation metrics (Petsiuk al. 2018)) including Shapley values and mask-based explana- tions. This motives the following definition of a masked- based explanation.
Definition 3.1 (Masking Explanation). A masking explanation for input x is a distribution A over subsets S of input coordinates ("salient sets of x²) as well as an input modification process I' that generates distribution of modified inputs å ~ I(z, S) under the constraint that F' matches x on every coordinate in set S. We use [(x, A) to denote the distribution of ă produced by when the input is x and the set S is sampled from A.
A salient set S could be the direct output of any saliency method, or the output heat map passed through a potentially randomized discretization (e.g., the insertion game in Section 3.3), leading to a distribution A over sets S. Simple examples of the input modification process I that stay in the intrinsic framework are: greying out the pixels outside S, replacing them by pixels from a Gaussian blurring of æ (Fong& & Vedaldi, 2017). Another example of 1 albeit non-intrinsic and hence not used in this paper- uses conditional image generative model to produce new pixel values in S conditional on pixels in S being consistent with æ (Agarwal & Nguyen 2020 Chang et al.) 2018). Definition 3.1 can also be changed allow I' to change the values of pixels S. In our proposed method in Section 4 we consider another distribution I, which replaces pixels outside of S with pixels from a random image from the training sef]] This amounts to grafting salient pixels of x on top of random image, reminiscent of BAM evaluations Yang & Kim 2019) for saliency methods.
3.1 COMPLETENESS AND SOUNDNESS
We now describe completeness and soundness for saliency evaluations. In a multiclass classification setting, denote by f(r a) the probability for label a returned by the model on input æ. The prediction of the model on input is ((z) := arg max,ec [xr a), where C' denotes all classes. For E [0. 1),
We think other random image distributions should work too.
4

---

Under review as a conference paper at ICLR 2022
masking explanation (A,T) for input is said to p-validate the label a E C if (x a)] 2 p.
(I)
We say that a masking explanation z p-validates the model prediction if it p-validates ((x).
Definition 3.2 (Completeness and Soundness), Fix an input modification process r. A saliency method given f. x and any label a E C produces a distribution A a) over salient sets. For a,3 S 1, the method is a-complete on f, 2, a if the masking explanation (A(z, (x),a). H') p-validates the label a for x with = a. [xr a), and the method is 3-sound on [, a if the masking explanation (A(r, cannot p-validate the label a with p > max a),
Intuitively, when a, are close to 1, a-completeness means that if the model outputs a high prob- ability for label a, then the probability for label after seeing only the coordinates in the salient sets is also high; 3-soundness means that if the model outputs a low probability for label a, then the probability for label after seeing only the coordinates in the salient sets is also low. Thus desirable saliency method should have a-completeness for labels predicted by the model and 3- completeness for other labels. This captures the idea that we want the method to be able to validate the model prediction ((z) but not other labels. It is worth noting that the saliency method that de- clares all pixels to be salient is a-complete and 3-sound with a = B = 1, but its size is too large to provide any useful information. Thus one should ask the saliency method to search over the salient sets of smaller sizes. But artifacts for incorrect labels may occur at small size (Figure D.. which hurts soundness. For this reason we put explicit or implicit constraints the salient sets (c.g., TV regularization) and ask saliency methods to be a-complete and 3-sound with a, 3 as close to 1 as possible. Ideally we should adjust the constraints achieve good tradeoff between completeness and soundness. In our experiments, we find that standard "tricks" like TV regularization and mask upscaling provide such good tradeofTs, as observed in Table and Figure 3
Important note. In the logic setting, from where we borrow the concept of soundness, one has to search among all possible proofs ensure that there is indeed valid proof for any false propo- sition. Similarly, our soundness metric is non-vacuous only if the saliency method "makes its best efforts" find masking explanations for every label. Otherwise the saliency method could for in- stance ignore the value of return the mask for ÿ which can p-validate ÿ(x) with a high value of and thus not validate any other label, leading to false notion of soundness. If the saliency method behaves like this on the incorrect label (i.e., is not working at producing an explanation) then as backup the evaluation can use some default method to produce saliency explanations for those la- bels. Our method for mask-based explanations, described later, is simple and not tied to any specific philosophy or a- a-priori definition of saliency. It can produce explanations for all labels.
3.2 CONSISTENCY SCORE FOR EVALUATING COMPLETENESS AND SOUNDNESS
Inspired by the notions of completeness and soundness, we propose to single number metric that evaluates the completeness and soundness simultaneously over a data distribution.
Definition 3.3 (Consistency Score). Let X' be the distribution of input and T be an input modifi- cation process. For a saliency method that produces salient sets A (z. a) on input x and label a, we define the "saliency score" for (xx a) as ga (Ir a) (r.,(rrA(z,a))) [f, a)). i.e. average probability the model assigns to a labels for modified versions of using the label-specific salient sets A(x, a). The consistency score for the saliency method is defined as the probability that the saliency score is consistent with the model prediction, i.e.,
Pr
arg max ga (z, a) ((x))
It is easy see that if a saliency method is 1-complete and 1-sound all inputs, then the consis- tency score is 1. The following lemma shows that even a-completeness and 3-soundness can imply consistency if there is a gap between the probabilities for the largest and second largest labels.
Lemma 3.4. For an input x, if saliency method is a-complete on the model prediction j(x) and 3-sound on all other labels, and if the ratio between the output probabilities for the largest label f(2,((=))
((x) and second largest labels max(maxxzzte)) (Z. is larger than 1 then the saliency method is consistent with the model input
ag
'The constant is used to avoid blowing up due tiny probabilities.
5

---

Under review as a conference paper at ICLR 2022
Proof. By a-completeness, the saliency method p-validates (zx) with p a f(z, (z)). Since f(x))(x))
max m0xa [(x,)))e)} > ag P has the lower bound p > 3 max maxxmaxx0=00o (1 a)}.c) c for
all o (z)) Then B-soundness implies that the saliency method cannot p-validate a. Therefore Eg (e ( a)] is maximized when a = j(x), and hence consistency.
3.3 CONNECTION TO POPULAR AREA- BLLEA-UNDER-TIE THEDERR-TEE--E CEEVE EVALUATIONS OF HEAT MAPS
As mentioned, our approach requires the method to output salient sets whereas existing methods of- ten return a heatmap of saliency values. However, popular Area- Under-the- Curve (AUC) evaluation metries (Petsiuk et al.) 2018) of saliency methods can be reinterpreted in terms of our completeness and soundness. An example the insertion game:
AUC of Insertion game: For s 1 to d take the top s saliency values, and plot the probability given by model to label a on the input where the top 8 pixels of x are retained and remaining pixels are assigned values from the input modification process T. Return AUC.
Lemma 3.5. Given a saliency heatmap with AUC p for label a, we can convert it into a masking explanation that p-validates the label a.
Proof. Given heatmap, produce the salient set S by picking uniformly at random from d and letting S be coordinates corresponding to top saliency values. Let A be the distribution of the produced salient sets. Then E a)] = p by definition.
4 PROCEDURES TO FIND MASKING EXPLANATIONS
Based on the definition 3.1 of masking explanations, we propose a very simple method to find saliency masks with good empirical completeness and soundness scores. We introduce our methods in this section concisely. The detailed intuitions and inplemennations are provided in Appendix A
Our method bears similarity to prior work on learning mask based saliency maps. but with subtle differences. The key difference from Dabkowski & Gal (2017); Phang et al. (2020) is that we do not neural network to learn the mask and the difference from Fong from|Fong & Vedaldi (2017): Agarwal & Nguyen (2020): Chang et al. (018) is that we use a different input modification process I (see Definition 3.1 to infill other pixels during training: given and S, generate hybrid inputs whereby the pixels in the set S match with x and the pixels outside S are set to those of a random image drawn from the training set X. Replacing with random image pixels instead of gray pixels will make it harder to find a mask. Despite this, we can still learn mask that predicts the correct label with high confidence in practice. Furthermore, it achieves superior performance over replacing with gray pixels, which is likely because the added hardness of task helps increase the robustness of the output. The final crucial difference is that our method can find a mask for every input-label pair (I, a) and does so by trying maximize the probability assigned to label a on modified images, as opposed to trying to match the model probability for a.
As is standard, we relax the domain of masks M from binary {0, 1) 1) to continuous [0, 1/sh,, and optimize M for input x and label a based on the following natural objective
L(M; (x,, a)) = Ex_x [- log(J( @ o + - M) Ø ã. a))] + X ||M|], Ib,
(3)
where the part of x on M is superimposed onto a distractor ă ~ X' as ã = M O x + (1- (11-M) and the E, norm penalty on M helps to reduce the size of masks.
M) 3
We further employ Total-Variation (TV) penalty (Fong & Vedaldi 2017) and upscaling of the mask from a lower resolution one (Petsiuk et al.] 2018) by learning a low-resolution mask at scale s, M E [0, 1] h/mm/", to minimize the following
O ā, a))] L(M; (z. a)) Ex [- log(f(M Oz+ (
+ XXvIV(MM**) + A ||M X I|
(4)
where X* E hhw is obtained by upscaling M by factor of s E [1,4] via bilinear interpolation.
While the motivation cited for these two "tricks" is to avoid artifacts, is not clear whether artifacts are a bad thing, since they might be relevant to the net's decision-making. Indeed, in our experi- ments, we show that while TV penalty or upscaling does produce better looking masks, they lead
A standard way maximize prohability is to minimize the negative log probability

---

Under review as a conference paper at ICLR 2022
Best label
Second best label
model nutput probability
1.0
model nutput probabiiity
1.0
TV
0.0
0.8
08-
0.01
0.6
0.6
0.1
TV
0.0
0.001
0.01
0.1
0.0
0.2 0.4 0.6
0.8 10
0.0
0.2
0.0 0.6 0.8
1.0
fraction of pixels used
fraction of pixels used
Figure 3: Plot of model output probability as more pixels from the original image are retained using learned masks. The remaining pixels are replaced with gray. Different curves correspond different values of TV regularization (Arv). Larger area- underr-urreue (AUC) for the left figure (best label) suggests good completeness. while lower AUC for the right figure suggests good soundness. Plots suggest that adding TVV significantly helps with soundness, while only slightly hurting completeness.
to drop in the completeness metric. However we also show that adding such tricks leads to signif- icant improvement in the soundness metric, thus providing novel justification for the use of such tricks, beyond just the heuristic argument of getting rid of artifacts. In Section [ we also provide theoretical justification for why TV penalty can help with soundness, even for the simple case of linear predictors on non-image data.
5 CLARIFYING BENEFIT OF TV REGULARIZATION
Our experiments show that the main benefit of TV regularization in saliency methods is that it improves soundness. Here we sketch simple example showing how this benefits even for simple linear models on non-image data. Details appear in Appendix Consider S, dataset of labeled data (z. y) where labels are binary, and linear classifier with weight w and margin > 0. Assume inputs are of unit norm and with bounded foe norm and so is w. The saliency method has to return a single subset S of salient coordinates to validate the label. The input modification process T will assign all coordinates outside S. Question whether the decision can be validated using a small S. A priori this seems impossible. Furthermore imposing TV constraint seems to help nothing, because the solution has no obvious continuity structure.
But now assume we randomly permute the coordinates. This (paradoxically) turns out to make big difference! For simplicity we consider salient sets among intervals, in other words sets of TV S 2. Theorem 5.1. For any (x, v) S, after random shuffle following holds for any L1 52 log }),
L2 = 2 log 2), where probability is over the random shuffle:
1. (Completeness) With probability - 6, there is an interval of length L, that validates
2. (Soundness) With probability 1 - 6, no interval of length Lo can validate -y-
6 EXPERIMENTS
In this section, we present results for our mask learning procedures described in Section 4 and our consistency score metric described in Definition 3.3 We also analyze the role of TV penalty and upsampling. We observe from the results that (1) our simple mask learning procedure achieves competitive performance with existing saliency methods on existing metries; (2) our procedure does better on our consistency score metric compared to previous methods, which suggests that consis- tency score has no major conflicts with previous metrics and that our method is more sound; (3) TV regularization and upscaling significantly aid soundness, while only slightly hurting completeness. Overall, consistency score increases with higher TV regularization and upscaling, thus showing the ability of our metrics provide intrinsic explanations for existing "tricks" that previously only had extrinsic justifications.
Experiments involving consistency score computationally infeasible to perform on ImageNet as it contains 1000 classes. Therefore, these experiments are performed either on CIFAR-10 or Imagenette (Howard. 2020), 10-class ImageNet subset. Even for CIFAR-10 and Imagenette, since
7

---

Under review as a conference paper at ICLR 2022
Table I: We compute AUC insertion game with gray infilling masking explanation. The last two columns are the mask-model consistency conditioned on whether the base classifier predicts the ground truth y. The best two of each column are marked bold. From the table, our method with upscaling factor achieves the best consisteney.
Deletion
Insertion
Saliency
Consistency Score
Consistency
Consistency
(gray) t
Metric +
when 1 y
when 1 o y
Gradient @ Input
0,42
0.56
0.31
0,841
0.977
0.27
Real time saliency
0,48
0.66
0.85
0.857
0,980
0.34
Fong Vedaldi 2017
0.64
0.63
0.41
0.845
0.990
0.23
Phang el al. (2020)
0.43
0.76
0.27
0.871
0.985
0.39
Ours (8 1)
0.52
0.68
0.91
0.862
0.968
0.41
Ours (s = 4)
0.47
0.57
0.66
0.880
0,980
0.46
Random
0,45
0.45
0.35
0.829
0.964
0,26
consistency score requires computing 10 times more saliency maps than other metrics, we test it on 1000 randomly drawn images from original test set. We also run experiments that do not involving consistency score on various datasets (including ImageNet and CIFAR-100) with various models. Results are shown Appendix Details of training procedure (beyond those Section are also in the Appendix C
6.1 FULL COMPARISON TO EXISTING METRICS AND METHODS ON IMAGENETTE
We compare our procedure to other methods including Real Time Saliency (Dakkosski & Gal 2017). Gradient C Input (Shrikumar et al., 2017). Fong & Vedaldi (017)) and Phang al. (2020)
on Imagenette. Apart from our new consistency score metric, we calculate the Deletion Game and Insertion Game metrics using the code provided by https: gitbhubbubuu coo//elliee eewiiuue//eeeeeecccccceeeeeeeeee
and Saliency metric (SM), an another intrinsic evaluation metric from Dabkowski & Gal (2017)
Detailed description of the different metrics can found in the Appendix.
For our method, we use the learned mask M from optimizing the objective function in Equation with \rv 0.01 and {1, 4) as the saliency map. For fairness, we use the identical ResNer50 pretrained on ImageNet as the base classifier for every saliency method. The salient sets used in our consistency score follows the Insertion game. Insertion scores were calculated by replacing remaining pixels with gray. We normalize the maps so that all values lie in [0, 1] before use.
To get a sense what the numbers mean, we also include the performance of a random mask, where the score for each pixel is sampled independently from standard normal distribution. The results are shown in Table I For the deletion metric, we note that most methods have comparable or worse performance than the random mask, which suggests that the metric does not give us much signal about the goodness of the saliency maps. For the Insertion scores and saliency metric, our method performs decently since these metrics mostly depend on variety of the completeness condition.
For our consistency score metric, most previous methods perform well on consistency score when the base classifier predicts the ground truth. However, their performance on samples that the base classifier predicts incorrectly is not as good as ours. Our method shows that soundness can be im- proved without much affecting performance on other known metries. Note that our metric assumes that testing methods take effort to generate different masks for each possible label. Saliency methods like Phang al. (2020) actually inherent advantage on our metric by not doing that. Considering that our method bears similarity to prior work but outperforms them on consistency score even with arguable disadvantage, implies there is room for improvement on soundness for previous works.
Table also shows that upscaling improves the consistency score. As mentioned in Section upscaling was used in previous work avoid "artifacts" and produce better looking masks. Our experiment shows new explanation for the upscaling that it improves the soundness of the masking.
To verify the connection between completeness/soundnes and consistency score, we compute the completeness and soundness (with 10- for test samples, and report the average in Table 2 It shows completeness/soundnss and consistency score are highly correlated. Even though saliency methods like Phang al. (2020) have inherent advantage in soundness, our method still achieves the best soundness. Our method also gains better completeness, which is likely contributed by maximizing the probability assigned the given label instead of matching the model probability.
8

---

Under review as a conference paper at ICLR 2022
Table Completeness and soundness for each saliency methods, averaged over test samples. Com- pleteness are computed on labels that the model predicts, and soundness are computed on the in- correct labels (w.r.t model prediction). For soundness, we compute both the average among the incorrect labels and the minimum one among them. Consistency scores are listed in the last column a comparison. The best two of each column are marked bold. Consistency scores are highly correlated to completeness/oundness in the table.
Completeness of Average soundness of
Worst soundness of Consistency the incorrect labels
model prediction
the incorrect labels
Score
Gradient Input
0.34
0,98
0.83
0.841
Real time saliency
0.56
0.98
0.85
0.857
Fong & Vedaldi (0177)
0.53
0,95
0.61
0.845
Phang ct al. 2020
0.71
0.98
0.87
0,871
Ours
0.85
0,95
0.61
0.862
Ours 4)
0.83
0.99
0.89
0.880
Random
0.35
0,96
0.70
0.829
Table 3: Consistency scores for different λrv in CIFAR-10. The last two rows are the mask-model consistency conditioned on whether the base classifier predicts the ground truth. The best one in each row is marked bold. \rv = 0.1 performs the best in both situations.
ATV
0
0,001
0.01
0.1
Consistency Score
0,46
0.37
0.49
0.62
Consistency (correct)
0.47
0.38
0.51
0.64
Consistency (incorrect)
0.24
0.21
0.16
0.36
6.2 EFFECT OF TV REGULARIZATION
In this subsection, we test our theoretical prediction from Section TV penalty improves sound- ness. We first visually inspect the masks learned by procedure we increase the TV regular- ization strength in Figure Masked images at 10 sparsity are depicted in odd columns and the full masks are shown in even columns. We plot the AUC insertion metric with blur evaluated the non-sparsely masked images on the axis of each plot. We find that as we increase the TV regularization strength, the model can still find saliency map for the correct label with high AUC score, however the AUC score for the mask learned to fit the second most confident label drops significantly. This, in conjunction with Lemma 3.5 suggests that the TV regularization method has slightly worse completeness but much higher soundness. Figure 3 plots the model output proba- bility for CIFAR-10 as more pixels from the original image are retained using the mask. Our high level finding is again that adding TV penalty and upsampling significantly aid soundness, while only slightly hurting completeness. To formally justify the effect of TV penalty, we measure our consis- tency score of different level of TV penalty on CIFAR-10. To further challenge the completeness and soundness, the salient sets of our consistency score follows the Insertion game but only takes sets of size 0.2d to 0.6d (d is the number of pixels). Table 3 the result.
7 CONCLUSIONS
Saliency explanations of ML models has proved nebulous and generated many controversies. By taking rooted in intrinsic definitions such as completenes/soundness and consistency score, this paper has tried to provide greater rigor to the intrinsic approaches to saliency. Other new contri- butions include clarifying the role of TV regularization (it hurts completeness slightly but greatly improves soundness); extensive experimental evaluations that bring new understanding using con- sistency score; and a simple saliency method for producing mask-base explanations whose perfor- mance is broadly competitive with good existing methods, and sometimes better.
The soundness notion is clearly useful for localization of objects in the image, and for classification in the wild -where multiple objects appear over the image. This was hinted in evaluation prior small dataset (see Appendix B). It may have other applications to handling distribution shift and other types of robustness, but such explorations are left for future work.
Similar images for scaling factors be found in Appendix

---

Under review as a conference paper at ICLR 2022
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances Neural Information Processing Systems, 31:9505- 9515, 2018.
Chirag Agarwal and Anh Nguyen. Explaining image classifiers by removing input features using generative models. In Proceedings of the Asian Conference on Computer Vision, 2020.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Roburt Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130100, 2015.
David Bachrens, Timon Schrocter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller. How explain individual classification decisions. The Journal of Machine Learn- ing Research, 11:1803 803-1831, 2010.
Alexander Binder, Grégoire Montavon, Sehastian Lapuschkin, Klaus-Roberr Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pp. 63-71. Springer, 2016.
Lukas Brunke, Prateek Agrawal, and Nikhil George. Evaluating imput perturbation methods for interpreting cnns and saliency map comparison. In European Conference on Computer Vision, pp. 134.134. Springer, 2020.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image clas- sifiers by counterfactual generation. In International Conference on Learning Representations, 2018.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6970-6979, 2017.
Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3243- 3255, 2020.
Ruth Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE International Conference Computer Vision, pp. 3429-34377 2017.
Daniel Fryer, Inga Strümke, and Hien Nguyen. Shapley values for feature selection: The good, the bad, and the axioms. ICLR, 2021.
Jindong Gu, Yinchong Yang, and Volker Tresp. Understanding individual decisions of cnns via contrastive backpropagation. In Asian Conference on Computer Vision, pp. 119-134. Springer, 2018.
Xin Hong, Pengfei Xiong, Renhe Ji, and Haoqiang Fan. Deep fusion network for image completion. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 2033-2042, 2019,
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretabil- ity methods in deep neural networks. In NeurIPS, 2019.
Jeremy Howard. imagenette, 2019. URL https://github. comffustail/mmggenntte 2020.
Alexey Ignatiev, Nina Narodytska, and Joao Marques-Silva. Abduction-based explanations for ma- chine learning models. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 33, pp. 1511-1519, 2019a.
Alexey Ignatiev, Nina Narodytska, and Joao Marques-Sikee On relating explanations and adversar- ial examples. 2019b.
10

---

Under review as a conference paper at ICLR 2022
Ashkan Khakzar, Soroosh Baselizadeh, Saurabh Khanduja, Christian Rupprecht, Seong Tae Kim, and Nassir Navab. Improving feature attribution through input-specific network pruning. arXiv e-prints, pp. arXiv-1911, 2019.
Beomsu Kim, Junghoon Seo, Seunghyeon Jeon, Jamyoung Koo, Jeongyeol Choe, and Taegyun Jeon. Why saliency maps noisy? cause of and solution noisy saliency maps. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 4149-4157. IEEE, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprini
arXiv: 1412.6999, 2014.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang. Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: A unified and generie model interpretability library for pytorch, 2020.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predic- tions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems. Curran Asso- ciates, Inc., 2017a. URL https: httpps://prcceeddnggg neurips cc/paperr201//flle/
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceed- ings of the 31st international conference on neural information processing systems, pp. 4768- 4777, 2017b.
Jan Macdonald, Stephan Wäldchen, Sascha Hauch, and Gitta Kutyniok. rate-distortion framework for explaining neural network decisions. arXiv e-prints, pp. arXiv-1905, 2019.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. 2020.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference (BMVC), 2018.
Jason Phang, Jungkyu Park, and Krzysztof J Geras. Investigating and simplifying masking- -bsed saliency methods for model interpretability. arXiv preprint arXiv:2010.09750, 2020.
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller. Evaluating the visualization what a deep neural network has learned. IEEE transactions on neural networks and learning systems, (6(1))26600267. 2016.
Wojeiech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Roben Müller. Explainable Al: interpreting, explaining and visualizing deep learning, volume 1 17700. Springer Nature, 2019.
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. In International Conference on Leurning Representations, 2019.
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? International Journal of Computer Vision (IJCV), 2019.
Junghoon Seo, Jeongyeol Choe, Jamyoung Koo, Seunghyeon Jeon, Beomsu Kim, and Taegyun Jeon. Noise Noise-adding methods of saliency map series of higher order partial derivative. 2018 ICML Workshop on Human Interpretability in Machine Learning, pp. arXiv-1806, 2018.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pp. 3145- 3153. PMLR, 2017.
11

---

Under review as a conference paper at ICLR 2022
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv e-prints, pp. arXiv-1706, 2017.
Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In Inter- national Conference on Machine Learning, pp. 9269-9278. PMLR, 2020.
Saeid Asgari Taghanaki, Mohammad Havaei, Tess Berthier, Francis Dutil, Lisa Di Jorio, Ghassan Hamarnh, and Yoshua Bengio. Infomask: Masked variational latent representation to localize chest disease. International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 739-747. Springer, 2019.
Mengjiao Yang and Been Kim. Benchmarking attribution methods with relative feature importance. arXiv e-prints, pp. ar0- 2019.
Chih-Kuan Ych, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. concept-based explanations in deep neural networks. Advances Neural Information Processing Systems, 33, 2020.
On completeness-awe
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5505-5514, 2018.
Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126 (10):1044-1122, 2018
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions pattern analysis and machine intelligence, (006)):4452-444 2017.
Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representations via network dissection. IEEE transactions pattern analysis and machine intelligence, 41(9): 2131-2145, 2018
12

---

Under review as a conference paper at ICLR 2022
A LNTUITIONS AND IMPLEMENTATIONS OF PROCEDURES TO FIND MASKING EXPLANATIONS
As introduced in Section 3.3 for evaluation we may interest in random binary masks due to its connection to AUC, but in our method for finding masking explanations we only focus on deter- ministic masks. Given network J. image E Rexkw and class a, we wish to find binary mask M E {0, 1/ such that when the part of z on M is superimposed onto distractor I ~ X as M @ x (1 (1-M) @ the output probability of the model f(ī, a) is high for the class a. This corresponds to the case where A is a singleton that assigns probability 1 to M and [ar M) is the class a over (f a)] high. To avoid the hard problem of optimizing over the hypercube {0, 1) low, a typical strategy (also employed in prior work) is to relax the domain of M to be [0, 1 Since we do not wish to learn masks of very large size, a & norm penalty on M (corresponding to size of the mask), leading to the following natural objective function
distribution of = As in Section 3.1 we compute the average probability assigned
the sampling of the distractor Σ, i.e. we are interested in making
L(M) Ez-xx [- log(f o o + (1- M) o 2, 2.))] + X 1|MM
(5)
However most masking-based methods employ additional "tricks" in order to avoid "artifacts" in the produced saliency maps, like VariitVaiittion (TV) penalty (Fong Vedaldi (017) and upscaling of the mask from a lower resolution one (Petsiuk et al. 2018). We also employ the same strategy by learning low-resolution mask at scale M E hg/// to minimize the following
L(M) Ex-x [- log(f(M* oz +
x O a.)] + (TV + -1IllM*
(6)
where xx* E Rhw is obtained by upscaling M by factor of {1,4] 4] via bilinear interpolation.
While the motivation cited for these "trick" is to avoid artifacts, it is not clear whether artifacts are a bad thing, since they might be relevant to the net's decision-making making.. Indeed, we show that while TV penalty or upscaling does produce better looking masks, they lead to a drop in the completeness metric. However we show that adding such tricks leads to significant improvement the soundness metric, thus providing a novel justification for the use of such tricks, beyond just the heuristic ar- gument of getting rid of artifacts. In Section 5 we also provide theoretical justification for why TV penalty can help with soundness, even for the simple case of linear predictors non-image data.
We optimize the objective in Equation (4) by parametrizing M as a sigmoid of real valued weights W E / i.e. M = (W). and run Adam (Kingma & Ba, 2014] optimizer for 2000 steps with learning rate 0.05 and by sampling 10 distractor images at every step. for different values of \rv and upscaling factor s. We report the effect of Xrv qualitatively in Figure 4 and quantitatively on various intrinsic saliency metrics in Table 10
B PRACTICAL BENEFITS OF SOUNDNESS FOR IMAGES OF MULTIPLE OBJECTS
Images may have multiple plausible labels. In Figure 2 images that previously used in Gu et al. 2018 can have both elephants and zebras present, but it may not be always clear from the model output if there is such a case, since the model can be much more confident on one label, e.g., elephant, than one would expect it to be. For this reason, finding masking explanations validating other labels, e.g.. zebra, could provide more information on how the model makes the prediction.
C EXPERIMENTAL DETAILS AND ADDITIONAL EXPERIMENTS
In this section we expand upon the experiments in Section 6 and complement them with more experiments on the ImageNet, CIFAR-10 and CIFAR-100 datasets. For each of the datasets test the following:
We slightly abuse the use of notation to use M as set over coordinates rather than binary mask.
A standard way maximize prohability is to minimize the negative log probability
13

---

Under review as a conference paper at ICLR 2022
1ym0.0
tvm0.00
tyme0.00 001
ly=0.001
tvm0.00
tvm0.01
tv
1v=0.1
pay.phone 0.9634
09766
0,257
0,287
0.885
0.056
aircroft carrier: 0.9950
0.8758
0.8756
0.8594
0.8508
chimparaee
0.033
0,933
0.905
0.505
10 % retained
100 1% retained
10 % retained
100 % retained
10 retained
100 % retained
10 % retained
100 retained
tv-0.01
tv-0.01
TV=0.11
tv-0.11
ges pump 0.0247
O 0094
19990
02342
0.342
0.0N
0.084
07735
0.235
0.1173
0.4135
0.0040
0.0049
siamang 0003
05707
05707
0.3246
03244
0,002
0,0022
10 reteined
100 1% ret retained
10 reteined
100 retained
% reteined
retaiied
retained
100 % retained
Figure 4: Top: Masking validating the correct label of ImageNet images using Section 4 Pixels out- side the salient set S are rendered grey. Size of S (as % of total pixels) appears below the images. Value of TV regularizer shown above each column corresponding to (0,.001,.01..11 Original image rendered in last column. True labels with original model probabilities column. Insertion metric when the mask is found by the procedure in Section 4 shown on axis Bottom: Masking validating the second-best label, which appears on the right along with original model probability of that label. Insertion metric when the mask is found by fitting the second best label under the procedure Section is shown the label of each plot. We find that increasing the TV regularizer makes the resulting assignment more sound.
shown in the rightmost
Visualization: For various values of TV regularization (and upscaling for ImageNet), we visualize the mask and also what part of the image sparse version of the mask highlights. We do so for masks learned for the correct label and also for the second most probable label as predicted by the model. The common trend that while TV regularization (and upscaling) make the masks more human interpretable, it also makes it harder to find good mask (partial assignment) for the incorrect label, thus improving soundness.
AUC curve: We plot the output model probability for a masked input as more pixels from the original image are selected. The plots denote replacing remaining pixels with grey pixels pixels from a random image, and masks to the correct or incorrect label. Again, we find the TV regularization and upsampling help with soundness. For mask M, if (p) denotes the discrete mask with top p fraction of the pixels from M picked. We plot E2 [f
(1 (p)) @ x', a)] v/s p. where either random image grey image, and a is either the correct label for x or the second best label.
Completeness/oonndnes:: We evaluate completeness and soundness metrics as defined in Definition 3.2) In particular, for every input evaluate completeness for the correct label a and soundness for the incorrect (second best) label a'. For any mask M, we de- fine AUC AUC(M; (I) E, [E
M(p) o x', a)]] Com-
14

---

Under review as a conference paper at ICLR 2022
pleteness and soundness are defined
AUC(M;
(MM ((,,,)
min
min(f(a,
max(f(r,d),e
S.(M) B(x,a")
min
(8)
(C,
where is the correct label and a' is the incorrect label and f(x, a) is the model probability for label for input ó and can be any reasonable constant stabilize the value.
Intrinsic metries: We evaluate our masks on other intrinsic metrics from prior work, and
compare to baseline saliency methods. Our baselines include Gradient @ Input (Shrikumar
2017). Smooth-Grad (Smilkov et al. 2017). Real Time Saliency (Dabkowski & Gal, 2017 (for ResNet-50 on ImageNet), and Random indicating random Gaussian mask as control. We use Captum (Kokhlikyan et al.] 2020) for Gradient @ Input and Smooth- Grad implementations and the original author code n1 for Real Time Saliency. When calculating the Saliency Metric (SM) (Dabkowski & Gal (017) we tune the threshold d on a holdout set of size 100 with between and in increments of 0.2 as in prior work.
et al.
For the saliency method of Fong & Vedaldi (2017) that we only used on the Imagenette, we GitHut The implementation contains minor
adapt the most popular implementation deviations from the original paper as described on its main page. For Phang et al. (2020), we used their best CA model pretrained and provided in original author code
C.1 CIFAR-10 EXPERIMENTS
We also run our method from Section 4 on the CIFAR-10 dataset using a pretrained ResNet- 164 architectur For all experiments we learn a mask M E R222322 thus using scaling factor of 8 1 (no upscaling). We train masks for 1600 images that were correctly classified by the pretrained ResNet- 164 using regularization parameter Xry E (0, 0.001, 0.01, 0.1). We use (fixe) L1 regularization value of .000 for all masks.
We visualize the masks learned for the correct label in Figure 6a and in Figure 6b we visualize the same for the second best label predicted by the ResNet-164 model. We also visualize the masks for all labels for some randomly picked images in F demonstrate the commonness of artifact, especially for the incorrect labels. The AUC curves in Figure 5S suggest similar trend that of ImageNet, adding TV regularization results in only a mild drop in completeness, but significantly improves soundness. Evaluation of our masks, compared to some gradient baselines, on intrinsic metrics can be found in Table [5] We place a downarrow after the name of the metric to indicate lower value considered better and an uparrow when higher value is considered better. We evaluate on randomly selected subset of 1000 data points where the model had correct top accuracy. We report the completeness and soundness results for CIFAR-10 in Table for TV values in (0.0,0.001,0.01,00 calculated using ResNet- 164 model.
C.2 CIFAR-100 EXPERIMENTS
We run the same experiment for CIFAR- using the corresponding ResNet164 model. We visu- alize the masks learned for the correct label in Figure Pand in Figure 9b we visualize the same for the second best label predicted by the ResNet-166 model. The AUC curves in Figure 13 suggest a similar trend to that of ImageNet, adding TVV regularization results in only mild drop in complete- ness, but significantly improves soundness. Evaluation of our masks, compared to some gradient baselines, intrinsic metrics can be found in Table 7 We place a downarrow after the name of the metric to indicate a lower value is considered better and an uparrow when a higher value is consid- ered better. We evaluate correct top I accuracy. When calculating the saliency metric we tune the threshold ō on a holdout set of size 100 with d between 0 and 5 in increments of 0.2.
randomly selected subset of 1000 data points where the model had
https:://
boxx
httpp::///itthbb.ccm//////s// repository is actually ResNet- model.
The ResNet-1 10 model in this
15

---

Under review as a conference paper at ICLR 2022
Table Completeness and soundness for ResNet-164 CIFAR-IO defined in Equation (). Each column contains tuple (Grey/Noose,, TV 0.0/ TV 0.001/ TV 0.01/ TV 0.1). Grey indicates pixels were greyed during calculation. Noise indicates they were replaced with other images. TV indicates TV regularization value of 0.0, 0.001, 0.01, or 0.1.
Grey
TV = 0.0
TV= 0.001
TV= 0.01
TV= 0.1
Correct label completeness (Co.s)
0.92
0.91
0.84
0.79
Second label soundness (So.2)
0.27
0.28
0.34
0.38
Noise
TV= = 0.0
TV= 0.001
TV= 0.01
TV= 0.1
Correct label completeness (Co.s)
0.90
0.88
0.81
0.75
Second label soundness (So.2)
0.30
0.31
0.37
0.42
Ground Truth Label,
Ground Truth Label, Noise
Second Best Label, Grey
Second Label, Noise
(scale, TV)
0,0.0
0.001
0.001
model output probabiitty
probability
Miliggooo Soooon andino japol
model output probability
0.))
(scale, TV)
(scale,
(scale,
0.8
T 0.:44)
(1, call
(1. 0.0031
IL 2.00)
c.001)
0.0000
D. 0.03)
0.03)
0.3)
0.:
0.31
fraction of pixels used
fraction of pixels used
fraction of pixels used
fraction of pixels used
Figure 5: [CIFAR-10] AUC curves with as the fraction of pixels retained from the original images based on the mask varies from 0 to 1.0 on the X-axis. The probabilities assigned by the model (averaged over 1600 images) on the Y-axis. Left: Mask learned for ground truth label, probabilities for ground truth label while replacing remaining pixels with grey. Center Left: Mask learned for ground truth label, probabilities for ground truth label while replacing remaining pixels with other image pixels. Center Right: Mask learned for second best label, probabilities for second best label while replacing remaining pixels with grey. Right: Mask learned for second best label, probabilities for second best label while replacing remaining pixels with other image pixels. We see that increasing regularization results in only mild drop in completeness, but significantly improves soundness.
Table 5: Performance of our method on CIFAR- 10 and some baselines on various intrinsic saliency metrics proposed in prior work. We find that while both our masks (learned with and without TV) have very good performance on the insertion metric. The deletion and saliency metrics are uninfor- mative in this case, since all methods are as good (or worse) compared to a random mask.
Gradient O Input
Our method
Our Method
Smooth-Grad saliency
Random
(Arv 0.01)
(ArV
Deletion & Insertion (blur)+ t
0.32
0.37
0.59
0.31
0.26
0.60
0.88
0.94
0.66
0.36
Insertion (grey)
0.51
0.83
0.92
0.55
0.26
Saliency Metric
0.22
0.22
0.22
0.23
0.22
We report the completeness and soundness results for CIFAR-100 in Table for TV values in (0.0.0.001.0.01.00)) calculated using ResNet- 164 model.
16

---

Under review as a conference paper at ICLR 2022
Teed
Tre0:000
Reed 021
Teet
Ne
Numoodde 0,9527
eutormate 0.99469
EXIT
fog
(a)
CID
Nuttmeeb 0.0
30 nitrred 100 retaired
meaned
100 % retored
10 rersined
102 reteined
% metiined
120 5. rebaieed
(b)
Figure 6: Details in Appendix C.I Panel 6a Partial statistical assignments validating the correct label of CIFAR-10 images using the procedure outlined in Section ResNet-1 Columns (1,3,5,7) depict masked images at 30 (retained) % mask sparseness. Columns (2,4,6,8) depict the original mask. TV values shown above. Original image shown in rightmost column. Model proba- bility of correct label for masked images the second most probable label of CIFAR- 10 images using the procedure outlined in Section 4 on ResNet- 164. Columns (1,3,5,7) depict masked images at 30 % mask sparseness. Columns (2,4,6,8) depict the original mask. TV values shown above. Original image shown in rightmost column. Model probability of second best label for masked images axis.
axis. 6a Partial statistical assignments validating
Table 6: Completeness and soundness for a ResNet-164 CIFAR-100 as defined in Equation (8) Each column contains a tuple (Grey/Noise, TV 0.0/ TV 0.001/ TV 0.01/ TV 0.1). Grey indicates pixels were greyed during calculation. Noise indicates they were replaced with other images. TV indicates TV regularization value of 0.0, 0.001, 0.01, or 0.1.
Grey
0.0
TV= 0.001
0.00.01
TV= 0.1
Correct label completeness (Co.8)
0.78
0.74
0.66
0.60
Second label soundness (So.2)
0.39
0.42
0.50
0.54
Noise
TV = 0.0
TV= 0.001
TV= 0.01
TV=
Correct label completeness (Co.s)
0.72
0.70
0.65
0.58
Second label soundness (So.2)
0.46
0.47
0.51
0.56
C.3 EXPERIMENTS ON IMAGENET
C.3.1 RESNET-18
In Figure 10a we depict the the masks for TV values in {0.0, 0.01} for ResNet-18 model on ImageNet for the ground truth label and in Figure 10b we depict the same for the second best label. We also experiment with the effect of upsampling (US) the mask, whereby we leam mask of size (56.56) and upsample to size (224,224). We use fixed LI regularization value 2e-5. We depict results ImageNet and ResNet-18 in Table
17

---

Under review as a conference paper at ICLR 2022
plane
bird
dog
truck
plame
bird
dog
frog
truck
plame aulo
biru
doy
Trog
ship
truck
ginal plane auto
bird
cat
ceer
dog
frog
horse
ship
truck
ginal plane
suiti hird
cat
cieer
dog
frog
horse ship
truck
ginal
plane
auto
bird
cat
ceer
dog
frog
horse
ship
truck
(a) Our method with no TV regularization
ginal plane
auto
bird
cat
coer
dog
frog
horse
ship
truck
ginal plane auto bird cal
cleer
dog
frog
horse ship
truck
ginal plane auto bird cat
ceer
dog
frog
horse
ship
truck
or ginal plane autto bird cat
dierr
dog
frog
horse ship
truck
ginal plane auto hird cat
ceer
dog
frog
horse ship truck
ginal
plane
auto
bird
cat
ceer
dog
frog
horse
ship
truck
(b) Our method with TV regularization Arv 0.01
Figure 7: A demonstration of artifacts created by masking. Pixels (partially) masked out are filled with gray based on the fractions they are masked out. Masks generated without or only with low level regularization easily produce artifacts. It is more common and/or severe for the incorrect label than correct label.
18

---

Under review as a conference paper at ICLR 2022
Ground Truth Label,
Ground Truth Label, Noise
Second Best Label, Grey
Second Label. Noise
(scale, TV)
(scale, TV)
(scale, TV)
2.000
8,000
0.0)
01000030
0,00003
0.011
0.0.011
0.0011
0.00
model outpur probabiitt
mode output probabilty
0.111
Chllieooo andino modei
12,
0.3)
mode outpur poo
(scale, TV)
0..0
0.0011
8.001
0.30
fraction of pixels used
fraction of pixels used
fraction of pixels used
fraction of pixels used
Figure 8: [CIFAR-100] AUC curves with the fraction of pixels retained from the original images based on the mask varies from 0 to 1.0 on the X-axis. The probabilities assigned by the model (averaged over 1600 images) on the Y-axis. Left: Mask learned for ground truth label, probabilities for ground truth label while replacing remaining pixels with grey. Center Left: Mask learned for ground truth label, probabilities for ground truth label while replacing remaining pixels with other image pixels. Center Right: Mask learned for second best label, probabilities for second best label while replacing remaining pixels with grey. Right: Mask learned for second best label, probabilities for second best label while replacing remaining pixels with other image pixels. We see that increasing TV regularization results in only a mild drop in completeness, but significantly improves soundness.
Table 7: Performance of our method on CIFAR-100 100 and some baselines on various intrinsic saliency metries proposed in prior work. We find that while both our masks (learned with and without TV) have very good performance the insertion metric. The deletion and saliency metrics are uninfor- mative in this case, since all methods are as good (or worse) compared to a random mask.
Gradient Input
Our method
Our Method
Smooth- Grad saliency
Random
(Arv 0.01)
(Arv
Deletion I Insertion (blur) t
0.10
0.17
0.10
0.29
0.11
0.36
0.71
0.82
0,39
0.20
Insertion (grey)
0.27
0.62
0.76
0.29
0.11
Saliency Metric
0.77
0.77
0.77
0.79
0.77
For the deletion metric, we note that most methods have comparable or worse performance than the random mask, which suggests that the metric does not give much signal about the goodness of the saliency maps. On the insertion metric, we find that mask learned by not adding the TV penalty significantly beats other methods. The mask learned using TV penalty, on the other hand, has impressive performance on both the insertion AUC and saliency metric (SM).
Completeness and Soundness on ImageNet and ResNet-18 We report our results in Table & for TV values in (0, 0.01) for both greying (Grey) and replacing with other image pixels (Noise). Addi- tionally, we investigate the effect of upsampling (US) where we derive a (56,56) and upsample by factor of 4 to (224, 224) mask.
order to investigate the effect ensembling we plot maps in Figure I2 vary the number of maps that ensembled over as K {1, 4). where learn multiple masks such that each of them are individually statistical assignments. We do not upsample (using a fixed L.1 regularization of 2e-5 and fixed TV regularization of 0.0.
Effect of ensembling.
scale of 1.0) and
C.3.2 50
We present our results on ImageNet and ResNet-50 Table 10 Using the same pretrained ResNet- 50 model Dabkowski & Gal (2017) lets compare method their real-time saliency method
19

---

Under review as a conference paper at ICLR 2022
Nee3001
Tradd
Tee0.
Neod
No51
Supoorar 0.0011
Hoard 0.9522
02
buor
6-1222
retarie
100 retarnot
20% rotained
130 ritored
retaired
100
32 retannaa 103 notaincd
(b)
Figure 9: Details in Appendix C.2 Panel 9a Partial statistical assignments validating the correct label of CIFAR-100 images using the procedure outlined in Section 4 on ResNet- 164. Columns (1,3,5,7) depict masked images at 30 (retained) % mask sparseness. Columns (2,4,6,8) depict the original mask. TV values shown above. Original image shown in rightmost column. Model proba- bility of correct label for masked images the second most probable label of CIFAR- 100 images using the procedure outlined in Section on ResNet- 164. Columns (1,3,5.7) depict masked images at 30 % mask sparseness. Columns 62,4.6,8) depict the original mask. TV values shown above. Original image shown in rightmost column. Model probability of second best label for masked images axis.
axis. Panel[9]
statistical assignments validating
Table 8: Completeness and soundness for a ResNet-18 18 model on ImageNet as defined in Equa- tion Each column contains tuple (Grey/Noise, TV 0.0/ TV 0.001/ TV 0.01/ TV 0.1). Grey indicates pixels were greyed during calculation. Noise indicates they were replaced with other im- ages. US indicates the full (224.224) mask was derived. US indicates (56, 56) mask was derived then upsampled by factor of 4. TV indicates a TV regularization value of 0.0 or 0.01.
Grey
TV 0.0
TV= =
US TV = 0.0
US TV = 0.01
Correct label completeness (C1)
0.93
0.82
0.75
0.72
Second label soundness (So)
0.21
0.39
0.51
0.57
Noise
TV = 0.0
TV = 0.01
US TV = 0.0
US TV 0.01
Correct label completeness (C1)
0.85
0.71
0.61
0.60
Second label soundness (So)
0.24
0.40
0.53
0.57
directly on intrinsic metrics. Dabkowski & Gal (2017) do not evaluate on insertion/delltion metrics, so we evaluate these using their pretrained mask model.
C.4 EFFECT ON SANITY CHECKS
Inspired by (Adebayo et al. 2018) we randomize the last layer of ResNet-13 18 network and visually inspect the resulting saliency maps in Figure II We find that the maps appear less coherent than those of a pre- mrentriined model. We use fixed Li regularization 2c-5 and depict maps with and without upsampling (US) at TVV values of (0, 0.01).
20

---

Under review as a conference paper at ICLR 2022
hreld
Tea0.01
Ieed
Irad US..
Iree us dhoun 0,56399
arciuht canter 0.993
Sumpaaaee
retained
resaived
Tym0
Dow0.02
01
pas cumOo 0,247
CAST
Sulmaari 0,00233
retained
(b)
Figure 10: Details in Appendix C.3.1 US stands for upsampled mask, where derive (56,56) mask and interpolate to (224,224). Panel 10a Partial statistical assignments validating the correct la- bel of ImageNet images using the procedure outlined in Section ResNet-50. Columns (1,3,5,7) depict masked images at 30 (retained) % mask sparseness. Columns (2,4,6,8) depict the original mask. TV values shown above. Original image shown in rightmost column. Model probability axis. Panel [Ob] Partial statistical assignments validating
of correct label for masked images the second most probable label of ImageNet images using the procedure outlined Section 4 on ResNet-50. Columns (1,3,5,7) depict masked images at 30 % mask sparseness. Columns (2,4,6,8) depiet the original mask. TV values shown above. Original image shown in rightmost column. Model probability of second best label for masked images regularization and upsampling make the mask more continuous and "human interpretable" and, more importantly, make it harder to find masks that can get high probability for the second best label, thus ensuring higher soundness.
axis. We find, unsurprisingly, that adding
Table 9: Performance of our method on ImageNet and ResNet-18 model and some baselines on various intrinsic saliency metrics proposed in prior work. We find that while both our masks (learned with and without TV) have very good performance on the insertion metric, the mask learned with TV has much better performance the saliency metric. The deletion metrie uninformative in most cases, since most methods are as good (or worse) compared to random mask.
Gradient Input
Our method
Our Method
Smooth- saliency
Random
(Ary = 0.01)
(Arv = 0)
Deletion Insertion (blur) f
0.1054
0.1337
0.2080
0.0757
0.1336
0.4443
0.7936
0,8507
0,5062
0.3118
Insertion (grey)
0.3056
0.6742
0.9213
0.3518
0.1325
Saliency Metric
0.3149
0.1507
0.3156
0.3157
0.3156
21

---

Under review as a conference paper at ICLR 2022
no US
TV na us
TV 0.01 na US
0V0.01 no US
TV u.
TV a US
0.000 US
TV 0.01 US pay-phone O 066339
8.595
Chimpanzee 0.99864
PHOTDS
30 % retained 100 retained
30 ratained
100 95 retained
5% retained
100 % rotained
30 recained
100 %. retained
Figure 11: Results of randomizing the last layer of a ResNet-18 model on ImageNet data for the pro- cedure described in Section US indicates (56, 56) map was learned and upsampled to (224, 224). We find the maps of this randomized network are less visually coherent than the analogous maps of a pre-trained model.
K-1
K-2
K=2
Km4
Km
pay-phone 0.96339
aircraft carrier 0.995
chimpanzee
OTTOS
CTTDS
PHOTOS CRYCO
30 % retained 100 % retained 30 % retained 100 % retained 30 % retained 100 % retained
Figure 12: Effeet of ensembling Partial statistical assignments validating the correct label of Ima- geNet and ResNet-18 images as we vary K, the number of maps. Details in Appendix C.3.1
Table 10: Performance of our method on ImageNet and ResNet-50 model and some baselines on various intrinsic saliency metrics proposed in prior work. We find that while both masks (learned with and without TV) have very good performance on the insertion metric, the mask learned with TV has much better performance the saliency metric. The deletion metric is uninformative in most cases, since most methods are as good (or worse) compared to a random mask.
Gradient @ Input
Our method
Our Method
Real Time saliency
Random
(Arv 0.01)
(Arv 0)
Deletion 1 Insertion (blur) t
0. 1451
0.1832
0.2067
0.1851
0.1843
0.5434
0.8363
0.8673
0.6857
0.3562
Insertion (grey)
0,3716
0.7401
0,8857
0.4873
0.1835
Saliency Metric
0.2549
0.2019
0.2604
0.2943
0.2584
D ADDITIONAL BACKGROUND INFORMATION
D.1 SALIENCY METHODS
We give a partial of extant saliency methods here. We broadly categorize explanations into three categories: Back-propagation based explanations, axiomatic methods, and masking methods.

---

Under review as a conference paper at ICLR 2022
Ground Truth Label,
Ground Truth Label, Noise
Second Best Label, Grey
Second Label, Noise
(scale, TV)
(scale, TV)
0,000)
0.)
11 8.31
model outpuc probabiitt
mopoo outpue probability
Saljgeccoui andano mopel
44. 0.011
64.0.0
model outpun biliy
0 6.0
66 0.53
(scale, TV)
(scale, TV)
13, 8.81
0.0
11. 8.81)
11.0011
8.01
fraction of pixels used
fraction of pixels used
fraction of pixels used
fraction of pixels used
Figure 13: ImageNet data. Left: AUC curves as fraction of pixels used varies from 0 1.0 for ground truth label when replacing with grey. Center Left: AUC curves as fraction of pixels used varies from 0 1.0 for ground truth label when replacing with other images. Center Right: AUC curves as fraction of pixels used varies from O to 1.0 for second best label when replacing with grey. Right: AUC curves as fraction of pixels used varies from to 1.0 for second best label when replacing with other images.
Backpropagation based explanations shape credit it propagated backwards through the neural network according to certain rules. These approaches include Layerwise Relevance Propagation (Binder al. 2016) which satisfies completeness, Reet-Grad which thresholds internal neuron activations (Kim et al. 2019). and DeepLIFT which satisfies the summation to delta rule.
Axiomatic methods. Axiomatic methods decompose the ouput (typically the logit) according to certain axioms like fairness in Shapley based methods SHAP (Lundberg & 2017 and concept- SHAP Yeh et al. 2020). We also include gradient based approaches like Gradient Pr (Baehrens) et al. 2010) which calculates the partial derivative of the logit with respect to the input. Gradient Input (Shrikumar et 2017
28 de zz which elementwise multiplies the gradient explanation by
the input, and Grad-CA.V Selvaraju et al. 2019) which takes the gradient of the logit with respect to the feature map of the last convolutional unit of DNN. Smooth-Gran (Smilkov et al. (017), which averages the Gradient @ Input explanation over several noisy copies of the input where 7 is some Gaussian. The previous methods are intrinsic in the sense that they aim to explain the model decision. The last category of saliency maps, namely masking methods, also aim to plain the model decision, but they frequently aim to do so a way that is interpretable by a human. Contrastive methods, such as contrastive layerwise propagation Gu al. (2018), also modify LRP by constructing class specific saliency maps, with the goal of object localization, i.e. in an image of elephant and zebra, the saliency map for elephant should have high overlap with the elephant, and similarly for the corresponding map for zebra.
17,
Masking Methods. Masking Methods are often evaluated using a pointing game WSOL metric, which measures overlap with human labeled bounding boxes or explanations. These masking meth- ods include techniques based on averaging over randomly sampled masks (Petsiuk et al. 2018), op- timizing over meaningful mask perturbations (Fong & Vedaldi 2017). and real time image saliency using a masking network (Dabkowski & Gall (017)). Pixels that have been removed from the image by the mask may be replaced by greying out, by Gaussian blurring as in Fong & Vedaldi (017).
or with infillers such as CA-GAN (Yu et al.) 2018) used in Chang et al. (2018), or DFNet (Hong eet al. 2019). De Cao al. (2020) find masks using differentiable masking. Taghanaki et al. (2019)
introduce a method that results in more accurate localization of discriminatory regions via mutual information.
Pruning and information theory Khakzar et al. (019) improve attribution via pruning. Schulz et al. (2019) improve attribution by adding noise to intermediate feature maps.
Saliency and Boolean Logic. Previous work has also drawn connections between saliency and notions in logic. [gnatiev et al. (209h)) relates saliency explanations and adversarial examples by
23

---

Under review as a conference paper at ICLR 2022
generalized form of hitting set duality. Ignatiev al. (2019a) develops constraint-anootte solution for computing explanations for any ML model. Macdonald et al. (2019) develop rate distortion explanation for saliency maps and prove hardness result. Mu & Andreas (2020) find a procedure for explaining neurons by identifying compositional logical concepts. /Zhou et al. (2018) describe
network dissection, which provides labels for the neurons of the hidden representations. We are unaware of frameworks like S
Arguments about saliency. For discussion including pro/cons of various methods some starting points are Seo et al. (0118) Fryer et al. (2021) Gu et al. 20188 Sundararajan & Najmi (2020).
Phang al. 2020) We describe separately the masking procedure used by Phang al. (2020). They begin by taking a pretrained model on ImageNet. The masker has access to the internal rep- resentations of the pre-trained model, and tries maximize masked in accuracy and masked out entropy. They do not provide the ground truth label to the masker.
D.2 SALIENCY EVALUATION METHODS
Saliency evaluation methods attempt to evaluate the quality of a saliency map. Many interpret the heatmap values as priority order of saliency. Extrinsic evaluation metrics include the WSOL metric, which aim to measure overlap of the saliency map with human annotated bounding box and the Pointing Game metric proposed by Zhang et al. (2018) in which a pixel count as hit it lies Hils
within bounding box and miss otherwise, and the metric is Ilits #Misses
Other more intrinsic
methods include early saliency evaluation techniques like MorF and LerF |Samek et al. (2016),
which involve removing pixels either in the order of highest importance lowest importance and observing the area of the resulting curve. Insertion and Deletion Games of [Petsiuk et al. (2018)
uses this too. The deletion game measures the drop in class probability important pixels
removed, while the insertion game measures the rise in class probability as important pixels are added. (Our AUC discussion in Section B relates this.) Remove and Retrain (ROAR) is saliency evaluation method proposed by Hooker et al. (2019). Input features are ranked and then removed according to a saliency map. A new model is trained on the modified training set, and larger degradation in accuracy on the modified test set compared to the original model on the original test set regarded as a better saliency method. (NB: retraining makes this a non-intrinsic method.) Previous work has also introduced datasets specifically designed to test saliency methods. BAM Yang & Kim (2019) creates saliency maps by pasting object pixels from MSCOCÓ Lin et al.) (014) into scene images from MiniPlaces Zhou et al. (2017). The Saliency Metric proposed by Dabkowski & Gal (017) thresholds saliency values above some a chosen on a holdout set, finds the smallest bounding box containing these pixels, upsamples and measures the ratio of bounding box area to model accuracy on the cropped image, s(a. p) = log(mn max 0.05)) log(p) where a is the area of the bounding box and is the class probability of the upsampled image.
Controversies.
There is extensive discussion of validity of saliency evaluation methods;
e.g., Brunke et al. (20suuuetsiuk et al. (2018).
D.3 SALIENCY COMPUTATIONS AND UNDERLYING MEANINGS OF SALIENCY
For simplicity this discussion assumes the datapoints are images and the classifier is deep net. The heatmap in the saliency method is trying to highlight the contribution of individual pixels to the final answer. This is analogous to how human may highlight relevant portions of the image with a plan. (Classic saliency methods in vision are inspired by studies of human cognition.) Saliency methods operationalize this intuitive definition in different ways, and we try to roughly categorise these as follows.
Variational interpretation. These interpret saliency in terms of effect on final output due to change in single pixel -captured either via partial derivative of output with respect to pixel value (i.e.. effect of infinitesimal change), or via change of output when this pixel is set to 0 or to a ran- dom (or "gray") value. Examples include Gradient, Gradient c Input Shrikumar al. (017)) Occlusion
Credit attribution guided by gradient. These use the gradient guide the assignment of saliency values. The gradient is interpreted as propagating values from the output to the input layer, and the values partitioned/recombinee internal nodes of the net following some conservation princi-
24

---

Under review as a conference paper at ICLR 2022
ples. A key goal is to ensure completeness, which means that the sum of the attributions equal the logit value. Examples include LRP. DeepLIFT Shrikumar et al. 2017). Rect-Grad Let ag be the activation of some node in layer 1, and R +1 be the backpropagated gradient up to a.. Rect-grad replaces the vanilla chain rule, R = laa > 0] with the rule that R = 1 [ a; > r] for some threshold r. Hence, during a backward pass preference is given to nodes with large margin.
Ensembling on top of above two ideas. Ensembling methods combine saliency estimates over multiple inputs an an attempt to reduce noise in the final map. Examples include Smooth-Grad, Occlusion based methods, etc. We also include Shapley Values in this list.
The Shapley value aims to fairly distribute credit among a coalition of N players. In the context of image saliency, each coordinate of the image input may be seen as player, and the Shapley value computes EscN Mii) 1 (v(S) {i}) - -())) can be interpreted as the marginal contribution of player over all possible orderings of the coalition. In this sense, it can be seen as an ensembling method, as it averages over all possible random permutations.
n!
Analysis of saliency methods. Previous work has analyzed ensembling methods like Smooth-grad, and found that it does not smooth the gradient [Seo et al.] (2018). They conclude that Smooth-Grad does not make the gradient of the score function smooth. Rather Smooth-grad is approximately the sum of a standard saliency map and higher order terms and the standard deviation of the Gaussian noise. It has also been found that Shapley values, despite having uniqueness result, can differ in the way they depend on the model, data, etc Sundararaaan & Najmi (020). |Fryer et al. (202T) highlight several nuances that should be taken into account when considering Shapley values. They introduce Shapley values as averaging over submodels, and note that "the performance of a feature across all submodels may not be indicative of the particular performance of that feature in the set of optimal submodels." They provide specific cases where satisfying the axioms of Shapley values works against the goal of feature selection.
E CLARIFYING BENEFT OF TV REGULARIZATION
This section illustrates Definitions 3.1 and 3.2 using linear classifiers. It also shows how TV regu- larizers help ensure soundness even in this setting.
Let S be a dataset of labeled data (a. y) where the inputs are of unit norm and labels are binary, i.., || la 1, y {+1). The model in question is a linear classifier (ax) == sgn( (w, a)) parameterized by the weight vector w E gá-1, and it achieves the perfect accuracy on the set S with margin 7 := min( y (w, > We assume that the coordinates of æ and are uniformly
bounded by Va i.e., ||=||>> S Vi ||w +| S va
(10 can be changed to any other constant).
Let P be the input modification process that sets all non-salient pixels to O. We are interested in masking explanations with deterministic salient set, i.e., A assigns probability 1 to some salient set S. According to Section 3.1 masking explanation validates label if EswI(e,)) [=)]==]]]]] is high. A simple calculation shows that this expectation equals |a Ecco w22 > and thus the goal to find S so that a Eies u0,74 > 0.
As we do not consider the full salient set informative, we are interested in masking explanations with size constraint |S|| = L for some 1 S < d. There is a simple saliency method that achieves this goal: Given an input and label a E {1 sort the coordinates according to awaz; and take the highest coordinates as the salient set S.
It is easy completeness, However, this method does not satisfy soundness: a salient set S with a Σ WiZ; > 0 can also be found for a + y!
that this method always produces S with a Eies w,', 0. Letting y proves the
Now we see how the TV constraint helps to ensure soundness (with good probability). A vec- tor can be seen as a 1D image, and the TV of a salient set S can be defined by TV(S) = Ed11 | - [[41Es For simplicity, we consider salient sets with TV at most 2. This means is just an interval. Given the size and TV constraints |S] = L, TV(S) < 2, it is easy to come out with the following saliency method: search over all the intervals of length T. and if an interval
25

---

Under review as a conference paper at ICLR 2022
S satisfies Lies W > O, return it the salient set. Fortunately, this method does satisfy both completeness and soundness, which is justified by the following theorem.
Theorem E.1. For any (ax y) E S. if we shuffle the coordinates of w and those of x according to the same random permulation, then
1. For L = -[ log }) with probability 1 - 6, there is an interval of length L that validates yi
2. For = = log )) with probability 6, no interval of length L can validate y.
Proof. Let S be any fixed interval of length L, then the distribution of Σies Was; is identical to the distribution of the sum of i.i.d. random variables drawn from {wiv1,..... Wggaa without replacement. Note that dyw1 ................................ dywayy are d numbers with mean 7, and their absolute values are bounded by 10 = O(1). By Chernoff bound,
1 dyw,r, - - S -
Σ iES
Pr L
Set e " ensures that Eies W;2; > 0 with probability e ((1) We can fix any interval S to prove Item Taking union bounds over intervals of length L, the probability of existing an interval of length L that validates yy should be no greater than Σ|s|= ((e(L) L) I < L) which proves Item 2.
This shows that such masking explanations make sense humans: if the model predicts y, then we can find an interval of length (2) so that computing the inner product only in that interval leads to the same prediction; otherwise if the model does not predict y, such interval cannot be found. Thus sufficient to convince humans that the model predicts y by only revealing the existence of such interval and the coordinate values in it.
Although the example given this section is simple, the conceptual message clear: saliency methods may not guarantee soundness in itself, but adding regularity constraints such as TV can mitigate this soundness issue effectively.
26


Reviews:
[{"review_id": "S46SPeAstKu", "reviewer": "Reviewer_Ge5Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper brings an interesting concept, i.e., soundness, to the model explainability research. There, however, remains a big motivation gap. Why do we need soundness? Given that we are interested to understand what causes the prediction output of a black-box model, why the source of wrong predictions shall matter?\n\nThe paper proposes a new score to evaluate the quality of heatmaps. The score is well designed and suited for the proposed approach. It, then, compares some methods with the proposed approach using the same evaluation score tailored for it's proposed algorithm. Thus, one could wonder, why are we lacking a careful comparison and discussions with other scores? Is it fair to compare multiple methods on a score tailored for one?\n\nOn the practicality of the method, based on the note in the paper about back-up plan of running other methods, is there any statistics of how often this could happen in the experiments? And how much this could be a risk to the applicability of the approach?\n\nOverall, the paper is interesting. One caveat is that it refers to the supplement so often. It would be nice to read a paper that is self-contained. ", "summary_of_the_paper": "The paper tackles the problem of model explainibility from the perspective of heatmap-based visualization approaches. It argues that achieving a good heatmap to explain the model decision is possible by considering two notions, completeness and soundness, borrowed from logic. In fact, the completeness condition is what already considered during the formulation of similar methods. Thus the novelty lies on studying and analyzing the usefulness of soundness. The paper, thus, proposes a metric to evaluate both completeness and soundness that is used to learn a mask that explains the model prediction.", "summary_of_the_review": "The paper is providing good insight about model explainability and proposes interesting concepts. Reading the paper, it seems the presentation, storyline and experiments could be improved. I find it a bit too much scattered between the supplement and main paper.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J-bkMCLaHKk", "reviewer": "Reviewer_4yoz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths：\n1.\tThe method proposed in this paper does not need to introduce additional manual labeling or training costs, which is an interesting and worth exploring perspective.\n2.\tThe paper provides a lot of intuitive visualization results, which is helpful to guide readers' understanding.\n\nWeaknesses:\n1.\tWe believe that the example in Figure 1 is not comprehensive and does not explain the difference between completeness and soundness well. We want to know what will happen if the saliency maps of the category labels do not overlap?\n2.\tSection.3 mentioned that this paper uses the images of the training set to fill the pixels outside the saliency map area S. Wouldn't other objects in the background area interfere with the result unpredictably? Perhaps more explanation should be given here.\n3.\tFrom the results in Table 1, compared with the method in Phang et al.(2020), the proposed method has no special advantages.\n4.\tThe data in Table 2 confuses me. The completeness of different methods is very different, but the soundness is very close, especially the second column of data. How should we understand this result?\n", "summary_of_the_paper": "This paper discusses the completeness of saliency maps and proposes the concept of soundness. Based on this, they propose consistency score to predict the possibility that the quality of the saliency map is consistent with the classification probability. Besides, the author uses these definitions to implement the intrinsic evaluation of masked-based saliency methods.", "summary_of_the_review": "The concept of the proposed method is innovative, but many details of the method have not been explained clearly. In addition, more persuasiveness is needed in the experimental performance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8-FfxXbLVPh", "reviewer": "Reviewer_LG7F", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n\n- The introduction of soundness and consistency score for use in evaluating saliency maps is a good idea and is explored in several experiments.\n- The introduction did a good job of setting up the distinction between saliency methods and saliency evaluation metrics and laying the groundwork for the paper\n- The related work gives good background on related saliency methods and evaluation methods; however, the paper could include more context for the past work and its relationship to the current paper, especially in the \"arguments about saliency\" and \"controversies\" sections. It would be helpful to have a brief description of what the main points of controversy are and perhaps a note about how this work relates to those arguments. \n- The paper proposes soundness as a quantitative way of discovering artifacts in saliency-based approaches and back this up empirically r.e. TV regularization. \n\n**Weaknesses:**\n- The impact of the work is unclear. The new saliency method is similar to prior work and has mixed results when evaluated against other baselines (Table 1). The introduction of new saliency metrics has the potential to be useful, but I do not understand what new information soundness or consistency score provide that are missing from other measures; furthermore, they are expensive to compute (Section 6). \n- I found the descriptions and equations for completeness and soundness--Definition 3.2 on page 4--to be difficult to understand. I think it would be useful for the paper to include an intuitive description of the equations, how they interacted with each other, and whether alpha and beta should be minimized or maximized. For example, it would be nice to follow up the definition with an intuitive explanation of these metrics to help the reader understand what they are capturing. It may also be preferable to break up these two concepts into separate definitions. \n- This is my current understanding of completeness and soundness: alpha provides a lower bound and 1/beta an upper bound on the ratio of the probability assigned to class a on the masked vs the unmasked input. It is desirable to have both alpha and beta close to 1 (however, additional constraints must be applied to avoid the trivial case where the whole image is considered salient). Alpha and beta close to 1 imply that for all classes, the model's scores on the masked input are similar to its scores on the unmasked input. Is my understanding correct? If not, could the authors please clarify? If so, it may help to include such a discussion in the paper. \n- The paper should justify why completeness is desirable. There is some discussion of this in the introduction, which ties increased soundness to reduction of artifacts in the mask itself. However, there are situations where it would make sense for different masks to significantly change a classifier output. Consider the case where a cat/dog classifier is trying to classify an image with both a cat and a dog in it, then masking out the dog should increase the probability of cat. Similarly, a non-robust model might output very different results in response to different masks. As such, does completeness measure the goodness of the saliency method or the robustness of the model? \n- For the experiments described in Section 6.1, are the authors trying to validate the usefulness of their metric or prove the benefit of their saliency method? The proposed saliency method does not outperform other baselines for most metrics. \n\n**Additional small notes:**\n- The approach was inspired by logical proof systems, which is an interesting source of inspiration! But the connection is not clear--perhaps adding a brief description of how soundness is derived from logical proof systems or some references on this point would be helpful. \n- It would be helpful to readers if the contributions were stated more clearly. The introduction contains a bulleted list of \"Other contributions\"; I recommend changing this to a list of the top contributions.", "summary_of_the_paper": "This paper addresses masking-based saliency methods for providing visual model explanations. It defines several new metrics for evaluating saliency methods: \"completeness\", \"soundness\", and \"consistency score\". It proposes a saliency method that is a variant of prior work. The paper shows through experiments that TV regularization improves soundness of saliency-based explanations.   ", "summary_of_the_review": "The contributions of the paper (the proposed saliency method and the new saliency metrics) are modest and their significance is not clearly explained in the paper or supported by experiments. The paper should explain the concepts of completeness, soundness, and consistency and justify why they are desirable. The experiments should support the main contributions: they should show that the proposed metrics are useful and that the proposed saliency method outperforms baselines on these metrics. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-14HAI3CCO3", "reviewer": "Reviewer_nxmx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper is well structured, and it is easy to read.\n\n- The proposed saliency explanation approach is simple and general. It leads to extensive results on three datasets: CIFAR-10, CIFAR-100, ImageNet. \n\n- The analysis of TV regularization is novel and interesting.\n\n\nWeaknesses:\n\n- While I generally agree with the idea of evaluating and improving the soundness of saliency methods. The premise that the top-1 prediction is considered to be the only correct label and the second prediction is considered to be incorrect, is problematic. Indeed, a well-trained model can achieve high classification accuracy on iconic images and the top-1 prediction usually matches the ground truth. However, in practice, the top-1 prediction is likely to be wrong （as in Table 1), or there are multiple labels that can be correct (as in Figure 4). In the case of Table 1, I am not sure whether the wrong model prediction or the ground truth is considered to be correct. In the case of Figure 4, it is not clearly stated which is the correct label, elephant or zebra or both, and what the completeness and soundness scores of the compared methods are. To avoid confusion, the definition of correctness needs to be clarified. Again, determining what labels are correct or incorrect is critical, which may require a thorough discussion.\n\n- Given this unclear definition, determining what labels are correct or incorrect will be critical. The metrics defined in Eq. (7)-(8) may need to be revised to consider multiple correct/incorrect labels. \n\n- The experiments show that TV penalty or upscaling generate more smooth saliency maps, which leads to decreased completeness and improved soundness. Can this observation be generalized to other smoothing techniques, such as a simple Gaussian filter? If the effect is mainly attributed to the smoothness of saliency maps, this approach looks a bit trivial. Other baseline methods, such as 1) a single Gaussian blog placed in the image center, 2) a randomly sampled saliency map from other images, or 3) a bottom-up saliency map independent of the labels, should be added, because the current pixel-wise randomization approach does not consider the distribution of image data.\n\n-  It would be interesting to quantitatively evaluate the performance of saliency methods on the ImageNet localization task, and to perform a correlation analysis between the completeness/soundness scores and the localization accuracy.\n\n", "summary_of_the_paper": "This paper introduces two intrinsic evaluation criteria for saliency-based model explanation: completeness and soundness. While completeness indicates a saliency method's ability to justify the correct label,  soundness requires the method to be unable to find a masked input that significantly increases the probability of an incorrect label. A consistency score is introduced to simultaneously evaluate both completeness and soundness. Further, taking into account the soundness factor, a saliency method is proposed to maximize the probability for a given label, rather than all labels. Experiments demonstrate that the heuristic TV regularization method can help with soundness, as suggested by high consistency scores.\n\n", "summary_of_the_review": "While I appreciate the motivation and the overall idea of measuring soundness of saliency, the definition of correctness and soundness is a bit unclear, and the effectiveness of the proposed method is not sufficiently demonstrated by the experiments. Considering the strengths and weaknesses of this paper, I don't think it is ready for publication at this moment. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission tackles the problem of model explainability from the perspective of masking-based saliency methods. 
Several metrics are proposed for evaluating saliency methods including  a new « soundness »  concept.
Experiments using a consistency score to simultaneously evaluate completeness and soundness are provided.  

Most of the reviewers were not convinced by the approach and have raised several issues. 

After rebuttal and despite the interest in the introduction of the concept of « soundness »  to better explain model decision, the current proposition needs to be improved. In particular, the interest of this soundness concept does not bring out, many details of the method are not clear enough and the effectiveness of the proposed measure is still questionable. It would be interesting that the authors consider the R’s comments as the ones for additional  experiments to demonstrate the relevancy of their contribution.
Decision: Reject