New Definitions and Evaluations for Saliency Methods: Staying Intrinsic, Complete and Sound
Authors: Arushi Gupta
Nikunj Saunshi
Dingli Yu
Kaifeng Lyu
Sanjeev Arora
Venue: NeurIPS 2022 Accept (2022)
Keywords: saliency
saliency methods
saliency evaluation
soundness
sanity checks
interpretability

Abstract: Saliency methods compute heat maps that highlight portions of an input that were most important for the label assigned to it by a deep net. Evaluations of saliency methods convert this heat map into a new masked input by retaining the $k$ highest-ranked pixels of the original input and replacing the rest with "uninformative" pixels, and checking if the net's output is mostly unchanged. This is usually seen as an explanation of the output, but the current paper highlights reasons why this inference of causality may be suspect. Inspired by logic concepts of completeness & soundness, it observes that the above type of evaluation focuses on completeness of the explanation, but ignores soundness.  New evaluation metrics are introduced to capture both notions, while staying in an intrinsic framework---i.e., using the dataset and the net, but no separately trained nets, human evaluations, etc. A simple saliency method is described that matches or outperforms prior methods in the evaluations. Experiments also suggest new intrinsic justifications, based on soundness, for popular heuristic tricks such as TV regularization and upsampling.

New Definitions and Evaluations for Saliency Methods: Staying Intrinsic, Complete and Sound
Arushi Gupta" ' Nikunj Saunshi" Dingli Yu"!, Kaifeng Lyu', Sanjeev Arora'
Princeton University
farushig, nsaunshi,
dingulsii,dnnliyll dingliy, klyu, arora] princeton.edu
"Denotes equal contribution
Abstract
Saliency methods compute heat maps that highlight portions of an input that were most important for the label assigned to it by deep net. Evaluations of saliency methods convert this heat map into new masked input by retaining the highest- ranked pixels of the original input and replacing the rest with "uninformative" pixels, and checking the net's output is mostly unchanged. This is usually seen an explanation of the output, but the current paper highlights reasons why this inference of causality may be suspect. Inspired by logic concepts of completeness & soundness, it observes that the above type of evaluation focuses completeness the explanation, but ignores soundness. New evaluation metries are introduced to capture both notions, while staying in an intrinsic framework-i.e using the dataset and the net, but no separately trained nets, human evaluations, etc. A simple saliency method is described that matches outperforms prior methods in the evaluations. Experiments also suggest new intrinsic justifications, based on soundness, for popular heuristic tricks such as TV regularization and upsampling.
1 Introduction
Saliency methods try to understand why the deep net gave a certain answer on a particular input, and are important component of explainability, fairness, robustness, etc., in deep learning. This paper restricts attention to the (large) class of saliency methods that return an importance score for each coordinate of the input often visualized heat map which captures the coordinate's importance to the final decision Early saliency methods used an axiomatic system of "credit attribu- tion" to individual input coordinates. using backpropagation-Ile
methods [5]
36 and cooperative
game theory, such as Shapley values (27] 45) See Section 2 and Samek et al.
3 for discussion of
strengths and limitations of such methods. More recent methods try find heat maps that are largely concentrated on smaller set of pixels/coordinates and are discussed further below.
There exists ecosystem of evaluation metrics evaluate saliency methods. Evaluations can be extrinsic, involving human evaluation [. and comparison to certain ground truth explanations [47] We restrict attention intrinsic evaluations, which computations involving the itself and the heat/saliency map - but no human evaluations or evaluations that involve training new deep nets. Popular intrinsic evaluations include saliency metric and insertion & deletion metrics K0
frequent idea in intrinsic evaluations (see Section S for references) is to create a new composite input or sequence of such inputs using the heat map and the original input, and to evaluate the original net on this composite input. For example, if is a binary vector with I's the coordinates with
Heat maps suffice for recognitiomcclasiffcaiio tasks; other tasks may require more complex explanations.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Original Image
Masked Image for Cat, Masked image Ship
Original Image
Masked Image for Frog Masked Image for Bird
Figure I: Masked CIFAR-IO images generated by our procedure with TV regularization. "Artifacts" exist for masks generated for incorrect labels; more examples can be found in Figure DT in Appendix. The base model outputs the correct label on the original image (ship and bird resp.) with probability at least 0.99, and assigns probability at most 10 for the incorrect label (cat and frog resp.). With the generated masks, the AUC metric (defined in Section B.T for the correct label remains high (around 0.94 and 0.90), which corresponds to completeness, but AUC metric for the incorrect label rises tremendously (around 0.18 for cat mask, 0.71 for frog mask.) This suggests violation of soundness.
the highest values in the heat map, then x & M (with @ denoting coordinate wise multiplication) can be viewed masked input where only & of the original coordinates of x remain. It is customary to replace the zeroes in this masked input with "uninformative" values, which we refer to as gray pixels. This masked input is fed into the original ne to check if the net outputs the same label as on the full input - and, if so, one should conclude that the unmasked coordinates of x were salient to the net's s output. Not surprisingly. recent methods
[ 9 B1 for finding heatmaps
objective that tries to methods.
directly optimize performance on such evaluations, greatly outperforming older
Should we trust such baask explanations? At first sight, this question may appear naive. In the descridd masked input x @ M, some le pixels were copied from the original image and the remaining pixels were made "gray." If the original net outputs essentially the same label as it did on the full image, does this not prove that the portion that came from the mask M must have caused the net's s output on the full image? The answer is "No", because the positions of the gray pixels can carry some signal.
Figure shows that in standard datasets. it is possible to create masks out of most inputs such that the masked input can cause the net to assign high probability for a different label than in its original prediction. Prior works referred to this phenomenon as "mask artifacts" and suggested using total variation (TV) regularization to mitigate this issue. But Figure shows TV regularization is not full solution to masks that cause the net to (incorrectly) flip the answer.
Soundness needed. The masked input method is trying prove that certain portion of the image "caused" the net's output, Usually logical reasoning must display both completeness (i.e., all correct statements are provable) and soundness (incorrect statements cannot be proved). Checking that the net's output is essentially unchanged with the masked input is akin to verifying completeness.
But this by itself should not be convincing because, mentioned, it is possible to use the method to find a different mask to justify another (i.c., wrong) label. Soundness would require verifying that the same saliency method cannot be used to produce masked inputs that make the net output different label. Together, completeness and soundness ensure that the evaluation of the masked inputs produced using various labels approximately tracks the model's probability of assigning the label (see Section B.2)
Main Contributions of this paper:
Section 3.2 draws connections to logical proof systems and formalizes completeness and sound- ness in the context of saliency methods. These are intrinsic evaluations of saliency without appealing to human judgements (e.g.. whether or not machine-generaue masks contain artifacts). Our notion of completeness and soundness requires saliency methods output a map for every label and not just the model prediction.
Section 4 revisits methods that learn masks through optimization. Incorporating soundness, we propose simple optimization method that maximizes the probability for any given labe rather than only the label deemed most likely by the net. key design choice of this method is the pixel
"This raises potential issue of distribution shift because the net never trained masked inputs. In practice, especially for image data, trained nets continue to work fine on masked inputs. 'Computation cost proportional to the number of labels is only incurred while designing/evaluating the saliency method but not deployment time. Another alternative is verify soundness condition for top labels.

---

replacement strategy during training: non-salient pixels are replaced with pixels of a random image, instead of using gray pixels, blurring or counterfactual models.
Experiments (in Section 2 estimate completeness and soundness scores for various methods. Our simple method does well on completeness and soundness as one would expect, but the slight surprise is that it performs comparably to prior masked-based methods on earlier saliency evaluations well, suggesting that soundness can be achieved without paying much in completeness. Heuristic choices in existing mask-based methods such as TV regularization and upsampling, which were hitherto justified extrinsically (i.e., they make masks look natural to humans), receive an intrinsic and precise justification: they improve soundness. A theoretical result in Section 5 complements this experimental finding by showing that TV regularization helps saliency methods even in a simple linear classification setting, not just for vision data.
2 Prior approaches
We relegate a more thorough description of prior work to Appendix - but mention some common saliency and saliency evaluation methods here. Saliency methods aim to explain a model's decision about an input. Saliency evaluation methods aim to evaluate the goodness of a saliency method.
Saliency methods include backpropagation based approaches such as Gradient C Input [3
iGOS++ (20). Robust Perturbations (22). LRP (5), GradCAM [66. Smooth-Grad BV Another line of work is masking methods which include techniques based on averaging over randomly sampled
masks (30) optimizing over meaningful mask perturbations M.. and real time image saliency using
masking network [9] Pixels that have been removed from the image by the mask may be replaced by greying out, by Gaussian blurring as in [., or with infillers such as CA-GAN 46 used [ [2
or DFNet 14. The pixel replacement strategy we used is closely related to hot deck imputation
where features may be replaced either by using the mean feature value (analogous to replacing with grey) sampling from the marginal feature distribution (analogous to replacing with image pixels sampled from other training images). Some prior work 77] has found that mean imputation does not significantly affect model output on the beer aroma review dataset. On Imagenette, by contrast, we found that replacement strategy can matter (See Figure 3 De Cao et al. [10) find masks using differentiable masking. Boolean logic is another approach for saliency methods
49
A Arguments about saliency. Discussions about the methods and the meaning of saliency appear, Some reveal situations where Shapley axioms work against feature selection or where Shapley values may be calculated in conflicting ways 12 Others question efficacy of saliency methods that add noise [77 making explanations non class discriminative [
among others, in 37 12
13 40
Saliency evaluation methods. Extrinsic evaluation metrics include the WSOL metric, and Pointing Game metric proposed by Zhang et al. [47] and ROAR [IS]. Other more intrinsic methods include early saliency evaluation techniques like MorF and LerF
33),, Insertion and Deletion game proposed by Petsiuk et al. [EC] which involve either inserting pixels in order of most importance or deleting pixels in order of most importance. BAM
44 creates saliency maps by pasting object pixels from MSCOCO 25 The Saliency Metric proposed by Dabkowski and Gal [ thresholds saliency values above some a chosen on holdout set, finds the smallest bounding box containing these pixels, upsamples and measures the ratio of bounding box area to model accuracy on the cropped image, s(a. = log(max((, 0.05)) - log(p) where a is the area of the bounding box and p is the class probability of the upsampled image. Adebayo et al. I proposed "sanity checks" for saliency maps. They note that if a model's ' weights are randomized, it has not Iearned anything, and therefore the saliency map should not look coherent. They also randomize the labels of the dataset and argue that the saliency maps for a model trained on this scrambled data should be different than the saliency maps for the model trained on the original data. We study the effect of model layer randomization on our method in Appendix section D.4 and find that randomizing the model weights does cause our saliency maps look incoherent. Tomsett et al. 4% also discover sanity checks for saliency metrics, finding that saliency evaluation methods can yield inconsistent results. They evaluate saliency maps on reliability, i.e. how consistent the saliency maps are. To measure method's reliability, because access to ground truth saliency maps are not available, they use three proxies 1) inter-rater reliability, i.c. how whether saliency evaluation metric able to consistently rank some saliency methods
*Code for computing completeness, soundness, and our masking method available at https://github com/ conn/agu//soundnesss _etesessssll_l___eeeee git
3

---

above others, 2) inter-method reliability, which indicates whether a saliency evaluation metric agrees across different saliency methods, and 3)nntennal consistency reliability, which measure whether different saliency methods are measuring the same underlying concept.
Discussions. Brunke al. o show that perturbation methods are sensitive baseline and Petsiuk et al. B00 point out that human centric explanations (based on bounding boxes) may not reveal why the model made certain decision. Our notion of intrinsic saliency method outputs the mask but introduces soundness to validate it.
3 Masking explanations and completeness/suunnnes
The goal of a saliency method to produce explanations for the outputs of a given deep net f. We think of saliency method (Definition 3.1) producing explanations providing a proof of the statement f outputs the label a for input x. Inspired by logical proof systems, propose that intrinsic evaluations for saliency methods should test for both completeness and soundness, and the rest of the section provides definitions for these notions.
heat map in this case as
3.1 Saliency methods and AUC metric
For net f. let fxx a) denote the (post-softmax) output probability of the net input x for label We are interested in saliency methods that output heatmap with a scalar per coordinate of the input. Definition 3.1 (Saliency Method), A saliency method M is an algorithm that takes 1) a deep net f. 2) an input x and 3) a label a and produces a heatmap m MM xr a) E (Rdim(s))
Note that in the above definition, .M can access any part of f iinluudig intermediate layers and gradients. Examples of methods that can be interpreted as returning heatmaps are mask-basee methods [ g 8
31
that output heatmaps in [0, [jdim(z), backpropagation-likk methods [5] (6).
Shapely values 2 55 and Gradient @ Input.
Given such a saliency method, an important question is how to evaluate the quality of the "expla- nations" or heat maps it produces. A common evaluation idea is to use the behavior of the net f on a "masked" input-onile
-0---on,, only the top few pixels based on the heat map in question are retained, while the oth- ers are hidden appropriately. Inspired by this idea we define evaluation metrics for saliency methods. We abstract the step of retaining the top few pixels into an input modification process, defined below.
Definition 3.2 (Input Modification Process T). Let I be potentially randomized procedure that takes an input x and a heatmap m and generates a modified input G. i.e. à ~ [( m).
This input modification process defines base metric that the starting point of the completeness and soundness metrics that will be defined later.
Definition 3.3 (Base Metric). Let & be an input (image), a a label, m a heatmap and T an input modification process. Then, base metric is a function gx,, a, m) Ez-T((m (F, a)), where the neural net of interes
The base metric measures the expected output of f acting modified inputs 7 and label and different choices for lead to different base metrics. Some examples of T that stay the intrinsic framework are: graying out the pixels outside some subset S of m. replacing them by pixels from a Gaussian blurring of x [Л]. Input modification through I' also shows up in saliency methods, including our proposed method in Section 4 discuss prior choices of I in that section. For our evaluation metrics, however, we choose T that grays out remaining pixels.
The popular Mrea-Unddr-theeCuure (AUC) evaluation metrics 330 of saliency methods can, in fact, be reinterpreted terms of our base metric. We state the insertion game and its AUC below, since forms the basis of our subsequent completeness and soundness metrics, and then define the AUC metric as an instantiation of Definition [
Definition 3.4 (AUC of insertion game). For to dim take the top s pixels as per m, and plot the probability f(z, a) given by model to label a on the input x, where the top pixels of x are retained and remaining pixels are assigned a default gray value. Return the area under the curve.
In general replace f with any target function / that depends on f

---

desk (Ours. 0.01)
desktop computer rOurs. TV 0.01)
desk (Ours. 0,001:
desétop computer (Ours. 0.001)
desk (Phang al.)
desktop computer (Phaing et all)
original Image
AUC: 0.65
AUC, 0.33
43
AUC: 0.08
txddesiopp Semputer/-013
Figure ImageNet image and masked versions for labels desk and desktop computer. Heatmaps gen- erated by method with λrv 0.01, \rv 0.001 and Phang et al. [3]]ss method, respectively.
Definition 3.5 (AUC metric), We denote by (Ave(r, a, m) the base metric when a modified input I ~ [[z m) is obtained as follows:
Sample s ~ {1...., ( and pick S to be top s pixels as per m.
Set a[S] = x[S] and =[S] T'gray [S] where Tgny is a "default" input with gray values.
We note that SAUc indeed equivalent to the AUC insertion game from Definition 3.4 While many choices of base metrics can be used, we use this AUC metric for subsequent definitions since it confers two benefits: 1) allows for fair comparison between methods since all methods are evaluated at the same sparsity level; 2) it considers the effect of multiple sparsity levels instead of picking an arbitrary one. We are now ready to present the formalization of completeness and soundness.
3.2 Completeness and soundness
Just as for proof systems, we would like a saliency method to be logically complete and sound, i.e., it should be able to justify the label output by the net, but no other labe The statements to be "proved" are of the form "does I think has label a?". We use the model output probability [(x, a) as fractional truth value of the "statement" (xr, For saliency map m that a saliency method M generates, we interpret the AUC metric 9Auc(1, m.) as the evaluation of truth value of the "proof" m, as adjudged by the base metric gAUC- Thus in this context, completeness for M would entail that whenever fz, a) is large (i.e., the statement is sufficiently truc), 9AUC(Z, a, m) should be large (a sufficient good proof can, and is, generated). Soundness, on the other hand, would entail that (auc(1, a, m) should be small whenever f(z, is small. Thus would like (Auc(2, a, m) to track the value fxx a) through a lower bound and upper bound as follows:
Definition 3.6 (Completeness and Soundness). Consider saliency method M. For a, S the method M is a-complete on f,z,e if the heatmap m = MMffIx, a) it produces satisfies (ave m) 2 fxx,a) while M is 3-sound on f,r, a if (ave a, m) S (x, Con- versely the completeness and soundness scores for a heap map (x, a) are defined as:
alx,, a) = min max (AAuC( a, m,m)),)) 11. B(x, a) min {
1] (1) gauc a, m))
f(x,
Definition 3.7 (Worst case completeness and soundness). For saliency method M, these metrics are defined by taking the worst case of the completeness/soundiee over labels a for given æ that is sam- pled from the underlying input distribution. More precisely, ((M) = Ex [min, a(x, a)] ((M) = Ex [ming ((x, a)).
Our metrics implicitly require a saliency method to output a meaningful map for every label (or least all labels assigned non-neglig probabilities by the net) and not just for the top model prediction. Notice, a- a-completeness means that if the model outputs a high probability for label then the probability for label a after seeing only the coordinates in the salient sets is also high, while 3-soundness means that this probability is not too high. Checking both conditions verifies if the
AUC score gAuc( (x, (r, a, m) of a map m is in the interval aff(z, a), f a)) We want a, as close to as possible. This notion is different from the usual AUC insertion game, which just requires (x G. m) to be as large as possible for the model prediction I = arg max, f(z a). Moreover,
We focus single label multi-class classification; extensions to multi-labed are left for future work.
5

---

our worst-case completeness and soundness metrics require the JAuc( m) to roughly match the model prediction f(z, a). for every pair (a, [ where the thresholds E1. ég in Equation (I) help in ignoring the case where the model outputs very small probabilities - for instance we can ignore any label for completeness if (xx a) < c1
3.3 Illustration of completeness and soundness on an ImageNet example
See Figure 2 For this image x, the model assigns reasonably high probabilities to labels d = "desk" and 02 = ddesttop computer", with f'x,, a1) & 0.67 and f(, 02) R 0.13. We compare the maps computed by three methods - MM our mask learning procedure from Section with TV regularization 0.01, M2: our method with TV of 0.001, and M3: the mask learned by Phang et al. B1J The corresponding maps generated are denoted by m., í E {1, 3). For this discussion we only consider the effect of labels an and 02 on completeness and soundness, and we assume the thresholds satisfy 61 = E2 = 0.
Completeness. For the label 01, the AUC scores 9AUc 01, m:) for the three methods are roughly 0.65, 0.70 and 0.43 respectively. Thus all three maps can "certify" the label a1 well enough, with m2 doing the best job. Conseguently the completeness scores (Equation ( for m. on the pair (x,01) a1) is or1 (2x, 01) min 0.65
0.67; 1} Rs 0.97: similarly 0:2 (2, a1) = and (33(2, 01) 0.67 R 0.64. 0.43
Similarly for label 02 that is assigned a probability of 0.13, the completeness scores 0s 42) are 1, 1, and 0.58. Thus the worst case completeness from Definition 3.7, i.e. (1.2) a(r, aj), for the three methods are 0.97, and 0.58. Based on completeness M2 seems like the best method; however soundness will paint a different picture.
Soundness. Using the same AUC scores we compute the soundness scores, defined in Equation
as (z a)= = min more interesting case is label (2, for which the model does not assign very high probability (0.13).
_ [(a.a)
axuc(z.a.m.)) 1}. This gives us soundness scores on ai of 1, 0.95 and 1. The
Here the mask from M2 still gets very high AUC score of 0.29, leading to a soundness score 32(1, a2) = 0.13 x 0.44. Mi on the other hand is not as overconfident in explanation as M2, 0.29
and gets 31 (Ir, a2) = 0.13
As 0.89. The worst case soundness for three methods ends up being 0.89,
0.15
0.44 and 1.
This examples highlights how completeness and soundness together can give more nuanced comparisons of saliency maps, including understanding the benefit of TV regularization, Experiments in Section g do more thorough comparison of many saliency methods on these and prior metrics.
4 Procedures for finding masking explanations
We propose a very simple method to find saliency maps with good empirical completeness and soundness scores. Detailed intuitions and implementations appear in Appendix A Our method is similar to prior work on mask-based saliency maps and is based on the idea of SSR (Smallest Sufficient Region @ In particular for input æ and label a, given network S, the main goal is to learn map (or mask) M E [0, 1 such that E=~I(< |- log(s (ă. a)) is minimized, i.e. the probability that the network assigns to modified (or composite) input e is high. The key difference from prior masked- based methods is our choice of I, which retains the pixels of z corresponding to M, but replaces the rest of the pixels with values from a randomly sampled image T from the training se X, i.c.
I ~ ((x,a) = 5 ~ X, 5 = M x + (1 M) @
Modifications procedures (I') that have been considered in prior work include: graying or blurring out the remaining pixels [II]. and using conditional image generative model to fill in the remaining pix- els 2]. Replacing with random image pixels forces the mask- frask-finding algorithm to solve a more diffi- cult task of having the network predict high confidence for the label, despite the presence of another image. This amounts to grafting salient pixels of x on top of a random image, reminiscent of BAM
Works like Gradient @ Input and LRP use completeness in a somewhat different sense. It requires the sum of the coordinate scores in the heatmap to exactly equal the logit of the label. Our completeness + soundness together try to approximately match the JAuc score to the output probability this label. á We not incorporate any SDR (Smallest Destroying Region) component in our method.
We think other random image distributions should work too.

---

Best label
Second best label
model output probability
model output probability
TV
0.0
0.001
0.6
0.6
TV
0.4
0.0
0.4
0,001
0.2
0.2
0.1
0.0
0.0
0.5
1.0
fraction of pixels used
fraction of pixels used
Figure 3: Plot of model output probability more pixels from the original image are retained using learned masks. The remaining pixels are replaced with gray. Different curves correspond to different values TV regularization (Arv). Larger Area- Seea--nndertthe Chrve (AUC) for the left figure (best label) suggests good completeness, while lower AUC for the right figure suggests good soundness. Plots suggest that adding TV significantly helps with soundness, while only slightly hurting completeness.
evaluations 44 for saliency methods. We also note a methodological difference from Dabkowski and Gal D.. Phang et al. 31 in that we do not train separate neural network to output the mask.
As is standard, we relax the domain of masks M from binary {0, 1 Jow continuous [0, 1 1 parametrized by sigmoid. We define composite input, where the part of x on M is superimposed onto a distractor E X' as ā M @ x + (1 M) @ We then have the following simple vanilla objective, where the L norm penalty on M helps to reduce the size of masks.
L(M; (zt, a)) = E [- log( ((, a))] + A M||,, where ã = M 11 M) o D..
(3)
approaches were tested for their effect on soundness. These include 1) Total- Variation (TV) penalty [T] and 2) upsampling of the mask from a lower resolution one 30 by learning a low-resolution mask at scale s, .M E [0. 1]/ww/s", For example, on ImageNet,
Promoting soundness. Tricks used in prior
I corresponds to learning a 224 x 224 mask, while = 4 corresponds to learning a 56 x 56 mask and upsampling by a factor of 4. Incorporating 1) and 2) leads to the following modified objective
LIM;(I, a)) Ez-x log(s a))] + (T A1ll **11
where ā x
x O 2x (1 xx9)
and X** E 1hw is obtained by upsampling M by factor of s E [1,4] via bilinear interpolation.
While the motivation cited for these two commonly employed "tricks" avoid artifacts [[]. in our intrinsic framework "artifacts" does not have an interpretation. What looks like an artifact to a human observer may be relevant to the net's greiisionmmkingg Indeed, our experiments show that while TV penalty or upsampling does produce better looking masks, they lead to drop in the completeness metric. But they significantly improve soundnexs. This provides an intrinsic justification for the use of such tricks, instead of extrinsic justifications related to human interpretability.
On benefit of TV regularization in saliency
Past justifications for TV regularization in saliency methods focused on its ability make heatmaps in images that look more natural to humans. But that is not an intrinsic justification. In the intrinsic framework proposed in this paper, experiments in Section show show TV regularization helps improve soundness of the saliency method. The current section reinforces this intrinsic benefit of TV by sketching simple example (details in Appendix (F) showing how TV helps even in a simple linear model æ + ao" x on non-image input E Rd with binary label E {=1}.
Suppose that the classifier has margin at least 7 0, that is, yw x Ed ii=1 yw;.; 2 7. We focus on saliency methods that can return binary heatmap E 10, to certify the label, where the coordinates marked as constitute a set of saliency coordinates. Let T(., m) be the input modification process that assigns 0 to all input coordinates outside the corresponding saliency set S.
7

---

0.90
0.90
Qursicheating)
Ourstcheating)
Worst Case Soundness
Worst Case Soundness
Phong et al. 12020)
Purs
0.05
Phang g). (2020) Hurs
Dabkowaki Gal (2017)
Dabkooski & Gal (2017)
0.80
Iradient@imput
0.80
Bradientoinput
Oursigray)
Ours(gray)
0.75
0.75
Centered
Centereg Gaussian G0S++
0.70
0.70
Random
Random
(us00VV000001)
@urs(TV=0.001)
0.65
0.60
gong Vedaldi (201/)
0.60
d Vedaldi (201/)
Orrsis
(urs((=1)
0.55
0.55
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0.4
0.5
0.6
0.7 0.8
0.9
1.0
Worst Case Completeness
Worst Completeness of incorrect labels
Figure 4: Worst case completeness v.s. soundness plot on Imagenette for different methods. (Upper right corner is best.) Each point represent either prior method [31 m 38 our method with different settings. By default, our methods use upsampling s = 4. TV penalty λrv 0.001 and use other images for composite image (see Equation 2 "Ours" uses the default setting: "Ours(s 1)" uses upsampling s = 1: "Ours(TV 0.001)" uses TV penalty factor Arv 0.001; "Ours(gray)" fills gray pixels modifications procedures I; and "Ours(cheating)" uses masks generated for the correct label as masks for all labels. The z-axis of the right figure is the average worst completeness of incorrect labels over samples where the second best label has model probability least 0.01. See interpretations of the plot in Section 6.2
Consider an input such that w x > 0 and 1. Firstly note that any set S of coordinates that contribute positively
wt TTa xx, i.e., the base metric g(x, y. m) := 1[E w*ay)00] 0] is 1, will be able to certify the label y = 1 and thus would satisfy completeness. However this might not be convincing to a user, since it is also possible to come up with set S" that certifies the label y = -1 by picking only negative coordinates of This would not satisfy soundness the original model only predicts y 1.
This issue of soundness can be solved by adding a regularization or constraint on the set of allowable maps, in way that ensures that it is easy to find mask for the correct label y, but not for -y- Ideally, the constraints should be task-specific, but for linear classification with no special structure, even TV constraint after random permutation suffices. We consider the constraint that salient sets S should be an interval; this corresponds to TV TV(m) < 2. The following result shows that if we further constrain the length of intervals be O 3/22 log d), then with high probability, we can find an interval certifying the model prediction y but we are unable do the same for yy (Details in Appendix F
Theorem 5.1. For (æ:,) y) Rdd (+1} with ||=||2 after random shuffling of the coordinates, the following holds for any L ( log }). L2 22( log 4):
1. (Completeness) With probability 6, there an interval m of length L1 s.t. g(x, y: m) 1;
2. (Soundness) With probability 1 - 5, g y, m) = holds for all intervals m of length L2-
Although this example simple, the conceptual message is clear: saliency methods may guarantee soundness in itself, but adding regularity constraints such as TV can improve soundness.
6 Experiments
In this section, we aim to 1) show TV penalty and upsampling have intrinsic justification that they significantly improve soundness
small cost in completeness; evaluate and compare prior saliency methods, and our method with different parameter settings, on our completeness and soundness metrics. Experiments are performed either on CIFAR-10 or Imagenette [6). a 10-class subset of ImageNet [0] We compute maps and evaluate metrics on 1000 randomly drawn images from the original test set. Additional experiments on various models and datasets (including ImageNet and
'CCopleteneess and soundness evaluation on all labels can get computationally expensive on ImageNet as it contains 1000 classes. This cost is only relevant when designing the method; not deployment time.

---

Table 1: Saliency methods evaluate on prior metrics, worst completeness and soundness on Imagenette. Insertion calculated with grey infilling. The best two of each column are marked bold. By default, our methods use upsampling image in the modification procedures r. Difference from default setting is noted parentheses. + indicates lower is better. "Strong performance of likely due centrality bias (ef. Appendix C
4, TV penalty factor λrv 0.01 and use randomly sampled
Deletion
Insertion
Saliency
Worst Case
Worst Case
(gray)
Metric 4
Completeness
Soundness
Gradient @ Input
0.42
0.56
-0.35
0.56
0.80
iGOS++ 20)
0.37
0.68
-036
0.61
0.71
Dubkowski and Gal 0
0,48
0.66
-085
0.58
0.82
Fong and Vedaldi [
0.58
0.59
-0.40
0.57
0.60
Phang et al. 31
0.41
0.75
-0.27
0.71
0.84
Ours
0.50
0.74
0.90
0.84
0.84
Ours (s = 1)
0.49
0.77
1.18
0.86
0.58
Ours (Arv 0.001)
0.45
0.80
0.87
0.65
Ours (gray infilling)
0.46
0.65
0.35
0.64
0.76
Random
0.45
0.45
0.35
0.38
0.69
Centered Gaussian
0.66
0.74
-0.97*
0.64
0.71
CIFAR-100) appear in Appendix D as do details of training procedure (beyond those in Section 3 For all our completeness and soundness evaluations, we use thresholds of é1 0.01 and e2 0.001 (see Equation
6.1 Effect of TV penalty and upsampling
In this section, we study the effect of "tricks" like TV penalty and upsampling on the learned maps and evaluation metrics. Figure shows how soundness and completeness of our masks changes with TV regularization. plots the model output probability for CIFAR-10 10 as more pixels from the original image are retained using the mask. Our high level finding is that adding TV penalty significantly aids soundness, while only slightly hurting completeness. Exact evaluation soundness and completeness for different λrv are listed in Table in Appendix C Relevant points of Figure g (which also contain other saliency methods) show effect of upsampling and TV penalty on completeness and soundness. with different settings Imagenette. Our method by default use 8 = 4 and \rv = 0.01. By comparing "Ours" against "Ours(s = 1)" and "Ours(7'V = 0.001)", observe that upsampling and TV penalty leads slightly worse completeness but significantly better soundness. The effect of upsampling on previous saliency evaluation metric may be seen in the relevant entries of Table I where it leads slightly worse performance on the insertion, deletion, and saliency metrics.
We compare our method from Section 4
6.2 Comparison to existing metrics and methods
We compared our procedure to other methods including Gradient O Input 38 iGOS++ 20), Dabkowski and Gal [9] Fong and Vedaldi [I]] and Phang et al. B1 on Imagenette. We also include simple baselines like "Random" map (each entry is a random Gaussian), and "Centered Gaussian" map (2-dimensional isotropic Gaussian distribution placed at the image center). Besides completeness and soundness metrics, we evaluate methods on the Deletion Game and Insertion Game metrics sattps://gite thtbb.....t.... com/ can/ecllqque /RISE), and Saliency metric (SM) 9 Detailed description of the different metrics can be found in the Appendix B
For our method, we learn map by optimizing the objective function in Equation with ÀTV E {0.01, 0.001) and s E {1, default settings are \Tv = 0.01, s = 4. We also evaluate our method when other pixels are replaced with gray rather than random image, denoted by "Ours (gray)". For uniformity, we use the identical ResNet50 pretrained on ImageNet as the base classifier for every saliency method. We normalize the maps so that all values lie in [0, 1] before use.
Completeness and soundness. We compare different methods on the metrics from Definition 3.7 through the visualization in Figure d (left) and in Table D We find that our method achieves better completeness and good soundness compared to other methods and baselines. We posit that completeness is better due to our improved composite input strategy from Equation (2). where non-salient pixels are filled with another image pixels as opposed to other strategies. As an ablation, see that "Ours (gray)" which uses gray replacement does not perform as well.

---

On the other hand, our soundness is good only for higher TV regularization Xrv = 0.01 and higher upsampling 4, as discussed in Section 6.1 Soundness very bad (but completeness slightly better) if these "tricks" are not used. We note that while Phang et al. [3] does as well as our method on soundness, does so via "cheating" in an interesting way: they compute mask for the top label and return the same mask for all labels. Since the method does not even try to generate masks for labels with low probability, it achieves good soundness for free. To delve deeper, we employ the same "cheating" with our method - return the same mask for all labels and observe a large bump in soundness, without any drop in completeness. While "cheating" is always possible (even if unintentionally) for any metric, compute maps for all labels. In this case we find, in Figure +
hope that our results inspire saliency method designers to (right), that looking at worst case completeness of incorrect labels (all except model prediction) gives a measure of the "effort" method employs in generating masks for all labels with probability at least 0.01. As evident methods that "cheat", like Phang et al. 31 and "Ours (cheating)", achieve bad "best effort" score.
Other metries. Table L suggests that our method can achieve comparable performances on most existing intrinsic metrics. This suggests that good completeness and soundness scores are achievable without compromising on other measures. Deletion metric is one where our method falls short, likely due to our method not employing SDR objective in Dabkowski and Gal [2] Although we note that "Random" baseline performs quite well on the Deletion metric, which suggests that deletion metric does not provide much signal on these datasets. Further investigation the good performance of "Centered Gaussian" " baseline can be found in Appendix C
Our settings that works best on other metrics are those without upsampling (s = 1) or lower TV (Arv 0.001). Soundness the only metric that strongly justifies the upsampling and TV "tricks", thus verifying its utility. Our method with = 4, Arv 0.01 demonstrates that good performance can be obtained on all metries simultaneously, with only little price to pay.
Limitations of our work. Completeness and soundness do not involve or address human inter- pretability of explanations. This can be strength when considering intrinsic evaluations, but is weakness when considering extrinsic evaluations. Additionally, though completeness and soundness should be taken into account while designing a saliency method, they are not meant as replacement for other evaluation methods. which can provide additional information on saliency map quality. For example, the axioms and sanity checks for saliency discussed in Appendix [.. should also be applied. Finally, at deployment time, there is risk that these evaluation methods may be used as a "proof" to justify a saliency method is safe to be used in high-risk applications. Completeness and soundness should be taken as sanity check rather than proof saliency method is safe for deployment.
7 Conclusions
Saliency explanations of ML models have proved nebulous and generated much discussion. By avoiding extrinsic considerations (e.g., human interpretability) and sticking to intrinsic notions such as completeness and soundness. this paper has tried to give rigorous notion of saliency that is intrinsic to the deep net, while avoiding the noisiness in carlier intrinsic ideas that motivated methods like Gradient and Shapley Values. Other new contributions include clarifying in the intrinsic view the role TV regularization and other tricks (it hurts completeness slightly but greatly improves soundness); simple saliency method (Section 4 for producing mask-based explanations that was designed solely with intrinsic considerations and yet has performance competitive with good existing methods, and sometimes better (Table IT. Our experiments suggest soundness provides new dimension to evaluate saliency methods on, and that good performance on it can be achieved without compromising on other metrics. Note that soundness needs to be considered only while designing and evaluating the method, not at deployment.
Evaluating soundness requires looking at maps for all labels instead of just the top label, and thus seems relevant for object localization in images, and for classification in the wild where multiple objects appear in image. This benefit is hinted in our evaluation pendix B and deserves further exploration. It may help with certain distribution shifts as well. While this work studies soundness in the context of mask-based saliency methods and evaluations, the con- cept of soundness could be applicable for the idea of saliency and interpretability in more generality.
prior small dataset (see Ap-
Acknowledgements. We thank Ruth Fong for feedback on earlier draft of this paper. We are also grateful to the valuable comments from various anonymous reviewers that helped improve the paper. This work is supported by funding from NSF, ONR, Simons Foundation, DARPA and SRC.
10

---

References
[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in Neural Information Processing Systems, 31: 9505-9515 2018.
[2] Chirag Agarwal and Anh Nguyen. Explaining image classifiers by removing input features using generative models. In Proceedings of the Asian Conference on Computer Vision, 2020.
[3] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller. How to explain individual classification decisions. The Journal of Machine Learning Research, 11:1803-1831, 2010.
[4] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov. Xiaohua Zhai, and Aäron van den Oord. Are done with imagenet? arXiv e-prints, pages arXiv-2006, 2020.
[5] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Woj- ciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pages 63-71. Springer, 2016.
[6] Lukas Brunke, Prateek Agrawal, and Nikhil George. Evaluating input perturbation methods for interpreting cnns and saliency map comparison. In European Conference on Computer Vision, pages 120-134. Springer, 2020.
[7] Brandon Carter, Jonas Mueller, Siddhartha Jain, and David Gifford. What made you do this? understanding black-box decisions with sufficient input subsets. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 567-576. PMLR, 2019.
[8] Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In International Conference on Learning Representa- tions, 2018
[9] Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 1970-6979, 2017.
[10] Nicola De Cao, Michael Sejr Schlichtkrull, Wilker Aziz, and Ivan Titov. How do decisions emerge across layers in neural models? interpretation with differentiable masking. In Proceed- ings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 3243-3255, 2020.
[11] Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation. Proceedings of the IEEE International Conference on Computer Vision, pages 3429 3437, 2017.
[12] Daniel Fryer, Inga Strümke, and Hien Nguyen. Shapley values for feature selection: The good, the bad, and the axioms. ICLR, 2021.
[13] Jindong Gu, Yinchong Yang, and Volker Tresp. Understanding individual decisions of cnns via contrastive backpropagation. In Asian Conference on Computer Vision, pages 119-134. Springer, 2018
[14] Xin Hong, Pengfei Xiong, Renhe Ji, and Haoqiang Fan. Deep fusion network for image completion. In Proceedings of the 27th ACM International Conference on Multimedia, pages 2033-2042. 2019.
[15] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for inter- pretability methods in deep neural networks. In NeurlPS, 2019.
[16] Jeremy Howard. imagenette, 2019, URL https://github. comffustai/imagenette, 2020.
[17] Alexey Ignatiev, Nina Narodytska, and Joao Marques-Silva. Abduction explanations for machine learning models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1511-1519, 2019.

---

[18] Alexey Ignatiev, Nina Narodytska, and Joao Marques-Silva. On relating explanations and adversarial examples. 2019.
[19] Ashkan Khakzar, Soroosh Baselizadeh, Saurabh Khanduja, Christian Rupprecht, Seong Tae Kim, and Nassir Navab. Improving feature attribution through input-specific network pruning. arXiv e-prints, pages arXiv-1911, 2019.
[20] Saeed Khorram, Tyler Lawson, and Li Fuxin. igos++ integrated gradient optimized saliency by bilateral perturbations. In Proceedings of the Conference on Health, Inference, and Learning, pages 174-182, 2021.
[21] Beomsu Kim, Junghoon Seo, Seunghyeon Jeon, Jamyoung Koo, Jeongyeol Choe, and Taegyun Jeon. Why are saliency maps noisy? cause of and solution to noisy saliency maps. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 4149 4157. IEEE, 2019.
[22] Junho Kim, Seongyeop Kim, Seong Tae Kim, and Yong Man Ro. Robust perturbation for visual explanation: Cross-checking mask optimization to avoid class distortion. IEEE Transactions Image Processing, 31:301-313, 2021.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint 1rX:v:14/2.6980, 2014.
[24] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz- Richardson. Captum: unified and generic model interpretability library for pytorch, 2020.
[25] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer. 2014.
[26] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predic- tions. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neurul Information Processing Systems. Curran Associates, Inc., 2017. URL
www.igs.neuiss... wwwpurpps.ccppppe///eeeeeee pptppppppe/////ieee pdf
[27] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st international conference on neural information processing systems, pages 4768 4777,4477,2111
[28] Jan Macdonald, Stephan Wäldchen, Sascha Hauch, and Gitta Kutyniok. A rate-distortion framework for explaining neural network decisions. arXiv e-prints, pages arXiv-1905, 2019.
[29] Jesse Mu and Jacob Andreas. Compositional explanations of neurons. 2020.
[30] Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation
of black-box models. In Proceedings of the British Machine Vision Conference (BMVC), 2018
[31] Jason Phang, Jungkyu Park, and Krzysztof J Geras. Investigating and simplifying masking- based saliency methods for model interpretability. arXiv preprint urXiv:2010.09750. 2020.
[32] Donald B Rubin. Inference and missing data. Biometrika, 63(3):581-592, 1976.
[33] Wojeiech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus- Robert Müller, Evaluating the visualization of what a deep neural network has learned. JEEE transactions neural networks and learning systems, 28(1 (1 1):2660-22661 2016.
[34] Wojeiech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Rober Müller. Explainable AI: interpreting, explaining and visualizing deep learning, volume 11700. Springer Nature, 2019.
[35] Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. In International Conference on Learning Representations, 2019.
12

---

[36] Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? International Journal of Computer Vision (IJCV), 2019.
[37] Junghoon Seo, Jeongyeol Choe, Jamyoung Koo, Seunghyeon Jeon, Beomsu Kim, and Taegyun Jeon. Noise-adding methods of saliency map as series of higher order partial derivative. 2018 ICML Workshop on Human Interpretability in Machine Learning, pages arXiv-1806, 2018.
[38] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International Conference on Machine Learning, pages 3 3145-3153. PMLR, 2017.
[39] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smooth- grad: removing noise by adding noise. arXiv e-prints, pages arXiv-1706, 2017.
[40] Mukund Sundararajan and Amir Najmi. The many shapley values for model explanation. In International Conference on Machine Learning, pages 9269-92777 PMLR, 2020.
[41] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference machine learning, pages 3319-3328. PMLR, 2017.
[42] Saeid Asgari Taghanaki, Mohammad Havaei, Tess Berthier, Francis Dutil, Lisa Di Jorio, Ghassan Hamarneh, and Yoshua Bengio. Infomask: Masked variational latent representation localize chest disease. In Interatiional Conference Medical Image Computing and Computer-Assisted Intervention, pages 739-747. Springer, 2019.
[43] Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun Preece. Sanity checks for saliency metrics. In Proceedings of the AAAI conference on artificial intelligence. volume 34, pages 6021-6029 2020.
[44] Mengjiao Yang and Been Kim. Benchmarking attribution methods with relative feature impor- tance. arXiv e-prints, pages arXiv-1907, 2019.
[45] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-awee aware concept-base based explanations in deep neural networks. Advances in Neural Information Processing Systems, 33, 2020.
[46] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5505-5514, 2018.
[47] Jianming Zhang, Sarah Adel Bargal, Zhe Lin, Jonathan Brandt, Xiaohui Shen, and Stan Sclaroff. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 1 126(10):1084-111 1002, 2018.
[48] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 4066):445211464, 2017.
[49] Bolei Zhou, David Bau, Aude Oliva, and Antonio Torralba. Interpreting deep visual representa- tions via network dissection. IEEE transactions on pattern analysis and machine intelligence, 2119)223312 2018.
13

---

1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Main con- tribution of this work is to provide suggestions and sanity checks to improve evaluation systems that aim to open up and explain the black-box box that is deep learning. Doing so can help us avoid unintended (or adversarial) negative impacts. Just like any other work, readers and users should be wary of treating this idea as a magic pill that solves all issues. One negative impact could be if these evaluation systems sanity checks, that systems are working well and are safe to deploy.
treated as "proofs", rather than
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you 're
using/curating? [N/A]
(e) Did you diseuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applica-
ble? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board
(IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "uNOnGRrHi8c", "reviewer": "Reviewer_Ea2U", "summary": "New Definitions and Evaluations for Saliency Methods introduces intrinsic evaluation metrics for saliency methods --- completeness and soundness --- that do not require additional models or human evaluation. These metrics are grounded in logical proof concepts and force the method to output a saliency map that only explains the class of interest. The paper proposes a mask-based saliency method that optimizes for soundness as well as completeness. Evaluations compare the proposed saliency method to other mask-based saliency methods on soundness and completeness, deletion and insertion game metrics, and saliency metric. ", "questions": "See the weaknesses section for questions and suggestions that would strengthen the paper.", "limitations": "Please include a discussion of limitations. Some questions I had were:\n* What is the tradeoff between intrinsic and extrinsic evaluations? Do they both have a place in evaluating saliency methods, or are intrinsic evaluations like yours always better?\n* Is there a tradeoff between completeness and soundness? Should we optimize for both equally, or is there ever a case where we should prioritize one over another? Can looking at completeness and soundness separately tell us anything different than looking at them together?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* *Intrinsic evaluation methods for saliency methods* --- This paper proposes that saliency methods should be evaluated on completeness and soundness. These attributes are grounded in logical proof systems and defined for saliency methods mathematically and textually in the paper. These concepts provide a formal intrinsic framework to evaluate saliency methods without requiring human evaluations. Human evaluators often measure how well the saliency maps to their representations, which may not always align with the model's representations. Intrinsic evaluations are better suited to evaluating saliency methods on their ability to reflect the underlying model.\n* *Defining saliency method requirements* --- The paper reframes saliency by introducing completeness and soundness as two necessary constraints for saliency methods. Previously we only required completeness; saliency justified a model's prediction. By requiring completeness and soundness, saliency justifies a model's prediction but can not justify any other possible prediction. These requirements improve the specificity of what a saliency method should output and make it more straightforward to interpret the results of a saliency method. It also ensures that the saliency map for each class is distinct, which can improve our ability to compare maps and draw meaningful insight between possible predictions.\n* *Clarity* --- The paper is very well written and precise. Section 3 is straightforward to follow, despite presenting complex definitions.\n\n**Weaknesses**\n* *Novelty of replacement strategy* --- The key novelty of the saliency method is its pixel replacement strategy, where a new pixel value is sampled from another random image. This strategy is known as hot-deck imputation, where replacement values are sampled from the marginal feature distribution. Existing work has used hot-deck imputation as a masking strategy. It has also shown hot-deck imputation and mean imputation (i.e., grey pixel replacement) result in similar changes to model outputs. (see \"What made you do this? Understanding black-box decisions with sufficient input subsets\" by Carter et al.). Given the similarities to this work, I suggest discussing it in the related work and including it in the comparison to existing metrics and methods.\n* *Missing related work* --- Related work on saliency evaluation methods should include model and data randomization tests from \"Sanity Checks for Saliency Maps\" by Adebayo et al.. Also, consider the saliency method axioms from \"Axiomatic Attribution for Deep Networks\" by Sundararajan et al.. \"Sanity Checks for Saliency Metrics\" by Tomsett et al. has a good evaluation of existing saliency evaluations.\n* *Lack of reproducibility* --- The checklist indicates the paper does not include the compute details, code, or data. Please include computing details and other details needed for reproducibility. If possible, also release the code.\n* *Limited limitations section* --- The paper does not discuss limitations. Understanding limitations is essential for readers who are looking to use this work. Please include a discussion on important considerations when using your method. Also, please incorporate the ethical considerations in Checklist 1 in the main text.\n\n**Minor Issues**\n* Line 19: missing space between emdash and words\n* Line 37: \"and if so one\" --> \"and, if so, one\"\n* Line 213: \"Procedures for find masking explanations\" --> \"Procedures for finding masking explanations\"\n* Line 278: \"from original test set\" --> \"from the original test set\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "n-YRCuVzZDF", "reviewer": "Reviewer_kmbL", "summary": "This paper presents a method for attributing saliency (in the sense of determining which pixels contribute to a classification outcome) to an image. It does so in a novel fashion that explores the tradeoff between the notions of completeness and soundness, pointing out that prior work in this domain does not address the latter.", "questions": "Figures 1 and 2 present a good case for the issues addressed by the paper. There are many quantitative results included as well. I would have liked to see some additional results on the qualitative side if possible but also understand that there are constraints to the size of the manuscript.\n\nCan the authors comment on the value of a measure of saliency being intrinsic rather than guided by e.g. an external neural network? I see merits for this, but would appreciate a succinct statement of why things should inherently be intrinsic.\n\nCan the authors comment on what is the right tradeoff between completeness and soundness in interpretation since there is an ability to create an arc of success on both fronts. E.g. how should one adjust things to exert the optimal tradeoff for human interpretability?", "limitations": "There are not strong societal impacts of this work insofar as I can see and to the extent that these do exist the authors have made a good case.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper itself is reasonably well written albeit with some typos (e.g. completeness is spelled wrong in some places). I find that this addresses an original angle of this type of assessment of how the neural network makes its determination and in a principled way that gives it an advantage over some of its predecessors. I think this is a significant result and generally view the conclusions drawn by this paper as positive.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "az5ZG2nDSn", "reviewer": "Reviewer_Ze6X", "summary": "The paper introduces and formalizes new evaluation metrics to ensure goodness of saliency methods, based on the logical concepts of completeness and soundness. The first ensures that the network's output is unchanged when using the masked (with the saliency map) input in place of the full image, which is what most of current evaluations methods for saliency methods require. The latter requires verifying that the same saliency method cannot be used to produce masked input that make the net output a different label, and therefore ensures that the evaluation of saliency maps appropriately track the model's probability of assigning labels. ", "questions": "The paper is very complete and accurate, and I don't have main questions or suggestions. Minor ones are:\n- In \"Prior approaches\" section, a number of extrinsic evaluation metrics are listed. Are there some intrinsic methods previously proposed?\n- At line 207, according to Fig. 2, M2 AUC score should be 0.29 and not 0.37.\n- In fig.4 caption, I think there is a mistake, since TV penalty factor Lambda_TV in line 5 should be 0.001 and not 0.01.\n\n", "limitations": "I think sharing code and data related to the paper would be beneficial for the scientific community.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper's contributions are clear and significant, and explained in a straightforward and accurate manner. Examples are significant and useful. The originality of the contribution lies in connecting the context of saliency methods to logical proof systems and to formalize an evaluation approach which overcomes limitations of current methods and helps making them more rigorous and theoretically grounded. A simple saliency method based on optimization is proposed, which thanks of a change in the pixel replacement strategy allows to satisfy soundness at a small price in completeness. This is proven to work as expected when validated on various datasets and compared to other saliency methods. Furthermore, thanks to their formal frameworks of definitions, authors provide an intrinsic justification about why methods used heuristically to improve the aspect of masks (TV regularization and upsampling) actually work, in that they improve the soundness.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IACLMOKrWPF", "reviewer": "Reviewer_9U1L", "summary": "This paper presents an additional dimension, *soundness*, for evaluating saliency methods for explainable AI. The authors define this concept, then use it to provide both explanations for why existing heuristic methods work, and to suggest new saliency methods. \n", "questions": "- Lines 44-46 clearly describe the \"mask shape\" problem. The method introduced in Section 4 mitigates this problem by using pixels from a random image rather than grey pixels or other methods. Does this solve the mask shape problem, or just make it more difficult to solve? The new method requires solving a difficult segmentation problem (that I'm not sure humans would be very good at) but one that is nevertheless solvable.\n- Lines 166-168: shouldn't other values for label $a$ enter in here? For example, Figure 3 (right) is for the second best label. Shouldn't there be an index over labels other than $a$?\n- Line 184: Should be $\\beta$ not $\\alpha$.\n- Are completeness and soundness similar to (or perhaps, the same as) sensitivity and specificity? If so, mentioning this would help to improve the clarity.\n- Figure 4 typo \"completenese\"", "limitations": "The authors have adequately addressed limitations.", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nI find this to be a useful and convincing paper. The paper is well written, but the presentation of the concepts could be made more crisp in parts (see questions, below).\n\n### Weaknessess\n\nNothing major I could see, but this is somewhat outside my area.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept