New Definitions and Evaluations for Saliency Methods: Staying Intrinsic and Sound

1 INTRODUCTION . Why did the deep net give a certain answer on a particular input , and can we trust the answer ? Saliency methods try to provide explanations , and are thus of great interest from viewpoint of human explainability , fairness , robustness , etc . This paper restricts attention to methods that return an importance score for each coordinate of the input —usually visualized as a heat map— that captures its importance to the final decision.1 We refer the reader to Samek et al . ( 2019 ) for an extensive survey and Section 2 for a short account of such methods and controversies . There are two important components in the research on saliency : saliency methods that produce such heat maps for explanations , and saliency evaluation metrics that aim to test and compare saliency methods . Numerous saliency methods have been proposed to learn such heat maps . Some methods learn maps through “ credit attribution ” to individual input coordinates by using methods reminiscent of backpropagation ( Binder et al. , 2016 ; Selvaraju et al. , 2019 ) , while some are derived using careful axiomatization of credit assignment using the idea of Shapley values from cooperative game theory ( Lundberg & Lee , 2017b ; Yeh et al. , 2020 ) . Some recent methods train another deep net to produce heat maps ( Dabkowski & Gal , 2017 ; Phang et al. , 2020 ) . Given the proliferation of saliency methods , an ecosystem of evaluation metrics has emerged to evaluate the quality of explanations produced by saliency methods , either through human evaluations ( Adebayo et al. , 2018 ) , comparison to certain ground truth explanations ( Zhang et al. , 2018 ) or other evaluations that do not require any external annotation or supervision ( Dabkowski & Gal , 2017 ) Petsiuk et al . ( 2018 ) . In this paper we restrict attention to a class of evaluations we refer to as intrinsic , that aim to evaluate saliency maps based on whether they are good explanations for the model prediction . These only involve computations using the provided heat map and the net , and do not involve extrinsic factors such as human judgements or retraining . A popular idea in such evaluations ( see Section 3 for references ) is to create a new composite input —or sequence of such inputs— using the heat map and the original input , and to evaluate the original net on this composite input . For example , if M is a binary vector with 1 ’ s in the k coordinates with the highest values in the heat map , then x M ( with denoting coordinate-wise multiplication ) can be viewed as a composite input2 ( aka 1Heat maps suffice for recognition/classification tasks ; other tasks may require more complex explanations . 2When x is an image the zeroes in this masked input are often replaced with gray values . “ masked input ” ) that can be fed into the original net . By varying k one obtains a sequence of masks M . Although there is an issue of distribution shift due to the net never having trained on composite inputs , in practice , especially for image data , trained nets work fine on masked inputs . ( This is also exploited in Shapley value based saliency methods . ) Inspired by logical reasoning , we identity a key component missing in existing intrinsic evaluations : soundness of the saliency method . In simple terms , checking soundness for mask-based explanations entails verifying that it is impossible for the method to produce well-performing ( in the compositeinput evaluations ) heat maps corresponding to the incorrect labels , i.e . labels not predicted as the most likely by the model . By contrast current masking evaluations focus on a property akin to completeness3 in logic : the heat map “ justifies ” the correct label , i.e . the label judged to be most likely by the model . We argue that soundness can be a useful criterion in addition to completeness . The concept of soundness is motivated by observing that the recent mask-based saliency methods are inherently different from the earlier gradient-based methods . They produce a heatmap that is interpreted as a masked image that , when fed into the net must make it output the same label that received the top logit value on the full image . Our paper observes that it is unclear at a logical level why this should be considered an “ explanation ” , since to be convinced one must verify that a different mask could not induce the net to output a label corresponding to other logits . Figure 1 shows that in fact this issue is real ; masks do exist that can “ justify ” the wrong labels . We introduce soundness as a quantitative measure for this issue ( definitions in Section 3.1 ) , and propose that evaluations of mask-based saliency methods should account for soundness . Many past works allude to difficulty of the soundness concept , mentioning that computations to find the “ best ” mask end up finding artifacts that can even justify labels that are obviously incorrect . To avoid uncovering artifacts , the methods to find masks/heat maps use a suitable regularization ( usually TV ) . However , Figure 1 illustrates that TV regularization is not a full solution to removing artifacts . Another reason for incorporating soundness with completeness in metrics is to implicitly encourage saliency methods to also produce saliency maps for incorrect labels . When using the classifier in the wild—as opposed to on a dataset with centered images— we might encounter multiple salient objects in the image , and the net may not have high confidence in a single label . It would then be reasonable to have saliency methods find evidence for all the labels which the net outputs with reasonable confidence . We find in Figure 2 that our saliency method that produces masks for all labels can produce different and meaningful masks for different labels when there are multiple objects present in the image4 , compared to prior state-of-the-art methods , that output a single mask for all labels that contains both objects ( more on this in Appendix B ) . 3In the literature devoted to saliency methods based upon backpropagation-like procedure , such as GRADIENT INPUT or LRP , the term completeness is used in a slightly different sense : the sum of the heat map values has to be equal to the logit value for the label . 4In this paper , multiple objects in one image is not formally considered , as we assume single object classification is the prior knowledge . Therefore , either zebra or elephant in Figure 2 is regarded as an incorrect label . A quick extension to the multi-object case is to define correct labels as the labels whose model probability exceeds a certain threshold . Even without the extended definition of correct labels , Definition 3.2 with suitable α , β already allows explanations to exist for both elephant and zebra as long as the model probability for elephant and zebra are high enough . Main contributions : In this paper we make a case for pushing saliency methods and metrics to incorporate soundness in addition to completeness . We propose metrics and a method and experimentally evaluate them in comparison to previous methods and metrics , leading to some novel findings . Our main contributions are as follows : • A new single number metric named consistency score that evaluates the completeness and sound- ness simultaneously . It measures the probability that the saliency map for the model prediction has the best quality compared to all other labels . See Section 3.2 for a formal definition . • A saliency method in Section 4 that bears similarity to prior works that learns masks through optimization , but with subtle differences . One of the differences is based on the logic of soundness : we learn a mask by trying to maximize the probability for the given label , rather than matching the probability , and learn the best mask for all labels this way . We find in experiments ( Section 6 ) that our method not only performs comparably to prior masked-based methods on earlier saliency metrics , but also does better than those methods on the proposed consistency score . • Previous works proposed “ tricks ” like TV penalty and upscaling of masks ( Fong & Vedaldi , 2017 ; Dabkowski & Gal , 2017 ) that are motivated by their ability to avoid “ artifacts ” and make the saliency maps visually look better . These justifications , however , are not in the spirit of an intrinsic evaluation of saliency maps . In this paper , we provide a novel intrinsic justification for these choices of TV penalty and upscaling , by showing that they improve the saliency maps by making them more sound , as empirically demonstrated in Section 6 . We complement this finding with a theoretical result in Section 5 that proves that TV regularization can help with soundness in a simple linear classification setting . Paper outline : Section 2 discusses prior works on saliency methods and evaluations . Section 3 delves further into the concept of completeness and soundness in the context of saliency evaluation , and describes our proposed metric of consistency score . Section 4 describes our saliency method and how it differs from prior approaches . Finally Section 5 describes various supportive experiments . 2 PRIOR APPROACHES . We delegate a more thorough description of prior work to Appendix D , but mention some common saliency and saliency evaluation methods here . Saliency methods aim to explain a model ’ s decision about an input . Saliency evaluation methods aim to evaluate the goodness of a saliency method . Saliency methods include backpropagation based approaches such as Gradient Input ( Shrikumar et al. , 2017 ) , LRP ( Binder et al. , 2016 ) , GradCAM ( Selvaraju et al. , 2019 ) , Smooth-Grad ( Smilkov et al. , 2017 ) . Another line of work is masking methods which include techniques based on averaging over randomly sampled masks ( Petsiuk et al. , 2018 ) , optimizing over meaningful mask perturbations ( Fong & Vedaldi , 2017 ) , and real time image saliency using a masking network ( Dabkowski & Gal , 2017 ) . Pixels that have been removed from the image by the mask may be replaced by greying out , by Gaussian blurring as in ( Fong & Vedaldi , 2017 ) , or with infillers such as CA-GAN ( Yu et al. , 2018 ) used in ( Chang et al. , 2018 ; Phang et al. , 2020 ) , or DFNet ( Hong et al. , 2019 ) . De Cao et al . ( 2020 ) find masks using differentiable masking . Boolean logic is another approach for saliency methods ( Ignatiev et al. , 2019b ; a ; Macdonald et al. , 2019 ; Mu & Andreas , 2020 ; Zhou et al. , 2018 ) . Arguments about saliency . As saliency methods have arisen , discussion about them has occurred e.g . in ( Seo et al. , 2018 ; Fryer et al. , 2021 ; Gu et al. , 2018 ; Sundararajan & Najmi , 2020 ) . Some works reveal situations where Shapley axioms work against feature selection or where Shapley values may be calculated in conflicting ways ( Fryer et al. , 2021 ; Sundararajan & Najmi , 2020 ) . Others question why noise adding saliency methods work ( Seo et al. , 2018 ) , with some pointing out that some explanations are not class discriminative ( Gu et al. , 2018 ) . Saliency evaluation methods . Extrinsic evaluation metrics include the WSOL metric , and Pointing Game metric proposed by Zhang et al . ( 2018 ) and ROAR ( Hooker et al. , 2019 ) . Other more intrinsic methods include early saliency evaluation techniques like MorF and LerF ( Samek et al. , 2016 ) , Insertion and Deletion game proposed by Petsiuk et al . ( 2018 ) , which involve either inserting pixels in order of most importance or deleting pixels in order of most importance . BAM ( Yang & Kim , 2019 ) creates saliency maps by pasting object pixels from MSCOCO ( Lin et al. , 2014 ) The Saliency Metric proposed by Dabkowski & Gal ( 2017 ) thresholds saliency values above some α chosen on a holdout set , finds the smallest bounding box containing these pixels , upsamples and measures the ratio of bounding box area to model accuracy on the cropped image , s ( a , p ) = log ( max ( a , 0.05 ) ) − log ( p ) where a is the area of the bounding box and p is the class probability of the upsampled image . Controversies . Brunke et al . ( 2020 ) show that perturbation methods are sensitive to baseline and Petsiuk et al . ( 2018 ) point out that human centric explanations ( based on bounding boxes ) may not reveal why the model made a certain decision . Our notion of intrinsic saliency method aligns with the latter idea , and we introduce a new criteria for evaluating saliency maps : soundness , which captures the notion that maps should not produce explanations for low probability labels .

Summary: This paper introduces two intrinsic evaluation criteria for saliency-based model explanation: completeness and soundness. While completeness indicates a saliency method's ability to justify the correct label,  soundness requires the method to be unable to find a masked input that significantly increases the probability of an incorrect label. A consistency score is introduced to simultaneously evaluate both completeness and soundness. Further, taking into account the soundness factor, a saliency method is proposed to maximize the probability for a given label, rather than all labels. Experiments demonstrate that the heuristic TV regularization method can help with soundness, as suggested by high consistency scores.