New Definitions and Evaluations for Saliency Methods: Staying Intrinsic and Sound

1 INTRODUCTION . Why did the deep net give a certain answer on a particular input , and can we trust the answer ? Saliency methods try to provide explanations , and are thus of great interest from viewpoint of human explainability , fairness , robustness , etc . This paper restricts attention to methods that return an importance score for each coordinate of the input —usually visualized as a heat map— that captures its importance to the final decision.1 We refer the reader to Samek et al . ( 2019 ) for an extensive survey and Section 2 for a short account of such methods and controversies . There are two important components in the research on saliency : saliency methods that produce such heat maps for explanations , and saliency evaluation metrics that aim to test and compare saliency methods . Numerous saliency methods have been proposed to learn such heat maps . Some methods learn maps through “ credit attribution ” to individual input coordinates by using methods reminiscent of backpropagation ( Binder et al. , 2016 ; Selvaraju et al. , 2019 ) , while some are derived using careful axiomatization of credit assignment using the idea of Shapley values from cooperative game theory ( Lundberg & Lee , 2017b ; Yeh et al. , 2020 ) . Some recent methods train another deep net to produce heat maps ( Dabkowski & Gal , 2017 ; Phang et al. , 2020 ) . Given the proliferation of saliency methods , an ecosystem of evaluation metrics has emerged to evaluate the quality of explanations produced by saliency methods , either through human evaluations ( Adebayo et al. , 2018 ) , comparison to certain ground truth explanations ( Zhang et al. , 2018 ) or other evaluations that do not require any external annotation or supervision ( Dabkowski & Gal , 2017 ) Petsiuk et al . ( 2018 ) . In this paper we restrict attention to a class of evaluations we refer to as intrinsic , that aim to evaluate saliency maps based on whether they are good explanations for the model prediction . These only involve computations using the provided heat map and the net , and do not involve extrinsic factors such as human judgements or retraining . A popular idea in such evaluations ( see Section 3 for references ) is to create a new composite input —or sequence of such inputs— using the heat map and the original input , and to evaluate the original net on this composite input . For example , if M is a binary vector with 1 ’ s in the k coordinates with the highest values in the heat map , then x M ( with denoting coordinate-wise multiplication ) can be viewed as a composite input2 ( aka 1Heat maps suffice for recognition/classification tasks ; other tasks may require more complex explanations . 2When x is an image the zeroes in this masked input are often replaced with gray values . “ masked input ” ) that can be fed into the original net . By varying k one obtains a sequence of masks M . Although there is an issue of distribution shift due to the net never having trained on composite inputs , in practice , especially for image data , trained nets work fine on masked inputs . ( This is also exploited in Shapley value based saliency methods . ) Inspired by logical reasoning , we identity a key component missing in existing intrinsic evaluations : soundness of the saliency method . In simple terms , checking soundness for mask-based explanations entails verifying that it is impossible for the method to produce well-performing ( in the compositeinput evaluations ) heat maps corresponding to the incorrect labels , i.e . labels not predicted as the most likely by the model . By contrast current masking evaluations focus on a property akin to completeness3 in logic : the heat map “ justifies ” the correct label , i.e . the label judged to be most likely by the model . We argue that soundness can be a useful criterion in addition to completeness . The concept of soundness is motivated by observing that the recent mask-based saliency methods are inherently different from the earlier gradient-based methods . They produce a heatmap that is interpreted as a masked image that , when fed into the net must make it output the same label that received the top logit value on the full image . Our paper observes that it is unclear at a logical level why this should be considered an “ explanation ” , since to be convinced one must verify that a different mask could not induce the net to output a label corresponding to other logits . Figure 1 shows that in fact this issue is real ; masks do exist that can “ justify ” the wrong labels . We introduce soundness as a quantitative measure for this issue ( definitions in Section 3.1 ) , and propose that evaluations of mask-based saliency methods should account for soundness . Many past works allude to difficulty of the soundness concept , mentioning that computations to find the “ best ” mask end up finding artifacts that can even justify labels that are obviously incorrect . To avoid uncovering artifacts , the methods to find masks/heat maps use a suitable regularization ( usually TV ) . However , Figure 1 illustrates that TV regularization is not a full solution to removing artifacts . Another reason for incorporating soundness with completeness in metrics is to implicitly encourage saliency methods to also produce saliency maps for incorrect labels . When using the classifier in the wild—as opposed to on a dataset with centered images— we might encounter multiple salient objects in the image , and the net may not have high confidence in a single label . It would then be reasonable to have saliency methods find evidence for all the labels which the net outputs with reasonable confidence . We find in Figure 2 that our saliency method that produces masks for all labels can produce different and meaningful masks for different labels when there are multiple objects present in the image4 , compared to prior state-of-the-art methods , that output a single mask for all labels that contains both objects ( more on this in Appendix B ) . 3In the literature devoted to saliency methods based upon backpropagation-like procedure , such as GRADIENT INPUT or LRP , the term completeness is used in a slightly different sense : the sum of the heat map values has to be equal to the logit value for the label . 4In this paper , multiple objects in one image is not formally considered , as we assume single object classification is the prior knowledge . Therefore , either zebra or elephant in Figure 2 is regarded as an incorrect label . A quick extension to the multi-object case is to define correct labels as the labels whose model probability exceeds a certain threshold . Even without the extended definition of correct labels , Definition 3.2 with suitable α , β already allows explanations to exist for both elephant and zebra as long as the model probability for elephant and zebra are high enough . Main contributions : In this paper we make a case for pushing saliency methods and metrics to incorporate soundness in addition to completeness . We propose metrics and a method and experimentally evaluate them in comparison to previous methods and metrics , leading to some novel findings . Our main contributions are as follows : • A new single number metric named consistency score that evaluates the completeness and sound- ness simultaneously . It measures the probability that the saliency map for the model prediction has the best quality compared to all other labels . See Section 3.2 for a formal definition . • A saliency method in Section 4 that bears similarity to prior works that learns masks through optimization , but with subtle differences . One of the differences is based on the logic of soundness : we learn a mask by trying to maximize the probability for the given label , rather than matching the probability , and learn the best mask for all labels this way . We find in experiments ( Section 6 ) that our method not only performs comparably to prior masked-based methods on earlier saliency metrics , but also does better than those methods on the proposed consistency score . • Previous works proposed “ tricks ” like TV penalty and upscaling of masks ( Fong & Vedaldi , 2017 ; Dabkowski & Gal , 2017 ) that are motivated by their ability to avoid “ artifacts ” and make the saliency maps visually look better . These justifications , however , are not in the spirit of an intrinsic evaluation of saliency maps . In this paper , we provide a novel intrinsic justification for these choices of TV penalty and upscaling , by showing that they improve the saliency maps by making them more sound , as empirically demonstrated in Section 6 . We complement this finding with a theoretical result in Section 5 that proves that TV regularization can help with soundness in a simple linear classification setting . Paper outline : Section 2 discusses prior works on saliency methods and evaluations . Section 3 delves further into the concept of completeness and soundness in the context of saliency evaluation , and describes our proposed metric of consistency score . Section 4 describes our saliency method and how it differs from prior approaches . Finally Section 5 describes various supportive experiments . 2 PRIOR APPROACHES . We delegate a more thorough description of prior work to Appendix D , but mention some common saliency and saliency evaluation methods here . Saliency methods aim to explain a model ’ s decision about an input . Saliency evaluation methods aim to evaluate the goodness of a saliency method . Saliency methods include backpropagation based approaches such as Gradient Input ( Shrikumar et al. , 2017 ) , LRP ( Binder et al. , 2016 ) , GradCAM ( Selvaraju et al. , 2019 ) , Smooth-Grad ( Smilkov et al. , 2017 ) . Another line of work is masking methods which include techniques based on averaging over randomly sampled masks ( Petsiuk et al. , 2018 ) , optimizing over meaningful mask perturbations ( Fong & Vedaldi , 2017 ) , and real time image saliency using a masking network ( Dabkowski & Gal , 2017 ) . Pixels that have been removed from the image by the mask may be replaced by greying out , by Gaussian blurring as in ( Fong & Vedaldi , 2017 ) , or with infillers such as CA-GAN ( Yu et al. , 2018 ) used in ( Chang et al. , 2018 ; Phang et al. , 2020 ) , or DFNet ( Hong et al. , 2019 ) . De Cao et al . ( 2020 ) find masks using differentiable masking . Boolean logic is another approach for saliency methods ( Ignatiev et al. , 2019b ; a ; Macdonald et al. , 2019 ; Mu & Andreas , 2020 ; Zhou et al. , 2018 ) . Arguments about saliency . As saliency methods have arisen , discussion about them has occurred e.g . in ( Seo et al. , 2018 ; Fryer et al. , 2021 ; Gu et al. , 2018 ; Sundararajan & Najmi , 2020 ) . Some works reveal situations where Shapley axioms work against feature selection or where Shapley values may be calculated in conflicting ways ( Fryer et al. , 2021 ; Sundararajan & Najmi , 2020 ) . Others question why noise adding saliency methods work ( Seo et al. , 2018 ) , with some pointing out that some explanations are not class discriminative ( Gu et al. , 2018 ) . Saliency evaluation methods . Extrinsic evaluation metrics include the WSOL metric , and Pointing Game metric proposed by Zhang et al . ( 2018 ) and ROAR ( Hooker et al. , 2019 ) . Other more intrinsic methods include early saliency evaluation techniques like MorF and LerF ( Samek et al. , 2016 ) , Insertion and Deletion game proposed by Petsiuk et al . ( 2018 ) , which involve either inserting pixels in order of most importance or deleting pixels in order of most importance . BAM ( Yang & Kim , 2019 ) creates saliency maps by pasting object pixels from MSCOCO ( Lin et al. , 2014 ) The Saliency Metric proposed by Dabkowski & Gal ( 2017 ) thresholds saliency values above some α chosen on a holdout set , finds the smallest bounding box containing these pixels , upsamples and measures the ratio of bounding box area to model accuracy on the cropped image , s ( a , p ) = log ( max ( a , 0.05 ) ) − log ( p ) where a is the area of the bounding box and p is the class probability of the upsampled image . Controversies . Brunke et al . ( 2020 ) show that perturbation methods are sensitive to baseline and Petsiuk et al . ( 2018 ) point out that human centric explanations ( based on bounding boxes ) may not reveal why the model made a certain decision . Our notion of intrinsic saliency method aligns with the latter idea , and we introduce a new criteria for evaluating saliency maps : soundness , which captures the notion that maps should not produce explanations for low probability labels .

Summary: The paper tackles the problem of model explainibility from the perspective of heatmap-based visualization approaches. It argues that achieving a good heatmap to explain the model decision is possible by considering two notions, completeness and soundness, borrowed from logic. In fact, the completeness condition is what already considered during the formulation of similar methods. Thus the novelty lies on studying and analyzing the usefulness of soundness. The paper, thus, proposes a metric to evaluate both completeness and soundness that is used to learn a mask that explains the model prediction.