Variational saliency maps for explaining model's behavior

1 INTRODUCTION . Since the advent of deep learning brought significant improvement in general machine learning tasks ( Krizhevsky et al. , 2012 ) , explaining deep networks have become an important issue ( Ribeiro et al . ( 2016 ) ) . Problems inherent in training a deep neural network , such as fairness ( Arrieta et al. , 2020 ) or the model classifying based on unintended features ( Ribeiro et al. , 2016 ) , can be mitigated when the model is finely explained . Therefore , the models that have gained users ’ trust through explanation are preferred in practical applications . Saliency maps , also called attribution maps or relevance maps , have been widely used for interpretability methods in classification tasks , typically in an image domain ( Simonyan et al. , 2013 ) . A saliency map represents the importance of each feature of given data that influences the model ’ s decision . There have been several approaches for obtaining the saliency map , which are backpropagation based methods ( Ancona et al. , 2017 ; Bach et al. , 2015 ; Lundberg & Lee , 2017 ; Montavon et al. , 2017 ; Selvaraju et al. , 2017 ; Shrikumar et al. , 2017 ; Simonyan et al. , 2013 ; Smilkov et al. , 2017 ; Srinivas & Fleuret , 2019 ; Sundararajan et al. , 2017 ) and perturbation based methods ( Chang et al. , 2019 ; Chen et al. , 2018 ; Dabkowski & Gal , 2017 ; Fong et al. , 2019 ; Fong & Vedaldi , 2017 ; Schulz et al. , 2020 ; Zeiler & Fergus , 2014 ; Zintgraf et al. , 2017 ) . Regardless of the approaches , the common implicit assumption shared by most of the previous interpretability methods is that a saliency map exists in a deterministic manner when a model and an input data are given : one attribution map is provided to explain the model ’ s decision for each data point . Instead of the implicit assumption , we propose a stochastic approach called Variational Saliency maps ( VarSal ) where it is assumed that the interpretation has inherent randomness . The intuition stems from the stochastic effect that makes interpretation methods more explainable . For instance , FIDO ( Chang et al. , 2019 ) expands the search space of the mask by drawing it from Bernoulli distribution . This approach prevents the mask to be searched in the local space when it is directly optimized ( Fong & Vedaldi , 2017 ) . The example informs us that the stochastic property draws better interpretation . We define the posterior distribution as the probability of the saliency map when the training data and the classifier are given . To make the posterior behave as the distribution of explanation , it is essential to carefully design the likelihood function and the prior distribution . We follow the idea of perturbation based methods to form the likelihood where the input that only contains features which correspond to high attribution in a saliency map is likely to describe the classifier ’ s behavior . For modeling the prior , we propose a new covariance matrix of Gaussian distribution that implies the property of having a positive correlation among attributions of adjacent pixels . As this property mimics total variation ( TV ) regularization , we name the prior as soft-TV Gaussian prior . After modeling the likelihood and the prior , the Variational Bayesian method ( Hoffman et al. , 2013 ; Kingma & Welling , 2013 ) is used since the posterior is intractable . After the optimization , unlike most of perturbation based methods , VarSal produces a real-time saliency map since only a single forward pass is required for generating it . Also , the VarSal method provides high quality in the visual inspection where sophisticated borderlines exist with objectoriented attention . We compare VarSal with baseline methods on the perturbation benchmark test to show the effectiveness of our approach . At the end , we examine the benefit of employing a posterior distribution , which is uncertainty over the explanation . 2 RELATED WORK . In this section , we take a look at perturbation based interpretability methods . Fong & Vedaldi ( 2017 ) optimize the cost function with respect to the mask which indicates the most important features in an image for the classifier ’ s prediction . This approach is further developed by Fong et al . ( 2019 ) where they introduce a new method for making a perturbed image which helps to reduce hyper-parameters and produces better qualitative results . Both methods should optimize the mask every time they receive input , which is computationally expensive . Dabkowski & Gal ( 2017 ) relax the problem of time complexity by using a trained network of which the output is a saliency mask . However , all three methods have a limitation for producing importance ranking among features of a given image since their objective is to produce a binary mask . PDA ( Zintgraf et al. , 2017 ) produces a saliency map from a different perspective . It computes the importance of each pixel by regarding it as an unobserved pixel and marginalizes it out to get the predictive probability output of the classifier . The same idea is used in FIDO ( Chang et al. , 2019 ) to generate a perturbed image that is regarded as a sample from training data distribution . It optimizes the parameters of a Bernoulli dropout distribution for making a saliency mask . It helps exploring the search space of binary mask rather than being limited to local search since the mask is sampled from the distribution for each training iteration . Our method is similar to FIDO in that VarSal also explores the search space by sampling the saliency map from the encoder in the training phase . There is an information theoretic approach for explaining the classifier ’ s prediction . Schulz et al . ( 2020 ) adopt an information bottleneck for restricting the flow of information in an intermediate layer by adding noise . They find the importance of each feature by calculating the information flow . Chen et al . ( 2018 ) also adopt mutual information concept and optimize its variational bound for training a network that maps an input image to a saliency map . VarSal is similar in that we also train the encoder network by optimizing the evidence lower bound ( ELBO ) . However , our method differs in that we regard the saliency map as a random variable and aim to calculate the posterior over the saliency map . 3 VARIATIONAL SALIENCY MAPS In this section , we introduce details of the VarSal method which provides stochastic saliency maps . Let us define a pre-trained classifier that we aim to interpret as M : Rc×h×w → Y where x ∈ Rc×h×w is an input with c , h , and w to be channel , height , and width of the input image , respectively , and Y = { 1 , 2 , . . . , K } is a set of classes . The classifier M provides categorical probability PM ( · ) = ŷ ∈ 4K−1 where 4K−1 is a K−1 simplex . Since the purpose of a saliency map s ∈ Rh×w is to describe the behavior of the classifier ’ s prediction , our goal is to calculate the posterior distribution of the saliency map , p ( s|x , ŷ ) ( solid lines in Figure 1 ) . By Bayes ’ rule , the posterior is stated as : p ( s|x , ŷ ) = p ( ŷ|x , s ) p ( s|x ) / Z , ( 1 ) where Z is the marginal likelihood . To calculate the posterior , we should model two terms : the likelihood p ( ŷ|x , s ) and the prior p ( s|x ) . 3.1 MODELING LIKELIHOOD . The likelihood should be well-designed to make the posterior over the saliency map explain the behavior of the classifier . We focus on the property that the importance of each feature in an image is determined by observing the response of the classifier ’ s output when the feature is perturbed ( Zeiler & Fergus , 2014 ; Fong & Vedaldi , 2017 ; Zintgraf et al. , 2017 ) . More specifically , important features are enough to correctly classify the input as target class with high confidence . This concept is first introduced by Dabkowski & Gal ( 2017 ) , and called smallest sufficient region ( SSR ) . The difference between SSR and our approach is that we do not consider the smallest sufficient region , but rather rank the features ( therefore , s ∈ Rh×w , not s ∈ { 0 , 1 } h×w ) . Moreover , we consider not the target class but the categorical probability to interpret the model itself . The likelihood is designed such that the aforementioned properties satisfy − log p ( ŷ|x , s , k ) = DKL [ PM ( x ) ‖ PM ( x τ ( k ) ( s ) ) ] + const , ( 2 ) p ( ŷ|x , s ) = Ep ( k ) [ p ( ŷ|x , s , k ) ] , ( 3 ) where DKL is a Kullback-Leibler ( KL ) divergence , τ ( k ) is a top-k operation , and is a perturb operation that makes local perturbation of input x . The top-k operation applied to the saliency map , τ ( k ) ( s ) ∈ { 0 , 1 } h×w , acts as a mask where [ τ ( k ) ( s ) ] i , j = 1 when si , j corresponds to the biggest k attributions in s. This way , the top-k operation makes the conditional likelihood in equation 2 to consider only the selected features in the input . By varying k , the amount of selected features is controlled , and we set p ( k ) as uniform distribution . To make the local perturbation of input x using perturb operation , we follow the method proposed by Fong & Vedaldi ( 2017 ) , x m = x ◦m + x̃ ◦ ( 1 −m ) , where x̃ is a baseline input , and ◦ is a pointwise multiplication . We bring three baseline settings in our experiment : blurred baseline1 , noise baseline2 , and mean baseline3 . The equation 2 states that the categorical probability ŷ is more likely when the distance between the classifier ’ s predictive probability of input x and that of perturbed input is close for given k. This makes sense since better saliency map that explains the classifier ’ s behavior would approximate the model ’ s prediction closer with the selected features of top-k attributions . Also , we do not consider the ground-truth class or the top-1 predicted class , but rather whole classes with predictive probability in order to examine the classifier ’ s behavior itself . To consider various values of k , we also take the expectation in equation 3 . 3.2 SOFT-TV GAUSSIAN PRIOR . The easiest way to model the prior distribution p ( s|x ) is to consider it as independent of x , p ( s|x ) = p ( s ) , and design it as standard Gaussian distribution N ( vec ( s ) ; 0 , I ) where vec ( s ) ∈ Rhw is a vectorized version of s. However , the standard Gaussian prior does not consider the belief over the saliency map that the attribution of neighbor pixels might have correlation ( Fong et al. , 2019 ) . Therefore , we propose a new prior distribution that expresses the belief . While Dabkowski & Gal ( 2017 ) ; Fong & Vedaldi ( 2017 ) proposed total variation ( TV ) regularization to prevent the saliency mask from being occurred adversarial artifacts , we mimic the TV method in building the prior distribution to provide the positive correlation between neighbor attributions . To be more specific , we design the prior distribution as zero mean Gaussian distribution , N ( vec ( s ) ; 0 , Σ ) , and infuse the TV knowledge to the covarianace matrix by setting Σi , j > 0 when pixel i and pixel j are identical or adjacent . We have Σi , j = 1 , if i=j α , if i−j ∈ { −w , −1 , 1 , w } and j ∈ Adji α2 , if i−j ∈ { −w−1 , −w+1 , w−1 , w+1 } and j ∈ Adji 0 , otherwise , ( 4 ) 1We define “ blurred baseline ” as an input image blurred with Gaussian kernel . 2The “ noise baseline ” is defined as Gaussian noise . 3We term “ mean baseline ” when the baseline is set to be the per channel mean of an original image and added by Gaussian noise . where α > 0 and Adji is the adjacent index set of pixel i ( Figure 9 in Appendix B ) . This way , we grant the TV knowledge to the Gaussian prior , and call it soft-TV Gaussian prior .

Summary: This paper proposed a method for generating saliency maps for image classifiers that are stochastic (instead of deterministic). The probabilistic model assumes a saliency map random variable that generates the data with a classifier. The inference is done by variational methods. The paper presents several qualitative examples and a comparison to previous work using the pixel perturbation benchmark.