Variational saliency maps for explaining model's behavior

1 INTRODUCTION . Since the advent of deep learning brought significant improvement in general machine learning tasks ( Krizhevsky et al. , 2012 ) , explaining deep networks have become an important issue ( Ribeiro et al . ( 2016 ) ) . Problems inherent in training a deep neural network , such as fairness ( Arrieta et al. , 2020 ) or the model classifying based on unintended features ( Ribeiro et al. , 2016 ) , can be mitigated when the model is finely explained . Therefore , the models that have gained users ’ trust through explanation are preferred in practical applications . Saliency maps , also called attribution maps or relevance maps , have been widely used for interpretability methods in classification tasks , typically in an image domain ( Simonyan et al. , 2013 ) . A saliency map represents the importance of each feature of given data that influences the model ’ s decision . There have been several approaches for obtaining the saliency map , which are backpropagation based methods ( Ancona et al. , 2017 ; Bach et al. , 2015 ; Lundberg & Lee , 2017 ; Montavon et al. , 2017 ; Selvaraju et al. , 2017 ; Shrikumar et al. , 2017 ; Simonyan et al. , 2013 ; Smilkov et al. , 2017 ; Srinivas & Fleuret , 2019 ; Sundararajan et al. , 2017 ) and perturbation based methods ( Chang et al. , 2019 ; Chen et al. , 2018 ; Dabkowski & Gal , 2017 ; Fong et al. , 2019 ; Fong & Vedaldi , 2017 ; Schulz et al. , 2020 ; Zeiler & Fergus , 2014 ; Zintgraf et al. , 2017 ) . Regardless of the approaches , the common implicit assumption shared by most of the previous interpretability methods is that a saliency map exists in a deterministic manner when a model and an input data are given : one attribution map is provided to explain the model ’ s decision for each data point . Instead of the implicit assumption , we propose a stochastic approach called Variational Saliency maps ( VarSal ) where it is assumed that the interpretation has inherent randomness . The intuition stems from the stochastic effect that makes interpretation methods more explainable . For instance , FIDO ( Chang et al. , 2019 ) expands the search space of the mask by drawing it from Bernoulli distribution . This approach prevents the mask to be searched in the local space when it is directly optimized ( Fong & Vedaldi , 2017 ) . The example informs us that the stochastic property draws better interpretation . We define the posterior distribution as the probability of the saliency map when the training data and the classifier are given . To make the posterior behave as the distribution of explanation , it is essential to carefully design the likelihood function and the prior distribution . We follow the idea of perturbation based methods to form the likelihood where the input that only contains features which correspond to high attribution in a saliency map is likely to describe the classifier ’ s behavior . For modeling the prior , we propose a new covariance matrix of Gaussian distribution that implies the property of having a positive correlation among attributions of adjacent pixels . As this property mimics total variation ( TV ) regularization , we name the prior as soft-TV Gaussian prior . After modeling the likelihood and the prior , the Variational Bayesian method ( Hoffman et al. , 2013 ; Kingma & Welling , 2013 ) is used since the posterior is intractable . After the optimization , unlike most of perturbation based methods , VarSal produces a real-time saliency map since only a single forward pass is required for generating it . Also , the VarSal method provides high quality in the visual inspection where sophisticated borderlines exist with objectoriented attention . We compare VarSal with baseline methods on the perturbation benchmark test to show the effectiveness of our approach . At the end , we examine the benefit of employing a posterior distribution , which is uncertainty over the explanation . 2 RELATED WORK . In this section , we take a look at perturbation based interpretability methods . Fong & Vedaldi ( 2017 ) optimize the cost function with respect to the mask which indicates the most important features in an image for the classifier ’ s prediction . This approach is further developed by Fong et al . ( 2019 ) where they introduce a new method for making a perturbed image which helps to reduce hyper-parameters and produces better qualitative results . Both methods should optimize the mask every time they receive input , which is computationally expensive . Dabkowski & Gal ( 2017 ) relax the problem of time complexity by using a trained network of which the output is a saliency mask . However , all three methods have a limitation for producing importance ranking among features of a given image since their objective is to produce a binary mask . PDA ( Zintgraf et al. , 2017 ) produces a saliency map from a different perspective . It computes the importance of each pixel by regarding it as an unobserved pixel and marginalizes it out to get the predictive probability output of the classifier . The same idea is used in FIDO ( Chang et al. , 2019 ) to generate a perturbed image that is regarded as a sample from training data distribution . It optimizes the parameters of a Bernoulli dropout distribution for making a saliency mask . It helps exploring the search space of binary mask rather than being limited to local search since the mask is sampled from the distribution for each training iteration . Our method is similar to FIDO in that VarSal also explores the search space by sampling the saliency map from the encoder in the training phase . There is an information theoretic approach for explaining the classifier ’ s prediction . Schulz et al . ( 2020 ) adopt an information bottleneck for restricting the flow of information in an intermediate layer by adding noise . They find the importance of each feature by calculating the information flow . Chen et al . ( 2018 ) also adopt mutual information concept and optimize its variational bound for training a network that maps an input image to a saliency map . VarSal is similar in that we also train the encoder network by optimizing the evidence lower bound ( ELBO ) . However , our method differs in that we regard the saliency map as a random variable and aim to calculate the posterior over the saliency map . 3 VARIATIONAL SALIENCY MAPS In this section , we introduce details of the VarSal method which provides stochastic saliency maps . Let us define a pre-trained classifier that we aim to interpret as M : Rc×h×w → Y where x ∈ Rc×h×w is an input with c , h , and w to be channel , height , and width of the input image , respectively , and Y = { 1 , 2 , . . . , K } is a set of classes . The classifier M provides categorical probability PM ( · ) = ŷ ∈ 4K−1 where 4K−1 is a K−1 simplex . Since the purpose of a saliency map s ∈ Rh×w is to describe the behavior of the classifier ’ s prediction , our goal is to calculate the posterior distribution of the saliency map , p ( s|x , ŷ ) ( solid lines in Figure 1 ) . By Bayes ’ rule , the posterior is stated as : p ( s|x , ŷ ) = p ( ŷ|x , s ) p ( s|x ) / Z , ( 1 ) where Z is the marginal likelihood . To calculate the posterior , we should model two terms : the likelihood p ( ŷ|x , s ) and the prior p ( s|x ) . 3.1 MODELING LIKELIHOOD . The likelihood should be well-designed to make the posterior over the saliency map explain the behavior of the classifier . We focus on the property that the importance of each feature in an image is determined by observing the response of the classifier ’ s output when the feature is perturbed ( Zeiler & Fergus , 2014 ; Fong & Vedaldi , 2017 ; Zintgraf et al. , 2017 ) . More specifically , important features are enough to correctly classify the input as target class with high confidence . This concept is first introduced by Dabkowski & Gal ( 2017 ) , and called smallest sufficient region ( SSR ) . The difference between SSR and our approach is that we do not consider the smallest sufficient region , but rather rank the features ( therefore , s ∈ Rh×w , not s ∈ { 0 , 1 } h×w ) . Moreover , we consider not the target class but the categorical probability to interpret the model itself . The likelihood is designed such that the aforementioned properties satisfy − log p ( ŷ|x , s , k ) = DKL [ PM ( x ) ‖ PM ( x τ ( k ) ( s ) ) ] + const , ( 2 ) p ( ŷ|x , s ) = Ep ( k ) [ p ( ŷ|x , s , k ) ] , ( 3 ) where DKL is a Kullback-Leibler ( KL ) divergence , τ ( k ) is a top-k operation , and is a perturb operation that makes local perturbation of input x . The top-k operation applied to the saliency map , τ ( k ) ( s ) ∈ { 0 , 1 } h×w , acts as a mask where [ τ ( k ) ( s ) ] i , j = 1 when si , j corresponds to the biggest k attributions in s. This way , the top-k operation makes the conditional likelihood in equation 2 to consider only the selected features in the input . By varying k , the amount of selected features is controlled , and we set p ( k ) as uniform distribution . To make the local perturbation of input x using perturb operation , we follow the method proposed by Fong & Vedaldi ( 2017 ) , x m = x ◦m + x̃ ◦ ( 1 −m ) , where x̃ is a baseline input , and ◦ is a pointwise multiplication . We bring three baseline settings in our experiment : blurred baseline1 , noise baseline2 , and mean baseline3 . The equation 2 states that the categorical probability ŷ is more likely when the distance between the classifier ’ s predictive probability of input x and that of perturbed input is close for given k. This makes sense since better saliency map that explains the classifier ’ s behavior would approximate the model ’ s prediction closer with the selected features of top-k attributions . Also , we do not consider the ground-truth class or the top-1 predicted class , but rather whole classes with predictive probability in order to examine the classifier ’ s behavior itself . To consider various values of k , we also take the expectation in equation 3 . 3.2 SOFT-TV GAUSSIAN PRIOR . The easiest way to model the prior distribution p ( s|x ) is to consider it as independent of x , p ( s|x ) = p ( s ) , and design it as standard Gaussian distribution N ( vec ( s ) ; 0 , I ) where vec ( s ) ∈ Rhw is a vectorized version of s. However , the standard Gaussian prior does not consider the belief over the saliency map that the attribution of neighbor pixels might have correlation ( Fong et al. , 2019 ) . Therefore , we propose a new prior distribution that expresses the belief . While Dabkowski & Gal ( 2017 ) ; Fong & Vedaldi ( 2017 ) proposed total variation ( TV ) regularization to prevent the saliency mask from being occurred adversarial artifacts , we mimic the TV method in building the prior distribution to provide the positive correlation between neighbor attributions . To be more specific , we design the prior distribution as zero mean Gaussian distribution , N ( vec ( s ) ; 0 , Σ ) , and infuse the TV knowledge to the covarianace matrix by setting Σi , j > 0 when pixel i and pixel j are identical or adjacent . We have Σi , j = 1 , if i=j α , if i−j ∈ { −w , −1 , 1 , w } and j ∈ Adji α2 , if i−j ∈ { −w−1 , −w+1 , w−1 , w+1 } and j ∈ Adji 0 , otherwise , ( 4 ) 1We define “ blurred baseline ” as an input image blurred with Gaussian kernel . 2The “ noise baseline ” is defined as Gaussian noise . 3We term “ mean baseline ” when the baseline is set to be the per channel mean of an original image and added by Gaussian noise . where α > 0 and Adji is the adjacent index set of pixel i ( Figure 9 in Appendix B ) . This way , we grant the TV knowledge to the Gaussian prior , and call it soft-TV Gaussian prior .

Summary: This paper proposes a new interpretability method for image classification networks. It considers a saliency map as a random variable and aims to calculate the posterior distribution over the saliency map.  The likelihood function and the prior distribution are then designed to make the posterior distribution over the saliency map explain the behavior of the classifier’s prediction. Quantitative evaluation on the perturbation benchmark as well as qualitative result show the effectiveness of the proposed method over baselines.