Variational saliency maps for explaining model's behavior
Authors: Jae Myung Kim
Eunji Kim
Seokhyeon Ha
Sungroh Yoon
Jungwoo Lee
Venue:  (2021)
Keywords: Interpretability
XAI
Variational Inference

Abstract: Saliency maps have been widely used to explain the behavior of an image classifier. We introduce a new interpretability method which considers a saliency map as a random variable and aims to calculate the posterior distribution over the saliency map. The likelihood function is designed to measure the distance between the classifier's predictive probability of an image and that of locally perturbed image. For the prior distribution, we make attributions of adjacent pixels have a positive correlation. We use a variational approximation, and show that the approximate posterior is effective in explaining the classifier's behavior. It also has benefits of providing uncertainty over the explanation, giving auxiliary information to experts on how much the explanation is trustworthy.

Under review as a conference paper at ICLR 2021
VARIATIONAL SALIENCY MAPS FOR EXPLAINING MODEL'S S BEHAVIOR
Anonymous authors Paper under double-blind review
ABSTRACT
Saliency maps have been widely used to explain the behavior of image classi- fier. We introduce a new interpretability method which considers a saliency map as random variable and aims to calculate the posterior distribution over the saliency map. The likelihood function is designed to measure the distance between the classifier's 's predictive probability of an image and that of locally perturbed image. For the prior distribution, we make attributions of adjacent pixels have a positive correlation. We use variational approximation, and show that the approximate posterior is effective in explaining the classifier's s behavior. It also has benefits of providing uncertainty over the explanation, giving auxiliary information to experts on how much the explanation is trustworthy.
INTRODUCTION
Since the advent of deep learning brought significant improvement in general machine learning tasks (Krizhevsky al. 2012), explaining deep networks have become important issue (Ribeiro et al. (0166)) Problems inherent in training a deep neural network, such as fairness (Arrieta et al.) 2020 or the model classifying based on unintended features (Ribeiro al. (016), can be mitigated when the model is finely explained. Therefore, the models that have gained users' trust through explanation are preferred in practical applications.
Saliency maps, also called attribution maps or relevance maps, have been widely used for inter- pretability methods in classification tasks, typically in an image domain (Simonyan et al. (013). A saliency map represents the importance of each feature of given data that influences the model's decision. There have been several approaches for obtaining the saliency map, which are back-
propagation based methods (Ancona et al., 2017; Bach et al. [005 Lundberg & Lee 2017 Mon-
lavon et al. 2017; Selvaraju et al. 2017]] Shrikumar et al., 2017 Simonyan et al. 2013; Smilkov
et al. 2017 Srinivas & Fleuret 2019) Sundararajan et al. 2017) and perturbation based methods
(Chang et al. 2019: Chen et al. 2018] Dabkowski & Gal 2017]] Fong et al. 2019 Fong & Vedaldi,
2017 Schulz et al. 2020 Zeiler & Fergus 2014 Zintgraf
al., 2017). Regardless of the ap- proaches, the common implicit assumption shared by most of the previous interpretability methods that a saliency map exists in a deterministic manner when a model and an input data are given: one attribution map is provided to explain the model's decision for each data point.
Instead of the implicit assumption, we propose stochastic approach called Variational Saliency maps (VarSal) where it is assumed that the interpretation has inherent randomness. The intuition stems from the stochastic effect that makes interpretation methods more explainable. For instance, FIDO (Chang et al., 2019) expands the search space of the mask by drawing it from Bernoulli distribution. This approach prevents the mask be searched in the local space when directly optimized (Fong & Vedaldi 2017). The example informs us that the stochastic property draws better interpretation.
We define the posterior distribution as the probability of the saliency map when the training data and the classifier are given. To make the posterior behave as the distribution of explanation, it is essential to carefully design the likelihood function and the prior distribution. We follow the idea of perturbation based methods to form the likelihood where the input that only contains features which correspond to high attribution in saliency map is likely to describe the classifier's behavior. For modeling the prior, we propose a new covariance matrix of Gaussian distribution that implies the property of having positive correlation among attributions of adjacent pixels. As this property

---

Under review as a conference paper at ICLR 2021
mimics total variation (TV) regularization, name the prior soft-TV Gaussian prior. After mod- eling the likelihood and the prior, the Variational Bayesian method (Hoffman et al. 2013 Kingma
& Welling 2013) is used since the posterior is intractable.
After the optimization, unlike most of perturbation based methods, VarSal produces a real-time saliency map since only single forward pass is required for generating it. Also, the VarSal method provides high quality in the visual inspection where sophisticated borderlines exist with object- oriented attention. We compare VarSal with baseline methods on the perturbation benchmark test to show the effectiveness of our approach. At the end, we examine the benefit of employing posterior distribution, which uncertainty over the explanation.
2 RELATED WORK
In this section, we take a look at perturbation based interpretability methods. Fong & Vedaldi (2017) optimize the cost function with respect to the mask which indicates the most important features an image for the classifier's prediction. This approach is further developed by Fong et al. (2019) where they introduce new method for making a perturbed image which helps to reduce hyper-parameters and produces better qualitative results. Both methods should optimize the mask every time they receive input, which is computationally expensive. Dabkowski & Gal (2017) relax the problem of time complexity by using trained network of which the output is saliency mask. However, all three methods have a limitation for producing importance ranking among features of a given image since their objective is to produce binary mask.
PDA Zinngrrff et al. 2017) produces a saliency map from a different perspective. It computes the importance of each pixel by regarding it unobserved pixel and marginalizes it out get the predictive probability output of the classifier. The same idea used in FIDO (Chang et al. 2019) to generate a perturbed image that is regarded as a sample from training data distribution. It optimizes the parameters of a Bernoulli dropout distribution for making a saliency mask. It helps exploring the search space of binary mask rather than being limited local search since the mask is sampled from the distribution for each training iteration. Our method is similar to FIDO in that VarSal also explores the search space by sampling the saliency map from the encoder in the training phase.
There is an information theoretic approach for explaining the classifier's prediction. [chulz et al. 2020) adopt an information bottleneck for restricting flow of information in an intermediate layer by adding noise. They find the importance of each feature by calculating the information flow. Chen et al. (2018) also adopt mutual information concept and optimize its variational bound for training a network that maps an input image to saliency map. VarSal is similar in that we also train the encoder network by optimizing the evidence lower bound (ELBO). However, our method differs in that regard the saliency map random variable and aim to calculate the posterior over the saliency map.
VARIATIONAL SALIENCY MAPS
In this section, introduce details of the VarSal method which provides stochastic saliency maps. Let us define pre-trained classifier that we aim interpret as M gexhxw - where E cx is an input with c, h, and w to be channel, height, and width of the input image, respectively, and y = {1, 2, .... K) is a set of classes. The classifier M provides categorical probability (. 9 E K-1 where A K-1 is a K-1 1 simplex. Since the purpose of saliency map 8 & is to describe the behavior of the classifier's prediction, our goal is to calculate the posterior distribution of the saliency map, p(s xx x (solid lines in Figure []. By Bayes' rule, the posterior is stated as:
0
M
i
Figure 1: Graphical model.
p(s = p(ū)0,s) p(s|x)) / Z,
(1)
where is the marginal likelihood. To calculate the posterior, we should model two terms: the likelihood p()) and the prior p(s(.)).
2

---

Under review as a conference paper at ICLR 2021
3.1 MODELING LIKELIHOOD
The likelihood should be well-designed to make the posterior over the saliency map explain the behavior of the classifier. We focus the property that the importance of each feature in an image is determined by observing the response of the classifier's s output when the feature is perturbed Zeiler & Fergus 2014; Fong & Vedaldi 2017: Zintgraf al., 2017). More specifically, important features are enough to correctly classify the input as target class with high confidence. This concept first introduced by [Dbkkoski & Gal (2017). and called smallest sufficient region (SSR). The difference between SSR and our approach is that we do not consider the smallest sufficient region, but rather rank the features (therefore, 8 h not & E {0, (0,1)*x0 1) Moreover, we consider not the target class but the categorical probability to interpret the model itself. The likelihood is designed such that the aforementioned properties satisfy
- (o, - DKI. (Mr()) || P + const,
(2)
p)))) = (p))) ((xk)p(((,,,
(3)
where DKL is a Kullback-Leible (KL) divergence, 7(K) is a top-k operation, and @ is a perturb operation that makes local perturbation of input æ. The top-k operation applied to the saliency map, (8) E {0, _/x acts as a mask where (s) k attributions in 8. This way. the top-k operation makes the conditional likelihood in equation to consider only the selected features in the input. By varying k. the amount of selected features controlled, and we set p(k) as uniform distribution. To make the local perturbation of input æ using perturb operation @. we follow the method proposed by Fong & Vedaldi (2017), æ @ m = a o m + o (1 m), where ă is baseline input, and is a pointwise multiplication. We bring three baseline settings in our experiment: blurred baselind noise baseling] and mean baseline
when 8t, corresponds to the biggest
The equation 2 states that the categorical probability ý is more likely when the distance between the classifier's 's predictive probability of input and that of perturbed input is close for given k. This makes sense since better saliency map that explains the classifier's 's behavior would approximate the model's prediction closer with the selected features of top-k attributions. Also, do not consider the ground-truth class or the top-1 predicted class, but rather whole classes with predictive probabil- ity in order to examine the classifier's behavior itself. To consider various values of k, also take the expectation in equation 3
3.2 SOFT-TV GAUSSIAN PRIOR
The easiest way to model the prior distribution p(s[z) to consider it as independent of æ. p(a(z) = p(s), and design it as standard Gaussian distribution N (vve((s); 0, I) where vec(s) ihw is vectorized version of s. However, the standard Gaussian prior does not consider the belief over the saliency map that the attribution of neighbor pixels might have correlation (Fong et al., 2019). Therefore, we propose a new prior distribution that expresses the belief. While Dabkowski & Gal Fong Fong Vedaldi (2017) proposed total variation (TV) regularization prevent the saliency mask from being occurred adversarial artifacts, we mimic the TV method in building the prior distribution provide the positive correlation between neighbor attributions. To be more specific, we design the prior distribution zero mean Gaussian distribution, A (vec(s); O, E), and infuse the TV knowledge to the covarianace matrix by setting Σ, when pixel i and pixel j are identical or adjacent. We have
i=j i-j E (-w, -1, 1, w) and j E Adj;
a2
(4)
a² if i-je f -----11 1, - w+1, w-1, w and j E Adj,
0, otherwise.
We define "blurred bascline" as input image blurred with Gaussian kernel.
"The "noise baseline" defined as Gaussian noise.
We term "mean haseline" when the baseline is set to he the per channel mean of an original image and added by Gaussian noise.
3

---

Under review as a conference paper at ICLR 2021
where 0 and Adj, is the adjacent index set of pixel i (Figure 9 in Appendix (B) This way, grant the TV knowledge to the Gaussian prior, and call it soft-TV Gaussian prior.
3.3 VARIATIONAL INFERENCE ON SALIENCY MAPS
Modeling the likelihood as equation and the prior as (Vvec(s); 0, Σ) makes the posterior of equation intractable. Therefore, approximate it with the distribution qo (8 (s|ær) parameterized by i (dotted lines in Figure I where the objective is to minimize the KI. divergence between qo (8|2r) (a))
and p( p.))
argmin PKI[(go(s(a) | p(s(x.j))
argmin E Alg DKL (s(x))) Il p(s[x))
(5)
(*+)
We apply a mean-field approximation with univariate Gaussian for each factorized term approxi- mate posterior, 4o (s(ar) = N vec(s); (o(a), (x))) ), where po(-) E Tikw is the mean of the distribution and diag( (-)() (:)) E Rhuxhe is the diagonal covariance matrix with the main diagonal to be (-) Rh" μi and My are collectively called encoder network parameterized by 0. The encoder network is optimized with the training dataset used for training the classifier M, and the reparameterization trick (Kingma & Welling 2013) is applied during optimization. The schematic description shown in Appendix|A
There are two problems in optimizing the equation 5 non-differentiable top- operation in equation (*) and computationally expensiveness in equation (*+). In case of (*), the top-k operation ((k) is non-differentiable where the gradient cannot flow backward. To overcome this issue, we approxi- mate it with differentiable SOFT operator proposed by Xie et al. (2020). This allows flowing the gradient from the classifier M to the encoder parameters 6. Note that the classifier M a pre-trained classifier that aim to interpret, and thus should be fixed.
Since the size of covariance
large, which he hnc, it is computationally ex expensive to calculate (**) when it is naively used (in case of Imagenet dataset (Russakovsky al. 2015), the size of Σ is 2244 !). We solve the problem by decomposing Σ with a Kronecker product:
Σ Kij X Kew
(6)
where @ is the Kronecker product, and K₁ Roxh and Kw Rex tridiagonal matrices with for main diagonal and a for first diagonal below and above the main diagonal (Figure 9 in Appendix After all, (**) be analytically derived as:
(o p(s[=)] diag (K=²))" rsh (vo) diag (KT sum (log va)
+ sum rsh (1t) N rsh (fer) (1=7 ()) + const,
where sum(-) is the summation of elements. For a vector b E Re we denote rsh(b) E Raxh as reshaping the vector to the matrix where [rsh(b)]as betwj- Also, for a square matrix B, diag B) is the vector where the ith entry is B,,- The derivation of equation 7 is provided in Appendix [D] The equation 7 shows that the computation is easily done in general deep learning frameworks such Pytorch (Paszke al. 2017).
EXPERIMENT
4.1 IMPLEMENTATION DETAIL
There are two terms in the objective function equation the reconstruction term (*) and the reg- ularization term (*4). The components the regularization term have high dimension, which is hw. This is usually too large that the regularization term becomes dominant in the loss function.
4

---

Under review as a conference paper at ICLR 2021
ilamingoo
bullifog
Image
Input Gradient
PDA
Real-Time
EP
FDD
VarSal (Ours)
Figure 2: Qualitative results. Compared with previous methods, VarSal visually captures the most sophisticated borderlines and object-oriented saliency map.
We avoid this phenomenon by introducing a hyper-parameter to the regularization term (Higgins al. 2017). We set the default value of 3 to be 1/(10hw:).
We test our method on the ImageNet dataset (Russakovsky et al., 2015). We use a pre-trained VGG16 (Simonyan & Zisserman, 2014) for the classifier M, and 16 convolutional layers for the encoder [4g and Va. As for the variable used in the top-k operator, we sample from the uniform distribution, & Uhow//1 10, 9hw/ 10), for each training iteration. The baseline input à is also domly selected among three baselines (blurred, noise, and mean baseline) for each training iteration. As for in the soft-TV Gaussian prior, proximate posterior used when performing qualitative and quantitative experiments. More details are given in Appendix C
0.4. Finally, if not mentioned, the mean of ap-
4.2 QUALITATIVE RESULTS
We perform visual inspection by comparing VarSal with previous interpretability methods. For fair comparison, with the data that are correctly classified, previous interpretability methods are performed based on the ground-truth target, while VarSal explains the predictive probability. In Figure 2 the first heatmap (Simonyan et al., 2013) are generated by the gradients of input. They visually highlights the object, but there exists sparsity. PDA (Zintgraf et al. 2017) usually provides unnecessary highlights since only considers spatially local parts in the optimization process. Real- time saliency (Dabkowski & Gal 2017) and EP (Fong al. 2019) show boolean mask for the saliency map with smoothed borderlines. This is because it optimizes the mask with the size smaller than the input, followed by performing upsampling for the final saliency mask. FIDO (Chang al.

---

Under review as a conference paper at ICLR 2021
2019)
provides the shape of the object to some extent, but it still has sparsity. Compared to the previous approaches, VarSal does not contain upsampling process, but rather directly provides the saliency map with the same size of input image. As the last heatmap shows, the VarSal highlights the object with more sophisticated borderlines. More results are provided in Appendix
To investigate the importance of modeling the prior distribution, we perform visual inspection between the VarSal method trained with standard Gaussian prior and that of soft-TV Gaussian prior. As Figure shows, VarSal trained with standard Gaussian prior provides saliency map with high frequency noise inside the ob- ject boundary. This is because the standard Gaussian prior does not constrain the adjacent attributions to have correlation. On the other side, VarSal optimized using the soft-TVV Gaussian prior shows smaller variation of attribution between adjacent pixels. Moreover, the noise on the background has been reduced when the soft-TV prior is used for modeling the prior distribution.
Sappa
Stuntie proo
SWI-TV pioo
Figure 3: Prior selection.
4.3 SANITY CHECK
The prerequisite for becoming an interpretability method to pass the sanity check (Adebayo et al. 2018). This is to identify whether the interpretability method provides saliency map dependent of classifier tested by randomizing the classifier's parameters. The difference between saliency maps obtained by the orig- inal classifier M and the parameter-randomizze classi- fier is measured by structural similarity index (SSIM) and Spearman rank correlation. It known that Guided Back- Prop (Springenberg al. 2014) and Guided GradCAM (Selvaraju al. 2017) do not pass the sanity check since the SSIM and the Spearman metric is close to 1. Instead, Figure shows that our method has lower values for each metric and different randomization of the classifier, indicating that VarSal has passed the sanity check.
data, and
Votia
lassn VGG16 initialized layers (top to bottom layers)
Figure 4: Sanity check.
4.4 QUANTITATIVE RESULTS
Low-k pixels perturbed
Inference time complexity
35
sampling
18a
Input Grodlent
intgg Grscient
divergeno
Smoothörad-sq
2.0
FIDO
Sccond
169
VarS
155
VarSal: soft-TV
VarSal wt sampling
165
0.6
10-2
10-4
10P
10
Vedill
Real Trme
Iapat-Geed
Fino
PDA
(%)
Iatg
Figure 5: (a) Quantitative results for pixel perturbation benehmark The results verify the usefulness of our method. (b) Time complexity of each interpretability methods. SG-sq. refers to SmoothGrad- squared.
Determining the state-of-the-art interpretability method is challenging since there is evaluation benchmark that exactly reflects the method's performance (Hooker et al. 2019). Commonly used benchmarks can split the superiority and inferiority of each method to some extent, but cannot actly rank them based on the quantitative evaluation indicator. We use pixel perturbation benchmark verify the usefulness of our method.

---

Under review as a conference paper at ICLR 2021
For the pixel perturbation metric, image pixels are erased that correspond the largest 4% saliency values (Ancona et al., 2017 Samek et al. 2016) or the smallest 4% saliency values (Srinivas & Fleuret 2019). and observe the response to the change of classifier's s output. In experiment, erase pixels with the latter procedure as the former is more prone to create unnecessary artifacts that lead to misunderstanding of the reason for the score drop (Srinivas & Fleuret 2019). We change pixel values of the input image to zero that correspond to the least k% values in the saliency map, and observe the KL divergence between the classifier's predictive probability of original input and that of perturbed input. The interpretability method is thought to be better when the distance is smaller.
It takes expensive computational time for PDA and FIDO method, about 25 minute and I minute per one image (Figure[5(t) Therefore, for evaluating PDA and FIDO, we randomly sample 100 data in the validation dataset to perform the top-k perturbation benchmark, with repeating the process 5 times. Other methods use entire validation dataset for the evaluation. For fair comparison, saliency map is generated from the predicted class for each interprebility methods except VarSal. Also, perturbing randomly drawn k pixels suggested as a control experiment. We omit drawing error range for this control experiment since the error (standard deviation) is less than 1e-2. The results are shown in Figure (5))). It is observed that Input- Gradient (Simonyan et al.] 2013), Integrated- Gradient (Sundararajan al., (0077), and PDA gets close to the control setting k gets larger. For others such as Smoduarrrad-qquare (Smilkov et al. [017]] Hooker et al.) 2019), FIDO, and VarSal, they have similar values throughout the change of k values. Moreover, VarSal performs better when using the soft-TV Gaussian prior than using the standard Gaussian prior.
An interesting point is that the saliency map obtained by sam- pling from the approximate posterior instead of the mean values results in low-quality (red dotted line in Figure a)). We spec- ulate this is because the sampling method produces artifacts in a perturbed image that cause score drop (Kurakin al., 2016). As shown in Figure 6 even though both the sampled saliency map and the mean saliency map captures objects with high val- ues, the sampled one has high-frequency noise. When the image pixels corresponding to low 1% of the noisy saliency map are perturbed, the perturbed image has the artifact of sharp color contrast between adjacent pixels that might cause score degrada- tion.
Inpel
Men
Figure 6: Sampled saliency map.
4.5 UNCERTAINTY OVER EXPLANATION
lapus
Input
Misss
Men
Standal dencorite
Stundadd davitiin
Stanaad devisation
(a) Origin
(b) Brightness
(4) Motion blur
Figure 7: Uncertainty over the explanation. The posterior distribution of saliency map gives two summaries, which are the explanation (mean) and the uncertainty over the explanation (standard deviation). They are compared qualitatively on the original dataset and two different shifted dataset.
One advantage of having posterior distribution over the saliency map is that it gives us the uncer- tainty of the explanation. This is done by summarizing the posterior with its covariance matrix. Since the VarSal approximates the posterior with factorized Gaussian, we observe the variance
7

---

Under review as a conference paper at ICLR 2021
of each attribution where the examples are shown in Figure 7a). While the explanation (second row) has higher attribution at the object, the uncertainty over the explanation (third row) presents at the borderline of the object.
We also qualitatively compare the posterior results on the shifted samples. Recently Hendrycks & Dietterich (2018) have established Imagenet-C dataset that is created by visually corrupting the data in the Imagenet dataset. The first row in Figure b and (c) respectively shows the example of images corrupted by "Brightness" and "Motion blur" with level of severity. While the explanation of corrupted images is similar to that of original images in that they all capture the object to some extent, the uncertainty of the explanation shows different appearance where the heatmaps of standard deviation are noisy in the corrupted images.
CONCLUSION AND FUTURE WORK
In this paper, we presented a new perspective on a saliency map where it is assumed to be a random variable. After designing the likelihood function and the prior distribution that makes the posterior distribution over the saliency map explain the behavior of the classifier's prediction, the approxi- mate posterior is optimized by maximizing ELBO. The experimental results were performed with the mean of the approximate posterior, and showed that our method has visually sharp borderlines with object-oriented saliency map. For quantitative results, the pixel perturbation benchmark is used to prove the effectiveness of our method. We verified that using the proposed soft-TV Gaussian distribution rather than the standard Gaussian distribution for modeling the prior has better perfor- mance in both qualitative and quantitative comparison. It was also shown that the proposed VarSal method has a strong advantage over other methods in terms of inference computation complexity. Finally, we showed that our method provides not only the explanation but also the uncertainty over the explanation.
There remain future works for producing better quality of the posterior distribution over saliency map. In modeling the likelihood, the problem of data distribution shift could be mitigated by gener- ating perturbed image that is expected to be sampled from training data distribution (Chang al. 2019). It would also interesting to consider axiomatic prior knowledge (Sundararajan et al., 2017 Srinivas & Fleuret 2019) in modeling the prior distribution and the likelihood. Finally, the uncer- lainty consideration over explaination is believed to be critical since it can tells us how much the explanation given by interpretability method be trusted, which needs to be studied further.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp- 9505-9515, 2018
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient-based altribution methods for deep neural networks. arXiv preprint 1=Xiv:1711.06110 2017.
Alejandro Barredo Arrieta, Natalia Dodz-Rodr,,,ez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Benjamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunitics and challenges to- ward responsible ai. Information Fusion, 58:82 115, 2020.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7), 2015.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image clas- sifiers by counterfactual generation. In International Conference on Learning Representations, 2019. httt:tttts::/ooeeee
net. bttpr//eweeeeeetteooooeeesseeiie
8

---

Under review as a conference paper at ICLR 2021
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning explain: An information-theoreticc perspective on model interpretation. International Conference on Ma- chine Learning, pp. 883-892 2018.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. Advances in Neural Information Processing Systems, pp. 6967-6976, 2017.
Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal per- turbations and smooth masks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 195002958,, 2019.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3429-3437, 2017.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. In International Conference on Learning Representations, 2018.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. veta-vae: Learning basic visual concepts with constrained variational framework. Iclr, 2(5):6, 2017.
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational infer- ence. The Journal of Machine Learning Research, 14(1):1003-1347, 2013.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretabil- ity methods in deep neural networks. In Advances in Neural Information Processing Systems, pp- 9734-9745, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint UrXiv:1412.69000 6980, 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint
arXiv:1312.61 14, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv: 1607.02533, 2016.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in neural information processing systems, pp. 4765-4774, 2017.
Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojeiech Samek, and Klaus-Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern
Recognition, 1-222, 2017.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1 1135-1144, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision. 115(3):211-252, 2015.
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Klaus-Rober Müller. Evaluating the visualization what a deep neural network has learned. IEEE transactions neural networks and learning systems, 28(1 1):2660-2673, 2016.

---

Under review as a conference paper at ICLR 2021
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. In International Conference on Learning Representations, 2020. URL
net. S
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-base based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Ma- chine Learning- Volume 70, pp. 3145-3153 JMLR. org, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiiv:149.1155 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: arXiv:/706.00 1706.03825, 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy. Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 2014.
Suraj Srinivas and François Fleuret. Full-gradient representation for neural network visualization. In Advances Neural Information Processing Systems, pp. 4126 4135, 2019.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference Machine Coorning-Volume 70, pp. 3319- 3328. JMLR. org, 2017.
Yujia Xie, Hanjun Dai, Minshuo Chen, Bo Dai, Tuo Zhao, Hongyuan Zha, Wei Wei, and Tomas Pfister. Differentiable top-k operator with optimal transport. arXiv preprint arXiv:2002.06504 2020.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818-833, Springer, 2014.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint 170.00002. 2017.
APPENDIX
A SCHEMATIC DESCRIPTION
Forigin
Classifier
Differentiable Top-k Opentor
(M)
MCX))
Sample
Encoder (e)
pertuuro
Fixed
Input data
Saliency
Mask
Regularization Less
Reconstmuction
Figure 8: Schematic description.
10

---

Under review as a conference paper at ICLR 2021
An imput image is fed into the encoder network that gives the mean and the variance of Guassian distribution (which is the approximate posterior). A saliency map is sampled from it, followed by passing differentiable top-k operator provide binary mask. With this mask, the imput image is perturbed. The perturbed image is passes classifier M that gives categorical probability Uperturb- The loss function is composed of two terms: the reconstruction term between Uperturt and the categorical probability obtained from the original image Yorigin. and the regularization term between the approximate posterior g()
(s; p (()), diag(v()))) and the prior distribution (s; O, E). As the classifier M is the model that we aim to interpret, it is fixed so as the parameter be updated while training the encoder network.
B SOFT-TV GAUSSIAN PRIOR
Σ
Kh
Kw
(a)
(b)
Figure 9: Soft-TV Gaussian prior.
We consider the prior knowledge that adjacent pixels in saliency map have positive correlation (Figure aaa). After modeling the prior as Gaussian distribution (s; 0, E), covariance matrix Σ is designed to infuse this prior knowledge into the prior distribution (Figure b)). The covari- ance matrix then decomposed by Kronecker product to better calculate the KL divergence of the regularization loss.
C IMPLEMENTATION DETAILS
Encoder architecture We use convolution layers for encoder network. To make the spatial size of the encoder's 's input and output to be same, do not pooling layer. Every convolution layer is comprised of a convolution with kernel size x 3, stride 1, and padding followed by batch normalization and rectified linear unit. The number of output channels for each convolution is as follows: (64, 64, 64, 64, 32, 32, 32, 32, 32, 16, 16, 16, 16, 16, 16, 2]. The encoder network provides hhx and 77 R for each channel of the output where μa is the mean of the Gaussian distribution and log y with / E hxw the variance of the Gaussian distribution.
hyper- parametasssssrss We use Adam (Kingma & Ba 2014 optimizer with learning rate to be 0.0001, weight decay to be 0.0005, and betas be (0.9, 0.99). We batch size of 128 while training the encoder network. We run 10 epochs for the Imagenet dataset, and save the network that has the lowest loss.
D PROOF OF REGULARIZATION LOSS EQUATION
Let us first define the notation. & is the Kronecker product, and is the element- wise multiplication. sum(-) i) is the summation of elements. For vector b h denote rsh(b) E Ruxb reshaping

---

Under review as a conference paper at ICLR 2021
the vector b to the matrix where the (i. entry is bli + wÿ] and diag(b) as the diagonal matrix where the diagonal is b. Also, for a square matrix B. diag () is the vector where the i-th entry is Bii, :]. For matrix B, vec( denotes the vectorization by stacking the columns of the matrix B to single column vector.
Recall that the approximate posterior is (() = (H, Σo) where Ep diag(v) with pt, Rhw and the prior distribution is p(s/2) = (O, Σ₁) where Σ = R4 @ Kew with Ks. E Rixh and
Kow Rwxx KL divergence between Gaussian distribution is:
DKL[g(s/2) p(s|=)] = DRL[N(H, En) |O En)
E
2 tr ( Ea) + nz Σ hw + log
(8)
We compute each term in RHS equation to make it computationally efficient.
(2) tr (AEA kw)) diag (v))
tr ((r)
2 Niw"" diag (v)
diag (K² @ K @ diag (V)
diag (k=))) rsh (V) diag (1
(9)
μ Ej u² ( & kww
n² (re @ ) vee (rsh (H))
n² vec (Kn) rsh (pL) x= -17)
sum rsh (fx) @ ( rsh (1x) (15 )7)
(10)
Σ
log Eo
log Krn, @ Kwwl
log |diag (N)
har
log [ec
New
log i=1
Vi
har
w log KR + h log [nw - log
s=1
sum v;) + const.
(11)
Therefore, the equation 8 is derived as:
DKL D || p(s[z)] = diag (m=_1 rsh (v) diag (M²
sum rsh (μ)) C ( rsh (4) 1M)
(12)
sum (log p:) + const.
12

---

Under review as a conference paper at ICLR 2021
E QUALITATIVE RESULTS
expootin
wolthaud
Image
Inperi Grndint
PDA
Real-Time
HP
FIDO
(Ours)
13


Reviews:
[{"review_id": "x94qEbAnE_D", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper presents a new saliency map interpretability method for the task of image classification. It considers the saliency map as a random variable and computes the posterior distribution over it. The likelihood measures the predictions of the classifier for an image and its perturbed counterpart. The prior encodes positive correlation among adjacent pixels. Variational approximation is used to approximate the posterior.  \n\n————————————————————————————————————————————————————————————————————————————————\n\n\nStrengths:\n\nS1) The paper is very well written and easy to understand.\n\nS2) The paper does a good job of combining the ideas of perturbation based saliency map methods and total variation regularization with variational approximation in proposing their interpretability approach.\n\nS3) The proposed approach is able to generate real-time saliency maps, and, therefore, is computationally cheap.\n\nS4) The paper shows that their proposed interpretability method passes the sanity check from [Adebayo et al. (2018)].\n\n————————————————————————————————————————————————————————————————————————————————\n\n\nWeaknesses:\n\nW1) My biggest concern with the paper is that the proposed approach entails training another (non-interpretable) network to explain a given pretrained classifier. One of the goals of interpretability is to ensure that the pretrained classifier doesn’t encode any undesired biases learnt from the data. How can we ensure that this newly trained network doesn’t encode any biases of its own, especially when this network is also trained on the same data?\n\n\nW2) Another key concern is the evaluation of the proposed approach. Details below:\n\nW2a) I am not sure if I understand the importance of qualitative examples correctly. The paper claims that VarSal is able to produce high quality object borderlines as compared to other methods. Is that a desirable property we should expect from interpretability methods? Ideally, we want the saliency maps to highlight regions which the classifier considered the most important. It is not clear to me how we can evaluate that from qualitative examples. \n\nW2b) Why is the case where pixels with the largest k% saliency values are erased more prone to creating unnecessary artifacts than the case where pixels with the smallest k% saliency values are erased? Especially, when compared to other methods, is there a reason to believe that these artifacts might be more common in some methods than others?\n\nW2c) Why is Real-Time [Dabkowski & Gal (2017)] omitted from the pixel perturbation benchmark (Figure 5a)? Given that it is computationally equally cheaper to the proposed approach, comparison with Real-Time seems to be the most important one. \n\nW2d) The proposed approach has many similarities to [Chen et al. (2018)]. How does it compare to the proposed approach / what are the advantages of the proposed approach over [Chen et al. (2018)]?\n\nW2e) The paper in its current form doesn’t provide an explicit take-away message. Given a number of saliency map based interpretability methods available at this point, why should a researcher choose the proposed approach over other works? I think it would be a good idea for the authors to discuss the advantages as well as the disadvantages explicitly, highlighting applications/tasks/conditions where their approach might be better suited than others as well as cases where it might be better to avoid the proposed approach.\n\n\nW3) A very clear difference between the proposed approach and previous works is that the proposed approach explains the predicted probability distribution while the previous works focus on the ground-truth target. The difference is clearly an advantage of the proposed approach. However, it is important to disassociate the contribution of this difference from the rest of the approach. Is it possible for the authors to create an ablation of their approach keeping this aspect same as the previous works and observe how that version compares (to previous works as well as the proposed approach) qualitatively and quantitatively? \n\nW4) Saliency maps highlight the parts of the input image that the classifier finds most important while making a prediction. In that sense, isn’t it more reasonable that for a given classifier and an image, the saliency map is deterministic? What does it mean intuitively for this saliency map to have randomness? It would be great if the authors can discuss why it makes more sense to treat it as a random variable instead of a deterministic quantity.\n\nW5) Can the authors please describe the construction of the graphical model in Figure 1? Shouldn’t there be a solid line from ‘M’ to ’s’?\n\nW6) The paper says that [Fong & Vedaldi (2017)], [Fong et al. (2019)] and [Dabkowski & Gal (2017)] -- “all three methods have a limitation for producing importance ranking among features of a given image since their objective is to produce a binary mask”. What is the limitation here? And, how is the proposed method overcoming that limitation?\n\nW7) The take-away message from the uncertainity over explanation subsection in unclear. What is the significance of the uncertainity saliency maps? And, what should one learn by generating these maps for the corrupted datasets? \n\nW8) How is the hyperparameter alpha in the soft-TV Gaussian prior selected? \n\n\n——————————————————————————————————————————————————————————————\n——————————————————————————————————————————————————————————————\n\nUpdate after rebuttal: I thank the authors for their responses to all my questions. They satisfactorily answer some of my concerns. However, I still have two major concerns: 1) the faithfulness of the proposed approach, and 2) I see the potential contribution of uncertainty saliency maps but without an application/evaluation, their significance is unclear. I disagree that uncertainty/confidence generated from the same mechanism that generated the explanation is more trustworthy than the explanation itself. Hence, I cannot recommend the paper for acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "gb7Los6KRuv", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposed a method for generating saliency maps for image classifiers that are stochastic (instead of deterministic). The probabilistic model assumes a saliency map random variable that generates the data with a classifier. The inference is done by variational methods. The paper presents several qualitative examples and a comparison to previous work using the pixel perturbation benchmark.\n\nStrengths:\n- Saliency map generation is an important problem of interest to the ICLR community.\n- Stochastic saliency map is interesting and novel to my knowledge.\n- The paper is generally easy to understand.\n\nConcerns:\n- Current empirical experiments compare to several saliency generation methods in the pixel perturbation benchmark. The authors should also compare to FullGrad and use the remove-and-retrain framework as in (Srinivas & Fleuret, 2019). This will help us fully understand the performance of this method.\n- A positive point is the improved inference time. This is shown in Figure 5b but not discussed in the text, only briefly mentioned in the conclusion. I also think that improved inference time is at the cost of longer training time (the authors can correct me on this).\n- A central claim is that stochastic saliency maps improve explanation. The authors have provided some intuition and examples, but I think more evidence is needed.\n- The presentation of the paper can be improved: the abstract and the introduction should highlight the contributions and summarize the main results, which the experiment section should fully discuss; the sanity check and implementation details can be cut or moved to supplementary.", "extra_scores": {}, "extra_text": {}}, {"review_id": "D8EOXOqDyr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nThis paper proposes a new interpretability method for image classification networks. It considers a saliency map as a random variable and aims to calculate the posterior distribution over the saliency map.  The likelihood function and the prior distribution are then designed to make the posterior distribution over the saliency map explain the behavior of the classifier’s prediction. Quantitative evaluation on the perturbation benchmark as well as qualitative result show the effectiveness of the proposed method over baselines. \n\nPro:\n+ The proposed method is very efficient at the inference stage after the training with extra a encoder.\n+ From results on the perturbation benchmark, the proposed method achieves better or comparable performance compared with baseline methods.\n\nConcerns:\n- The evaluation is a kind of weak. Since it is hard to evaluate the interpretability method on one single benchmark, why not following existing explanation works e.g. Schulz et al. (2020) to conduct evaluations on multiple tasks with different metrics and network backbone. These tasks could be object localization, image degradation (Schulz et al. 2020), etc.\n- For the current evaluation on perturbation benchmark, the proposed method is comparable and sometime worse than some recent perturbation-based methods such as SmoothGrad-squared, FIDO. Although the proposed method is quite efficient at the inference stage, it requires extra training steps and extra encoder with parameters. It is a kind of trade-off compared with other  perturbation-based methods. \n- The selected visualization examples have quite simple background. Is it possible to show example with complicated scenes such as samples from PASCAL VOC, MSCOCO, Visual Genome datasets.\n- It is not clear whether the proposed method can be generalized to other kinds of data rather than images, such as text data, video data, vision-language tasks etc.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall the reviewers had various positive things to say about the paper, including that it was well written and easy to understand, topical, that the method was sensible, novel and interesting and that the computational efficiency (i.e. real time) was appealing.  However, all the reviewers thought it wasn't quite ready for acceptance, mainly citing concerns with the empirical evaluation.  It seems they had trouble interpreting the empirical results and placing the work with respect to other relevant methods.  

It seems in the author response, the authors did much to add to the experiments, but ultimately the reviewers were not comfortable with acceptance.  Taking the reviewers' feedback into account and adding the desired empirical evaluation would make this a much stronger submission to a subsequent conference.
Decision: Reject