Robust saliency maps with distribution-preserving decoys
Authors: Yang Young Lu
Wenbo Guo
Xinyu Xing
William Stafford Noble
Venue:  (2020)
Keywords: explainable machine learning
explainable AI
deep learning interpretability
saliency maps
perturbation
convolutional neural network

Abstract: Saliency methods help to make deep neural network predictions more interpretable by identifying particular features, such as pixels in an image, that contribute most strongly to the network's prediction. Unfortunately, recent evidence suggests that many saliency methods perform poorly when gradients are saturated or in the presence of strong inter-feature dependence or noise injected by an adversarial attack. In this work, we propose a data-driven technique that uses the distribution-preserving decoys to infer robust saliency scores in conjunction with a pre-trained convolutional neural network classifier and any off-the-shelf saliency method.  We formulate the generation of decoys as an optimization problem, potentially applicable to any convolutional network architecture. We also propose a novel decoy-enhanced saliency score, which provably compensates for gradient saturation and considers joint activation patterns of pixels in a single-layer convolutional neural network. Empirical results on the ImageNet data set using three different deep neural network architectures---VGGNet, AlexNet and ResNet---show both qualitatively and quantitatively that decoy-enhanced saliency scores outperform raw scores produced by three existing saliency methods.

Under review as a conference paper at ICLR 2020
ROBUST SALIENCY MAPS PRESERVING DECOYS
WITH DISTRIBUTION-
Anonymous authors Paper under double-bling review
ABSTRACT
Saliency methods help to make deep neural network predictions more interpretable by identifying particular features, such as pixels in an image, that contribute most strongly to the network's prediction. Unfortunately, recent evidence suggests that many saliency methods perform poorly when gradients are saturated or in the presence of strong inter-feature dependence or noise injected by an adversarial attack. In this work, we propose a data-driven technique that uses the distribution- preserving decoys to infer robust saliency scores in conjunction with pre-trained convolutional neural network classifier and any off-the-shelf saliency method. We formulate the generation of decoys as an optimization problem, potentially applicable to any convolutional network architecture. We also propose a novel decoy-enhanced saliency score, which provably compensates for gradient saturation and considers joint activation patterns of pixels in a single-layer convolutional neural network. Empirical results on the ImageNet data set using three different deep neural network architecture- qualitatively and quantitatively that decoy- enchnnen saliency scores outperform raw scores produced by three existing saliency methods.
AlexNet and ResNet- both
INTRODUCTION
Deep neural networks (DNNs) deliver remarkable performance in an increasingly wide range of application domains, but they often do so an inscrutable fashion, delivering predictions without accompanying explanations. In practical setting such as automated analysis of pathology images, if a patient sample classified as malignant, then the physician will want to know which parts of the image contribute to this diagnosis. Thus, in general, a DNN that delivers explanations alongside its predictions will enhance the credibility and utility of its predictions for end users (Lipton) 2016).
Two primary classes of methods have been developed for interpreting CNN models. The first class of methods, often referred to as saliency maps, use gradients of classification outcome as a per-pixel importance score to characterize the influence of individual pixels to the prediction (Simonyan
ct al. 2013: Selvaraju et al. 2016 Binder et al.] 2016 (Shrikumar et al., 2016; (Smilkov et al.
2017 Sundararajan al., 2017: Levine et al. 2019). Despite providing apparently meaningful interpretations in practice, these methods exhibit bit fundamental limitations. First, saliency maps evaluate pixel-wise importance in an isolated fashion by design, implicitly assuming that other pixels are fixed (Singla et al.) 2019). Worse still, the presence of gradient saturation often breaks the assumption that important features in general correspond to large gradients (Sundararajan et al., 2016 Shrikumar et al. 2016 Smilkov et al. 2017). Specifically, an important pixel may contribute a strong joint effect on the prediction when considered in conjunction with neighboring pixels. However, that important feature may have tiny gradient locally. in the sense that its marginal contribution is diminishing due to the flattened output of the layer. Finally, Ghorbani et al. (2017) and Kindermans) ct al. (2017) systematically revealed the fragility of widely-used saliency methods by showing that even an imperceivable perturbation or simple shift transformation of the input data can lead to large change in the resulting saliency scores. In light of this observation. a key challenge for any saliency method is ensuring that the saliency scores are robust to gradient saturation and perturbations.
A second class of interpretability methods, often based on counterfactual, perturb (i.e., nullifying, blurring, adding noise, or inpainting) small regions of the image to modify the classifier prediction
(Fong& & Vedaldi 2017 Dabkowski & Gall 2017 Chang et al. 2017 Fan al. 2017 Chang et al.

---

Under review as a conference paper at ICLR 2020
2019
Yousefzadeh & O' O'Leary 2019 Goyal et al. 2019 Despite identifying meaningful regions in practice, these methods exhibit several limitations. First, counterfactual-batt -ased methods implicitly assume that regions containing the object the ones most contributed to the prediction (Fan et al.) methods are also vulnerable to the adversarial attacks, which force these explanation methods to output non- related backgrounds rather than the meaningful objects as important subregions, In addition, the counterfactual images may be potentially far away from the distribution from which the training samples were drawn [Burns et al. 2019), causing the classifier behavior ill-defined. Though some efforts have been made to train the saliency extractor and the classifier simultaneously (Fan et al. 2017 Zoina et al. 2019). the robustness of the resulting explanations are not guaranteed in most cases (Hendrycks & Dietterich 2019). In light of this, for any perturbation- -based method, a key challenge is ensuring that the perturbations are effective yet preserving the training distribution.
2017). However, Moosavi-Deezooli et al.] (017)) revealed that counterfactual
In this work, we propose a data-driven technique to infer robust saliency maps with distribution- preserving perturbations. Given image of interest, the core idea of this method is to generate a population of perturbed images, referred to as decoy images, that resemble the neural network's s intermediate representation of the original image but are conditionally independent of its label. The resulting decoy images capture the variation of the image data originating from either sensor noise or adversarial attacks. The major differences between the proposed and counterfactual-bee based methods are threefold. First, unlike counterfactual images that seek the minimum set of features to exclude so as to minimize the prediction score or to include so as to maximize the prediction score (Fong & Vedaldi 2017 saliency map methods like the one we propose in this paper aim to characterize the influence of each feature on the prediction score. Second, unlike counterfactual images that are optimized toward the aforementioned objective with respect to the decision boundary, decoys are designed to be independent of labels, without changing the decision boundary at all. Finally, unlike counterfactual images which could potentially be out-of-distribution by adding noise (Smilkov) jet al., 2017), rescaling (Sundararajan et al.) 2017), blurring (Fong Vedaldi (017). or inpainting the image (Chang al. (017). decoys are plausibly constructed in the sense that their intermediate representations are non discriminative from the original input data by design.
In addition, we propose novel decoy-enhanced saliency score, which considers ensemble of saliency maps across multiple decoy images. We also derive a closed-form formula for the decoy- enhanced saliency score generated from a single-layer convolutional neural network. This analysis shows that the proposed saliency score for each pixel can be decomposed into two components: the range of values for given pixel among the decoys and the number of neurons jointly activated by that pixel and its neighbors. The former compensates for gradient saturation (Sundararajan et al.) 2016) in the sense that an important pixel can vary without strongly affecting the prediction, due to the joint effect of nearby pixels. The latter compensates for the way in which saliency maps treat each pixel independently (Singla et al. 2019) in the sense that the importance of a pixel is jointly determined by the importance of pixels in its vicinity. This closed-form result may provide insights to help design better defensive strategies against adversarial attacks.
We apply the proposed method to the ImageNet dataset (Russakovsky et al. 2015 in conjunction with three standard saliency methods. We demonstrate empirically that decoy- saliency scores perform better than the original saliency scores, both qualitatively and quantitatively, even in the presence of various adversarial perturbations to the image.
PROBLEM SETTING
Consider a multi-label classification task in which pre-trained neural network model implements a function F: R2d + C that maps from the given input x E Rd to predicted classes. The score for each class E {I..... C} is Fx)), and the predicted class is the one with maximum score, i.e., arg max ,c (x).
A common instance of this setting is image classification, in which case the input corresponds to the pixels of an image. A saliency method aims to find the subset of features in the input x that most strongly lead the neural network make its prediction. Thus, each pixel is assigned saliency score, in which the pixels with higher scores represent higher
encoded in an explanation map E : Rdd + I "importance" to the final prediction.
2

---

Under review as a conference paper at ICLR 2020
Patch masks
Pretrained network
Label-dependent gradients
Decoy
Saliency maps x
Input image
Output saliency
Decoy Generator
Saliency
Methods
Figure 1: Workflow for creating decoy eechn-enhanced saliency maps.
variety of saliency methods have been proposed the literature. Some, such edge or other generic feature detectors (Adebayo et al. (0018). are independent of the predictive model. In this work, we focus on three methods that do depend on the predictor. The vanilla gradient method (Simonyan et al. 2013) simply calculates the gradient of the class score with respect to the input x, which is defined as (x) = VxF*(x), The SmoothGrad method (Smilkov et al. 2017) seeks reduce noise in the saliency map by averaging over explanations of noisy copies of an input, defined as (x) + >m (x with noise vectors 9i [0, o²). The integrated gradients method (Sundararajan et al. 2017) aims to tackle the problem of gradient saturation. The method starts from a baseline input x" and sums over the gradient with respect to scaled versions of the input ranging from the baseline to the observed input, defined as (x) = (x x²) x So (xF"(x" + a(x - x )dx.
In this setting, we are given three inputs: a pre-trained neural network model F with L layers, an image of interest x, and a saliency method E such that E(x: F) is a saliency map of the same dimensions As illustrated in Figure [ the robust saliency map can be obtained in two steps: 1) constructing decoy images, and 2) computing decoy-enhance saliency scores.
DEFINITION OF DECOYS
Say that F. Rd Rd the function instantiated by the given network, which maps from input image x E IRdd to its interme- diate representation Fx(x) E gd at layer E {1, 2.... L}. At
specified layer É, the random vector x E RRd is said to be decoy of x E Rdd the following swappable condition satisfied:
Original image
Decoy image
(x = Fe Fx(Xxwup(x.P) for image patches P C f1.-.. d}
Figure 2: An illustrative exam- ple of the swap operator swap- ping image patches between riginal and decoy images.
(1)
The swap(X & P) operation swaps image patches between and (Figure 2 Hence, if P = (10), then Tswap(X.P) is a new
image which is almost equivalent to x except that the 10th patch is exchanged from x. A "valid" patch is a local region receptive to the convolutional filter of a given CNN, for example, the 3 x area VGGNet (Simonyan Zisserman, 2014). The swappable condition ensures that the original image x and its decoy x are indistinguishable in terms of the intermediate representation at layer Note in particular
the construction decoys relies solely on the first layers of the neural network Fi, P22- ... F& and is independent of the succeeding layers F , FLz In other words, x is conditionally independent of the classification task F(x) given the input x; i.e., & IL F(x) x. See Supplementary Section A6 for the explanation why decoys exist.
3.1 GENERATE DECOYS BY OPTIMIZATION
Given an image of interest x E Rd and patch mask M E 13 we want to construct a decoy image MM E Rd, with respect specified neural network layer f, that as different as possible from x. For this purpose, we optimize the following objective function:
maximizeg ||((x x)+ s) +111-
s.t. [[((()) Fx(x)l Se e, (x - xx) M) = 0, and x E [0, 1jd.
3

---

Under review as a conference paper at ICLR 2020
where (+)+ max(-.0) and
or -1 depends whether intend to investigate the upward or downward limit of x deviating from x, respectively. The operators ||-Il, and I1 correspond to the L1 and Lo norms, respectively. The constraint (x -x (1 (1- = 0 ensures that and x differ only in the area marked by the patch mask M, where denotes entry-wise multiplication. The final constraint ensures that the values in x fall into an appropriate range for image pixels, [0, 1] in the normalized image case. In principle, the formulation in Equation i applicable any convolutional neural network architecture ranging from simple convolutional networks to more sophisticated ones (e.g.. AlexNet (Krizhevsky et al. 2012), Inception network (Szegedy et al. 2016) and ResNet (He et al., 2016). etc.).
As illustrated in Figure ! we can construct population of n independently-sumpee patch masks {M',
.M"), }, so that each possible image patch is covered by at least one patch mask. Thus we can construct a population of n decoy images {x', ,,,... ,.) with respect to different patch masks. Note that the input image is also trivially valid decoy, by definition; iee. it satisfies Equation and the conditional independence requirement (i.e., X F(x))x).
3.2 DECOY- DECOY-ENHANCED SALIENCY SCORES
Given the generated population of decoy images {x] be applied to each decoy image, following the fundamental rule that the saliency method should be agnostic to information revealing which image is a decoy and which is not. In this way, we obtain
, the given saliency method E can
corresponding population of decoy saliency maps { [B(x); ; F), E(x;; F),.... E(x"; F)).
Now the pixel- wise variation for each feature x, can be characterized by a population of feature values X, = [x], ... .x] Accordingly, instead of merely quantifying the saliency of each feature x, via a single value E(x; F) we can now characterize the uncertainty of the feature's 's saliency via the corresponding population of saliency scores Ej = Ex F)5, F), E(X";
We define the decoy-enhanced saliency score Zj for each feature x, as
Z, max E,) min(E,), That is, Zy is determined by the empirical variation of the decoy's saliency scores.
(3)
3.3 EXAMPLE: SINGLE-LAYER CNN
Consider a single-layer convolutional neural network with decoy swappable patch size x 1 and convolutional size x3. The input of this CNN is x E Rdd unrolled from a vd x v grayscale image matrix. Similarly. we have a x 3 convolutional filter unrolled into E R², where is indexed as gx
for A E KC, here K. = v - - vd, - v + 1, 1,, v vd, vd + 1 corresponding the index shift in matrix form from the top-left bottom-right pixel, respectively. We denote (g*x) E IRd as the output of the convolution operation on the imput where (g*x); = >kEK g For simplicity, we further assume that there are no pathological cases such as (g * x); = 0. Such a neural network can be represented
m = relu(g * x) o W m -
softmax(o),
(4)
where relu(-) is the entry-wise ReLU operator GGorot et al. 2011). W E RdxC represents the combined weights of the neural network, and H represents the biases. The terms Eº and IRC are the logits and the predicted class probabilities, respectively. The entry-wise softmax operator for target class is defined as Pe ΣS Po for E (1.2..... CC}.
In this case, we have the following result:
Proposition Z, bounded by:
2, 18 x),l
Σ kEK
ājtb + C1.
(5)
where x is the decoy which maximizes E(x; F), E(x: F) à; = {(g x)) 0}. and C > and C2 0 are bounded constants. See Supplementary Section A7 for the full proof.
4

---

Under review as a conference paper at ICLR 2020
Dindiiattssii indicates that the saliency statistic is determined by two factors: the range of values associated with pixel j among the decoys (( - x)yl) and the number of neurons jointly activated by pixel j and its neighbors (KEc aj-k). The former explains the gradient saturation problem (Sundararajan et al., 2016) in the sense that important features may have more room to fluctuate without influencing the joint effect the prediction. 'he latter compensates for way in which saliency maps treat each pixel independently (Singla et al. (0199) in the sense that the importance of pixel x, jointly determined by the surrounding pixels (i.e., the 3 3 localized region our setting), potentially capturing meaningful patterns such as edges, texture, etc. In this way, the proposed saliency statistic not only compensates for gradient saturation but takes joint activation patterns into consideration as well. Indeed, the bound provided by Proposition can be considered a motivation for defining the decoy-enhanced saliency score as we have in Equation
Proposition
3.4 IMPLEMENTATION DETAILS
We solve Equation 2 by augmenting constraint penalty in the objective function:
minimizeą
- ||((( x) s)* 11 c [E(( - (x)
(6)
s.t. (x (1 M) 0, and 10.
When the constant e > 0 is properly chosen, Equations 2 and 6 are equivalent in the sense that the optimal solution to Equation equuls the optimal solution to Equation 6 Our strategy is to set e to small value initially and run the optimization. If it fails, then we double and repeat until success.
The sensitivity of choosing e reported in Supplementary Section A9
To eliminate the constraint on pixel values (X E [0, 1](), we use the change-of-variable trick (Car lini & Wagner 2017). Note that other transformations are possible yet unexplored in this paper. Specifically, instead of optimizing over X, we introduce & satisfying X; = & (tanh( ) + 1), for all
i E (1,2, ,d}. Because tanh(x;) E [-1, implies x; E [0, 1]. any solution to & is naturally valid.
To eliminate of the patch mask constraint ((x - x) (1 (1-111100 MM = 0), we use projected gradient descent during the optimization stage. Specifically, after performing each step of standard gradient descent, we enforce x x o M + x o (I-M).
Putting these ideas together, we minimize the following objective function:
minimizez - || ((arctanh (2x - 1) - x) s) Il, + e. 1 (arctanh))) - 1)) F Fx)
(7)
Because the L. norm is not fully differentiable, we adopt the trick introduced by Carlini & Wagner (017)). Specifically, in each iteration we solve:
minimize@ |max((arctann| (2x- 1) - x) 0) +e (arctanh - 1)) 1 [x +)+
where > 0 is initialized with large value (1 in our experiment). After each iteration, if the second term in Equation is zero, indicating that r too large, then we reduce 7 by factor of 0.95 and repeat; otherwise, we terminate the optimization. The run time of optimization is reported in Supplementary Section A10
4 EXPERIMENTS
To evaluate the effectiveness of our proposed method, we perform extensive experiments on deep learning models that target image classification tasks. The performance of our method is assessed both qualitatively and quantitatively. The results show that our proposed method identifies intuitively more coherent saliency maps than existing ofate-of-the-ar saliency methods alone. The method also achieves quantitatively better alignment to truly important features and demonstrates stronger robustness to adversarial manipulation.
Our experiments primarily use the VGG16 model (Simonyan & Zisserman 2014 pretrained on the
ImageNet dataset (Russakovsky et al.) 2015). We also demonstrate the applicabilii of our method to other well-studied CNN architectures such as AlexNet (Krizhevsky et al., 2012) and ResNet (He
et al. 2016). The settings for all experiments are reported in Supplementary Section A13
5

---

Under review as a conference paper at ICLR 2020
Grad+ Decoys
Grad+
Grad+ Inpainting
IntGrad+ Decoys
IntGrad+
IntGrad+ Inpainting
SGrad+ Decuys
SGrad+
SGrad+
Grad
Blarring
IntGrad
Blurring
SGrad
Blurring
Inpainting
Tettier
Se:
SF: S0:60.26 SF: -0.11 SP: SF: -0.36 S..666--0.265.
SF. -0.09 SE: -0.07
SF:40.48 Se: -0.43 St: -0.11
Figure 3: Visualization of saliency maps comparing three state-of-the-art saliency methods with and without decoy images generated from three techniques (i.e., our decoy generation method, blurring, and inpainting). More examples can be found in Supplementary Section A14
4.1 BENCHMARKS AND VISUALIZATION METHODOLOGIES
To benchmark the performance of proposed method, considered the following baseline saliency methods: the vanilla gradient method (Simonyan et al. 2013), SmoothGrad (Smilkov et al. 2017) and integrated gradients (Sundararajan al. 2017).
To evaluate our decoy generation method, we used the following two methods to replace the proposed decoy generation method: blurring and inpainting. In particular, given image with few masked patches (i.e., X function in Equation D we fill the missing parts by replacing the original missing patches with blurring or using the tfae-offthe--a inpainting method Yu et al. 2018) Using the decoy images generated by these two substitute methods, we then computed the encanned saliency scores for the three baseline saliency methods. We compared the results of blurring and inpainting with our method, both qualitatively and quantitatively.
M)), instead of generating a decoy image (i.e. x) by solving the optimization
A heatmap is typical way of displaying saliency map. Our strategy for projecting a saliency map onto a heatmap is as follows. Saliency methods produce signed values for input features (pixels in experiment), where the sign of the value indicates the direction of influence for the corresponding feature to the predicted class. We first took the absolute value of the saliency maps, as suggested by Smilkov al. (201)). Then we used the maximum value across all color channels, suggested by Simonyan et al. (2013), to arrive at a single saliency value for each pixel. To avoid outlier pixels with extremely high saliency values leading to almost entirely black heatmap, winsorized those outlier saliency values to a relatively high value (e.g., 95th percentile), as suggested by Smilkov et al. (2017). to achieve better visibility. Finally, the values are linearly rescaled the range [0, 1].
4.2 COMPARISON CRITERIA
To comprehensively evaluate our proposed method against the aforementioned baseline saliency methods, we focus the following three goals.
First, we aim to achieve visual coherence of the identified saliency map. Intuitively, we would prefer saliency method that produces saliency map that aligns cleanly with the object of interest.
Second, we use a saliency fidelity metric (Dabkowski & Gal 2017) that quantifies the influ- ence of important features identified by saliency method, defined as (x(FF(x; F), c) -(log(A"<(E) F') x)) log( ('x(xx;; Fmmmn o x))), where e indicates the predicted class of input x and the saliency map E(x; F') is normalized into [0, 1] range, described in Section 4.1 The function (EE(x: F) o x) performs entry-wiss multiplication between E(x; F) and x, encoding the overlaps between the object of interest and the concentration of information by the saliency map. The rationale here is that, by viewing the saliency score of the feature as its contribution to the predicted class, a good saliency method will be able to weight important features more highly than less important ones, giving rise to better predicted class scores. Accordingly, a lower value for the saliency fidelity metric implics a faithful saliency method. Note that we subtract the influence of mean saliency Exx; mean (i.e.. the mean of Exx; F) on each channel) to climinate the influence of bias in Exx; F). As a result, we can exclude some trivial cases (e.g., Ex;; F) 1).
Third, aim for robustness adversarial manipulations (Ghorbani al. 2017 Adversarial attacks aim to apply imperceptible perturbations to an input image that do not affect the predicted class but produce a very different saliency map relative to the map from the unperturbed image. Ghorbani et al.
We directly used the well-trained inpainting model provide by Yu 2018

---

Under review as a conference paper at ICLR 2020
Grad Attimage
Grad Decoys
InrGrad Attimage
IntGrad +
SGrad Attimage
SGrad + Deenys
IntGrad
Decays
SGrad
Top-k
Ss: 13.66
Ss. 12.87
Ss 18,28
Ssi 16.41
Ss. 0,86
Ss. 0.80
Mass
Center
Ss. 33.43
7.38
Ss: 11.12
Ss: 10.69
Target
Ss 9.21
Ss: 6.63
Se: 10.56
Ss 9.09
Ss 8.86
Ss 7.75
Figure 4: Visualization saliency maps under adversarial attacks. More examples can be found in Supplementary Section A14
(217)) propose three different attacks: (1) the top-k attack seeks to decrease the important scores of the top-k important features; (2) the target attack aims to increase the feature importance of pre-specified region in the input image; (3) the mass-center attack aims to spatially change the center of mass of the original saliency map. In this paper, we specify the bottom-right 4 4 region of the original image for the target attack and select 5000 in the top-k attack.
To quantify the robustness of saliency method E adversarial attack, we use the sensitivity metric (((x,)))((sF))ll where xx-8||2
-Alvarez-Melis Jaakkola 2018), defined as Ss (F(:: F), x)
is the perturbed image of X. A small value means that similar inputs do not lead to substantially
different saliency maps.
4.3 RESULTS
We applied all three state-of-the-ar saliency methods and their decoys to a dozen randomly sampled images from ImageNet dataset. side-by-side comparison of the resulting saliency maps (Figure 3 suggests that decoys consistently help reduce noise and produce more visually coherent saliency maps. For example, Figure the gradient method without decoys highlights the region correspond- ing the dog's head, but in scattered format. In contrast, the decoy-enhancee only highlights the missing body but identifics the dog head with more details such as cars, check, and nose. The visual coherence is also quantitatively supported by the saliency fidelity metric Sy. which is consistently higher with decoys than without decoys. The sanity check (Adebayo et al.) 2018 of decoy-enhanced saliency method is reported in Supplementary Section A8
gradient method not
Figure 3 also shows the comparison of our decoy generation method with blurring and inpainting. Our decoy generation method consistently outperforms both competing methods, achieving higher fidelity scores on all of the three baseline saliency methods. The reason is as follows. As is discussed Proposition to obtain higher decoy score, set of decoy images should satisfy the following two requirements: first, they produce similar intermediate representation when provided as input to the target neural network; second, they exhibit large decoy variation. In this paper, we design our decoy generation objective function based on these two requirements. In contrast, neither blurring nor inpainting is designed to meet these requirements. Specifically, blurring may violate the intermediate representation constraint and inpainting may not be able to produce enough decoy variation.
is worth mentioning that combining image blurring or inpainting with our decoy-enhanced saliency scores can still improve upon the bascline saliency maps. The empirical validity of using image blurring is explained in Supplementary Section A1l From the practitioners' perspective, blurry images be used as more efficient alternatives to combine with the decoy-enhanced saliency score to achieve comparable performance. However, it is worth mentioning that decoy images are still necessary to justify the theoretical soundness of the recoo-enhancee saliency score.
Next, to test the robustness of the methods we subjected each to the three aforementioned adversarial attacks (Figure 4. Though not fully resistant to adversarial attacks, observe that enhancing
7

---

Under review as a conference paper at ICLR 2020
Bottom layers to top layers
M Maspool 5
Terrier
Massuen
Maspool
comn
camel
can'
Max Maxpool
comrd
N Maxpool
cans
Grad
IntGrad
SGrad
Figure 5: Generating decoys with respect to different DNN layers.
IntGrud Decoys
Grad
IntGrad
SGrad
Grad
IntGrad
SGrad
Decoys
Decoys
Decoys
Decnys
Decoys
Sr: 4.64
-298
SF. 2.99
SE:-2.18
SF: 6.85
SF: 3,86
S.:
0.24
5,80
0.26
S.7
Sr: 0.01
(a) Saliency maps generated AlexNet.
(h) Saliency maps generated on ResNet.
Figure 6: Visualization of saliency maps under different CNN architectures.
existing saliency methods with decoys consistently mitigates the impact of adversarial manipulations, leading to visually more coherent saliency maps as well as lower sensitivity scores. believe extra benefits in the robustness is brought by our decoy-enhanced saliency score. In normal situation (when the image doesn't It suffer from an adversarial attack), an important pixel is not important in an isolated fashion. Instead, the important pixel tends to contribute strong joint effect in conjunction with neighboring important pixels, to potentially capture meaningful patterns such as edges, texture, etc. In light of this observation, this particular important pixel will have
room to fluctuate without influencing the joint effect the prediction. In such case, some elements of Ej will be high and others will be low, contributing a large Zj. In the unusual situation when an isolated important pixels is indeed observed (i.e., pixel is very important for all decoy images and has room for fluctuation.), we tend to believe that the pixel has been adversarial attacked. In this case, all elements of E, will be high and the proposed decoy- eechneed saliency score Zj will be low, which is what we want.
4.4 SENSITIVITY OF DECOY SETTINGS
We also conduct experiments to understand the impact of decoy settings the performance of different saliency methods. Specifically, we focus on the choice of network layer é, because the intermediate representations can vary significantly different layers. Accordingly, we vary the value of ( for VGG16 and compare the differences of decoy-enhanced saliency scores from the three saliency methods, ranging from the first convolutional layer the last pooling layer. As shown in Figure 5) and Supplementary Section A14 the decoy-enhanced saliency scores generated from different layers for the same image are similar qualities. We also compute the Sr score for each saliency map. The mean and standard deviation, respectively, for the scores obtained by the gradient, integrated gradient and SmoothGrad methods are as follows: (-0.11. 0.02). (-0.18, 0.02), (-0.21, 0.01). These quantitative results also support the conclusion that our method is not sensitive layer. This likely because, as previous research has shown (Chan et al.) 2015 Saxe et al., 2011). ), the final classification results of DNN not highly related to the intermediate representations. As result, decoys for the same sample with the same label from different layers should yield similar results.
4.5 APPLICABILITY TO OTHER NEURAL ARCHITECTURES
In addition to the VGG16 model (Simonyan & Zisserman 2014). we generate saliency maps for other neural architectures: AlexNet (Krizhevsky al. (012)) and ResNet (He al. (016). We visualize their saliency maps in Figure and Supplementary Section A14 From Figure 6 we observe that all the saliency maps accurately highlight the contour of the dog and achieve higher fidelity score
8

---

Under review as a conference paper at ICLR 2020
than the baseline saliency methods. This indicates that apply our method to various neural architectures and expect consistent performance. It should noted that, in comparison with saliency maps derived from other neural architectures, the maps tied to AlexNet are relatively vague. We hypothesize that this is due to the relative simplicity the AlexNet architecture.
5 DISCUSSION AND CONCLUSION
In this work, we have used distribution-preserving decoys as a way to produce more robust saliency scores, First, we formulate decoy generation as an optimization problem, in principle applicable to any network architecture. We demonstrate the superior performance of our method relative to three standard saliency methods, both qualitatively and quantitatively, even in the presence of various adversarial perturbations to the image. Second, by deriving closed-fore formula for the decoy- enhanced saliency score, we show that our saliency scores compensate for the frequently violated assumption in saliency methods that important features in general correspond to large gradients.
Although we have shown that decoys can be used to produce more robust saliency scores, there remain some interesting directions for future work. For example, one could potentially reframe interpretability as hypothesis lesting and use decoys to deliver set of salient pixels, subject to false discovery rate control some pre- pre-specified level (Burns et al., 2019 Lu et al. 2018).
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Advances in Neural Information Processing Systems, pp. 9524-9535, 2018.
David Alvarez-Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems, 2018.
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In International Conference on Artificial Neural Networks, pp. 63-71, 2016.
Collin Burns, Jesse Thomason, and Wesley Tansey. Interpreting black box models via hypothesis testing. arXiv: 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 20/7 IEEE Symposiun on Security and Privacy (SP), pp. 39-57, 2017.
Tsung-Han Chan, Kui Jia, Shenghua Gao, Jiwen Lu, Zinan Zeng, and Yi Ma. PCANet: A simple deep learning baseline for image classification. IEEE Transactions on Image Processing, 24(12): 5017-5032, 2015.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Interpreting neural network classifications with variational dropout saliency maps. In Advances in Neural Information Processing Systems, volume I, pp. 6, 2017.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In International Conference on Learning Representations, 2019.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Advances in Neural Information Processing Systems, pp. 6967-6976, 2017.
Lijie Fan, Shengjia Zhao, and Stefano Ermon. Adversarial localization network. In Learning with limited labeled data: weak supervision and beyond, Advances in Neural Information Processing Systems Workshop, 2017.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3429-3437 2017.

---

Under review as a conference paper at ICLR 2020
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. arXiv:1710.10547, 2017.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth International Conference on Artificial Intelligence and Statistics, pp. 315-323, 2011.
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. International Conference on Machine Learning, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
Picter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (Un) reliability of saliency methods. arXiv: 11X.v:/71..0067.. 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Alexander Levine, Sahil Singla, and Soheil Feizi. Certifiably robust interpretation deep learning. arXiv preprint arXiv:1905.121000 2019.
Zachary C Lipton. The mythos of model interpretability, arXiv: 1606.6600.0333 034000 2016.
Yang Lu, Yingying Fan, Jinchi Lv, and William Stafford Noble. DeepPINK: reproducible feature selection in deep neural networks. In Advances in Neural Information Processing Systems, pp. 8676-8686, 2018
Seyed-Mohsen Moosavi-Dezfoolii Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1773- 2017.
Olga Russakovsky. Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 1 115(3):211-22 2015.
Andrew M Saxe, Pang Wei Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, and Andrew Y Ng. On random weights and unsupervised feature learning. In International Conference on Machine Learning, volume 2. pp. 6, 2011.
Ramprasaath R Selvaraju, Abhishek Das. Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-CAM: Why did you say that? arXiv:1611.07450 2016.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. arXiv: 1605.01713, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.155 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv:1312.6034 2013.
Sahil Singla, Eric Wallace, Shi Feng, and Soheil Feizi. Understanding impacts of high-order loss approximations and features in deep learning interpretation. UrXiv:7902.004400 2019.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fermanda Viégas, and Martin Wattenberg, Smoothgrad: removing noise by adding noise. arXiv: /006. 03825, 2017.
10

---

Under review as a conference paper at ICLR 2020
1
2
4
6
4
5
6
7
8
conv with 2x2 filter and stride 2
5
5
with 2x2 filter and stride 2
0
8
4
3
2
1
0
filter=
2
4
filter=
2
1
1
2
3
4
2
4
2
1
2
4
6
1
0
0
5
6
7
8
max pool with 2x2 filter and stride
6
8
max pool with 2x2 filter and stride 2
0
8
0
3
2
1
0
3
4
3
3
1
4
1
2
3
4
3
3
3
4
Original image
Decoy image
Figure A7: A toy example to illustrate why decoys exist.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Gradients of counterfactuals. arXiv:1611.02639 2016.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. arX- iv: 10:3003.1,35,, 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. 1312: 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings the IEEE conference Computer Vision and Pattern Recognition, pp. 1-26228,,, 2016.
Roozbeh Yousefzadeh and Dianne P O' O'Leary. Interpreting neural networks using flip points, arXiv preprint arXiv: 08789, 2019.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5505-5514, 2018.
Konrad Zoina, Krzysztof J Geras, and Kyunghyun Cho. Classifie--annoticc saliency map extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 10087-10088, 2019.
A6 EXISTENCE OF DECOYS
As illustrated in Figure A7 the original image and the decoy image shares exactly the same intermediate representation with respect to convolutional and max pooling.
A7 PROOF OF PROPOSITION
Consider a single-layer convolutional neural network with decoy swappable patch size 1 and convolutional size 3. The input of this CNN is x E Rdd unrolled from a vd v grayscale image matrix. Similarly, we have a x 3 convolutional filter unrolled into R9, where is indexed as ga
for k E IC, here K. = { v - - -d, vd 1,0, vd v +
corresponding
11

---

Under review as a conference paper at ICLR 2020
to the index shift in matrix form from the top-left bottom-right pixel, respectively. We denote (g*x) E IR" as the output of the convolution operation on the imput x, where (g+x), = Ekek &+ For simplicity, we further assume that there are no pathologic cases such (g x); 0. Such neural network can be represented as
= relu(g g x),
W m + b. = softmax(o).
where relu(-) is the entry-wise ReLU operator, W RdxC represents the combined weights of the neural network, and Rº represents the biases. The terms o E R and E gº are the logits and the predicted class probabilities, respectively. The entry-wise softmax operator for target class is 80c defined as Pe- Σø_1 2 for (1,2,-..,C,
The gradient of Pe with respet to can be written as follows, using the denominator layout notation of the derivative vector:
dm do Òp.
VxPc dx dm do
(10)
where
do W,
(11)
dm
and
(De - P if =c, 0p.
do.
(12)
8ps Jo
P.De otherwise.
Then, write dp. follows:
S0
Öp. P
(13)
do.
where P. corresponds to the e-th column P and f diag(p) - ppp..
We then define
0m 8x as E Raxd in which
Bij dmy
O(g + x), dmj 0x; 0(g x);
0(g x)j dx:
ag
(14)
0x:
where aj is denoted as aj [gg x)j 0}. Then, we have
{
Bij = a,gg-j if ifi-jEK,
(15)
B 0 otherwise.
Given Equations II 13 and
15
Equation 10 can be rewritten as
VxPe BWP .c:
(16)
Further, we define the Hessian
H. where H.j can be written as follows:
a(B.WPP..)
H Vx, (Vx Do)
Ox:
a jk P..
Ox:
a j4h P..)
dx,
(17)
KEK
OP.
Σ B BJJt&
Ox
KEA
Op.
Σ
aj+hgA W jth,
0x:
kER
12

---

Under review as a conference paper at ICLR 2020
and
PP =
(1 (1-28) 2x.e = PxVx, + Vx otherwise.
(18)
0x;
After computing the gradient and the Hessian, we now derive the decoy-enhanced saliency score Zj for X,, given population of saliency scores E, = {E(x'; F),, (( Exx; F)2)).
According to Equation 3 Zj can always be decomposed into Zj 2 + 2 where Zj
max Fiy) - Ex; F)5 and 7 = Exx F), min min( (Fy), respectively. Without loss generality, we only focus Zj", noting that smilar analysis is applicable to Zj as well.
Let
{x x2, x") denote the decoy which maximizes E(x; F), E(x; F);-
The second-order Taylor expansion of the predicted F*(x) for target class around is follows:
(x) RJ (x) + F (x) A H²
(19)
where A = X and H& is the Hessian of the neural network model on the decoy & whose entries
ppp= are (Hx)).. 88.08, By definition of the swappable condition in Equation 1 F*(x (x)..
Therefore, the following equation holds:
Px) ~ A=
(20)
Consider the case that the swappable patches are of size x 1 at position i; then Equation 20 can be decomposed into
F(x; F): = (V* (V (x)); (x (Hx)))
(21)
Then, according Equation
21 we have
Zj
(X xH)(Hz) - Vx
2
1
22 (8 x)j Σ
ājtngR W P. +
(22)
KEK
C2 (X - x),l
ājth + C1
KEK:
where C1 = Vx,Pe and C2 = maxtek gx W (i+)) P. respectively. Note that both C₁ and C2 are linear combinations of the gradient, which bounded by some Lipschitz constant (Szegedy et al. 2013).
A8 SANITY CHECK FOR DECOY-ENHANCED SALIENCY MAPS
As suggested by Adebayo et al. (2018). any valid saliency methods should pass the sanity check in the sense that the salrency method should be dependent the learned parameters of the predictive model, instead of edge or other generic feature detectors. We performed the model parameter randomization test (Adebayo et 2018) by comparing the output the proposed saliency method on a pretrained VGG16 network with the output of the proposed saliency method weight-randomized VGG16 network, If the proposed saliency method indeed depends on the learned parameters of the model, expected that the outputs between the two cases differ substantially.
Following the cascading randomization strategy (Adebayo et al., 2018), the weights of pretrained VGGI6 network are randomized from the top to bottom layers in cascading fashion. This cascading randomization procedure is designed to destroy the learned weights successively. As illustrated in Figure A8 the cascading randomization destroys the decoy-enhanced saliency maps combined with three existing saliency methods, qualitatively. The conclusion is also supported by quantitative comparison measured by the structural similarity index (SSIM), shown in Figure A9
13

---

Under review as a conference paper at ICLR 2020
Cascading randomization from to bottom layers
Original
2 cunn
2
Terrier
logit
R
fe
enav
cumvs
con
cunt
can cunr cont
Grad
+Decoys
IntGrad
+Decoys
SGrad
+Decoys
Figure A8: Cascading randomization on VGG16 network. The figure shows the original saliency map (first column) for the terrier. Progression from left to right corresponds to complete randomization of the pretrained VGG16 network weights from the top layer to the bottom layer.
1.0
Grad++Decoys
IneGrad+Decoys
0.8
SGrad+Decoys
0.4
original
52
32
11
layer
Figure A9: Structural similarity index (SSIM) for Cascading Randomization on VGG16 network.
A9 SENSITIVITY OF HYPERPARAMETER SELECTION
The optimization formulated in Equation 6 involves one hyperparamer, c, the initial coefficient to weigh the intermediate representation difference between the decoy and the original image. Note that we use the same hyperparameter setting across all the images, rather than for each image separately. To evaluate how the selection of c affects the performance, to wide range of E {10', 102, 10 10a, 10y }. As illustrated in Figure A A10 the choice initial coefficient makes negligible difference of three state-of-the-ard saliency methods, both qualitatively and quantitatively.
carried out optimizations with respect
A10 RUN TIME OF DECOY OPTIMIZATION
To evaluate the computational cost of optimizing Equation 6 we carried out the run time comparison between optimizing one decoy and calculating three types of state-of-the comparison is repeated 500 times with respect to different patch masks. As illustrated in Figure All on average optimizing one decoy takes 37. 87.7% of the run time of the fastest, vanilla gradient- based saliency method. For other methods, the optimization is even less expensive, in relative sense.
saliency methods. The
A11 VALIDITY OF GENERATING DECOY BY BLURRING
As shown in Figure combining blurry images with our decoy-enhanced saliency scores achieves empirically good performance even though the blurry images violate the design of decoys which resemble the intermediate representation of the original image. To understand why blurry images are empirically good, we compared the relative difference between the intermediate representation of the original images and the decoy/blurry images. Here, the relative difference is defined as the
14

---

Under review as a conference paper at ICLR 2020
C=10
C=100
C=1000
C=10000
C=100000
Terrier
Grad +Decoys
SF: -0.20
SF: -0.23
SF: -0.18
SF: -0.26
SF: -0.16
IntGrad +Decoys
SF: -0.37
SF: -0.36
SF: -0.37
SF: -0.36
SF: -0.36
SGrad +Decoys
SF: -0.51
SF: -0.44
SF: -0.47
SF: -0.48
SF:-0.50
Figure A10: Visualization of saliency maps optimized using different hyperparamer
10a
Running time in seconds
102.
Decoy generation
Grad
IntGrad
SGrad
Figure All: Run time to optimize one decoy and calculate three state-ofthh--s saliency maps of the decoy. The comparison conducted the same CPU to guarantee fairness.
L-infinity norm between two intermediate representations divided by the maximum absolute value of any intermediate representation.
15

---

Under review as a conference paper at ICLR 2020
Because the decoy images are designed preserve the intermediate representation of the original images, the relative difference is expected to be small. On the other hand, for the blurry images, the relative difference should be arbitrary since there is such constraint. As illustrated in Figure Al2 in the first maxpool layer (i.e., layer near the bottom of the network), the relative difference is large for blurry images (0.307 average) and very small for decoy images (0.006 on average) as expected. However, in the last fully-connected layer (i.e., a layer near the top of the network), the relative difference is much smaller for blurry images (0.034 on average) and remains small for decoy images (0.002 on average). In conclusion, even though the blurry images violate the constraint of decoys, this violation would be mitigated deeper layers of the network.
Original decoy Blurry decoy
0.5
0.4
Relative difference
0.3
0.1
0.0
vg9_ v99_16ffc6/squeezed
VGG16 layers
Figure A12: Relative difference between the intermediate representation of the original images and decoy/blurry images.
A12 OBJECT LOCALIZATION
standard method to evaluate saliency maps is by object localization Dabkowski & Gal 2017
Fong & Vedaldi 2017). where the model was trained with the class label only without access any localization data. We carried out Imagenet ILSVRC"14 14 localization task (Russakovsky et al. 2015) which contains 50K ImageNet validation images with annotated bounding boxes ground truth. For each of the image in the dataset, we first calculated the gradient-based saliency maps with and without using blurry decoys, based
pretrained VGG16 network. Following the preprocessing
steps suggested by Dabkowski & Gal (2017) Fong & Vedaldi (2017). for each calculated saliency maps, bounding box for the most dominant object is predicted by thresholding the corresponding saliency map and extracting the tightest bounding box that contains the whole remaining saliency map.
We investigated three different thresholding strategies suggested Fong & Vedaldi (017). including value thresholding, energy thresholding, and mean thresholding. Following the evaluation protocol suggested by Dabkowski & Gal 2017 Fong & Vedaldi 2017). the extracted localization box has to have Intersect over Union (loU) greater than 0.5 with any of the ground truth bounding boxes in order to consider the localization successful, otherwise it is regarded as failure. As shown in Table Al] in terms of accuracy, decoy-enhane enhanced saliency maps perform better than vanilla saliency maps without decoys.
16

---

Under review as a conference paper at ICLR 2020
Accuracy Gradient
Value thresholding (0.25)
Energy thresholding (0.25)
Mean thresholding (0.25)
0.662
0.715
0.662
Gradient+Decoy:
0.722
0.723
0.665
Table Al: ImageNet localization accuracy on VGG16 network using different thresholding strategies.
A13
EXPERIMENTAL SETTINGS
patch_siae size
stride
AlexNet
10000
200
1
VGG16
10000
200
1
ResNet
10000
200
Table A2: The experimental settings of proposed methods for all the target models.
The experimental settings of the proposed approach three convolutional neural networks (i.e, AlexNet, VGGI6, ResNet) are shown Table A2 In the table, is the index of the lay- within the target model that is selected to generate the decoy images. The hyperparameter controls the weight of (x) - Fx))) and e refers to the tolarence of [x) (x loc. in Equation (6). The patch_size and stride control the size and the stride step of each decoy patch.
A14 ADDITIONAL EXPERIMENTAL RESULTS
Figure A13 A16 provide examples of the fidelity, stability and generalizability of the decoy-
enhanced saliency methods. These results are consistent with those shown in the Figures
17

---

Under review as a conference paper at ICLR 2020
Grud Decoys
IntGrud Decoys
SGrad Decoys
Grad
SGrad
T-shirt
Sr: 3.90
SF: 1.76
Sf: 3.42
Si: 2.77
Sf: 1.42
SF: 1.26
Lccthopeer
SF: 1.27
Sf: 0.75
SF: 1.45
SF: 0.36
SF: 1.54
SF: 1.48
T Trailer Truck
Sr: 2.23
SF: 1.41
Sr: 3.07
SE: 2.33
SE: 0.55
SF: 0.36
Shield
SF:
SF: 2.44
SF: 3.48
SF:
SF: 2.10
SF: 1.90
Bustard
SF: 0.11
SF: 0.05
SF:-0.09
-0.04
SF: 1.76
SF: 1.61
Vending Machine SF:3.59
SF: 1.34
SF: 1.87
SF: 1.47
SF: 0.93
SF: -0.41
Volcano
SF: 3.79
SF: 3.27
SF: 3.17
SF: 0.36
SF: 1.79
SF: 0.72
Cliff
SF: 5.57
SF: 0.12
SF: 2.98
SF: 0.49
SF: 0.33
SF: 0.18
Scashore
SF: 2.76
SF: 2.73
SF: 2.66
SE: 2.11
SF: 2.23
1.89
Alp
SF: 6.12
SF: 1.03
SF: 4.95
SF: 2.66
SE: 8.28
SE: 7.57
Figure A13: Fidelity evaluation results of four more images on VGG16.
18

---

Under review as a conference paper at ICLR 2020
Grad
Grad Decoys
IntGrad Attlmagee
IntGrad Deceys
SGrad Altlmage
SGrad Decuys
Grad
IntGrad
SGrad
Top-k
42.28
Ss.30.98
Ss. 9.088
Ss: 8.68
19.25
17.76
Mass Center
Ss: 132,42
83.07
Ss 9.72
Ss: 9.65
Ss: 12.25
Ss.9.74
Target
Ss: 103.73
Ssi 66.67
Ss 8.84
Ssi 7.27
Ss. 13.54
Ss. 11.01
(a) Stability results shield.
Grad Attimage
Grad
IntGrad Attimage
IntGrad Decoys
SGrad AttImage
SGrad + Decoys
Decoys
IntGrad
Top-k
Ss: 26.87
Ss: 20.21
Ss.5.97
Ss: 4.60
3.68
Ss: 2,81
Mass Center
Ss: 62.29
4s:40.01
$o:5.44
Ss:3.87
5.57
$s:5.09
Target
Ss 42.51
Ss: 27.67.67
7.92
Ss: 7.08
5s
8x:4,28
(b) Stability results of bustard.
Grad Decoys
IntGrad Attimage
IntGrad Decoys
SGrad AttImage
SGrad Decuys
Grad
IntGrad
SGrad
Attimage
Top-k
Ss: 14.61
Ss. 9.78
Ss. 5.33 Ss 6.19
Ss. 8.73
Ss: 7.63
Mass Center
47.28
Ss. 36.91
8.37
Ss. 7.10
6.61
Target
10.88
Ss. 8.61
Ss. 10.16
Ss. 6.81
Ss. 4,63
Ss. 4.33
(c) Stability results of vending machine.
Figure A14: Stability evaluation results of three more images on VGG16.
19

---

Under review as a conference paper at ICLR 2020
Grail Decays
IntGred
IntGrad Doans
SGrad
SGrad
Deeays
SF:
0.004
0.12
SP:-0.04
SF: 1.20
Macaw
SF: 7.80
SF: 2.22
SF: 4.19
SF: -0.03
SF: 2.30
SP: -1.26
Taillrr Truck
4.44
S3: 2.64
4.36
1.96
3.20
S1:
(a) Original and decoy-enhanced saliency maps Net.
IntGrad
T-shirt
Sa: 6.83
So: 2.66
O:20..3
Se: 0.05
Ss: 0:01
SF: 0.01
Shield
Sr: 3.84
Sp: 2.28
SF: 1.97
SF: 1.15
Se: 0.26
SF: -0.02
Macaw
Scooler
SF: 8.99
SF: 7.35
SF: 10.58
SF: 6.96
SF: 7.64
SF: 7.44
(b) Original and decoy-enhanced saliency maps ResNet.
Figure A15: The illustration of the performances of the proposed approach of more images on AlexNet and ResNet.
Butoon layor hpp bayers
Merponl
Maspel
Hostand
Inttirad
SGrad
(a) The mean and standard derivation of Se score for gradient, integrated gradient and SmoothGrad (-0.01, 0.01). (-0.03, 0.01). (0.09, 0.01).
Bation layers to top layars
Shidd
Grad
SCrad
(b) The mean and standard derivation of Sp score for gradient, integrated gradient and SmoothGrad are: (3.34, 0.73), (2.21, 1.33), (1.49, 0.18).
Figure A16:
Demonstrations of decoy-enhanced saliency maps generated from each convolutional and pooling layer in VGG16.
20


Reviews:
[{"review_id": "r1xwnPtItS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors tackle the important problem of generating saliency maps. First, a decoy image is defined (in short, it is a perturbed version of the image that leads to very similar activations) and then, the method that leverages decoy images is proposed. The method can be understood as a improvement that can be applied to enhance an existing saliency extractor. For a given image, the method generates a bunch of decoys images, and then a given saliency extractor is applied to not only the original image but all generated decoy images. The resulting saliency maps are aggregated to output the final saliency map. I found this idea technically sound.\n\nHowever, there are two keys reasons why this paper should be rejected.\n(1) The idea of using decoy images is interesting but computationally expensive. In practice (as showed in the paper) using blurred images leads to comparable results.\n(2) The paper misses one very important experiment which is a quantitative comparison with the existing works on Imagenet ILSVRC’14 localization task. This is the standard experiment and is used in a few works cited (Fong & Vedaldi, 2017; Dabkowski & Gal, 2017).\n\nEven though the idea of using decoy images is technically sound, I find the algorithm for generating these decoy images very complex and computationally expensive. First, under Eq.6 one can find \"Our strategy is to set c to a small value initially and run the optimization. If it fails, then we double c and repeat until success.\" And then below Eq.8 there is \"After each iteration, if the second term in Equation 8 is zero, indicating that τ is too large, then we reduce τ by a factor of 0.95 and repeat\". Hence, the algorithm is nested and the paper does not provide any details how long the procedure is in practice. On top of that, since a population of decoy images is needed (12 in the experiments), this expensive procedure has to be run a few times. When decoy images are replaced with blurry images the results are almost the same and hence, unfortunately, using decoy images is not justified.\n\nIn general, the experimental results are decent and prove the possible benefits of using the population of images (decoys or blurred ones). However, I think that Imagenet ILSVRC’14 localization task is the standard experiment that provides the quantitative view and should be performed. The paper evaluates on Imagenet data set already, and hence adding this experiment should be straightforward.\n\nThe paper does a good job motivating the problem and covering related work. The paper states that the key limitation of existing saliency maps is that they \"evaluate pixel-wise importance in an isolated fashion by design, implicitly assuming that other pixels are fixed\" and \"the presence of gradient saturation\". While it is valid for gradient-based methods, it is not for perturbation-based methods. The latter are just briefly mentioned and then it is stated that \"for any perturbation-based method, a key challenge is ensuring that the perturbations are effective yet preserving the training distribution\". However, I do not find this reason to be strong enough to exclude these work from consideration, because there are works that train saliency extractor and the classifier simultaneously and then the argument mentioned does not hold (Fan et al., 2017; Zolna et al., 2018).\n\nThere is one more thing that I would like to ask about. In Eq.3, Z_j is defined as max(E˜_j ) − min(E˜_j ). Hence, if a given pixel is very important for all decoy images, all elements of E˜_j will be high and then Z_j will be very small. As a result, a saliency value assigned to this pixel will be low which seems to be counter-intuitive. Can you please elaborate on that?\n\n\n(Fong & Vedaldi, 2017): Ruth C Fong, and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturbation\n(Dabkowski & Gal, 2017): Piotr Dabkowski, and Yarin Gal. Real time image saliency for black box classifiers\n(Fan et al 2017): Lijie Fan, Shengjia Zhao, and Stefano Ermon. Adversarial localization network\n(Zolna et al 2018): Konrad Zolna, Krzysztof J. Geras, and Kyunghyun Cho. Classifier-agnostic saliency map extraction", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "Bye7qbVUqH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I Summary\nThis paper has two major contributions:\n- A method to infer robust saliency maps using distribution preserving decoys (generated perturbated images that resemble the intermediate representation of the original image in the neural network)\n- A decoy-enhanced saliency score which compensates for gradient saturation and takes into account joint activation patterns\nThe authors show the performance of their methods on three different saliency methods, quantitatively and qualitatively, but also when it is submitted to adversarial perturbations.\n\nII Comments\n1. Content\nThe paper is very clear and easy to read, I would like to point out how well structured it is. \nThe method yields very interesting results, the whole work is very complete experimentally (saliency methods used, adversarial perturbations, models etc) and the process coherent. I would love to see it applied to different datasets: Imagenet focuses more on the foreground, whereas \"Places\" focuses more on background objects and could be interesting to use. \nThe authors referred to Sanity Checks for Saliency Maps (Adebayo et al) without using it for their results, it would be nice to add it to the experiments.\n\n\n2. Typos\n- intro, paragraph 2, l2: use -> uses\n- paragraph 5, l9: indendently ->  independently\n- 4.2, paragraph 4, last line: target word twice\n- 4.3, paragraph 2, l5: fisrt -> first\n- l9: not able -> may not be able\n- 4.5, l5: salinency -> saliency\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1xzEKtJ9r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Given a model and an image, the proposed method generates perturbations of the image, such that the model output in intermediate layers of the network does not change. Their method first generates such perturbed images, and then uses those to generate a saliency map in order to interpret the classification.  The idea is overall interesting, but its relationship with many other methods is not adequately examined. Significant portion of recent literature is ignored, especially studies on counterfactuals and decision boundaries in the context of image classification.  For numerical results, they compare their results with raw scores generated by three other methods, but the results are quite thin and disorganized with respect to adversarial robustness and interpretability.\n\nYour proposed computational method possibly needs to be changed, too.   Details are explained below:  \n\nAuthors have cited the paper on \"Sanity Checks for Saliency Maps\", but have not used it to verify their own results, not with respect to saliency map, and not with respect to computational cost.  Several recent methods on saliency maps have not been considered, for example: \n1. Certifiably robust interpretation in deep learning \n2. Interpreting Neural Networks Using Flip Points \n3. Explaining Image Classifiers by Counterfactual Generation\n 4. Counterfactual Visual Explanations \n\nIt's not clear how your proposed saliency scores relate to methods that consider counterfactuals and the decision boundaries. For any image, there is a counterfactual image closest to it. One can also compute the closest image on the decision boundary of the model. Such image can reveal which pixels should be changed in order to change or to keep the classification. The relationship between the current method and those methods needs to be established.\n Assume that for an image, you compute the closest image to it on the decision boundary and let’s call the distance between them r (distance measured in l1 or l2 norm). Then, consider a ball centered at that image with radius r. All images inside that ball would have the same label as the original image and you can obtain all of them only by solving one optimization problem for finding closest point on the decision boundary. Would this be a less expensive computation compared to solving your proposed decoy optimization many times?\n\nThere are many hyper parameters in your optimization problem, and it seems that for each image, hyper-parameters should be tuned separately. However, if you seek the closest image on the decision boundary, your results would be independent of hyper-parameters. \n\nAs you have mentioned, your optimization problem seems quite hard to solve, but no details are given on how long it takes to solve it for a single image. You have mentioned sometimes you cannot solve it if c is not chosen well. This information needs to be reported. Is it solvable at all with a large mask? How many trials did it take to obtain your saliency maps?\n\nYour optimization is nonlinear, non-convex, non-differentiable, …. . How did you deal with non-convexity? Do you arrive at the same solution each time you solve it? These are vital information to about an optimization method.\n\nProposition 1 is quite obvious. Its discussion seems obvious, too, given how you have defined the Z. In your proof, you have considered a single layer network, i.e. a simple linear transformation. It might be better to explain your proof for Proposition 1 as an observation and use it as the motivation on how you have defined Z.\n Overall, you are taking a data-driven approach to interpret the classification of an image. The additional computational cost of your proposed method needs to be compared to other methods that are not data-driven. For example, how does your computational cost compare with the references above?\n\nOn page 12, one equation is referenced with ??.\n\nAt the beginning of section 3.4, you are using a standard optimization technique to augment a constraint as a penalty in the objective function. You can explain it as such, instead of saying we solve an alternative formulation. See Numerical Optimization by Nocedal and Wright.\n\nThe necessity to transform the variable via tanh, instead of performing projection is not explained. Does it make the process faster?\n\nStrategy for choosing a good initial value for c (in the penalty term) and how to increase it is not explained well. How many iterations does it take to find the solution?\n\nAny classification model is defined by its decision boundaries, so interpreting the classification without considering the decision boundaries seems inconclusive.\n\nIn summary, it would be best to compare your results with other methods, some of which are mentioned above. It would also be necessary to justify the additional data-driven cost that you are prescribing and to explain why your goals cannot be achieved otherwise with less expensive computation.\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes a method to explain deep vision models using saliency maps that are robust to certain input perturbations.

Strengths:
-The paper is clear and well-written.
-The approach is interesting.

Weaknesses:
-The motivation and formulation of the approach (e.g. coherence vs explanation and the use of decoys) was not convincing.
-The validation needs additional experiments and comparisons to recent works.

These weaknesses were not sufficiently addressed in the discussion phase. AC agrees with the majority recommendation to reject.
Decision: Reject