Year: 2019

Abstract:
Saliency methods help to make deep neural network predictions more interpretable by identifying particular features, such as pixels in an image, that contribute most strongly to the network’s prediction. Unfortunately, recent evidence suggests that many saliency methods perform poorly when gradients are saturated or in the presence of strong inter-feature dependence or noise injected by an adversarial attack. In this work, we propose a data-driven technique that uses the distributionpreserving decoys to infer robust saliency scores in conjunction with a pre-trained convolutional neural network classifier and any off-the-shelf saliency method. We formulate the generation of decoys as an optimization problem, potentially applicable to any convolutional network architecture. We also propose a novel decoy-enhanced saliency score, which provably compensates for gradient saturation and considers joint activation patterns of pixels in a single-layer convolutional neural network. Empirical results on the ImageNet data set using three different deep neural network architectures—VGGNet, AlexNet and ResNet—show both qualitatively and quantitatively that decoy-enhanced saliency scores outperform raw scores produced by three existing saliency methods.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) deliver remarkable performance in an increasingly wide range of application domains, but they often do so in an inscrutable fashion, delivering predictions without accompanying explanations. In a practical setting such as automated analysis of pathology images, if a patient sample is classified as malignant, then the physician will want to know which parts of the image contribute to this diagnosis. Thus, in general, a DNN that delivers explanations alongside its predictions will enhance the credibility and utility of its predictions for end users (Lipton, 2016).
Two primary classes of methods have been developed for interpreting CNN models. The first class of methods, often referred to as saliency maps, use gradients of classification outcome as a per-pixel importance score to characterize the influence of individual pixels to the prediction (Simonyan et al., 2013; Selvaraju et al., 2016; Binder et al., 2016; Shrikumar et al., 2016; Smilkov et al., 2017; Sundararajan et al., 2017; Levine et al., 2019). Despite providing apparently meaningful interpretations in practice, these methods exhibit fundamental limitations. First, saliency maps evaluate pixel-wise importance in an isolated fashion by design, implicitly assuming that other pixels are fixed (Singla et al., 2019). Worse still, the presence of gradient saturation often breaks the assumption that important features in general correspond to large gradients (Sundararajan et al., 2016; Shrikumar et al., 2016; Smilkov et al., 2017). Specifically, an important pixel may contribute a strong joint effect on the prediction when considered in conjunction with neighboring pixels. However, that important feature may have a tiny gradient locally, in the sense that its marginal contribution is diminishing due to the flattened output of the layer. Finally, Ghorbani et al. (2017) and Kindermans et al. (2017) systematically revealed the fragility of widely-used saliency methods by showing that even an imperceivable perturbation or simple shift transformation of the input data can lead to a large change in the resulting saliency scores. In light of this observation, a key challenge for any saliency method is ensuring that the saliency scores are robust to gradient saturation and perturbations.
A second class of interpretability methods, often based on counterfactual, perturb (i.e., nullifying, blurring, adding noise, or inpainting ) small regions of the image to modify the classifier prediction (Fong & Vedaldi, 2017; Dabkowski & Gal, 2017; Chang et al., 2017; Fan et al., 2017; Chang et al.,
2019; Yousefzadeh & O’Leary, 2019; Goyal et al., 2019). Despite identifying meaningful regions in practice, these methods exhibit several limitations. First, counterfactual-based methods implicitly assume that regions containing the object are the ones most contributed to the prediction (Fan et al., 2017). However, Moosavi-Dezfooli et al. (2017) revealed that counterfactual-based methods are also vulnerable to the adversarial attacks, which force these explanation methods to output nonrelated backgrounds rather than the meaningful objects as important subregions. In addition, the counterfactual images may be potentially far away from the distribution from which the training samples were drawn (Burns et al., 2019), causing the classifier behavior ill-defined. Though some efforts have been made to train the saliency extractor and the classifier simultaneously (Fan et al., 2017; Zołna et al., 2019), the robustness of the resulting explanations are not guaranteed in most cases (Hendrycks & Dietterich, 2019). In light of this, for any perturbation-based method, a key challenge is ensuring that the perturbations are effective yet preserving the training distribution.
In this work, we propose a data-driven technique to infer robust saliency maps with distributionpreserving perturbations. Given an image of interest, the core idea of this method is to generate a population of perturbed images, referred to as decoy images, that resemble the neural network’s intermediate representation of the original image but are conditionally independent of its label. The resulting decoy images capture the variation of the image data originating from either sensor noise or adversarial attacks. The major differences between the proposed and counterfactual-based methods are threefold. First, unlike counterfactual images that seek the minimum set of features to exclude so as to minimize the prediction score or to include so as to maximize the prediction score (Fong & Vedaldi, 2017), saliency map methods like the one we propose in this paper aim to characterize the influence of each feature on the prediction score. Second, unlike counterfactual images that are optimized toward the aforementioned objective with respect to the decision boundary, decoys are designed to be independent of labels, without changing the decision boundary at all. Finally, unlike counterfactual images which could potentially be out-of-distribution by adding noise (Smilkov et al., 2017), rescaling (Sundararajan et al., 2017), blurring (Fong & Vedaldi, 2017), or inpainting the image (Chang et al., 2017), decoys are plausibly constructed in the sense that their intermediate representations are non-discriminative from the original input data by design.
In addition, we propose a novel decoy-enhanced saliency score, which considers an ensemble of saliency maps across multiple decoy images. We also derive a closed-form formula for the decoyenhanced saliency score generated from a single-layer convolutional neural network. This analysis shows that the proposed saliency score for each pixel can be decomposed into two components: the range of values for a given pixel among the decoys and the number of neurons jointly activated by that pixel and its neighbors. The former compensates for gradient saturation (Sundararajan et al., 2016) in the sense that an important pixel can vary without strongly affecting the prediction, due to the joint effect of nearby pixels. The latter compensates for the way in which saliency maps treat each pixel independently (Singla et al., 2019) in the sense that the importance of a pixel is jointly determined by the importance of pixels in its vicinity. This closed-form result may provide insights to help design better defensive strategies against adversarial attacks.
We apply the proposed method to the ImageNet dataset (Russakovsky et al., 2015) in conjunction with three standard saliency methods. We demonstrate empirically that decoy-enhanced saliency scores perform better than the original saliency scores, both qualitatively and quantitatively, even in the presence of various adversarial perturbations to the image.

heading: 2 PROBLEM SETTING
text: Consider a multi-label classification task in which a pre-trained neural network model implements a function F : Rd 7→ RC that maps from the given input x ∈ Rd to C predicted classes. The score for each class c ∈ {1, · · · , C} is F c(x), and the predicted class is the one with maximum score, i.e., arg maxc∈{1,··· ,C} F c(x).
A common instance of this setting is image classification, in which case the input x corresponds to the pixels of an image. A saliency method aims to find the subset of features in the input x that most strongly lead the neural network to make its prediction. Thus, each pixel is assigned a saliency score, encoded in an explanation map E : Rd 7→ Rd, in which the pixels with higher scores represent higher “importance” to the final prediction.
A variety of saliency methods have been proposed in the literature. Some, such as edge or other generic feature detectors (Adebayo et al., 2018), are independent of the predictive model. In this work, we focus on three methods that do depend on the predictor. The vanilla gradient method (Simonyan et al., 2013) simply calculates the gradient of the class score with respect to the input x, which is defined as Egrad(x) = OxF c(x). The SmoothGrad method (Smilkov et al., 2017) seeks to reduce noise in the saliency map by averaging over explanations of noisy copies of an input, defined as Esg(x) = 1 N ∑N i=1Egrad(x+gi) with noise vectors gi ∼ N(0, σ2). The integrated gradients method (Sundararajan et al., 2017) aims to tackle the problem of gradient saturation. The method starts from a baseline input x0 and sums over the gradient with respect to scaled versions of the input ranging from the baseline to the observed input, defined as Eig(x) = (x− x0)× ∫ 1 0 OxF c(x0 + α(x− x0))dα.
In this setting, we are given three inputs: a pre-trained neural network model F with L layers, an image of interest x, and a saliency method E such that E(x;F ) is a saliency map of the same dimensions as x. As illustrated in Figure 1, the robust saliency map can be obtained in two steps: 1) constructing decoy images, and 2) computing decoy-enhanced saliency scores.
3 DEFINITION OF DECOYS
Say that F` : Rd 7→ Rd` is the function instantiated by the given network, which maps from an input image x ∈ Rd to its intermediate representation F`(x) ∈ Rd` at layer ` ∈ {1, 2, · · · , L}. At a specified layer `, the random vector x̃ ∈ Rd is said to be a decoy of x ∈ Rd if the following swappable condition is satisfied: F`(x) = F`(xswap(x̃,P)), for image patches P ⊂ {1, · · · , d} , (1) The swap(x̃,P) operation swaps image patches between x and x̃ (Figure 2). Hence, if P = {10}, then xswap(x̃,P) is a new image which is almost equivalent to x except that the 10th patch
is exchanged from x̃. A “valid” patch is a local region receptive to the convolutional filter of a given CNN, for example, the 3 × 3 area in VGGNet (Simonyan & Zisserman, 2014). The swappable condition ensures that the original image x and its decoy x̃ are indistinguishable in terms of the intermediate representation at layer `. Note in particular that the construction of decoys relies solely on the first ` layers of the neural network F1, F2, · · · , F` and is independent of the succeeding layers F`+1, · · · , FL. In other words, x̃ is conditionally independent of the classification task F (x) given the input x; i.e., x̃ |= F (x)|x. See Supplementary Section A6 for the explanation why decoys exist.

heading: 3.1 GENERATE DECOYS BY OPTIMIZATION
text: Given an image of interest x ∈ Rd and a patch maskM ∈ {0, 1}d, we want to construct a decoy image x̃ = G`(x,M) ∈ Rd, with respect to a specified neural network layer `, so that x̃ is as different as possible from x. For this purpose, we optimize the following objective function:
maximizex̃ ∥∥((x̃− x) · s)+∥∥
1 ,
s.t. ‖F`(x̃)− F`(x)‖∞ ≤ , (x̃− x) ◦ (1−M) = 0, and x̃ ∈ [0, 1] d.
(2)
where (·)+ = max(·, 0), and s = 1 or s = −1 depends on whether we intend to investigate the upward or downward limit of x̃ deviating from x, respectively. The operators ‖·‖1 and ‖·‖∞ correspond to the L1 and L∞ norms, respectively. The constraint (x̃− x) ◦ (1−M) = 0 ensures that x̃ and x differ only in the area marked by the patch mask M, where ◦ denotes entry-wise multiplication. The final constraint ensures that the values in x̃ fall into an appropriate range for image pixels, [0, 1] in the normalized image case. In principle, the formulation in Equation 2 is applicable to any convolutional neural network architecture ranging from simple convolutional networks to more sophisticated ones (e.g., , AlexNet (Krizhevsky et al., 2012), Inception network (Szegedy et al., 2016) and ResNet (He et al., 2016), etc.). As illustrated in Figure 1, we can construct a population of n independently-sampled patch masks{ M1,M2, · · · ,Mn } , so that each possible image patch is covered by at least one patch mask.
Thus we can construct a population of n decoy images { x̃1, x̃2, · · · , x̃n } with respect to different patch masks. Note that the input image x is also a trivially valid decoy, by definition; i.e., it satisfies Equation 1 and the conditional independence requirement (i.e., x̃ |= F (x)|x).

heading: 3.2 DECOY-ENHANCED SALIENCY SCORES
text: Given the generated population of decoy images { x̃1, x̃2, · · · , x̃n } , the given saliency method E can be applied to each decoy image, following the fundamental rule that the saliency method should be agnostic to information revealing which image is a decoy and which is not. In this way, we obtain a corresponding population of decoy saliency maps { E(x̃1;F ), E(x̃2;F ), · · · , E(x̃n;F ) } .
Now the pixel-wise variation for each feature xj can be characterized by a population of feature values X̃j = { x̃1j , x̃ 2 j , · · · , x̃nj } . Accordingly, instead of merely quantifying the saliency of each feature xj via a single value E(x;F )j , we can now characterize the uncertainty of the feature’s saliency via the corresponding population of saliency scores Ẽj = { E(x̃1;F )j , E(x̃ 2;F )j , · · · , E(x̃n;F )j) } .
We define the decoy-enhanced saliency score Zj for each feature xj as
Zj = max(Ẽj)−min(Ẽj) , (3) That is, Zj is determined by the empirical variation of the decoy’s saliency scores.

heading: 3.3 EXAMPLE: SINGLE-LAYER CNN
text: Consider a single-layer convolutional neural network with decoy swappable patch size 1 × 1 and convolutional size 3×3. The input of this CNN is x ∈ Rd, unrolled from a √ d× √ d grayscale image matrix. Similarly, we have a 3× 3 convolutional filter unrolled into g ∈ R9, where g is indexed as gk for k ∈ K, here K = { − √ d− 1,− √ d,− √ d+ 1,−1, 0, 1, √ d− 1, √ d, √ d+ 1 } , corresponding
to the index shift in matrix form from the top-left to bottom-right pixel, respectively. We denote (g∗x) ∈ Rd as the output of the convolution operation on the input x, where (g∗x)i = ∑ k∈K gkxi+k. For simplicity, we further assume that there are no pathological cases such as (g ∗ x)i = 0. Such a neural network can be represented as
m = relu(g ∗ x) , o = WTm + b , p = softmax(o) , (4)
where relu(·) is the entry-wise ReLU operator (Glorot et al., 2011), W ∈ Rd×C represents the combined weights of the neural network, and b ∈ RC represents the biases. The terms o ∈ RC and p ∈ RC are the logits and the predicted class probabilities, respectively. The entry-wise softmax operator for target class c is defined as pc = e
oc∑C c′=1 e o c′ , for c ∈ {1, 2, · · · , C}.
In this case, we have the following result: Proposition 1. Zj is bounded by:
Zj ≤ C2 |(x̃− x)j | ∑ k∈K ãj+k + C1 , (5)
where x̃ is the decoy which maximizes E(x̃;F )j −E(x;F )j , ãj = 1 {(g ∗ x̃)j ≥ 0}, and C1 > 0 and C2 > 0 are bounded constants. See Supplementary Section A7 for the full proof.
Proposition 1 indicates that the saliency statistic is determined by two factors: the range of values associated with pixel j among the decoys (|(x̃− x)j |) and the number of neurons jointly activated by pixel j and its neighbors ( ∑ k∈K ãj+k). The former explains the gradient saturation problem (Sundararajan et al., 2016) in the sense that important features may have more room to fluctuate without influencing the joint effect on the prediction. The latter compensates for way in which saliency maps treat each pixel independently (Singla et al., 2019) in the sense that the importance of a pixel xj is jointly determined by the surrounding pixels (i.e., the 3×3 localized region in our setting), potentially capturing meaningful patterns such as edges, texture, etc. In this way, the proposed saliency statistic not only compensates for gradient saturation but takes joint activation patterns into consideration as well. Indeed, the bound provided by Proposition 1 can be considered a motivation for defining the decoy-enhanced saliency score as we have in Equation 3.

heading: 3.4 IMPLEMENTATION DETAILS
text: We solve Equation 2 by augmenting a constraint as a penalty in the objective function: minimizex̃ − ∥∥((x̃− x) · s)+∥∥
1 + c · ‖F`(x̃)− F`(x)‖∞ ,
s.t. (x̃− x) ◦ (1−M) = 0, and x̃ ∈ [0, 1]d . (6)
When the constant c > 0 is properly chosen, Equations 2 and 6 are equivalent in the sense that the optimal solution to Equation 2 equals the optimal solution to Equation 6. Our strategy is to set c to a small value initially and run the optimization. If it fails, then we double c and repeat until success. The sensitivity of choosing c is reported in Supplementary Section A9.
To eliminate the constraint on pixel values (x̃ ∈ [0, 1]d), we use the change-of-variable trick (Carlini & Wagner, 2017). Note that other transformations are possible yet unexplored in this paper. Specifically, instead of optimizing over x̃, we introduce x̂ satisfying x̃i = 12 (tanh(x̂i) + 1), for all i ∈ {1, 2, · · · , d}. Because tanh(x̂i) ∈ [−1, 1] implies x̃i ∈ [0, 1], any solution to x̂ is naturally valid.
To eliminate of the patch mask constraint ((x̃−x)◦ (1−M) = 0), we use projected gradient descent during the optimization stage. Specifically, after performing each step of standard gradient descent, we enforce x̃ = x̃ ◦M+ x ◦ (1−M). Putting these ideas together, we minimize the following objective function:
minimizex̂ − ∥∥((arctanh(2x̂− 1)− x) · s)+∥∥
1 + c · ‖F`(arctanh(2x̂− 1))− F`(x)‖∞ , (7)
Because the L∞ norm is not fully differentiable, we adopt the trick introduced by Carlini & Wagner (2017). Specifically, in each iteration we solve:
minimizex̂−‖max((arctanh(2x̂− 1)− x) · s, 0)‖1+c· ∥∥(|F`(arctanh(2x̂− 1))− F`(x)| − τ)+∥∥22 , (8) where τ > 0 is initialized with a large value (1 in our experiment). After each iteration, if the second term in Equation 8 is zero, indicating that τ is too large, then we reduce τ by a factor of 0.95 and repeat; otherwise, we terminate the optimization. The run time of optimization is reported in Supplementary Section A10.

heading: 4 EXPERIMENTS
text: To evaluate the effectiveness of our proposed method, we perform extensive experiments on deep learning models that target image classification tasks. The performance of our method is assessed both qualitatively and quantitatively. The results show that our proposed method identifies intuitively more coherent saliency maps than existing state-of-the-art saliency methods alone. The method also achieves quantitatively better alignment to truly important features and demonstrates stronger robustness to adversarial manipulation.
Our experiments primarily use the VGG16 model (Simonyan & Zisserman, 2014) pretrained on the ImageNet dataset (Russakovsky et al., 2015). We also demonstrate the applicability of our method to other well-studied CNN architectures such as AlexNet (Krizhevsky et al., 2012) and ResNet (He et al., 2016). The settings for all experiments are reported in Supplementary Section A13.

heading: 4.1 BENCHMARKS AND VISUALIZATION METHODOLOGIES
text: To benchmark the performance of our proposed method, we considered the following baseline saliency methods: the vanilla gradient method (Simonyan et al., 2013), SmoothGrad (Smilkov et al., 2017) and integrated gradients (Sundararajan et al., 2017).
To evaluate our decoy generation method, we used the following two methods to replace the proposed decoy generation method: blurring and inpainting. In particular, given an image with a few masked patches (i.e., x ◦ (1−M)), instead of generating a decoy image (i.e., x̃) by solving the optimization function in Equation 2, we fill the missing parts by replacing the original missing patches with blurring or using the state-of-the-art inpainting method (Yu et al., 2018).1 Using the decoy images generated by these two substitute methods, we then computed the decoy-enhanced saliency scores for the three baseline saliency methods. We compared the results of blurring and inpainting with our method, both qualitatively and quantitatively.
A heatmap is a typical way of displaying a saliency map. Our strategy for projecting a saliency map onto a heatmap is as follows. Saliency methods produce signed values for input features (pixels in our experiment), where the sign of the value indicates the direction of influence for the corresponding feature to the predicted class. We first took the absolute value of the saliency maps, as suggested by Smilkov et al. (2017). Then we used the maximum value across all color channels, as suggested by Simonyan et al. (2013), to arrive at a single saliency value for each pixel. To avoid outlier pixels with extremely high saliency values leading to an almost entirely black heatmap, we winsorized those outlier saliency values to a relatively high value (e.g., 95th percentile), as suggested by Smilkov et al. (2017), to achieve better visibility. Finally, the values are linearly rescaled to the range [0, 1].

heading: 4.2 COMPARISON CRITERIA
text: To comprehensively evaluate our proposed method against the aforementioned baseline saliency methods, we focus on the following three goals.
First, we aim to achieve visual coherence of the identified saliency map. Intuitively, we would prefer a saliency method that produces a saliency map that aligns cleanly with the object of interest.
Second, we use a saliency fidelity metric (Dabkowski & Gal, 2017) that quantifies the influence of important features identified by a saliency method, defined as SF (E(x;F ),x, c) = −(log(F c(E(x;F ) ◦ x)) − log(F c(E(x;F )mean ◦ x))), where c indicates the predicted class of input x and the saliency map E(x;F ) is normalized into a [0, 1] range, as described in Section 4.1. The function F c(E(x;F ) ◦ x) performs entry-wise multiplication between E(x;F ) and x, encoding the overlaps between the object of interest and the concentration of information by the saliency map. The rationale here is that, by viewing the saliency score of the feature as its contribution to the predicted class, a good saliency method will be able to weight important features more highly than less important ones, giving rise to better predicted class scores. Accordingly, a lower value for the saliency fidelity metric implies a faithful saliency method. Note that we subtract the influence of mean saliency E(x;F )mean (i.e., . the mean of E(x;F ) on each channel) to eliminate the influence of bias in E(x;F ). As a result, we can exclude some trivial cases (e.g., E(x;F ) = 1).
Third, we aim for robustness to adversarial manipulations (Ghorbani et al., 2017). Adversarial attacks aim to apply imperceptible perturbations to an input image that do not affect the predicted class but produce a very different saliency map relative to the map from the unperturbed image. Ghorbani et al.
1We directly used the well-trained inpainting model provide by Yu et al. (2018).
(2017) propose three different attacks: (1) the top-k attack seeks to decrease the important scores of the top-k important features; (2) the target attack aims to increase the feature importance of a pre-specified region in the input image; (3) the mass-center attack aims to spatially change the center of mass of the original saliency map. In this paper, we specify the bottom-right 4× 4 region of the original image for the target attack and select k = 5000 in the top-k attack.
To quantify the robustness of a saliency method E to adversarial attack, we use the sensitivity metric (Alvarez-Melis & Jaakkola, 2018), defined as SS(E(·, F ),x, x̂) = ‖(E(x,F )−E(x̂,F ))‖2‖x−x̂‖2 , where x̂ is the perturbed image of x. A small value means that similar inputs do not lead to substantially different saliency maps.

heading: 4.3 RESULTS
text: We applied all three state-of-the-art saliency methods and their decoys to a dozen randomly sampled images from ImageNet dataset. A side-by-side comparison of the resulting saliency maps (Figure 3) suggests that decoys consistently help to reduce noise and produce more visually coherent saliency maps. For example, in Figure 3, the gradient method without decoys highlights the region corresponding to the dog’s head, but in a scattered format. In contrast, the decoy-enhanced gradient method not only highlights the missing body but identifies the dog head with more details such as ears, cheek, and nose. The visual coherence is also quantitatively supported by the saliency fidelity metric SF , which is consistently higher with decoys than without decoys. The sanity check (Adebayo et al., 2018) of decoy-enhanced saliency method is reported in Supplementary Section A8.
Figure 3 also shows the comparison of our decoy generation method with blurring and inpainting. Our decoy generation method consistently outperforms both competing methods, achieving higher fidelity scores on all of the three baseline saliency methods. The reason is as follows. As is discussed in Proposition 1, to obtain a higher decoy score, a set of decoy images should satisfy the following two requirements: first, they produce similar intermediate representation when provided as input to the target neural network; second, they exhibit large decoy variation. In this paper, we design our decoy generation objective function based on these two requirements. In contrast, neither blurring nor inpainting is designed to meet these requirements. Specifically, blurring may violate the intermediate representation constraint and inpainting may not be able to produce enough decoy variation.
It is worth mentioning that combining image blurring or inpainting with our decoy-enhanced saliency scores can still improve upon the baseline saliency maps. The empirical validity of using image blurring is explained in Supplementary Section A11. From the practitioners’ perspective, blurry images can be used as more efficient alternatives to combine with the decoy-enhanced saliency score to achieve comparable performance. However, it is worth mentioning that decoy images are still necessary to justify the theoretical soundness of the decoy-enhanced saliency score.
Next, to test the robustness of the methods we subjected each to the three aforementioned adversarial attacks (Figure 4). Though not fully resistant to adversarial attacks, we observe that enhancing
existing saliency methods with decoys consistently mitigates the impact of adversarial manipulations, leading to visually more coherent saliency maps as well as lower sensitivity scores. We believe the extra benefits in the robustness is brought by our decoy-enhanced saliency score. In a normal situation (when the image doesn’t suffer from an adversarial attack), an important pixel is not important in an isolated fashion. Instead, the important pixel tends to contribute a strong joint effect in conjunction with neighboring important pixels, to potentially capture meaningful patterns such as edges, texture, etc. In light of this observation, this particular important pixel will have more room to fluctuate without influencing the joint effect on the prediction. In such case, some elements of Ẽj will be high and others will be low, contributing a large Zj . In the unusual situation when an isolated important pixels is indeed observed (i.e., a pixel is very important for all decoy images and has no room for fluctuation.), we tend to believe that the pixel has been adversarial attacked. In this case, all elements of Ẽj will be high and the proposed decoy-enhanced saliency score Zj will be low, which is what we want.

heading: 4.4 SENSITIVITY OF DECOY SETTINGS
text: We also conduct experiments to understand the impact of decoy settings on the performance of different saliency methods. Specifically, we focus on the choice of network layer `, because the intermediate representations can vary significantly at different layers. Accordingly, we vary the value of ` for VGG16 and compare the differences of decoy-enhanced saliency scores from the three saliency methods, ranging from the first convolutional layer to the last pooling layer. As shown in Figure 5 and Supplementary Section A14, the decoy-enhanced saliency scores generated from different layers for the same image are of similar qualities. We also compute the SF score for each saliency map. The mean and standard deviation, respectively, for the scores obtained by the gradient, integrated gradient and SmoothGrad methods are as follows: (-0.11, 0.02), (-0.18, 0.02), (-0.21, 0.01). These quantitative results also support the conclusion that our method is not sensitive to layer. This is likely because, as previous research has shown (Chan et al., 2015; Saxe et al., 2011), the final classification results of a DNN are not highly related to the intermediate representations. As a result, decoys for the same sample with the same label from different layers should yield similar results.

heading: 4.5 APPLICABILITY TO OTHER NEURAL ARCHITECTURES
text: In addition to the VGG16 model (Simonyan & Zisserman, 2014), we generate saliency maps for other neural architectures: AlexNet (Krizhevsky et al., 2012) and ResNet (He et al., 2016). We visualize their saliency maps in Figure 6 and Supplementary Section A14. From Figure 6, we observe that all the saliency maps accurately highlight the contour of the dog and achieve higher fidelity score
than the baseline saliency methods. This indicates that we can apply our method to various neural architectures and expect consistent performance. It should be noted that, in comparison with saliency maps derived from other neural architectures, the maps tied to AlexNet are relatively vague. We hypothesize that this is due to the relative simplicity of the AlexNet architecture.

heading: 5 DISCUSSION AND CONCLUSION
text: In this work, we have used distribution-preserving decoys as a way to produce more robust saliency scores. First, we formulate decoy generation as an optimization problem, in principle applicable to any network architecture. We demonstrate the superior performance of our method relative to three standard saliency methods, both qualitatively and quantitatively, even in the presence of various adversarial perturbations to the image. Second, by deriving a closed-form formula for the decoyenhanced saliency score, we show that our saliency scores compensate for the frequently violated assumption in saliency methods that important features in general correspond to large gradients.
Although we have shown that decoys can be used to produce more robust saliency scores, there remain some interesting directions for future work. For example, one could potentially reframe interpretability as hypothesis testing and use decoys to deliver a set of salient pixels, subject to false discovery rate control at some pre-specified level (Burns et al., 2019; Lu et al., 2018).

heading: A6 EXISTENCE OF DECOYS
text: As illustrated in Figure A7, the original image and the decoy image shares exactly the same intermediate representation with respect to convolutional and max pooling.

heading: A7 PROOF OF PROPOSITION 1
text: Consider a single-layer convolutional neural network with decoy swappable patch size 1 × 1 and convolutional size 3×3. The input of this CNN is x ∈ Rd, unrolled from a √ d× √ d grayscale image matrix. Similarly, we have a 3× 3 convolutional filter unrolled into g ∈ R9, where g is indexed as gk for k ∈ K, here K = { − √ d− 1,− √ d,− √ d+ 1,−1, 0, 1, √ d− 1, √ d, √ d+ 1 } , corresponding
to the index shift in matrix form from the top-left to bottom-right pixel, respectively. We denote (g∗x) ∈ Rd as the output of the convolution operation on the input x, where (g∗x)i = ∑ k∈K gkxi+k. For simplicity, we further assume that there are no pathologic cases such as (g ∗ x)i = 0. Such a neural network can be represented as
m = relu(g ∗ x) , o = WTm + b ,
p = softmax(o) , (9)
where relu(·) is the entry-wise ReLU operator, W ∈ Rd×C represents the combined weights of the neural network, and b ∈ RC represents the biases. The terms o ∈ RC and p ∈ RC are the logits and the predicted class probabilities, respectively. The entry-wise softmax operator for target class c is defined as pc = e
oc∑C c′=1 e o c′ , for c ∈ {1, 2, · · · , C}.
The gradient of pc with respct to x can be written as follows, using the denominator layout notation of the derivative of a vector:
Oxpc = ∂m
∂x
∂o
∂m ∂pc ∂o , (10)
where ∂o
∂m = W , (11)
and { ∂pc ∂oc′
= (pc − p2c) if c′ = c , ∂pc ∂oc′ = −pcpc′ otherwise . (12)
Then, we can write ∂pc∂oc′ as follows: ∂pc ∂oc′ = P̂·c , (13)
where P̂·c corresponds to the c-th column of P̂ and P̂ = diag(p)− ppT .
We then define ∂m∂x as B ∈ R d×d, in which
Bij = ∂mj ∂xi = ∂(g ∗ x)j ∂xi ∂mj ∂(g ∗ x)j = aj ∂(g ∗ x)j ∂xi , (14)
where aj is denoted as aj = 1 {(g ∗ x)j ≥ 0}. Then, we have{ Bij = ajgi−j if i− j ∈ K , Bij = 0 otherwise .
(15)
Given Equations 11, 13 and 15, Equation 10 can be rewritten as
Oxpc = BWP̂·c, (16)
Further, we define the Hessian as H, where Hij can be written as follows:
Hij = Oxi(Oxjpc) = ∂(Bj·WP̂·c)
∂xi
= d∑ k=1 ∂(BjkWk·P̂·c) ∂xi
= ∑ k∈K ∂(Bj,j+kWj+k,·P̂·c) ∂xi
= (∑ k∈K Bj,j+kWj+k,· ) ∂P̂·c ∂xi
= (∑ k∈K aj+kgkWj+k,· ) ∂P̂·c ∂xi
(17)
and ∂P̂c′c ∂xi = { (1− 2pc)Oxipc if c′ = c , pcOxipc′ + pc′Oxipc otherwise .
(18)
After computing the gradient and the Hessian, we now derive the decoy-enhanced saliency score Zj for xj , given a population of saliency scores Ẽj = { E(x̃1;F )j , E(x̃ 2;F )j , · · · , E(x̃n;F )j) }
. According to Equation 3, Zj can always be decomposed into Zj = Z+j + Z − j where Z + j = max(Ẽj)− E(x;F )j and Z−j = E(x;F )j −min(Ẽj), respectively. Without loss of generality, we only focus on Z+j , noting that a smilar analysis is applicable to Z − j as well.
Let x̃ ∈ { x̃1, x̃2, · · · , x̃n } denote the decoy which maximizes E(x̃;F )j − E(x;F )j .
The second-order Taylor expansion of the predicted F c(x) for target class c around x is as follows:
F c(x) ≈ F c(x̃) + Ox̃F c(x̃)T ∆ + 1
2 ∆THx̃∆ (19)
where ∆ = x− x̃ and Hx̃ is the Hessian of the neural network model on the decoy x̃ whose entries are (Hx̃)i,j = ∂ 2F c
∂x̃i∂x̃j . By definition of the swappable condition in Equation 1, F c(x) = F c(x̃).
Therefore, the following equation holds:
Ox̃F c(x̃)T ∆ ≈ −1
2 ∆THx̃∆ (20)
Consider the case that the swappable patches are of size 1× 1 at position i; then Equation 20 can be decomposed into
E(x̃;F )i = (Ox̃F c(x̃))i ≈
1 2 (x̃− x)i(Hx̃)i,i, (21)
Then, according to Equation 21, we have
Zj = 1
2 (x̃− x)j(Hx̃)jj − Oxjpc
≤ ∣∣∣∣∣12(x̃− x)j ∑ k∈K ãj+kgkW(j+k)·P̃·c ∣∣∣∣∣+ C1 ≤ C2 |(x̃− x)j |
∑ k∈K ãj+k + C1
(22)
where C1 = ∣∣Oxjpc∣∣ and C2 = maxk∈K ∣∣∣gkW(j+k)·P̃·c∣∣∣, respectively. Note that both C1 and C2
are linear combinations of the gradient, which is bounded by some Lipschitz constant (Szegedy et al., 2013).

heading: A8 SANITY CHECK FOR DECOY-ENHANCED SALIENCY MAPS
text: As suggested by Adebayo et al. (2018), any valid saliency methods should pass the sanity check in the sense that the saliency method should be dependent on the learned parameters of the predictive model, instead of edge or other generic feature detectors. We performed the model parameter randomization test (Adebayo et al., 2018) by comparing the output of the proposed saliency method on a pretrained VGG16 network with the output of the proposed saliency method on a weight-randomized VGG16 network. If the proposed saliency method indeed depends on the learned parameters of the model, it is expected that the outputs between the two cases differ substantially.
Following the cascading randomization strategy (Adebayo et al., 2018), the weights of pretrained VGG16 network are randomized from the top to bottom layers in a cascading fashion. This cascading randomization procedure is designed to destroy the learned weights successively. As illustrated in Figure A8, the cascading randomization destroys the decoy-enhanced saliency maps combined with three existing saliency methods, qualitatively. The conclusion is also supported by quantitative comparison measured by the structural similarity index (SSIM), shown in Figure A9.
co nv 5_ 3 co nv 5_ 2 co nv 5_ 1 co nv 4_ 3 co nv 4_ 2 co nv 4_ 1 co nv 3_ 3 co nv 3_ 2 co nv 3_ 1 co nv 2_ 2 co nv 2_ 1
Grad +Decoys
Terrier
Cascading randomization from top to bottom layers
log it fc_ 7 fc_ 6 Or igi na l
co nv 1_ 2 co nv 1_ 1
IntGrad +Decoys
SGrad +Decoys
Figure A8: Cascading randomization on VGG16 network. The figure shows the original saliency map (first column) for the terrier. Progression from left to right corresponds to complete randomization of the pretrained VGG16 network weights from the top layer to the bottom layer.
original logit fc7 fc6 53 52 51 43 42 41 33 32 31 22 21 12 11 layer
0.0
0.2
0.4
0.6
0.8
1.0
SS IM
Grad+Decoys IntGrad+Decoys SGrad+Decoys
Figure A9: Structural similarity index (SSIM) for Cascading Randomization on VGG16 network.

heading: A9 SENSITIVITY OF HYPERPARAMETER SELECTION
text: The optimization formulated in Equation 6 involves one hyperparamer, c, the initial coefficient to weigh the intermediate representation difference between the decoy and the original image. Note that we use the same hyperparameter setting across all the images, rather than for each image separately. To evaluate how the selection of c affects the performance, we carried out optimizations with respect to a wide range of c ∈ { 101, 102, 103, 104, 105 } . As illustrated in Figure A10, the choice of initial coefficient c makes a negligible difference of three state-of-the-art saliency methods, both qualitatively and quantitatively.

heading: A10 RUN TIME OF DECOY OPTIMIZATION
text: To evaluate the computational cost of optimizing Equation 6, we carried out the run time comparison between optimizing one decoy and calculating three types of state-of-the-art saliency methods. The comparison is repeated 500 times with respect to different patch masks. As illustrated in Figure A11, on average optimizing one decoy takes 37.7% of the run time of the fastest, vanilla gradient-based saliency method. For other methods, the optimization is even less expensive, in a relative sense.
A11 VALIDITY OF GENERATING DECOY BY BLURRING
As shown in Figure 3, combining blurry images with our decoy-enhanced saliency scores achieves empirically good performance even though the blurry images violate the design of decoys which resemble the intermediate representation of the original image. To understand why blurry images are empirically good, we compared the relative difference between the intermediate representation of the original images and the decoy/blurry images. Here, the relative difference is defined as the
L-infinity norm between two intermediate representations divided by the maximum absolute value of any intermediate representation.
Because the decoy images are designed to preserve the intermediate representation of the original images, the relative difference is expected to be small. On the other hand, for the blurry images, the relative difference should be arbitrary since there is no such constraint. As illustrated in Figure A12, in the first maxpool layer (i.e., a layer near the bottom of the network), the relative difference is large for blurry images (0.307 on average) and very small for decoy images (0.006 on average) as expected. However, in the last fully-connected layer (i.e., a layer near the top of the network), the relative difference is much smaller for blurry images (0.034 on average) and remains small for decoy images (0.002 on average). In conclusion, even though the blurry images violate the constraint of decoys, this violation would be mitigated in deeper layers of the network.

heading: A12 OBJECT LOCALIZATION
text: A standard method to evaluate saliency maps is by object localization (Dabkowski & Gal, 2017; Fong & Vedaldi, 2017), where the model was trained with the class label only without access to any localization data. We carried out Imagenet ILSVRC’14 localization task (Russakovsky et al., 2015) which contains 50K ImageNet validation images with annotated bounding boxes as ground truth. For each of the image in the dataset, we first calculated the gradient-based saliency maps with and without using blurry decoys, based on a pretrained VGG16 network. Following the preprocessing steps suggested by Dabkowski & Gal (2017); Fong & Vedaldi (2017), for each calculated saliency maps, a bounding box for the most dominant object is predicted by thresholding the corresponding saliency map and extracting the tightest bounding box that contains the whole remaining saliency map.
We investigated three different thresholding strategies suggested by Fong & Vedaldi (2017), including value thresholding, energy thresholding, and mean thresholding. Following the evaluation protocol suggested by Dabkowski & Gal (2017); Fong & Vedaldi (2017), the extracted localization box has to have Intersect over Union (IoU) greater than 0.5 with any of the ground truth bounding boxes in order to consider the localization successful, otherwise it is regarded as failure. As shown in Table A1, in terms of accuracy, decoy-enhanced saliency maps perform better than vanilla saliency maps without decoys.
Accuracy Value thresholding (0.25) Energy thresholding (0.25) Mean thresholding (0.25) Gradient 0.662 0.715 0.662
Gradient+Decoys 0.722 0.723 0.665
Table A1: ImageNet localization accuracy on VGG16 network using different thresholding strategies.

heading: A13 EXPERIMENTAL SETTINGS
text: ` c patch_size stride τ AlexNet 6 10000 200 3 1 1 VGG16 3 10000 200 3 1 1 ResNet 2 10000 200 3 1 1
Table A2: The experimental settings of proposed methods for all the target models.
The experimental settings of the proposed approach on three convolutional neural networks (i.e., AlexNet, VGG16, ResNet) are shown in Table A2. In the table, ` is the index of the layer within the target model that is selected to generate the decoy images. The hyperparameter c controls the weight of ‖F`(x̃)− F`(x)‖∞, and refers to the tolarence of ‖F`(x̃)− F`(x)‖∞ in Equation (6). The patch_size and stride control the size and the stride step of each decoy patch.

heading: A14 ADDITIONAL EXPERIMENTAL RESULTS
text: Figure A13–A16 provide more examples of the fidelity, stability and generalizability of the decoyenhanced saliency methods. These results are consistent with those shown in the Figures 3–6.

Review summaries:
Given a model and an image, the proposed method generates perturbations of the image, such that the model output in intermediate layers of the network does not change. Their method first generates such perturbed images, and then uses those to generate a saliency map in order to interpret the classification.  The idea is overall interesting, but its relationship with many other methods is not adequately examined. Significant portion of recent literature is ignored, especially studies on counterfactuals and decision boundaries in the context of image classification.  For numerical results, they compare their results with raw scores generated by three other methods, but the results are quite thin and disorganized with respect to adversarial robustness and interpretability.