Decoy-enhanced Saliency Maps 
Authors: Yang Young Lu
Wenbo Guo
Xinyu Xing
William Noble
Venue:  (2021)
Keywords: Deep neural network
Explainable AI
Saliency methods
Decoys

Abstract: Saliency methods can make deep neural network predictions more interpretable by identifying a set of critical features in an input sample, such as pixels that contribute most strongly to a prediction made by an image classifier. Unfortunately, recent evidence suggests that many saliency methods poorly perform, especially in situations where gradients are saturated, inputs contain adversarial perturbations, or predictions rely upon inter-feature dependence. To address these issues, we propose a framework that improves the robustness of saliency methods by following a two-step procedure. First, we introduce a perturbation mechanism that subtly varies the input sample without changing its intermediate representations. Using this approach, we can gather a corpus of perturbed data samples while ensuring that the perturbed and original input samples follow the same distribution. Second, we compute saliency maps for the perturbed samples and propose a new method to aggregate saliency maps. With this design, we offset the gradient saturation influence upon interpretation. From a theoretical perspective, we show that the aggregated saliency map not only captures inter-feature dependence but, more importantly, is robust against previously described adversarial perturbation methods. Following our theoretical analysis, we present experimental results suggesting that, both qualitatively and quantitatively, our saliency method outperforms existing methods, in a variety of applications.

Under review as a conference paper at ICLR 2021
DECOY-ENHANCED SALIENCY MAPS
Anonymous authors
Paper under double-blind review
ABSTRACT
Saliency methods can make deep neural network predictions more interpretable by identifying set of critical features in an input sample, such pixels that contribute most strongly to prediction made by an image classifier. Unfortunately, recent evidence suggests that many saliency methods poorly perform, especially situations where gradients are saturated, inputs contain adversarial perturbations, or predictions rely upon interffeauree dependence. To address these issues, we propose framework that improves the robustness of saliency methods by following a two-step procedure. First, we introduce perturbation mechanism that subtly varies the input sample without changing its intermediate representations. Using this approach, we can gather a corpus of perturbed data samples while ensuring that the perturbed and original input samples follow the same distribution. Second, we compute saliency maps for the perturbed samples and propose a new method aggregate saliency maps. With this design, we offset the gradient saturation influence upon interpretation. From a theoretical perspective, we show that the aggregated saliency map not only captures inter-feature dependence but, more importantly, is robust against previously described adversarial perturbation methods. Following our theoretical analysis, we present experimental results suggesting that, both qualitatively and quantitatively, our saliency method outperforms existing methods, in variety of applications.
INTRODUCTION
Deep neural networks (DNNs) deliver remarkable performance in an increasingly wide range of application domains, but they often do so in an inscrutable fashion, delivering predictions without accompanying explanations. In practical setting such as automated analysis of pathology images, patient sample is classified as malignant, then the physician will want to know which parts of the image contribute to this diagnosis. Thus, in general, a DNN that delivers interpretations alongside its predictions will enhance the credibility and utility of its predictions for end users (Lipton) 2016).
In this paper, we focus on a popular branch of explanation methods, often referred to as saliency methods, which aim to find input features (e.g., image pixels or words) that strongly influence the network predictions (Simonyan an et al. 2001 Selvaraju et al.) 20166] Binder et al. 2016; [hhiiumarr al. 2017 Smilkov et al.] 2017 Sundararajan et al. 2017; Ancona et al. 2018). Saliency methods typically rely on back-propagation from the network ' output back to its input to assign a saliency score to individual features that higher scores indicate higher importance the output prediction. Despite attracting increasing attention, saliency methods suffer from several fundamental limitations:
Gradient saturation (Sundararajan et al. 2017 Shrikumar et al. 2017 Smilkoy et al. 2017) may lead to the problem that the gradients of important features have small magnitudes, breaking down the implicit assumption that important features. in general, correspond to large gradients. This issue can be triggered when the DNN outputs are flattened in the vicinity of important features.
Importance isolation (Singla et al. 2019) refers to the problem that gradient- saliency methods evaluate the feature importance an isolated fashion, implicitly assuming that the other features are fixed.
Perturbation sensitivity (Ghorbani et al. 2017 Kindermans al. 2017 Levine et al. 2019) refers to the observation that even imperceivable, random perturbations or a simple shift transformation of the input data may lead to a large change in the resulting saliency scores.

---

Under review as a conference paper at ICLR 2021
In this paper, we tackle these limitations by proposing decoy-enhanced saliency score. At a high level, our method generates the saliency score of an input by aggregating the saliency scores of multiple perturbed copies of this imput. Specifically, given input sample of interest, method first generates a population of perturbed samples, referred to as decoys, that perfectly mimic the neural network's intermediate representation of the original input. These decoys used to model the variation of an input sample originating from either sensor noise or adversarial attacks. The decoy construction procedure draws inspiration from the knockoff's proposed recently by BBarber & Candès (2015) in the setting of error-controlled feature selection, where the core idea is to generate knockoff features that perfectly mimic the empirical dependence structure among the original features.
In brief, the current paper makes three primary contributions, First, we propose framework to perturb input samples to produce corresponding decoys that preserve the input distribution, in the sense that the intermediate representations of the original input data and the decoys are indistinguishable. We formulate decoy generation as an optimization problem, applicable to diverse deep neural network architectures. Second, we develop a decoy-enhanced saliency score by aggregating the saliency maps of generated decoys. By design, this score naturally offsets the impact of gradient saturation. From a theoretical perspective, we show how the proposed score simultancously reflect the joint effects of other dependent features and achieve robustness to adversarial perturbations. Third, we demonstrate empirically that the decoy-en -echnnee saliency score outperforms existing saliency methods, both qualitatively and quantitatively, on three real-world applications. We also quantify our method's advantage over existing saliency methods in terms of robustness against various adversarial attacks.
2
RELATED WORK
variety of saliency methods have been proposed in the literature. Some, such as edge detectors and
Guided Backpropagation (Springenberg et al. (2014) are independent from the predictive model (Nic)
al. 2018 Adebayo al. 2018). Others are designed only for specific architectures (i.e.. Grad-
CAM (Selvaraju et al. 2016) for CNNs, DeConvNet for CNNs with ReI.U activations (Zeiler &
Fergus 2014)). In this paper, instead of exhaustively evaluating all saliency methods, we apply our
method to the three saliency methods that do depend on the predictor (i.e., passing the sanity checks in Adebayo et al. (2018) and (ixt et al. 2020 and are applicable to diverse DNN architectures:
The vanilla gradient method (Simonyan et al. 2013) simply calculates the gradient of the class score with respect to the input x, which defined as (x; F*) (x).
The SmoothGrad method (Smilkov et al. 2017) seeks to reduce noise in the saliency map by averaging over explanations of the noisy copies of input, defined as Eeg (x: F*)
N ΣA=1 Eggra( + gs; Po) with noise vectors gi ~ N(0, o2).
The integrated gradient method (Sundararajan et al. 2017) starts from baseline input x² and sums over the gradient with respect to scaled versions of the input ranging from the baseline to the observed input, defined as Fig (x; Fe) (x x²) ))> fo vx Fx + a(x xxx-xxl )d)
We do not empirically compare to several other categories of methods. Counterfactual-ban based methods work under the same setup as saliency methods, providing explanations for the predictions of a pre- trained DNN model (Sturmfels et al. 2020). These methods identify the important subregions within an input image by perturbing the subregions (by adding noise, rescaling (Sundararajan et al.) 2017).
blurring (Fong& & Vedaldi 2017), or inpainting (Chang et al., 2019)) and measuring the resulting changes in the predictions (Ribeiro et al. 2016 Lundberg & Leel 2017 Chen et al. 2018 Fong & Vedaldi 2017: (Dabkowski & Gal 2017 7 Chang et al.12019 Yousefzadeh O'I O'Leary [019 Goyal et al.) 2019). Although these methods do identify meaningful subregions in practice, they exhibit several limitations. First, counterfactual -based methods implicitly assume that regions containing the object most contribute to the prediction (Fan et al. [2017). However, Mooosavi-Dezfoo et al. (207)) showed that counterfactual -based methods are also vulnerable to adversarial attacks, which force these methods output unrelated background rather than the meaningful objects as important subregions.
Sixt et al. (020)) shows that LRP (Binder et al. 2016) is independent of the parameters of certain layers. Ancona et al. (2018) shows that input @ gradient and Deepl.IFT (Shrikumar et al. 2017) are strongly related to the integrated gradient. As such, we only select the integrated gradient.
2

---

Under review as a conference paper at ICLR 2021
(A)
Pretrained Label-dependent gradients
Decoy-anhanced saliency maps
(B Original image
Identical intermediate representation
Decoy Generator
Saliency Methods
Original image
Decoy images
Saliency maps of decoys
Decay image
Figure 1: The overview of the proposed method. (A) The workflow of creating Recoy-enhancee saliency maps. (B) The operation of swapping image patches between original and decoy images.
Second, the counterfactual images may be potentially far away from the training distribution, causing ill-defined classifier behavior (Burns et al. 2019 Hendrycks & Dietterich] 2019).
methods and our decoy-based method are fundamentally different in three ways. First, the former seeks the minimum set of features to exclude in order to minimize the prediction score include in order maximize the prediction score (Fong) & Vedaldi 2017), whereas our approach aims to characterize the influence of each feature on the prediction score. Second, counterfactual comparing each image to the closest image the other side of the boundary. In contrast, the proposed method only considers the decision boundary implicitly by calculating the gradient's variants. Third, unlike counterfactual images, which could potentially be out-of citto--distribution. decoys plausibly constructed in the sense that their intermediate representations are indistinguishable from the original input data by design. Because of these limitations and differences, do not compare method with counterfactual-basee methods.
In addition these limitations, counterfactual-
methods explicitly consider the decision boundary by
In addition to saliency methods and counterfactual-basee methods, several other types of interpretation methods have been proposed that either aim for a different goal or have a different setup. For example, recent research (e.g., Ribbiiro al. (2016); Lundberg & Lee (2017): ) Chen al. (2018)
2019b)) designed techniques to explain a black-box model, where the model's 's internal weights are
inaccessible. Koh Liang (0117) and some follow-up work (Yeh al. 2018 Koh et al. 201
tried to find the training points that are most influential for given test sample. Some other e efforts have been made to train more interpretable DNN classifier (Fan et al., 2017 Zolna et al. 2019
Alvarez-Melis & Jaakkola, 2018 Toneva & Wehbe 2019), synthesize samples that represent the
model predictions (Ghorbani al.
2019 Chen et al. 2019a)), ) or identifying noise-tolerant features
(Ikeno & Hara 2018: Schulz et al. 2020). However, due the task and setup differences, do not
consider these methods in this paper.
3 METHODS
3.1 PROBLEM SETUP
Consider a multi-label classification task in which a pre-trained neural network model implements a function F: gd + C that maps from the given input x E Rd to C predicted classes. The score for each class E {1..... C} is F F*(x), and the predicted class is the one with maximum score, i.e, arg max. (1,"..1) (x).. saliency method aims to assign each feature saliency score, encoded in a saliency map Exx; F²): gd + Rdd in which the features with higher scores represent higher "importance" relative to the final prediction.
Given pre-trained neural network model with L layers, an input x, and saliency method E such that Exx; F) is saliency map of the same dimensions as the proposed scores can be obtained in two steps: generating decoys and aggregating the saliency maps of the decoys (See Fig. A that illustrates the workflow of creating eecayced saliency maps).
3

---

Under review as a conference paper at ICLR 2021
3.2 DECOY DEFINITION
Say that Pr: : Rd + Rd is the function instantiated by the given network, which maps from an input Rd to its intermediate representation Fx(x) E R& layer 8 E (1.2..... L}. A vector x Rd
is said to be a decoy of x E IR at specified layer e if the following swappable condition is satisfied:
Fx(x) Fi (X) for swappable features KC (L....
(I)
Here, the swap( & K.) operation swaps features between and X based on the elements in K. In this work, IC represents a small meaningful feature set, which represents a small region/segment in an image group of words (embeddings) in sentence. Take image recognition task for example. Assume IC = {10) and x is a zero matrix, then Xswap(&X.C) indicates a new image that is identical to except that the tenth pixel is set to zero. An illustrative explanation of swap operator is shown in Fig. BB. Using the swappable condition, we aim to ensure that the original image x and its decoy x are indistinguishable in terms of the intermediate representation
layer L. Note in particular that the construction of decoys relies solely on the first l layers of the neural network F1, F2, ... . Fe and is independent of the succeeding layers Fi+1.- FL. As such, X is conditionally independent of the classification task F(x) given the input x; i.e,, X Il F(x)[x.
3.3 DECOY GENERATION
To identify decoys satisfying the swappable condition, we solve the following optimization problem:
maximize Xmi Xna]d
||((8 x) - +))+
[[]F((x) - -e(())l Se { 1 x) (1- - MM) =0
(2)
Here, ())+ = mmx((;))) 0), and the operators I|-II, and 11 correspond to the Ln and Loo norms, respectively. M {0, 13 2 is a specified binary mask. And the value of each feature in the decoy is restricted to lie in a legitimate value range i.e., [Xmin: Xmax] (e.g., the pixel value should lie in [0, 255]). We impose the constraint [|F((X) - Fx)) I c, wwiich ensures that the generated decoy satisfies the swappable condition described in Eqn. [1] It should be noted that we take x and x to be indistinguishable except for the swappable features indicated by the mask (i.., Xswap(& = x).
As is shown later in Section 3.4 range of the decoy saliencies. Here, we first need to estimate the upper/lower ends of the legitimate decoys. To achieve this, in Eqn. [ we maximize the deviation between & and x from both the positive and negative directions, i.e., = +1 and 8 -1. By using this objective function, for each mask M, we can compute two decoys-one for the positive deviation (i.e. +1) and the other for the negative one (i.e. s = =1).
decoy-enhanced saliency score is defined to capture the empirical
To solve the optimization function in Eqn. 2 we employ three commonly adopted methods - lagrange multiplier, projected gradient, and change-of-variabe (Carlini & Wagner 2017 to transform the original objective function into the following form:
minimizex - max(( (tanh(x) + +1) - x) s, o0/| + |ara 1 (tanh(()) -))-))+(x() 7)
(3)
where À >> 0 is the lagrange multiplier. &; arctanh(2X; 1), for all i E {1, d). 0 is
introdued to approximate the Los norm in Eqn. 2 After obtaining & by solving Eqn. 3 we compute x and map it back to the original feature value range [Xmin Xmax]. More details about how to transform
Eqn. into Eqn. 3 can be found in Section A6
3.4 DECOY-ENIIACED SALIENCY SCORES
Given an input sample x and a swappable patch with size P. we can obtain (Vd s unique masks by sliding the swappable patch across the input with certain stride. For computational efficiency, we aggregate m masks into one decoy sample and optimize these masks jointly by solving one decoy sample from Eqn. Then, we can generate 2n decoys for that sample. We
denote these decoys as { ,,.,.... 22 Here, n = (vd - P + stride))
For these decoys,
4

---

Under review as a conference paper at ICLR 2021
then apply given saliency method E to yield the corresponding decoy saliency maps {E(x); F), F) ;(x²";F)}
With these decoy saliency maps in hand, for each feature X, in characterize its saliency score variation by using a population of saliency scores E ( FO);, ((x2:F)); E(x²", ] In this work, we define the decoy-enhanced saliency score Z for each feature x; as
Z, max(E.)
min E.).
(4)
Here, Z, is determined by the empirical range of the decoy saliency scores. Ideally, important features will have large values and unimportant ones will have small values. Note that the proposed method is designed specifically for nonlinear models in need interpretation. As is discussed in Section [A7] cannot output meaningful saliency maps linear models. It should also be noted that by sliding the swappable patch across the input and ensembling the obtained decoy-enhancee saliency maps, we could capture the saliency of each feature. The motivations of manipulating patch level rather than the entire input are capturing the local dependency structure and enabling batch operations for better efficiency.
3.5 THEORETICAL INSIGHTS
In this section, we analyze the saliency score method in a theoretical fashion. In particular, we take a convolutional neural network with the ReLU activation function as an example to discuss why the proposed interpretation method can account for inter-feature dependence while also improving explanatory robustness. It should be noted that, while we conduct our theoretical analysis the setting of CNNs with specific activation function, the conclusions drawn from the theoretical analysis can easily be extended to other feed-forward neural architectures and other activation functions (e.g., sigmoid and tanh). For analysis of other neural architectures, see Section A9
Consider a CNN with I. hidden blocks, with each layer containing a convolutional layer with a filter of size / x /se and max pooling layer with pooling size /st X /st. (We set the pooling size the same as the kernel size in each block for simplicity.) The input to this CNN is E R4, unrolled from vd vd matrix. Similarly, we also unroll each convolutional filter into ge E R where ge is indexed as (g²); for j E Jr. Here, Ti corresponds to the index shift in matrix form from the top-left to bottom- right element. For example, 3 x convolutional filter (i.e., 8, 9) is indexed by J: - the probability vector p e R generated by the softmax function, where C' is the total number of classes. Such a network can be represented as
1, - vd, vd 1, - 1, 0, vd 1, Vd, va + 1). The output of the network is
m, pool(relu(g:: * m)-1) for 1,2,3,....
W + b
softmax(o),
where relu(-) and pool(-) indicate the ReLU and pooling operators, m, E Rd is the output of the block& l (m, = x), and (g: * m(-1) Id represents convolutional operation on that block. We assume for simplicity that the convolution retains the input shape.
Consider an input and its decoy generated by swapping features in K., For each feature i K. we have the following theorem for the decoy-enhanced saliency score Z₁
Theorem 1. In the aforementioned setting, Z; is bounded by
Z,
Σ ( (HH)),) kek
C1.
(6)
Here, C1 > 0 is bounded constant and H₂ is the Hessian of Fx(x on x where (H&)i, a² pe
X and x refer to the decoy that maximizes and minimizes E(x; F²), respectively. See Section A7 for the proof. Theorem [ implies that the proposed saliency score is determined by the second- order Hessian (H) in the same swappable feature set. The score explicitly models the feature
Note that developed the theoretical properties by using the vanilla gradient as the base saliency method.
5

---

Under review as a conference paper at ICLR 2021
dependencies in the swappable feature set via this second-order Hessian, potentially capturing meaningful patterns such as edges, texture, etc.
In addition to enabling representation of inter- nnterrfeature dependence, Theorem ! sheds light on the robustness of the proposed saliency score against adversarial attack. To illustrate the robustness improvement of method, we introduce the following proposition. The proof of this proposition as well as in-depth analysis can be found Section A8
Proposition 1. Given an input x and the corresponding adversarial sample x, if both |x; - X; S
O2δ; and X X. < Cză, can be obtain where C2 > 0 is a bounded constant and 6 [E F); E(x, F).l then the following relation can be guaranteed. (Zx); (2x);l (E P), E(x, F).l-
Given an adversarial sample & (i.e., the perturbed x), we say a saliency method is not robust against & if the deviation of the corresponding explanation 5; [ F); - E(x, F):l (for all i E 21, dd) is large. According to the proposition above, we can easily discover that the deviation of our decoy-enhanced saliency score is always no larger than that of other saliency methods when a certain condition is satisfied. This indicates that, when the condition holds, our saliency method can guarantee stronger resistance to the adversarial perturbation. To ensure the satisfication
of conditions |x; x < C2o; and &; X S Czō,, we can further introduce the corresponding condition as constraint to Eqn. In the following section, without further clarification, the saliency scores used in our evaluation are all derived with this constraint imposed.
4 EXPERIMENTS
To evaluate the effectiveness of our proposed method, we perform extensive experiments on deep learning models that target three tasks: image classification, sentiment analysis, and network intrusion detection. The performance of our approach is assessed both qualitatively and quantitatively, The results show that our proposed method identifies intuitively more coherent saliency maps than the state-of-fe oh-the-art saliency methods alone. The method also achieves quantitatively better alignment to truly important features and demonstrates stronger robustness to adversarial manipulation. The description of the datasets and experimental setup can be found in Section A10)
4.1 SALIENCY BENCHMARK
As mentioned in Section 2, we apply our decoy enhancement method to three saliency methods: vanilla gradient, SmoothGrad, and integrated gradient. Here, we applied the default setup for the integrated gradient (a zero baseline) and SmoothGrad. Section A14 shows that our method can also improve the performance of the variants of the integrated gradient/SmotthGral and GradCAM. In each case, the decoy-enhanced saliency scores are post-processed in the following way before qualitative and quantitative evaluations. To rule out the bias introduced by the saliency values and ensure fair comparison, we constructed a binary saliency map by retaining only the top- features ranked by each method. Specifically, we set the saliency value of the selected features as 1 and the rest features as O. In this section, we choose the K as the top 20 percent of all features. Note that Section A 16 shows that subtly varying K does not influence the experiment conclusions. To demonstrate that all three methods, when enhanced with decoys, still depend on the predictor, we carry out sanity check on the ImageNet dataset. The results show that our decoy lahanced-salienge methods pass the sanity check (see Section AlI for details).
4.2 PERFORMANCE IN VARIOUS APPLICATIONS
To comprehensively evaluate our proposed approach against the baselines mentioned above, focus on two criteria. First, we aim to achieve qualitative coherence of the identified saliency map. Intuitively, we prefer saliency method that highlights features that align closely with the predictions (e.g., highlights the object of interest in an image or the words indicating the sentiment of the sentence). Second, we aim to quantify the correctness of the saliency maps produced by the corresponding method. To do it, we use the fidelity metric (Dabkowski & Gal 2017) defined as:
SF(E(s F*),)
log Fc (F((xx; F9) xx)
6

---

Under review as a conference paper at ICLR 2021
(A)
wil decay of decoped
vader Oradien Gredint
via decoy deczpt Siteeree
IeSiral
InGred
Sürdd
decty
S0rie
cacoga Baronce
SCeed
(B)
Decay variefors:
eppropaien
remge spproge Coooga wi memn app sgatiin
Wittol Beary
Coroltat range aganipation
Foregrounde objects
$1-341
A.S
Fidelity
Gradient
IntGrad
SGradd
S.alard
(C)
paich stow
pwech
patch
Backerouun objects
Fidelity
10.0
5.0
2.5
0.0
26
32
54
120
25.5
512
1024
salency decoys sallency wio decoy
Number of used decoys
Figure 2: Performance evaluation on ImageNet. (A) Visualization of saliency maps on foreground and background objects. (B) Fidelity comparison of original saliency method (i.e., "Without decoys"), our method (i.e. "Decoys range aggregation"), and its alternatives: replacing the decoy generation (Eqn. 2 with constant perturbation (i.e., "Constant w/ range aggregation") or noise perturbation (i.e.. "Noise w/ range aggregation")) replacing the decoy aggregation (Eqn. 4 with mean aggregation (i.e., "Decoys w/ mean aggregation") (See Tab, Tab.A4 more statistics about the performance differ- ences between our method and the baselines). (C) Performance with regard to variant patch size and different number of decoys.
where e indicates the predicted class input x, and E(x; Fs) is the top-K -rretiiedd binary saliency map described above. E(x; F*) performs entry-wise multiplication between E(x; F*) and x, encoding the overlap between the object of interest and the concentration of the saliency map. The rationale behind this metric utilization is as follows. By viewing the saliency score of the feature as its contribution to the predicted class, a good saliency method will highlight more important features and thus give rise to higher predicted class scores and lower metric values.
4.2. PERFORMANCE ON THE IMAGENET DATASET
We applied our decoy-enhanced saliency score to randomly sampled images from the ImageNet dataset (Russakovsky et al. 2015), with a pretrained VGG16 model (Simonyan & Zisserman, 2014). See Section 12 for applicability of our method to diverse CNN architectures such as AlexNet (Krizhevsky al. 2012) and ResNet (He al., 2016). The 3 image patches are treated as swappable features in generating decoys.
syde-by-side comparison (Fig. A)) suggests that decoys consistently help to reduce noise and produce more visually coherent saliency maps. For example, the original integrated gradient method highlights the region of dog head in a scattered format, which is also revealed by the difference plot. In contrast, the decoy-enhanced integrated gradient method not only highlights the missing body but also identifies the dog head with more details such as ears, cheek, and nose (See Section A18 for more visualization examples). The visual coherence is also quantitatively supported by the saliency fidelity score.
To further evaluate the necessity of the two steps (i.e., decoy generation and aggregation) in our method, we carried out a control experiment by replacing either step with alternatives. Specifically, as alternatives to the decoy generation, we used an image in which all pixel values are either replaced with a single mean pixel value or contaminated with Gaussian white noise. Regarding the decoy aggregation,
calculated the mean saliency score as the alternative. As shown in Fig. B), our method, which incorporate both steps, reports the best performance. This validates the effectiveness of each of our designs.
Recall that the number of decoys n is decided by the patch size (P), stride, and the number of multiple masks in one decoy (m)) Here, keep stride as 1 and vary by selecting different P and
7

---

Under review as a conference paper at ICLR 2021
(A)
Grodiont w/o doddy
5F:1 1.000
(B)
Decay variatione:
Decrye so ronge segregetion
Nixiaw range aegresstioon
Unad decoya
anyetin
Witoutt
Comsant acgregation
wi aparegation
Rocomment
50:11
Intorad coogye
mocemment
aryhhng
5F:0.064
Slonad we dwayy envemment
<f anytlng
0F:0.111
500red deoops
mosermant
Fidelity
0:127
dwaops
Infired wio decay
Insd new
3P:0.5001
IntOred decoya
Incel
Im0 movee
0.000
BCrod we dooyy
moal
8F: 0458
SOrod
5P:00007
Gradient
IntGrad
SGrad
Figure 3: Evaluation results obtained from the SST dataset. (A) Visualization of saliency maps in each word, where the normalized saliency values are shown for better distinction. (B) Fidelity comparison of the original saliency method, our method, and its alternatives. Here, the alternative methods represent the practice of replacing the decoy generation (Eqn. 2 with constant perturbation or noise perturbation as well the practice of replacing the decoy aggregation (Eqn. 4 with mean aggregation (See Tab. A5 for more statistics about the performance differences).
M. Fig. C)) shows that our method achieves stable fidelity scores across the substantial variations of decoy numbers. The sensitivity test of other hyper-parameters can be found in Section A16
4.2.2 PERFORMANCE ON THE STANFORD SENTIMENT TREEBANK (SST) DATASET
We also applied our decoy-enhanced saliency score to randomly sampled sentences from the Stanford Sentiment Treebank (SST) (Russakovsky et al. 2015). We train a two-layer CNN (Kim 2014) which takes the pretrained word embeddings as input (Pennington et al.) 2014) (see A10 for experimental details). As suggested by Guan et al. (2019), ) the average saliency value of all dimensions of a word embedding is regarded the word-level saliency value. The embeddings of the words treated as swappable features when generating decoys.
As shown in Fig. AA), side-by-side comparison suggests that decoys consistently help to produce semantically more meaningful saliency maps. For example, in a sentence with negative sentiment, keywords associated with negation, such 'no" and 'not', are more highlighted by decoy-enhanced saliency methods. The semantic coherence is also quantitatively supported by the saliency fidelity (Fig. B)). We also tested the alternatives mentioned above: constant (replacing the decoy generation with the mean embedding of the whole dictionary) and noise perturbation with range aggregation, and decoy with mean aggregation. Fig. () shows that our method outperforms these alternatives.
4.3 ROBUSTNESS TO ADVERSARIAL ATTACKS
Next we investigate the robustness of our method to adversarial manipulations of images. In particular, we focus on three popular adversarial attacks (Ghorbani et al. 2017): (1) the top-k attack, which seeks decrease the scores of the top k most important features, (2) the target attack, which aims to increase the importance of a pre-specified region in the input image, and (3) the mass-center attack, which aims to spatially change the center of mass of the original saliency map. Here, we specify the bottom-right 4 x 4 region of the original image for the target attack and select k = 5000 in the top-k attack. We use the sensitivity metric (Alvarez-Men) & Jaakkola 2018) to quantify the robustness of a saliency method E to adversarial attack, defined as:
SS(E(: F x, x) [[(E(x, Fe) F9)112
||x- x||2
where & is the perturbed image of x. A small sensitivity value means that similar inputs do not lead to substantially different saliency maps.
As shown in Fig. [A)), side-by-side comparison suggests that decoys consistently yield low sensitivity scores and help to produce more visually coherent saliency maps, mitigating the impact of various adversarial attacks. More examples can be found in Section A18 The visual coherence and robustness to adversarial attacks are also quantitatively supported by Fig. B (D), As is mentioned above, we also did experiments results in Section A13 The results are consistent with those on CNNs, which confirm our method's applicability the widely-used feed-forward networks.
a MLP trained with a network intrusion dataset and show the
8

---

Under review as a conference paper at ICLR 2021
(A)
wio
deoeya ohherres
Beazy
dapop oPhenecee
(B)
Top-k Attack
Without decoy
Sensilivity
With deccys
Top Alsh
200
DC SE DNCTES
STAT
200
55 30 30-51-3 S1at
35
50 2015
yad
(C)
300
Mass Center
Without
Sensitvvtty
Attack
-1.0
selency decays- saliency decay
T
(D)
60
Target Attack
Without docoy
Maes
Sensitvity
With decoys
Center
Aeac
40
Grad
IntGrad
SGrad
Figure 4: Robustness to adversarial attacks on images. (A) Visualization of saliency maps under adversarial attacks. (B)~(D) The decoy-enhanced saliency score compared to the original saliency score under adversarial attacks, evaluated by sensitivity (See Tab. A6 for more statistics about the performance differences).
5 DISCUSSION AND CONCLUSION
In this work, we propose a method for computing, from a given saliency method, decoy-enhanced saliency scores that yield more accurate and robust saliency maps. We formulate the decoy generation as an optimization problem, applicable to diverse deep neural network architecture. We demonstrate the superior performance of method relative to three standard saliency methods, both qualitatively and quantitatively, even in the presence of various adversarial perturbations to the image. From theoretical perspective, by deriving closed-form solution, show that the proposed score can provably compensate for the limitations of existing saliency methods by reflecting the joint effects from other dependent features and maintaining robustness to adversarial perturbations.
Fig. ( shows our method can achieve a decent performance with only small number of decoys. Section A15 further shows the runtime of generating one decoy is marginal compared to existing saliency methods. This indicates that our technique can improve the existing saliency methods without introducing too much computational overhead. With the parallel computing enabled by multiple GPUs, our method can be much faster, which further decreases the overhead and esca- lates our method's 's practicability (See Section A1 for a detailed runtime analysis). Our method mainly introduce three hyperparameters: swappable feature size K, network layer f, and initial Lagrange multiplier X. In Section A16 we show that our method is insensitive to the substantial variation of hyperparameters. We generate decoys by using Eqn. 2 While there are other widely used perturbation methods (e-g., random noise, blurring, and inpainting), they are not suitable for generating decoys. First, Section 4 shows that some general pertrbations (i.e., random noise and constant perturbation) obtain worse fidelity than decoy. Second, without ensuring the swappable condition in Eqn. they cannot provide a theoretical guarantee for robustness improvement. Third, methods like blurring and inpainting are not well-defined for applications beyond computer vision.
This work points several promising directions for future research. First, (E(x; F*) o x) may be out-of onttof-distribution and thus fails our fidelity metric. We will investigate more rigorous metrics and use other benchmark datasets (e.g., BAM (Yang Kim 2019)) for evaluation. Second, a possible extension is to customize our method to recurrent neural networks and to inputs with categorical/discrete features. Third, recent work (Bansal et al. 2020 Chen et al. (019c) shows that adversarial training improve the interpretability of DNN model. is worth exploring whether our method could further enhance the quality of saliency maps derived from these adversarially retrained classifiers. A fourth promising direction could be reframing interpretability hypothesis testing and using decoys to deliver a set of salient features, subject to false discovery rate control at some pre-specified level (Burns et al.) 2019
Lu al. 2018).

---

Under review as a conference paper at ICLR 2021
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Proc. of NeurIPS, 2018.
David Alvarez-Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Proc. of NeurlPS, 2018.
Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. Towards better understanding of gradient- attribution methods for deep neural networks. In Proc. of ICLR, 2018.
Naman Bansal, Chirag Agarwal, and Anh Nguyen. Sam: The sensitivity of attribution methods to hyperparameters. arXiv preprint arXiv:2003.0875: 2020.
Rina Foygel Barber and Emmanuel Candès. Controlling the false discovery rate via knockoffs. The Annals of Statistics, 2015.
Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In Proc. of ICANN, 2016.
Collin Burns, Jesse Thomason, and Wesley Tansey. Interpreting black box models via hypothesis testing, arXiv:1904, 00045, 2019.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Proc. of S&P, 2017.
Tsuug-Haa Chan, Kui Jia, Shenghua Gao, Jiwen Lu, Zinan Zeng, and Ma. PCANet: simple deep learning baseline for image classification. IEEE Transactions on Image Processing, 2015.
Chun-Hao Chang, Elliot Creager, Anna Goldenberg, and David Duvenaud. Explaining image classifiers by counterfactual generation. In Proc. of ICLR, 2019.
Chaofan Chen, Oscar I.i, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Proc. of NeurIPS, 2019a.
Jianbo Chen, Le Song, Martin Wainwright, and Michael Jordan. Learning to explain: An information-theorrtiic perspective on model interpretation. In Proc. of ICMI,, 2018.
Jianbo Chen, Le Song, Martin Wainwright, and Michael I Jordan. L-shapley and e-shapley: Efficient model interpretation for structured data. In Proc. of ICLR, 2019b.
Jiefeng Chen, Xi Wu, Vaibhav Rastogi, Yingyu Liang, and Somesh Jha. Robust attribution regular- ization. In Proc. of NeurlPS, 2019c.
Piotr Dabkowski and Yarin Gal. Real time image saliency for black box classifiers. In Proc. of NeurlPS, 2017.
Lijie Fan, Shengjia Zhao, and Stefano Emnon. Adversarial localization network. Proc. of NeurIPS LLD Workshop, 2017.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proc. of ICCV, 2017.
Canadian Institute for Cybersecurity. Cse-cic-ids2018 on aws.
unb. wtt.://wwbbccc..iaa.
datasets 2018 htmll 2018.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks fragile. 1:Xi::770011044 2017.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. In Proc. of NeurlPS, 2019.
Yash Goyal, Ziyan WU, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. Proc. of ICMI, 2019.
10

---

Under review as a conference paper at ICLR 2021
Chaoyu Guan, Xiting Wang, Quanshi Zhang, Runjin Chen, Di He, and Xing Xie. Towards deep and unified understanding of deep neural models in nlp. In Proc. of IIML,, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. of CVPR, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In Proc. of ICLR, 2019.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. In Proc. of NeurlPS, 2019.
Kouichi Ikeno and Satoshi Hara. Maximizing invariant data perturbation with stochastic optimization. arXiv preprint arXiv: (807.05077 2018.
Yoon Kim. Convolutional neural networks for sentence classification. Proc. of EMNLP,, 2014.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (Un) reliability of saliency methods. 1rXiv:1711.00880 2017.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. Proc. of ICML, 2017.
Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. On the accuracy of influence functions for measuring group effects. In Proc. of NeurlPS, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In Proc. of NeurlPS, 2012.
Alexander Levine, Sahil Singla, and Soheil Feizi. Certifiably robust interpretation in deep learning. arXiv preprint 1905v 12t.:/9955.115, 2019.
Zachary C Lipton. The mythos of model interpretability. wrXiv:1606.03499 2016.
Yang Lu, Yingying Fan, Jinchi Lv, and William Stafford Noble. DeepPINK: reproducible feature selection in deep neural networks. In Proc. of NewrIPS, 2018.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proc. of NeurlPS, 2017.
Seyee-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal adversarial perturbations. In Proc. of CVPR, 2017.
Weili Nie, Yang Zhang, and Ankit Patel. A theoretical explanation for perplexing behaviors of backpropagation--based visualizations. In Proc. of ICML, 2018.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proc. of EMNLP, 2014.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should trust you?: predictions of any classifier. In Proc. of KDD, 2016.
Explaining the
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
Andrew M Saxe, Pang Wei Koh, Zhenghao Chen, Maneesh Bhand, Bipin Suresh, and Andrew Y Ng. On random weights and unsupervised feature learning. In Proc. of ICML, 2011.
Karl Schulz, Leon Sixt, Federico Tombari, and Tim Landgraf. Restricting the flow: Information bottlenecks for attribution. 2020.
Ramprasaath R Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localiza- tion. arXiv: /6/ 1.04500 2016.
11

---

Under review as a conference paper at ICLR 2021
Iman Sharafaldin, Arash Habibi Lashkari, and Ali A Ghorbani. Toward generating intrusion detection dataset and intrusion traffic characterization. In Prof. of ICISSP, 2018.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proc. of ICML, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv: 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv:1312.6034 2013.
Sahil Singla, Eric Wallace, Shi Feng, and Soheil Feizi. Understanding impacts of high-order loss approximations and features in deep learning interpretation. arXiv:/902.000) 2019.
Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified bp attributions fail. In Proc. of ICML, 2020.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv: 1707770007.222 2017.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014.
Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 2020.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proc. of ICMI, 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. 1rX6v9331..619, 2013.
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural
(in the brain). In Proc. of NeurIPS, 2019.
Mengjiao Yang and Been Kim. Benchmarking Attribution Methods with Relative Feature Importance. CoRR, abs/1907. 0000009701 2019.
Chih- Yeh, Joon Kim, Ian En-Hsu Yen, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. In Proc. of NeurIPS, 2018.
Roozbeh Yousefzadeh and Dianne P O'Leary. Interpreting neural networks using flip points. arXiv preprint arXiv:1903.08788 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proc. of ECCV, 2014.
Konrad Zomn, Krzysztof J Geras, and Kyunghyun Cho. Classifier-annostic saliency map extraction. In Proceedings of AAAI, 2019.
A6 IMPLEMENTATION DETAILS
The optimization function proposed to generate decoys is non-differentiable and very difficult to solve; hence, we instead solve an alternate formulation with the help of the following tricks. First, we introduce Lagrange multiplier À O and augment the first constraint in the optimization function as penalty in the objective function. This will rule out the hyper-parameter in Eqn. 2 Second, use projected gradient descent during the optimization eliminate the mask constraint (i.e., (x x) o (1 M) = 0). Specifically, after each standard gradient descent step, we enforce the change-of-variabe eliminate the feature value constraint (i.e., x E [Xmin: Xmax] Instead of directly optimizing x, we first normalize [0, and introduce & satisfying &; (tanh(x;) + 1), for all E (1,2, ... d).
x X M + x (1 M). Third,
trick (Carlini & Wagner 2017) to
12

---

Under review as a conference paper at ICLR 2021
Because tanh(&;) [-1, 1] implies X; [0, 1], any solution to x is naturally valid. It should be noted that other transformations for this third step are also possible but were not explored in this paper. Putting these ideas together, we minimize the following objective function:
minimizes - -1 (Lan))()) + 1) - x)
x)) x)+
+ A.
Fe(2
(lanh(x) + +1) - -xx))((x))
(10)
where À 0 is initialized small and repeatedly doubled until the optimization succeeds. Because the Too norm is not fully differentiable, we adopt the approximation trick introduced by Carlini & Wagner (2017) and solve the following formulation:
max(( (tanh(() + xx) s, 0) A | (tanh(&) 1)) - FX(x) 7)
minimizes
where 7 > O. In this paper, we follow the selection strategy proposed in Carlini & Wagner (2017) and initialize of 0.95 and repeat; otherwise, we terminate the optimization. After obtaining X, we compute x and map back to the original feature value range 'Mminn Xmax]- Note that Eqn. can be efficiently solved by any first-order optimization method without introducing too much computational overhead. In practice, the average run time solving 62.3% shorter than the fastest, vanilla gradient method.
1. After each iteration, if the second
zero, then we reduce by factor
A7 PROOF OF THEOREM
Before proving Theorem 1, we first state and prove the following lemma.
Lemma 1. Consider an input and its decoy X. generated by replacing the original features with swappable features K., |K] = K. The partial derivative of (x()) w.rd. 8; for i E K.
(v2F(x)))
Σ (Rx (H SC.
(12)
2
REK
Proof. The second-order Taylor expansion the predicted Fx(x) for target class around x is as follows:
F*(x) R F* 7 A
(13)
where A x x. By definition of the decoys in Section 2.2 (i.e., Fx(x) = FX))), the following equation holds:
V& ~
(14)
From the above equation, we can see that, for linear model, the linearity zeroes out the gradient of the decoys, causing our method to output zero saliency scores for all input features. We clarified in Section 3.4 that method is mainly defined for non-linear complicated models.
Given swappable patch of size K x starting from position i1, then = [0, ..., Xi, - -Ki,. Xix Xig O,.., 0j. As such, we have
Vx F F*
(x)²
Σ
(Vx
(VxF((x))
-x;,,
iEK
(15)
A A
Σ
[x: x,) Σ
(Hx),,(x - 5xx)
IEK
kEK
Plugging Eqn. 115 into Eqn.
14 have
> (()=F(()); +
Σ
(H&),,(x {x))((x; ==)=00 00
(16)
2
Then derive
(V)F+(8))+
1
Σ
(x+ - xxx)
SC,
2
kek:
(17)
1
(V)F((x));
(8t - xx)x))(H22i..
SC.
2
kek
13

---

Under review as a conference paper at ICLR 2021
First, derive |x; x4 is bounded by 2max(XXmax |Xmia)). We also have [i+4 x,+*| in that we can always find a small perturbation to each feature in x such that [[F((X) Fx(x)) loo S c. In addition, both gradient and Hessian are bounded by some Lipschitz con- stant (Szegedy et al. 2013). [ As a result, we can always find a constant C, such that C 2 ( + Deee dng (8ig Xx2 (801 8nn)
For the case K 1, we have
(x-=x)
(V*F(())) = (Hx),, x,).
Now we prove Theorem from Section 3.5
Consider CNN with L hidden blocks, with each layer containing convolutional layer with a filter of size v8e x /8g and max pooling layer with pooling size /8t /se. The input to this CNN x E Rd, unrolled from a vd x vd matrix. Similarly, we also unroll each convolutional filter into ge R" where ge indexed (ge), for j Tg. Here, Te corresponds to the index shift in matrix form from the top-left to bottom-right element. The output of the network is the probability vector p e RC generated by the softmax function, where C/ is the total number of classes. Such network can be represented as
m; pool(relu(g) m for l = 1,2,3,...,
0 = W m2 + bL41,
(18)
p = softmax(o),
where relu(-) and pool(.) indicate the ReLU and pooling operators, my gd is the output of the block e (mg = x), and (ge m,-1) E Rdc-1 represents a convolutional operation on that block.
Consider input and its decoy x, generated by swapping features in A. For each feature i K. we have the following theorem for the decoy-enhancee saliency score Z₁:
Theorem 1. the aforementioned setting, Zg bounded by
1
Zi
(x
(Hx)k
SC1.
(19)
2
kEK
Proof. The gradient of P. with respect to x can be written as follows, using the denominator layout notation of the derivative of vector:
0m: do 0p.
VxPa
II
(20)
dm,-1 dm, do (=1
where
do 0mp
WL+1,
(21)
and
(Pe P²) if = 8pc
Do.'
8pc
PoPa otherwise.
(22)
Do
Then we can write 0ps d0
as follows:
Öp. P.c.
(23)
where P. corresponds to the c-th column P and P diag(p)-pp We then define B, as B, E Rd xxde In the following, we compute B,,
8m,
8m
First, we can have
0(mr);
if - -n-n Te, and argmax, argmax wei+J, (ge m
Trelo ge-me 1) (m)))
(24)
otherwise,
Dreelu gr Dreelu
FFollowing other works that also utilized Lipschitz continuity to analyze DNNs (Szegdyy et al. 2013
Ghorbani al. 2017) we assume that locally continuous around x, for & 1,2,
14

---

Under review as a conference paper at ICLR 2021
where represents the center of the pooling patch in relu(ge m which results in (mz).. Then we can compute
O(relug ( mmc-1))m (at)n (ge)n-i if n - E Jt.
a(my-1);
(25)
(ucbugg-mmm1))) = otherwise,
0(m;-1);
where (az)n {rrelu(g my-1111 (on-1))) )) 2 0}. we change the activation function to either sigmoid
or tanh, then (ar)n in Eqn. 25 will be replaced with the derivative of either function. For the sigmoid activation function (zz), the derivative is azz)(1 - (z)), with a range of [0, l. For the tanh
activation function tanh(zr) the derivative
tanh(x)² with a range of [0, 1]. We conclude that the derivative of both sigmoid and tanh bounded by value no larger than 1.
Combining Eqn. with 25 we have
(Be)
8im; (g)) in i Je, - n Jt, and n "rgmax,.ee++ (gr me-1)n', a(ma);
0(mz);
(Be)
otherwise.
a[m 17;
(26
For simplicity, rewrite the non-zero condition E Te. Plugging Bg, 1, L, into Eqn. we can obtain the partial derivative VxP.-
20
Further, we compute each element in the Hessian matrix H.j as follows:
Hij = Vx. (x, P.) aT Be), WL+1P.
Ox;
(27)
1 II
JP. 0x,
di
L
O..
BB), W
Σ
IIII
[B)
(WL+1)e
Ox.
E=1
(PL=1
E=1
and
OP c/c
(1 (1-2p.)Vx,PP =
(28)
0x;
P + Vx,e otherwise.
Now compute
(TT/_) Be)jnz
Be) (Bı) II
(BL)
(29)
t=1
E=2
where
(Bi), B2 = 10, ..., Caa (u2)
Σ (a1)a 0], niE.
(30)
and where Cnz (92)ng-2 Enj 8n-1. Here, we redefine J1 as the set of indices such that (B1)jn, # for nı Jt. As such, we can compute (B1),.
11,_2 B as
L-1 (Bi),
L-2
II 2
B. = 10...,
(ae) O].
(31)
8m meEe
Plugging Eqn. into Eqn.
29 we have
L-1
B jn.. (Bi,, II B, (Ct)ar (ar.ms.
Σ Σ (at)ne
(32)
{=1 nEER
Plugging Eqn. 32 into Eqn.
27 we have
oP
H
C2
E E=1 my
Σ
(ae)
(33)
where C, is a linear combination of g1, gr. W Wi+1 which bounded. Hi equals the multiplication of two components-thee the summation of neurons activated by and gradient
15

---

Under review as a conference paper at ICLR 2021
BP.,.
Given the total number of neurons in CNN a constant (denoted by CT), we have 0x:
0 S (Ei=1 Emcef, (ae)))) I Cr. Then, we have (Hx))y S Cr|C, 2xr Since the deriva- tives of both sigmoid and tanh are no larger than this inequality also applies to the network with these two functions as the activation function. Similarly, for the Hessian (Hx)iş of a decoy
x, we also have (H&)ss & Cr|C, 1. Given the inequality of (Hx)ı and (H&)is. we can obtain that (Hx)ay - (Hx)al I 2Cymax(1C, ds, I, (R;)) where aF
8x; is given by Equ. 28
Recalling that P. is within [o, I], the gradient OP. is bounded by some Lipschitz constant (Szegedy Tx
et al. 2013), we can obtain that af.c is bounded by some constant. Finally, we can derive that 2x.
(Hx) hi (Hx)i)l I Cc. where Cc represents the upper bound
Now, we derive the decoy-enhancee saliency score Z, for x,, given a population of saliency scores = ; E(x;; F).. ... _Ex="; F):}. Let x x E x........ denotes the decoy which maximizes and minimize ( F)i, respectively. According to Lemma 1, the partial derivative Vx, Pe has the following relationship
E.
(VxF
Σ KEK
(H SC, (
(34)
2
Then, derive
Σ ( xx) -C }); S Σ
(x - (H) (35)
2
kEK
kE
x - (Hxx )i. - C - F (8 ));
Σ
{ - xx) CC, (36)
2
2
kEK
kER
Then, we have
Z = (V2+ (F=*F((())) - (V2- F* });
( xxx) )(, Σ (xx -xx -Hx) )1, +0
1
{x xx)((Hx)
+Cc)
Σ (xT - )(. Cc. 2C
(37)
kE
kEK
1
1
(x x (Hx)i,
Cc Σ
[x1t -xx) +
2
2
kek
kEK
And
Z, = (v*+ F* 1); - (Vx- F });
( Xx (H )i, Σ (x-- -xx) 2C
kE
KEK
{x xx) (H) -Cc)
Σ (x xx)((HH )i. + Cc) +C
(38)
kek
kek
IV
x
(Hx)i
Co Σ
1
(x -x,) - 20,
2
KEK
kEK
Combining Eqn.
37
with Eqn. 38 have
Z;
X
(H
SC..
(39)
2
kek
Note that this inequality cannot be directly obtained by the Lipschitz inequality, hecause the gradient may be continuous.
16

---

Under review as a conference paper at ICLR 2021
Recall that (xx² - -x,) is bounded by a upper-bound, can obtain that there exist constant C1. such that Ci 2 Cc Σ x xx + 2C. Note that this upper bound is data specifie, and leave the exploration on its tightness as part of future works.
A8 PROOF OF PROPOSITION
Proposition 1. Given an input x and its corresponding adversarial sample & if both |x [21 2 C2o; and &4 & & C2ds can obtain where C2 > is a bounded constant and = [ F); E(x; F),l, then the following relation can guaranteed.
(2x): (2x)al < (E(8, P): F)).
(40)
Proof. Recall the goal of the attack against saliency maps is to subtly perturb input sample such that the added perturbation does not change the output of the classifier (Ghorbani et al. 2017) but force saliency method to output less meaningful saliency map (i.e., highlighting features that irrelevant to the classifier prediction). To achieve this goal, when generating an adversarial sample & from the given input x. an attacker needs to impose the following constraint || x||00 we have an adversarial sample X satisfies this constraint. Then, we can assume (x - x); é. where |é, < e, for 1, 2, .... In addition. we can compute saliency maps E(x, F) and E F) for x and x by using an existing saliency method. Given both saliency maps, we can further compute the difference between E(x, F) and E(x, F) as
Suppose
d (E P) F(x, V Vx) = (H& x)); Σ (Hx) j=1
(41)
Based on the Eqn. in Section 3.3 when generating the decoys x, we ensure the classifier's predictions for those decoys the difference between the hidden representations of x and x. As is discussed in Section A7 to preserve the same prediction e for and x, one has to ensure [F(()) [*xxx] is bounded. This implies the difference between X and x is bounded within e. Here, e, represents the maximum difference between X, and x; at the 2th dimension. As mentioned above, the adversarial sample does not change the classifier's prediction. Therefore, we could imply é; S C,, for = 1,2,....4 d.
same that of the x. In this work, achieve this by bounding
Now, suppose we obtain a set of decoys for and have their corresponding saliency maps. i.e.. {E(*'; F)., E(x;; F).. ... E(x?"; F):)]. Let x E [x' x ... x" denote the decoys which maximize E F); and let x denote the decoys which minimize E(x; F);. Similarly, we can also have the corresponding decoys X and x for the adversarial sample & well as their corresponding saliency maps. With both the decoys and saliency maps for the input sample x and its adversarial sample &, we compute the difference between (Zx); and (Zx); as
(Zx); (Zx);
Ex²- F); F F)::) - (F((*t, F); ; F(x F):))
(Ha - -x)); - -xx): ((Hx(** x)); (H+( - x))i)
(42)
Hxx) (55 x (x) x3))
Forr simplicity, we use the vanilla gradient method. The conclusion can be generalized to the other saliency methods considered in this paper
17

---

Under review as a conference paper at ICLR 2021
To guarantee improvement in robustness against the adversarial perturbation, we have to ensure that (Zz)s (Zx) - [ F) Exx, F)) I 0, for i = 1, 2..... d. That is,
Σ (Hx)) (8)
H H2 6
50,
(43)
Hx )is (8) (x- - x3)
(x-- - x7))
Σ H )iye
-
As is discussed in Section (Hx)isl S Cc. With this, we can have
d Σ (Hx)s) k (3k] - - (x} - x7))
(Hx) )ol (3x) - x2) - (x) x5,)
(44)
d
Co
[x] + - x2) (x} - x5))
==
By plugging Eqn. 44 into Eqn. 43 we conclude that as long as - -x5 -x)- x7)
S
Cld Σ (Hx) Jije our method could guarantee to improve the robustness against the adversarial
perturbations. Let & EE F); E(x, F-)a-- If we can ensure that |X; X;| Tे. O and
x; x _ dü. we can have x3 x S _d_ di and
X - 2
< TC 5, Thus, the
aforementioned condition can be satisfied, i.e., (5 - -x x5)
S d By setting
C2 Id.d could obtain the robustness conditions in Proposition 1.
A9 COROLLARY
Consider multilayer perceptron with L fully-connected hidden layers and a decoy swappable size K x 1. The input of this MLP is x E Rd. For each hidden layer, we use the ReLU activation function. Similar the CNN mentioned above, the output of this CNN p E PC The network can be represented as:
m, relu(W)) m/ + by), For = 1,3,,...,
W m, + b
(45)
p softmax(o).
where We Rdd- for é (1..... + 1) represents the weights of the neural network, and b, E &d represents the biases, where do d and d = C. m E R2 is the output of each hidden layer, with m = x and o E R,C the logits. The entry- wise softmax operator for target class de
defined as P. Ee_ e for E (1.2.....C0
Corollary 1. For the above MLP, Z, is also bounded by:
Z, <
(81+h - X(++) (Hx))) + Cz.
(46)
kEK
Proof. Based on the proof of Theorem 1, the gradient of Pe with respect to can be written as follows
1
V xxP.
I
B.W L+1 P.c.
(47)
1=1
18

---

Under review as a conference paper at ICLR 2021
where B, 8mg By Rdd-ixdd P.c. is also defined as P diag(p) pp".. In the following,
0m
we compute By, First, we can compute (B1)ss )is, in which
(Bi) a(mi)y 0(Wfx bi)s
(mm)) = (Wi))s(a1)s O(W x bi)j
(48)
8x;
ox,
where (a1)) 1[(W[x[[]]]) b1)1 2 0}. Similar, we can also compute (Bz)aj, for = 2, 3, ....
(Br) = (Wa):)(ar)s
(49)
where (ae)) = 1(( x bb); 2 0}.
Then, compute the each element in the Hessian matrix Haj- Specifically, based Eqn. 27
have
dr.
OP.
H =
Σ
II
Bz)jno (W D+nne
(50)
0x:
mL 1=1
where UP... 0x;
is the same with Equ. 28
Now, compute (T Bi)jnz
L
L-1
II Be) (B1), II B (BL)
(51)
where (Bz)g- = (W1) (W1)ja, (0) )d, and
di
di
(B1), B2 = [(02)
Σ m1=1
(Cz)am (a))) (02)ds m=
Σ
(C2)da. (a1)mi,
(52)
where (C2) = (W2)ni.n (i)jmm For simplicity, we can rewrite Edj= (C2)ma (a1)m
(C2)n; Ed { Then, we have
(Bi), B2 ((22)
Σ
(C2)dd (a2)dz
Σ (a))m]]
(53)
m1=1
m1==1
As such, we can compute (Bi);. 1
L-1
Bs
1-- de
Σ de
(B j II B. ((r--))
Σ
Σ
(CI- (C (ar-1)dd
(ae)me)
E=
(=1 neml
(54)
Plugging Eqn. 54 into Eqn. 23
we have
L-1
L-1 de
II Br) = (Bi),
II
(,
=Cr)) (aL)ms
Σ
(at)me-
(55)
1=1
t=2
1=1 me=1
Finally, we can obtain that
de
L-1 de
O
H =
Σ
(at)m
(ar)ne (W (WL+)me.
Ox,
n==1
1=1 n==1
(56)
OP.
C
(at)me
Ox;
2=1 =1
where Cj is a linear combination of the elements in (W1),. W W
Note that the Hessian derived from the MLP has similar form with the Hessian derived from the CNN in Eqn. [33] i.e., the summation of neurons activated by x multiplying the gradient. Here, the summation of neurons activated by is again bounded by the total number of neurons in the
19

---

Under review as a conference paper at ICLR 2021
network. The gradient 8P is bounded by Lipschitz constant. Similarly, we also have the following dx. inequality for (Hx)y and (Hx)g i.e., (Hx)is i) - (Hx),gl S Cm-
Similar Theorem 1.let x* x minimize ((x;; F),, respectively. Based Eqn. 34 Eqn. 39 have
{x x², .... x²n) denotes the decoy which maximizes and
2
(Hx)R,
S CO2.
(57)
2
k
AEK
C2 Cas E ( xx) 2C. Slightly different for CNN, MLP sometimes is used to process the imput that does not have a strong local dependency. In this case, we can set the swappable path
size K Then, Eqn. (7 can reformulated as Z (x] x, (Hx)),) S C2. As we observe from this equation, our proposed saliency score is still able to compensate for the gradient saturation problem.
Table A1: The hhper-paramenn parameter choices of the proposed method different target models.
À
patch_ size (P)
stride
ImageNet AlexNet
10000
ImageNet VGG16
10000
ImageNet ResNet
10000
SST CNN
2
10000
IDS MLP
2
10000
1
A10 DATASETS AND EXPERIMENT SETUP
In this section, we introduce the datasets used in our experiments and the neural network trained on each dataset, followed by our choices of seper-pprameters when explaining each model.
ImageNet. We randomly select a subset of samples from the ImageNet validation set, which can be downloaded from the following link: http:: most widely used preprocessing method for the selected images. Specifically, for each image, we resized it to 227 X 227, converted it BGR format, and subtract the mean value of each channel [103.93] 113,,99, 116.779 779, 123.68] from the image. Rather than training our own networks, we downloaded a pretrained VGG16 model, AlexNet model, and ResNet v1 555 model from the fol- lowing link: http::::///tttttut slim and http://www toronto. wwtp:/uueehoy//aeeeeu alexnet/ We applied our proposed method to explain the predictions of these networks on the selected samples.
mage-net. org/] We adopt the
SST. We downloaded the Stanford Sentiment Treebank (SSTI) from the following link: https 7/github. cww.coo/coa/ao//oo com/harve wwwaaaaarree 1 p wtw.raaaanll/eettcooooooo
masttr/daaaa.... The data is spited into a training set of 76, 961 samples and testing set of 1, 8281 samples. We used a pre- trained glove embedding represent each word in the sentences (sample). The embedding of each word is a vector of 100 dimensions. The pretrained embedding matrix can be downloaded from the following link: http::/nnil trained two-layer CNN with the embeddings as inputs. The model achieves about 80% accuracy on the testing set. The preprocessed testing data and the pretrained model can be downloaded from the following link: https:: / /tttttnnlrrrrr/... com/ homm/y9mm 99ooqi61 We run our explanation method on the pretrained model with the testing samples.
stanford.. edu/ / http://wwdvvss//eeeee httpe//aaaaaaat////eee////////// glove.
zip We
Network intrusion detection (IDS). We use a subset of CSE-CIC-IDS2018 dataset (Sharafaldin et al., 2018 for Cybersecurity) 2018), a network intrusion dataset contains the benign network traffic traces and malicious traces generated by three types of attacks: Denial Service (DoS)-Hulk, SSH- BruteForce, and Infiltration. The training set contains 88, 661 samples and the testing set has 22, 165 samples. Each sample represented as a vector of 83 dimensions, where each feature represents the statisties of network traffic flows (e.g.. Number of packets, Number of bytes, Length of packets, etc). The features are normalized within [0, 1] by using the scikit- -learn MinMax function. We trained two-layer MI.P P classify whether an input is benign traffic attack (intrusion).
20

---

Under review as a conference paper at ICLR 2021
Cascading randomization botlom
2
Terrirr
Gradient
decoys
Intärad
decoys
SGrad
decoys
Figure A5: Cascading randomization on VGG16 network. The figure shows the original saliency map (first column) for the terrier. Progression from left to right corresponds complete randomization of the pretrained VGGI6 network weights from the top layer to the bottom layer. Note that, here, we followed the visualization method Adebayo al. 2018 to show the saliency maps, i.e., 0-1 normalization. The row labels share the same meanings as the column labels in Fig.
Goslent Intient dectys
Sorad deugy
Figure A6: Structural similarity index (SSIM) for Cascading Randomization VGG16 network. Note that the legends have the same meaning as the column labels in Fig.
The model reaches 99% accuracy on the testing set. After training the model, we randomly sampled subset of 000 testing samples and used method to derive explanations from the model predictions of samples in this subset. The dataset, model, and the descriptions of each feature can be found in
com cow.ynooojy.oll
Hyper-parameter choices. The hyper-parameter choices of the proposed method on three datasets are shown in Table Al In the table, is the index of the layer within the target model that is selected generate the decoy images. The Lagrange multiplier controls the weight of F| The patch_ size and stride control the size and the stride step of each decoy patch. 7 is introduced by Eqn. Section A6 Note that we set the swappable patch size of SST and IDS data as 1, because their features may not have a strong local correlation. It should also be noted that we selected the swappable patch size of ImageNet data as the widely used convolutional kernel size and stride size 1. We set the number of patches (masks) in each decoy m 100 for ImageNet, 1 for SST and IDS. When generating adversarial attack images, we applied the code relcased by the corresponding work (Ghorbani et al. 2017) and followed their default setup in our implementation.
(x)
A11 SANITY CHECK FOR DECOY-ENHANCED SALIENCY MAPS
As suggested by Adebayo et al. (2018), any valid saliency methods should pass the sanity check in the sense that the saliency method should be dependent on the learned parameters of the predictive model, instead of edge or other generic feature detectors. We performed the model parameter randomization test (Adebayo et al. 2018) on the ImageNet dataset by comparing the output of the proposed saliency method on pretrained VGG16 network with the output of the proposed saliency method on weight-randomized VGG16 network. If the proposed saliency method indeed depends on the learned parameters of the model, it is expected that the outputs between the two cases differ substantially.
Following the cascading randomization strategy (Adebayo et al. 2018), the weights of pretrained VGGI6 network are randomized from the top to bottom layers in cascading fashion. This cascading randomization procedure is designed to destroy the learned weights successively. As illustrated in Fig. A5 the cascading randomization destroys the decoy-enhanced saliency maps combined with three existing saliency methods, qualitatively. The conclusion is also supported by quantitative comparison measured by the structural similarity index (SSIM), shown in Fig. A6
21

---

Under review as a conference paper at ICLR 2021
All de
cecon she with decoy
oe decrel Biberesee Alt dewdeccyaaa desnn oee
el decio del Caizen dreenace with decty do decogt Beraane Alla decie dested oPPAWNA
(a) Saliency maps generated on AlexNet.
(b) Saliency maps generated on ResNet.
Figure A7: Visualization of saliency maps under different CNN architectures. Here, the column
labels are same as those in Fig. 2 The difference figures share the same colorbar as those in Fig. 2 Table A2: Quantitative comparisorrof our method and baselines on the network intrusion dataset.
report the means and standard errors of the fidelity scores.
Fidelity (SF)
Salinecy method
Without deney
Decoys with range
Constant with range
Noise with range
Decoys with mean
Gradient
1.80 0.39
1.64
0.40
1.68 + 0.40
1.78 0.43
2.04 + 0.40
IntegratedGrad
1.68 0.39
1.57 0.40
1.68 0.44
1.79 0.43
2.19 =
SmoothGrad
1,59 0.39
1.57 0.40
1.74 0.44
1,73 0.44
1.87 0.45
A12 APPLICABILITY TO OTHER CNN ARCHITECTURES
In addition the VGG16 model, generated saliency maps for AlexNet (Krizhevsky al. 2012) and ResNet (He et al. 2016) trained from the ImageNet dataset. We visualize their saliency maps in Fig. A7 We observe that our method consistently outperforms the baseline methods, both quantitatively and qualitatively. Together with the results in Section 4 these results suggest that we can apply our decoy-enhanced saliency methods to various feed-forward network architectures and expect consistent performance.
A13 PERFORMANCES ON THE NETWORK INTRUSION DATASET.
Rather than visualizing the saliency scores through heatmaps, we apply the following to compare the saliency scores obtained by different methods qualitatively. We ranked the features based on their saliency scores and compared the ranking obtained by the existing methods with that obtained by our decoy-enhanced method. "Minimum size of packet in forward direction", "Minimum length of packet", "Minimum time between two packets sent in the forward direction" are ranked higher by our methods than the baselines. These features could capture the differences between benign and malicious traffics. This is because attackers usually tend to rapidly send small packages to discover the backdoors in the victim network system, while the benign users may send much larger that
packages with a longer interval between two packages. On the contrary, features that useful for intrusion detection (e.g., timestamp, Download and upload ratio) are wrongly pinpointed by the existing method. However, our methods correctly assign lower importance these features. Table A2 shows the fidelity comparisons of different saliency methods. We can observe that our decoys-enhanced methods outperform the original saliency methods. These results show that our method could pinpoint more accurate features and achieve a higher fidelity than baselines. We also evaluated three alternatives used in Section 4 constant perturbation with range aggregation, noise perturbation with range aggregation, decoys generation with mean aggregation. The results in Table A2 are consistant with those Fig. 2 and Fig. 3 i.e., our method outperforms these baselines. In summary, the results this dataset align with those the other datasets. This confirms our method's applicability to multilayer perceptrons.
A14 DECOYS ON OTHER BASELINES.
In Section 4
we evaluated our methods on three ofate-of-the-art saliency methods. Recent re- search (Sturmfels al. 2020 Hooker et al. 2019) suggests some variants that improve the perfor-
22

---

Under review as a conference paper at ICLR 2021
Goldal
Cilere
Oftericce
SP U4F
Figure A8: Visualization of saliency maps obtained by original saliency methods and our decoy- enhanced versions. "ExpGrad" refers to Expected Gradient, "SGradRage" stands for Smoothgrad with range aggregation, and "IntUniform" represents integrated gradient with uniform baseline. The difference figures share the same colorbar those in Fig. 2
Fidelity
10
5
Without decoy
With deroys
ExpGrad
Varšrad
SGradRange IntUniform
GradCAM
Figure A9: Fidelity comparision of saliency maps obtained by original saliency methods and our decoy-enhanced versions. "ExpGrad" refers to Expected Gradient, "SGradRage" stands for Smooth- grad with range aggregation, and "IntUniform" represents integrated gradient with uniform baseline (See Tab. A7 for more statistics about the performance differences).
mance of these baseline methods. Here, by using ImageNet data, evaluate whether decoy method could further improve these variants and another widely used saliency method. Specifically, we consider two variants of the integrated gradient: integrated gradient with uniform baseline (Sturm- Tels et al.] 2020) and Expected Gradient (Sturmfels et al. 2020); two variants of the SmoothGrad:
VarGrad (Hooker et al.) 2019) and Smoothgrad with range aggregation; and one existing saliency method: Grad-CAM (Selvaraju et al. 2016). For the variants of the integrated gradient and Smooth- Grad, we kept the number samples the same as the original version and used the default number suggested by existing works 25 (See https: /tpp:://ht htg.t.......... com/ / PAIR- / sa 1 Lency). We will investigate whether increasing the sample numbers improve the existing saliency methods' fidelity and robustness in future work.
Fig. A8 and Fig. A9 shows the qualitatively and quantitatively comparison of each method with/without decoys. As is depicted in Fig. A8 our method helps knock off the noises and im- prove the visual quality of the saliency maps. Fig. A9 further demonstrates the advantage of our method in explanation fidelity. Together with the results Section 4 they demonstrate the generaliz- ability of our technique to different saliency methods. Note that our method only imposes a minor improvement Grad-CAN both qualitatively and quantitatively. As part of future work, will explore how to customize our method for Grad-CAM and investigate the effectiveness of applying our technique to more saliency methods.
A15 RUNTIME OF DECOY GENERATIONS
To evaluate the computational cost of our decoy generations, we carried out the run time comparison between optimizing one decoy and calculating three types of saliency methods, repeated 500 times with respect to different patch masks. As illustrated in Fig. A10al on average, optimizing one decoy is 62.3% faster than the fastest vanilla gradient- optimization is even less expensive, in relative sense.
saliency method. For other methods, the
As is mentioned in Section 3.4] our decoy generation, and the saliency map computation can be run parallelly in batch mode. In the optimal case, where have enough resources to compute each
23

---

Under review as a conference paper at ICLR 2021
10a
Running time seconds
Rurniing poccnnss
10?-
Without deroy
With decoys
104
Grad
IntGrad
SGrad
Secoy gereration
Indord
Sorad
(a) Run time optimize decoy and caleu- late saliency map with the existing methods.
(b) Run time to compute saliency maps with and without optimizing one decoy.
Figure A10: Run time of decoy generation. The comparison is conducted in the same CPU/GPU to ensure fairness. Note that "Grad", "IntGrad", and "SGrad" stands for the vanilla gradient, the integrated gradient, and the SmoothGrad, respectively.
A-10
Am100
1-1000
>=10000
1-100000
1-10
1=100
1-1000
A-10000
1=100000
A-10
X-100
1-1000
1=10000
10100000
2:30
SF
2:30
SP
22:
SP.2
SP-22:00
BFF 0E7
3F
U.32
3FF
0:38
3F. 0:30
SF. 0:30
BF 0.30
dF way
SF: 0:34
BF
Grodient wi decays
InlegratedGrad wl decays
SmoothErad wi decoys
Figure A11: Visualization of saliency maps optimized using different initial A.
decoy saliency map (i.e., F((x;; F)) parallelly, the overall runtime of generating one decoy-enhanced saliency map is the total time of generating decoy and computing saliency map using the existing methods. Fig. A shows the comparisons between the optimal decoy-enhanced saliency map generation time and the original saliency map generation time across three saliency methods. As we can observe from the figure, our method introduces negligible computational overhead over the existing methods. Taking a step back, when the users have limited resources for running the decoy-enhanced saliencies in a fully parallel fashion. As is shown in Fig. 2 our method is not sensitive the variations the number of generated decoys. More specifically, Fig. 2 shows that can obtain decent performance by only solving 16 decoys on the ImageNet dataset. In the worst-case scenario where a user cannot run decoy generation in parallel, our method's 's computational overhead over the baselines is 24X for the Gradient approach, 16 X for the Integrated gradient method, and 16 for the SmoothGrad method. In most cases, where users could afford partial parallel computing, this overhead will be decreased linearly with the available computational resources. For example, if user has GPUs, the overhead will drop to 6X for Gradient, 4X for Integrated gradient, and 4X for SmoothGrad. We argue that for ensemble method, this overhead is acceptable. Besides, saliency generation is much lighter weight than training deep neural networks. Even with 4X~6X overheads, the time of computing saliency maps is still much less than network training. In addition, our method can be even faster on more powerful machines, which escalates the practicality of our method.
A16 HYPER-PARAMETER SENSITIVITY
We also conduct experiments on the VGG16 to understand the impact of hyper-parameter choices on the performance of our optimization-based decoy generation method. Specifically, we focus on the choice of three hyper-paraman parameters: network layer é, initial Lagrange multiplier X, and patch size.
Accordingly, first varied the value of l for VGG16 and compared the differences of the generated decoy saliencies from the three aforementioned saliency methods. In particular, we set it to range from the first convolutional layer to the last pooling layer and demonstrate the generated decoy saliencies in Fig. A99 Note that according to our design, only the convolutional layers and the pooling layers can be used to generate decoy images. For each saliency method, Fig. A19 demonstrates that the decoy saliencies generated from different layers for the same image are of similar qualities. Fig. A19 also shows the mean and standard derivation the SF scores for each saliency method. These quantitative results also support the conclusion that approach is not sensitive the layer. This likely because, as previous research has shown (Chan et al.) 2015 Saxe et al., 2011). the final classification results of DNN not highly related to the hidden representations. As result,
24

---

Under review as a conference paper at ICLR 2021
P23
P=5
P=7
P=8
P= 11
P=3
P
P=7
P
P=11
P=3
P=5
P=7
P=9
Ps 11
2.42.4
SF: 1,81
SF:
SF-235
SF:291
6.:4.34
SF: 0.30
SE: 0.29
SF. 5F.0.38
SF:0.32
SF: 0.34
SF. 0.40
SF: 0.38
ST: 0.25
SF:035
Gradient decoys
Indegratedrrad wl denoys
SmoothGrad
Figure A12: Visualization of saliency maps optimized using different patch size P.
15
Flaciity
10
Witheut decay
Decoys wi range aggregation
Constant vo range agregettion
Grad
IntGrad
SGrad
Noke range aggregation
(a) Fidelity comparison when selecting top 10% features on ImageNet.
20
Fidelty
Witheal aggregattinn
range aggregation
IntGrad
SGrad
Nosve sange agoregation Decoys wi meon aggregation
(b) Fidelity comparison when selecting top 40% features on ImageNet.
Figure A13: Fidelity comparison of our methods and baselines under different choices of (See Tab. A8 and A9 for more statistics about the performance differences).
generating decoy saliencies for the same sample with the same label from different layers should yield similar results.
We also varied the initial Lagrange multiplier À to be {10*, 102 100, 104, 10a} and compared the differences of the generated decoy saliencies. Fig. A1l depicts the quantitative and qualitative comparison results. As shown in the figure, the different choices initial all produce similar saliency maps, indicating negligible influence upon method.
Then, we fixed m and increased the patch size to be {3, 5, 7, 9, 11} and showed the generated decoy saliencies in Fig. A12 The results show that varying the patch size within certain range only imposes negligible influence upon our method.
mention that decoy masks generated by sliding the swappable patch across given input. With a given constant stride 1, the number of sliding windows is equal to (Vd-P - +1+ 1)2. our implementation, to enable batch computing, we introduce m, which controls
Recall that in Section 3.4
the number of sliding windows each decoy. Then, the number decoys is (Vd- -Vd- P 1)2
Fig. shows the results of fixing m 100 and varying P. In Fig. [C), substantially varied both P and m and showed that our method is insensitive to the variations in the number of decoys n. Note that the box bars with the same color in Fig. 2 drawn by fixing P and varying m. Their slight difference indicates the robustness of our method in the variations of m.
The results in Fig. (C) All and A12 indicate can expect obtain stable decoy saliencies when the syper-parameted are subtly varied. This a critical characteristic because users do not need overly worry about setting very precise peree-pprsaeters obtain a desired saliency map.
In addition to the parammtrrssetrrs introduced by our methods, we also test the sensitivity of fidelity evaluation results the choice of K in the topK normalization. Specifically, we varied K select top 10% and 40% important features and redrawn the fidelity/sensitivity comparison figures in Fig. B)/ Fig. (D), The results in Fig. A13
A14 and A15 are aligned with those in Fig and
A17 OBJECT LOCALIZATION
We compare method and the vanilla gradient the object localization task (Dabkowski & Gal 2017 Fong & Vedaldi 2017). where the model was trained with the class label only without access any localization data. We carried out Imagenet ILSV 14 localization task (Russakovsky et al.)
25

---

Under review as a conference paper at ICLR 2021
200
whis dereo
(a) Top-k attack.
(b) Mass center attack.
(c) Target attack.
Figure A 1:: Sensitivity comparison when selecting top 10% features on ImageNet (See Tab. 110 for more statistics about the performance differences).
whot dedte
Wittout cadly destus
sce
(a) Top-k attack.
(b) Mass center attack.
(c) Target attack.
Figure A15: Sensitivity comparison when selecting top 40% features on ImageNet (See Tab. All for more statistics about the performance differences).
Table A3: ImageNet localization accuracy on VGG16 network using different thresholding strategies.
Accuracy
Value thresholding (0.25)
Energy thresholding (0.25)
Mean thresholding (0.25)
Gradient
0,662
0.715
0.662
Gradient w/ decoys
0.722
0.723
0.665
2015) which contains 50K ImageNet validation images with annotated bounding boxes as ground truth. For each image, we first calculated the gradient- saliency maps with and without using decoys, based on the pretrained model. Following the preprocessing steps suggested by Dabkowski & Gal (2017): Fong & Vedaldi (2017). we then obtained a bounding box from each calculated saliency maps based on certain thresholds. Specifically, we investigated three thresholding strategies suggested by Fong & Vedaldi (2017): value thresholding. energy thresholding, and mean thresholding. Following the evaluation protocol of Dabkowski & Gal (2017); Fong Vedaldi (2017), we then computed the Intersect over Union (IoU) of the extracted box and the ground truth. If an IoU is greater than 0.5, the corresponding box is marked as correct. (33]e/A3shhw that decoy-enhanced saliency maps achieve higher accuracy than those of the vanilla gradient.
A18 ADDITIONAL EXPERIMENTAL RESULTS
Fig. A17 Fig. A16 and Fig. A18 provide more results of the fidelity and robustness evaluation. These results are consistent with those shown in the Section 4
A19 STATISTICS OF THE PERFORMANCE DIFFERENCES
In section Section A14 and Section A16 we varied the choice of K in the top-K normalizations, compared our method with each baseline approach, and showed the fidelity/sensitivity of each approach in the box-plots. To demonstrate the advantage of our method over the basclines, we further compared the fidelity/sensitivity difference between our method and the corresponding baseline approach. To be more specific, given two sets of fidelity/sensitivity scores (Sour and 8base) obtained from method and baseline approach respectively, we first computed their difference, i.e., diff = Sour Shase- Then, we conducted a statistical measure on the values of dif f by computing the mean, the standard error, and the p-value of the paired t-test. For the paired t-test, our null hypothesis is Ho E[di fl 2 O. This indicates that, if the value of p is larger than a threshold, we cannot reject this null hypothesis, and have to conclude that our method cannot outperform the corresponding baseline approach. As we present in Table A4 /Talle A11 the overall experiment results align with those shown in the box plots, demonstrating the superiority of our method over the baselines.
26

---

Under review as a conference paper at ICLR 2021
Gradient w/o decoy Ihis one of polanski bes
films
SF: 0.457
Gradient w/ decoys
this is one of polanski best
rilms
SF: 0.038
IntGrad w/o decoy
this one of polanski best
films
SF: 0.457
IntGrad w/ decoys this one of polanski
best
films
SF: 0.038
SGrad w/o decoy this one of polanski best
0.073 films
SGrad w/ decoys this one of polanski best films
SF: 0.062
Gradient w/o decoy
movement no yuks not
of anything
SF: 1.003
Gradient decoys
No movement
yuks not much of anything
SF: 0.075
IntGrad w/o decoy
No movement
no yuks no much of anything SF: 1.003
IntGrad decoys
yuks
much of anything SF: 0.084
SGrad w/o decoy
movement
yuks not much of anything
SF: 0.111
SGrad w/ decoys
No
movement
no yuks
much of anything
SF: 0.105
Gradient wlo decoy
movies
bright sheen
SF: 0.457
Gradient w/ decoys
most
movies have
bright sheen
SF: 0.049
IntGrad w/o decoy
most
new movies have bright
sheen
SF: 0.457
IntGrad w/ decoys
most new
movies have
bright
sheen
SF: 0.049
SGrad w/o decoy
most movies have
bright sheen
SF: 0.069
SGrad w/ decoys
most new movies have
bright
sheen
SF: 0.031
Gradient w/o decoy
singular character study
perfect
SF: 0.457
Gradient w/ decoys
as
singular character
study
perfect
SF: 0.039
IntGrad w/o decoy
as
singular
character
study
perfect
IntGrad w/ decoys
singular
character
study
perfect
SF: 0.039
SGrad w/o decoy
singular character
study
perfect
SF: 0.076
SGrad w/ decoys as
singular character
study perfect
SF: 0.056
Gradient w/o decoy
well made and often ovely
depiction the mysteries of friendship
SF: 0.457
Gradient w/ decoys
well made and often lovely
depiction of the mysteries of hendshig
0.013
IntGrad w/o decoy
well made and often lovey
depiction
mysteries friendship
SF: 0.457
IntGrad w/ decoys
well made and often lovely
depiction of the mysteries of mendship
SF: 0.007
SGrad w/o decoy
well made
and often lovely
depiction of the mysteries of friendship
SF: 0.006
SGrad w/ decoys
made and
often
loelyy depiction
mysteries friendship
SF: 0.006
Visualization of saliency maps the sentences in SST dataset. The labels and Figure A16:
colorbar are the same with those in Fig. A).
wi Ehow
recoys Bavnne
rifeence
ceoovt alference
Ail diccy cROWE cilewress
Serme
Figure A17: Visualization of saliency maps the images in ImageNet dataset. The column labels and colorbar are the same with those in Fig. A).
27

---

Under review as a conference paper at ICLR 2021
Gracket
Grnderi Godert
Infond
Infied
Intind
Göed
50ml
Geadier
Grodant
Gracler
Spar
DOmal
deage
Top Alick
Mise
Mazs
Alick
Center
RU--S 50 91.31
2010 10:37
TA. IOS DATY
SEA4
Togal
Top-
Topak
Mowx Ceeter
Conde
Abxck
Figure A18: Visualization of saliency maps on the perturbed images generated by using three attacks in VGG16. The column labels are the same with those in Fig. A
Bottom layers to top layers
cone11
comvil .2
Mamepooi
comv2
comvt_2
Manpord
comi
Maxporol
comed
comia
comet
Marpooa 4
comvo
comes
convi
Mexpool
Shield Gradient
w/ decoys
IntGrad
w/ decoys
SGrad
w/ decoys
(a) The mean and standard derivation of SF score for gradient, integrated gradient and SmoothGrad are: (10.23, 0.29), (10.37, 0.84), (9.34, 0.51).
Bottom layers to top layers
canvi 1
Masppod
3
Marpoot 2
Marpood
crme 1
comd .2
Maxppood
cronvs 1
comdo 2
Marppotl
com-l
com2
com'
coatdd
camdd
Bustard Gradient w/ decoys
IntGrad
w/ decoys
SGrad
w/ decoys
(b) The mean and standard derivation of SF score for gradient, integrated gradient and SmoothGrad are: (0.07, 0.02), (0.01, 0.003), (0.06, 0.007).
Bottom layers to top layers
comvt
cantr
Maspool
came
Marpool
camiaa
comg
camoa
Maxpoool
come
camel
Maxpasl
comod
came
come .3
Marpod
Temir Gradient w/ decoys
IntGrad
w/ decoys
SGrad
w/ decoys
(c) The mean and standard derivation of S score for gradient, integrated gradient and SmoothGrad (2.15, 0.50). (0.97, 0.56). (0.19. 0.06).
Figure A19: Demonstrations of decoy-enhanced saliency maps generated from each convolutional and pooling layer in VGG16.
28

---

Under review as a conference paper at ICLR 2021
Table A4: Mean, standard error, and p-value of the difference Fig. B).
Without decny
Constant with range
Noise with range
Decnys with mean
Salinecy method
Mcan=Sid
P-valuc
Scd
P-value
Scan=Sid
P-value
Mcd
P-value
Gradient
1.61=3.24
0.014
-1.26+22.29
0.009
-0.51=1.74
0.093
-1.80 2.15
<0.001
IntegratedGirad
1.14+3.82
0.087
-0.71+3411
0.170
-0.06+3.03
0.440
-2.53+ 2.25
<0.001
SmoothGrad
1..121..23
0.068
-0,44:1..27
0.058
-0.79-1.22
0.003
-1.00 2.65
0.002
Table A5: Mean, standard error, and P-value of the difference in Fig. B
Salinecy method
Without decoy
Constant with range
Noise with range
Decoys with mean
Mcan+Sid
P-value
Std
P-value
Mean+-Srd
P-value
Mca
Pvvauue
Gradient
0.2970.57
< <0.001
+0.09+0.09
0.921
0.003+0.099
0.912
-0.1740.51
< <0.001
IntegratedGrad
0..1200.56
<.0001
0.001=0.07
0,744
-0,20:00.4
<0.001
-0.0940.52
< 000001
SmoothGrad
0..02=0552
0.043
-0.02=#0.52
0.043
-0.02+0.51
0.029
0.006=+0.15
0.959
Table A6: Mean, standard error, and P-value of the difference in Fig. B (D))
Gradient
Integrated gradieni
SmoothCrrad
Attack
Mean
P-value
Sed
P-value
Sid
P-value
Top-k
-23552 1: 57.02
0.008
-3.89
2.47
<0.001
-2.32 1: 21.00
0,349
Mass Center
+30.3+ 25.48
<0.001
-6.06
+ 4.56
<0.001
-2.75
1.85
< 4..001
Target
-7.66 + 3,03
< 0.001
4.77 # 1.29
<0.001
-2.81
+ 2.88
0.002
Table A7: Mean, standard error, and P-value of the difference Fig. A9
ExpGrad
VarGrad
SGradKange
Intnniom
GraCCAM
Mean Std
Vvllue
Mean Sid
P-yauue
Mean Std
Pvaluee
Mean: Std
P-value
Mean
Std
V-vuue
-,26 + 4.11
0.009
-0.53 + 118
0.001
-0.00 = 131
0.026
-2.8 - 3.18
0.001
-0.08
0.23
0.121
Table A8: Mean, standard error, and p-value of the difference in Fig. A13a
Without decoy
Constant with range
Noisc with range
Decoys with
Salinecy method
StaniStd
P-value
Sta
P-value
Mta
P-value
Std
P-value
Gradient
0.8712213
0.034
-0.3011.21
0.126
-0.29-1011
0.100
-0.911 1.96
0.021
IntegratedGrad
1.3942.29
0.005
11.31+1.64
0.001
-0.79=1.50
0.011
-2.024 1.91
0.001
SmoothGral
0.7910.97
<0.001
-1.16.11.17
<00001
-0.881100 1,01
0.007
-1.76.1 1,69
< 0.001
Table A9: Mean, standard error, and P-value of the difference in Fig.
A13b
Wilhout decny
Constant with range
Noise with range
Decnys with mean
Salinecy method
Mcan=Sid
P-valuc
Sta
P-value
Mca
P-value
Std
P-value
Gradient
3.88
<0.001
00.3713.74
0.320
-1.27 2,51
0.014
-3.331 2.75
< 0.001
IntegratedGraad
-2.31+3.70
0.004
-0.2142.99
0.374
-1.87=3.08
0.005
-4.33+ 3.41
< 0.001
SmoothGrad
-0.9411.21
0.001
-0.9411.09
< 0.001
-0.48-0.70
0.002
-2.774 2.92
< 0.001
Table A10: Mean, standard error, and P-value of the difference in Fig. A14
Grudient
Integrated gradient
SmoothGrau
Attack
Sid
P-yalue
McaniSId
P-value
Mcan:ESd
P-value
Top-k
-8.04
49.69
0.285
-1,58
1.75
0.003
-1.34
16.30
0.386
Mass Center
144.48 15.68
0.003
-2.98
2.41
0.001
-1.87
1.26
0.001
Target
-3,95 1. 2.42
0.001
-2.30
1.06
<.0001
-1.81 1: 1.96
0.003
Table A11: Mean, standard error, and P-value of the difference in Fig.
A15
Gradient
Integrated gradient
GmoothGrad
Attack
Mid
P-value
Mid
P-value
Midn
P-value
Top-k
-2..44 1 76.08
0.052
-8.37 4.26
<.0001
-2.81
23.61
0.338
Mass Center
-36.542 58.27
< 0.001
10.288431.8
0.133
-2.51
1.49
<0.001
Tangel
-13.09 3,60
< 0.000
-8.99 12.08
< 0.001
-3.12 1: 3.69
0.005
29


Reviews:
[{"review_id": "rhRJFHJTDKQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper introduces a technique to improve current gradient-based attribution methods by 1. generating perturbed data, i.e. decoys, that maintain the same feature maps as the original data and 2. an aggregation strategy of the saliency maps across the decoys. They provide theoretical as well as empirical support for their method. This paper show a great improvement over saliency maps and integrated gradients with a smaller gain for smoothgrad. This is well written and an interesting technique that could add to the repertoire of interpretability methods that exist. \n\nComments:\n* One questions that arose while reading: why is there a significant boost with range aggregation compared to mean aggregation? What is the intuition here? \n* Visually, smoothgrad (without decoys) seems to make a big improvement over saliency maps and integrated gradients — almost as much as decoy-enchanced saliency maps. Smoothgrad perturbations are drawn from Gaussian noise, not optimized, with a mean aggregation. The performance gain of smoothgrad seems to be better than the decoys with mean aggregation (Fig. 1). Certainly the decoys with range aggregation improve Grad (saliency maps). Does it make sense to try smoothgrad with the range aggregation strategy to see if it improves its performance?  This experiment can help isolate which attributes are more noteworthy, the decoy generation method itself or the aggregation strategy.  Also, since smoothgrad can be applied to integrated gradients, a similar comparison should be made for integrated gradients as well. \n* The fidelity metric is a way to quantitatively compare the attribution methods, but it is unclear (at least visually) how sensitive the score is. For instance, in Fig. 1, the picture of the alps has a SF score of 7.57 for SGrad_Decoys, while the the volcano image has an SF score of 0.72. Moreover, the intgrad_decoys for the volcano has an SF score of 0.36, but it seems to be capturing the base of the mountain in addition to the sky. Are there certain aspects of the saliency maps that boosts/biases this fidelity score? A brief mention of the limitations of this SF score would be helpful to put this in context for a reader. \n* The comparison between grad and intgrad with deocys is visually very clear. However, the difference between sgrad and sgrad+decoys is much less pronouced. Perhaps a difference plot would be better at revealing what the decoy methods are able to better capture. \n* The robustness to adversarial attacks on images is not as convincing as the other aspects of the story. In Ghorbani et al. 2017, they demonstrate that adversarial attacks can maintain the same predictions but lead to very different attribution maps.  Each attack that is explored here (Top-k, mass center, and target) largely leads to the same feature importance maps as the original image without any adversarial attacks (see Fig. 3a). This may be why the sensitivity (while consistently lower) is very similar to the feature importance maps without decoys.\n* On a smaller note, this paper mentions that the three hyperparameters are swappable feature size, network layer, and initial lagrange multiplier. They never mention lagrange multiplier in the main text — it is first introduced in A6.  It should probably be mentioned in the main text for clarity.  \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "h459oV-g-eF", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of work: a new saliency map method is presented, along with empirical work showing that it beats SoTA by some metrics.\n\nRecommendation: I lean toward accepting this paper.\n\nReasoning: \n\nStrengths: the paper represents an advance in state of the art, based on a variety of quantitative metrics. The authors have come up with an interesting new idea, and done a fair amount of work to back up their claims. It would be useful to have this paper in the literature.\n\nWeak points: Interpretability methods ultimately stand or fall based on how useful they are to humans, not quantitative metrics. To my eye, the results for images and sentences aren't noticeably better than the alternatives. Furthermore, I found the writing fairly confusing--I needed to read the \"decoy\" definition several times.\n\nAreas for improvement:\n* I'd like to see a better description of the basic decoy method, maybe with some figures walking us through the construction. I would also like to see some motivation for what feels like a fairly complex technique, beyond Theorem 1.\n* Theorem 1 is suggestive, but I'd like to see a critical analysis of how it applies in the real world. Do we have any idea how big the constant $C_1$ is, for instance? Furthermore, $F^c$ is only piecewise differentiable, and on very small pieces in practice. (Also, if it weren't for the softmax layer, the function would be piecewise linear--it's not clear to me what the Hessian tells us about anything that happens before that final layer.) Given this, it's not obvious that the Hessian at a given point captures significant information about robust inter-feature interactions.\n* I find Figure 3 confusing (e.g., there are undefined abbreviations) and visually it's not clear that the decoy method is better. Maybe having one large diagram, with callout arrows and annotations, would help make it clear what the new method is adding.\n* I did not check the math in the appendix (if I understand correctly, this is not necessary in a review, but I wanted to make that fact explicit.)", "extra_scores": {}, "extra_text": {}}, {"review_id": "60d_PkVDOr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(I have updated my review to raise my reviewer score by two points after discussion with the authors; this is still a preliminary evaluation as I have not discussed the paper with the other reviewers)\n\n--\n\nSUMMARY OF METHOD\n\nThe paper discusses an approach to create \"decoys\", which are slightly altered versions of the input that preserve the activations of an intermediate layer. The proposed \"decoy enhancement\" consists of applying a saliency method to calculate importance scores for the decoys, and then subtracting the maximum saliency across decoys from the minimum saliency. The authors benchmark their method on several datasets and argue that their saliency maps are superior according to the fidelity metric (Dabkowski & Gal, 2017). They also demonstrate that the decoy-enhanced maps are more robust to adversarial attacks.\n\nSTRENGTHS\n\n- This approach of enhancing saliency maps is very unique compared to other methods in the literature. The generation of decoys could be useful in other contexts (e.g. to understand which sets of inputs can compensate for or \"buffer\" each other).\n- The authors have benchmarked their method on several types of data - images, the Stanford sentiment treebank, and network intrusion detection.\n- The authors have included several valuable experimental analyses, such as making sure the method is robust to the choice of hyperparameters, performing sanity checks to make sure the saliency maps respond to cascading randomization (a sanity check from Adebayo et al.), investigating different baselines for integrated gradients, comparing the decoys to a constant perturbation, etc.\n- To me, the authors have made the case that the maps are more robust to adversarial attacks (caveat: I am not an expert on adversarial attacks)\n\nWEAKNESSES\n\n(1) The biggest weakness of this method would be the runtime to compute the saliency maps on the decoys. The authors generate $2n$ decoys, where $n$ is the number of unique masks. The number of masks is determined by the patch size $K$, and the authors write (in section A16 in the supplement) \"given a constant stride 1, the number of decoys $n$ is unique to the patch size $K$. Specifically, $n$ is inversely proportional to $K$\". The claim that $n$ is inversely proportional to $K$ is a little confusing given that the stride is 1 - I would expect the number of unique masks to be $(\\text{image_len} - $K$ + 1)^2$, but the phrase \"inversely proportional\" implies that $n$ is proportional to $1/K$. Assuming that the authors meant  $(\\text{image_len} - $K$ + 1)^2$ (which is consistent with a stride of 1) - this could potentially be a very large number of decoys per image, given that the patch sizes $K$ are fairly small ({3, 5, 7, 9, 11} according to section A16 in the supplement) - e.g. the ImageNet images are resized to 227 x 227 (as per section A10 in the supplement). The authors say in section A15 of the supplement that the \"saliency map computation can be run parallelly in batch mode\", but regular saliency maps can also be computed in batches on a GPU, so I am confused how the authors claim that their method  \"introduces about 2X ~ 5X overhead over the existing saliency methods\" when it seems like the number of decoys per image is much larger than 5. Can the authors explicitly state how many decoys are used for particular tasks, and (if the number is larger than 5) how they reconcile this with the claim of \"2X - 5X overhead\"? It would additionally be good to clarify whether the interpretation improvements from having the decoys are greater than what one would achieve by instead investing the equivalent computation time in other techniques (e.g. more samples for SmoothGrad or ExpectedGradients - have SmoothGrad and ExpectedGradients saturated in the number of samples?).\n\n(2) I have a set of concerns pertaining to the evaluation using the fidelity metric. The authors define a normalized saliency score as follows: \"First, we follow the existing methods (Simonyan et al., 2013) and compute the absolute saliency scores...to avoid outlier features with extremely high saliency values leading to almost zero saliency scores for the other features, we then winsorized outlier saliency values to a relatively high value (the 95th percentile)...before linearly scaling to the range [0,1]\". They then define the fidelity metric as $- \\log \\frac{F^c (E(x; F^c) \\odot x) }{F^c (\\bar{E(x; F^c}) \\odot x) }$, where $c$ is the predicted class of the input. The authors write \"by viewing the saliency score of the feature as its contribution to the predicted class, a good saliency method will weight important features more highly than less important ones and thus give rise to higher predicted class scores and lower metric values. Note that we subtract the mean saliency $\\bar{E(x; F^c)}$ to eliminate the influence of bias in $E(x; F)$ and exclude trivial cases such as $E(x; F^c) = 1$\". Here are my concerns:\n\n2a) In Figure 1, the fidelity scores are predominantly all positive - this implies that $F^c (E(x; F^c)$ was *lower* than $F^c (\\bar{E(x; F^c}) \\odot x)$, which is concerning - the average saliency map across all images was better than a saliency map that was generated specifically for the example at hand? To me, this suggests that the fidelity metric is not working as intended on image data. My guess is that there is a lot of *high-frequency noise* in the saliency maps, which is causing $E(x; F^c) \\odot x$ to be out-of-distribution, and that is the reason $F^c(E(x; F^c) \\odot x)$ is so poor. According to this hypothesis, I suspect the authors may find that simply *smoothing* the saliency map scores would achieve better results via the fidelity metric. The authors should thus include simple smoothing as a baseline. I note that the fidelity metric values are predominantly negative for the SST dataset (as shown in Figure 2c), which a sign that the fidelity metric makes sense there; however, in Table A2 (pertaining to the network intrusion dataset), the fidelity scores are once again predominantly positive (also Grad performs better than IntGrad and SGrad, which is unexpected). \n\n2b) The authors acknowledge that extreme values can result in near-zero importance for many features in the normalized saliency maps and propose clipping as a way to address this; however, even with clipping, the problem could still persist; if the 95th percentile of a saliency method is more on the extreme side, then the resulting masks would have a **lower L1 norm after the linear scaling normalization** - thus, saliency maps that had less extreme variation in the scores would get an unfair advantage in the evaluation (because they would be able to retain a larger portion of the original image after applying the mask $E(x; F^c)$). It is not clear to me that subtracting the mask created by the mean saliency would be enough to correct for this potential advantage; the rigorous way to correct for it would be to retain something like the top $k$ positions as ranked by a particular saliency method (with sign information included for the methods that provide them), or to find some other way of ensuring that the L1 norm of the masks were comparable.\n\n2c) As alluded to in 2b, the approach of taking the absolute value of the saliency scores would handicap methods where the sign information is valuable. Given that the authors are specifically trying to identify regions that are important for the target class, they should retain the sign information for the methods that provide them (in conjunction with switching to a rank-based masking strategy described above). I recognize that Simonyan et al., 2013 discarded the sign information, but they were not trying to optimize the fidelity metric. I also recognize that the proposed decoy-enhanced saliency scores are always positive, but this is a shortcoming of the decoy-enhanced saliency scores (in that they don't distinguish regions that are contributing positively to a class from regions that are contributing negatively to the class); I don't think that's a good enough reason to handicap methods that are providing valuable information in the sign of the saliency.\n\n2d) (this point is lower priority) Again regarding the benchmarking on images: it has frequently been observed that pixel-level attribution methods do not perform very well on image data (e.g. they tend to be outperformed by Grad-CAM, which computes the attributions at a higher conv layer). Accordingly, to make a compelling case that the proposed method can be helpful in practice, the authors should show that the decoy enhancement helps even when the saliency maps are computed at some higher convolutional layer (rather than computed at the pixel level). Alternatively, they could add Grad-CAM (or a related method - see [1]) as a baseline. A suggestion regarding benchmarking on image data: the authors could consider using the BAM dataset (https://github.com/google-research-datasets/bam) to sidestep the issues with the fidelity metric altogether.\n\n(3) In section A7 of the supplement, the authors write \"from the above equation, we can see that, for a linear model, the linearity zeroes out the gradients of the decoys, causing the ill-defined behavior of our method. This does not dilute the significance of our method because linear models are self-explainable and do not need extra post-hoc explanation\". I think this is a bit misleading (either that, or it could use more clarification); the equation in question is $\\nabla_{\\tilde{\\boldsymbol{x} }} F^c(\\tilde{\\boldsymbol{x}})^T \\Delta \\approx - \\frac{1}{2} \\Delta^T \\boldsymbol{H}_{ \\tilde{\\boldsymbol{x} } } \\Delta$, where $\\Delta = \\boldsymbol{x} - \\tilde{ \\boldsymbol{x} }$ and $ \\tilde{ \\boldsymbol{x} }$ denotes the decoy of $x$. To me, this equation just seems to convey that decoys are constructed such that the change in the output due to first-order effects (the left-hand side) is approximately balanced by the change in the output caused by second-order effects (the right-hand side). If the right-hand side were zero (as is the case in a linear model), then that would just mean the first-order effects would have to be such that they canceled out - why does this make the method's behavior \"ill-defined\"? However, more to the point: it seems clear that the decoy-construction process will only perturb a feature if they effect of this perturbation can be cancelled out by perturbing other features. But *a feature can be important even if the effect of perturbing the feature is not easily cancelled out by perturbing other features* - particularly given that the decoys are constructed in order to preserve the representation at some intermediate hidden layer. Also, regarding the assertion that \"linear models are self-explainable and do not need extra post hoc explanation\" - a more complex model is still capable of learning linear effects for some of its input features, and it would not be obvious that the complex model has learned these linear effects without some post-hoc explanation.\n\n(4) Since these decoy-enhanced saliency maps look at $\\max(\\tilde{E}) - \\min(\\tilde{E})$, where $\\tilde{E}$ is a population of saliency scores, it seems appropriate to benchmark against a method that also looks at the variation across a population of saliency scores - e.g. the VarGrad method, which was one of the top-performing methods in https://arxiv.org/abs/1806.10758. Also, while I appreciate that the authors added a baseline of the \"constant perturbation\" decoy, I would be interested to see a similar baseline that consists of random perturbations that are of comparable magnitude to the perturbations made by decoys.\n\nMINOR\n\n- Why do the authors write (in the introduction, when discussing their proposed method): \"By design, this score naturally offsets the impact of gradient saturation\" - if the gradients on a particular set of inputs is zero, would the decoy-generation process perturb those inputs? My understanding is no, since the decoy generation proceeds via gradient descent. In what sense is the saturation problem addressed?\n- In table A3, how many samples were used for ExpectedGradients?\n- It would be good if the authors clarified what $\\delta_i$ is in Proposition 1 in the main text itself, without the reader having to refer to the supplement\n- There are a few typos/grammatical errors - \"it exhibits ill-defined on linear models\" on page 4, \"can obtain where\" in Proposition 1, \"obtain, we can further\" in the subsequent paragraph.\n- I did not read the supplementary proof of the adversarial robustness in detail (I am also not an expert on these adversarial attacks), so one high-level question I have for the authors is whether we expect *any* method that is based on an ensemble of saliency maps to be more robust to adversarial perturbations compared to the original (non-ensembled) saliency maps, and if so, is the proposed decoy-enhanced approach *more* robust than what would be provided by devoting the equivalent computational resources to drawing more samples for an ensemble (say with a method like VarGrad)?\n\nSUMMARY OF RATING\n\nI think the concept of using decoys to enhance saliency scores is unique, and the authors have clearly invested considerable effort in the work. However, the concerns regarding the benchmarking (point (2) under weaknesses) mean that I hesitate to fully believe the empirical improvements (the results of course look aesthetically pleasing to the human eye, but we as a field have learned our lesson that aesthetically pleasing results can be deceiving). If the authors can make the benchmarking more compelling, I would be willing to revise my score towards acceptance. Overall, my current feeling is that the decoy generation process proposed by the authors may turn out to have good uses (e.g. in identifying sets of inputs that can compensate for each other), but I'm not yet convinced that the application presented here of enhancing saliency maps is the right fit for the method (particularly when coupled with the runtime concerns in point (1) and the potential failure cases in point (3) where important features might not be highlighted).\n\n[1] https://openaccess.thecvf.com/content_CVPR_2020/papers/Rebuffi_There_and_Back_Again_Revisiting_Backpropagation_Saliency_Methods_CVPR_2020_paper.pdf\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a novel approach to producing saliency maps for interpreting deep neural networks.  In general this paper seems quite close to borderline, although on the positive side, with some low confidence reviews.  The reviewers felt that the proposed approach could be useful to the community and they seemed to feel that the qualitative results in the experiments demonstrated convincing saliency maps.  There were some concerns, however, about the quantitative experiments as the reviewers (e.g. AnonReviewer2) found that while the mean results seemed better, it wasn't clear if they were statistically significant.  Naturally, the qualitative experiments are highly subjective and there was disagreement between reviewers whether the proposed approach did indeed produce better saliency maps than existing approaches such as smoothgrad.  One reviewer indicated that they found it difficult to follow the paper and to understand the decoy concept given the writing.  During discussion AnonReviewer2 updated their score (and very thorough review) by 2 points to indicate a weak preference toward accept.  None of the reviewers argued particularly strongly for acceptance and "championed" the paper.

The low confidence, slightly above borderline reviews seem to suggest that the reviewers thought the paper was above the bar but were reluctant to argue strongly for acceptance.  The method seemed like it could be useful to them but they weren't clearly convinced that it set a new state-of-the-art given the quantitative and qualitative empirical results.  
Decision: Reject