Certifiably Robust Interpretation in Deep Learning
Authors: Alexander Levine
Sahil Singla
Soheil Feizi
Venue:  (2020)
Keywords: deep learning interpretation
robustness certificates
adversarial examples

Abstract: Deep learning interpretation is essential to explain the reasoning behind model predictions. Understanding the robustness of interpretation methods is important especially in sensitive domains such as medical applications since interpretation results are often used in downstream tasks. Although gradient-based saliency maps are popular methods for deep learning interpretation, recent works show that they can be vulnerable to adversarial attacks. In this paper, we address this problem and provide a certifiable defense method for deep learning interpretation. We show that a sparsified version of the popular SmoothGrad method, which computes the average saliency maps over random perturbations of the input, is certifiably robust against adversarial perturbations. We obtain this result by extending recent bounds for certifiably robust smooth classifiers to the interpretation setting. Experiments on ImageNet samples validate our theory.

Under review as a conference paper at ICLR 2020
CERTIFIABLY ROBUST INTERPRETATION IN DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep learning interpretation is essential to explain the reasoning behind model predictions. Understanding the robustness of interpretation methods is important especially in sensitive domains such as medical applications since interpretation results are often used in downstream tasks. Although gradient-base based saliency maps are popular methods for deep learning interpretation, recent works show that they can be vulnerable to adversarial attacks. In this paper, we address this problem and provide a certifiable defense method for deep learning interpretation. We show that sparsified version the popular SmoothGrad method, which com- putes the average saliency maps over random perturbations of the input, is certi- fiably robust against adversarial perturbations. We obtain this result by extending recent bounds for certifiably robust smooth classifiers to the interpretation setting. Experiments on ImageNet samples validate our theory.
INTRODUCTION
1.1 MOTIVATION
The growing use of deep learning wide range of highly-sensitive applications. such au- tonomous driving, medicine, finance, and even the legal system (Teichmann et al., 2018; Quellec al., 2017; Fischer Krauss, 2018; Nissan, 2017), raises concerns about human trust in machine learning systems. Interpretations, which explain why certain predictions are made, are critical for establishing this trust between users and the machine learning system. Moreover, the interpreta- tion results themselves are often used for downstream tasks. For example, gradient-based saliency maps have been used in medical applications: Quellec et al. (2017) uses gradient-based interpreta- tion techniques to highlight signs of retinal disease in retinal fundus photographs (see Figure 1-b.) Similarly, BenTaieb et al. (2016) uses gradient-bart based saliency map as part of a pipeline to auto- matically analyse histopathology slides of colon tumours. Techniques like these also used in medical research to annotate datasets (Lahiani et al., 2019). Outside of the medical field, gradient- based interpretations can be used in general-purposs image segmentation tasks (Hong al., 2015).
However, Ghorbani et al. (2019) has shown that several gradient-based interpretation methods are sensitive adversarial examples, obtained by adding a small perturbation to the input image. These adversarial examples maintain the original class label while greatly distorting the saliency map (Fig- ure 1-a: see 'Gradient" interpretation method). Although adversarial attacks and defenses on image classification have been studied extensively in recent years (Szegedy et al., 2014; Uesato et al., 2018; Goodfellow et al., 2015; Athalye et al., 80;;; Buckman et al., 2018; Kurakin et al., 2017: Paper- not & McDaniel, 2016; Carlini & Wagner, 2017; Madry et al., 2018), less attention has been paid to effectively defending deep learning interpretation against adversarial examples (Lee et al., 2019; Etmann et al., 2019). This is partially due to the difficulty of protecting high-dimensional saliency maps compared to defending a class label, as well as to the lack ground truth for interpretation. Interestingly, we have observed that the standard adversarial training used for the classification bustness does not lead to robust interpretations (see Section 4). This calls for better understanding the robustness of interpretation methods and developing defenses with performance guarantees.

---

Under review as a conference paper at ICLR 2020
(a)
Interpretation Method
Interpretation of Unperturbed Image
Adversarial Attack on Interpretation
Interpretation after Adversarial Attack
Image
Gradient:
Proposed
Method:
(b)
(c)
Scaled Smoothhred
Sparsified Gre
Cradient-Basee interpretation
Certificate
0.5
Showing Evidence of Disease
0.
8D
Medical Image
(Retinal Fundus)
0.0
Sparsification Parameter
Figure 1: (a) An illustration of the sensitivity of gradient-based saliency maps adversarial perturbation of an image from CIFAR-10. Our proposed method, Sparsified SmoothGrad (relaxed variant shown here; adversarial attack than plain gradient saliency maps. (b) A medical use of gradient-basee based saliency maps. Figures are borrowed from Quellee al. (2017). Signs of lesions indicative of diabetic retinopathy are automatically highlighted.
Section 2.3), produces saliency maps which considerably are robust to
(c) A comparison of robustness certificate values Reen of Sparsified SmoothGrad vs. scaled SmoothGrad, on ImageNet samples.
1.2 PROPOSED APPROACH AND MAIN CONTRIBUTIONS
In the last couple of years, several approaches have been proposed for interpreting neural network outputs (Simonyan et al., 2014; Sundararajan et al., 2017; Alvarez-Melis & Jaakkola, 2018; Adebayo al., 2018; Huang et al., 2017). Specifically, Simonyan absolute value of the gradient of the largest class score with respect to the input. To define some notation, gxx) be this most basic form of the gradient- based saliency map. for an input image с RR. For simplicity, we also assume that elements of g(x) have been linearly normalized to be between and 1. g(x) represents, to a first order linear approximation, the importance each pixel in determining the class label. Numerous variations of this method have been introduced, which review in the appendix.
(2014) computes the elementwise
A popular saliency map method which extends the basic gradient method is SmoothGrad (Smilkov et al., 2017), which takes the average gradient over random perturbations of the input. Formally, we define the smoothing function
g(x) == E[ e)],
(1.1)
where e has a normal distribution (i.c. e~ N0, IT)). We will discuss other smoothing functions in Section 2.3 while the empirical smoothing function which computes the average finitely many
2

---

Under review as a conference paper at ICLR 2020
perturbations of the input will be discussed in Section 2.2. We refer to the basic method described in the above equation as the scaled SmoothGrad
Since ground truth label for interpretation is not available, use a similarity metric between the original and perturbed saliency maps as an estimate of the interpretation robustness. Ghorbani et al. (2019) introduced a top-K overlap metric for this purpose: define R(x, X, K) the number of overlapping elements between top K largest elements of saliency maps of x and the perturbed image x. Ghorbani et al. (2019) developed an Loo attack on R[(,, X, K), which we adapt to the La case to test empirical robustness: see the appendix for details. This top-K overlap metric is naturally motivated: absolute magnitude has little meaning in saliency maps, which are nearly always pre- sented as normalized. Moreover, it is common (Sundararajan et al., 2017) to clip the highest values when displaying saliency maps, so that more than few pixels are clearly visible. This suggests that relative ranks of salience values (and not their absolute values) are important for interpretation.
Note that for an input x, R(x, X, K) depends on the specific perturbation We define R (x, K) as the robustness measure with respect to the worst perturbation of x. That is,
Rx K) = min R(x, x, K)
(1.2)
1 - x1|2 S.
For deep learning models, this optimization is non-convex in general. Thus, characterizing the true robustness of interpretation methods is likely intractable for most modern classifier models.
In this paper, we show that a lower bound on the true robustness value of an interpretation method (i.e. robustness certificate) can be computed efficiently. In other words, for given input x, we compute a robustness certificate Rean such that (x K) S R (x, K). To establish the robustness certificate for saliency map methods, we prove the following result for a general function h(.) whose range is between O and 1:
Theorem 2. Let h(xx x) be the outpul of an interpretation method whose range is between 0 and I and let h be its smoothed version defined as in equation 1.1. Let h2(x) and (y (x)(x) be the i-th element and the i-th largest elements of h(x),, respectively. Let &b be the edf of the normal distribution. If
2p) σ
(h(((())))
h12k-11
(1.3)
then for the smoothed interpretation method, we have Re (x, K) i
Intuitively, this means that, if there is a sufficiently large gap between the i-th largest element of the smoothed saliency map and its (2K - -ih largest element, then we can certify that at least i elements in the top K largest elements of the original smoothed saliency map will also be in the top K elements of adversarially perturbed saliency map. We present more general version of this result with empirical expectations for smoothing in Section 2, as well as another rank-based robustness certificate in Section 3. The proof of this bound relies on an extension of Cohen et al. (2019) which addresses certified robustness in the classification case. Proofs for all theorems are given in the appendix.
Evaluating the robustness certificate for the scaled SmoothGrad method on ImageNet samples pro- duces vacuous bounds (Figure 1-c). This motivated us to develop variations of SmoothGrad with larger robustness certificates. One such variation is Sparsified SmoothGrad which is defined by smoothing a sparsification function that maps the largest elements of g(x) one and the rest to zero. Sparsified SmoothGrad obtains a considerably larger value of the robustness certificate (Figure 1-c) while producing high-quality saliency maps (Figure 3). We study other variations of Sparsified SmoothGrad in Section 2.
considerably different certification result for robust interpretation is provided by Lee et al. (2019). That work provides a certified L2 radius in which a first-orded gradient saliency map is exactly iden- tical to the salieney map for the unperturbed image. The method depends on the network structure being locally linear (Rel. RRel.Ubbased) the inner radius of the locally linear region is directly calculated.
'The original definition of SmoothGrad does not normalize and take the absolute values of gradient elements before averaging. We start with the definition of equation 1.1 because it can more easily be compared to our certifiably robust interpretation scheme discussed in Section 2.
3

---

Under review as a conference paper at ICLR 2020
By contrast, our method makes demands the structure of the classifier, and could be applied to saliency maps generated by methods other than computing simple gradients: any bounded saliency function can be used h(x) in our Theorem 1. Additionally, tionally inexpensive enough to allow certificates to be computed on ImageNet-scale samples.
certification method is computa-
2
SMOOTHING FOR CERTIFIABLY ROBUST INTERPRETATION
2.1 NOTATION
We introduce the following notations to indicate Gaussian smoothing: for function h, we define population and empirical smoothed functions, respectively, as:
h(x) = ([-x00,onn((((xx() h(x)
1 ' M((+))) a~N(0,001)
(2.1)
In other words, h(x) represents the expected value of h(x) when smoothed under normal pertur- bations of with some standard deviation a while h(xx represents an empirical estimate of hix) using q samples. We call o² the smoothing variance and the number of smoothing perturbations.
We V; denote the jth element of the vector v. Similarly (x(x denotes the jth element of the output h(x). We also define, for any hxx), rank(h ( as the ordinal rank of hx(x) in h(x) (in the descending order): rank(h(x)) =j denotes that h2(x) is the j'h largest element in h( (x). We use [ to denote the 1th largest element in x. If not an integer, the ceiling of i is used.
2.2 ROBUSTNESS CERTIFICATE
In order to derive robustness certificate for saliency maps, we present an extension the classifi- cation robustness result of Cohen al. (2019) to real-valued functions, rather than discrete classifi- cation functions. In our case, we will apply this to the saliency map vector g. First, we define a floor function to simplify notation.
Definition 2.1. The Floor function is a function L: [:[0,1] 1]. such that
④ -1 (z) 2p)
L(z) = +
where p denotes the L2 norm of the adversarial distortion and o² denotes the smoothing variance. T is the edf function for the standard normal distribution and t is its inverse.
Below (Theorem 1) is general result which be used to derive robustness certificates for inter- pretation methods. In particular, it used to derive our robustness certificate, Theorem 2, later in the section:
[0, 1] be real-valued function. Let L(.) be the floor function defined as in equation 2.1 with parameters o² and p. Using o² e R as the smoothing variance for h,
Theorem 1. h:
A i,j e [n], x, ee R" where ||x x
(f. (x)x))
(x).)
Note that this theorem is valid for any general function. However, we will use for our case where h(x) is a smoothed saliency map. Theorem states that, for given saliency map vector h(x), if [x); 2 (x), then if x is perturbed inside an T.2 norm ball of radius at most p. h; 2 h, This result extends Theorem 1 in Cohen et al. (2019) in two ways: first, it provides a guarantee about the difference in the values of two quantities, which in general might not be related, while the original result compared probabilities of two mutually exclusive events. Second, we are considering a real-valued function h, rather than a classification output which can only take discrete values. This bound be compared directly to Lecuyer et al. (2019)'s 's result which similarly concerns unrelated elements in a vector. Just as in the classification case (as noted by Cohen et al. (2019)), Theorem gives significantly tighter bound than that of Lecuyer et al. (2019) (see details in the appendix).
4

---

Under review as a conference paper at ICLR 2020
86
D82
0.05
ub-
--337
0.35
Robustriess Certificate
Robustress Certificale
05
04
04
02
02
00
00-
Sparsification Parameter
Sparsification Parameter
(a) Sparsified SmoothGrad
(b) Relaxed Sparsified SmoothGrad
Certified robustness bounds ImageNet for different values of the sparsification param- eter т. The lines shown are for the 60th percentile guarantee, meaning that 60 percent of images had guarantees least tight those shown. For both examples, K 0.2n, and p= 0.03 (in units where pixel intensity varies from to 1.)
Figure 2:
We can extend Theorem to use empirical estimates of smoothed functions. Following Lecuyer et al. (2019), derive upper and lower bounds of the expected value function h(x) in terms of h(x), by applying Hoeffding's Lemma. To present result for the empirical case, we first define an empirical floor function to derive a similar lower bound when the population mean is estimated using a finite number of samples:
Definition 2.2. The Empirical Floor function is function L: [0, - [0. 1], such that for given values of p, 0, p, q, n, where denotes the maximum L2 distortion, d² denotes the smoothing variance, denotes the probability bound, denotes the number of perturbations, and n is the size of input of the function:
i(x) + 10(0-16
-
2p
where e
In(2n(1 1)
24
Corollary 1. Ler h: R" - []
" be function such for given values of a, Vi,je [n],
R", ||x [|x-x]] x||22 P. with probability at least p.
((,( h2(x) h,(x) (X).
(2.2)
Note that unlike the population case, this certificate bound probabilistic. Also note that this bound compares the empirical expectation of the saliency map at an observed point to the true population expectation all nearby points. This convention in line with recent works providing probabilistic certificates for smoothed classifiers (Cohen et al., 2019; Salman al., 2019).
Theorem allows us to derive certificates for top-K K overlap (denoted by R). In particular:
Theorem 2. x. R" xx x12 S p, R, q é NN define R (x, K) the largest i K such that (()n())) 2 h12x-1(x) (x). Then, with probability at least p.
RCe" (x, K) R(x, &, K).
(2.3)
where R(x, 8, K) denotes the top-K overlap between h(xx) and hnxx).
Intuitively, if there is sufficiently large gap between the ith and (2K i largest elements of an empirical smoothed saliency map. then we can certify that the overlap between top K elements of the observed saliency map and any possible perturbed population smoothed saliency map least i with probability at least p. If the observed saliency map is possibly adversarially corrupted up to radius of p, it thus still guaranteed be similar to the "ground truth" population saliency map.
5

---

Under review as a conference paper at ICLR 2020
Note that it requires some modifications of SmoothGrad (Smilkov et al., 2017) for bounds to be directly applicable. Smilkov et al. (2017) in particular defines two methods, which we will call SmoothGrad and Quadratic SmoothGrad. SmoothGrad takes the mean gradient values, with absolute value typically taken after smoothing for visualization. Quadratic SmoothGrad takes the mean of the elementwise squares of gradient values. Both methods, therefore, require modification for our bounds to be applied: we define scaled SmoothGrad g(x), such that g(x) is the elementwise absolute value of the gradient, linearly scaled so that the largest element one. We can similarly define scaled Quadratic SmoothGrad. We can then apply Theorem 2 directly scaled SmoothGrad (or scaled Quadratic SmoothGrad), simply by scaling the components of g(x) (or g(x (x) @ g( x)) to lie in the interval [0, 1J. However, we observe that this gives vacuous bounds for both of them when using the suggested hyperparameters from Smilkov et al. (2017). One issue is that the suggested value for (number of perturbations) is 50 which is too small to give useful bounds in Corollary 1. For a standard size image from the ImageNet dataset (n 224 x 224 x 150, 528), with p 0.95, this gives 0.395 (using Definition 2.2). Note that even for small p:
samples of the signed
i(=) (0)))($>'() (2-c) 20) - c ($ (x- - (==c)) 2c
Thus the gap between 2 and [=x) is at least 0.79. We can see from Corollary that gap of 0.79 (on a scale of 1) is far too large to be of any practical use. We instead take q 2 which gives more manageable estimation error of e = 0.031. However, we found that even with this adjustment, the bounds computed using Theorem 2 are not satisfactory for either scaled SmoothGrad and or scaled Quadratic SmoothGrad (see details in the appendix). This prompted the development of Sparsified SmoothGrad described in Section 2.3.
2.3 SPARSIFIED SMOOTHGRAD AND ITS RELAXATION
Scaled SmoothGrad and Quadratic SmoothGrad give vacuous robustness certificates using our pro- posed method, as demonstrated in Figure 1. We therefore develop method, Sparsified Smooth- Grad, which has (1) non-vacuous robustness certificates at ImageNet scale (Figure 2a), (2) similar high-quality visual output to SmoothGrad (Figure 3), and (3) theoretical guarantees that aid in set- ting its hyper-parameters (Section 3).
The Sparsified SmoothGrad is defined as g g[rl, where g! [r] is defined as follows:
0, if gi (x) < g[rn] (x)
(2.4)
1, if g: (x) 2 B[mm] (x)
In other words, т controls the degree of sparsification: a fraction 7 of elements (the largest tn elements of g(x)) are assigned to 1, and the
set to 0.
For some applications. it may be desirable to have at least some differentiable elements in the com- puted saliency map. For this purpose, also propose Relaxed Sparsified SmoothGrad:
0,
if ffg:(() < g[rn]
[x5)(()
1 (x)
1,
if (x) g/ (x)
(2.5)
(x)
otherwise
(x))
Here, controls the degree of sparsification and controls the degree of clipping: fraction y of elements are clipped to Elements neither clipped nor sparsified are linearly scaled between 0 and 1. Note that Relaxed Sparsified SmoothGrad is generalization of Sparsified SmoothGrad. With no clipping (7 0), we again achieve nearly-vacuous results. However, with only a small degree of clipping (+ 0.01), we achieve results very similar (although slightly worse) than sparsifed Smooth- Grad: see Figure 2b. We use Relaxed Sparsified SmoothGrad in this paper to test the performance of first-order adversarial attacks against Sparsified SmoothGrad-lin techniques.
3 RANK CERTIFICATE FOR THE PROPOSED SPARSIFIED SMOOTHGRAD
In this section, we show that if the median rank of a saliency map element over smoothing pertur- bations is sufficiently small (i.e. near the top rank), then for adversarially perturbed input, that

---

Under review as a conference paper at ICLR 2020
Base Image
Quadratic SmoothGrad
Sparsified SmoothGrad
SmoothGrad
Figure 3: Qualitative comparison of Sparsified SmoothGrad (with the sparsification parameter T 0.1) with the SmoothGrad methods defined by Smilkov al. (2017). All methods lead to high- quality saliency maps while our proposed Sparsified SmoothGrad is certifiably robust to adversarial examples well. Additional examples have been presented in the appendix.
element will certifiably remain near the top rank of the proposed Sparsified SmoothGrad method with high probability. This provides a theoretical justification for the certifiable robustness of the proposed Sparsified SmoothGrad method.
To present this result, we first define the certified rank of element in the saliency map as follows:
Definition 3.1 (Certified Rank). For given input and given saliency map method (denoted by h: + RR",, let the maximum adversarial distortion be p, i.e. | Xxxl|12 $ p. Then, for a probability p. the certified rank for an element at index (denoted by rank (x, i)) is defined as the minimum A such that the following condition holds:
L (h,()))) (x).
the i-th element of the saliency map has a certified rank of k. using Corollary 1, we will have:
h (x) hra with probability at least p.
That the 7th element of the population smoothed saliency map is guaranteed be large as the smallest n - 1: + elements of the smoothed saliency map of any adversarially perturbed input.
Note that certified rank depends on the particular perturbations used generate the smoothed saliency map h(x). In the following result, show that if the median rank of a gradient ele- ment at index i, over a set of randomly generated perturbations, is less than specified threshold value, then the certified rank of that element in the Sparsified SmoothGrad saliency map generated using those perturbations can be upper bounded.
Theorem 3. Let be the set of q random perturbations for a given input x using the smoothing variance a Using the Sparsified SmoothGrad method, with probability p, we have
Tn
Median (rann)((aaa((x))))...()) [T rank cent (x, i)
(3.1)
cel)
where is the sparsification parameter of the Sparsified SmoothGrad method.

---

For instance, if p and for sufficiently large number of smoothing perturbations (i.e. q oo), have T221 1/2. If set (2n),, then for indices whose median ranks are less than or equal to K/2, their certified ranks will be less than or equal to K. That is, even after adversarially perturbing the imput, they will certifiably remain among the top K elements of the Under review as a conference paper at ICLR 2020
Sparsified SmoothGrad saliency map. We present more general form of this result in the appendix.
2,0
Origisal Image
Bodine Mork!
Robux Clecifisation Model g(x)
0.9
g(
R(S)
0.8
5
0.6
0.5
0.-
0.
30
10
20
25
30
Altack Ragnitude
Dracient
StecethGrad sempless
SmontthGrrd 82 samples)
Figure 5: Adversarial training for robust classi- fication is not effective at producing robust in- terpretations. We attack the interpretation (us- ing an L2 attack magnitude p = 10) of a base- line CNN and CNN trained for robust clas- sification. The classification-rohie average robustness of 0.5853, compared to a baseline of 0.5727, over the MNIST test set. Robustness here is measured as R(x, x, K)/K with K = m/44
Quedratic Smoothdred (8192 samples)
Sparsited SmeothGrae
8192 samorest
Relowed Sparsified Smeothhrrd it 0.1, 0192 samples)
Reloxad Sparsifid Smoothörad it 0.2, 8192 sampics)
Figure 4: Empirical robustness of variants of SmoothGrad to adversarial attack, tested on CIFAR-10 with ResNet-18. The attack mag- nitude is in units of standard deviations pixel intensity. Robustness is measured R(x, &, K)/K with n/4.
model has
4 EMPIRICAL RESULTS
To test the empirical robustness of Sparsified SmoothGrad, we used an L2 attack on R(x, K) adapted from the Los attack defined by Ghorbani et (2019); see the appendix for details of our proposed attack. We test Relaxed Sparsified SmoothGrad (7 .01;7= .05, .1, .2), rather than Spar- sified SmoothGrad because our attack gradient-based and Sparsified SmoothGrad has no defined gradients. We tested on ResNet-18 with CIFAR-10 with the attacker using reparately-traines differentiable version of ResNet-18, 18, with SoftPlus activations in place of ReLU. We present our re- sults in Figure 4. We observe that our method is significantly more robust than the SmoothGrad method while its robustness on par with the Quadratic SmoothGrad method with the same num- ber of smoothing perturbations. We note that robustness certificate appears to be loose for the large perturbation magnitudes used in these experiments; see the appendix for a direct comparison.
fully
The attack introduced by Ghorbani al. (2019) (and its Le extension that introduced) does not change the the original classification the image, so protecting against adversarial attacks classification is insufficient defence. To demonstrate this, compare model trained adversarially for classification with a baseline model. Specifically, we adversarially train a simple CNN classification model MNIST (see appendix for model architecture) using state-of-the-arr L2 adversarial training procedure provided by Rony et al. (2019). We used the code available for this adversarial training procedure directly. While. as predicted by Tsipras et al. (2018), the resulting model produced high-quality saliency maps, the model was not significantly more robust to adversarial perturbation tailored for the interpretation than naively trained classifier (Figure 5, top-K overlap of 58.5% with adversarial training for classification vs. 57.3% on the baseline). This highlights that adversarial training for classification does not suffice to provide adversarial robustness for interpretation. In appendix D, we propose adversarial training approach which successfully defends interpretations against adversarial attack (top-K K overlap greater than 80% under the same attack) while also producing high-quality saliency maps. However, unlike the proposed Sparsified SmoothGrad, this approach is not certifiably robust.
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
In this work, we studied the robustness of deep learning interpretation against adversarial attacks. We introduced a sparsified variant of the popular SmoothGrad method which computes the average saliency maps over random perturbations of the input. By establishing casy-to-compute robust- ness certificate for the interpretation problem, we showed that the proposed Sparsified SmoothGrad is certifiably robust to adversarial attacks while producing high-quality saliency maps. We provided experiments on ImageNet samples validating our theory.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 9525-9536, USA, 2018. Curran Associates Inc.
D. Alvarez-Melis and T. S. Jaakkola. Towards Robust Interpretability with Self-Explaining Neural Networks. Neural Information Processing Systems, 2018.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018a.
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In ICML, 2018b.
A. BenTaicb, J. Kawahara, and G. Hamarnch. Multi-loss convolutional networks for gland analysis in microscopy. In 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp- 642-645, April 2016. doi: D0.1099/SBBL2011.74993333
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian J. Goodfellow. Thermometer encoding: One hot way to resist adversarial examples, In ICLR. OpenReview.net, 2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEEE Symposium on Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pp. 1310-1320, 2019.
Christian Etmann, Sebastian Lunz, Peter Maass, and Carola Schoenlieb. On the connection between adversarial robustness and saliency map interpretabilityy In International Conference on Machine Learning. pp. 1823-1832, 2019.
Thomas Fischer and Christopher Krauss. Deep learning with long short-term memory networks for financial market predictions. European Journal of Operational Research, 270(2):654 669, 2018.
Amirata Ghorbani, Abubakar Abid, and James Zou, Interpretation of neural networks is fragile. In the Proceedings the AAAI Conference on Artificial Intelligence, volume 33, pp. 3681-3688 2019.
Ian J. Goodfellow. Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. CoRR, abs/1412.6572, 2015.
Seunghoon Hong, Hyeonwoo Noh, and Bohyung Han. Decoupled deep neural network for semi- supervised semantic segmentation. In C. Cortes, N. D. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems 28, pp. 1495-1503. Curran Associates, Inc., 2015.
Kan Huang, Chunbiao Zhu, and Ge Li. Robust saliency detection via fusing foreground and back- ground priors. CoRR, abs/171 11771110033 1.00322, 2017.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dühne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. Springer, 2019.

---

Under review as a conference paper at ICLR 2020
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical world. CoRR, abs/1607.02533. 2017.
Amal Lahiani, Jacob Gildenblat, Irina Klaman, Nassir Navab, and Eldad Klaiman. Generalis- ing multistain immunohistochemistty tissue segmentation using end-to-end colour deconvolution deep neural networks. IET Image Processing, 13(7):1066-1073, 2019.
Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In 20/9 IEEE Symposium Security and Privacy (SP), pp. 656-672. IEEE, 2019.
Guang-He Lee, David Alvarez-Melis, and Tommi S. Jaakkola. Towards robust, locally linear deep networks. In International Conference Learning Representations, 2019.
Bai Li, Changyou Chen, Wenlin Wang, and Lawrence Carin. Second-order adversarial attack and certifiable robustness. arXiv preprint arXiv:1809.03111 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. CoRR, abs/1 0tts/106.0000 706.06083, 2018.
Ephraim Nissan. Digital technologies and artificial intelligences present and foresceable impact lawyering. judging, policing and law enforcement. Ai & Society, 32(3):441 4644, 2017.
Nicolas Papernot and Patrick D. McDaniel. On the effectiveness of defensive distillation. CoRR, abs//607.00 2016.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS- W, 2017.
G Quellec, K Charrière, Y Boudi, В Cochener, and M Lamard. Deep image mining for diabetic retinopathy screening. Medical image analysis, 39:178, 2017.
Jérôme Rony, Luiz G Hafemann, Luiz S Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and nomn for efficient gradient-based 12 adversarial attacks and defenses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1322-4330 2019.
Hadi Salman, Greg Yang. Jeny Li, Pengchuan Zhang, Huan Zhang, Ilya Razenshteyn, and Sebastien Bubeck. Provably robust deep learning via adversarially trained smoothed classifiers. arXiv preprint arXiv::106.044
2019.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Ma- chine Learning-Volure 70, pp. 3145-3153. JMLR. org, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In ICLR, 2014.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Vićgas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: 0706.038255 2017.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. CoRR, abs/1312.6199 2014.
Marvin Teichmann, Michael Weber, Marius Zoellner, Roberto Cipolla, and Raquel Urtasun. Multi- net: Real-time joint semantic reasoning for autonomous driving. In 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1013-1020. IEEE, 2018
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. There is no free lunch in adversarial robustness (but there are unexpected benefits). arXiv preprint arXiv: 1805. /2/22, 2018.
10

---

Under review as a conference paper at ICLR 2020
Jonathan Uesato, Brendan O' Donnoghue, Pushmeet Kohli, and Aäron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In JCML, 2018.

---

Under review as a conference paper at ICLR 2020
A PROOFS
Theorem 1. Let : RR - [0.1]" be a bounded, real-valued function, o² e R be the smoothing variance for h, then V i,j [n], x, XE Rn where & 5 such that ||0| p.
①b x (h.())())) 2p)
hi -x) hx)
where & denotes the cdf function for the standard normal distribution and - 1 is its inverse.
We will prove this by first proving more general lemma
Lemma 1. For any bounded function h: IR - [0, 1] and smoothing variance o² R, x- (h(x)) is
Lipschitz-contimuuus with respect x, with Lipschitz constant
Proof. By the definition of Lipschitz continuity, we must show that Vg e RR".
| < 4 (h(x*+ 6)) x (h(xx)))) + |
s (>((xx)
(A.1)
σ
We first define new, randomized function H :R {0, 1}.
H(x) Bern (Ix(x)
Then A x R
L[ [)] [Lw [X [ LE c)] h(x)
(A.2)
Now, we apply the following Lemma (Lemma from Cohen al. (2019)):
Lemma (Lemma from (Cohen et al., 2019)). Let X N(x, o²1) and Y N(xx + 0²1). Let f: Rn {0, 1) be any deterministic or random function, Then:
1. If {2e (2ER" : 2 5 for some 33 and Pr(f(X) = = Pr(X 8), then Pr(f(Y) Pr(Y ES)
2. If S = {2e R=: 8T 2 33 for some 3 and Pr(f(X) = =) Pr(X e S), then Pr(f(Y)-
1) s Pr(Y 5)
Using the same technique as used in the proof of Theorem in Cohen et al. (2019), we fix 6 and define,
8 = 3=0/0122 (E[H(xx e)])
Also define the salf-spaces:
S_ (z: 8 12::2=z (2: x) 3)
S, = (2: 5 2 - 5- (z x) 2 -9)
Applying algebra from the proof of Theorem 1 in Cohen al. (2019), we have,
8
Pr(X e S.) ①b
E [[(xx)))
(A.3)
|||6|||2
2
-3
0 ))
3
Pr(Xe = 1
= 1
-E[H(x+
(A.4)
8 2
B
|
|01l2
Pr(Ye
T
(E (E[H(x+
(A.5)
a 2
(--8)
|
|8112
e =
[E[H c)])
(A.6)
0|602
We note that Salman al. (2019) has recently and independently published different proof of this lemma.
12

---

Under review as a conference paper at ICLR 2020
Using equation A.3
(x(H(X)=11 1) [[(()) - [ Pr(X c
Applying Statement I of Cohen's 's lemma, using = H and S = S_:
E[H(x 6 - Prre))))))(((++bb))) - = 1) 2 Pr(Y
(A.7)
Using equation A.4,
(x( =) +| P S+)
Applying Statement 2 of Cohen's lemma, using H and 5 S,:
E[H(x 8 c)] + Pr(H 1) s &
(A.8)
Using equation A.7 and equation A.8:
Pr(Y S-) s
e.)
Then by equation A.5 and equation A.6:
c + (E[H(x+e))) |
+E[H(x+8++)] s & ( |
Noting that -- is monotonically increasing function, we have:
|öه2 s x-' x (E[[Hx* < (E| e)])+ |
Φ (E[H(x+
Using equation A.2 yields equation A.I, which completes the proof.
We now proceed with the proof of Theorem 1.
Proof. Applying Lemma to h, and h, gives (recalling that x = 5+ x ):
-ô12 s Φ s0-00hhhhh (hh())) s (h,((x)) - < x- (h.(x))
(A.9)
Φ (h,(x)) s क (h,(x))
(h)x)) +
|01|
s <
Φ (h,((() P
(A.10)
Then we have:
4 ( (hx(x))
2p
0 2 (x)
Φ- (fi(x)) L $- ' (h,(x)) 2
(by monotonicity of x-1)
4 (h(()) x- (h,(x)) P
(by equation A.9)
Φ (h((() 2 x- (h;(x)
(by equation A.10)
hx(x) 2 hy(x)
(by monotonicity of ()
which proves the implication.
Corollary 1. Let h: R" [ 13a be finction such that for given values a:
n((x) = 1 Σ h(xx+e),, 6 N(,,00)
(A.11)
V i, e [n], RE Xx x1|2 S p. with probability at least
ixh((x)) 2 h,(x) (x))) hx)
13

---

Under review as a conference paper at ICLR 2020
Proof. By Hoeffding's Inequality, for any <> 0, V d:
Pr [x[[a(x): - -x)()))x) 2 c) s 2e 200
(A.12)
Then:
U [h,
hx,))) - e 2ne 2qc
Pr
(A.13)
Since we are free to choose c, define such that p 2ne 29c² then:
Im (m -p)-1)
c=
(A.14)
2q
Pr U [x((x) - h,(x)) 2 (x)(
2ne 220² =1-1
1
1- Pr (u((u(((xx)) -
(x())) 2 (x,xx()))xxx)))22)))
2p
Pr n (h,x)))),,,x))303 2p
Then with probability at least y::
h;(x) cx c(xx))) h,(x) c> (x)
(A.15)
So:
2p) 2 Φ (f,(((( -c) 20) h কb ( (0 (h:(x)) ( h,(x)
(A.16)
a
o
The result directly follows from Theorem 1.
Theorem 2. V xx RR" |x- | $ p, R, q IN, define neen K) the largest is K such that ((rgg(x)) (x). Then, with probability at least p.
Rxe" K) R(x, &, K).
(A.17)
where R(x, x, K) denotes the top-K K overlap between h(x) and h(x).
Proof. Note that the proof of Corollary 1 guarantees that with probability at least p, all estimates li(x) are within the approximation bound e of h(x).. So we can assume that Corollary I will apply simultaneously to all pairs indices i,j, with probability p. We proceed to prove by contradiction.
Leti= R K)
(x4))(x)) 2 hrak (x),
Suppose there exists such that:
R(x, 8, K) <i, Since L a monotonically increasing function.
((b(g(x)) (x)
((h))))))) 2 (x), vi'si, 2K
and therefore by Corollary 1:
n) V m,n n rfn), m) s i, n))) 2K h (x)) 2 h (x)
(A.18)
Let X be the set of indices in the top K elements in hi(x), and X be the set of indices the top K elements in h(x)). By assumption, X and X share fewer than elements, there will be least K i elements in
14

---

Under review as a conference paper at ICLR 2020
X which are not in X.
All of these elements have rank at least K + l(xx..
Thus by pigeonhole principle, there some index I X X, such that rank(f((x), 1) K + i 2K + 2 2K i.
Thus by Equation equation A.18,
Vm, where rank s hmm(1) hx(x)
(A.19)
Hence, there are i such elements where rank rank(h(x). (((x), m) s i: these elements are clearly in X. Because / X, Equation equation A.19 implies that these elements are all also in X. Thus X and X share at least elements, which contradicts the premise.
(In this proof we have implicitly assumed that the top K elements of a vector can contain more than K elements, if ties occur, but that rank is assigned arbitrarily in cases of ties. In practice, ties in smoothed scores will be very unlikely.)
A.1 GENERAL. FORM AND PROOF OF THEOREM
We note that Theorem 3 can be used to derive more general bound for any saliency map method that for an input x, first maps g(x) to an elementwise function that only depends on the rank of the current element in gxx) and not the individual value of the element. We denote the composition of the gradient function and this elementwise function as g[rank], The only properties that the function must satisfy is that it must be monotonically decreasing and non-negative. Thus, we have the following statement:
Theorem 3. Let T be the threshold value and let U be the set of q random perturbations for a given input x using the smoothing variance o² and let p be the probability bound. If is an element index such that:
Median (rank ()) <EU
(A.20)
Then:
Σz
[rank] (x)
rank (x, i)s
(A.21)
baank) (x)
i 2
Furthermore:
raakk] [
(x)
[rank] Big (x): i.
are both independent of x. Thus RHS is a constant.
(A.22)
2
R* i. takes the rank of the element as the input and outputs a real number. Furthermore, we assume that f is non-negative monotonically decreasing function. Thus g. [rank] (x) (rann((g(x),))
Proof. Let the elementwise function be f
We use f() to denote the constant value that maps elements rank to.
Note that B[t] (x) is the tt largest element of g! g[rauk)(x))
[rauk]
Since f is monotonically decreasing function:
[rank]
(x) f(i) V Vie [n]
Bia]
Thus B[i]
[rank] (x) is independent of x, we simply use [rank)() g/
(.) to denote f(i), i.e:
[rank]]
(.) = f(i Vie
Because Median [rank(g(x + e), i)] < T, for at least half of sampling instances e in U. rank(g(x+ e), T. [rank] (x + e) 2 JT)), So in these instances g
The remaining half or fewer elements mapped to other nonnegative values. Thus the sample mean:
[rank]
(x)
1
Eg
g (x
[rank] e) 2 8(1]
[rank] (-)/2
c
15

---

Under review as a conference paper at ICLR 2020
Using Corollary 1, B. (x) is certifiably large all elements with indices such that:
[(
[rank]
(+/2) 2 Bj
[rank] (x)
Now we will find an upper bound on the number of elements with indices such that:
|rank]
(x) (
[rank] (-)/2)
Because all the ranks from n will occur in every sample in U, we have:
VeEU,
Bk
[rank]
xx+
k=1 [()]
Σ
[rank]
())
8k
[rank]
(x)
Σ
12
Σ
8i
[rank]
(x +))
Σ
8[k]
[rank] (.)
k=1
1
cEl
Thus strictly fewer than Ek-1
[rank] Kik]
())/i
[rank] ()/2) elements will have mean greater than 8|11
i. [rank] ()/2) Bi1]
Hence, B. (x) is certifiably at least as large
Σ [rank] (raakk(())
[rank] ()/2) + elements, (3)
which by the definition of rank (x, i) yields the result. Theorem 3 in the main text follows trivially, because in the Sparsified SmoothGrad case, Note that this represents the tightest possible realization of
Σk=1 () T, and [r1] this general theorem.
B RELATED WORKS
Sundararajan et al. (2017) defines a baseline, which represents an input absent of information and determines feature importance by accumulating gradient information along the path from the base- line to the original input. Alvarez-Melis & Jaakkola (2018) builds interpretable neural networks by learning basis concepts that satisfy an interpretability criteria. Adebayo et al. (2018) proposes meth- ods to assess the quality of saliency maps. Although these methods can produce visually pleasing results, they can be sensitive to noise and adversarial perturbations.
Szegedy et al. (2014) introduced adversarial attacks for classification in deep learning. That work dealt with La attacks, and uses L-BFGS optimization to minimize the norm of the perturbation. Carlini & Wagner (2017) provide an L2 attack for classification which is often considered state of the art.
One strategy to make classifiers more robust to adversarial attacks is randomized smoothing. Lecuyer al. (2019) use randomized smoothing to develop certifiably robust classifiers in both the L1 and L2 norms. They show that if Gaussian smoothing is applied to class scores, a gap be- tween the highest smoothed class score and the next highest smoothed score implies that the highest smoothed class score will still be highest under all perturbations of some magnitude. This guarantees that the smoothed classifier will be robust under adversarial perturbation.
Li et al. (2018) and Cohen et al. (2019) consider related formulation. Cohen gives a bound that is tight in the case of linear classifiers and gives significantly larger certified radii. their formulation, the unsmoothed classifier is treated a black box outputting just discrete class label. The smoothed classifier outputs the class observed with greatest frequency over noisy samples. Salman et al. (2019) considers adversarial attacks against smoothed classifiers.
In the last couple of years, several approaches have been proposed to for interpreting neural network outputs. Simonyan et al. (2014) computes the gradient of the class score with respect to the input. Smilkov et al. (2017) computes the average gradient- based importance values generated from sev- eral noisy versions of the input. Sundararajan et al. (2017) defines baseline, which represents
16

---

Under review as a conference paper at ICLR 2020
input absent of information and determines feature importance by accumulating gradient informa- tion along the path from the baseline to the original input. Alvarez-Melii & Jaakkola (2018) builds interpretable neural networks by learning basis concepts that satisfy an interpretability criteria. Ade- bayo et al. (2018) 1 proposes methods to assess the quality of saliency maps. Although these methods can produce visually pleasing results, they be sensitive noise and adversarial perturbations (Ghorbani et al., 2019; Kindermans et al., 2019).
As mentioned in Section 1, several approaches have been introduced for interpreting image classi- fication by neural networks (Simonyan et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017; Shrikumar et al., 2017).
C L2 ATTACK ON SALIENCY MAPS
We developed an Le norm attack on Re based on Ghorbani et al. (2019)'s Loo attack. Our algo- rithm is presented as Algorithm We deviate from Ghorbani et al. (2019)'s attack in the following ways.:
We use gradient descent, rather than gradient sign descent: this is direct adaptation to the L2 norm.
We initialize learning rate (eा())) and then decrease learning rate with increasing iteration count, proportionately (for the most part) to the reciprocal of the iteration count. These are both standard practices for gradient descent.
We use random initialization and random restarts, also standard optimization practices.
If gradient descent step would cross decision boundary, we use backtracking line search to reduce the learning rate until the step stays on the correct-class side. This allows the optimization to get arbitrarily close decision boundaries without crossing them.
We measured the effectiveness of our attack (Q 100, 20, 5 against a slight modification of Ghorbani et al. (2019)'s attack, in which the image was projected (if necessary) onto the T.2 ball at every iteration, and also clipped to fit within image box constraints (this was not mentioned in Ghorbani al. (2019)'s original algorithm). For this attack, we set the (L..) learning rate parameter at p/500, and ran for to 100 iterations. We also tested against random perturbations. For random perturbations, up to 100 points were tested until a point in the correct class was identified. We tested these attacks on both "vanilla gradient" and SmoothGrad saliency maps. See Figure 6. Experimental conditions are described in Section F for experiments on CIFAR- 10. In this figure, for each attack magnitude, we discard any image on which any optimization method failed.
0,50
Ra sdor perturtation
0.46
8m-
0.42
03-
0.-
0.58
Charbani attack, projected asto ta
witack
0.3
Random pertu bation
10
12:
20
12
20
23
30
ATTOCK Magnituse
ABDOk Magnituse
(a) Attacks on vanilla gradient method.
(b) Attacks on SmoothGrad method (q=64).
Figure Comparison of attack methods on images in CIFAR-10. See text of section C.
17

---

Under review as a conference paper at ICLR 2020
Algorithm L2 attack lop-k overlap
Input: k, image x, saliency map function h, iteration number P, random sampling iteration number
Q. Le perturbation norm constraint p. classifier c, restarts number
Output: Adversarial example x
1: Define ((z) = rank((((()))))) (z)
2: for == L, .... T do
3:
loop
4:
8 Uniformly random vector L2 p sphere.
5:
x + x+6
6:
Clip x" such that falls within image box constraints.
7:
if f[(x")== xxx")=co =((x) then break inner loop
8:
if total iterations have passed over all & cycles of random sampling then
9:
break outer loop
end if end loop
11:
12: a DV
13:
for p 1,... do
loop
15:
x" + x" +VD(xx
necessary, project x" such that |x*
17:
Clip x" such that it falls within image box constraints. c(x²) cxx) then
19:
break inner loop
20:
else
21:
22:
end end loop
23:
pa 24: a p+l
25: end for
x arg m x") D(z)
27: end for
if random sampling failed at every iteration then fail
29: X arg max (x...) ((z)
D ADVERSARIAL TRAINING FOR ROBUST SALIENCY MAPS
Adversarial training has been used extensively for making neural networks robust against adversarial attacks on classification (Madry et al., 2018). The key idea is to generate adversarial examples for classification model, and then re-train the model on these adversarial examples.
In this section, we present an adversarial training approach for fortifying deep learning interpreta- tions so that the saliency maps generated by the model (during test time) are robust against adversar- ial examples. We focus on "vanilla gradient" saliency maps, although the technique presented here can potentially be applied to any saliency map method which differentiable w.r.t. the input. We solve the following optimization problem for the network weights (denoted by 8):
min ((xx))
(xn(X,y)
A (g(x) -g(x())-(x)11
(D.1)
Classification loss
Robustness loss
where an adversarial perturbation for the saliency map generated from x. To generate x, we developed an Le attack saliency maps by extending the Los attack of Ghorbani et al. (2019) (see the details Section C). (xs y) is the standard cross entropy loss, and À is the regularization pa- rameter encourage consistency between saliency maps of the original and adversarially perturbed images.
We observe that the proposed adversarial training significantly improves the robustness of saliency maps. Aggregate empirical results are presented in Figure 7, and examples of saliency maps are presented in Figure 8. It is notable that the quality of the saliency maps is greatly improved for
18

---

Under review as a conference paper at ICLR 2020
1.0
0.9
0.9
Robustness
0.8
Accuracy
0.7-
0.7
0.5
0.5
200
400
600
800 1000
Figure 7: Effectiveness of adversarial training MNIST. Increasing the regularization parame- ter À in the proposed adversarial training optimization (Equation D.1) sigmificantly increases the robustness of gradient-based saliency maps while it has little effect on the classification accuracy. Robustness here is measured as R(x, &, K)/ K, where K = n/4.
unperturbed inputs, by adversarial training- We note that this observation related to the observation made in Tsipras et al. (2018) showing that adversarial training for classification improves the quality of the gradient-basee based saliency maps. We observe that even for very large value of X, only slight reduction in classification accuracy occurs due to the added regularization term.
Without Adversarial Training
With Adversarial Training
(xx)
x
g(X)
g(x)
g(X)
5
5
5
5 5
8
8
8
Figure 8: An illustration the proposed adversarial training to robustify deep learning interpretation on MNIST. We observe that the proposed adversarial training not only enhances the robustness but it also improves the quality of the gradient-basse saliency maps.
E ADVERSARIAL TRAINING ARCHITECTURE DETAILS
We use the Adam optimizer and generate new adversarial examples after each batch of training according to the updated model. We use a simple convolutional neural network, with SoftPlus acti- vations ensure differentiability of the saliency map, on the MNIST data set (Figure 9). Adversarial perturbations norm up to p= 10 standard deviations of pixel intensity were used. The adversarial attack used was the L2 attack described in Algorithm with P 15, Q 100, T 3. Training was performed for 30 epochs using 48,000 images from the MNIST training set, testing was on the entire MNIST test set. Instances where Algorithm I failed were counted in the averages of
19

---

Under review as a conference paper at ICLR 2020
saliency map robustness, and were rare. (Highest frequency for À 0, at 0.11%). We used the implementation of Adam Optimizer provided with Py PyTorch (Paszke et al., 2017), with default training parameters. These are (Table 2):
Table Hyper-paramese used in model training for MNIST experiments.
Learning Rate
0.001
La regularization parameter
B
(.9,999)
10-8
Batch Size
512
157
9828826
Bdl 14x
10 28428
1x20 1xt0
Convolution
Max-Pool
Convolution
Max-Pool
Flatton
Figure 9: Network architecture used for the MNIST classification. SoftPlus activations are applied after both convolutional layers, and after the first fully connected layer.
F DESCRIPTION OF EXPERIMENTS IN MAIN TEXT
For ImageNet experiments (Figures 1-b and 2-a.b), we use ResNet-50, using the model pre-trained on ImageNet that provided by prchvision.modek and images were pre-processed according to the recommended procedure for that model. In all of these figures, data are from the JLSV RC2012 validation set, sample size is 64, and the main data lines represent the 60th percentile in the sample of the calculated robustness certificate. Error bars represent the 48th and 72 percentile values, corresponding to a 95% confidence interval for the population quantile. For CIFAR-10 experiments, intensities normalized to a 1, = 0 in each channel) using Stochastic Gradient Descent with Momentum implemented by PyTorch (Paszke al., 2017). The following training parameters were used:
train ResNet-18 model on the CIFAR-10 training set (with pixel
Table Hyper-parameters used in model training for CIFAR experiments.
Momentum
0.9
L2 regularization parameter
.0005
Epochs (max)
375
Learning Rate Schedule
(epoch < 150), .01 (epoch 150)
Batch Size
128
Twenty percent of the training data was used validation, and early stopping was used to maximize accuracy relative to this validation set. For adversarial attacks CIFAR samples, we train version of ResNet-18 with SoftMax activations instead ReLU. The adversarial attack used was the Le attack described in Algorithm 1, with P 20, Q 100, T 5. When adversarially attacking images smoothed with = 8192 perturbations with this model, fewer perturbations are are used (512). In these experiments, the sample size is 144. Error bars represent the 95% confidence interval of the population mean. In Figure 4, instances where the adversarial attack failed were not counted at each point. For MNIST experiments in Section 4, the model architecture, attack method, and baseline
20

---

Under review as a conference paper at ICLR 2020
training method are the same used for the adversarial training experiments, detailed in appendix above. The adversarially training for classification uses the implementation provided in Rony et al. (2019), with default arguments.
G ADDITIONAL EXAMPLE IMAGES
See Figures 10 and 11.
images from ImageNet. The input image shown in the first column (far left), with interpretations using SmoothGrad (second col- umn from left), Quadratic SmoothGrad (third column), and Sparsified SmoothGrad (r .1, fourth column). a .2, in units where pixel intensity ranges from to 1.
Figure 10: An illustration of different saliency maps
An illustration of different saliency maps on some images from CIFAR-10. The input image is shown in the first column (far left), with interpretations using SmoothGrad (second column from left), Quadratic SmoothGrad (third column), and Relaxed Sparsified SmoothGrad (r= .1,7 .01, fourth column). .2, and the noise is scaled to the range of pixel intensities of the image.
Figure 11:
21

---

Under review as a conference paper at ICLR 2020
H BOUNDS FOR SCALED SMOOTHGRAD, QUADRATIC SMOOTHGRAD, AND RELAXED SPARSIFIED SMOOTHGRAD WITH 7 0
In the text, we mention that we achieve vacuous bounds for Scaled SmoothGrad, Quadratic Smooth-
Grad, and Relaxed Sparsified SmoothGrad with O. Here are these bounds (Figure 12);
0.5
044
03-
0.
0.02
0.05
0.20
0.20
0.50
100
Sparsification Parameter
Figure 12: Bounds for Scaled SmoothGrad and and Relaxed Sparsified SmoothGrad with ? O. Note that Scaled SmoothGrad is equivalent to Relaxed Sparsified SmoothGrad with = 1. Directly comparable Figure For the Quadratic case, the certificates were negligibly small: the largest robustness certificate achieved was 0.00023, at = 0.2.
I ADDITIONAL IMAGES FROM ADVERSARIAL TRAINING EXPERIMENT (APPENDIX D)
See Figure 13.
A=
200
A=400
g(x)
R
9(X)
g(x)
R
g())
g(x)
g())
8
8
2
600
A= 800
A= 1000
g(x)
g(R)
g(R)
g(x)
R
g(R)
0
8
8
g
2
Figure 13: Additional figures from adversarial training on MNIST, for various X. Note that Figure shows for À= 200.
22

---

Under review as a conference paper at ICLR 2020
J COMPARISON OF BOUNDS TO EMPIRICAL PERFORMANCE FOR RELAXED SPARSIFIED SMOOTHGRAD.
We present detailed view of Figure 4, for small magnitude perturbations, with the robustness certificate shown. (Figure 14)
Gradient
Relaxed Sparsified Smoothōrad (t 0.1, 8192 samples)
SmoothGrad (64 samples)
Relaxed Sparsified SmoothGrad (t 0.2, 8192 samples)
SmoothGrad samples]
Certified
Sparsified Smoothörad 0.05)
Quadratic SmoothGrad (8192 samples)
Certihedd Robustness of Relaxed Sparsitted SmaothGrad (t- 0.1)
Relaxed Sparsified SmoothGrad (t 0.05, 8192 samples)
Certified Robustness of Relaxed Sparsified SmaothGrad (t 0.2)
0.8
Robustness
0.4
0.2
Attack Magnitude
Figure 14: Empirical robustness of variants of SmoothGrad to adversarial attack, tested on CIFAR- 10 with ResNet-18. 18. Attack magnitude units of standard deviations of pixel intensity. Robustness is measured as R(x.3, x, K)/K, where K = n/4
K COMPARISON TO BOUNDS IN LECUYER ET AL. (2019)
Lecuyer et al. (2019) approaches the classification case for certified robustness by smoothing, by using bounds directly comparably to Theorem 1, but applying them to the class score elements, rather than the saliency map elements: bounds are certified by demonstrating that the top class score certifiably larger than all other class scores. However, as noted by Cohen et al. (2019), these bounds rather loose, and Cohen et al. (2019) gives significantly tighter bounds specifically for classification case, which we extend to apply to interpretation. In Figure 15, we compare our bounds for interpretation straightforward application of Lecuyer al. (2019)'s results for class scores to saliency scores. Note that Lecuyer et al. (2019)'s results have a free parameter, for which we numerically maximize the bound.
23

---

Under review as a conference paper at ICLR 2020
Lecuyer, el., plo -1
Theorem 1, =1
Lecuyer, al., plo =0.5
Throrem 1, pio =0.5
0.6
(h:
0.4
L
0.0
0.2
0.4
0.6
0.8
1.0
h (x)
Figure 15: Comparison of Theorem 1 to results from Lecuyer et al. (2019)
24


Reviews:
[{"review_id": "r1epdz-RKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a way to testify how much a SmoothGrad saliency can vary from the true saliency attesting to the adversarial robustness but with the goal of interpretation.\n\nAt the premise of this work I do not think the paper motivates the value of such a robustness certificate. Using the gradient (with SmoothGrad), while providing a reasonable interpretation of the model, is just a linear approximation of the true explanation of the prediction. So saying we have the correct approximation is not so useful. I also am not sure we need such a method. For example imagine a doctor is looking at a saliency map and we are sure that it is correct first order approximation because of some method. What were the negative cases where this would fail? How would this method improve that? I believe right now just the basic gradient is sufficient to indicate the region of interest.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJlg24cQcr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work addresses an important problem of robustness of interpretation methods against adversarial perturbations. The problem is well motivated as several gradient-based interpretations are sensitive to small adversarial perturbations. \n\nThe authors present a framework to compute the robustness certificate (more precisely, a lower bound to the actual robustness) of any general saliency map over an input example. They further propose variants of SmoothGrad interpretation method which are claimed to be more robust.    \n\nThe empirical validation of the underlying theory and use of the sparsified (and relaxed) SmoothGradient interpretation methods is unconvincing because of the following reasons:\n\n1. In the demonstrated experiment, the proposed alternative to SmoothGrad involves setting the lowest 90% of the saliency values to zero, and the top 10% (for sparsified SmoothGrad) or top 1% (in the case of relaxed sparsified SmoothGrad) to one. The problem with clamping most of the lower values to zero and the remainder (or most of the remainder) higher values to one is that it defeats the purpose of having a saliency map in the first place, which exist to characterize the relative importance of the input features. \n\n2. The paper claims that the proposed variant maintains the high visual quality of SmoothGrad, however, the claim is unsubstantiated. With the current setup, there is a clear trade-off between robustness and fidelity of interpretation, which the paper fails to acknowledge. In principle, one can always build extremely sparse or dense interpretation methods (close to all zeros or all ones), which would produce high robustness certificates but would be much less meaningful as they are not faithful to the underlying mechanism of prediction, and the characteristics of the input.\n\n3. The authors present empirical evidence on just one set of sparsification parameters and K. It would be more conclusive to evaluate the robustness of the proposed variations with different values of sparsification parameters, and K.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Hkxsru03FB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an extension of Cohen et al. (2019)’s result that allows one to derive robustness certificates for interpretation methods, as well as a bound on the top-K overlap of saliency methods. These results motivate the introduction of Sparsified SmoothGrad and a relaxation of this method that has differentiable elements. These introduced approaches adapt previous methods so the derived bounds are applicable. The proposed methods are shown to perform as well as Quadratic SmoothGrad (Smilkov et al. 2017) in CIFAR-10 experiments.\n\nI’m not familiar with the field so it is hard for me to judge how novel the presented results are or whether the used baselines are the proper ones. That being said, the paper presents an interesting idea and it is relatively easy to read (I really appreciate the fact that for every theorem there is an interpretation, in words, for it). The only thing that sometimes makes the paper hard to read is when it starts to refer to too many constants without remind the reader what they are about. I have two complaints/questions about the relevance of the introduced bounds though. Right now, to me, it seems that the derived theoretical guarantees are not that relevant, hopefully the questions below will help clarify that.\n\nIn page 6, before introducing the “Sparsified SmoothGrad and its Relaxations”, it is said that q is set to 2^13 because otherwise the gap would be too large in images from ImageNet, for example, when comparing to traditional values of q. However, ImageNet is never revisited in the paper. I was expecting to see ImageNet results in the experimental section but they are not there (or maybe some correlation between the gap and performance -- robustness). More than that, the Quadratic SmoothGrad, which doesn’t have any theoretical guarantee, seems to perform as well as the proposed methods. So where is the gap/theoretical result relevant? What are the settings in which having a method with the derived theoretical guarantees shine? What are the limitations of Quadratic SmoothGrad? Right now, it seems to me that the “Sparsified SmoothGrad and its Relaxations” and its empirical analysis weaken the paper, because they take a big chunk of it when there is not enough evidence to claim them as an important contribution. Am I missing something? I gave this paper a relatively low score because I’m not certain about the relevance of its results, but if my questions are satisfactory answered, I’ll be happy to update my score.\n\n------\n\n\n>>> Update after rebuttal: I stand by my score after the rebuttal. \n\nUnfortunately I'm not an expert in this area and I don't feel confident in having a very strong opinion about this paper. That being said, enough presentation issues were raised that make me uneasy about raising my score. I do agree with some of the concerns raised by other reviewers. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper discusses new methods to perform adversarial attacks on salience maps.

In its current form, this paper in its current form has unfortunately has not convinced several of the reviewers/commenters of the motivation behind proposing such a method. I tend to share the same opinion. I would encourage the authors to re-think the motivation of the work, and if there are indeed solid use cases to express them explicitly in the next version of the paper.
Decision: Reject