Visual Interpretability Alone Helps Adversarial Robustness
Authors: Akhilan Boopathy
Sijia Liu
Gaoyuan Zhang
Pin-Yu Chen
Shiyu Chang
Luca Daniel
Venue:  (2020)
Keywords: adversarial robustness
visual explanation
CNN
image classification

Abstract: Recent works have empirically shown that there exist   adversarial examples that can be hidden from neural network interpretability, and  interpretability is itself susceptible to adversarial attacks. In this paper, we theoretically show  that with the correct measurement of interpretation, it is actually difficult to hide adversarial examples, as confirmed by experiments on MNIST, CIFAR-10 and Restricted ImageNet. Spurred by that, we develop a novel defensive scheme built only on robust interpretation (without resorting to adversarial loss minimization). We show that our defense achieves similar classification robustness to state-of-the-art robust training methods while attaining higher interpretation robustness under various settings of adversarial attacks.

Under review as a conference paper at ICLR 2020
PROPER NETWORK INTERPRETABILITY HELPS AD- VERSARIAL ROBUSTNESS IN CLASSIFICATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent works have empirically shown that there exist adversarial examples that can be hidden from neural network interpretability (namely, making network interpretation maps visually similar), and interpretability itself susceptible to adversarial attacks. In this paper, we theoretically show that with the correct measurement of interpretation, it is actually difficult to hide adversarial examples, MNIST, CIFAR-10 and Restricted ImageNet.
confirmed by experiments Spurred by that, we develop a novel defensive scheme built only on robust in- terpretation (without resorting to adversarial loss minimization). We show that our defense achieves high classification robustness, outperforming state-of-the-an adversarial training methods against attacks of large perturbation while attaining high interpretation robustness under various settings of adversarial attacks.
INTRODUCTION
has become widely known that convolutional neural networks (CNNs) are vulnerable to adver- sarial examples, namely, perturbed inputs with intention to mislead networks' prediction (Szegedy) cet al.
2014; Goodfellow et al. 2015 Papernot et al. 2016aa Carlini & Wagner 2017; Chen et al. Su al., 2018). The vulnerability of CNNs has spurred extensive research adversarial attack and defense. To design adversarial attacks, most work has focused on creating either im- perceptible input perturbations (Goodfellow al. 2015 Papernot et al.) 2016a Carlini & Wagner Chen et al. 2018) or adversarial patches robust to the physical environment (Eykholt et al., 2018 Brown et al. 2017 Athalye et al.) 2017). Many defense methods have also been developed to prevent CNNs from misclassification when facing adversarial attacks. Examples include defensive distillation (Papernot et al.) 2016b), training with adversarial examples (Goodfellow al.) 2015). in- put gradient or curvature regularization (Ross & Doshi Velez 2018 MoosavviDezzoool et al.) 2019), adversarial training via robust optimization (Madry et al. 2018). and TRADES to trade ad adversarial robustness off against accuracy (Zhang et al. 2019). Besides studying adversarial effects network prediction decisions, this work explores the connection between adversarial robustness and network interpretability, and provides novel insights when and how interpretability helps the robustness.
2018
2017
Having a prediction might not be enough for many real-world machine learning applications. It is also crucial to demystify why CNNs make certain decisions. Thus, the problem of network interpre- tation arises. Various methods have been proposed to understand the mechanism of decision making by CNNs. One category of methods justify prediction decision by assigning importance values to reflect the influence of individual pixels or image sub-regions on the final classification. Exam-
ples include pixel-space sensitivity map methods (Simonyan al. 2013 Zeiler & Fergus 2014 Springenberg et al.) 2014; Smilkov et al. 2017]] (Sundararajan et al., 2017) and class-discriminativ
localization methods (Zhou et al. [0166] Selvaraju et al. [017]] Chattopadhay al.) 2018 Petsiuk et al. 2018). where the former evaluates the sensitivity of network classification decision to pixel variations at the input, and the latter localizes which parts of an input image were looked at by the network for making a classification decision. Sensitivity map methods include vanilla gradient (Si- monyan et al. 2013) deconvolution (Zeiler & Fergus 20014), guided backpropagation (Springenberg et al.) 2014). SmoothGrad (Smilkov et al. 2017), integrated gradient (IG) (Sundararajan al. (017) to name few. They highlight fine-graines details in the image but are not class diikssiiscriminative for visual explanation. By contrast, localization approaches like class activation map (CAM) (Zhou (et al, (016)). GradCAM (Selvaraju et al. (0077), GradCAM+ (Chattopadhay et al.) 2018) and RISE (Petsiuk et al. 2018) are highly classdiisrimminative, namely, localizing image sub-regions reasoned

---

Under review as a conference paper at ICLR 2020
for a prediction class. We refer readers to Sec. for some representative interpretation methods. Besides interpreting CNNs via feature importance maps, some methods zoom into the internal re- sponse of neural networks. Examples include network dissection (Bau et al., 2017), which evaluates the alignment between individual hidden units and semantic concepts, and learning perceptually- aligned representations from robust training (Engstrom et al.
2019).
Very recently, some works (Xu et al.
2019b
Zhang et al. 2018 Subramanya et al. 2018 Ghor-
bani et al. 2019 Dombrowski et al.
2019 Chen et al. 2019) are beginning to study adversarial
robustness by exploring the spectrum between classification accuracy and network interpretability. was shown in (Xu et al., 2019b a that an imperceptible adversarial perturbation
fool classi- fiers can lead to significant change in a class-specific network interpretability map, e.g., CAM. Thus, it was argued that such an interpretability discrepancy can be used as a helpful metric to differentiate adversarial examples from benign inputs. However, the work (Zhang et al.) 2018:: Subb- ramanya et al.) 2018) showed that under certain conditions, generating an attack (which we call interpretability sneaking attack, ISA) that fools the classifier as well its coupled interpreter (in terms of keeping interpretability map highly similar to that of benign input) is not significantly more difficult than generating adversarial inputs deceiving the classifier only. Besides investigating ro- bustness in classification through the lens of interpretability, the work (Ghorbani et al., (2019; Dombrowski et al.) (019)) studied the robustness in network interpretation maps, showing that which can significantly be manipulated via imperceptible input perturbations but keeping the classifier's decision intact. We call this type of threat model attack against interpretability (AAI). The existing work had no agreement on the relationship between robustness in interpretation and robustness in classification. Spurred by that, we attempt explore this relationship from both attack and defense perspectives.
The most relevant work to ours (Chen et al. 2019), which robustified network interpretation with the aid of integrated gradient (IG), an axiomatic attribution map. It proposed robust attribution training, which was shown as a principled generalization of previous formulations of robust clas- sification and an effective defense against AAI. In this paper, we first investigate when ISA is possible, and then relate our insights on ISA to robust classification and robust interpretabil- ity. Different from the previous work, our paper contains the following contributions.
1. We provide an answer to the question of when adversarial examples can bypass in- terpretability discrepancy. We show that enforcing stealthiness of adversarial examples from network interpretation could be challenging. Its difficulty relies how one measures the interpretability discrepancy caused by input perturbations.
2. We propose an & norm based 2-class interpretability discrepancy measure and theoretically show that constraining it helps adversarial robustness.
3. We develop an interpretability-wweel interpretability alone can be used to defend adversarial attacks for both misclassifcation and misinterpretation. Compared to the IG-based robust attribution training (Chen et al. 2019) approach is simpler in implementation, and provides better robustness even as facing a strong adversary.
robust training method and empirically show that
2 PRELIMINARIES AND MOTIVATION: INTERPRETABILITY OF CNNS FOR JUSTIFYING A CLASSIFICATION DECISION
To explain what and why CNNs predict, we consider two types of network interpretation methods: a) class activation map (CAM) (Zhou al. 2016 Selvaraju al. 2017 Chattopadhay et al.] 2018) and b) pixel sensitivity map (PSM) (Smonnva et al. 2013 Springenberg et al. 2214:] Smilkov et al.) 2017 Sundararajan et al. 2017 Yeh et al., 2019). Let (x) Rec denote a CNN- based predictor that maps an input x E Red to a probability vector of C classes. Here fx), the eth element of f fx),,
denotes the classification score (given by logit before the softmax) for class c. Let L(x, c) denote an interpreter (CAM or PSM) that reflects where in contributes to the classifier's 's decision on
CAM-type methods. CAM (Zhou et al. 2016) produces a glass-discrmminative localization map for CNNs, which performs global averaging pooling over convolutional feature maps prior to the softmax. Let the penultimate layer output K feature maps, each of which is denoted by vector
2

---

Under review as a conference paper at ICLR 2020
representation A₁ R for channel & [K]. The ith entry of CAM (xAM( is given by
Here [K] represents the integer set {1,2. K}.
[xCAM c)}=
wfAA. i [u].
(I)
RE(K]]
where w is the linear classification weight that associates the channel & with the class c, and A denotes the ith element of Ag. The rationale behind (T) is that the classification score fc(x) can be written as the average of CAM values (Zhou et al. 2016), fc(x) Σ [LCAM c)]. For visual explanation, LCAM(X. e) often up-sampled to the input dimension d using bi-linear interpolation.
GradCAM (Selvaraju et al. 2017) generalizes CAM for CNNs without the architecture 'global av- erage pooling - softmax layer' over the final convolutional maps. Specifically, the weight w in (1) is given by the gradient of the classification score fa(x) with respect to (w....) the feature map Ak. " 1 Σ==1 0x.(x), GradCAM++ (Chattopadhay et 2018), generalized formulation of GradCAM, utilizes a more involved weighted average of the (positive) pixel-wise gradients but provides better localization map if an image contains multiple occurrences of the same class. In this work, we focus on CAM since it is computationally light and our models used in experiments follow the architecture "global average pooling - softmax layer'.
AAA.
PSM-type methods. PSM uses calculations with gradients to assign importance scores to individ- ual pixels toward explaining the classification decision about an input. Examples of commonly used approaches include vanilla gradient (Simonyan al. 2013), guided backpropogation (Springenberg al. 2014), SmoothGrad (Smilkov et al. 2017); and integrated gradient (IG) (Sundararajan
al.,
2017). In particular, IG satishes the completeness attribution axiom that PSM ought to obey. Specif- ically, it averages gradient saliency maps for interpolations between an input x and a baseline image
[La (x, c)}: = (æ: ai) S 0fc(a + a(x - a)) da 20 (s: 0s) Ox,
Øfe(a + _ (x-aa)
a)) 1 m
i [d].
Oz.
(2)
where m is the number of steps in the Riemman approximation of the integral. The completeness axiom (Sundararajan et al., 2017 Proposition 1) states that Σ (TGG(xx where the baseline image is often chosen such that fc(a) R 0, e.g., the black image. Note that CAM also satisfies the completeness axiom. PSM is able to highlight fine-grained details in the image, but is computationally intensive and not very llassdiiscimminaive compared to CAM Selvaraju al. 2017).
fa(x) (a),
Interpretability discrepancy caused by adversarial perturbation. Let = x Ô represent an adversarial example w.r.t. x, where 5 denotes an adversarial perturbation. By replacing the input image x with x', the CNN will be fooled from the true label & to the target (incorrect) label t'. It was recently shown in (Xu et al., 2019b a) that the adversary could introduce an evident interpretability discrepancy w.r.t. both the true and the target label in terms of L(x,t) vs. L(x',t'). ,. An illustrative example is provided in Figure We see that an adversary suppresses the network interpretation w.r.t. the true label but promotes the interpretation w.r.t. the target la- bel. We also observe that compared to IG, CAM and GradCAM+ better localize class-specific discriminative regions. These results reveal two observations how measuring interpretation discrepancy affects classification robustness: a) interpretability discrepancy may be used to detect adversarial examples, interpretability discrepancy itself may be vulnerable adver- sarial perturbations. In what follows, we explore the spectrum between adversarial robustness and interpretability from a unified perspective considering both the adversarial vulnerability of interpretability discrepancy and the value of interpretability discrepancy in defense
t) vs. L(x', t), and L(x, t')
3 ROBUSTNESS IN CLASSIFICATION VS. INTERPREATION MAP: AN ATTACK PERSPECTIVE
In this section, we examine two types of threat models, interpretabilityy sneaking attack (ISA) and atlack against interpretability (AAI). Since an adversarial example designed for misclassification
3

---

Under review as a conference paper at ICLR 2020
Input Image
CAM
GradCAM+*
IG
46.4
Original Original caampe
Adverranall cuuppees"
comrelatoon: 0,4782 correlatio:: 0.5039
correlatio:: 0.5013 correlmtio: 0.5472
conrelation: 0.40ND conrelation: 0.3911
Interpretation (I.) of benign (x) and adversarial (x') image from restricted ImageNet (Engstrom al. 2019) with respect to the true label t= t='monkey' and the target label t'='fish" Here the adversarial example is generated by 10-step PGD attack with perturbation size 0.02 (Madry et al. 2018). and we consider three types of interpretation maps, CAM, GradCAM+ and IG. Given an interpretation method, the first column is T(xx, t) versus [(x', f), the second column is [xxx, t') versus [(x', t'), and all maps under each category are normalized w.r.t their largest value. At the bottom of each column, quantify the resulting interpretability 2017 between every pair of L(x, i) and
Figure 1:
discrepancy by Kendall's Tau order rank correlation (Selvaraju [xx', i)) for
gives rise to interpretability discrepancy (which could then be used as detector for the adversarial input), the problem of ISA arises: One may wonder whether or not it is easy to generate adversarial examples that mistake classification but keep interpretation intact. Such adversarial vulnerability could have serious consequences when classification and interpretation are jointly used in tasks like medical diagnosis (Subramanya al., 2018), and call into question the faithfulness of interpretation to network classification. One the other hand, it is suggested from interpretability discrepancy that an interpreter could be quite sensitive to input perturbations. Thus, the problem of AAI arises: One may wonder if perturbed inputs could induce differing explanations but without altering predictions.
3.1 RETHINKING ISA FROM PERSPECTIVE OF INTERPRETABILITY DISCREPANCY
Previous work Zhang et al. 2018 Subramanya al., 2018) showed that it is not difficult hide adversarial examples from network interpretation when the interpretability discrepancy is measured w.r.t. a single class label (either the true label / or the target label ('). However, we see from Figure[ that the adversary (against classification) alters interpretability maps w.r.t. both t and t'. This motivates us to rethink whether the single-class interpretability discrepancy measure is proper, and whether ISA is truly easy to bypass an interpretability discrepancy check.
We consider the following generic form of interpretability discrepancy
D (x,x''
|C| Σ Lxx, i) L(x L(x, 10),,
(3)
where recall that x and x' are natural and adversarial examples respectively, represents an inter- preter, e.g., CAM or IG, C denotes the set of class labels used in L, [C] is the cardinality of C, and we consider p E (1,2) } in this paper. Clearly, a specification of (3) relies on the choice of C and p. We specify with = t') and p = 1, leading to E 2-class interpretability discrepancy measure,
Da.5 (x,x)) = ([] L(x' .01, [[((x.)) - L(xx(())..
1
Rationale behind é 2-class discrepancy measure. Compared to the previous work (hhang et al., 2018 Subramanya et al. 2018) using a single class label, we choose = (4, L' H motivated by the fact that an interpretability discrepancy occurs w.r.t. both l. and L' (F. Moreover, although Euclidean distance (namely, l norm or its square) is arguably one of the most commonly-used dis- crepancy metrics (Zhang et al., 2018),
show in Proposition[]] that the proposed interpretability discrepancy measure D2.6, (x, x') has a non-trivial lower bound for any successful adversarial at- tack. This provides explanation why it could be difficult to hide adversarial examples from
In addition to the 2-class case, our experiments will also cover the all-class case C [C].
4

---

Under review as a conference paper at ICLR 2020
network interpretability methods. Moreover, l is upper bound of the &g norm and promotes the sparsity of interpretability discrepancy, which enforces pixels of L to stay intact when facing input perturbations.
Proposition 1. Given a classifier s(x) E Re and its interpreter L(x, c) for e E [C]. suppose that the interpreter satisfies the completeness axiom, namely, EdLxx, fx)x). For natural example x and an adversarial example x' with prediction I. and 1' respectively, D2,6, (x,x') in ④ has the lower bound
1 D2.1, (x,x)2 20)()) (((x))fe(x)).
(5)
Proof: See proof in Appendix
Proposition I connects 22.6, xx) with the classification margin f((x) - fx Thus, if a classi- fier has large classification margin on the natural example x, it will be difficult to find successful adversarial attack with small interpretability discrepancy. In other words, constraining the inter- pretability discrepancy prevents misclassification since generating adversarial examples becomes infeasible under 22,0, (x, x') < ( fx Also, the completeness condition of L sug- gests specifying (4) with CAM ( IG 2 Here we focus on CAM due to its light computation.
Design of ISA. We pose the following optimization problem for design of ISA, which not only fools a classifier's 's decision but also minimizes the resulting interpretability discrepancy,
minimize
À max fx(x 8) fe (x 8), -7} D2.6, 6x
(6)
subject to || S,
where À >>> 0 is regularization parameter that strikes a balance between the success of an attack and its resulting interpretability discrepancy, T 2 O (e.g., 0.1 used in the paper) is a tolerance on the classification margin of a successful attack between the target label t' and the non-target top-1 prediction label arg maxj#r fy(x + 8). D2,, was defined by (4. and e > 0 is a (pixel- level) perturbation size. In (6) the first term of the objective corresponds loss (Carlini & Wagner 2017). which reaches -r if the attack succeeds in misclassification. To find ISA of minimum interpretability discrepancy, we perform a bisection on À until there exists no successful attack that can be found when À further decreases. Although we focus on foo attack in (6. but similar formulation applies to & and lz attacks. Attacks are found using PGD, with sub- gradients taken at din-differentiable points. We consider only targeted attacks to better evaluate the effect on interpretability of target classes, although this approach can be extended to an untargeted setting (e.g.. by using the target label-free interpretability discrepancy measure introduced in the next section).
a W- attack
Difficulty of generating ISA versus interpretability discrepancy measure. Through an illustra- tive example in Figure we elaborate on how the choice of interpretability discrepancy measure plays crucial role on the difficulty of hiding adversarial examples from network interpretation. We refer readers to Sec.5) for more experimental results. We generate ISA under different specifications of (3) for different values of perturbation size e. Compared to f,162 1-class (true class t), &g 2-class, and fj/l2 all-class, we see that the & 2-class interpretability discrepancy (4) cannot be easily miti- gated even as the attack power (in terms of c) increases. This is verified by a) high interpretability (c)
discrepancy score and b) its flat slope of discrepancy score against in Figure (a)& further shows CAMS of adversarial examples w.r.t. the true label 1 and the target label l' generated by & 1/2/all-class ISAs. We note that both 1-class measure and all-class measure could give a false sense of ease of hiding adversarial examples. For & 1-class ISA, although the interpretability discrepancy w.r.f t is minimized, the discrepancy w.r.t. t' remains large, supported by the observation that E 1-class ISA even yields a smaller correlation between L(x', t') and L t') than PGD attack. Similarly, although the averaged discrepancy over all classes is minimized for & all-class ISA, discrepancies w.r.t. specific labels such as f and do not necessarily re- duce. This illustrates that although ISA may be found, with the proper choice of discrepancy measure, ISA with low discrepancy becomes quite difficult.
3.2 ATTACK AGAINST INTERPRETABILIT (AAI)
Different from ISA, AAI produces input perturbations to maximize the interpretability discrepancy while keeping the classification decision intact. Thus, AAI provides means to evaluate the ad-
5

---

Under review as a conference paper at ICLR 2020
1e4
Squared I, CAM Discrepancy
CAM Disereper eeereppeny
twie closs
1.2
Two class
7-
one class
1.0
one class
12.8
6-
0.6
0.4
0.2
0.002
0.004
(a)
(b)
original image x
10-tlop PGD atteck x
2-class 1SA
4 I L-clas ISA
all cluss ISA
46.0
0.5213
0,5342
correaaiinn 0.5039
correlation: 0.5416
correlation: 0.4129
correlation: 0.5561
Seeing the effect of discrepancy measure on hiding adversarial examples from network interpreta- tion. The same benign image in Figure is considered. (a) ISA using CAM-based f 1/2/all-class discrepancy measure versus perturbation size e, (b) ISA using CAM-based squared & 1/2/all-class discrepancy measure versus e, (c) CAM interpretation different specifications of ISA. All interpretation maps normalized w.r.t. their largest value. At the bottom of each interpretation map L(x', -). we quantify the interpretability discrepancy by Kendall's Tau order rank correlation between every pair L(x', 1) and L(x, for i {t, f'}, where x' is obtained from PGD attack or each specification of ISA.
Figure 2:
example in Figure its adversarial counterparts from PGD attack and
versarial robustness in interpretability. Since t = arg max; fi(x) arg max; f((x') f((x')) = t' in AAI, the 2-class interpretability discrepancy measure (4 reduces to its 1-class version. The problem of generating AAI then cast
minimize 8 subject
A imax/maxxst fx(x -) (x,
||8||~0 Se
where the first term is hinge loss to enforce fe(x + 8) 2 maxj41 ff + 8), namely, arg max; fi(x) = (unchanged prediction under input perturbations), and Dj denotes 1-class interpretability discrepancy measure, e.g., Di.e, from { or the top-k pixel difference between in- terpretability maps (Ghorbani et al. 2019). Similar (6). the regularization parameter À in ( strikes balance between stealthiness in classification and variation in interpretability. Experiments Sec.5 5 show that the state-of- of the-ar art defense methods against adversarial examples do not nec- essarily preserve robustness in interpretability increases, athough the prediction is not altered.
4 INTERRRETABILIT-AWAANR ROBUST TRAINING: DEFENSE PERSPECTIVE
recall from Sec. Sec.3.1 that adversarial examples that intend to fool a classifier could be difficult to evade the & 2-class interpretability discrepancy. Thus, constraining the interpretability discrepancy helps to prevent misclassification. Motivated by this observation about ISA, we introduce interpretability based defense method that penalizes interpretability discrepancy to achieve high classification robusiness. As an additional benefit, also robustifies the classifier against AAI, where the adversary maximizes the interpretability discrepancy.

---

Under review as a conference paper at ICLR 2020
Target label-free interpretability discrepancy. Different from attack generation, the f 2-class discrepancy measure (4) cannot directly be used by defender since the target label t' specified by the adversary is not known a priori. To circumvent this issue, we propose to approximate the in- terpretability discrepancy w.r.t. the target label by weighting discrepancies from all non-true classes according to their importance in prediction. This modifies (4) to
D (x,x')=
- 1 T(xx,
,1)1 +
Σ 6791
e((o E, ex(o) [I,(x,)) ((xx,))l.
(8)
where the softmax function e(x') Σ, E,(e) adjusts the importance of non-true labels according to their classification confidence. Clearly, when x' succeeds in misclassification, the top-1 predicted class of x' becomes the target label and the resulting interpretability discrepancy is most penalized.
Interprefahility-awar robust training. We propose to train a classifier against the worst-case interpretability discrepancy (8, yielding the min-max optimization problem
minimize ((x,t))PPrain fteai (0; x, t) +
maximize D (xxxx) x': xx
(9)
where 0 denotes the model parameters, Dursin denotes the training dataset, furain is the training loss (e.g., cross-entropy loss), 7 >> O is a regularization parameter, and for ease of notation we omit the parameters A and t in D (x, x'). Here y controls the tradeoff between clean accuracy and robustness; see Appendix [ for experiments analyzing this tradeoff.
In problem (9), the inner maximization is only used to evaluate the worst-case interpretability dis- crepancy. Thus, is different from adversarial training (Madry et al. 2018), where the training loss is replaced with the adversarial loss maximizex::ll>'-> - Se Jtrain (0, x l). The formulation (9)
allows us to examine whether or not robust interpretation is directly beneficial to robust classifica- tion. For completeness, with the use of the adversarial loss.
will also provide experiment results on modified formulation of (9)
Difference to (Chen et al. 2019). The recent work (Chen et al. 2019) proposed improving adver- sarial robustness by leveraging robust IG attributions. However. different from (Chen et al. 2019), our approach is motivated by the importance of l 2-class interpretability discrepancy measure. We will show in Sec.51 [3] that the incorporation of interpretability discrepancy w.r.t. target class labels, namely, the second term in (8) plays a key role in boosting classification and interpretation robust- ness. This is because robust interpretability implies robust classification only when interpre- tation maps are measured with a proper metric. We will also show that our proposed method is sufficient to improve adversarial robustness even in the absence of adversarial loss, while the robust attribution regularization method (Chen et al., 2019) becomes ineffective when the attack becomes stronger. Last but not the least, beyond IG, our proposed theory and method apply to any network interpretation method with completeness axiom. The use of CAM avoids Riemman approximation used in IG and thus simplifies the implementation during robust training.
EXPERIMENTS
In this section, we empirically show the effectiveness of proposed methods in various attack and defense settings. For ISA, we examine how the interpretability discrepancy measure plays a role in hiding adversarial examples from network interpretation. For AAI, we evaluate its attack success rate under the natural and various robust models. For interpretabilityawwan we demonstrate its advantages in a) defending against PGD attacks with different steps and pertur- bation sizes (Madry et al. 2018: Athalye et al.) 2018)), b) defending against unforeseen adversarial attacks (Kang et al.) 2019). c) rendering robustness in interpretability, and d) computation efficiency compared to the based robust attribution regularization method (Chen et al. 2019).
robust training,
Datasets, CNN models, and experiment setting. We evaluate networks trained on the MNIST, CIFAR-10 10 and restricted ImageNet ImagNeeet dataset used in (Engstrom al. 2019). We consider three models, Small (for MNIST and CIFAR), Pool (for MNIST) and WResnet (for CIFAR and R-ImageNet): 1) a small CNN architecture consisting of three convolutional layers of 16, 32
7

---

Under review as a conference paper at ICLR 2020
and 100 filters (Small), 2) CNN architecture with two convolutional layers of 32 and 64 filters each followed by max- -pooling which is adapted from (Madry et al. 2018) (Pool), and 3) a Wide Resnet from (Madry et al. 2018) (WResnet). We refer readers to Appendix B for more details.
In the following experiments, we consider 5 baselines from the literature: i) standard training (Nor- mal), ii) adversarial training (Adv) (Madry et al. 2018), iii) TRADES (Zhang al. 2019), iv) IG-Norm that uses IG-based robust attribution regularization (Chen et al., 2019). v) IG-Sum-Norm (namely, IG-Norm with adversarial loss). Additionally, we consider 3 variants of our method: vi) the proposed interpretability-awaa robust training method (9) (we call Int), vii) Int using & 1-class discrepancy (Int-Icllass, and viii) Int with adversarial loss (Iut-Adv). For interpretability-unne methods, the regularization parameter y is set to 0.01. In Appendix [ settings of the parameter y for our method to demonstrate that we can achieve a range of different points on the robustness- tradeoff.
we explore different
Ditaset
Interpretation
Fs morm
Fg nomm
method
I-class
2-class
cllsclass
I-class
2:class
all class
CAM GradCAM+
3,07230,0810
3.2672/0.0233
2.5289010444
O 0,305TKI 1505
0.565470.6121
0.43200 (455
MNIST
IG
30304000330
12244800814
6,78940.0238
3.18670.0221
2.839480.0566
4.3667002514 W/a
0.330000 1447
0.447680.0022
0.5766/0.004
0.5531/0.0289
0.2100000337
0,4922014456
Repr
W/
2.3668/0.0404
0.4129/0.0429
WVA
CAM 1452
2,502000.0499
1 1.7898000774
0.131340.2369
0.361340.0668
0.2746000000
CIFAR-10
GraICAM+
IG
4.999900188
14355001490
4.97940.0177
2.478%0.0313
2.8548001336
0.12300.01 WV
O 0.1355012346
0. 0.3577/1-6676
0.130900.0092
O 0.2758007769
0.05780.023
Repr
1.7049/0.0785
288/08000666
IVa
CAM
49 18.2860 1005
61.975000031
8777005577
1 9933300 1526
2. 2,6036/0.0791
2,0055000865
R- ImageNNe
GraICAM++
Repr
39. 70.76111110 1028
50. 04330330
46.892000657
42,
1 1969
2. 0891
2.0730000078
2.1151/0.0896
Table I: NDS and NSL (format given by NDS/NSL) of successful ISAs generated under different specifi- cations of interpretbiility discrepancy measure (3) and datasets MNIST, CIFAR-10 and R- R-ImageNet. Here a discrepancy measure with large NDS and small NSI. indicates a strong resistance to ISA.
Evaluating ISA. We evaluate the effect of interpretability discrepancy measure on ease of finding ISAs. Spurred by F such an effect is quantified by calculating minimum discrepancy required in generating ISAs against different values of perturbation size e in (6). We conduct experiments over 4 network interpretation methods: i) CAM. ii) GradCAM++, iii) IG, and iv) internal representation at the penultimate (pre-softmax) layer (denoted by Repr). In order to fairly compare among different interpretation methods, we compute normalized discrepancy score (NDS) extended from (3) and normalized slope (NSL) that measures the relative change of NDS for c E [č, c). We refer readers to Appendix D for more details.
In Table we present NDS and NSL of ISAS generated under different realizations of interpretabil- ity discrepancy measure 33, each of which is given by a combination of interpretation method (CAM, GradCAM++. IG or Repr). Lp norm (p E {1, 2)) and number of interpreted classes. Note that Repr independent of the number of classes, and thus we report NDS and NSL correspond- ing to Repr in the 2-class column of Table Given an Ep norm and an interpretation method, we consistently find that the use of 2-class measure achieves the largest NDS and smallest NSL the same time. This implies that the 2-class discrepancy measure increases the difficulty of ISA to evade network interpretability check. Moreover, given a class number and an interpretation method, we see that NDS under & norm is greater than that under l norm, since the former is naturally an upper bound of the latter. Also, the use of & norm often yields a smaller value of NSL, implying that the L fo-rrmm based discrepancy measure is more resistant to ISA. Furthermore, by fixing the combination of & norm and 2 classes, we observe that IG is the most resistant to ISA due to their relatively high NDS and low ISA, and Repr yields the worst performance. However, compared to CAM, the computation cost of IG increases dramatically as the input dimension, the model size, and the number of steps in Riemman approximation increase. We find that it becomes infeasible to generate ISA using IG for WResnet under R-ImageNet within 200 hours.
Evaluating AAL We specify problem 7 under the top-k attack setting GGhoranni et al. 2017 where the top-k intersection between the original and adversarial interpretability maps is minimized. The strength of AAI is then measured by the Kendall's Tau order rank correlation between the aforementioned two interpretability maps (Chen et al.; 2019). The higher the correlation is, the more robust the model is against AAI. Reported rank correlations are averaged over a test set.
8

---

Under review as a conference paper at ICLR 2020
In Table we present the performance of AAI under multiple perturbation sizes to attack mod- els trained using different training methods. For the model Small under MNIST, we evaluate AAI over 5 baselines (Normal, Ady, TRADES, IG-Norm, IG-Sum-Norm) and 3 variants of our method (Int, Int-I-class and Int-Adv). For a larger model WRes- net under CIFAR-10, the training methods using IG (IG-Norm and IG-Norm-Summ Sum) are excluded due to their prohibitive computation cost. The insights learnt from T are summarized as below. First, the normally trained model (Normal) does not auto- matically give robustness guarantees in interpretabil- ity, particularly for AAI with 2 0.2. Second, the methods Adv, TRADES, IG-Sum-Norm and Int- Adv that uses adversaria loss offer certain robust- ness against AAI but the performance gets worse as the perturbation size increases. Third, in the ab- sence of adversarial loss, the baseline IG-Norm be- comes less robust increases. By contrast, our proposed method Int consistently more robust and its advantage becomes more evident as increases, including when e is increased beyond the value used for training.
Method
<= 0.05
0.1
0.2
0.3
0.35
0.4
MNIST:
Small
Normal
0.007
0.797
0.366
0.085
0,855
0.085
Adc
0.978
0.955
0.910
0.857
0.67
0.136
TRADES
(L978
0.55
0.905
0.547
11400
6.115
IG-Norm
0U555
0.991
0.662
0.278
0098
0,093
IG Norn Sum
01976
0.511
0.901
0,30
0659
0,589
Int
0.983
0.968
0.941
0,913
0.261
0.078
Int-Ady
(L980
0.965
EL936
0,912
0.50
0.331
Int-I-class
0.874
0.818
0.754
0.992
0.527
0.348
< 2/285
4/255
62555
8/255
9/255
10/255
CIFAR, Wiooodd
Numaa
0.595
0.067
-0.000
-0.14
Adl
IL912
0.734
0.629
0.600
TRADES
0.918
0.747
0.652
0.619
0,578
0,899
0.763
0.746
0.682
0.679
0.661
Int-Adv
0.883
0
0.751
0.696
0.663
0.640
Table 2: Performance of AAI for different values of perturbation size in terms of Kendall's Tau der rank correlation between the original and versarial interpretability maps. Best robustness re- sults (corresponding to highest correlation values) are highlighted (1st, 2nd, 3rd) under each column of dataset- model pair.
robust training. We previously showed in Table that interpretabilit-awaar robust training (Int, Int-Adv) often achieves more robust interpretability than state-of-the -fr adversarial training methods particularly for large perturbation size. We next provide a thorough evaluation on how interpretability helps adversarial robustness against misclassification.
Evaluating interpretability-awee
Robustness versus efficiency. In Figure 3 we
present the training time (left y-axis) and the ad- versarial test accuracy (right y-axis) for differ- ent training methods (x-axis), which are ranked in decreasing order of computation complex- ity. Here the adversarial test accuracy (ATA) is measured using 200-step (norm) perturbation sizes = 0.3 and 0.4 on the Small MNIST model (Madry et al., 2018). Note that all AT-type methods (GG-Norm-Sum Int-Ady, TRADES and Adv) offer robust classification at c = 0.3 with ATA around 80%. Among non-AT but interpretability pro- moted defensive schemes (IG-Norm, Int, Int-1- class), we find that only the proposed Int yields competitive ATA, and outperforms all AT-type methods except Int-Adv at 0.4. Particularly, IG-Norm degrades significantly in robust classification when PGD attack becomes stronger (also see results in T Moreover, the non-robustness of Int-I-class verifies the importance of 2-class interpretability discrepancy measure on rendering robust classification. Last but not the least, Adv, TRADES and Int-based methods have similar computational complexity, but IG-based methods make training time (per epoch) significantly higher, 3 times more than Int-Ady even under the Small MNIST model.
000
3 500-
Advernarel TiME Acoursy
400
Twising Time pe
300
-38
200
300
0.0
G-AoormSuu Sum
inb Nomm
ind
Inc -ias
TRADES
Hormar
Figure 3: Computation time per epoch and ad- versarial test accuracy for Small MNIST model trained with different methods.
Int does not cause obfuscated gradients and importance of 2-class measure. It was shown in (Athalye et al. 2018; Carlini, that some defense methods cause obfuscated gradients, which give a false sense of security. There exist two characteristic behaviors of obfuscated gradients: a) One-step attacks perform better than iterative attacks; b) Increasing distortion bound does not increase success. Motivated by that, we evaluate our interpretabilit--wware robust training methods under PGD attacks with a) different steps and b) different perturbation sizes. Table reports ATA of interpretability-wvar 200-step PGD attacks are conducted for E {0, 0.05, 0.1, 0.2, 0.3, 0.35, 0.4) for MNIST and (0. 2/255, 4/255, 6/255, 8/255, 9/ 9/255, 10/255) for CIFAR. We use 0.3 on MNIST
robust training relative to various baselines, where
9

---

Under review as a conference paper at ICLR 2020
and 8/255 CIFAR for robust training. As we can see, ATA decreases as increases. Thus, our methods do not exhibit the behavior a) of obfuscated gradients. We also observe that com- pared to Adv and TRADES, Int achieves reasonable but worse ATA as < 0.3 MNIST and S 8/255 on CIFAR. As used in PGD attack achieves the value used for robust training, Int achieves ATA 0.790 against ATA 0.890 from Ady on MNIST, but outperforms Adv on CIFAR (0.270 vs 0.170). Interestingly, the advantage of Int becomes more evident as the adversary becomes stronger, L..., E > 0.3 on MNIST and > 8/255 on CIFAR. We highlight that such a robust classification is achieved by promoting robustness of interpretability alone (without using adversarial loss). worth mentioning that IG-Norm fails to defend PGD attack with 0.3 for the Small MNIST model. We further note that Int-l-class performs much worse than Int, supporting the importance of using 2-class discrepancy measure (see Prop.[). Besides robust classification, IG-Norm and Int-lsclass are also not sufficient to render robustness in interpretation (Table
T shows ATA of interpretability-awaaa robust training against k-step PGD attacks (examining behavior b) of obfuscated gradients), where & E {1,10 10, 100, 200}. As we can see, ATA decreases as k increases. This also suggests that the high robust accuracy from our methods is not a result of obfuscated gradients. Similar to Table 3 we that compared to Int, IG-Norm and Int-l-class are insufficient to defend PGD attacks with h 2 100.
Beyond too norm PGD attacks. In 5 we present ATA of interpretabillty-aweee aware robust train- ing and various baselines for defending attacks (Gabor, Snow, JPEG fx., JPEG f2, and JPEG 4 recently proposed in (Kang et al.] 2019). These attacks are called 'unforseen attacks' since they are not met by PGD-based robust training and often induce larger perturbations than conventional PGD attacks. For robust training methods without resorting adversarial loss, we find that Int sig- nificantly outperforms IG-Norm especially under Snow and JPEG lp attacks. Int also yields quite competitive robustness compared AT-type methods on most attacks.
Method
C-O
0.1
Method
Steps
10
100
200
MNIST.
MNIST, Small, c - 0.3
Normal
Adv
TRADES
0,560
0.920
1,400
0.970
0.960
0.530
0.940
0.955
0.045
0.930
0,000
0.925
0.890
0.885
0.000
0.000
0.010
0.000
0.000
0,000
Normal
Adv
0.990
0.975
0.070
0.000
0.890
0,000
IG-Meem
0.965
(,950
0.895
0.410
0005
0.000
0.000
TRADES
0.970
0.955
0.885
0.885
IG Norm Sum
0,975
0.955
0.935
0.910
0.850
0.115
0,000
IG-Norm
0.970
0.905
0.005
0.005
Int class
0,975
0.635
0.330
0.140
0.125
0.115
0.880
[G-Norm- Sum
0.970
0.940
0.580
0.880
Int
0,935
0.530
0.905
0,840
0.79)
0.180
0.140
Ini-I-class
0.950
0.125
Int-Ady
0,950
0.955
0.905
0,880
0,855
0.355
0.178
Int
0.935
0.910
0.790
0.790
C-0
2/255
4/255
6/255
8/255
91255
102258
Int-Adv
0.950
0.905
0.855
0.855
CIFAR WRooor
CIFAR, Wresnet,
8/255
0.485
0.085
Normal
0.470
TRADES
0.460
0.205
0.100
Adv
0.590
0.205
0.188
0,185
0.455
0.210
TRADES
0.590
0.180
0.165
0.165
Int Int-Ady
0.065
0.555
0.510
0.385
0.339
0.560
0.280
Int
0.620
0.310
0.275
0.275
0.685
0,565
0.360
0.190
0.040
0.025
Int-Ady
0.580
0.335
[nt-] lalss-class
0.505
0.100
0.060
0.060
Table 3: 200-step PGD accuracy for different val- ues of perturbation size e. Best ATA results are highlighted (Ist, 2nd, 3rd) at each column. Note that ATA with reduces natural accuracy.
Table 4: Multi-step PGD accuracy. Best ATA results are highlighted (1st, 2nd, 3rd) at each column.
CONCLUSION
Method
Gabor
Snow
JPEG f.
JPPG fg
JPEG f.
In this paper, we investigate the connection be- tween network interpretability and adversarial robustness. We show theoretically and empiri- cally that with the correct choice of discrepancy measure, it difficult to hide adversarial exam- ples from interpretation. We leverage this dis- crepancy measure to develop interpretability- aware robust training method that displays 1) high classification robustness in a variety of set- tings and 2) high robustness of interpretation.
CIFAR- 10, Small
Normal
0.000
0.030
0.000
Adv
0.190
0..155
(1.660
0.380
0,230
TRADES
0.220
0.425
0.300
0.070
IG-Norm
0.155
0.015
0.000
0.000
0.000
IG-Norm-Sum
0.185
0.110
0.480
0.375
0.215
Int
0.160
0.105
0.440
0.345
0.260
Int-Adv
0.150
0.120
0.340
0.310
0.235
Table 5: ATA different unforeseen attacks in (Kang)
al. 2019). Best results in each column are high-
lighted (Ist, 2nd, 3rd)
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust ad adversarial examples. arXiv preprint (7X1::1707003997 2017.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense se- curity: Circumventing defenses to adversarial examples. ICML, arXiv preprint arXiv:1802.004200 2018.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 6541-6549, 2017.
Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint 171.::::11..0065,, 2017.
Nicholas Carlini. Is ami (attacks meet interpretability) robust to adversarial examples? arXiv 2019.
preprint arXiv:1902.022
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (SP), pp. 39-57, IEEE, 2017.
Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian. Grad- CAM++ Generalized madient-based visual explanations for deep convolutional networks. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 839 847. IEEE, 2018.
Jiefeng Chen, Xi Wu, Vaibhav Rastogi, Yingyu Liang, and Somesh Jha. Robust attribution regular- ization, 2019.
Pin-Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich. Ead: clastic-net attacks to deep neural networks via adversarial examples. AAAI, 2018.
Ann-Kathrin Dombrowski, Maximilian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. arXiv preprint arXiv: /906.0988, 2019.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Learning perceptually-alinnne representations via adversarial robustness. arXiv preprint arXiv:1906.00945. 2019.
K. Eykholt, I. Evtimov, E. Fernandes, B. Li, A. Rahmati, C. Xiao, A. Prakash, T. Kohno, and D. Song. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1625-1634, 2018.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. arXiv preprint 1710. 2017.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3681-3688, 2019.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, arXiv preprint arXiv:1412.6572, 2015.
Daniel Kang, Yi Sun, Dan Hendrycks, Tom Brown, and Jacob Steinhardt. Testing robustness against unforeseen adversaries. arXiv preprint mwxXiv:/08..000 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ICLR, arXiv preprint arXiv: 106.017,, 2018.
Seyed-Mohsen Deziovll,,,,, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9078-9086, 2019.

---

Under review as a conference paper at ICLR 2020
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson. Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In IEEE European Symposium Security and Privacy (EuroS&P), pp. 372-387. IEEE, 2016a.
Nicolas Paperno, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP). pp. 582-597. IEEE, 2016b.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint (Tf.:/800.0722,, 2018.
Andrew Slavin Ross and Finale Doshi- Velez. Improving the adversarial robustness and interpretabil- ity of deep neural networks by regularizing their input gradients. In Thirty-second AAAI confer- ence on artificial intelligence, 2018.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:13/2.6034 2013.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint 1rXiv:1706.03825, 2017.
Jost Tobias Springenberg. Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: 1412,6806 2014.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy?-a comprehensive study on the robustness of 18 deep image classification models. arXiv preprint 1exivv1808.. 01688, 2018.
Akshayvarun Subramanya, Vipin Pillai, and Hamed Pirsiavash. Towards hiding adversarial exam- ples from network interpretation. arXiv preprint arXiv:1812.0284. 2018.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference Machine Learning- Kaarning-Volume 70, pp. 3319- 3328. JMLR. org, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. ICLR, arXiv preprint arXiv:1312.6199 2014.
Kaidi Xu, Sijia Liu, Gaoyuan Zhang, Mengshu Sun, Pu Zhao, Quanfu Fan, Chuang Gan, and Xue Lin. Interpreting adversarial examples by activation promotion and suppression. arXiv preprint arXiv: NKK4:999400005 2019a.
Kaidi Xu, Sijia Liu, Pu Zhao, Pin- Yu Chen, Huan Zhang, Quanfu Fan, Deniz Erdogmus, Yanzhi Wang, and Xue Lin. Structured adversarial attack: Towards general implementation and better interpretability. In International Conference on Learning Representationss 2019b.
Chih-Kuan Yeh, Cheng- Yu Hsich, Arun Sai Suggala, David Inouye, and Pradeep Ravikumar. On the (in)fidelity and sensitivity for explanations, 2019.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818 08-3..3.35 Springer, 2014.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv:1901.08573 2019.
Xinyang Zhang, Ningfei Wang, Shouling Ji, Hua Shen, and Ting Wang. Interpretable deep learning under fire. CoRR, abs/181 2,00891 2018. URL http://tt/////xiv.oo http://xxxxw..oo///baaa////a/a/s 00891
12

---

Under review as a conference paper at ICLR 2020
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discrim- inative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921-2929, 2016.
13

---

Under review as a conference paper at ICLR 2020
APPENDIX
A PROOF OF PROPOSITION
For Ve [C], by the completeness axiom we have fe(x) Ex c)}; Using this fact, we obtain that
1x(x')_f(xx) Σ
Σ [[L(K, 17), [I(x, 1 f(x,
(10)
Similarly, we have
f((x) f((x) [|) L(x' t)]l
(11)
Adding 10 and 11 rearranging yields
(xe [x(') + [/((x) - (x)) & [[ 11) - T(x, 11) + [[((x()) - 1(x, (12)
Since (x') - ft(x) 2 0, we then have [[((x()) - L(x, + +)]l++++ L(xxx 2 f((x) fe (x), which is equivalent to (5)
14

---

Under review as a conference paper at ICLR 2020
B ADDITIONAL DETAILS ON MODEL AND DATASET
The considered network model has a global average pooling layer followed by fully connected layer at the end of the network. For R-ImageNNet, we use only a normally trained network, used for evaluating ISA. During adversarial training of all baselines, 40 adversarial steps are used for MNIST and 10 steps for CIEAR. To ensure stability of all training methods, the size of perturbation is increased during training from 0 to final value of 0.3 on MNIST and 8/255 on CIFAR. MNIST and R- R-ImageNet networks are trained for 100 epochs and CIFAR networks are trained for 200 epochs. A batch size of 50 is used for MNIST, 128 for CIFAR and 64 for R- Imagenet. For all methods, training is performed using Adam with initial learning rate of 0.0001 for MNIST and 0.001 for CIFAR and R- I-mmaggnet, with the learning decayed by x1/10 at training steps 40000 and 60000 for CIFAR and 8000 and 16000 for R- E--mmgenet.
15

---

Under review as a conference paper at ICLR 2020
C EXPERIMENTS ON REGULARIZATION PARAMETER 7
We conduct experiments for evaluating the sensitivity of the regularization parameter in our propased apporoach (namely, Int) under a Small MNIST model. Adversarial test accuracy (ATA) and clean test accuracy results are plotted in Figure[AT] As illustrated, using different values of the hyperparameter controls the tradeoff between clean accuracy and ATA, with smaller y yielding higher clean accuracy, but lower ATA (a value of ? normal training). We note that with the model tested, ATA stops increasing at value of y 0.01. Beyond this value, clean accuracy continues decrease while ATA slightly decreases. These results indicate that by choosing an appropriate 7, it is possible to smoothly interpolate between normal training and maximally robust Int training.
corresponds to
1.00
0.8
0.7
0.98
Clean Accuracy
0.96
0.5
0.94
0.4
Adverserial Teat
0.3
0.90
0.88
0.0
0,000
0,005
0.010
0.015
0.020
0,025
0.030
Figure AI: Clean test accuracy and adversarial test accuracy for a Small MNIST model trained with Int using different values of regularization parameter
16

---

Under review as a conference paper at ICLR 2020
D DETAILS ON NDS AND NSL
In order to fairly compare among different interpretation methods, compute normalized discrepancy score (NDS) extended from 33: Dnorm
[c] Eiec maxy ((x.)yy-xmm ming L(x,))
-(x.i)-((xx.)
A smaller value of NDS implies the less difficulty for ISA to hide adversarial examples. A strong ISA is also expected to resist interpretability discrepancy even as the perturbation size increases; see (a)
To quantify this, compute additional quan- tity called normalized slope (NSL) that measures the relative change of NDS for E []
Snorm (D( DD -Paeml//earm parm. The smaller NDS is, the less strong ISA is to evade network (E-é)/E
norm
interpretation change. In our experiment, we choose ? = c" and = 1.6c", where e" is the minimum perturbation size required for a successful PGD attack for each image x. Here perform binary search over to find its smallest value for misclassification. Reported NDS and NSL statistics are averaged over a test set.
17

---

Under review as a conference paper at ICLR 2020
E
ADDITIONAL TABLES
Method
= 0
0.05
0.1
0.2
0.3
MNIST, Pool
Normal
0.990
0.435
0,070
0.000
0.000
Adv
0.930
0.885
0.835
0.695
0.535
TRADES
0.955
0.910
0.870
0.720
0.455
IG-Norm
0.980
0.940
0.660
0.050
0.000
IG-Norm-Sum
0,920
0.885
0.840
0.700
0.540
Int
0,950
0.930
0,875
0.680
0.390
Int-Adv
0.870
0.840
0,810
0.755
0.690
Int- Int-l-class
0.975
0.885
0.720
0.200
0.130
0
2/255
4/255
6/255
8/255
CIFAR-IO, Small
Normal
0.650
0.015
0.000
0.000
0.000
Adv
0.505
0.470
0.380
0.330
0.285
TRADES
0.630
0.465
0.355
0.235
0.140
IG-Norm
0.525
0.435
0.360
0.295
0.230
IG-Norm-Sum
0.390
0.365
0.325
0.310
0.285
Int
0.530
0.450
0.345
0.290
0.215
Int-Ady
0.675
0.145
0.005
0.000
0.000
Int- --el-s
0.515
0.450
0.380
0.315
0.265
Table A1: 200 steps PGD accuracy, additional results.
Method
c= 0
0.05
0.1
0.2
0.3
MNIST, Pool
Normal
0.990
0.435
0.070
0.000
0.000
Adv
0.930
0.885
0.835
0.695
0.535
TRADES
0.955
0.910
0,870
0.720
0.460
IG-Norm
0.980
0.945
0,660
0.060
0,000
IG-Norm-Sum
0.920
0.885
0.840
0.700
0.540
Int
0.950
0.930
0.875
0.680
0.385
Int-Adv
0,870
0.840
0,810
0.755
0.700
Int-I-class
0.975
0.885
0.720
0.200
0.130
2/255
4/255
6/255
8/255
CIFAR- 10, Small
Normal
0.650
0.015
0.000
0.000
0.000
Adv
0.505
0.470
0.380
0.330
0.285
TRADES
0.630
0.465
0.355
0.235
0.140
IG-Norm
0.525
0.435
0.360
0.295
0.230
IG-Norm-Sum
0.390
0.365
0.325
0.310
0.285
Int
0.530
0.450
0.345
0.290
0.215
Int-Adv
0,675
0.145
0,005
0.000
0.000
Int- 1-class
0.515
0.450
0.380
0.315
0.265
Table A2: 100 step PGD accuracy, additional results.
18

---

Under review as a conference paper at ICLR 2020
Method
0.05
0.1
0.2
0.3
MNIST, Pool
Normal
0.990
0.470
0.135
0.135
0.135
Adv
0.930
0.885
0.845
0.845
0.845
TRADES
0.955
0.910
0.870
0.870
0.870
IG-Norm
0,980
0.945
0.705
0.705
0.705
IG-Norm-Sum
0.920
0.885
0.850
0.850
0.850
Int
0.950
0.930
0.885
0.885
0.885
Int-Adv
0.870
0.840
0,810
0.810
0,810
Int-I-class
0.975
0.885
0.750
0.750
0.750
6 0
2/255
4/255
6/255
8/255
CIFAR- 10, Small
Normal
0,650
0.015
0.000
0.000
0.000
Adv
0.505
0.470
0.380
0.325
0.280
TRADES
0.630
0.465
0.360
0.240
0.145
IG-Norm
0.675
0.145
0.005
0.000
0.000
IG-Norm-Sum
0.515
0.450
0.380
0.315
0.265
Int
0.525
0.435
0.360
0.295
0.235
Int-Adv
0.390
0.365
0.325
0.310
0.285
Int- -nt---llass
0.530
0.450
0,345
0.290
0.220
Table A3: 10 step PGD accuracy, additional results.
Method
c 0.05
0.1
0.2
0.3
MNIST, Pool
Normal
0.934
0.876
0.719
0.482
Adv
0.976
0.951
0.896
0.824
TRADES
0.976
0.952
0.891
0.815
IG-Norm
0.942
0.872
0.648
0.341
IG-Norm-Sum
0.976
0.951
0.895
0.824
Int
0.964
0.928
0.852
0.771
Int-Adv
0.977
0.957
0.921
0.891
Int-I-class
0.930
0.871
0.779
0.704
2/255
4/255
6/255
8/255
CIFAR-10 10, Small
Normal
0.694
0.350
0.116
-0.031
Ady
0.958
0.907
0.849
0.783
TRADES
0.940
0.867
0.781
0.689
IG-Norm
0.810
0.552
0.308
0,131
IG-Norm-Sum
0.958
0.907
0.847
0.779
Int
0.965
0.926
0.883
0.840
Int-Adv
0.979
0.956
0.931
0,904
Int-I-class
0.961
0.918
0.871
0.820
Table A4: Kendall rank correlation coefficients of top-k CAM attacks against interpretability found using 200 steps of PGD, additional results.
19

---

Under review as a conference paper at ICLR 2020
Method
0.05
0.1
0.2
0.3
MNIST, Pool
Normal
0.957
0.912
0.811
0.594
Adv
0.978
0.955
0.903
0.839
TRADES
0.978
0.955
0.900
0.822
IG-Norm
0.979
0.955
0.879
0.771
IG-Norm-Sun Sum
0.979
0.956
0.905
0.843
Int
0,949
0.893
0.759
0.588
Int-Adv
0.983
0.967
0.938
0.909
Int-1-class
0.934
0.858
0.747
0.717
2/255
4/255
6/255
8/255
CIFAR-10, Small
Normal
0.691
0.335
0.118
-0.031
Adv
0.958
0.908
0.850
0.781
TRADES
0.939
0.866
0.779
0.690
IG-Norm
0.805
0.546
0.303
0.117
IG-Norm-Sum
0.957
0.906
0.844
0.773
Int
0,964
0.925
0.881
0,831
Int-Adv
0.975
0.949
0.919
0.886
Int-l-class
0.959
0.914
0.861
0.803
Table A5: Kendall rank correlation coefficients top-k GradCAM+ attacks against interpretability found using 200 steps of PGD, additional results.
20


Reviews:
[{"review_id": "rkeeRAvCtB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Interesting work and good contribution\n#Summary:\nThe paper demonstrated that by having an l1-norm based 2-class interpretability discrepancy measure, it can be shown both empirically and theoretically that it is actually difficult to hide adversarial examples. Furthermore, the authors propose an interpretability-aware robust training method and show it can be used to successfully defend adversarially attacks and can result in comparable performance compared to adversarial training.\n\n#Strength\nThe paper is well written and structured, with a clear demonstration of technical details. Compared with other works that tried to use model interpretation to help improve the model’s robustness, the authors not only consider the saliency map computed for the actual target label but also the label that corresponds to the adversarial example. The proposed interpretability discrepancy measure is novel and has been proven effective to defend interpretability sneaking attacks that aiming to fool both classifiers and detectors and against interpretability-only attacks. Furthermore, extensive experiments have been done to prove the effectiveness of interpretability-aware training, which strengthens the claims of the entire paper.\n\n#Presentation\nGood coverage of the literature in both adversarial robustness and model interpretation.\nSome minor typos need to be fixed. For example, in the second last line of the caption of Figure. 2, one L(x’,i) should be L(x,i) if I understand correctly. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "rJlQeu8GiB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In summary, this paper studies if interpretation robustness (i.e., similar examples should have similar interpretation) can help enhance the robustness of the model, especially in terms of adversarial attacks. The study direction itself is interesting and very useful for the interpretation and adversarial attack community. Moreover, some promising results can be observed in part of the empirical study. However, this paper can be improved a lot as follows.\n\n1. This paper states several times that \"adversarial examples can be hidden from neural network interpretability\". It is not clear on the definition of \"hidden\" in terms of  \"interpretability\". Therefore, how this \"hidden\" is related and why this \"hidden\" is important are unclear too.\n\n2. Many details are missing, which makes the proposal suspicious. For example, the proposed method has a tradeoff parameter \\lambda. However, the settings and affects are not discussed at all. Without a clear setup, the reproducibility and applicability is in doubt.\n\n3. Some empirical results are overstated. For example, why 0.790 vs 0.890 and 0.270 vs 0.170  are comparable results? These results show the weakness of the proposed method. Further explanations can be provided. From the reported results, it could be useful to see results when the perturbation is even higher to check the limitation of the proposed method.\n\n4. Besides the clarification in the writing mentioned above, some typos or errors should be fixed, e.g., f_t'(x') - - f_t(x') >=0 in the proof of proposition 1.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eSBR_CtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The present work considers adversarial attacks that also yield similar outputs for \"interpretability methods\", which are methods that output some vector corresponding to a given classification (usually the vector is e.g. an image or a similar object). It also shows that by regularizing nearby inputs to have similar interpetations (instead of similar classifications), robustness can be achieved similar to adversarial training. \n\nI did not understand the motivation of the paper. Why is it important for adversarial attacks to yield similar interpretations? A human would need to assess the interpretations to detect the attack, but it would already be trivial for an attack to be detected given human oversight (just check whether the classification of the image matches the human-assigned label). It also wasn't clear how this was related to the other observation that regularizing based on interpretability yields robustness; these seem like two fairly separate results.\n\nFinally, I found the claim that \"interpretability alone helps robustness\" to be misleading and not substantiated by the paper. The purported justification is that regularizing nearby inputs to have the same interpretation yields robustness. But a better summary of this observation is that \"robustness of interpretability implies robustness of classification\", which is not surprising, and is in fact a trivial corollary of the fact that the metric on interpretations dominates the classification error metric (an observation which is made in the paper).\n\nMore minor, but I found it hard to follow the writing in the paper (this is related to the motivation being unclear). This is exacerbated by the paper being longer than unusual (10 pages instead of 8).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work focuses on how one can design models with robustness of interpretations. While this is an interesting direction, the paper would benefit from a more careful treatment of its technical claims.


Decision: Reject