Title: PROPER NETWORK INTERPRETABILITY HELPS AD-VERSARIAL ROBUSTNESS IN CLASSIFICATION

Abstract: Recent works have empirically shown that there exist adversarial examples that can be hidden from neural network interpretability (namely, making network interpretation maps visually similar), and interpretability is itself susceptible to adversarial attacks. In this paper, we theoretically show that with the correct measurement of interpretation, it is actually difficult to hide adversarial examples, as confirmed by experiments on MNIST, CIFAR-10 and Restricted ImageNet. Spurred by that, we develop a novel defensive scheme built only on robust interpretation (without resorting to adversarial loss minimization). We show that our defense achieves high classification robustness, outperforming state-of-the-art adversarial training methods against attacks of large perturbation while attaining high interpretation robustness under various settings of adversarial attacks.

1. INTRODUCTION:

It has become widely known that convolutional neural networks (CNNs) are vulnerable to adversarial examples, namely, perturbed inputs with intention to mislead networks' prediction (Szegedy et al., 2014; Goodfellow et al., 2015; Papernot et al., 2016a; Carlini & Wagner, 2017; Chen et al., 2018; Su et al., 2018) . The vulnerability of CNNs has spurred extensive research on adversarial attack and defense. To design adversarial attacks, most work has focused on creating either imperceptible input perturbations (Goodfellow et al., 2015; Papernot et al., 2016a; Carlini & Wagner, 2017; Chen et al., 2018) or adversarial patches robust to the physical environment (Eykholt et al., 2018; Brown et al., 2017; Athalye et al., 2017) . Many defense methods have also been developed to prevent CNNs from misclassification when facing adversarial attacks. Examples include defensive distillation (Papernot et al., 2016b) , training with adversarial examples (Goodfellow et al., 2015) , input gradient or curvature regularization (Ross & Doshi-Velez, 2018; Moosavi-Dezfooli et al., 2019) , adversarial training via robust optimization (Madry et al., 2018) , and TRADES to trade adversarial robustness off against accuracy (Zhang et al., 2019) . Besides studying adversarial effects on network prediction decisions, this work explores the connection between adversarial robustness and network interpretability, and provides novel insights on when and how interpretability helps the robustness.

Having a prediction might not be enough for many real-world machine learning applications. It is also crucial to demystify why CNNs make certain decisions. Thus, the problem of network interpretation arises. Various methods have been proposed to understand the mechanism of decision making by CNNs. One category of methods justify a prediction decision by assigning importance values to reflect the influence of individual pixels or image sub-regions on the final classification. Examples include pixel-space sensitivity map methods (Simonyan et al., 2013; Zeiler & Fergus, 2014; Springenberg et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017) and class-discriminative localization methods (Zhou et al., 2016; Selvaraju et al., 2017; Chattopadhay et al., 2018; Petsiuk et al., 2018) , where the former evaluates the sensitivity of a network classification decision to pixel variations at the input, and the latter localizes which parts of an input image were looked at by the network for making a classification decision. Sensitivity map methods include vanilla gradient (Simonyan et al., 2013) , deconvolution (Zeiler & Fergus, 2014) , guided backpropagation (Springenberg et al., 2014) , SmoothGrad (Smilkov et al., 2017) , integrated gradient (IG) (Sundararajan et al., 2017) to name a few. They highlight fine-grained details in the image but are not class-discriminative for visual explanation. By contrast, localization approaches like class activation map (CAM) (Zhou et al., 2016) , GradCAM (Selvaraju et al., 2017) , GradCAM++ (Chattopadhay et al., 2018) and RISE (Petsiuk et al., 2018) are highly class-discriminative, namely, localizing image sub-regions reasoned for a prediction class. We refer readers to Sec. 2 for some representative interpretation methods. Besides interpreting CNNs via feature importance maps, some methods zoom into the internal response of neural networks. Examples include network dissection (Bau et al., 2017) , which evaluates the alignment between individual hidden units and semantic concepts, and learning perceptuallyaligned representations from robust training (Engstrom et al., 2019) .

Very recently, some works (Xu et al., 2019b; a; Zhang et al., 2018; Subramanya et al., 2018; Ghorbani et al., 2019; Dombrowski et al., 2019; Chen et al., 2019) are beginning to study adversarial robustness by exploring the spectrum between classification accuracy and network interpretability. It was shown in (Xu et al., 2019b; a) that an imperceptible adversarial perturbation to fool classifiers can lead to a significant change in a class-specific network interpretability map, e.g., CAM. Thus, it was argued that such an interpretability discrepancy can be used as a helpful metric to differentiate adversarial examples from benign inputs. However, the work (Zhang et al., 2018; Subramanya et al., 2018) showed that under certain conditions, generating an attack (which we call interpretability sneaking attack, ISA) that fools the classifier as well as its coupled interpreter (in terms of keeping interpretability map highly similar to that of benign input) is not significantly more difficult than generating adversarial inputs deceiving the classifier only. Besides investigating robustness in classification through the lens of interpretability, the work (Ghorbani et al., 2019; Dombrowski et al., 2019) studied the robustness in network interpretation maps, showing that which can significantly be manipulated via imperceptible input perturbations but keeping the classifier's decision intact. We call this type of threat model attack against interpretability (AAI). The existing work had no agreement on the relationship between robustness in interpretation and robustness in classification. Spurred by that, we attempt to explore this relationship from both attack and defense perspectives.

The most relevant work to ours is (Chen et al., 2019) , which robustified network interpretation with the aid of integrated gradient (IG), an axiomatic attribution map. It proposed a robust attribution training, which was shown as a principled generalization of previous formulations of robust classification and an effective defense against AAI. In this paper, we first investigate when ISA is possible, and then relate our insights on ISA to robust classification and robust interpretability. Different from the previous work, our paper contains the following contributions.

1.:

We provide an answer to the question of when adversarial examples can bypass interpretability discrepancy. We show that enforcing stealthiness of adversarial examples from network interpretation could be challenging. Its difficulty relies on how one measures the interpretability discrepancy caused by input perturbations.

2. We propose an 1 norm based 2-class interpretability discrepancy measure and theoretically show that constraining it helps adversarial robustness.

3. We develop an interpretability-aware robust training method and empirically show that interpretability alone can be used to defend adversarial attacks for both misclassifcation and misinterpretation. Compared to the IG-based robust attribution training (Chen et al., 2019) , our approach is simpler in implementation, and provides better robustness even as facing a strong adversary.

2. PRELIMINARIES AND MOTIVATION: INTERPRETABILITY OF CNNS FOR JUSTIFYING A CLASSIFICATION DECISION:

To explain what and why CNNs predict, we consider two types of network interpretation methods: a) class activation map (CAM) (Zhou et al., 2016; Selvaraju et al., 2017; Chattopadhay et al., 2018) and b) pixel sensitivity map (PSM) (Simonyan et al., 2013; Springenberg et al., 2014; Smilkov et al., 2017; Sundararajan et al., 2017; Yeh et al., 2019) . Let f (x) ∈ R C denote a CNN-based predictor that maps an input x ∈ R d to a probability vector of C classes. Here f c (x), the cth element of f (x), denotes the classification score (given by logit before the softmax) for class c. Let L(x, c) denote an interpreter (CAM or PSM) that reflects where in x contributes to the classifier's decision on c. 

EQUATION

where w c k is the linear classification weight that associates the channel k with the class c, and A k,i denotes the ith element of A k . The rationale behind (1) is that the classification score f c (x) can be written as the average of CAM values (Zhou et al., 2016) , f c (x) = u i=1 [L CAM (x, c)] i . For visual explanation, L CAM (x, c) is often up-sampled to the input dimension d using bi-linear interpolation.

GradCAM (Selvaraju et al., 2017) generalizes CAM for CNNs without the architecture 'global average pooling → softmax layer' over the final convolutional maps. Specifically, the weight w c k in (1) is given by the gradient of the classification score f c (x) with respect to (w.r.t.) the feature map

A k , w c k = 1 u u i=1 ∂fc(x)

∂A k,i . GradCAM++ (Chattopadhay et al., 2018) , a generalized formulation of GradCAM, utilizes a more involved weighted average of the (positive) pixel-wise gradients but provides a better localization map if an image contains multiple occurrences of the same class. In this work, we focus on CAM since it is computationally light and our models used in experiments follow the architecture 'global average pooling → softmax layer'.

PSM-type methods. PSM uses calculations with gradients to assign importance scores to individual pixels toward explaining the classification decision about an input. Examples of commonly-used approaches include vanilla gradient (Simonyan et al., 2013 ), guided backpropogation (Springenberg et al., 2014) , SmoothGrad (Smilkov et al., 2017) , and integrated gradient (IG) (Sundararajan et al., 2017) . In particular, IG satisfies the completeness attribution axiom that PSM ought to obey. Specifically, it averages gradient saliency maps for interpolations between an input x and a baseline image a:

EQUATION

where m is the number of steps in the Riemman approximation of the integral. The completeness axiom (Sundararajan et al., 2017 , Proposition 1) states that

d i=1 [L IG (x, c)] i = f c (x) -f c (a)

, where the baseline image a is often chosen such that f c (a) ≈ 0, e.g., the black image. Note that CAM also satisfies the completeness axiom. PSM is able to highlight fine-grained details in the image, but is computationally intensive and not very class-discriminative compared to CAM (Selvaraju et al., 2017) .

Interpretability discrepancy caused by adversarial perturbation. Let x = x + δ represent an adversarial example w.r.t. x, where δ denotes an adversarial perturbation. By replacing the input image x with x , the CNN will be fooled from the true label t to the target (incorrect) label t . It was recently shown in (Xu et al., 2019b; a) that the adversary could introduce an evident interpretability discrepancy w.r.t. both the true and the target label in terms of L(x, t) vs. L(x , t), and L(x, t ) vs. L(x , t ). An illustrative example is provided in Figure 1 . We see that an adversary suppresses the network interpretation w.r.t. the true label but promotes the interpretation w.r.t. the target label. We also observe that compared to IG, CAM and GradCAM++ better localize class-specific discriminative regions. These results reveal two observations on how measuring interpretation discrepancy affects classification robustness: a) interpretability discrepancy may be used to detect adversarial examples, b) interpretability discrepancy itself may be vulnerable to adversarial perturbations. In what follows, we explore the spectrum between adversarial robustness and interpretability from a unified perspective considering both the adversarial vulnerability of interpretability discrepancy and the value of interpretability discrepancy in a defense .

3. ROBUSTNESS IN CLASSIFICATION VS. INTERPREATION MAP: AN ATTACK PERSPECTIVE:

In this section, we examine two types of threat models, interpretability sneaking attack (ISA) and attack against interpretability (AAI). Since an adversarial example designed for misclassification et al., 2019) with respect to the true label t='monkey' and the target label t ='fish'. Here the adversarial example is generated by 10-step PGD attack with perturbation size 0.02 (Madry et al., 2018) , and we consider three types of interpretation maps, CAM, GradCAM++ and IG. Given an interpretation method, the first column is L(x, t) versus L(x , t), the second column is L(x, t ) versus L(x , t ), and all maps under each category are normalized w.r.t. their largest value. At the bottom of each column, we quantify the resulting interpretability discrepancy by Kendall's Tau order rank correlation (Selvaraju et al., 2017) between every pair of L(x, i) and L(x , i) for i = t or t .

gives rise to interpretability discrepancy (which could then be used as a detector for the adversarial input), the problem of ISA arises: One may wonder whether or not it is easy to generate adversarial examples that mistake classification but keep interpretation intact. Such adversarial vulnerability could have serious consequences when classification and interpretation are jointly used in tasks like medical diagnosis (Subramanya et al., 2018) , and call into question the faithfulness of interpretation to network classification. One the other hand, it is suggested from interpretability discrepancy that an interpreter could be quite sensitive to input perturbations. Thus, the problem of AAI arises: One may wonder if perturbed inputs could induce differing explanations but without altering predictions.

3.1. RETHINKING ISA FROM PERSPECTIVE OF INTERPRETABILITY DISCREPANCY:

Previous work (Zhang et al., 2018; Subramanya et al., 2018) showed that it is not difficult to hide adversarial examples from network interpretation when the interpretability discrepancy is measured w.r.t. a single class label (either the true label t or the target label t ). However, we see from Figure 1 that the adversary (against classification) alters interpretability maps w.r.t. both t and t . This motivates us to rethink whether the single-class interpretability discrepancy measure is proper, and whether ISA is truly easy to bypass an interpretability discrepancy check.

We consider the following generic form of interpretability discrepancy

EQUATION

where recall that x and x are natural and adversarial examples respectively, L represents an interpreter, e.g., CAM or IG, C denotes the set of class labels used in L, |C| is the cardinality of C, and we consider p ∈ {1, 2} in this paper. Clearly, a specification of (3) relies on the choice of C and p. We specify (3) with C = {t, t } and p = 1, leading to 1 2-class interpretability discrepancy measure,

D 2, 1 x, x = 1 2 L(x, t) -L(x , t) 1 + L(x, t ) -L(x , t ) 1 . (4)

Rationale behind 1 2-class discrepancy measure. Compared to the previous work (Zhang et al., 2018; Subramanya et al., 2018 ) using a single class label, we choose C = {t, t }1 , motivated by the fact that an interpretability discrepancy occurs w.r.t. both t and t (Figure 1 ). Moreover, although Euclidean distance (namely, 2 norm or its square) is arguably one of the most commonly-used discrepancy metrics (Zhang et al., 2018) , we show in Proposition 1 that the proposed interpretability discrepancy measure D 2, 1 (x, x ) has a non-trivial lower bound for any successful adversarial attack. This provides an explanation on why it could be difficult to hide adversarial examples from network interpretability methods. Moreover, 1 is an upper bound of the 2 norm and promotes the sparsity of interpretability discrepancy, which enforces pixels of L to stay intact when facing input perturbations.

Proposition 1. Given a classifier f (x) ∈ R C and its interpreter L(x, c) for c ∈ [C], suppose that the interpreter satisfies the completeness axiom, namely, i [L(x, c)] i = f c (x). For a natural example x and an adversarial example x with prediction t and t respectively, D 2, 1 (x, x ) in (4) has the lower bound

EQUATION

Proof: See proof in Appendix A.

Proposition 1 connects D 2, 1 (x, x ) with the classification margin f t (x) -f t (x). Thus, if a classifier has a large classification margin on the natural example x, it will be difficult to find a successful adversarial attack with small interpretability discrepancy. In other words, constraining the interpretability discrepancy prevents misclassification since generating adversarial examples becomes infeasible under

D 2, 1 (x, x ) < 1 2 (f t (x) -f t (x))

. Also, the completeness condition of L suggests specifying (4) with CAM (1) or IG (2). Here we focus on CAM due to its light computation.

Design of ISA. We pose the following optimization problem for design of ISA, which not only fools a classifier's decision but also minimizes the resulting interpretability discrepancy,

EQUATION

where λ > 0 is a regularization parameter that strikes a balance between the success of an attack and its resulting interpretability discrepancy, τ ≥ 0 (e.g., 0.1 used in the paper) is a tolerance on the classification margin of a successful attack between the target label t and the non-target top-1 prediction label arg max j =t f j (x + δ), D 2, 1 was defined by (4), and > 0 is a (pixellevel) perturbation size. In ( 6), the first term of the objective corresponds to a C&W-type attack loss (Carlini & Wagner, 2017) , which reaches -τ if the attack succeeds in misclassification. To find ISA of minimum interpretability discrepancy, we perform a bisection on λ until there exists no successful attack that can be found when λ further decreases. Although we focus on ∞ attack in (6), but a similar formulation applies to 1 and 2 attacks. Attacks are found using PGD, with subgradients taken at non-differentiable points. We consider only targeted attacks to better evaluate the effect on interpretability of target classes, although this approach can be extended to an untargeted setting (e.g., by using the target label-free interpretability discrepancy measure introduced in the next section).

Difficulty of generating ISA versus interpretability discrepancy measure. Through an illustrative example in Figure 2 , we elaborate on how the choice of interpretability discrepancy measure plays a crucial role on the difficulty of hiding adversarial examples from network interpretation. We refer readers to Sec. 5 for more experimental results. We generate ISA under different specifications of (3) for different values of perturbation size . Compared to 1 / 2 1-class (true class t), 2 2-class, and 1 / 2 all-class, we see that the 1 2-class interpretability discrepancy (4) cannot be easily mitigated even as the attack power (in terms of ) increases. This is verified by a) its high interpretability discrepancy score and b) its flat slope of discrepancy score against in Figure 2-(a) &(b). Figure 2-(c ) further shows CAMs of adversarial examples w.r.t. the true label t and the target label t generated by 1 1/2/all-class ISAs. We note that both 1-class measure and all-class measure could give a false sense of ease of hiding adversarial examples. For 1 1-class ISA, although the interpretability discrepancy w.r.t. t is minimized, the discrepancy w.r.t. t remains large, supported by the observation that 1 1-class ISA even yields a smaller correlation between L(x , t ) and L(x, t ) than PGD attack. Similarly, although the averaged discrepancy over all classes is minimized for 1 all-class ISA, discrepancies w.r.t. specific labels such as t and t do not necessarily reduce. This illustrates that although ISA may be found, with the proper choice of discrepancy measure, ISA with a low discrepancy becomes quite difficult.

3.2. ATTACK AGAINST INTERPRETABILITY (AAI):

Different from ISA, AAI produces input perturbations to maximize the interpretability discrepancy while keeping the classification decision intact. Thus, AAI provides a means to evaluate the ad- versarial robustness in interpretability. Since t = arg max i f i (x) = arg max i f i (x ) = t in AAI, the 2-class interpretability discrepancy measure (4) reduces to its 1-class version. The problem of generating AAI is then cast as

EQUATION

where the first term is a hinge loss to enforce f t (x + δ) ≥ max j =t f j (x + δ), namely, arg max i f i (x ) = t (unchanged prediction under input perturbations), and D 1 denotes a 1-class interpretability discrepancy measure, e.g., D 1, 1 from (4), or the top-k pixel difference between interpretability maps (Ghorbani et al., 2019) . Similar to (6), the regularization parameter λ in (7) strikes a balance between stealthiness in classification and variation in interpretability. Experiments in Sec. 5 will show that the state-of-the-art defense methods against adversarial examples do not necessarily preserve robustness in interpretability as increases, athough the prediction is not altered.

4. INTERPRETABILITY-AWARE ROBUST TRAINING: DEFENSE PERSPECTIVE:

We recall from Sec. 3.1 that adversarial examples that intend to fool a classifier could be difficult to evade the 1 2-class interpretability discrepancy. Thus, constraining the interpretability discrepancy helps to prevent misclassification. Motivated by this observation about ISA, we introduce a interpretability based defense method that penalizes interpretability discrepancy to achieve high classification robustness. As an additional benefit, it also robustifies the classifier against AAI, where the adversary maximizes the interpretability discrepancy.

Target label-free interpretability discrepancy. Different from attack generation, the 1 2-class discrepancy measure (4) cannot directly be used by a defender since the target label t specified by the adversary is not known a priori. To circumvent this issue, we propose to approximate the interpretability discrepancy w.r.t. the target label by weighting discrepancies from all non-true classes according to their importance in prediction. This modifies (4) to

EQUATION

where the softmax function e f (x ) i i e f (x ) i adjusts the importance of non-true labels according to their classification confidence. Clearly, when x succeeds in misclassification, the top-1 predicted class of x becomes the target label and the resulting interpretability discrepancy is most penalized.

Interpretability-aware robust training. We propose to train a classifier against the worst-case interpretability discrepancy (8), yielding the min-max optimization problem minimize θ E (x,t)∼D train ftrain(θ; x, t) + γ maximize

EQUATION

where θ denotes the model parameters, D train denotes the training dataset, f train is the training loss (e.g., cross-entropy loss), γ > 0 is a regularization parameter, and for ease of notation we omit the parameters θ and t in D (x, x ). Here γ controls the tradeoff between clean accuracy and robustness; see Appendix C for experiments analyzing this tradeoff.

In problem (9), the inner maximization is only used to evaluate the worst-case interpretability discrepancy. Thus, it is different from adversarial training (Madry et al., 2018) , where the training loss is replaced with the adversarial loss maximize x : x -x ∞≤ f train (θ, x ; x, t). The formulation (9) allows us to examine whether or not robust interpretation is directly beneficial to robust classification. For completeness, we will also provide experiment results on a modified formulation of ( 9) with the use of the adversarial loss.

Difference to (Chen et al., 2019) . The recent work (Chen et al., 2019) proposed improving adversarial robustness by leveraging robust IG attributions. However, different from (Chen et al., 2019) , our approach is motivated by the importance of 1 2-class interpretability discrepancy measure. We will show in Sec. 5 that the incorporation of interpretability discrepancy w.r.t. target class labels, namely, the second term in (8), plays a key role in boosting classification and interpretation robustness. This is because robust interpretability implies robust classification only when interpretation maps are measured with a proper metric. We will also show that our proposed method is sufficient to improve adversarial robustness even in the absence of adversarial loss, while the robust attribution regularization method (Chen et al., 2019) becomes ineffective when the attack becomes stronger. Last but not the least, beyond IG, our proposed theory and method apply to any network interpretation method with completeness axiom. The use of CAM avoids Riemman approximation used in IG and thus simplifies the implementation during robust training.

5. EXPERIMENTS:

In this section, we empirically show the effectiveness of our proposed methods in various attack and defense settings. For ISA, we examine how the interpretability discrepancy measure plays a role in hiding adversarial examples from network interpretation. For AAI, we evaluate its attack success rate under the natural and various robust models. For interpretability-aware robust training, we demonstrate its advantages in a) defending against PGD attacks with different steps and perturbation sizes (Madry et al., 2018; Athalye et al., 2018) , b) defending against unforeseen adversarial attacks ( Kang et al., 2019) , c) rendering robustness in interpretability, and d) computation efficiency compared to the IG-based robust attribution regularization method (Chen et al., 2019) .

Datasets, CNN models, and experiment setting. We evaluate networks trained on the MNIST, CIFAR-10 and a restricted ImageNet (R-ImageNet) dataset used in (Engstrom et al., 2019) . We consider three models, Small (for MNIST and CIFAR), Pool (for MNIST) and WResnet (for CIFAR and R-ImageNet): 1) a small CNN architecture consisting of three convolutional layers of 16, 32 and 100 filters (Small), 2) a CNN architecture with two convolutional layers of 32 and 64 filters each followed by max-pooling which is adapted from (Madry et al., 2018) (Pool), and 3) a Wide Resnet from (Madry et al., 2018) (WResnet) . We refer readers to Appendix B for more details.

In the following experiments, we consider 5 baselines from the literature: i) standard training (Normal), ii) adversarial training (Adv) (Madry et al., 2018) , iii) TRADES (Zhang et al., 2019) , iv) IG-Norm that uses IG-based robust attribution regularization (Chen et al., 2019) , v) IG-Sum-Norm (namely, IG-Norm with adversarial loss). Additionally, we consider 3 variants of our method: vi) the proposed interpretability-aware robust training method (9) (we call Int), vii) Int using 1 1-class discrepancy (Int-1-class), and viii) Int with adversarial loss (Int-Adv). For interpretability-based methods, the regularization parameter γ is set to 0.01. In Appendix C, we explore different settings of the parameter γ for our method to demonstrate that we can achieve a range of different points on the robustness-accuracy tradeoff. Evaluating ISA. We evaluate the effect of interpretability discrepancy measure on ease of finding ISAs. Spurred by Figure 2 , such an effect is quantified by calculating minimum discrepancy required in generating ISAs against different values of perturbation size in (6). We conduct experiments over 4 network interpretation methods: i) CAM, ii) GradCAM++, iii) IG, and iv) internal representation at the penultimate (pre-softmax) layer (denoted by Repr). In order to fairly compare among different interpretation methods, we compute a normalized discrepancy score (NDS) extended from (3) and a normalized slope (NSL) that measures the relative change of NDS for ∈ [ˇ , ˆ ]. We refer readers to Appendix D for more details.

In Table 1 , we present NDS and NSL of ISAs generated under different realizations of interpretability discrepancy measure (3), each of which is given by a combination of interpretation method (CAM, GradCAM++, IG or Repr), p norm (p ∈ {1, 2}) and number of interpreted classes. Note that Repr is independent of the number of classes, and thus we report NDS and NSL corresponding to Repr in the 2-class column of Table 1 . Given an p norm and an interpretation method, we consistently find that the use of 2-class measure achieves the largest NDS and smallest NSL at the same time. This implies that the 2-class discrepancy measure increases the difficulty of ISA to evade network interpretability check. Moreover, given a class number and an interpretation method, we see that NDS under 1 norm is greater than that under 2 norm, since the former is naturally an upper bound of the latter. Also, the use of 1 norm often yields a smaller value of NSL, implying that the 1 -norm based discrepancy measure is more resistant to ISA. Furthermore, by fixing the combination of 1 norm and 2 classes, we observe that IG is the most resistant to ISA due to their relatively high NDS and low ISA, and Repr yields the worst performance. However, compared to CAM, the computation cost of IG increases dramatically as the input dimension, the model size, and the number of steps in Riemman approximation increase. We find that it becomes infeasible to generate ISA using IG for WResnet under R-ImageNet within 200 hours.

Evaluating AAI. We specify problem (7) under the top-k attack setting (Ghorbani et al., 2017) , where the top-k intersection between the original and adversarial interpretability maps is minimized. The strength of AAI is then measured by the Kendall's Tau order rank correlation between the aforementioned two interpretability maps (Chen et al., 2019) . The higher the correlation is, the more robust the model is against AAI. Reported rank correlations are averaged over a test set. In Table 2 , we present the performance of AAI under multiple perturbation sizes to attack models trained using different training methods. For the model Small under MNIST, we evaluate AAI over 5 baselines (Normal, Adv, TRADES, IG-Norm, IG-Sum-Norm) and 3 variants of our method (Int, Int-1-class and Int-Adv). For a larger model WResnet under CIFAR-10, the training methods using IG (IG-Norm and IG-Norm-Sum) are excluded due to their prohibitive computation cost. The insights learnt from Table 2 are summarized as below. First, the normally trained model (Normal) does not automatically give robustness guarantees in interpretability, particularly for AAI with ≥ 0.2. Second, the methods Adv, TRADES, IG-Sum-Norm and Int-Adv that uses adversarial loss offer certain robustness against AAI but the performance gets worse as the perturbation size increases. Third, in the absence of adversarial loss, the baseline IG-Norm becomes less robust as increases. By contrast, our proposed method Int is consistently more robust and its advantage becomes more evident as increases, including when is increased beyond the value used for training.

Evaluating interpretability-aware robust training. We previously showed in Table 2 that interpretability-aware robust training (Int, Int-Adv) often achieves more robust interpretability than state-of-the-art adversarial training methods particularly for large perturbation size. We next provide a thorough evaluation on how interpretability helps adversarial robustness against misclassification. Robustness versus efficiency. In Figure 3 , we present the training time (left y-axis) and the adversarial test accuracy (right y-axis) for different training methods (x-axis), which are ranked in a decreasing order of computation complexity.

Here the adversarial test accuracy (ATA) is measured using 200-step ( ∞ -norm) perturbation sizes = 0.3 and 0.4 on the Small MNIST model (Madry et al., 2018) . Note that all AT-type methods (IG-Norm-Sum, Int-Adv, TRADES and Adv) offer robust classification at = 0.3 with ATA around 80%. Among non-AT but interpretability promoted defensive schemes (IG-Norm, Int, Int-1class), we find that only the proposed Int yields competitive ATA, and outperforms all AT-type methods except Int-Adv at = 0.4. Particularly, IG-Norm degrades significantly in robust classification when PGD attack becomes stronger (also see results in Table 4 ). Moreover, the non-robustness of Int-1-class verifies the importance of 2-class interpretability discrepancy measure on rendering robust classification. Last but not the least, Adv, TRADES and Int-based methods have similar computational complexity, but IG-based methods make training time (per epoch) significantly higher, 3 times more than Int-Adv even under the Small MNIST model.

Int does not cause obfuscated gradients and importance of 2-class measure. It was shown in (Athalye et al., 2018; Carlini, 2019) that some defense methods cause obfuscated gradients, which give a false sense of security. There exist two characteristic behaviors of obfuscated gradients: a) One-step attacks perform better than iterative attacks; b) Increasing distortion bound does not increase success. Motivated by that, we evaluate our interpretability-aware robust training methods under PGD attacks with a) different steps and b) different perturbation sizes. Interestingly, the advantage of Int becomes more evident as the adversary becomes stronger, i.e., > 0.3 on MNIST and > 8/255 on CIFAR. We highlight that such a robust classification is achieved by promoting robustness of interpretability alone (without using adversarial loss). It is worth mentioning that IG-Norm fails to defend PGD attack with = 0.3 for the Small MNIST model. We further note that Int-1-class performs much worse than Int, supporting the importance of using 2-class discrepancy measure (see Prop. 1). Besides robust classification, IG-Norm and Int-1-class are also not sufficient to render robustness in interpretation (Table 2 ).

Table 4 shows ATA of interpretability-aware robust training against k-step PGD attacks (examining behavior b) of obfuscated gradients), where k ∈ {1, 10, 100, 200}. As we can see, ATA decreases as k increases. This also suggests that the high robust accuracy from our methods is not a result of obfuscated gradients. Similar to Table 3 , we see that compared to Int, IG-Norm and Int-1-class are insufficient to defend PGD attacks with k ≥ 100.

Beyond ∞ -norm PGD attacks. In Table 5 , we present ATA of interpretability-aware robust training and various baselines for defending attacks (Gabor, Snow, JPEG ∞ , JPEG 2 , and JPEG 1 ) recently proposed in (Kang et al., 2019) . These attacks are called 'unforseen attacks' since they are not met by PGD-based robust training and often induce larger perturbations than conventional PGD attacks. For robust training methods without resorting to adversarial loss, we find that Int significantly outperforms IG-Norm especially under Snow and JPEG p attacks. Int also yields quite competitive robustness compared to AT-type methods on most attacks. lighted (1st, 2nd, 3rd) In this paper, we investigate the connection between network interpretability and adversarial robustness. We show theoretically and empirically that with the correct choice of discrepancy measure, it is difficult to hide adversarial examples from interpretation. We leverage this discrepancy measure to develop a interpretabilityaware robust training method that displays 1) high classification robustness in a variety of settings and 2) high robustness of interpretation. 

B ADDITIONAL DETAILS ON MODEL AND DATASET:

The considered network model has a global average pooling layer followed by a fully connected layer at the end of the network. . The smaller NDS is, the less strong ISA is to evade network interpretation change. In our experiment, we choose ˇ = * and ˆ = 1.6 * , where * is the minimum perturbation size required for a successful PGD attack for each image x. Here we perform binary search over to find its smallest value for misclassification. Reported NDS and NSL statistics are averaged over a test set.

:

In addition to the

2-class case, our experiments will also cover the all-class case C = [C].

Figures and Tables:

Figure 1: Interpretation (L) of benign (x) and adversarial (x ) image from restricted ImageNet (Engstrom

Figure 2: Seeing the effect of discrepancy measure on hiding adversarial examples from network interpretation. The same benign image in Figure 1 is considered. (a) ISA using CAM-based 1 1/2/all-class discrepancy measure versus perturbation size , (b) ISA using CAM-based squared 2 1/2/all-class discrepancy measure versus , (c) CAM interpretation of example in Figure 1 and its adversarial counterparts from PGD attack and different specifications of ISA. All interpretation maps are normalized w.r.t. their largest value. At the bottom of each interpretation map L(x , •), we quantify the interpretability discrepancy by Kendall's Tau order rank correlation between every pair of L(x , i) and L(x, i) for i ∈ {t, t }, where x is obtained from PGD attack or each specification of ISA.

Figure 3: Computation time per epoch and adversarial test accuracy for a Small MNIST model trained with different methods.

For R-ImageNet, we use only a normally trained network, used for evaluating ISA. During adversarial training of all baselines, 40 adversarial steps are used for MNIST and 10 steps for CIFAR. To ensure stability of all training methods, the size of perturbation is increased during training from 0 to a final value of 0.3 on MNIST and 8/255 on CIFAR. MNIST and R-ImageNet networks are trained for 100 epochs and CIFAR networks are trained for 200 epochs. A batch size of 50 is used for MNIST, 128 for CIFAR and 64 for R-Imagenet. For all methods, training is performed using Adam with an initial learning rate of 0.0001 for MNIST and 0.001 for CIFAR and R-Imagenet, with the learning decayed by ×1/10 at training steps 40000 and 60000 for CIFAR and 8000 and 16000 for R-Imagenet.D DETAILS ON NDS AND NSLIn order to fairly compare among different interpretation methods, we compute a normalized discrepancy score (NDS) extended from (3):D norm = 1 |C| i∈C L(x,i)-L(x ,i) maxj L(x,i)j -minj L(x,i)j p .A smaller value of NDS implies the less difficulty for ISA to hide adversarial examples. A strong ISA is also expected to resist interpretability discrepancy even as the perturbation size increases; see Figure2-(a)&(b). To quantify this, we compute an additional quantity called normalized slope (NSL) that measures the relative change of NDS for ∈ [ˇ , ˆ ]:

CAM-type methods. CAM(Zhou et al., 2016)  produces a class-discriminative localization map for CNNs, which performs global averaging pooling over convolutional feature maps prior to the softmax. Let the penultimate layer output K feature maps, each of which is denoted by a vectorrepresentation A k ∈ R u for channel k ∈ [K].Here [K] represents the integer set {1, 2, . . . , K}. The ith entry of CAM L CAM (x, c) is given by

Table content: <table/>

NDS and NSL (format given by NDS/NSL) of successful ISAs generated under different specifications of interpretability discrepancy measure (3) and datasets MNIST, CIFAR-10 and R-ImageNet. Here a discrepancy measure with large NDS and small NSL indicates a strong resistance to ISA.

Table content: <table><tr><td>Dataset</td><td>Interpretation method</td><td>1-class</td><td>1 norm 2-class</td><td>all-class</td><td>1-class</td><td>2 norm 2-class</td><td>all-class</td></tr><tr><td/><td>CAM</td><td colspan="6">3.0723/0.0810 3.2672/0.0223 2.5289/0.0414 0.3061/0.1505 0.5654/0.0321 0.4320/0.0459</td></tr><tr><td>MNIST</td><td>GradCAM++ IG</td><td colspan="6">3.1264/0.0814 3.1867/0.0221 2.5394/0.0366 0.3308/0.1447 0.5531/0.0289 0.4392/0.0456 6.3604/0.0330 6.7884/0.0233 4.3667/0.2314 0.4476/0.0082 0.5766/0.0064 0.2160/0.0337</td></tr><tr><td/><td>Repr</td><td>n/a</td><td>2.3668/0.0404</td><td>n/a</td><td>n/a</td><td>0.4129/0.0429</td><td>n/a</td></tr><tr><td/><td>CAM</td><td colspan="6">1.9523/0.1450 2.5020/0.0496 1.7898/0.0774 0.1313/0.2369 0.3613/0.0668 0.2746/0.0809</td></tr><tr><td>CIFAR-10</td><td>GradCAM++ IG</td><td colspan="6">1.9355/0.1439 2.4788/0.0513 1.8020/0.0745 0.1375/0.2346 0.3577/0.0676 0.2758/0.0769 4.9499/0.0188 4.9794/0.0177 2.8541/0.1356 0.1230/0.0110 0.1309/0.0092 0.0878/0.0235</td></tr><tr><td/><td>Repr</td><td>n/a</td><td>1.7049/0.0785</td><td>n/a</td><td>n/a</td><td>0.1288/0.0056</td><td>n/a</td></tr><tr><td/><td>CAM</td><td colspan="6">49.286/0.1005 61.975/0.0331 49.877/0.0557 1.9373/0.1526 2.6036/0.0791 2.0935/0.0863</td></tr><tr><td>R-ImageNet</td><td>GradCAM++ Repr</td><td colspan="6">39.761/0.1028 50.303/0.0453 42.390/0.0552 1.9185/0.1609 2.5869/0.0891 2.1151/0.0896 n/a 46.892/0.0657 n/a n/a 2.0730/0.0781 n/a</td></tr></table>

Table 3 reports ATA of interpretability-aware robust training relative to various baselines, where 200-step PGD attacks are conducted for ∈ {0, 0.05, 0.1, 0.2, 0.3, 0.35, 0.4} for MNIST and ∈ {0, 2/255, 4/255, 6/255, 8/255, 9/255, 10/255} for CIFAR. We use = 0.3 on MNIST and 8/255 on CIFAR for robust training. As we can see, ATA decreases as increases. Thus, our methods do not exhibit the behavior a) of obfuscated gradients. We also observe that compared to Adv and TRADES, Int achieves reasonable but worse ATA as < 0.3 on MNIST and ≤ 8/255 on CIFAR. As used in PGD attack achieves the value used for robust training, Int achieves ATA 0.790 against ATA 0.890 from Adv on MNIST, but outperforms Adv on CIFAR (0.270 vs 0.170).

Table content: <table/>



Table content: <table><tr><td>Method</td><td>Steps= 1</td><td>10</td><td>100</td><td>200</td></tr><tr><td colspan="3">MNIST, Small, = 0.3</td><td/><td/></tr><tr><td>Normal</td><td>0.990</td><td colspan="3">0.070 0.000 0.000</td></tr><tr><td>Adv</td><td>0.975</td><td colspan="3">0.945 0.890 0.890</td></tr><tr><td>TRADES</td><td>0.970</td><td colspan="3">0.955 0.885 0.885</td></tr><tr><td>IG-Norm</td><td>0.970</td><td colspan="3">0.905 0.005 0.005</td></tr><tr><td>IG-Norm-Sum</td><td>0.970</td><td colspan="3">0.940 0.880 0.880</td></tr><tr><td>Int-1-class</td><td>0.950</td><td colspan="3">0.365 0.125 0.125</td></tr><tr><td>Int</td><td>0.935</td><td colspan="3">0.910 0.790 0.790</td></tr><tr><td>Int-Adv</td><td>0.950</td><td colspan="3">0.905 0.855 0.855</td></tr><tr><td colspan="4">CIFAR, Wresnet, = 8/255</td><td/></tr><tr><td>Normal</td><td>0.470</td><td colspan="3">0.075 0.060 0.060</td></tr><tr><td>Adv</td><td>0.590</td><td colspan="3">0.205 0.185 0.185</td></tr><tr><td>TRADES</td><td>0.590</td><td colspan="3">0.180 0.165 0.165</td></tr><tr><td>Int</td><td>0.620</td><td colspan="3">0.310 0.275 0.275</td></tr><tr><td>Int-Adv</td><td>0.580</td><td colspan="3">0.345 0.335 0.335</td></tr><tr><td>Int-1-class</td><td>0.505</td><td colspan="3">0.100 0.060 0.060</td></tr><tr><td>: 200-step PGD accuracy for different val-</td><td/><td/><td/><td/></tr><tr><td>ues of perturbation size . Best ATA results are</td><td/><td/><td/><td/></tr><tr><td>highlighted (1st, 2nd, 3rd) at each column. Note</td><td/><td/><td/><td/></tr><tr><td>that ATA with = 0 reduces to natural accuracy.</td><td/><td/><td/><td/></tr></table>

Multi-step PGD accuracy.

Table content: <table><tr><td>Best ATA results are highlighted (1st,</td></tr><tr><td>2nd, 3rd) at each column.</td></tr></table>

ATA on different unforeseen attacks in(Kang et al., 2019). Best results in each column are high-

Table content: <table/>

B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921-2929, 2016.

Table content: <table/>

Peer review (iclr2020):
In summary, this paper studies if interpretation robustness (i.e., similar examples should have similar interpretation) can help enhance the robustness of the model, especially in terms of adversarial attacks. The study direction itself is interesting and very useful for the interpretation and adversarial attack community. Moreover, some promising results can be observed in part of the empirical study. However, this paper can be improved a lot as follows. 1. This paper states several times that *adversarial examples can be hidden from neural network interpretability*. It is not clear on the definition of *hidden* in terms of *interpretability*. Therefore, how this *hidden* is related and why this *hidden* is important are unclear too. 2. Many details are missing, which makes the proposal suspicious. For example, the proposed method has a tradeoff parameter lambda. However, the settings and affects are not discussed at all. Without a clear setup, the reproducibility and applicability is in doubt. 3. Some empirical results are overstated. For example, why 0.790 vs 0.890 and 0.270 vs 0.170 are comparable results? These results show the weakness of the proposed method. Further explanations can be provided. From the reported results, it could be useful to see results when the perturbation is even higher to check the limitation of the proposed method. 4. Besides the clarification in the writing mentioned above, some typos or errors should be fixed, e.g., f_t*(x*) - - f_t(x*) >=0 in the proof of proposition 1.