Title: EVADING DEFENSES TO TRANSFERABLE ADVERSAR-IAL EXAMPLES BY MITIGATING ATTENTION SHIFT

Abstract: Deep neural networks are vulnerable to adversarial examples, which can mislead classifiers by adding imperceptible perturbations. An intriguing property of adversarial examples is their good transferability, making black-box attacks feasible in real-world applications. Due to the threat of adversarial attacks, many methods have been proposed to improve the robustness, and several state-of-the-art defenses are shown to be robust against transferable adversarial examples. In this paper, we identify the attention shift phenomenon, which may hinder the transferability of adversarial examples to the defense models. It indicates that the defenses rely on different discriminative regions to make predictions compared with normally trained models. Therefore, we propose an attention-invariant attack method to generate more transferable adversarial examples. Extensive experiments on the ImageNet dataset validate the effectiveness of the proposed method. Our best attack fools eight state-of-the-art defenses at an 82% success rate on average based only on the transferability, demonstrating the insecurity of the defense techniques.

1. INTRODUCTION:

Recent progress in machine learning and deep neural networks has led to substantial improvements in various pattern recognition tasks such as image understanding (Simonyan & Zisserman, 2015; He et al., 2016a) , speech recognition (Graves et al., 2013) , and machine translation (Sutskever et al., 2014) . However, deep neural networks are highly vulnerable to adversarial examples (Biggio et al., 2013; Szegedy et al., 2014; Goodfellow et al., 2015) . They are maliciously generated by adding small perturbations to legitimate examples, but make deep neural networks produce unreasonable predictions. The existence of adversarial examples, even in the physical world (Kurakin et al., 2016; Eykholt et al., 2018; Athalye et al., 2018b) , has raised concerns in security-sensitive applications, e.g., self-driving cars, healthcare and finance.

Attacking deep neural networks has drawn an increasing attention since the generated adversarial examples can serve as a surrogate to evaluate the robustness of different models (Carlini & Wagner, 2017) and help to improve the robustness (Goodfellow et al., 2015; Madry et al., 2018) . Several methods have been proposed to generate adversarial examples with the knowledge of the gradient information of a given model, such as fast gradient sign method (Goodfellow et al., 2015) , basic iterative method (Kurakin et al., 2016) , and Carlini & Wagner (2017) 's method, which are known as white-box attacks. Moreover, it is shown that adversarial examples have cross-model transferability (Liu et al., 2017) , i.e., the adversarial examples crafted for one model can fool a different model with a high probability. The transferability of adversarial examples enables practical black-box attacks to real-world applications and induces serious security issues.

The threat of adversarial examples has motivated extensive research on building robust models or techniques to defend against adversarial attacks. These include training with adversarial examples (Goodfellow et al., 2015; Kurakin et al., 2017; Tramèr et al., 2018; Madry et al., 2018) , image denoising/transformation (Liao et al., 2018; Xie et al., 2018a; Guo et al., 2018) , leveraging generative models to move adversarial examples towards data manifold (Song et al., 2018; Samangouei et al., 2018) , and theoretically-certified defenses (Raghunathan et al., 2018; Wong & Kolter, 2018) . Although the non-certified defenses have demonstrated robustness against common attacks, they do so by causing obfuscated gradients, which can be easily circumvented by new attacks (Athalye et al., 2018a) . However, some of the defenses (Tramèr et al., 2018; Liao et al., 2018; Xie et al., 2018a Figure 1 : Demonstration of the attention shift phenomenon of the defense models compared with normally trained models. We adopt class activation mapping (Zhou et al., 2016) to visualize the attentive regions of three normally trained models-Inception v3 (Szegedy et al., 2016) , Inception ResNet v2 (Szegedy et al., 2017) , ResNet 152 (He et al., 2016a) and four defense models (Tramèr et al., 2018; Liao et al., 2018; Xie et al., 2018a; Guo et al., 2018) . These defense models focus their attention on slightly different regions compared with normally trained models, which may affect the transferability of adversarial examples. Guo et al., 2018) claim to be resistant to transferable adversarial examples, making black-box attacks difficult to evade these defenses.

In this paper, we identify attention shift, that the defenses make predictions based on slightly different discriminative regions compared with normally trained models, as a phenomenon which may hinder the transferability of adversarial examples to the defense models. For example, we show the attention maps of several normally trained models and defense models in Fig. 1 , to represent the discriminative regions for their predictions. It is apparent that the normally trained models have similar attention maps while the defenses induce shifting attention maps. The attention shift of the defenses is caused by either training under different data distributions (Tramèr et al., 2018) or transforming the inputs before classification (Liao et al., 2018; Xie et al., 2018a; Guo et al., 2018) . Therefore, the transferability of adversarial examples is largely reduced to the defenses since the structure information hidden in adversarial perturbations may be easily overlooked if a model focuses its attention on different regions.

To mitigate the effect of attention shift and evade the defenses by transferable adversarial examples, we propose an attention-invariant attack method. In particular, we generate an adversarial example for an ensemble of examples composed of an legitimate one and its shifted versions. Therefore the resultant adversarial example is less sensitive to the attentive region of the white-box model being attacked and may have a bigger chance to fool another black-box model with a defense mechanism based on attention shift. We further show that this method can be simply implemented by convolving the gradient with a pre-defined kernel under a mild assumption. The proposed method can be integrated into any gradient-based attack methods such as fast gradient sign method and basic iterative method. Extensive experiments demonstrate that the proposed attention-invariant attack method helps to improve the success rates of black-box attacks against the defense models by a large margin. Our best attack reaches an average success rate of 82% to evade eight state-of-the-art defenses based only on the transferability, thus demonstrating the insecurity of the current defenses.

2. RELATED WORK:

Adversarial Examples. Deep neural networks are shown to be vulnerable to adversarial examples first in the visual domain (Szegedy et al., 2014) . Then several methods are proposed to generate adversarial examples for the purpose of high success rates and minimal size of perturbations (Goodfellow et al., 2015; Kurakin et al., 2016; Carlini & Wagner, 2017) . They also exist in the physical world (Kurakin et al., 2016; Eykholt et al., 2018; Athalye et al., 2018b) . Although adversarial examples are recently crafted for many domains, we focus on image classification tasks in this paper.

Black-box Attacks. Black-box adversaries have no access to the architecture or parameters of the target model, which are under a more challenging threat model to perform attacks. The transferability of adversarial examples provides an opportunity to attack a black-box model (Liu et al., 2017) .

Several methods (Dong et al., 2018; Xie et al., 2018b) (Athalye et al., 2018a) . Although these defenses are not robust in the white-box setting, some of them (Tramèr et al., 2018; Liao et al., 2018; Xie et al., 2018a; Guo et al., 2018) empirically show the resistance against transferable adversarial examples in the black-box setting. In this paper, we focus on generating more transferable adversarial examples against these defenses.

3. METHODOLOGY:

In this section, we provide the detailed description of our algorithm. Let x real denote a real example and y denote the corresponding ground-truth label. Given a classifier f (x) : X → Y that outputs a label as the prediction for an input, we want to generate an adversarial example x adv which is visually indistinguishable from x real but fools the classifier, i.e., f (x adv ) = y.1 In most cases, the L p norm of the adversarial perturbation is required to be smaller than a threshold as ||x advx real || p ≤ . In this paper, we use the L ∞ norm as the measurement. For adversarial example generation, the objective is to maximize the loss function J(x adv , y) of the classifier, where J is often the cross-entropy loss. So the constrained optimization problem can be written as arg max

EQUATION

To solve this optimization problem, the gradient of the loss function with respect to the input needs to be calculated, termed as white-box attacks. However in some cases, we cannot get access to the gradient of the classifier, where we need to perform attacks in the black-box manner. We resort to transferable adversarial examples which are generated for a different white-box classifier but have high transferability for black-box attacks.

3.1. GRADIENT-BASED ADVERSARIAL ATTACK METHODS:

Several methods have been proposed to solve the optimization problem in Eq. ( 1). We give a brief introduction of them in this section.

Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2015) generates an adversarial example x adv by linearizing the loss function in the input space and performing one-step update as

EQUATION

where ∇ x J is the gradient of the loss function with respect to x. sign(•) is the sign function to make the perturbation meet the L ∞ norm bound. FGSM can generate more transferable adversarial examples but is usually not effective enough for attacking white-box models (Kurakin et al., 2017) .

Basic Iterative Method (BIM) (Kurakin et al., 2016) extends FGSM by iteratively applying gradient updates multiple times with a small step size α, which can be expressed as

EQUATION

To restrict the generated adversarial examples within the -ball of x real , we can clip x adv t after each update or set α = /T with T being the number of iterations. It has been shown that BIM induces much more powerful white-box attacks than FGSM at the cost of worse transferability (Kurakin et al., 2017; Dong et al., 2018) .

Momentum Iterative Fast Gradient Sign Method (MI-FGSM) (Dong et al., 2018) proposes to improve the transferability of adversarial examples by integrating a momentum term into the iterative attack method. The update procedure is

EQUATION

where g t gathers the gradient information up to the t-th iteration with a decay factor µ.

Diverse Inputs Iterative Fast Gradient Sign Method (Xie et al., 2018b) applies random transformations to the inputs and feeds the transformed images into the classifier for gradient calculation.

The image transformation includes random resizing and padding with a given probability. This method can be combined with the momentum-based method to further improve the transferability.

Carlini & Wagner (2017)'s method is a powerful optimization-based method. It uses an auxiliary variable v adv as x adv = 1 2 (tanh(v adv ) + 1), and optimizes v adv by solving

EQUATION

where the loss function J could be different from the cross-entropy loss. This method aims to find adversarial examples with minimal size of perturbations, to measure the robustness of different models. It also lacks the efficacy for black-box attacks like BIM.

3.2. ATTENTION-INVARIANT ATTACK METHOD:

Although many attack methods (Dong et al., 2018; Xie et al., 2018b) can generate adversarial examples with very high transferability across normally trained models, they are less effective to attack defense models in the black-box manner. Some of the defenses (Tramèr et al., 2018; Liao et al., 2018; Xie et al., 2018a; Guo et al., 2018) are shown to be quite robust against black-box attacks. So we want to answer that: Are these defenses really free from transferable adversarial examples?

We identify the attention shift phenomenon which may inhibit the transferability of adversarial examples to the defenses. The attention shift refers to that the discriminative regions used by the defenses to identify object categories are slightly different from those used by normally trained models, as shown in Fig. 1 . The adversarial examples generated for one model can be hardly transferred to another model with attention shift since that the structure information in adversarial perturbations may be easily destroyed if the model focuses its attention on different regions.

To reduce the effect of attention shift, we propose an attention-invariant attack method. In particular, rather than optimizing the objective function at a single point as Eq. ( 1), the proposed method uses a set of shifted images to optimize an adversarial example as arg max

EQUATION

where T ij (x) is a transformation operation that shifts image x by i and j pixels along the twodimensions respectively, i.e., each pixel (a, b) of the transformed image is

T ij (x) a,b = x a-i,b-j ,

and w ij is the weight for the loss J(T ij (x adv ), y). We set i, j ∈ {-k, ..., 0, ..., k} with k being the maximal number of pixels to shift. With this method, the generated adversarial perturbations are less sensitive to the attentive regions of the white-box model, which may be transferred to another model with a higher success rate. However, we need to calculate the gradients for (2k + 1) 2 images, which introduces much more computations. Sampling a small number of shifted images for gradient calculation is a feasible way (Athalye et al., 2018b) . But we show that we can perform attacks by calculating the gradient for only one image under a mild assumption.

Convolutional neural networks are known to have the shift-invariant property (LeCun & Bengio, 1995) , that an object in the input can be recognized in spite of its position. Pooling layers contribute resilience to slight transformation of the input. Therefore, we make an assumption that the shifted image T ij (x) is almost the same as x as inputs to the models, as well as their gradients

EQUATION

)

Based on this assumption, we calculate the gradient of the loss defined in Eq. ( 6) at a point x as

∇ x i,j w ij J(T ij (x), y) x= x = i,j w ij ∇ x J(T ij (x), y) x= x = i,j w ij ∇ Tij (x) J(T ij (x), y) • ∂T ij (x) ∂x x= x = i,j w ij T -i-j ∇ x J(x, y) x=Tij ( x) ≈ i,j w ij T -i-j (∇ x J(x, y) x= x).

(8)

Given Eq. ( 8), we do not need to calculate the gradients for (2k + 1)2 images. Instead, we only need to get the gradient for the unchanged image x and then average all the shifted gradients. This procedure is equivalent to convolving the gradient with a kernel composed of all the weights w ij as i,j

EQUATION

)

where W is the kernel matrix of size (2k + 1) × (2k + 1) with W i,j = w -i-j . In this paper, we generate the kernel W from a two-dimensional Gaussian function because: 1) the images with bigger shifts have relatively lower weights to make the adversarial perturbation fool the model at the unshifted image effectively; 2) by using a Gaussian function, this procedure is known as Gaussian blur, which is widely used in image processing.

Note that we only illustrate how to calculate the gradient of the loss function defined in Eq. ( 6), but do not specify the update algorithm for generating adversarial examples. This indicates that our method can be integrated into any gradient-based attack methods including FGSM, BIM, MI-FGSM, etc. Specifically, in each step we calculate the gradient ∇ x J(x adv t , y) at the current solution x adv t , then convolve the gradient with the pre-defined kernel W , and finally get the new solution x adv t+1 following the update rule in different attack methods (In FGSM, there is only one step of update).

4. EXPERIMENTS:

In this section, we present the experimental results to demonstrate the effectiveness of the proposed method on improving the transferability of adversarial examples to the defense models.

4.1. EXPERIMENTAL SETTINGS:

We use an ImageNet-compatible dataset 2 comprised of 1000 images to conduct experiments. This dataset was used in the NIPS 2017 adversarial competition. We include eight defense models which are shown to be robust agsinst black-box attacks on the ImageNet dataset. These are • Inc-v3 ens3 , Inc-v3 ens4 , IncRes-v2 ens (Tramèr et al., 2018) ; • high-level representation guided denoiser (HGD, rank-1 submission in the NIPS 2017 defense competition) (Liao et al., 2018 ); • input transformation through random resizing and padding (R&P, rank-2 submission in the NIPS 2017 defense competition) (Xie et al., 2018a ); • input transformation through JPEG compression or total variance minimization (TVM) (Guo et al., 2018 ); • rank-3 submission3 in the NIPS 2017 defense competition (NIPS-r3).

To attack these defenses based on the transferability, we also include four normally trained models-Inception v3 (Inc-v3) (Szegedy et al., 2016) , Inception v4 (Inc-v4), Inception ResNet v2 (IncRes-v2) (Szegedy et al., 2017), and ResNet v2-152 (Res-v2-152) (He et al., 2016b) , as the white-box models to generate adversarial examples. In our experiments, we integrate our method into the fast gradient sign method (FGSM) (Goodfellow et al., 2015) , momentum iterative fast gradient sign method (MI-FGSM) (Dong et al., 2018) and diverse input iterative fast gradient sign method with momentum (DIM) (Xie et al., 2018b) . We do not include the basic iterative method and Carlini & Wagner (2017)'s method since that they are not good at generating transferable adversarial examples (Dong et al., 2018) . We denote the attacks combined with our attention-invariant method as A-FGSM, A-MI-FGSM and A-DIM respectively.

For the settings of hyper-parameters, we set the maximum perturbation to be = 16 among all experiments with pixel value in [0, 255] . For the iterative attack methods, we set the number of iteration as 10 and the step size as α = 1.6. For MI-FGSM and A-MI-FGSM, we adopt the default dacay factor µ = 1.0. For DIM and A-DIM, the transformation probability is set to 0.7. Please note that the settings for each attack method and its attention-invariant version are the same, because our method is not concerned with the specific attack precedure.

4.2. SINGLE-MODEL ATTACKS:

We first perform adversarial attacks for Inc-v3, Inc-v4, IncRes-v2 and Res-v2-152 respectively using FGSM, MI-FGSM, DIM and their extensions by combining with the proposed attention-invariant attack method as A-FGSM, A-MI-FGSM and A-DIM. We then use the generated adversarial examples to attack the eight defense models we consider based only on the transferability. We report the success rates of black-box attacks in Table 1 , Table 2 and Table 3 , where the success rates are the misclassification rates of the corresponding defense models with adversarial images as inputs. In the attention-invariant based attacks, we set the size of the kernel matrix W as 15 × 15 across all experiments, and we will study the effect of kernel size in Section 4.4.

From the tables, we observe that the success rates against the defenses are improved by a large margin when using the proposed method regardless of the attack algorithms or the white-box models being attacked. In general, the attention-invariant based attacks consistently outperform the baseline attacks by 5% ∼ 30%. In particular, when using A-DIM, the combination of our method and DIM, to attack the IncRes-v2 model, the resultant adversarial examples have about 60% success rates against the defenses (as shown in Table 3 ). It demonstrates the vulnerability of the current defenses against black-box attacks. The results also validate the effectiveness of the proposed method. Although we only compare the results of our attack method with baseline methods against the defense models, our attacks remain the success rates of baseline attacks in the white-box setting and the black-box setting against normally trained models, which will be shown in the Appendix.

We show several adversarial images generated for the Inc-v3 model by FGSM and A-FGSM in Fig. 2 . It can be seen that by using A-FGSM, in which the gradients are convolved by a kernel W before applying to the raw images, the adversarial perturbations are much smoother than those generated by FGSM. The smooth effect also exists in other attention-invariant based attacks.

4.3. ENSEMBLE-BASED ATTACKS:

In this section, we further present the results when adversarial examples are generated for an ensemble of models. Liu et al. (2017) have shown that attacking multiple models at the same time can improve the transferability of the generated adversarial examples. It is due to that if an example remains adversarial for multiple models, it is more likely to transfer to another black-box model.

We adopt the ensemble method proposed by Dong et al. (2018) , which fuses the logit activations of different models. We attack the ensemble of Inc-v3, Inc-v4, IncRes-v2 and Res-v2-152 with equal ensemble weights using FGSM, A-FGSM, MI-FGSM, A-FGSM, DIM and A-DIM respectively. We also set the kernel size in the attention-invariant based attacks as 15 × 15.

In Table 4 , we show the results of black-box attacks against the eight defenses. The proposed method also improves the success rates across all experiments over the baseline attacks. It should by noted that the adversarial examples generated by A-DIM can fool the state-of-the-art defenses at an 82% success rate on average based on the transferability. And the adversarial examples are generated for normally trained models unaware of the defense strategies. The results in the paper demonstrate that the current defenses are far from real security, and cannot be deployed in real-world applications.

4.4. THE EFFECT OF KERNEL SIZE:

The size of the kernel W plays a key role for improving the success rates of black-box attacks. If the kernel size equals to 1 × 1, the attention-invariant based attacks degenerate to their vanilla versions. Therefore, we conduct an ablation study to examine the effect of different kernel sizes.

We attack the Inc-v3 model by A-FGSM, A-MI-FGSM and A-DIM with the kernel length ranging from 1 to 21 with a granularity 2. In Fig. 3 , we show the success rates against five defense models-

:

This corresponds to untargeted attack. The method in this paper can be simply extended to targeted attack.

https://github.com/tensorflow/cleverhans/tree/master/examples/nips17_ adversarial_competition/dataset

https://github.com/anlthms/nips-2017/tree/master/mmd

Figures and Tables:

Figure 2: The adversarial examples generated for Inc-v3 using FGSM and A-FGSM.

have been proposed to improve the transferability, which enable powerful black-box attacks. Besides the transfer-based black-box attacks, there is another line of works that perform attacks based on adaptive queries. For example,Papernot et al. (2017) use queries to distill the knowledge of the target model and train a surrogate model. It therefore turns the black-box attacks to the white-box attacks. Recent methods use queries to estimate the gradient or the decision boundary of the black-box model (??) to generate adversarial examples. However, these methods usually require tremendous number of queries, which may be impractical in real-world applications. In this paper, we resort to transferable adversarial examples for black-box attacks.Defend against Adversarial Attacks. A large variety of methods have been proposed to increase the robustness of deep learning models. Besides directly making the models produce correct predictions for adversarial examples, some methods attempt to detect them instead(?Metzen et al., 2017;  ?). However most of the non-certified defenses demonstrate the robustness by causing obfuscated gradients, which are successfully circumvented by new developed attacks

Table content: <table/>

The success rates (%) of black-box attacks against eight defenses. The adversarial examples are crafted for Inc-v3, Inc-v4, IncRes-v2 and Res-v2-152 respectively using FGSM and A-FGSM.

Table content: <table><tr><td/><td>Attack</td><td>Inc-v3ens3</td><td>Inc-v3ens4</td><td>IncRes-v2ens</td><td>HGD</td><td>R&amp;P</td><td>JPEG</td><td>TVM</td><td>NIPS-r3</td></tr><tr><td>Inc-v3</td><td>FGSM A-FGSM</td><td>15.6 28.2</td><td>14.7 28.9</td><td>7.0 22.3</td><td>2.1 18.4</td><td>6.5 19.8</td><td>19.9 25.5</td><td>18.8 30.7</td><td>9.8 24.5</td></tr><tr><td>Inc-v4</td><td>FGSM A-FGSM</td><td>16.2 28.2</td><td>16.1 28.3</td><td>9.0 21.4</td><td>2.6 18.1</td><td>7.9 21.6</td><td>21.8 27.9</td><td>19.9 31.8</td><td>11.5 24.6</td></tr><tr><td>IncRes-v2</td><td>FGSM A-FGSM</td><td>18.0 32.8</td><td>17.2 33.6</td><td>10.2 28.1</td><td>3.9 25.4</td><td>9.9 28.1</td><td>24.7 32.4</td><td>23.4 38.5</td><td>13.3 31.4</td></tr><tr><td>Res-v2-152</td><td>FGSM A-FGSM</td><td>20.2 34.6</td><td>17.7 34.5</td><td>9.9 27.8</td><td>3.6 24.4</td><td>8.6 27.4</td><td>24.0 32.7</td><td>22.0 38.1</td><td>12.5 30.1</td></tr></table>

The success rates (%) of black-box attacks against eight defenses. The adversarial examples are crafted for Inc-v3, Inc-v4, IncRes-v2 and Res-v2-152 respectively using MI-FGSM and A-MI-FGSM.

Table content: <table><tr><td>Attack</td><td>Inc-v3ens3</td><td>Inc-v3ens4</td><td>IncRes-v2ens</td><td>HGD</td><td>R&amp;P</td><td>JPEG</td><td>TVM</td><td>NIPS-r3</td></tr></table>

Peer review (iclr2019):
 *Summary:* The present paper propose a new way of overcoming state of the art defences against adversarial attacks on CNN. This is done efficiently by noticing that many defences agains adversarial attacks have a an attention shift, and exploiting it efficiently via CNN invariance to translations. The authors shows that the proposed method can fool many state of the art defences. *Clarity:* The paper is well written and easy to follow. To the best of my knowledge the maths are correct. However the shift mechanism is still not clearly understood. See answer to R1. *Originality and Significance:* to the best of my knowledge the proposed method is new and the paper looks complete enough to be fully reproducible. The experiments show that on a wide selection of dataset, many state of the art method are significantly sensitive to the proposed attack. If the paper is accepted ot would be really nice to provide a link to the code used. As the experiments showed that the proposed attacks fools many state of the art defences and the method is new I believe this paper has a good significance/impact. *Questions:* - Have you tried to fool the defences with other invariants that CNN are also known to be invariant too? eg rotations or diffeomorphism? Would it be possible to obtain the same computational *trick* as in eq 9? - I would see the continuous version of the method as integrating over all the possible translation T_{uv} over a measure mu(u, v) (instead of T_{ij}, w_{ij}) is that correct? - Have you tried something else than the gaussian kernel for the convolution? Do you have a proof / explaination / experimental evidence that other kernels such as a uniform / laplacian are less performant than a Gaussian kernel? (point 1) page 5 is convincing but point 2) is irrelevant in my opinion). - Could you propose new defences agains your attack? Typos: some references (eg p 3) are broken (ie ??)