Year: 2018

Abstract:
State-of-the-art deep neural networks are vulnerable to adversarial examples, formed by applying small but malicious perturbations to the original inputs. Moreover, the perturbations can transfer across models: adversarial examples generated for a specific model will often mislead other unseen models. Consequently the adversary can leverage it to attack deployed systems without any query, which severely hinders the application of deep learning, especially in the safety-critical areas. In this work, we empirically study how two classes of factors those might influence the transferability of adversarial examples. One is about model-specific factors, including network architecture, model capacity and test accuracy. The other is the local smoothness of loss surface for constructing adversarial examples. Inspired by these understandings on the transferability of adversarial examples, we then propose a simple but effective strategy to enhance the transferability, whose effectiveness is confirmed by a variety of experiments on both CIFAR-10 and ImageNet datasets.

heading: 1 INTRODUCTION
text: Recently Szegedy et al. (2013) showed that an adversary is able to fool deep neural network models into producing incorrect predictions by manipulating the inputs maliciously. The corresponding manipulated samples are called adversarial examples. More severely, it was found that adversarial examples have cross-model generalization ability, i.e., that adversarial examples generated from one model can fool another different model with a high probability. We refer to such property as transferability. By now, more and more deep neural network models are applied in real-world applications, such as speech recognition, computer vision, etc.. Consequently, an adversary can employ the transferability to attack those deployed models without querying the systems (Papernot et al., 2016b; Liu et al., 2017), inducing serious security issues.
Understanding the mechanism of transferability could potentially provide various benefits. Firstly, for the already deployed deep neural network models in real systems, it could help to design better strategies to improve the robustness against the transfer-based attacks. Secondly, revealing the mystery behind the transferability of adversarial examples could also expand the existing understandings on deep learning, particularly the effects of model capacity (Madry et al., 2018; Fawzi et al., 2015) and model interpretability (Dong et al., 2017; Ross and Doshi-Velez, 2018). Therefore, studying the transferability of adversarial examples in the context of deep networks is of significant importance.
In this paper, we numerically investigate two classes of factors that might influence the adversarial transferability, and further provide a simple but rather effective strategy for enhancing the transferability. Our contributions are summarized as follows.
• We find that adversarial transfer is not symmetric, which means that adversarial examples generated from model A can transfer to model B easily does not implies the reverse is also natural. Second, we find that adversarial examples generated from a deep model appear less transferable than those from a shallow model. We also explore the impact of the nonsmoothness of the loss surface. Specifically, we find that the local non-smoothness of loss surface harms the transferability of generated adversarial examples. • Inspired by previous investigations, we propose the smoothed gradient attack to improve
the adversarial transferability, which employs the locally averaged gradient instead of the
original one to generate adversarial examples. Extensive numerical validations justify the effectiveness of our method.

heading: 2 RELATED WORK
text: The phenomenon of adversarial transferability was first studied by Szegedy et al. (2013). By utilizing the transferability, Papernot et al. (2016b;a) proposed a practical black-box attack by training a substitute model with limited queried information. Liu et al. (2017) first studied the targeted transferability and introduced the ensemble-based attacks to improve the transferability. More recently, Dong et al. (2018) showed that the momentum can help to boost transferability significantly.
Meanwhile, there exist several works trying to explain the adversarial transferability. Papernot et al. (2016a) attributed the transferability to the similarity between input gradients of source and target models. Liu et al. (2017) proposed to use the visualization technique to see the large-scale similarity of decision boundaries. However, our numerical investigations imply that these similarity-based explanations have their intrinsic limitation that they cannot explain the non-symmetric property of adversarial transferability.
Our smoothed gradient attacks that enhance the transferability by utilizing the information in the small neighborhood of the clean example is inspired by the works on shattered gradients (Balduzzi et al., 2017) and model interpretability (Smilkov et al., 2017). Recently, similar strategies are also explored by Athalye et al. (2018a) and Warren He (2018) for white-box attacks, whereas we focus on the black-box settings. Our method is also related to the work by Athalye et al. (2018b), which introduced the expectation over transformation (EOT) method to increase robustness of adversarial examples. The EOT formulation is similar to our objective (7), but they did not study the transferability. Also our motivations are totally different from theirs.

heading: 3 BACKGROUND

heading: 3.1 ADVERSARIAL ATTACK
text: Let f(x) : Rd 7→ RK denote the classifier. In deep learning, it is found that for almost any sample x and its label ytrue, there exists a small perturbation η that is nearly imperceptible to human such that
argmax i
fi(x) = y true, argmax i fi(x+ η) 6= ytrue. (1)
We call η the adversarial perturbation and correspondingly xadv := x+ η the adversarial example. The attack (1) is called a non-targeted attack since the adversary has no control over which class the input x will be misclassified to. In contrast, a targeted attack aims at fooling the model to produce a wrong label specified by the adversary, i.e. argmaxi fi(x+ η) = y target.
In this paper, we consider the pure black-box attacks. This means the adversary has no knowledge of the target model (e.g. architecture and parameters) and is also not allowed to query any input-output pair from the target model. On the contrary, the white-box attack means that the target model is available to the adversary.

heading: 3.2 GENERATING ADVERSARIAL EXAMPLES
text: Modeling In general, crafting adversarial examples can be modeled as the following optimization problem,
maximizex′ J(x′) := J(f(x′), ytrue)
s.t. ‖x′ − x‖∞ ≤ ε, (2)
where J is some loss function measuring the discrepancy between the model prediction and ground truth; the `∞ norm ‖ · ‖∞ is used to quantify the magnitude of the perturbation. Other norm is also possible, but we focus on `∞ norm in this paper. To improve the strength of adversarial transferability, instead of using a single model, Liu et al. (2017) proposed the ensemble attack, which generates
adversarial examples against a model ensemble f1(x), f2(x), · · · , fM (x):
maximizex′ J
( 1
M M∑ k=1 fk(x′), ytrue ) s.t. ‖x′ − x‖∞ ≤ ε.
(3)
Optimizer We use the following iteration to solve (2) and (3), xt+1 = projD (xt + α sign(∇xJ(xt))) . (4) where D = [0, 255]d ∩ {x′ | ‖x′ − x‖ ≤ ε} and α is the step size. We call the attack that evolves (4) for T steps iterative gradient sign method (IGSM) attack (Kurakin et al., 2017; Madry et al., 2018). Furthermore, the famous fast gradient sign method (FGSM) is a special case with α = ε, T = 1.
Dong et al. (2018) recently proposed the momentum-based attack as follows gt+1 = µ gt +∇J(xt)/‖∇J(xt)‖1 xt+1 = projD (xt + α sign(gt)) ,
(5)
where µ is the decay factor of momentum. By using this method, they won the first-place in NIPS 2017 Non-targeted Adversarial Attack and Targeted Adversarial Attack competitions. We will call it mIGSM attack in this paper.

heading: 4 EVALUATION OF ADVERSARIAL TRANSFERABILITY
text: Datasets To evaluate the transferability, three datasets including MNIST, CIFAR-10 and ImageNet are considered. For ImageNet, evaluations on the whole ILSVRC2012 validation dataset are too timeconsuming. Therefore, in each experiment we randomly select 5, 000 images that can be correctly recognized by all the examined models to form our new validation set.
Models (i) For MNIST, we trained fully connected networks (FNN) ofD hidden layers, with the width of each layer being 500. (ii) For CIFAR-10, we trained five models: lenet,resnet20, resnet44, resnet56, densenet. The test accuracies of them are 76.9%, 92.4%, 93.7%, 93.8% and 94.2%, respectively. (iii) For ImageNet, the pre-trained models provided by PyTorch are used. The Top-1 and Top-5 accuracies of them can be found on website1. To increase the reliability of experiments, all the models have been tested. However, for a specific experiment we only choose some of them to present since the findings are consistent among all the tested models.
Criteria Given a set of adversarial pairs, {(xadvi , ytruei )}Ni=1, we calculate the Top-1 success rate (%) fooling a given model f(x) by 100N ∑N k=1 1[argmaxi fi(x adv k ) 6= ytruek ]. For targeted attacks, each image xadv is associated with a pre-specified label ytarget 6= ytrue. Then we evaluate the performance of the targeted attack by the following Top-1 success rate: 100N ∑N k=1 1[argmaxi fi(x adv k ) = y target k ]. The corresponding Top-5 rates can be computed in a similar way.

heading: 5 HOW MODEL-SPECIFIC FACTORS AFFECT TRANSFERABILITY
text: Previous studies on adversarial transferability mostly focused on the influence of attack methods (Liu et al., 2017; Dong et al., 2018; Tramèr et al., 2017; Kurakin et al., 2017). However it is not clear how the choice of source model affects the success rate transferring to target models. In this section, we investigate this issue from three aspects including architecture, test accuracy and model capacity.

heading: 5.1 ARCHITECTURE
text: We first explore how the architecture similarity between source and target model contributes to the transferability. This study is crucial since it can provide us guidance to choose the appropriate source models for effective attacks. To this end, three popular architectures including ResNet, DenseNet and VGGNet are considered, and for each architecture, two networks are selected. Both one-step and multi-step attacks are performed on ImageNet dataset. Table 1 presents the experiment results, and the choice of hyper-parameters is detailed in the caption.
1http://pytorch.org/docs/master/torchvision/models.html
We can see that the transfers between two models are non-symmetric, and this phenomenon is more obvious for the models with different architectures. For instance, the success rates of IGSM attacks from densenet121 to vgg13 bn is 58.7%, however the rate from vgg13 bn to densenet121 has only 15.9%. The lack of symmetry implies that previous similarity-based explanations of adversarial transferability are quite limited.
Another interesting observation is that success rates between models with similar architectures are always much higher. For example the success rates of IGSM attacks between vgg13 bn and vgg16 bn are higher than 90%, nearly twice the ones of attacks from any other architectures.

heading: 5.2 MODEL CAPACITY AND TEST ACCURACY
text: We first study this problem on ImageNet dataset. A variety of models are used as source models to perform both FGSM and IGSM attacks against vgg19 bn, and the results are displayed in Figure 1. The horizontal axis is the Top-1 test error, while the vertical axis is the number of model parameters that roughly quantifies the model capacity.
We can see that the models with powerful attack capability concentrate in the bottom left corner, whereas for those models with either large test error or large number of parameters, the fooling rates are much lower. We also tried other models with results shown in Appendix C , and the results show no difference.
We suspect that the impact of test accuracy is due to that the decision boundaries of high-accuracy models are similar, since they all approximate the ground-truth decision boundary of true data very well. On the contrary, the model with low accuracy has a decision boundary relatively different from the high-accuracy models. Here the targeted network vgg19 bn is a high-accuracy model. Therefore it is not surprising to observe that high-accuracy models tend to exhibit stronger attack capability. This phenomenon implies a kind of data-dependent transferability, which is different from the architecture-specific transfers observed in the previous section.
It is somewhat strange that adversarial examples generated from deeper model appear less transferable. To further confirm this observation, we conduct additional experiments on MNIST and CIFAR-10. Table 2 shows the results, which is basically consistent. This observation suggests us not to use deep models as the source models when performing transfer-based attacks, although we have not fully understand this phenomenon.

heading: 6 NON-SMOOTHNESS OF THE LOSS SURFACE
text: In this section, we consider that how the smoothness of loss surface J(x) affects the transferability. For simplicity, let g(x) = ∇xJ(x) denote the gradient. Smilkov et al. (2017) showed that gradient g(x) is very noisy and uninformative for visualization, though the model is trained very well. Balduzzi et al. (2017) studied a similar phenomenon that gradients of deep networks are extremely shattered. Both of them implies that the landscape is locally extremely rough. We suspect that this local non-smoothness could harm the transferability of adversarial examples.
6.1 INTUITION
For simplicity, assume model A and B are the source and target models which are well trained with high test accuracies, respectively. Previous methods use gA(x) to generate adversarial perturbations, so the transferability mainly depends on how much sensitivity of gA for model A can transfer to model B. As illustrated in Figure 2, where three curves denote the level sets of three models, we can see that the non-smoothness can hurt the transferability, since both model A and B have very high test accuracy, their level sets should be similar globally, and JB(x) is probably unstable along gA. As illustrated, the local oscillation of gA makes the sensitivity less transferable. One way to alleviate this is to smooth the landscape JA, thereby yielding a more transferable gradient GA, i.e. 〈ĜA, ĝB〉 > 〈ĝA, ĝB〉. Here we use G denote the gradient of the smoothed loss surface.

heading: 6.2 JUSTIFICATION
text: To justify the previous arguments, we consider smoothing the loss surface by convolving it with a Gaussian filter, then the smoothed loss surface is given by Jσ(x) := Eξ∼N (0,I)[J(x + σξ)]. The corresponding gradient can be calculated by
Gσ(x) = Eξ∼N (0,I)[g(x+ σξ)]. (6)
The extent of smoothing is controlled by σ, and please refer to Appendix A to see the smoothing effect. We will show that Gσ(x) is more transferable than g(x) in the following.
Gradient similarity We first quantify the cosine similarity between gradients of source and target models, respectively. Two situations are considered: vgg13 bn→vgg16 bn, densenet121→vgg13 bn, which correspond the within-architecture and cross-architecture transfers, respectively. We choose σ = 15, and the expectation in (6) is estimated by using 1m ∑m i=1 g(x+ ξi). To verify the averaged gradients do transfer better, we plot the cosine similarity against the number of samples m. In Figure 3a, as expected, we see that the cosine similarity between GA and gB are indeed larger than
the one between gA and gB . Moreover, the similarity increases with m monotonically, which further justifies that GA is more transferable than gA.
Visualization In Figure 3b we visualize the transferability by comparing the decision boundaries of model A (resnet34 ) and model B (densenet121). The horizontal axis represents the direction of GA of resnet34, estimated by m = 1000, σ = 15, and the vertical axis denotes orthogonal direction hA := gA − 〈gA, ĜA〉ĜA. This process means that we decompose the gradient into two terms: gA = αGA + βhA with 〈GA, hA〉 = 0 . Each point in the 2-D plane corresponds to the image perturbed by u and v along each direction, clip(x+u ĜA+ v ĥA, 0, 255), where x is the clean image. The color corresponds to the label predicted by the target model.
It can be easily observed that for model A, a small perturbation in both directions can produce wrong classification. However, when applied to model B, the sensitivities of two directions dramatically change. The direction of hA becomes extremely stable, whereas to some extent GA preserves the sensitivity, i.e. GA do transfer to model B better than hA. This suggests that for gradient gA, the noisy part hA is less transferable than the smooth part GA. We also tried a variety of other models shown in Appendix C, and the results are the same.

heading: 7 SMOOTHED GRADIENT ATTACK

heading: 7.1 METHOD
text: Inspired by the previous investigations, enhancing the adversarial transferability can be achieved by smoothing the loss surface. Then our objective becomes,
maximize Jσ(x′) := Eξ∼N (0,I)[J(x′ + σξ)] s.t. ‖x′ − x‖ ≤ ε. (7)
Intuitively, this method can also be interpreted as generating adversarial examples that are robust to Gaussian perturbation. Expectedly, the generated robust adversarial examples can still survive easily in spite of the distinction between source and target model.
If use the iterative gradient sign method to solve (7), we have the following iteration:
Gt = 1
m m∑ i=1 ∇J(xt + ξi), ξi ∼ N (0, σ2I)
xt+1 = projD (xt + α sign (Gt)) ,
(8)
where Gt is a mini-batch approximation of the smoothed gradient (6). Compared to the standard IGSM, the gradient is replaced by a smoothed version, which is endowed with stronger transferability. Therefore we call this method sg-IGSM. The special case T = 1, α = ε, is accordingly called sg-FGSM. Any other optimizer can be used to solve the (7) as well, and we only need to replace the original gradient with the smoothed one.

heading: 7.2 CHOICE OF HYPER PARAMETERS
text: We first explore the sensitivity of hyper parameters m,σ when applying our smoothed gradient technique. We take ImageNet dataset as the testbed, and sg-FGSM attack is examined. To increase the reliability, four attacks are considered here. The results are shown in Figure 4.
We see that sg-FGSM consistently outperforms FGSM for all distortion level, although the improvement varies for different ε. Furthermore, larger m leads to higher success rate due to the better estimation of the smooth part of gradient, and the benefit starts to saturate after m ≥ 30. For the smoothing factor σ, we find neither too large nor too small value can work well, and the optimal σ is about 15. Overly large σ will introduce a large bias in (8), and extremely small σ is unable to smooth the landscape enough.
Therefore, in the subsequent sections, we will use m = 20, σ = 15 to estimate the smoothed gradient and only report the result for one distortion level.

heading: 7.3 EFFECTIVENESS
text: Single-model attack We first test the effectiveness of our method for single-model based attacks on non-targeted setting. To make a fair comparison, we fix the number of gradient calculation per sample at 100. Specifically, for sg-IGSM we have T = 5 due to m = 20, whereas T = 100 for IGSM. The results are shown in Table 3. We see that smoothed gradients do enhance the transferability dramatically for all the attacks considered. Please especially note those bold rates, where the improvements have reached about 30%.
Ensemble attack In this part, we examine the ensemble-based attack on the targeted setting2. For targeted attacks, generating an adversarial example predicted by target models as a specific label is
2Compared to non-targeted attack, we find that a larger step size α is necessary for generating strong targeted adversarial examples. Readers can refer to Appendix B for more details on this issue, though we cannot fully understand it. Therefore a much larger step size than the non-target attacks is used in this experiment.
too hard, resulting in a very low success rate. We instead adopt both Top-1 and Top-5 success rates as our criteria to better reflect the improvement of our method. A variety of model ensembles are examined, and the results are reported in Table 4.
As we can see, it is clear that smoothed gradient attacks outperform the corresponding normal ones by a remarkable large margin. More importantly, the improvement never be harmed compared to single-model case in Table 3, which implies that smoothed gradient can be effectively combined with ensemble method without compromise.
Momentum attack In this experiment, three networks of different architectures are selected. As suggested in Dong et al. (2018), we choose µ = 1, and all attacks are iterated for T = 5 with step size α = 5. In Table 5, we report the results of non-targeted attacks of three attacks including mIGSM, sg-IGSM and mIGSM with smoothed gradient (sg-mIGSM). It is shown that our method clearly outperforms momentum-based method for all the cases. Moreover, by combining with the smoothed gradient, the effectiveness of momentum attacks can be further improved significantly.

heading: 7.4 ROBUSTNESS
text: Smoothed gradient attacks can be viewed as generating adversarial examples robust against Gaussian noise perturbations. Therefore, we are interested in how robust is the adversarial example against other image transformations. To quantify the influence of transformations, we use the notion of destruction rate defined by Kurakin et al. (2017). The lower is this rate, the more robust are the adversarial examples.
Densenet121 and resnet34 are chosen as our source and target model, respectively; and four image transformations are considered: rotation, Gaussian noise, Gaussian blur and JPEG compression. Figure 5 displays the results, which show that adversarial examples generated by our methods are more robust than those generated by vanilla methods. This numerical result is interesting, since we only explicitly increase the robustness against Gaussian noise in generating adversarial examples. We speculate that the robustness can also transfer among different image transforms.

heading: 8 CONCLUSION
text: In this paper, we first investigated the influence of model-specific factors on the adversarial transferability. It is found that the model architecture similarity plays a crucial role. Moreover models with lower capacity and higher test accuracy are endowed with stronger capability for transfer-based attacks. we second demonstrate that the non-smoothness of loss surface hinders the transfer of adversarial examples. Motivated by these understandings, we proposed the smoothed gradient attack that can enhance the transferability of adversarial examples dramatically. Furthermore, the smoothed gradient can be combined with both ensemble and momentum based approaches rather effectively.

heading: C ADDITIONAL RESULTS
text: How Model Capacity and Test Accuracy Affect Transferability In this part, we additionally explore the impact of model capacity and test accuracy by using resnet152 as our target model and perform transfer-based attacks against from a variety of models and the results shown in Figure 8. The observations are consistent with the observation in Section 5.2.
Visualization of Decision Boundary In this part, we provide additional results on the visualization of the decision boundary. Different Figure 3b, we here consider the sign of two directions, since the attack method actually updates using the sign (g) rather than g. In Figure 9, we show the decision boundary. Each point corresponds to perturbed image:
clip(x+ usign (GA) + vsign (hA) , 0, 255).
Here the model A is resnet34, and the definitions of GA and hA are the same as before. The color denotes the label predicted by the target model, and the gray color corresponds to the ground-truth label. We can see that the smoothed gradient GA is indeed more transferable than the noise part hA.
Additional Results for the Effectiveness of Smoothed Gradient Attacks In this part, we present some additional results to show the effectiveness of smoothed gradient attack.

heading: D THE EXAMINED IMAGE FOR VISUALIZATION OF DECISION BOUNDARY

Review summaries:
The paper explores how the architecture, smoothness of the decision boundary and test accuracy of a model impacts the transferability of examples produced from it.  The paper provides a couple of novel insights, such as the asymmetry when transferring adversarial examples from one model to another. In addition, a novel method is proposed to enhance the transferability of adversarial examples from any model, through using smoothed gradients.