Exploring and Enhancing the Transferability of Adversarial Examples

1 INTRODUCTION . Recently Szegedy et al . ( 2013 ) showed that an adversary is able to fool deep neural network models into producing incorrect predictions by manipulating the inputs maliciously . The corresponding manipulated samples are called adversarial examples . More severely , it was found that adversarial examples have cross-model generalization ability , i.e. , that adversarial examples generated from one model can fool another different model with a high probability . We refer to such property as transferability . By now , more and more deep neural network models are applied in real-world applications , such as speech recognition , computer vision , etc .. Consequently , an adversary can employ the transferability to attack those deployed models without querying the systems ( Papernot et al. , 2016b ; Liu et al. , 2017 ) , inducing serious security issues . Understanding the mechanism of transferability could potentially provide various benefits . Firstly , for the already deployed deep neural network models in real systems , it could help to design better strategies to improve the robustness against the transfer-based attacks . Secondly , revealing the mystery behind the transferability of adversarial examples could also expand the existing understandings on deep learning , particularly the effects of model capacity ( Madry et al. , 2018 ; Fawzi et al. , 2015 ) and model interpretability ( Dong et al. , 2017 ; Ross and Doshi-Velez , 2018 ) . Therefore , studying the transferability of adversarial examples in the context of deep networks is of significant importance . In this paper , we numerically investigate two classes of factors that might influence the adversarial transferability , and further provide a simple but rather effective strategy for enhancing the transferability . Our contributions are summarized as follows . • We find that adversarial transfer is not symmetric , which means that adversarial examples generated from model A can transfer to model B easily does not implies the reverse is also natural . Second , we find that adversarial examples generated from a deep model appear less transferable than those from a shallow model . We also explore the impact of the nonsmoothness of the loss surface . Specifically , we find that the local non-smoothness of loss surface harms the transferability of generated adversarial examples . • Inspired by previous investigations , we propose the smoothed gradient attack to improve the adversarial transferability , which employs the locally averaged gradient instead of the original one to generate adversarial examples . Extensive numerical validations justify the effectiveness of our method . 2 RELATED WORK . The phenomenon of adversarial transferability was first studied by Szegedy et al . ( 2013 ) . By utilizing the transferability , Papernot et al . ( 2016b ; a ) proposed a practical black-box attack by training a substitute model with limited queried information . Liu et al . ( 2017 ) first studied the targeted transferability and introduced the ensemble-based attacks to improve the transferability . More recently , Dong et al . ( 2018 ) showed that the momentum can help to boost transferability significantly . Meanwhile , there exist several works trying to explain the adversarial transferability . Papernot et al . ( 2016a ) attributed the transferability to the similarity between input gradients of source and target models . Liu et al . ( 2017 ) proposed to use the visualization technique to see the large-scale similarity of decision boundaries . However , our numerical investigations imply that these similarity-based explanations have their intrinsic limitation that they can not explain the non-symmetric property of adversarial transferability . Our smoothed gradient attacks that enhance the transferability by utilizing the information in the small neighborhood of the clean example is inspired by the works on shattered gradients ( Balduzzi et al. , 2017 ) and model interpretability ( Smilkov et al. , 2017 ) . Recently , similar strategies are also explored by Athalye et al . ( 2018a ) and Warren He ( 2018 ) for white-box attacks , whereas we focus on the black-box settings . Our method is also related to the work by Athalye et al . ( 2018b ) , which introduced the expectation over transformation ( EOT ) method to increase robustness of adversarial examples . The EOT formulation is similar to our objective ( 7 ) , but they did not study the transferability . Also our motivations are totally different from theirs . 3 BACKGROUND . 3.1 ADVERSARIAL ATTACK . Let f ( x ) : Rd 7→ RK denote the classifier . In deep learning , it is found that for almost any sample x and its label ytrue , there exists a small perturbation η that is nearly imperceptible to human such that argmax i fi ( x ) = y true , argmax i fi ( x+ η ) 6= ytrue . ( 1 ) We call η the adversarial perturbation and correspondingly xadv : = x+ η the adversarial example . The attack ( 1 ) is called a non-targeted attack since the adversary has no control over which class the input x will be misclassified to . In contrast , a targeted attack aims at fooling the model to produce a wrong label specified by the adversary , i.e . argmaxi fi ( x+ η ) = y target . In this paper , we consider the pure black-box attacks . This means the adversary has no knowledge of the target model ( e.g . architecture and parameters ) and is also not allowed to query any input-output pair from the target model . On the contrary , the white-box attack means that the target model is available to the adversary . 3.2 GENERATING ADVERSARIAL EXAMPLES . Modeling In general , crafting adversarial examples can be modeled as the following optimization problem , maximizex′ J ( x′ ) : = J ( f ( x′ ) , ytrue ) s.t . ‖x′ − x‖∞ ≤ ε , ( 2 ) where J is some loss function measuring the discrepancy between the model prediction and ground truth ; the ` ∞ norm ‖ · ‖∞ is used to quantify the magnitude of the perturbation . Other norm is also possible , but we focus on ` ∞ norm in this paper . To improve the strength of adversarial transferability , instead of using a single model , Liu et al . ( 2017 ) proposed the ensemble attack , which generates adversarial examples against a model ensemble f1 ( x ) , f2 ( x ) , · · · , fM ( x ) : maximizex′ J ( 1 M M∑ k=1 fk ( x′ ) , ytrue ) s.t . ‖x′ − x‖∞ ≤ ε . ( 3 ) Optimizer We use the following iteration to solve ( 2 ) and ( 3 ) , xt+1 = projD ( xt + α sign ( ∇xJ ( xt ) ) ) . ( 4 ) where D = [ 0 , 255 ] d ∩ { x′ | ‖x′ − x‖ ≤ ε } and α is the step size . We call the attack that evolves ( 4 ) for T steps iterative gradient sign method ( IGSM ) attack ( Kurakin et al. , 2017 ; Madry et al. , 2018 ) . Furthermore , the famous fast gradient sign method ( FGSM ) is a special case with α = ε , T = 1 . Dong et al . ( 2018 ) recently proposed the momentum-based attack as follows gt+1 = µ gt +∇J ( xt ) /‖∇J ( xt ) ‖1 xt+1 = projD ( xt + α sign ( gt ) ) , ( 5 ) where µ is the decay factor of momentum . By using this method , they won the first-place in NIPS 2017 Non-targeted Adversarial Attack and Targeted Adversarial Attack competitions . We will call it mIGSM attack in this paper . 4 EVALUATION OF ADVERSARIAL TRANSFERABILITY . Datasets To evaluate the transferability , three datasets including MNIST , CIFAR-10 and ImageNet are considered . For ImageNet , evaluations on the whole ILSVRC2012 validation dataset are too timeconsuming . Therefore , in each experiment we randomly select 5 , 000 images that can be correctly recognized by all the examined models to form our new validation set . Models ( i ) For MNIST , we trained fully connected networks ( FNN ) ofD hidden layers , with the width of each layer being 500 . ( ii ) For CIFAR-10 , we trained five models : lenet , resnet20 , resnet44 , resnet56 , densenet . The test accuracies of them are 76.9 % , 92.4 % , 93.7 % , 93.8 % and 94.2 % , respectively . ( iii ) For ImageNet , the pre-trained models provided by PyTorch are used . The Top-1 and Top-5 accuracies of them can be found on website1 . To increase the reliability of experiments , all the models have been tested . However , for a specific experiment we only choose some of them to present since the findings are consistent among all the tested models . Criteria Given a set of adversarial pairs , { ( xadvi , ytruei ) } Ni=1 , we calculate the Top-1 success rate ( % ) fooling a given model f ( x ) by 100N ∑N k=1 1 [ argmaxi fi ( x adv k ) 6= ytruek ] . For targeted attacks , each image xadv is associated with a pre-specified label ytarget 6= ytrue . Then we evaluate the performance of the targeted attack by the following Top-1 success rate : 100N ∑N k=1 1 [ argmaxi fi ( x adv k ) = y target k ] . The corresponding Top-5 rates can be computed in a similar way . 5 HOW MODEL-SPECIFIC FACTORS AFFECT TRANSFERABILITY . Previous studies on adversarial transferability mostly focused on the influence of attack methods ( Liu et al. , 2017 ; Dong et al. , 2018 ; Tramèr et al. , 2017 ; Kurakin et al. , 2017 ) . However it is not clear how the choice of source model affects the success rate transferring to target models . In this section , we investigate this issue from three aspects including architecture , test accuracy and model capacity . 5.1 ARCHITECTURE . We first explore how the architecture similarity between source and target model contributes to the transferability . This study is crucial since it can provide us guidance to choose the appropriate source models for effective attacks . To this end , three popular architectures including ResNet , DenseNet and VGGNet are considered , and for each architecture , two networks are selected . Both one-step and multi-step attacks are performed on ImageNet dataset . Table 1 presents the experiment results , and the choice of hyper-parameters is detailed in the caption . 1http : //pytorch.org/docs/master/torchvision/models.html We can see that the transfers between two models are non-symmetric , and this phenomenon is more obvious for the models with different architectures . For instance , the success rates of IGSM attacks from densenet121 to vgg13 bn is 58.7 % , however the rate from vgg13 bn to densenet121 has only 15.9 % . The lack of symmetry implies that previous similarity-based explanations of adversarial transferability are quite limited . Another interesting observation is that success rates between models with similar architectures are always much higher . For example the success rates of IGSM attacks between vgg13 bn and vgg16 bn are higher than 90 % , nearly twice the ones of attacks from any other architectures . 5.2 MODEL CAPACITY AND TEST ACCURACY . We first study this problem on ImageNet dataset . A variety of models are used as source models to perform both FGSM and IGSM attacks against vgg19 bn , and the results are displayed in Figure 1 . The horizontal axis is the Top-1 test error , while the vertical axis is the number of model parameters that roughly quantifies the model capacity . We can see that the models with powerful attack capability concentrate in the bottom left corner , whereas for those models with either large test error or large number of parameters , the fooling rates are much lower . We also tried other models with results shown in Appendix C , and the results show no difference . We suspect that the impact of test accuracy is due to that the decision boundaries of high-accuracy models are similar , since they all approximate the ground-truth decision boundary of true data very well . On the contrary , the model with low accuracy has a decision boundary relatively different from the high-accuracy models . Here the targeted network vgg19 bn is a high-accuracy model . Therefore it is not surprising to observe that high-accuracy models tend to exhibit stronger attack capability . This phenomenon implies a kind of data-dependent transferability , which is different from the architecture-specific transfers observed in the previous section . It is somewhat strange that adversarial examples generated from deeper model appear less transferable . To further confirm this observation , we conduct additional experiments on MNIST and CIFAR-10 . Table 2 shows the results , which is basically consistent . This observation suggests us not to use deep models as the source models when performing transfer-based attacks , although we have not fully understand this phenomenon .

Summary: The paper explores how the architecture, smoothness of the decision boundary and test accuracy of a model impacts the transferability of examples produced from it.  The paper provides a couple of novel insights, such as the asymmetry when transferring adversarial examples from one model to another. In addition, a novel method is proposed to enhance the transferability of adversarial examples from any model, through using smoothed gradients.