Rethinking Adversarial Transferability from a Data Distribution Perspective

1 INTRODUCTION . Deep neural networks ( DNNs ) are widely used in various safety-critical fields , but they are vulnerable to adversarial examples ( Szegedy et al. , 2013 ) . Adversarial attacks are imperceptible to humans but catastrophic for the DNNs and can be transferred between different models ( Goodfellow et al. , 2015 ; Liu et al. , 2017 ) . Adversarial transferability enables attackers to generate adversarial examples from the source model to attack unknown target models , which has raised security concerns about the deployment of DNNs in practice . Understanding the essence of adversarial transferability is a fundamental problem in deep learning . On the one hand , some works show that the characteristics of the source model , such as model architecture ( Wu et al. , 2019 ) , model capacity ( Tramèr et al. , 2017 ) , and test accuracy ( Wu & Zhu , 2020 ) , influence adversarial examples ’ transferability . On the other hand , some works think that the data-relevant information may be the key factor for adversarial transferability . Ilyas et al . ( 2019 ) explain that adversarial perturbations are non-robust features and not meaningless bugs , but it is hard to specifically define non-robust features . We want to further study transferability quantitatively from the data distribution perspective . It has been empirically observed that DNNs are relatively robust to random noise ( Fawzi et al. , 2016 ) . However , in this work we find an intriguing phenomenon : some samples are sensitive to Gaussian noise , in the sense that injecting small Gaussian noise into these samples can fool different models trained on the same dataset . Furthermore , their adversarial counterparts generated by different source models have much stronger transferability against different target models than other samples . We hypothesize that these samples are in the low-density regions of the ground truth distribution both source and target models are trained on , and models are not well trained in these regions . Thus predictions of these samples are easy to be perturbed and even not robust to small random noises . We denote this kind of data as Low-Density Data ( LDD ) , while others as High-Density Data ( HDD ) . As shown in Fig . 1 ( Left ) , the attack success rate against different target models of LDD with different strengths of Gaussian noise is much higher than that of HDD . Furthermore , in Fig . 1 ( Right ) , the adversarial counterparts of LDD have much stronger transferability than the adversarial counterparts of HDD ( see Appendix B for details ) . This phenomenon reveals that the location of data plays a vital role in adversarial transferability and the adversarial examples of samples in the low-density region are strongly transferable . The most efficient direction towards the low-density region is −∇x log pD ( x , y ) , where pD ( x , y ) is the ground truth density of natural data . We name this direction Intrinsic Attack because it doesn ’ t depend on the models and only depends on the ground truth distribution . Thus , we propose to match the adversarial attack with intrinsic attack for generating strong transferable adversarial examples . We explore the potential of a classifier pθ , Λ ( y|x ) with parameters θ and structure hyper-parameters Λ ( see Sec . 3.1 ) to generate more transferable adversarial examples by aligning adversarial attack with intrinsic attack −∇x log pD ( x , y ) . The adversarial attack of pθ , Λ ( y|x ) is usually generated by PGD/FGSM method , and is determined by −∇x log pθ , Λ ( y|x ) . We match the Alignment between the Adversarial attack and Intrinsic attack ( AAI ) , EpD ( x , y ) [ ∇x log pθ , Λ ( y|x ) ‖∇x log pθ , Λ ( y|x ) ‖2 · ∇x log pD ( x , y ) ] , by modifying the structure parameters Λ for a pre-trained network . In order to maximize AAI , we should make pθ , Λ ( y|x ) smoother . Otherwise , ∇x log pθ , Λ ( y|x ) will oscillate frequently and hard to match∇x log pD ( x , y ) . For the commonly used ReLU network , we can smooth it by replacing ReLU activation with Softplus ( Nair & Hinton , 2010 ) with little change of the model ’ s output . Maennel et al . ( 2020 ) show that the early layers of a network learn the local statistics of the data distribution better than the later layers , which motivates us to decrease the impact of later layers when generating adversarial examples to utilize the data distribution-relevant information . We can closely match the adversarial attack with the intrinsic attack−∇x log pD ( x , y ) and improve the adversarial transferability by optimizing structure hyper-parameters Λ to maximize AAI as the objective function . We name our method as Intrinsic Adversarial Attack ( IAA ) . There are some interesting observations in our IAA experiments . Firstly , we find that the test accuracy of the source model may not be important . As shown in Fig 2 , the accuracy of the pre-trained model with Softplusβ=15 is around 60 % , but the adversarial transferability of this model is much stronger than the model with Softplusβ=45 . Secondly , although the existing methods ( Madry et al. , 2018 ; Wu et al. , 2019 ) can significantly decrease the top-1 accuracy of the target models , the top-5 accuracy is still high . IAA can both decrease the top-1 accuracy and top-5 accuracy . Furthermore , the existing methods ( Xie et al. , 2019 ; Wu et al. , 2019 ) can just slightly improve the one-step attack under different strengths , while our IAA surpasses the existing methods by a large margin . These phenomena verify our hypothesis that IAA pulls examples to the low-density region , which causes prediction difficulty to the target models . Our main contributions are summarized below : • We propose an effective metric , AAI , to evaluate the alignment of the model ’ s adversarial attack with intrinsic attack −∇x log pD ( x , y ) . Furthermore , we show that AAI is also an effective metric for adversarial transferability . • We propose the Intrinsic Adversarial Attack ( IAA ) by maximizing AAI to generate more transferable adversarial examples . • We conduct comprehensive transfer attack experiments from different source models against nine naturally trained models and three ensemble secured models , showing that IAA can significantly improve the state-of-the-art transferability ( both targeted and untargeted attack ) of adversarial examples ( even improve 20 % under some settings ) . 1The AAI metric for LDD ( 0.1264 ) is much larger than HDD ( 0.0457 ) , which shows that the direction of PGD attack on LDD aligns better than that on HDD ( The AAI metric on all test samples is 0.052 ) . 2 RELATED WORK . There are two types of adversarial attacks : white-box attacks and black-box attacks . White-box attacks assume that the attacker can completely access the structure and parameters of the target model . Typical examples of white-box attacks are FGSM ( Goodfellow et al. , 2015 ) , PGD ( Madry et al. , 2018 ) , and CW ( Carlini & Wagner , 2017 ) . The black-box attack assumes that the attacker only knows the output of the target model ( prediction or confidence ) . Black-box attacks are roughly divided into two types : estimating gradient with queries to the target model ( Papernot et al. , 2017 ; Su et al. , 2019 ; Yang et al. , 2020 ) and attacking a surrogate model ( Xie et al. , 2019 ; Dong et al. , 2018 ; Wu et al. , 2019 ) . Attacking a surrogate model is much more efficient and can reduce the risk of exposure . Thus , many existing works focused on adversarial transferability . Su et al . ( 2018 ) explore the factors influencing the transferability and show the architecture has greater influence than model capacity . Dong et al . ( 2018 ) show that the momentum of gradients can be used to improve the adversarial transferability . Xie et al . ( 2019 ) show the diversity of input data will enhance the adversarial transferability . Huang et al . ( 2019 ) fine-tune the adversarial examples by increasing perturbation on a pre-specified layer . Wang et al . ( 2020 ) propose a loss to decrease interactions between perturbation units during attacking . Wu et al . ( 2019 ) propose that reducing gradients from the residual modules is effective for improving transferability . Guo et al . ( 2020 ) removes ReLU activations in the later layers to get linear backpropagation and decreases the influence of intermediate layers . They only modify the backpropagation when generating adversarial examples while keeping the forward prediction as the original model . Based on this , Zhang et al . ( 2021 ) conjecture that backpropagating smoothly might be sufficient for improving transferability . There are also some works on adversarial attack and defense using generative models . Naseer et al . ( 2019 ) and Yang et al . ( 2021a ) learn adversarial perturbation through a conditional generative attacking model , which needs to be carefully designed for certain classes . Samangouei et al . ( 2018 ) ; Song et al . ( 2018 ) use GANs or autoregressive models to detect and purify adversarial examples . Du & Mordatch ( 2019 ) ; Hill et al . ( 2021 ) ; Srinivasan et al . ( 2021 ) ; Yoon et al . ( 2021 ) purify adversarial examples by EBM or score-based generative models . JEM ( Grathwohl et al. , 2020 ) shows that combining a classifier with EBM can help to obtain some robustness . However , adversarial attacks or purification based on generative models are computationally costly . We want to modify a normal classifier with little computation cost to enhance its adversarial transferability by maximizing our AAI metric . 3 METHODS . 3.1 ALIGNMENT BETWEEN THE ADVERSARIAL ATTACK AND INTRINSIC ATTACK For a classifier fθ , Λ parameterized by θ with structure hyper-parameters Λ ( e.g. , hyper-parameters for architecture , activation function , etc . ) , and data x , label y , total possible classes n , then fθ , Λ ( x ) [ k ] represents the kth output of the last layer . The conditional density pθ , Λ ( y|x ) can be expressed as : pθ , Λ ( y|x ) = exp ( fθ , Λ ( x ) [ y ] ) ∑n k=1 exp ( fθ , Λ ( x ) [ k ] ) . ( 1 ) The adversarial attack is usually based on−∇x log pθ , Λ ( y|x ) ( Madry et al. , 2018 ; Goodfellow et al. , 2015 ) . The most effective direction towards low-density region is intrinsic attack−∇x log pD ( x , y ) . To improve the adversarial transferability , we need to match model ’ s adversarial attack direction with intrinsic attack . We define the inner product of normalized −∇x log pθ , Λ ( y|x ) and −∇x log pD ( x , y ) to quantify the matching of the direction of adversarial attack and the intrinsic attack . Definition 1 ( AAI ) . For a classifier pθ , Λ ( y|x ) , the Alignment between its Adversarial attack and the Intrinsic attack is : AAI , EpD ( x , y ) [ ∇x log pθ , Λ ( y|x ) ‖∇x log pθ , Λ ( y|x ) ‖2 · ∇x log pD ( x , y ) ] , ( 2 ) where pD ( x , y ) is the ground truth joint distribution . Remark . ( 1 ) We use the normalized adversarial attack to remove the influence of scaling factor when comparing different models . ( 2 ) This definition is equivalent with a modified score matching objective as : 1 2 EpD ( x , y ) ∥∥∥∥ ∇x log pθ , Λ ( y|x ) ‖∇x log pθ , Λ ( y|x ) ‖2 −∇x log pD ( x , y ) ∥∥∥∥2 2 = −AAI + CpD , where CpD is a constant only depend on the ground truth distribution pD . As getting the gradient of pD ( x , y ) is not feasible , we use integration by parts ( Hyvärinen & Dayan , 2005 ) to move the gradient on pD ( x , y ) to model ’ s adversarial attack . With the smoothness assumption on ∇xpθ , Λ ( y|x ) , we have the following theorem : Theorem 1 . If ∇xpθ , Λ ( y|x ) is differentiable almost everywhere , then AAI = −EpD ( x , y ) [ ∇x · ∇xpθ , Λ ( y|x ) ‖∇xpθ , Λ ( y|x ) ‖2 ] . ( 3 ) Moreover , AAI = −EpD ( x , y ) Ep ( v ) [ vT∇x vT∇xpθ , Λ ( y|x ) ‖∇xpθ , Λ ( y|x ) ‖2 ] , ( 4 ) where p ( v ) is a distribution of random vector v such that Ep ( v ) [ vvT ] = I ( e.g. , the multivariate standard normal N ( 0 , I ) ) . This theorem makes it possible to calculate AAI without knowing the gradient of ground truth distribution when the model is smooth , see Appendix E for the proof . Combined with sliced score matching ( Song et al. , 2020 ) we can efficiently approximate AAI on discrete samples . However , for ReLU networks , the model ’ s second derivative is not well defined , which prevents us from using this theorem . Thus , we need to smooth the ReLU models for better modifying the model to maximize AAI and improve the transferability . One obvious way to improve the smoothness is to replace ReLU activation with some smooth activation . In this paper , we use Softplus to show that smoothness can greatly help us to improve transferability .

Summary: This paper proposes Intrinsic Adversarial Attack (IAA), a transfer attack method based by jointly matching data distribution. The key assumption of this paper is that the DNN might not be well trained on low-density regions (LDD). Therefore, taking data distribution into consideration during attack could potentially improve the transferability. Empirical evaluation on different model architectures, normal models, robustly trained models, and ensemble-based attack context, demonstrates the advantage of IAA.