Rethinking Adversarial Transferability from a Data Distribution Perspective

1 INTRODUCTION . Deep neural networks ( DNNs ) are widely used in various safety-critical fields , but they are vulnerable to adversarial examples ( Szegedy et al. , 2013 ) . Adversarial attacks are imperceptible to humans but catastrophic for the DNNs and can be transferred between different models ( Goodfellow et al. , 2015 ; Liu et al. , 2017 ) . Adversarial transferability enables attackers to generate adversarial examples from the source model to attack unknown target models , which has raised security concerns about the deployment of DNNs in practice . Understanding the essence of adversarial transferability is a fundamental problem in deep learning . On the one hand , some works show that the characteristics of the source model , such as model architecture ( Wu et al. , 2019 ) , model capacity ( Tramèr et al. , 2017 ) , and test accuracy ( Wu & Zhu , 2020 ) , influence adversarial examples ’ transferability . On the other hand , some works think that the data-relevant information may be the key factor for adversarial transferability . Ilyas et al . ( 2019 ) explain that adversarial perturbations are non-robust features and not meaningless bugs , but it is hard to specifically define non-robust features . We want to further study transferability quantitatively from the data distribution perspective . It has been empirically observed that DNNs are relatively robust to random noise ( Fawzi et al. , 2016 ) . However , in this work we find an intriguing phenomenon : some samples are sensitive to Gaussian noise , in the sense that injecting small Gaussian noise into these samples can fool different models trained on the same dataset . Furthermore , their adversarial counterparts generated by different source models have much stronger transferability against different target models than other samples . We hypothesize that these samples are in the low-density regions of the ground truth distribution both source and target models are trained on , and models are not well trained in these regions . Thus predictions of these samples are easy to be perturbed and even not robust to small random noises . We denote this kind of data as Low-Density Data ( LDD ) , while others as High-Density Data ( HDD ) . As shown in Fig . 1 ( Left ) , the attack success rate against different target models of LDD with different strengths of Gaussian noise is much higher than that of HDD . Furthermore , in Fig . 1 ( Right ) , the adversarial counterparts of LDD have much stronger transferability than the adversarial counterparts of HDD ( see Appendix B for details ) . This phenomenon reveals that the location of data plays a vital role in adversarial transferability and the adversarial examples of samples in the low-density region are strongly transferable . The most efficient direction towards the low-density region is −∇x log pD ( x , y ) , where pD ( x , y ) is the ground truth density of natural data . We name this direction Intrinsic Attack because it doesn ’ t depend on the models and only depends on the ground truth distribution . Thus , we propose to match the adversarial attack with intrinsic attack for generating strong transferable adversarial examples . We explore the potential of a classifier pθ , Λ ( y|x ) with parameters θ and structure hyper-parameters Λ ( see Sec . 3.1 ) to generate more transferable adversarial examples by aligning adversarial attack with intrinsic attack −∇x log pD ( x , y ) . The adversarial attack of pθ , Λ ( y|x ) is usually generated by PGD/FGSM method , and is determined by −∇x log pθ , Λ ( y|x ) . We match the Alignment between the Adversarial attack and Intrinsic attack ( AAI ) , EpD ( x , y ) [ ∇x log pθ , Λ ( y|x ) ‖∇x log pθ , Λ ( y|x ) ‖2 · ∇x log pD ( x , y ) ] , by modifying the structure parameters Λ for a pre-trained network . In order to maximize AAI , we should make pθ , Λ ( y|x ) smoother . Otherwise , ∇x log pθ , Λ ( y|x ) will oscillate frequently and hard to match∇x log pD ( x , y ) . For the commonly used ReLU network , we can smooth it by replacing ReLU activation with Softplus ( Nair & Hinton , 2010 ) with little change of the model ’ s output . Maennel et al . ( 2020 ) show that the early layers of a network learn the local statistics of the data distribution better than the later layers , which motivates us to decrease the impact of later layers when generating adversarial examples to utilize the data distribution-relevant information . We can closely match the adversarial attack with the intrinsic attack−∇x log pD ( x , y ) and improve the adversarial transferability by optimizing structure hyper-parameters Λ to maximize AAI as the objective function . We name our method as Intrinsic Adversarial Attack ( IAA ) . There are some interesting observations in our IAA experiments . Firstly , we find that the test accuracy of the source model may not be important . As shown in Fig 2 , the accuracy of the pre-trained model with Softplusβ=15 is around 60 % , but the adversarial transferability of this model is much stronger than the model with Softplusβ=45 . Secondly , although the existing methods ( Madry et al. , 2018 ; Wu et al. , 2019 ) can significantly decrease the top-1 accuracy of the target models , the top-5 accuracy is still high . IAA can both decrease the top-1 accuracy and top-5 accuracy . Furthermore , the existing methods ( Xie et al. , 2019 ; Wu et al. , 2019 ) can just slightly improve the one-step attack under different strengths , while our IAA surpasses the existing methods by a large margin . These phenomena verify our hypothesis that IAA pulls examples to the low-density region , which causes prediction difficulty to the target models . Our main contributions are summarized below : • We propose an effective metric , AAI , to evaluate the alignment of the model ’ s adversarial attack with intrinsic attack −∇x log pD ( x , y ) . Furthermore , we show that AAI is also an effective metric for adversarial transferability . • We propose the Intrinsic Adversarial Attack ( IAA ) by maximizing AAI to generate more transferable adversarial examples . • We conduct comprehensive transfer attack experiments from different source models against nine naturally trained models and three ensemble secured models , showing that IAA can significantly improve the state-of-the-art transferability ( both targeted and untargeted attack ) of adversarial examples ( even improve 20 % under some settings ) . 1The AAI metric for LDD ( 0.1264 ) is much larger than HDD ( 0.0457 ) , which shows that the direction of PGD attack on LDD aligns better than that on HDD ( The AAI metric on all test samples is 0.052 ) . 2 RELATED WORK . There are two types of adversarial attacks : white-box attacks and black-box attacks . White-box attacks assume that the attacker can completely access the structure and parameters of the target model . Typical examples of white-box attacks are FGSM ( Goodfellow et al. , 2015 ) , PGD ( Madry et al. , 2018 ) , and CW ( Carlini & Wagner , 2017 ) . The black-box attack assumes that the attacker only knows the output of the target model ( prediction or confidence ) . Black-box attacks are roughly divided into two types : estimating gradient with queries to the target model ( Papernot et al. , 2017 ; Su et al. , 2019 ; Yang et al. , 2020 ) and attacking a surrogate model ( Xie et al. , 2019 ; Dong et al. , 2018 ; Wu et al. , 2019 ) . Attacking a surrogate model is much more efficient and can reduce the risk of exposure . Thus , many existing works focused on adversarial transferability . Su et al . ( 2018 ) explore the factors influencing the transferability and show the architecture has greater influence than model capacity . Dong et al . ( 2018 ) show that the momentum of gradients can be used to improve the adversarial transferability . Xie et al . ( 2019 ) show the diversity of input data will enhance the adversarial transferability . Huang et al . ( 2019 ) fine-tune the adversarial examples by increasing perturbation on a pre-specified layer . Wang et al . ( 2020 ) propose a loss to decrease interactions between perturbation units during attacking . Wu et al . ( 2019 ) propose that reducing gradients from the residual modules is effective for improving transferability . Guo et al . ( 2020 ) removes ReLU activations in the later layers to get linear backpropagation and decreases the influence of intermediate layers . They only modify the backpropagation when generating adversarial examples while keeping the forward prediction as the original model . Based on this , Zhang et al . ( 2021 ) conjecture that backpropagating smoothly might be sufficient for improving transferability . There are also some works on adversarial attack and defense using generative models . Naseer et al . ( 2019 ) and Yang et al . ( 2021a ) learn adversarial perturbation through a conditional generative attacking model , which needs to be carefully designed for certain classes . Samangouei et al . ( 2018 ) ; Song et al . ( 2018 ) use GANs or autoregressive models to detect and purify adversarial examples . Du & Mordatch ( 2019 ) ; Hill et al . ( 2021 ) ; Srinivasan et al . ( 2021 ) ; Yoon et al . ( 2021 ) purify adversarial examples by EBM or score-based generative models . JEM ( Grathwohl et al. , 2020 ) shows that combining a classifier with EBM can help to obtain some robustness . However , adversarial attacks or purification based on generative models are computationally costly . We want to modify a normal classifier with little computation cost to enhance its adversarial transferability by maximizing our AAI metric . 3 METHODS . 3.1 ALIGNMENT BETWEEN THE ADVERSARIAL ATTACK AND INTRINSIC ATTACK For a classifier fθ , Λ parameterized by θ with structure hyper-parameters Λ ( e.g. , hyper-parameters for architecture , activation function , etc . ) , and data x , label y , total possible classes n , then fθ , Λ ( x ) [ k ] represents the kth output of the last layer . The conditional density pθ , Λ ( y|x ) can be expressed as : pθ , Λ ( y|x ) = exp ( fθ , Λ ( x ) [ y ] ) ∑n k=1 exp ( fθ , Λ ( x ) [ k ] ) . ( 1 ) The adversarial attack is usually based on−∇x log pθ , Λ ( y|x ) ( Madry et al. , 2018 ; Goodfellow et al. , 2015 ) . The most effective direction towards low-density region is intrinsic attack−∇x log pD ( x , y ) . To improve the adversarial transferability , we need to match model ’ s adversarial attack direction with intrinsic attack . We define the inner product of normalized −∇x log pθ , Λ ( y|x ) and −∇x log pD ( x , y ) to quantify the matching of the direction of adversarial attack and the intrinsic attack . Definition 1 ( AAI ) . For a classifier pθ , Λ ( y|x ) , the Alignment between its Adversarial attack and the Intrinsic attack is : AAI , EpD ( x , y ) [ ∇x log pθ , Λ ( y|x ) ‖∇x log pθ , Λ ( y|x ) ‖2 · ∇x log pD ( x , y ) ] , ( 2 ) where pD ( x , y ) is the ground truth joint distribution . Remark . ( 1 ) We use the normalized adversarial attack to remove the influence of scaling factor when comparing different models . ( 2 ) This definition is equivalent with a modified score matching objective as : 1 2 EpD ( x , y ) ∥∥∥∥ ∇x log pθ , Λ ( y|x ) ‖∇x log pθ , Λ ( y|x ) ‖2 −∇x log pD ( x , y ) ∥∥∥∥2 2 = −AAI + CpD , where CpD is a constant only depend on the ground truth distribution pD . As getting the gradient of pD ( x , y ) is not feasible , we use integration by parts ( Hyvärinen & Dayan , 2005 ) to move the gradient on pD ( x , y ) to model ’ s adversarial attack . With the smoothness assumption on ∇xpθ , Λ ( y|x ) , we have the following theorem : Theorem 1 . If ∇xpθ , Λ ( y|x ) is differentiable almost everywhere , then AAI = −EpD ( x , y ) [ ∇x · ∇xpθ , Λ ( y|x ) ‖∇xpθ , Λ ( y|x ) ‖2 ] . ( 3 ) Moreover , AAI = −EpD ( x , y ) Ep ( v ) [ vT∇x vT∇xpθ , Λ ( y|x ) ‖∇xpθ , Λ ( y|x ) ‖2 ] , ( 4 ) where p ( v ) is a distribution of random vector v such that Ep ( v ) [ vvT ] = I ( e.g. , the multivariate standard normal N ( 0 , I ) ) . This theorem makes it possible to calculate AAI without knowing the gradient of ground truth distribution when the model is smooth , see Appendix E for the proof . Combined with sliced score matching ( Song et al. , 2020 ) we can efficiently approximate AAI on discrete samples . However , for ReLU networks , the model ’ s second derivative is not well defined , which prevents us from using this theorem . Thus , we need to smooth the ReLU models for better modifying the model to maximize AAI and improve the transferability . One obvious way to improve the smoothness is to replace ReLU activation with some smooth activation . In this paper , we use Softplus to show that smoothness can greatly help us to improve transferability .

Summary: This paper presents a new adversarial attack method which could generate adversarial examples with higher transferability. The proposed method is based on the observation that low-density region of the training data is not well trained. To utilize this, the authors try to align the adversarial direction with the direction to decrease the ground truth density. The proposed method is theoretically support. Their experiments show that the proposed method outperforms multiple adversarial attack method in almost all evaluated scenario. I believe this could be a useful attack method for generating transferable adversarial examples, and providing a strong counterpart for future research on adversarial defense.