RETHINKING ADVERSARIAL TRANSFERABILITY
Year: 2021

Abstract:
Adversarial transferability enables attackers to generate adversarial examples from the source model to attack the target model, which has raised security concerns about the deployment of DNNs in practice. In this paper, we rethink adversarial transferability from a data distribution perspective and further enhance transferability by score matching based optimization. We identify that some samples with injecting small Gaussian noise can fool different target models, and their adversarial examples under different source models have much stronger transferability. We hypothesize that these samples are in the low-density region of the ground truth distribution where models are not well trained. To improve the attack success rate of adversarial examples, we match the adversarial attacks with the directions which effectively decrease the ground truth density. We propose Intrinsic Adversarial Attack (IAA), which smooths the activation function and decreases the impact of the later layers of a given normal model, to increase the alignment of adversarial attack and the gradient of joint data distribution. We conduct comprehensive transferable attacks against multiple DNNs and show that our IAA can boost the transferability of the crafted attacks in all cases and go beyond state-of-the-art methods.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) are widely used in various safety-critical fields, but they are vulnerable to adversarial examples (Szegedy et al., 2013). Adversarial attacks are imperceptible to humans but catastrophic for the DNNs and can be transferred between different models (Goodfellow et al., 2015; Liu et al., 2017). Adversarial transferability enables attackers to generate adversarial examples from the source model to attack unknown target models, which has raised security concerns about the deployment of DNNs in practice. Understanding the essence of adversarial transferability is a fundamental problem in deep learning. On the one hand, some works show that the characteristics of the source model, such as model architecture (Wu et al., 2019), model capacity (Tramèr et al., 2017), and test accuracy (Wu & Zhu, 2020), influence adversarial examples’ transferability. On the other hand, some works think that the data-relevant information may be the key factor for adversarial transferability. Ilyas et al. (2019) explain that adversarial perturbations are non-robust features and not meaningless bugs, but it is hard to specifically define non-robust features. We want to further study transferability quantitatively from the data distribution perspective.
It has been empirically observed that DNNs are relatively robust to random noise (Fawzi et al., 2016). However, in this work we find an intriguing phenomenon: some samples are sensitive to Gaussian noise, in the sense that injecting small Gaussian noise into these samples can fool different models trained on the same dataset. Furthermore, their adversarial counterparts generated by different source models have much stronger transferability against different target models than other samples. We hypothesize that these samples are in the low-density regions of the ground truth distribution both source and target models are trained on, and models are not well trained in these regions. Thus predictions of these samples are easy to be perturbed and even not robust to small random noises. We denote this kind of data as Low-Density Data (LDD), while others as High-Density Data (HDD). As shown in Fig. 1 (Left), the attack success rate against different target models of LDD with different strengths of Gaussian noise is much higher than that of HDD. Furthermore, in Fig. 1 (Right), the adversarial counterparts of LDD have much stronger transferability than the adversarial counterparts of HDD (see Appendix B for details).
This phenomenon reveals that the location of data plays a vital role in adversarial transferability and the adversarial examples of samples in the low-density region are strongly transferable. The most efficient direction towards the low-density region is −∇x log pD(x, y), where pD(x, y) is the ground truth density of natural data. We name this direction Intrinsic Attack because it doesn’t depend on the models and only depends on the ground truth distribution. Thus, we propose to match the adversarial attack with intrinsic attack for generating strong transferable adversarial examples.
We explore the potential of a classifier pθ,Λ(y|x) with parameters θ and structure hyper-parameters Λ (see Sec. 3.1) to generate more transferable adversarial examples by aligning adversarial attack with intrinsic attack −∇x log pD(x, y). The adversarial attack of pθ,Λ(y|x) is usually generated by PGD/FGSM method, and is determined by −∇x log pθ,Λ(y|x). We match the Alignment between the Adversarial attack and Intrinsic attack (AAI), EpD(x,y) [ ∇x log pθ,Λ(y|x) ‖∇x log pθ,Λ(y|x)‖2 · ∇x log pD(x, y) ] , by modifying the structure parameters Λ for a pre-trained network.
In order to maximize AAI, we should make pθ,Λ(y|x) smoother. Otherwise, ∇x log pθ,Λ(y|x) will oscillate frequently and hard to match∇x log pD(x, y). For the commonly used ReLU network, we can smooth it by replacing ReLU activation with Softplus (Nair & Hinton, 2010) with little change of the model’s output. Maennel et al. (2020) show that the early layers of a network learn the local statistics of the data distribution better than the later layers, which motivates us to decrease the impact of later layers when generating adversarial examples to utilize the data distribution-relevant information. We can closely match the adversarial attack with the intrinsic attack−∇x log pD(x, y) and improve the adversarial transferability by optimizing structure hyper-parameters Λ to maximize AAI as the objective function. We name our method as Intrinsic Adversarial Attack (IAA).
There are some interesting observations in our IAA experiments. Firstly, we find that the test accuracy of the source model may not be important. As shown in Fig 2, the accuracy of the pre-trained model with Softplusβ=15 is around 60%, but the adversarial transferability of this model is much stronger than the model with Softplusβ=45. Secondly, although the existing methods (Madry et al., 2018; Wu et al., 2019) can significantly decrease the top-1 accuracy of the target models, the top-5 accuracy is still high. IAA can both decrease the top-1 accuracy and top-5 accuracy. Furthermore, the existing methods (Xie et al., 2019; Wu et al., 2019) can just slightly improve the one-step attack under different strengths, while our IAA surpasses the existing methods by a large margin. These phenomena verify our hypothesis that IAA pulls examples to the low-density region, which causes prediction difficulty to the target models. Our main contributions are summarized below:
• We propose an effective metric, AAI, to evaluate the alignment of the model’s adversarial attack with intrinsic attack −∇x log pD(x, y). Furthermore, we show that AAI is also an effective metric for adversarial transferability.
• We propose the Intrinsic Adversarial Attack (IAA) by maximizing AAI to generate more transferable adversarial examples.
• We conduct comprehensive transfer attack experiments from different source models against nine naturally trained models and three ensemble secured models, showing that IAA can significantly improve the state-of-the-art transferability (both targeted and untargeted attack) of adversarial examples (even improve 20% under some settings).
1The AAI metric for LDD (0.1264) is much larger than HDD (0.0457), which shows that the direction of PGD attack on LDD aligns better than that on HDD (The AAI metric on all test samples is 0.052).

heading: 2 RELATED WORK
text: There are two types of adversarial attacks: white-box attacks and black-box attacks. White-box attacks assume that the attacker can completely access the structure and parameters of the target model. Typical examples of white-box attacks are FGSM (Goodfellow et al., 2015), PGD (Madry et al., 2018), and CW (Carlini & Wagner, 2017). The black-box attack assumes that the attacker only knows the output of the target model (prediction or confidence). Black-box attacks are roughly divided into two types: estimating gradient with queries to the target model (Papernot et al., 2017; Su et al., 2019; Yang et al., 2020) and attacking a surrogate model (Xie et al., 2019; Dong et al., 2018; Wu et al., 2019). Attacking a surrogate model is much more efficient and can reduce the risk of exposure. Thus, many existing works focused on adversarial transferability.
Su et al. (2018) explore the factors influencing the transferability and show the architecture has greater influence than model capacity. Dong et al. (2018) show that the momentum of gradients can be used to improve the adversarial transferability. Xie et al. (2019) show the diversity of input data will enhance the adversarial transferability. Huang et al. (2019) fine-tune the adversarial examples by increasing perturbation on a pre-specified layer. Wang et al. (2020) propose a loss to decrease interactions between perturbation units during attacking. Wu et al. (2019) propose that reducing gradients from the residual modules is effective for improving transferability. Guo et al. (2020) removes ReLU activations in the later layers to get linear backpropagation and decreases the influence of intermediate layers. They only modify the backpropagation when generating adversarial examples while keeping the forward prediction as the original model. Based on this, Zhang et al. (2021) conjecture that backpropagating smoothly might be sufficient for improving transferability.
There are also some works on adversarial attack and defense using generative models. Naseer et al. (2019) and Yang et al. (2021a) learn adversarial perturbation through a conditional generative attacking model, which needs to be carefully designed for certain classes. Samangouei et al. (2018); Song et al. (2018) use GANs or autoregressive models to detect and purify adversarial examples. Du & Mordatch (2019); Hill et al. (2021); Srinivasan et al. (2021); Yoon et al. (2021) purify adversarial examples by EBM or score-based generative models. JEM (Grathwohl et al., 2020) shows that combining a classifier with EBM can help to obtain some robustness. However, adversarial attacks or purification based on generative models are computationally costly. We want to modify a normal classifier with little computation cost to enhance its adversarial transferability by maximizing our AAI metric.

heading: 3 METHODS
text: 3.1 ALIGNMENT BETWEEN THE ADVERSARIAL ATTACK AND INTRINSIC ATTACK
For a classifier fθ,Λ parameterized by θ with structure hyper-parameters Λ (e.g., hyper-parameters for architecture, activation function, etc.), and data x, label y, total possible classes n, then fθ,Λ(x)[k] represents the kth output of the last layer. The conditional density pθ,Λ(y|x) can be expressed as:
pθ,Λ(y|x) = exp(fθ,Λ(x)[y])∑n k=1 exp(fθ,Λ(x)[k]) . (1)
The adversarial attack is usually based on−∇x log pθ,Λ(y|x) (Madry et al., 2018; Goodfellow et al., 2015). The most effective direction towards low-density region is intrinsic attack−∇x log pD(x, y). To improve the adversarial transferability, we need to match model’s adversarial attack direction with intrinsic attack. We define the inner product of normalized −∇x log pθ,Λ(y|x) and −∇x log pD(x, y) to quantify the matching of the direction of adversarial attack and the intrinsic attack.
Definition 1 (AAI). For a classifier pθ,Λ(y|x), the Alignment between its Adversarial attack and the Intrinsic attack is:
AAI , EpD(x,y) [ ∇x log pθ,Λ(y|x) ‖∇x log pθ,Λ(y|x)‖2 · ∇x log pD(x, y) ] , (2)
where pD(x, y) is the ground truth joint distribution.
Remark. (1) We use the normalized adversarial attack to remove the influence of scaling factor when comparing different models. (2) This definition is equivalent with a modified score matching objective as:
1 2 EpD(x,y) ∥∥∥∥ ∇x log pθ,Λ(y|x)‖∇x log pθ,Λ(y|x)‖2 −∇x log pD(x, y) ∥∥∥∥2
2
= −AAI + CpD ,
where CpD is a constant only depend on the ground truth distribution pD.
As getting the gradient of pD(x, y) is not feasible, we use integration by parts (Hyvärinen & Dayan, 2005) to move the gradient on pD(x, y) to model’s adversarial attack. With the smoothness assumption on ∇xpθ,Λ(y|x), we have the following theorem: Theorem 1. If ∇xpθ,Λ(y|x) is differentiable almost everywhere, then
AAI = −EpD(x,y) [ ∇x ·
∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2
] . (3)
Moreover,
AAI = −EpD(x,y)Ep(v) [ vT∇x
vT∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2
] , (4)
where p(v) is a distribution of random vector v such that Ep(v)[vvT] = I (e.g., the multivariate standard normal N (0, I)).
This theorem makes it possible to calculate AAI without knowing the gradient of ground truth distribution when the model is smooth, see Appendix E for the proof. Combined with sliced score matching (Song et al., 2020) we can efficiently approximate AAI on discrete samples. However, for ReLU networks, the model’s second derivative is not well defined, which prevents us from using this theorem. Thus, we need to smooth the ReLU models for better modifying the model to maximize AAI and improve the transferability. One obvious way to improve the smoothness is to replace ReLU activation with some smooth activation. In this paper, we use Softplus to show that smoothness can greatly help us to improve transferability.

heading: 3.2 INTRINSIC ADVERSARIAL ATTACK FOR NORMAL MODELS

heading: 3.2.1 SMOOTHING THE CLASSIFIER BY SOFTPLUS
text: Srinivas & Fleuret (2021) claim that the input-gradient of a classifier is somewhat aligned with the gradient of the ground truth data distribution pD(x, y). On the other hand, Dombrowski et al. (2019) show that the information in gradients may be hidden for that the gradient of the ReLU activation function is discontinuous and will frequently change when the input closes to zero. Thus better gradient with less noise can contribute to matching the input-gradients −∇x log pθ,Λ(y|x) and the gradients of ground truth distribution −∇x log pD(x, y). We provide a simple implementation that replacing the widely used activation function ReLU with Softplusβ (Nair & Hinton, 2010):
Softplusβ(x) = 1
β log(1 + exp(βx)), (5)
where β is the shape-related hyper-parameter. The larger β is, the better Softplusβ approximates ReLU. The Softplusβ can also be expressed as the expectation of ReLU in a neighborhood as follows:
Softplusβ(x) = E ∼pβ [ReLU(x− )], (6)
where pβ( ) = β(eβ /2+e−β /2)2 (Dombrowski et al., 2019). See Appendix F for a short proof. Smilkov et al. (2017); Dombrowski et al. (2019) show that the local average of gradient values can make the gradient smoother. This means that the gradient obtained with Softplusβ is less noisy and more meaningful than ReLU. We use Softplusβ as an approximation of the activation function ReLU in the pre-trained model. As shown in Fig. 2 (Left), AAI first increases then decreases when decreasing β in Softplusβ (the smaller the β, the smoother the classifiers) and AAI achieves the largest value at β = 15. Thus there exits a trade-off between AAI and smoothness, and a proper β is needed.
We conducted an experiment with the PGD attack on the randomly selected 5000 ImageNet validation images which are correctly classified to investigate how the shape-related hyper-parameter β affects the adversarial transferability. As shown in Fig. 2 (Right), when the hyper-parameter β is too small, Softplusβ cannot approximate ReLU well, which reduces the effectiveness of adversarial perturbation and prediction accuracy. With a proper selection of β = 15 which is also the maximum of AAI, using Softplusβ as an approximation of ReLU can significantly improve the transferability of adversarial examples.

heading: 3.2.2 UTILIZING DISTRIBUTION-RELEVANT INFORMATION FROM EARLY LAYERS
text: Maennel et al. (2020) and Boopathy & Fiete (2021) show that the early layers can better capture the local statistics of the inputs than the layer layers. This discovery inspires us to observe whether the gradient of the early layers of the model is better aligned with the gradient of the ground truth data distribution. The general ResNet consists of four blocks and the residual module (He et al., 2016) can be expressed as: zi+1 = zi + F(zi), (7) where zi+1 is the output of residual modules and zi is the input of residual modules,F is the residual function. Considering that the information can transfer from shallow layers to deep layers through the skip connection in ResNet-like neural networks, we decrease the weight for certain residual modules to reduce the impact of these modules and enhance the impact of the previous modules from the skip connection. Thus, we modify the residual module as:
zi+1 = zi + λi · F(zi), (8) where 0 ≤ λi ≤ 1 can reduce the impact of the certain residual modules. We use the λi to relatively adjust the influence of different layers.
As shown in Fig. 3, when decreasing λ1:4 at the same rate on pre-trained ResNet-50 (Paszke et al., 2019), the AAI metric first increases then decreases with reaching its maximum at λ1:4 = 0.6. At the same time, different target models obtain the best attack success rate also at λ1:4 = 0.6 (see Fig. 3 (Middle)). This implies that adjusting the influence of different residual branches can help to increase AAI as well as transferability. Moreover, we find that decreasing the value of λ4 can significantly improve the transferability a lot (from 53% to 71.32%) (Fig. 3 (Right)). This shows that the last block does restrict adversarial transferability and reducing the impact of the last block can greatly improve the transferability. Decreasing λ1,2,3 for Block1, Block2, and Block3 can slightly improve the adversarial transferability. If we want to achieve the best transferability, we need carefully tuning the hyper-parameters λ1:4. As our AAI metric can be effectively predict the transferability, we use Bayesian optimization to search the best λ1:4 to maximize AAI.
We can also make full use of the distribution-relevant information of a given classifier to increase AAI without skip connections. As shown in Appendix J, there are many other ways that can enhance the impact of the early layers to increase AAI and can be applied to models without skip connection, but this is not the focus of this paper. Our main contribution is to rethink the adversarial transferability from a novel perspective and reveal that the distribution-relevant information may be the key for boosting adversarial transferability.
epoch
-A AI
3.2.3 THE ALGORITHM IAA
As analyzed in previous sections, we have verified that the AAI is an effective metric for transferability. The smoothness and data distribution information from different layers can be important factors for maximizing AAI. In order to find the best parameters, we use Bayesian optimization to optimize
min β,λ
EpD(x,y)Ep(v) [ vT∇x
vT∇xpθ,β,λ(y|x) ‖∇xpθ,β,λ(y|x)‖2
] . (9)
We propose Intrinsic Adversarial Attack (IAA) as shown in Alg. 1, which replaces the activation function ReLU with Softplusβ and applies corresponding decay parameters λi searched by Eq. (9) to different residual modules. Fig. 4 shows the training curve on pre-trained ResNet-50, see Appendix C for setting details. The experiments are shown in next section. We analyze the computation cost in Appendix M and show other optimization methods to find the proper structural parameters in Appendix O.

heading: 4 EXPERIMENTS

heading: 4.1 IMPLEMENTATION
text: Attack Setting. We use a black-box threat model to test the adversarial transferability. Adversarial examples are generated by attacking the source models, and then these examples are applied to attack target models. The architecture of the source model is different from that of the target model. When the architecture is the same, this setting is similar to the white-box attack. In terms of attack strength,
we follow the same standard setting for all attack methods (Xie et al., 2019; Wu et al., 2019). We constrain the adversarial perturbation within the `∞ ball of radius = 16/255 with respect to pixel value in [0, 1] and set the step size α to 2/255. The iteration steps in all the experiments are set to 10. All experiments in this paper are run on Tesla V100. The experiments are repeated 5 times with different random seeds and we just show the maximum standard deviations in the caption of tables due to page limitations. See Appendix C for more attack setting.
Target Models and Source Models. We conduct experiments on both normal target models and robust target models. For normal target models, we choose 9 models: VGG19 (Simonyan & Zisserman, 2015), ResNet-50 (RN50), ResNet-152 (RN152) (He et al., 2016), DenseNet-121 (DN121), DenseNet-201 (DN201) (Huang et al., 2017), 154 layers Squeeze-and-Excitation network (SE154) (Hu et al., 2018), Inception V3 (IncV3) (Szegedy et al., 2016), Inception V4 (IncV4), and InceptionResNet V2 (IncRes) (Szegedy et al., 2017), and we use the pre-trained models in PyTorch (Paszke et al., 2019). For robust target models, we consider 3 robustly trained models using ensemble adversarial training (Tramèr et al., 2018): IncV3ens3 (ensemble of 3 IncV3 models), IncV3ens4 (ensemble of 4 IncV3 models) and IncResV2ens3 (ensemble of 3 IncResV2 models). We choose 4 models as source models: ResNet-50, ResNet-152, DenseNet-121, DenseNet-201. We also choose VGG16, VGG19 (models without skip connection) as source model in Appendix J. All models are trained on ImageNet training set.

heading: 4.2 TRANSFERABILITY AGAINST NORMAL MODELS
text: Firstly, we focus on untargeted adversarial attacks. We compare different attack methods on four source models against nine normal target models. In all transfer scenarios, IAA outperforms existing methods by a large margin consistently against different target models. The results are reported in Tab. 1. For transfer ResNet-50→ VGG19, IAA achieves a success rate of 96.40% which is 43.40% and 13.68% higher than PGD and SGM respectively. For transfer ResNet-50→ Inception-ResNet V2, IR performs the best in the existing methods (attack success rate 46.78%) while our IAA can achieve a new state-of-the-art success rate 64.32%.
In Tab. 1, we show that IAA surpasses the other existing methods when attacking against normal models under Top-1 accuracy. Top-5 accuracy, which checks if the ground truth label is among the five predictions with the highest probability, is also an important metric. In Tab. 2, we show the first five predictions of the target model. For transfer ResNet-50→ Inception V3, the top-5 accuracy on adversarial examples crafted by PGD is 96.66%. SGM is an efficient method to enhance adversarial transferability, but the top-5 accuracy is as high as 84.28% for transfer ResNet-50 → Inception V3. Our IAA drives the examples to the low-density region by matching the adversarial attack and intrinsic attack. Thus the target models can hardly give a proper prediction, and the top-5 accuracy is significantly lower than other methods.
Secondly, we focus on targeted attacks. Liu et al. (2017); Yang et al. (2021b) show that generating transferable targeted adversarial examples is much more difficult than generating transferable untargeted adversarial examples. As we analyzed before, when crafting the untargeted attack, the intrinsic adversarial attack can drive adversarial examples to low-density regions. Similarly, when crafting the targeted attack, IAA can move the adversarial images towards the high-density region of the target class along the direction of ∇x log pθ(x, ytarget). Thus IAA can also improve the adversarial transferability of targeted examples, and we show the results in Tab. 3. For transfer ResNet-50 → VGG19 (targeted), IAA achieves a success rate of 33.68% which is 33.28% and 29.20% higher than PGD and SGM respectively.
We also evaluate the transferability of one-step attacks in Fig. 5. SGM can greatly enhance the transferability of the multi-step attack, but it can only slightly improve the effect of the one-step attack. DI performs better in the transferability of the one-step attack than SGM. We think that diversity inputs may help to reduce the noise in gradient and get better adversarial perturbation. Our IAA can significantly improve the adversarial transferability of the one-step attack. This is because that the perturbation crafted by IAA is aligned with the intrinsic attack −∇x log pD(x, y) and can efficiently drive the examples towards the low-density region. It is difficult for the target models to predict accurately for examples in the low-density region.

heading: 4.3 TRANSFERABILITY AGAINST ROBUSTLY TRAINED MODELS
text: The success rates of our IAA and other methods against the robust target models (using ensemble adversarial training) are reported in Tab. 4. TI was originally proposed for the robust models. Compared with the existing methods, IAA has a large improvement in all scenarios. For transfer ResNet-50→ IncV 3ens3, IAA achieves a success rate of 38.72% which is 31.2% and 5.91% higher than PGD and SGM respectively. We think that both the robust model and the normal model are trained on the same dataset. IAA perturbs the distribution-relevant information, which leads to misclassification of the robust models. We show that IAA can also effectively penetrate many other defense methods in Appendix N.

heading: 4.4 ENSEMBLE-BASED ATTACKS
text: Wu et al. (2019); Wang et al. (2020) show that the ensemble-based strategy (Liu et al., 2017) can improve the performance of their method. This section, shows the results in Tab. 5, and the attacks are crafted on an ensemble of ResNet-34, ResNet-152, and DenseNet-201. We show that our IAA with the ensemble-based strategy can generate more transferable perturbation than other methods. For transfer Ensemble model→ IncV4, IAA achieves a success rate of 90.70% which is 7.73% and 10.63% higher than SGM and IR respectively.

heading: 5 CONCLUSION
text: In this paper, we rethink the adversarial perturbation from the distribution perspective and show that the alignment between the adversarial attack and intrinsic attack (AAI) is an effective metric for predicting the adversarial transferability. Furthermore, we propose a novel algorithm (IAA) which maximizes AAI for generating adversarial examples in the low-density region by Bayesian optimization. We have conducted experiments on four source models, nine normal target models (including untargeted attack and targeted attack) and three robust target models and show that our methods surpass other existing methods by a large margin.
Our findings can motivate new research into the adversarial transferability and adversarial examples from the perspective of data distribution and open up further challenges for the defense against black-box attacks. What’s more, IAA can also fool the image search engines (Google Reverse Image Search and Baidu Reverse Image Search) as shown in Appendix A.
ETHICS STATEMENT
In this paper, we propose a novel method IAA to boost adversarial transferability. IAA can craft stronger adversarial examples from the source model to attack unknown target models than other existing methods. Our findings can motivate new research into the adversarial transferability and adversarial examples from the data distribution perspective and open up further challenges for the defense against black-box attacks. Our goal is to understand current deep learning’s weaknesses and make deep models more robust and transparent. We did not use crowdsourcing and did not conduct research with human subjects in our experiments. We cited the creators when using existing assets (e.g., code, data, models).
REPRODUCIBILITY STATEMENT
We show the proofs of Theorem 1 in Appendix E. Our IAA is an appealingly simple method. We specify the settings of hyper-parameters and how they were chosen in Appendix C. We repeat experiments 5 times with different random seeds and show the standard deviation in the caption of tables. We plan to open the source code to reproduce the main experimental results later.

heading: A FOOLING THE IMAGE SEARCH ENGINES
text: The previous experiment mainly verified the attack effect of IAA on the pre-trained classification model. In this section we show that IAA can fool the image search engines successfully. As shown in Fig. 6, the original image can be correctly identified by both Google Reverse Images Search and Baidu Reverse Images Search. However, the adversarial images generated by our IAA can fool the image search engines.

heading: B HIGH TRANSFERABLE EXAMPLES
text: We use a source model RN50 to generate adversarial examples with widely used PGD attack. If the original image can be correctly classified by different targeted models (VGG19, RN50, RN152, DN121, DN201 and SE154) and its adversarial counterpart can fool all the target models, we save this original image. In this way, we randomly selected 1000 samples from ImageNet evaluation datasets. We named these samples Boundary Data (B Data). If the original images can be correctly
classified by different targeted models (VGG19, RN50, RN152, DN121, DN201 and SE154) and their adversarial counterparts can just fool the source models, we save this original image. In this way, we randomly selected 1000 samples from ImageNet evaluation datasets. We named these samples Interior Data (I Data). We find an intriguing phenomena: LDD with random noise can fool different target models, while the attack success rate is much lower when applying the same strength of random noise to HDD. Specifically when we injecting Gaussian noise ( =16/255) in LDD, 23.3% of the samples can fool the target model VGG19. However, when we injecting Gaussian noise ( =16/255) in HDD, only 0.7% of the samples can fool the target model VGG19. What’s more, LDD can be correctly classified by different models but their adversarial counterparts generated by different source models have strong transferability against different target models (as shown in

heading: C ATTACK SETTING
text: We choose the baseline attack PGD (Madry et al., 2018) and 5 state-of-the-art transfer attacks: MI attack (Dong et al., 2018), DI attack (Xie et al., 2019), ILA attack (Huang et al., 2019), SGM attack
(Wu et al., 2019) and IR attack (Wang et al., 2020). And we take TI attack (Dong et al., 2019) which is proposed to attack robust model into consideration when attacking against robust models.
The scikit-optimize 1 is a simple and efficient library to minimize black-box functions, which helps us to search the best structure hyper-parameters for a pre-trained classifier to minimize the Eq. (9). We use Batesian optimization to optimize the Eq. (9) to search the hyper-parameters β and λi. For ResNet-50, the search results are β = 20, and λ1 applied to the residual modules in Block1 is 0.98, λ2 applied to the residual modules in Block1 is 0.87, λ3 applied to the residual modules in Block3 is 0.73, λ4 applied to the residual modules in Block4 is 0.19. For ResNet-152, the search results are β = 32, and λ1 applied to the residual modules in Block1 is 0.89, λ2 applied to the residual modules in Block1 is 0.88, λ3 applied to the residual modules in Block3 is 0.70, λ4 applied to the residual modules in Block4 is 0.20. For DenseNet-121 and DenseNet-201, the search results are β = 35, and λ1,2,3 applied to the residual modules in Block1, Block2 and Block3 is 0.80, λ4 applied to the residual modules in Block4 is 0.44.

heading: D SIMILARITY BETWEEN ADVERSARIAL PERTURBATION
text: We illustrate frequency histogram of cosine similarity between adversarial perturbation generated by different models in Fig. 9. We also illustrate frequency histogram of Pearson Correlation Coefficient (PCC) between adversarial perturbation generated by different models in Fig. 10. Perturbation generated by different models using IAA have stronger correlation than other methods (PGD and SGM), which means that IAA reduces the dependence on models and generate more distribution relevant perturbation.
1The website of scikit-optimize is https://scikit-optimize.github.io/stable/

heading: E PROOF: THEOREM 1
text: The proof is similar as Hyvärinen & Dayan (2005) and Song et al. (2020). AAI , EpD(x,y) [ ∇x log pθ,Λ(y|x) ‖∇x log pθ,Λ(y|x)‖2 · ∇x log pD(x, y) ]
(I) = ∫ dy ∫ dx [ ∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2 · ∇x log pD(x, y) ] pD(x, y)
= ∫ dy ∫ dx [ ∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2 · ∇xpD(x, y) ]
(II) = ∫ dy { lim R→∞ ∫ dSR n · ∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2 pD(x, y)− ∫ dx [ ∇x · ∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2 ] pD(x, y) } = ∫ dy { lim R→∞ ∫ dSR n · ∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2 pD(x, y) } − EpD(x,y) [ ∇x · ∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2
] (III) = −EpD(x,y) [ ∇x ·
∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2
] .
(10)
The log function can be moved for the equality (I) because the normalized ∇xp(y|x) and ∇x log p(y|x) are the same direction, as:
∇x log p(y|x) ‖∇x log p(y|x)‖2 = p(y|x)−1∇xp(y|x) ‖p(y|x)−1∇xp(y|x)‖2 = p(y|x)−1∇xp(y|x) p(y|x)−1‖∇xp(y|x)‖2 = ∇xp(y|x) ‖∇xp(y|x)‖2 , (11)
where we use the formula: ∇x log f(x) = f(x)−1∇xf(x). Take 1d case for example, we use integration by part formula for the equality (II):∫ +∞
−∞ f ′ (x)g ′ (x)dx = lim R→∞ f ′ (x)g(x)|+R−R − ∫ +∞ −∞ f ′′ (x)g(x)dx, (12)
The equality (III) holds by Holder Inequality (Bartle, 2014):∣∣∣∣ ∫ dSRn · ∇xpθ,Λ(y|x)‖∇xpθ,Λ(y|x)‖2 pD(x, y) ∣∣∣∣ ≤ [∫ p2D(x, y)dSR] 12 → 0, R→∞.
For a distribution of a random vector v such that Ep(v)[vvT] = I , then we have AAI = −EpD(x,y) [ ∇x ·
∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2 ] = −EpD(x,y)Ep(v) [ ∇x ·
vvT∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2 ] = −EpD(x,y)Ep(v) [ vT∇x
vT∇xpθ,Λ(y|x) ‖∇xpθ,Λ(y|x)‖2
] .
(13)
We use the property that random vector v is independent of (x, y) in Eq. 13, thus we can switch∇x and v.

heading: F PROOF: SOFTPLUS IS THE EXPECTATION OF RELU
text: Softplusβ(x) = 1
β log(1 + exp(βx)), (14)
where β is the shape-related hyper-parameter. The derivative of this function is d
dx Softplusβ(x) =
1
1 + exp(−βx) . (15)
When the pβ( ) is implicitly defined, we express Softplusβ(x) as:
Softplusβ(x) = ∫ +∞ −∞ pβ( )ReLU(x− )d . (16)
With respect to x, the differential of the equation is:
d
dx Softplusβ(x) = ∫ +∞ −∞ pβ( )Θ(x− )d = ∫ x −∞ pβ( )d , (17)
where Θ(x) = I(x > 0) is the Heaviside step function. From Eq. (15), the differential of Eq. (17) is:
β
(eβ /2 + e−β /2)2 = pβ( ). (18)
Thus, the Softplusβ can be expressed as the expectation of ReLU in a neighborhood as follows:
Softplusβ(x) = E ∼pβ [ReLU(x− )], (19)
where pβ( ) = β(eβ /2+e−β /2)2 .

heading: G THE INFLUENCE OF STRUCTURE PARAMETERS ON DENSENET-121
text: We first show the influence of the shape parameter β when we use replace ReLU with Softplusβ in Fig. 11 (Left). A proper β can improve the adversarial transferability. The general ResNet and DenseNet consists of four blocks. We illustrate the influence of applying λi on different blocks of ResNet-50 in paper. Here, we illustrate the influence of applying λi on different blocks of DenseNet121 in Fig. 11 (Right). The target model is VGG19. λi applied to Block4 has great influence on adversarial transferability, and λi=0.4 can improve the attack success rate from 56.78% to 67.70%. λi applied to Block1, Block2, and Block3 can slightly improve the adversarial transferability. When we apply the best λi to each block, the attack success rate can reach 85.08%, which has exceeded state-of-the-art methods SGM (80.18%) and IR (82.56%).

heading: H ADVERSARIAL EXAMPLES SHOW RECONIZABLE FETURES
text: We insert and fine-tune classifiers (fully connected layer) after each block of ResNet-50. Attack Block1 means attacking the classifier after block1. As show in Fig. 12, the perturbation generated
(e) Robust Model
by attacking normal ResNet-50 like unrecognizable noise. However the perturbation generated by attacking the classifiers deployed after block2 and block3 of normal ResNet-50 show recognizable features of tench. This shows that the normal model has also learned some distribution-relevant information.

heading: I CONDITIONAL GENERATING EXPERIMENT
text: We also show the input-gradients in the normal model are somewhat aligned with the gradient of the ground truth data distribution pdata(x|y) through a conditional generating experiment using the method proposed by Santurkar et al. (2019). The general ResNet consists of 4 blocks. We insert classifiers (fully connected layers) after each block of ResNet-50 to utilize information from different layers, then fine-tune these classifiers and use these classifiers to generate images. As shown in
Fig. 13(given the label tench), the classifier on Block4 (the last block) of the model cannot generate images with recognizable semantic features. The images generated by the classifiers on Block2 and Block3 are not good enough but can be recognized with high confidence. The generative capability of Block1 is poor maybe it does not learn enough class-relevant information. This experiment intuitively shows that manipulating input-gradients can generate images with the features of target distribution (tench).

heading: J OTHER METHODS TO ENHANCE THE IMPACT OF EARLY LAYERS
text: In our paper, IAA gives a appealingly simple implementation on ResNet-like neural networks. There are many other ways to adjust the impact of different layers to improve the adversarial transferability. These methods are slightly complex to implement, but they are also effective.
J.1 TAKING ADVANTAGE OF LOW-LEVEL INFORMATION
In our paper, we find that the later layers may reduce the adversarial transferability and we apply hyper-parameters λi to reduce the influence of the later layers. We can also use the new loss to enhance the influence of the early layers. We propose a loss to take advantage of low-level information during attacking :
`feature = 1
N N−1∑ i=0 KL(featureadv[i], featureori[i]), (20)
where KL is KL divergence, N is the number of blocks in source model, featureori is the feature of the natural examples extracted by the block i, and featureadv is the feature of the adversarial examples extracted by the block i. The general ResNet and DenseNet consists of four blocks. We optimize the classification loss and the low-level feature loss jointly during attacking:
maximize ‖δ‖p≤ [`(fθ(x+ δ), y) + γ`feature], (21)
where is a constant of the norm constraint and γ is a constant weight for the low-level feature loss. We call this method Enhancing the Early Blocks (EEB). As shown in Fig. 14, we show the influence of the hyper-parameter γ. Increasing the hyper-parameter γ (increasing the weight of low-level information) tends to improve adversarial transferability until it exceeds a threshold, e.g., γ=100. This may due to γ encourages the attack to utilize more transferable low-level information. However, the attack can be insufficient when just using low-level information and ignoring all highlevel class-relevant information.
EEB can also improve the adversarial transferability of the deep neural networks without skip connection (VGG16, VGG19). As shown in Tab.
J.2 ONLY ATTACK THE EARLY LAYERS
We insert a classifier after the Blocks3 (the penultimate block) of ResNet-50, then fine-tune this classifier, and attack this classifier to generate adversarial examples. Such a simple operation can compete with state-of-the-art methods (SGM). We call this method Dropping the Last Block (DLB).
K IAA UTILIZE THE LOW-LEVEL INFORMATION
Centered kernel alignment (CKA) can reliably identify correspondences between representations in networks (Kornblith et al., 2019; Nguyen et al., 2020). In order to show the difference between natural examples and adversarial examples, we show the CKA similarity between the features of natural examples and adversarial examples extracted by the same normal ResNet-50 (ImageNet). As shown in Fig. 15 (a), the adversarial examples generated by attacking the normal model using PGD show more difference from natural examples in features extracted by deep layers. In Fig. 15 (b), the adversarial example generated by attacking the normal model using our IAA are different from the natural example in the shallow layers. For example, in the shallow layers (layer 20 to layer 30), the CKA similarity between the adversarial examples generated by the normal model using PGD and the natural example is about 0.9, while the similarity of the adversarial example generated by IAA with the natural example is about 0.7. That is to say, IAA uses more low-level information when generating adversarial examples and IAA enhances the influence of the shallow layers on generating adversarial perturbation.

heading: L OTHER CORRUPTION CASES FOR THE MOTIVATION EXAMPLE
text: As shown in Tab. 8, we added other types of corruption like CIFAR-10-C and ImageNet-C (Hendrycks & Dietterich, 2019), including noise, blur, weather, etc. Consistent with the observations in Fig. 1, the target models are more likely to misclassify corrupted LDD data. We hypothesize that these samples are in the low-density region of the ground truth distribution where both source and target models are not well trained on. Moreover, we find that the adversarial counterparts of LDD have stronger adversarial transferability than that of HDD, which inspires us to match the direction of adversarial perturbation and the direction towards the low-density region to generate adversarial examples with high transferability.

heading: M COMPUTATION COST
text: Our proposed method includes two steps. The first step is adjusting the pre-trained source model to match the adversarial attack with the ground truth density decreasing direction by optimizing AAI, and the second step is generating adversarial samples by the modified model similar to the normal PGD attack.
Optimizing AAI to get a strong source model by Bayesian optimization on the pre-trained model needs some extra computation. As shown in Eq. 4, the computation cost for AAI can be reduced to O(d) from O(d2) by using ideas from sliced score matching (Song et al., 2020), where d is the dimension of data. The computation cost of AAI optimization is 10 hours on Tesla-V100 using ResNet-50.
However, the cost of generating adversarial examples is more important than the cost of adjusting the source model in practice. Once we get the modified model, generating different adversarial
examples by the modified model shares the same computation with normal PGD adversarial attacks. That is to say, it is very efficient to use our modified model to generate adversarial examples.

heading: N ATTACK OTHER DEFENSE METHODS
text: In Tab. 4, we evaluate that our IAA can fool the robust model by ensemble adversarial training (Tramèr et al., 2018). Moreover, we show that the adversarial examples generated by IAA can also effectively penetrate many other defense methods in Tab. 9. We further add neural representation purifier (NRP) (Naseer et al., 2020) which is a state-of-the-art input processing defense method, and some other robust training methods, including training with AugMix (Hendrycks et al., 2020), Styled ImageNet (SIN) (Geirhos et al., 2019), the mixture of Styled and natural ImageNet (SIN-IN), and adversarial examples (Salman et al., 2020) as defense models. In Tab. 9, we show the top-1 accuracy and top-5 accuracy (ACC1/ACC5) on different defense models when predicting the blackbox adversarial examples crafted by different methods. We use the ResNet-50 as the source model. IAA surpasses other attacks in different cases. Our IAA drives the examples to the low-density region by matching the adversarial attack and intrinsic attack. Thus the target models can hardly give a correct prediction on IAA examples, and both the top-1 accuracy and the top-5 accuracy are lower than other methods.

heading: O FURTHER DISCUSSION ON DIRECTLY OPTIMIZING THE TRANSFERABILITY.
text: In our paper, we used the method of optimizing AAI to choose hyperparameters for two reasons. On the one hand, we hope to verify the effectiveness of the metric we proposed. On the other hand, we think that using the transferability for a certain target model to optimize may overfit the target model. Interestingly, Fig.2 and Fig.3 show that the attack success rate among all the target models seems like aligns well under different structural hyperparameters. Maybe it won’t be so over-fitted to the target model that it is difficult to select the proper hyperparameters. We conduct experiments on the source model ResNet-50. We use four methods to choose the proper hyperparameters, including using the AAI as the optimization objective (”optimize-AAI”), using the transferability against the target model VGG19 as the optimization objective (”optimize-vgg19”), using transferability against the target model SE154 as the optimization objective (”optimize-senet154”), and using transferability against the ensemble of VGG19 and SE154 as the optimization objective (”optimize-ensemble”). The following table represents the top-1 accuracy and top-5 accuracy (ACC1/ACC5) when predicting the adversarial examples crafted by different methods. Lower accuracy indicates better adversarial transferability. We found that directly using the transferability as the optimization objective can also find proper structural hyperparameters and outperforms the existing methods PGD and SGM. The ensemble-based strategy can slightly improve the adversarial transferability. The computation cost of directly optimizing transferability is half that of optimizing AAI.

Review summaries:
This paper presents a new adversarial attack method which could generate adversarial examples with higher transferability. The proposed method is based on the observation that low-density region of the training data is not well trained. To utilize this, the authors try to align the adversarial direction with the direction to decrease the ground truth density. The proposed method is theoretically support. Their experiments show that the proposed method outperforms multiple adversarial attack method in almost all evaluated scenario. I believe this could be a useful attack method for generating transferable adversarial examples, and providing a strong counterpart for future research on adversarial defense.

This paper identifies that adversarial examples in the low-density region of the groud truth distribution have much stronger transferability. This observation leads to the AAI metric which evaluate the alignment of the model’s adversarial attack with intrinsic attack direction. The paper further identifies a set of model hyperparameters that can influence the AAI metric, and find the optimal hyperparameter choices to maximize AAI and generate more transferable adversarial examples.

This paper proposes Intrinsic Adversarial Attack (IAA), a transfer attack method based by jointly matching data distribution. The key assumption of this paper is that the DNN might not be well trained on low-density regions (LDD). Therefore, taking data distribution into consideration during attack could potentially improve the transferability. Empirical evaluation on different model architectures, normal models, robustly trained models, and ensemble-based attack context, demonstrates the advantage of IAA.