Rethinking Adversarial Transferability from a Data Distribution Perspective
Authors: Yao Zhu
Jiacheng Sun
Zhenguo Li
Venue: ICLR 2022 Poster (2022)
Keywords: Adversarial Attack
Adversarial Transferability
Black-box Attack

Abstract: Adversarial transferability enables attackers to generate adversarial examples from the source model to attack the target model, which has raised security concerns about the deployment of DNNs in practice. In this paper, we rethink adversarial transferability from a data distribution perspective and further enhance transferability by score matching based optimization. We identify that some samples with injecting small Gaussian noise can fool different target models, and their adversarial examples under different source models have much stronger transferability. We hypothesize that these samples are in the low-density region of the ground truth distribution where models are not well trained. To improve the attack success rate of adversarial examples, we match the adversarial attacks with the directions which effectively decrease the ground truth density. We propose Intrinsic Adversarial Attack (IAA), which smooths the activation function and decreases the impact of the later layers of a given normal model, to increase the alignment of adversarial attack and the gradient of joint data distribution. We conduct comprehensive transferable attacks against multiple DNNs and show that our IAA can boost the transferability of the crafted attacks in all cases and go beyond state-of-the-art methods.

Published as a conference paper at ICLR 2022
RETHINKING ADVERSARIAL TRANSFERABILITY FROM A DATA DISTRIBUTION PERSPECTIVE
Yao Zhu': Jiacheng Sun Zhenguo Li
Zhejiang University, "Huawei Noah's Ark Lab ee zhuy tauy@zju.edde @zju..du.cn, (sunjiacheng zienuoo @ @huawei.com
ABSTRACT
Adversarial transferability enables attackers to generate adversarial examples from the source model to attack the target model, which has raised security con- cerns about the deployment of DNNs in practice. In this paper, we rethink ad- versarial transferability from data distribution perspective and further enhance transferability by score matching based optimization. We identify that some sam- ples with injecting small Gaussian noise can fool different target models, and their adversarial examples under different source models have much stronger transfer- ability. We hypothesize that these samples are in the low-density region of the ground truth distribution where models are not well trained. To improve the at- tack success rate of adversarial examples, we match the adversarial attacks with the directions which effectively decrease the ground truth density. We propose Intrinsic Adversarial Attack (IAA), which smooths the activation function and de- creases the impact of the later layers of a given normal model, to increase the alignment of adversarial attack and the gradient of joint data distribution. We con- duct comprehensive transferable attacks against multiple DNNs and show that our IAA can boost the transferability of the crafted attacks in all cases and go beyond state-of- sfate-of-the-ate art methods.
INTRODUCTION
Deep neural networks (DNNs) are widely used in various safety-critical fields, but they are vulnera- ble adversarial examples (Szegedy et al., 2013). Adversarial attacks are imperceptible humans but catastrophic for the DNNs and can be transferred between different models (Goodfellow et al.) 2015 Liu al., 2017). Adversarial transferability enables attackers to generate adversarial exam- ples from the source model to attack unknown target models, which has raised security concerns about the deployment of DNNs in practice. Understanding the essence of adversarial transferability is a fundamental problem in deep learning. On the one hand, some works show that the character- istics of the source model, such as model architecture (Wu et al.] 2019), model capacity (Tramer) et al. 2017), and test accuracy (Wu & Zhu 2020), influence adversarial examples' transferability. On the other hand, some works think that the data-relevant information may be the key factor for adversarial transferability. Πyas et al. (2019) cxplain that adversarial perturbations non-robust features and not meaningless bugs, but it is hard to specifically define non-robust features. We want to further study transferability quantitatively from the data distribution perspective.
has been empirically observed that DNNs are relatively robust to random noise (Fawzi et al., 2016 However, in this work we find intriguing phenomenon: some samples sensitive to Gaussian noise, in the sense that injecting small Gaussian noise into these samples can fool dif- ferent models trained on the same dataset. Furthermore, their adversarial counterparts generated by different source models have much stronger transferability against different target models than other samples. We hypothesize that these samples are in the low-density regions of the ground truth distribution both source and target models are trained on, and models are not well trained in these regions. Thus predictions of these samples are easy to be perturbed and even not robust to small ran- dom noises. We denote this kind of data as Low-Densie
Data (LDD), while others as High-Denste
"This work was done when Yao Zhu was a research intern in Huawei Noah's Ark Lab. Corresponning to: Jiacheng Sun

---

Published as a conference paper at ICLR 2022
Data (HDD). As shown in Fig. 1 (Left), the attack success rate against different target models of LDD with different strengths of Gaussian noise is much higher than that of HDD. Furthermore, in Fig. (Right), the adversarial counterparts of LDD have much stronger transferability than the adversarial counterparts of HDD (see Appendix [ for details).
This phenomenon reveals that the location of data plays vital role in adversarial transferability and the adversarial examples of samples in the low-density region are strongly transferable. The most efficient direction towards the low-density region is V log pp (ar, y), where pp (ar,) the ground truth density of natural data. We name this direction Intrinsic Attack because it doesn't t depend on the models and only depends on the ground truth distribution. Thus, we propose match the adversarial attack with intrinsic attack for generating strong transferable adversarial examples.
LSE Vaico
4: kall LOE ENSE1
100
Quiteee Bate
9G21 -2- 400 9:/M
60-
30
20-
3.5
8/205
16/255
24/255
32/265
45/255
48/255
V9314
PED Attack
Figure 1: (Left) The attack success rate when injecting Gaussian noise into LDD and HDD. See Appendix D for more corruption experiments. (Right) The attack success rate of the adversarial examples for LDD and HDD by PGD (foe. e=16/255) against different target models (VGG19, RN152, DN201, SE154). The source model is ResNett-5.
We explore the potential of a classifier Pa.A (ylar) with parameters 0 and structure hyper-parameters A (see Sec. 3.I) to generate more transferable adversarial examples by aligning adversarial attack with intrinsic attack - V log (x, y). The adversarial attack of Pe.A(ylzz is usually generated by
PGD/FGSM method, and determined by ~ log Po.A (y).))) We match the Alignment between
the Adversarial attack and Intrinsic attack (AAI), (pp(w.)) [V po by modifying the structure parameters A for pre-trained network.
V. Pe.llr))
V lo v)
In order to maximize AAI, we should make Pa.A (y/az) smoother. Otherwise, v log P will oscillate frequently and hard to match V logPD PD az. y). For the commonly used ReL.U network, we can smooth it by replacing ReLU activation with Softplus (Nair & Hinton] 2010) with little change of the model's output. Maennel et al. (2220) show that the early layers of a network learn the local statistics of the data distribution better than the later layers, which motivates us to decrease the impact of later layers when generating adversarial examples to utilize the data iestriuttion-relevan information. We can closely match the adversarial attack with the intrinsic attack - V, log Pn (x. y) and improve the adversarial transferability by optimizing structure hyper-parameters A to maximize AAT as the objective function. We name our method as Intrinsic Adversarial Attack (IAA).
There are some interesting observations in our AAA experiments. Firstly, we find that the test aceu- racy of the source model may not be important. As shown in Fig 2 the accuracy of the pre-trained model with Softpluss fpflpuus:115 is around 60%, but the adversarial transferability of this model is much stronger than the model with foftpluss=45- Secondly, although the existing methods (Madry et al., 2018 Wu et al.) 2019) can significantly decrease the top-1 accuracy of the target models, the top-5 accuracy is still high. IAA can both decrease the top-1 accuracy and top-5 accuracy. Furthermore, the existing methods (Xie al., 2019: Wu et al., (019) can just slightly improve the one-step attack under different strengths, while our TAA surpasses the existing methods by large margin. These phenomena verify our hypothesis that IAA pulls examples to the low-density region, which causes prediction difficulty to the target models. Our main contributions are summarized below:
We propose an effective metric, AAI, to evaluate the alignment of the model's adversarial attack with intrinsic attack V log (m y). Furthermore, we show that AAI is also an effective metric for adversarial transferability.
'The AAT metric for LDD (0.1264) is much larger than HDD (0.0457), which shows that the direction of PGD attack on L.D aligns better than that on HDD (The AAT metric on all test. samples 0.052).
2

---

Published as a conference paper at ICLR 2022
We propose the Intrinsic Adversarial Attack (IAA) by maximizing AAI to generate more transferable adversarial examples.
We conduct comprehensive transfer attack experiments from different source models against nine naturally trained models and three ensemble secured models, showing that IAA can significantly improve the state-of-the-an transferability (both targeted and untar- geted attack) of adversarial examples (even improve 20% under some settings).
2 RELATED WORK
There are two types of adversarial attacks: white-box attacks and black-box attacks. White-box attacks assume that the attacker completely access the structure and parameters of the target model. Typical examples of white-box attacks are FGSM (Goodfellow et al.) (015), PGD (Madry al. 2018), and CW (Carlini Wagner] 2017). The black-box attack assumes that the attacker only knows the output of the target model (prediction or confidence). Black-box attacks are roughly divided into two types: estimating gradient with queries the target model (Papernot et al., 2017 Su et al. 2019 Yang et al., (020) and attacking surrogate model (Xie et al. [019 Dong et al, 2018 Wu et 2019p. Attacking surrogate model is much more efficient and can reduce the risk of exposure. Thus, many existing works focused on adversarial transferability.
Su et al. 2018) explore the factors influencing the transferability and show the architecture has greater influence than model capacity. Dong et al. (2018) show that the momentum of gradients can be used to improve the adversarial transferability. Xie et al. (2019) show the diversity of input data will enhance the adversarial transferability. Huang et al. (2019) fine-tune the adversarial examples by increasing perturbation on pre-specifeed layer. Wang et al. (2020) propose a loss to decrease interactions between perturbation units during attacking. Wu al. (2019) propose that reducing gradients from the residual modules is effective for improving transferability. Guo et al. (2020) re- moves ReLUU activations in the later layers get linear backpropagation and decreases the influence of intermediate layers. They only modify the backpropagation when generating adversarial exam- ples while keeping the forward prediction the original model. Based on this, Zhang et al. (2021) conjecture that backpropagating smoothly might be sufficient for improving transferability.
There are also some works adversarial attack and defense using generative models. [aseer al. (019) and Yang et al. (2021a) learn adversarial perturbation through conditional generative at- tacking model, which needs to carefully designed for certain classes. (Samangouei et al.) (2018); Song et (2018 8) use GANs or autoregressive models to detect and purify adversarial examples. Du Mordatch (201 9): Hill et al. (22T); (Srinivasan et al. (202T); Yoon et al. (202T) purify adversar- ial examples by EBM or score-based generative models. JEM (Grathwohl et al. 2020) shows that combining a classifier with EBM can help to obtain some robustness. However, adversarial attacks purification based on generative models are computationally costly. We want to modify normal classifier with little computation cost to enhance its adversarial transferability by maximizing our AAI metric.
3 METHODS
3.1 ALIGNMENT BETWEEN THE ADVERSARIAL ATTACK AND INTRINSIC ATTACK
For classifier fa,A parameterized by with structure hyper-parameters (e.g., hyper-parameters for architecture, activation function, etc.), and data æ, label total possible classes then fa.a(=) represents the t output of the last layer. The conditional density Pa.A (2.(3//2) x)) can be expressed as:
exp(fe (x)
( = ER exp( SoL (x)
The adversarial attack is usually based on -R 2015). The most effective direction towards low-density region is intrinsic attack - V log PD (z, y). To improve the adversarial transferability, we need to match model's adversarial attack direc- tion with intrinsic attack. We define the inner product of normalized -V log (1) x>) and -V, log pn (x. y) to quantify the matching of the direction of adversarial attack and the intrinsic attack.
Madry et al. 2018 Goodfellow et al,
3

---

Published as a conference paper at ICLR 2022
Definition 1 (AAI). For classifier Pe.A (y)zz), the Alignment between its Adversarial attack and the Intrinsic attack is:
log Po Vn log
AAT - R (a..))
(2)
log Po.A (1)
where PD y) is the ground truth joint distribution. Remark. (1) We use the normalized adversarial attack to remove the influence of scaling factor when comparing different models.
(2) This definition is equivalent with a modified score matching objective as:
log Po.A (
2 E
(w..))
V log (2. v) = AAI + Opds
V
Po. 2
where Cpp is constant only depend on the ground truth distribution pp-
As getting the gradient of pp y) is not feasible, we use integration by parts (Hyvärinen & Dayan 2005) to move the gradient on (a( to model's adversarial attack. With the smoothness assump- tion on 2aPo.A(U ar), we have the following theorem:
Theorem 1. uPo.A (y) a:) is differentiable almost everywhere, then
VaPo,A(yla)
AAI -EpD(
V
(3)
V N (1/ 1|2
Moreover,
T TV V
AAI (a.
(m. Ep(m)
v T V
(4)
V (3/ x))
where p(v) is distribution of random vector such that Ep(w) vve *1 = 1 (e.g., the multivariate standard normal (0,
This theorem makes it possible to calculate AAI without knowing the gradient of ground truth distribution when the model is smooth, see Appendix for the proof. Combined with sliced score matching (Song et al., 2020) we can efficiently approximate AAI on discrete samples. However, for ReLU networks, the model's second derivative is not well defined, which prevents from using this theorem. Thus, we need to smooth the ReLU models for better modifying the model to maximize AAI and improve the transferability. One obvious way to improve the smoothness is to replace ReLU activation with some smooth activation. In this paper, we use Softplus to show that smoothness greatly help us improve transferability.
3.2
INTRINSIC ADVERSARIAL ATTACK FOR NORMAL MODELS
3.2. 1
SMOOTHING THE CLASSIFIER BY SOFTPLUS
Srinivas & Fleuret (202) claim that the input- of classifier is somewhat aligned with the gradient of the ground truth data distribution (z( y). On the other hand, Dombrowski et al. (2019) show that the information in gradients may be hidden for that the gradient of the ReL.U activation function is discontinuous and will frequently change when the input closes to zero. Thus better gradient with less noise contribute matching the input-gradients V x log Pe. (yla) (z) and the gradients of ground truth distribution - V log pD (æ, y). We provide simple implementation that replacing the widely used activation function ReLU with Softplus (Nair & Hinton 2010):
1 Softplus log exp(Bx)),
where B shape-related hyper- hyper-parameter. The larger is, the better Softplus, approximates ReLU. The Softplus, can also be expressed the expectation of ReLU in neighborhood fol- lows:
Softplus Ee-py ReLU(( <)),
where Pa(c) = (elle 2eee//t---ee de/33a
Dombrowski et al. 2019 See Appendix F for a short proof.
Smilkov al. (017): Dombrowski al. (2019) show that the local average of gradient values
4

---

Published as a conference paper at ICLR 2022
make the gradient smoother. This means that the gradient obtained with Softplus is less noisy and more meaningful than ReLU. We use Softplus, as an approximation of the activation function ReLU in the pre-trained model. As shown in Fig. (Left), AAI first increases then decreases when decreasing 33 in Softplus, (the smaller the B, the smoother the classifiers and AAT achieves the largest value at B 15. Thus there exits trade-off between AAI and smoothness, and proper is needed.
0.6
90-
3.6
Sussett Rate
0.4
40
0.2
20
12111
30
1912
10
20
25
90 40
10
15
29
betta
Figure 2: (Left) AAI and accuracy of the normal ResNet-50 on ImageNet by replacing ReLU with different Softphus.. (Right) The attack success rate of adversarial examples generated by the pre- trained ResNet-50 modified with different plus.. 'he baseline each target model similar to the success rate at B 45. See Appendix G for illustration on DenseNet-121.
We conducted an experiment with the PGD attack on the randomly selected 5000 ImageNet vali- dation images which are correctly classified to investigate how the shape-related
B affects the adversarial transferability. As shown in Fig. _ (Right), when the hyper-parameter 33 is too small, Softplus, cannot approximate ReLU well, which reduces the effectiveness of ad adversarial perturbation and prediction accuracy. With a proper selection of /3 = 15 which is also the maximum of AAI, using Softplus, approximation of ReLU can significantly improve the transferability of adversarial examples.
3.2.2 UTILIZING DISTRIBUTION-REIEEE
INFORMATION FROM EARLY LAYERS
Maennel al. (2020) and Boopathy & Fiete (202I) show that the early layers can better capture the local statistics of the inputs than the later layers. This discovery inspires to observe whether the gradient of the early layers the model better aligned with the gradient of the ground truth data distribution. The general ResNet consists of four blocks and the residual module (He et al. 2016) can be expressed as:
21+1 2; F(z,),
where 1 is the output of residual modules and z; the input residual modules, F is the residual function. Considering that the information can transfer from shallow layers to deep layers through the skip connection in ResNet-like neural networks, we decrease the weight for certain residual modules reduce the impact of these modules and enhance the impact of the previous modules from the skip connection. Thus, we modify the residual module as:
Zi+1 = + X; F(z.),
(
where 0 < X; < 1 can reduce the impact of the certain residual modules. We use the A; to relatively adjust the influence of different layers.
As shown in Fig. 3. when decreasing A1-1 at the same rate on pre-trained ResNet-50 (Paszke et al. 2019) the AAI metric first increases then decreases with reaching its maximum at A1:1 0.6. At the same time. different target models obtain the best attack success rate also at λ = 0.6 (see Fig. 3 (Middle)). This implies that adjusting the influence of different residual branches can help to increase AAT as well as transferability. Moreover, we find that decreasing the value of Xa can significantly improve the transferability a lot (from 53% to 71.32%) (Fig. 3 (Right)). This shows that the last block does restrict adversarial transferability and reducing the impact of the last block can greatly improve the transferability. Decreasing 21,2.3 for Block1, Block2, and Block3 can slightly improve the adversarial transferability. If want achieve the best transferability, need carefully tuning the hyper-parameters A1.4- As our AAI metric can be effectively predict the transferability, Bayesian optimization search the best 71:4 maximize AAI.
5

---

Published as a conference paper at ICLR 2022
Mi
UB
135
168
Black
0. 50
Bleak
Blach
Figure 3: (Left) We illustrate the impact of applying the same À on all the residual modules of the normal ResNet-50 (ImageNet). X1-4 0.6 reaches the maximum of AAI while the accuracy of the modified model is only around 60%. (Middle) We illustrate the attack success rate when applying the same À all the residual modules of the normal ResNet-50. The A1:4 0.6 has the best success rate, which also has the maximum AAI. (Right) The impact of applying different X; on each block. The source model is ResNet-50 and the target model is VGG19. The horizontal line shows the attack success rate when combining the best X for each block.
Algorithm 1 Intrinsic Adversarial Attack (IAA)
Input: A pre-trained classifier fe: original im- age æ; for perturbation radius e; step size os iterations N; the classification loss E;
Output: The modified model fe. adver- sarial perturbation 5.
4.0
Solve Eq. to find the modified model fe,s-.>x ,x*, replace ReLU with Softplusg. and apply X to residual modules of fe. Initialize Uniform((ccc)
-.2
Figure 4: The training loss of Eq. with Bayesian optimization on pre-trained ResNet-50. The search results are 8 20, A = 0.98, A2 = 0.87, λ = 0.73, \4 0.19.
for i 1,2, .... N do =
...,
(agn(VVl(Jo,,,s 8),)), mar ( end for return 6
can also make full use of the distribution-relevanl information given classifier to increase AAI without skip connections. As shown in Appendix there are many other ways that can en- hance the impact of the early layers to increase AAI and can be applied to models without skip connection, but this is not the focus of this paper. Our main contribution is to rethink the adversarial transferability from a novel perspective and reveal that the distributioon-rllvann information may be the key for boosting adversarial transferability.
3.2.3 THE ALGORITHM IAA
As analyzed in previous sections, we have verified that the AAI is an effective metric for transfer- ability. The smoothness and data distribution information from different layers can be important factors for maximizing AAI. In order to find the best parameters, we use Bayesian optimization to optimize
min Epp (2o.9) Bp(w) Bp(() 3.X
T V
21
We propose Intrinsic Adversarial Attack (IAA) as shown in Alg which replaces the activation function ReLU with Softplus and applies corresponding decay parameters X; searched by Eq. (9) to different residual modules. Fig. 4 shows the training curve on pre-trained ResNet-50, see Ap- pendix for setting details. The experiments shown in next section. We analyze the computation cost in Appendix M and show other optimization methods to find the proper structural parameters in Appendix

---

Published as a conference paper at ICLR 2022
Table 1: Transfcrability against normal models: the success rates of black-box attacks (untargeted) crafted on RN50, RN152, DN121 and DN201.
Source
Attack
VGG19
RN50
RN152
DN121
DN201
SE154
IncV3
IncV4
IncRes
FGD
53.00%
99.94%
61.26%
55.62%
53.36%
24.78%
20.86%
21.96%
17.60%
MI
64.86%
99.72%
73.22%
73.50%
64.33%
47.20%
39.08%
37.35%
25 25.26%
DI
75.06%
99.78%
81.65%
81.98%
74.80%
52.42%
42.58%
39.30%
27.12%
RN50
ILA 83.56%
99.75%
92.46%
88.40%
85.24%
61.44%
49.94%
48.34%
35.74%
SGM
82.72%
99.82%
88.40%
83.56%
80.34%
61.30%
53.72%
49.83%
42.86%
IR
82.46%
99.80%
85.24%
84.35%
82.10%
64.20%
54.60%
51.05%
46.78%
IAA 96.40%
99.88%
98.32%
97.46%
96.64%
84.74%
76.28%
71.38%
64.32%
PGD
49.32%
72.72%
99.91%
53.44%
51.00%
26.32%
23.50%
22.58%
18.72%
MI
65.42%
83.40%
99.82%
77.60%
75.79%
53.00%
46.50%
43.32%
33.08%
DI 74.01%
88.18%
99.78%
79.46%
77.81%
57.49%
50.28%
47.16%
30%10%
RN152
ILA 66.20%
90.44%
99.88%
75.48%
73.80%
50.32%
42.32%
41.30%
29.98%
SGM
80.40%
96.10%
99.87%
85.80%
82.76%
61.90%
53.16%
49.24%
43.30%
IR 73.20%
92.70%
99.84%
83.43%
80.60%
64.00%
53.60%
50.30%
48.00%
IAA 94.46%
99.32%
99.87%
96.58%
95.06%
82.46%
76.34%
71.04%
58.34%
PGD
56.78%
63.22%
52.76%
99.94%
71.98%
31.46%
24.92%
26.82%
20.64%
MI
68.36%
74.18%
72.88%
99.80%
89.56%
58.58%
52.22%
45.35%
35.24%
DI 73.68%
79.56%
74.72%
99.85%
89.40%
53.34%
53.65%
47.94%
37.72%
DN121
ILA 87.76%
90.38%
83.42%
99.82%
95.32%
65.02%
58.64%
57.36%
40.76%
SGM
80.18%
88.54%
80.54%
99.79%
92.70%
64.92%
54.62%
49.82%
37.76%
IR 82.56%
86.14%
85,20%
99.82%
95.30%
72.20%
62.22%
62.10%
56.00%
IAA 96.80%
96.78%
93.90%
99. 88%
97.70%
88.34%
88.24%
87.36%
77.18%
FGD
57.76%
70.68%
59.08%
83.06%
99.89%
40.60%
33.80%
32.46%
23.80%
MI
75.09%
82.46%
76.39%
88.18%
99.84%
64.38%
59.62%
54.85%
39.40%
DI 78.11%
85.34%
78.18%
90.20%
99.81%
61.75%
60.04%
56.15%
40.56%
DN201
ILA 88.56%
94.78%
90.02%
98.02%
99.72%
76.34%
67.78%
65.36%
49, 49,50%
SGM
82.72%
91.72%
86.60%
96.40%
99.67%
72.20%
62.34%
56.36%
45.42%
IR 76.74%
90.46%
85.40%
95.39%
99.74%
73.60%
59.80%
63.00%
56.60%
IAA
96.32%
96.98%
93.82%
98.10%
99.78%
87.98%
88.26%
87.02%
79.12%
4 EXPERIMENTS
4.1 IMPLEMENTATION
Attack Setting. We use a black-box threat model to test the adversarial transferability. Adversarial examples are generated by attacking the source models, and then these examples are applied to attack target models. The architecture of the source model is different from that of the target model. In terms of attack strength, we follow the same standard setting for all attack methods (Xie et al. 2019 Wu et al. 2019). We constrain the adversarial perturbation within the Eoo ball of radius e 16/255 /55 with respect to pixel value in [0, 1] and set the step size a to 2/2 2/255. The iteration steps in all the experiments are set to 10. All experiments in this paper are run on Tesla V100. The experiments are repeated 5 times with different random seeds and we just show the maximum standard deviations in the caption of tables due to page limitations. See Appendix C for more attack setting.
Target Models and Source Models. We conduct experiments on both normal target models and robust target models. For normal target models, we choose models: VGG19 (Simonyan Zisser- man 2015), ResNet-50 (RN50), 152 (RN152) (He et al.) 2016), DenseNet-121 (DN 12), network (SE154) at. 2018), Inception V3 (Inc (Szegedy et al. 2016), Inception V4 (4n, and Inception-
DenseNet-200 201 (DN201) (Huang et al. 2017). 154 layers Squeeze-and-Excitatien
ResNet V (IncRes) (Szegedy et al., 2017). and we use the pre-trained models in PyTorch (Paszke)
al. 2019). For robust target models, we consider 3 robustly trained models using ensemble adver- sarial training (Tramèr et al. 2018): Inen33 (ensemble of 3 IncV3 models), V3en44 (ensem- ble of IncV3 models) and IncRes Venss3 (ensemble of IncResV2 models). We choose models as source models: ResNet-50, ResNet-152, DenseNet-121, DenseNet-201. We also choose VGG16, VGG19 (models without skip connection) as source model in Appendix
4.2 TRANSFERABILITY AGAINST NORMAL MODELS
Firstly, focus untargeted adversarial attacks. We compare different attack methods four source models against nine normal target models. In all transfer scenarios, IAA outperforms existing methods by large margin consistently against different target models. The results reported in
7

---

Published as a conference paper at ICLR 2022
Tab. [ For transfer ResNet-50 - VGG19, IAA achieves success rate of 96.40% which is 43.40% and 13.68% higher than PGD and SGM respectively. For transfer ResNet-50 + Creption-ResNer V2, IR performs the best in the existing methods (attack success rate 46.78%) while our IAA can achieve a new state-of-the-art success rate 64.32%.
In Tab. T show that IAA surpasses the other existing methods when attacking against normal models under Top-1 accuracy. Top-5 accuracy, which checks if the ground truth label is among the five predictions with the highest probability, is also an important metric. In Tab. 2 we show the first five predictions of the target model. For transfer ResNet-50 + Inception V3, the top-5 accuracy on adversarial examples crafted by PGD is 96.66%. SGM an efficient method to enhance adversarial transferability, but the top-5 accuracy is as high as 84.28% for transfer ResNet-50 + Inception V3. Our TAA drives the examples to the low-density region by matching the adversarial attack and intrinsic attack. Thus the target models can hardly give proper prediction, and the top-5 accuracy significantly lower than other methods.
Table 2: Top-5 accuracy on different target models when predicting the black-box adversarial exam-
ples crafted (untargeted) on RN50, RN152, DN121 and DN201. The best results are in bold.
Attack
VGGT9
DNT21
DN201
SET54
IncV3
Attack
VGGT9
RN50
RNI52
SE154
IncV3
RN152 RNS0
PGD
83,32
84.16
86.92
93.00
96.66
DN20 DN121
PGD
83.86
82.84
90.70
95.00
95.78
SGM
58.68
63.52
68.46
75.62
84.28
SGM
61.16
55.36
67.18
71.46
81.24
IAA
5.67
4.37
6.60
18.68
29.36
IAA
7.14
7.82
14.04
22.50
24.80
PGD
89.68
87.70
89.94
95.18
96.60
PGD
79,45
73.36
81.62
87.92
92,94
SGM
62.30
61.62
67.4
74.60
82.00
SGM
60.76
52.46
61.46
68.64
78.78
IAA
8.46
6.32
9.92
25.42
33.4
IAA
10.08
7.10
15,40
24.48
23,82
Table 3: Transferability against normal models: the success rates (%) of black-box attacks crafted
(targeted) on RN50, RN152, DN121 and DN201. The best results are in bold. The results are averaged on 8 different target classes.
Attack
VGG19
RN50
RN152
DN121
DN201
Attack
VGGT9
RN50
RN152
DN121
DN201
RNI52 RNS0
PGD
0.40
98.96
1,18
0.78
0,4
DN200 DN121
PGD
0.12
0.15
0.10
99.64
0.14
SGM
4.48
99.98
6.92
6.50
3.14
SGM
0.55
1.26
0.75
99.73
1.03
IAA
33.68
100
53.72
52.46
38.02
IAA
8,80
14.16
6.58
99.92
16.74
PGD
0.24
2.78
99.76
0.74
0.38
PGD
0.15
0.10
0.12
0.57
99.75
SGM
1.62
5.14
100
4.28
1.08
SGM
0.63
1.84
1.14
3.68
99.84
IAA
15.04
48.18
99.90
26.70
14.82
IAA
16.34
27.64
17.72
31,94
99.84
Secondly, we focus on targeted attacks. Liu et al. (2017):
Yang et al. (202Tb) show that generat-
ing transferable targeted adversarial examples is much more difficult than generating transferable untargeted adversarial examples. As we analyzed before, when crafting the untargeted attack, the intrinsic adversarial attack can drive adversarial examples to low-density regions. Similarly, when crafting the targeted attack, IAA can move the adversarial images towards the high-density region of the target class along the direction of V. log Pa (z. Vtarget). Thus IAA can also improve the adver- sarial transfcrability of targeted examples, and we show the results in Tab. 3 For transfer ResNet-50
+ VGG19 (targeted), iAA achieves a success rate of 33.68% which is 33. 33.28% and 29.20% higher than PGD and SGM respectively.
We also evaluate the transferability of one-step attacks in Fig. 5 SGM can greatly enhance the transferability of the multi-step attack, but it can only slightly improve the effect of the one-step attack. DI performs better in the transferability of the one-step attack than SGM. We think that diversity inputs may help to reduce the noise in gradient and get better adversarial perturbation. Our IAA can significantly improve the adversarial transfcrability of the one-step attack. This is because that the perturbation crafted by IAA is aligned with the intrinsic attack - -a PogPD( (x, v) and can efficiently drive the examples towards the low-density region. It is difficult for the target models to predict accurately for examples in the low- low-density region.
4.3 TRANSFERABILIT AGAINST ROBUSTLY TRAINED MODELS
The success rates of our IAA and other methods against the robust target models (using ensemble adversarial training) are reported in Tab. TI was originally proposed for the robust models. Compared with the existing methods, IAA has a large improvement in all scenarios. For transfer ResNet-50 IncV3 IAA achieves a success rate of 78% which is 31.2% and 5.91% higher than PGD and SGM respectively. We think that both the robust model and the normal model are trained on the same dataset. TAA perturbs the distributioon-elevant information, which leads to
8

---

Published as a conference paper at ICLR 2022
Sporise Baxe
30
8.30
15/955
16/255
20/255
26/25
19/255
16:955
20/15
26/15
4725
13/225
16/255
26/255
43132100
lest notS
The attack success rate (one-step attack) of different methods against different target models. The source model RN50. The horizontal axis represents different attack strengths. (Left) The target model is DN121. (Middle) The target model SE154. (Right) The target model is V3c
Figure 5:
misclassification of the robust models. We show that IAA can also effectively penetrate many other defense methods in Appendix N
Table 4: Transferability against robustly trained models: the success rates of black-box attacks
crafted on source models: ResNet-50, ResNet-152, DenseNet121 and DenseNet201.
Source
Attack
IncRes IncV3enut IncV3
Source
Attack
/ncV/Jena
IncV 3
IneRtssass
PGD
7.52%
9.50%
5,58%
PGD
13.80%
13.10%
7.90%
T1
30.06%
23.48%
16.201
11
28.10%
27.50%
21.20%
RNS0
SGM
32.81%
26.80%
20.02%
DN121
SGM
40.20%
36.83%
27.86%
IR
29.56%
22.18%
13.90%
IR
26.00%
24.10%
15.90%
IAA
38.72%
33.62%
24.32%
IAA
59.52%
43.38%
43.16%
FGD
12.20%
10.80%
3.70%
PGD
18.16%
13.30%
10.40%
35.97%
20.16%
42.76%
42.01%
34.28%
RN152
SGM
31.57%
27.77%
20.84%
DN201
SGM
41.45%
37.85%
29.41%
IR
29.34%
24.33%
13.76%
IR 38.90%
38.45%
28.70%
IAA
43.28%
37.88%
26.78%
IAA
61.02%
53.80%
46.34%
4.4 ENSEMBLE-BASED ATTACKS
Wu et al. (2019); Wang al. (220)) show that the ensemble-based strategy (Liu al., 2017) can improve the performance of their method. This section, shows the results in Tab. and the attacks are crafted on an ensemble of ResNet-34, ResNet-152, and DenseNet-201. We show that our IAA with the ensemble- For transfer Ensemble model + Inc V4, IAA achieves a success rate of 90.7000 90.70% which is 7.73% and 10.63% higher than SGM and IR respectively.
strategy can generate more transferable perturbation than other methods.
Table Transferability against different models: the success rates black-bor box attacks (untargeted) crafted on ensemble of models (RN34, RN152 and DN201). The best results are in bold.
Model
Attack
VGG16
VGG19
SE154
IncV3
IncV4
IncRes
PGD
86.60%
86.69%
69.65%
69.95%
59.30%
53.91%
TI 85.84%
84.35%
71.67%
67.22%
66.02%
56.83%
DI 96.85%
96.34%
89.72%
87.53%
85.04%
81.11%
Ensemble
SGM
96.84%
97.36%
90.40%
87.86%
82.97%
80.93%
IR 91.50%
91.16%
86.12%
81.69%
80.07%
79.34%
IAA
99.60%
99.68%
95.34%
93.60%
90.70%
81.54%
5 CONCLUSION
In this paper, we rethink the adversarial perturbation from the distribution perspective and show that the aligmment between the adversarial attack and intrinsic attack (AAI) an effective metric for predicting the adversarial transferability. Furthermore, we propose novel algorithm (IAA) which maximizes AAT for generating adversarial examples in the low-density region by Bayesian optimization. We have conducted experiments on four source models, nine normal target models (including untargeted attack and targeted attack) and three robust target models and show that our methods surpass other existing methods by large margin. Our findings motivate new research into the adversarial transferability and adversarial examples from the perspective data distribution and open up further challenges for the defense against black-brr box attacks. What's more, IAA can also fool the image search engines (Google Reverse Image Search and Baidu Reverse Image Search) shown in Appendix

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
In this paper, we propose novel method IAA boost adversarial transferabiliity. IAA can craft stronger adversarial examples from the source model to attack unknown target models than other existing methods. Our findings can motivate new research into the adversarial transferability and adversarial examples from the data distribution perspective and open up further challenges for the defense against black-box attacks. Our goal is to understand current deep learning's weaknesses and make deep models more robust and transparent. We did not use crowdsoureing and did not conduct research with human subjects in our experiments. We cited the creators when using existing assets (e.g., code, data, models).
REPRODUCIBILLT STATEMENT
We show the proofs of Theorem in Appendix E Our IAA is appealingly simple method. We specify the settings of hyper-parameters and how they were chosen in Appendix C We repeat exper- iments times with different random seeds and show the standard deviation the caption of tables. We plan to open the source code to reproduce the main experimental results later.
REFERENCES
Robert G Bartle. The elements of integration and Lebesgue measure. John Wiley & Sons, 2014.
Akhilan Boopathy and lla Fiete. Gradient--rained weights in wide neural networks align layerwise to error-scaled input correlations. arXiv e-prints, pp. arXiv-2106, 2021.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. IEEE, 2017.
Ann- Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry blame. Advances in Neural Information Processing Systems, 32: 32:13589-13600, 2019.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boost- ing adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018.
Yinpeng Dong, Tianyu Pang. Hang Su, and Jun Zhu. Evading defenses to transferable adversar- ial examples by rraslation-invarian attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4312 4321, 2019.
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
Alhussein Fawzi, Seyed-Mohsee Moosavi Moosavi-Dezfooli, and Pascal Frossard. Robustness of classifiers: from adversarial to random noise. Advances in Neural Information Processing Systems, 1631 1640, 2016
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019. URL /attps://openreviiee net Ledee//ooo...d-/gg111
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Will Grathwohl, Kuan-Chieh Wang, Jömn Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. International Conference on Learning Representations, 2020.
Yiwen Guo, Qizhang Li, and Hao Chen. Backpropagating linearly improves transferability of ad- versarial examples. Advances in Neural Information Processing Systems, 2020.
10

---

Published as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. Proceedings of the International Conference on Learning Represen- futions, 2019.
Dan Hendrycks, Norman Mu*, Mu", Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple method to improve robustness and uncertainty under data shift. In International Conference on Learning Representations, 2020. BRLhttpp::/ ptttts:://operevviee
1ttp gmrxHFvR
Mitch Hill, Jonathan Mitchell, and Song-Chun long-run dynamics of energy-based models. International Conference Learning Representa- lions, 2021.
Chun Zhu. Stochastic security: Adversarial defense using
Jie Hu, Li Shen, and Gang Sun. ggueeze-andd-exiiatin networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhanc- ing adversarial example transferability with an intermediate level attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4733-4742, 2019.
Aapo Hyvärinen and Peter Dayan, Estimation of non-normalized statistical models by score match- ing. Journal of Machine Learning Research, 6(4), 2005.
Andrew Ilyas, Shibani Santurkar, Logan Engstrom, Brandon Tran, and Aleksander Madry. Ad- versarial examples are not bugs, they features. Advances in neural information processing systems, 32, 2019.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial exam- ples and black-box attacks. International Conference on Learning Representations, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Hartmut Maennel, Ibrahim Alabdulmohsin, Ilya Tolstikhin, Robert JN Baldock, Olivier Bousquet, Sylvain Gelly, and Daniel Keysers. What do neural networks learn when trained with random labels? ing Systems, 2020. URL https: httpps:/papers..ot wctppppee//2///111.e... 11111
Advances in Neural Information Process-
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning, pp. 807-814, 2010. URL URL https://immm.
http://110/////// 432.
Muhammad Muzammal Naseer, Salman H Khan, Muhammad Haris Khan, Fahad Shahbaz Khan, and Fatih Porikli. Cross-domain transferability of adversarial perturbations. Advances in Neural Information Processing Systems, 32: 2990--12991 2019.
Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self- supervised approach for adversarial robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 262-271, 2020.
11

---

Published as a conference paper at ICLR 2022
Thao Nguyen, Maithra Raghu, and Simon Korbblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In Interna- tional Conference on Learning Representations, 2020.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pp. 506-519, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high high-performance deep learning library. Advances in Neural Information Processing Systems 32, pp.
2019.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? In Advances in Neural Information Processing Systems, 2020.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan Protecting classifiers against adversarial attacks using generative models. International Conference on Learning Representa- tions, 2018.
Shibani Santurkar, Andrew Ilyas, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Image synthesis with a single (robust) classifier. Advances in Neural Information Process- ing Systems, 32: 122--1773 2019. URL URUhttpss
www.ineuippp. cc/paper/ Paper.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 00119//lee/222661111333333bb2bb2
2015.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv: 700::03005. 2017.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. Interna- tional Conference on Learning Representations, 2018.
Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pp. 574-584. PMLR, 2020.
Suraj Srinivas and Francois Fleuret. Rethinking the role of gradient- attribution methods for model interpretabilitty In International Conference on Learning Representations, 2021. URL
.et/ http://wwwww..eeeformdddddd---e /tte://rrmmmmmddeedemmaaaa
Vignesh Srinivasan, Csaba Rohrer, Arturo Marban, Klaus-Robert Müller, Wojeiech Samek, and Shinichi Nakajima. Robustifying models against adversarial attacks by langevin dynamics. Neu- ral Networks, 2021.
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, Pin- Yu Chen, and Yupeng Gao. Is robustness the cost of accuracy?-a comprehensive study on the robustness of 18 deep image classification models. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 631-648, 2018.
Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 23(5):828 841, Oct 2019. ISSN 1941-0026. doi: 22110099ee22219.2900000 URL http:///x/x doi org. 1109/ TEVC. 2019 2890858
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference on Learning Representations, 2013.
12

---

Published as a conference paper at ICLR 2022
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning, In Thirty-forst AAAI con- ference on artificial intelligence, 2017.
F'Tammer D Boneh, Kurakin, Goodfellow, Papernot, and P McDaniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations, 2018.
Florian Tramer, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space 2017.
of transferable adversarial examples. arXiv preprint
Xin Wang, Jie Ren, Shuyun Lin, Xiangming Zhu, Yisen Wang, and Quanshi Zhang, A unified approach interpreting and boosting adversarial transferability. International Conference on Learning Representations, 2020.
Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In International Conference on Learning Representations, 2019.
Lei Wu and Zhanxing Zhu. Towards understanding and improving the transferability of adversarial examples in deep neural networks. In Asian Conference on Machine Learning, pp- 837-850.
PMLR, 2020.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang. Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp. 2730-2739, 2019.
Jiancheng Yang, Yangzhou Jiang, Xiaoyang Huang, Bingbing Ni, and Chenglong Zhao. Learning black-box attackers with transferable priors and query feedback. Advances in Neural Information Processing Systems, 33, 2020.
Xiao Yang, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Boosting transferability of targeted adversarial examples via hierarchical generative networks. arXiv preprint arXiv:2107.01880 10999 2021a.
Xiao Yang, Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Boosting transferability of targeted adversarial examples via hierarchical generative networks. CoRR, abs/2107. 018009 202
Jongmin Yoon, Sung Ju Hwang, and Juho Lec. Adversarial purification with score-based generative models. International Conference Machine Learning, 2021.
Chaoning Zhang, Philipp Benz, Gyusang Cho, Adil Karjauv, Soomin Ham, Chan-Hyun Youn, and In So Kweon. Backpropagating smoothly improves transferability of adversarial examples. In CVPR 2021 Workshop Workshop on Adversarial Machine Learning in Seal-World Computer Vi- sion Systems and Online Challenges (AML-CV), 2021. 2021.
13

---

Published as a conference paper at ICLR 2022
A
FOOLING THE IMAGE SEARCH ENGINES
Origiani Iesage Bluz
Google Reverse Image Seach. Blue Jay
Baiku Reverse Image Search, Blus Jay
PGD lay
Google Reverse Search: bo
Baidu Reverse Search:
Image Joy
Google
image
Origiual Imagee Red-spatted
Image aodee
Image
Guople Image Red-spoted
Image
Image suotee
Google Image Bulliing
Image Chensinensi
Figure 6: We show that the adversarial perturbation generated by our IAA is imperceptible to human observers but can fool the images search engines. The first three lines show the recognition results of the image Blue Jay in different image search engines. Baidu Reverse Images Search can't give the classification results when identifying our adversarial examples, and the similar pictures searched are not matched with Blue Jay. The last three lines show the recognition results of Red-spotted Toad the image search engines. Both Google Reverse Images Search and Baidu Reverse Images Search give wrong recognition results.
The previous experiment mainly verified the attack effect of TAA on the pre-trained classification model. this section we show that TAA can fool the image search engines successfully. shown in Fig. 6 the original image can be correctly identified by both Google Reverse Images Search and Baidu Reverse Images Search. However, the adversarial images generated by our IAA can fool the image search engines.
B HIGH TRANSFERABLE EXAMPLES
We use source model RN50 to generate adversarial examples with widely used PGD attack. If the original image can be correctly classified by different targeted models (VGG19, RN50, RN152, DN121, DN201 and SE154) and its adversarial counterpart this original image. Then, we randomly save 1000 samples from ImageNet evaluation datasets. We named these samples Low-Density Data (I. Data). If the original images can be correctly classified
fool all the target models,
14

---

Published as a conference paper at ICLR 2022
by different targeted models (VGGI9, RN50, RN152, DN121, DN201 and SE154) and their adver- sarial counterparts can only fool the source models, we save this original image. Then, we randomly save 1000 samples from ImageNet evaluation datasets. We named these samples High-Density Data (H Data). We find an intriguing phenomena: LDD with random noise can fool different target mod- els, while the attack success
is much lower when applying the same strength of random noise to HDD. Specifically when we injecting Gaussian noise (e=16/255) in LDD, 23.3% of the samples can fool the target model VGG19. However, when we injecting Gaussian noise (e= 16/255) HDD, only 0.7% of the samples can fool the target model VGG19. What's more, LDD can be correctly classified by different models but their adversarial counterparts generated by different source models have strong transferability against different target models (as shown in Fig. 3.1 and Fig. 7 In Fig. 8 we show some samples of L.D.
1.00
LO0
95.2
80.8
Success Rate
Attack (DM121)
PGD Attaok (V9616)
Figure 7: The histogram shows the attack success rate of the adversarial examples for LDD and HDD using PGD (loo, 6=16/555) against different target models (VGG19, RN152, DN201 and SE154). The source model DenseNet-12 121 (Left) and VGG16 (Right).
Rhodesian ridgeback
Otierbound
Water bon
Bighorn sheep
Loggerhead
Wolf soidsr
Ieecream
Lighthouse
Bridegroom
Cliff
Bullesproof
Photocapier
Upright piano
West Highland white
American lobster
Burrito
Marnot
Long's
Figure 8: Some samples which have high adversarial transferable counterparts. These samples can be correctly classified by different target models.
C ATTACK SETTING
We choose the baseline attack PGD (Madry al. 2018) and 5 state-of-theeart transfer attacks: MI
attack (Dong et al.) 2018). DI attack (Xie et al. 2019). ILA attack (Huang et al.] (019), SGM attack
Wu et al. 2019) and IR attack (Wang et al. 2020). And take TI attack (Dong et al. 2019) which
15

---

Published as a conference paper at ICLR 2022
is proposed to attack robust model into consideration when attacking against robust models. All the source models are trained on ImageNet training set.
The siikitoooiii is simple and efficient library to minimize black-box functions, which helps us to search the best structure perrmpareetterss for pre-trained classifier to minimize the Eq. (9). We use Batesian optimization to optimize the Eq. (9) search the nyper-parameters 3 and X₃- For ResNet-50, the search results are 3 = 20, and A applied to the residual modules in Block1 is 0.98, 22 applied the residual modules in Blockl is 0.87, X3 applied to the residual modules in Block3 is 0.73, b4 applied to the residual modules in Block4 is 0.19. For ResNet- 152, the search results are 32, and Å applied the residual modules Block1 0.89, X2 applied to the residual modules in Blockl is 0.88, A3 applied to the residual modules in Block3 is 0.70, x applied to the residual modules in Block4 0.20. For DenseNet-12 121 and DenseNet-201, the search results are 3 35, and 21, applied to the residual modules in Block1, Block2 and Block3 is 0.80, X applied to the residual modules in Block4 0.44.
D SIMILARITY BETWEEN ADVERSARIAL PERTURBATION
We illustrate frequency histogram of cosine similarity between adversarial perturbation generated by different models in Fig. We also illustrate frequency histogram of Pearson Correlation Coeffi- cient (PCC) between adversarial perturbation generated by different models Fig. 10 Perturbation generated by different models using IAA have stronger correlation than other methods (PGD and SGM), which means that IAA reduces the dependence on models and generate more distribution relevant perturbation.
16
01
5.2
0.3
3.2
8.3
(0) ResNet-54 ResNet-50
(90 ResNes 34 & ResNee115
(c) ResNee-50 A Rechin- 150
Figure 9: Frequency histogram of cosine similarity hetween adversarial perturbations generated by ResNet-34, ResNet-50 and ResNet-152. The perturbations generated by our IAA methods are closer to each other with different source model than perturbations generated by PGD and SGM.
99-
SM
Figure 10: Frequency histogram of Pearson Correlation Coefficient between adversarial perturba- tions generated by ResNet-34, ResNet-50 and ResNet-152. The perturbations generated by our IAA methods are closer to each other with different source model than perturbations generated by PGD and SGM.
'The wehsite of scikit-optimize http:///sskii--ppiiiegggttuuihhi
16

---

Published as a conference paper at ICLR 2022
E PROOF: THEOREM 1
The proof is similar as Hyvärinen & Dayan (2005) and Song et al. (0200)
log ,a. x)
AAT - (x.))
V logpp(z.u))
m log Pa,A (3/ 22
Po. A (U/ x>)
dy
dr
V log 3)
PD( v)
V TPPe (1))
VmPo.A (y/ xx)
dy
dr
(=.D(..)))
V aaa. a 22
dS n VaPa. (D 1) - f fdx V
V A x)
dy
lim
pr v)
R1+00
=oo az)
V A 22)
V V
V TaPo_A(ylaz
dy
lim
dS R
PD (x. 3) (po(
V
V TP
(
V (=Po.A(y(2))
(x.0)
V
(
(10)
The log function can be moved for the equality (I) because the normalized Va log p(ylat) are the same direction, as:
V>p(y/>) and
V logp(ylz)
[ep(yl>)
V Vxr(v/>))22 (11)
V 2 p(y())))
[[p(yl)) -p(y/=)11
where use the formula: V. log (x:) fxx)
Take 1d case for example, we use integration by part formula for the equality (II):
+00
(z)g lim f (=)g(=))
(x))g(z))
(12)
The equality (III) holds by Holder Inequality (Bartle 2014):
|
(yyle)
[ pfb y)dS
dSn [V (D(2,,)) v) s
0, 00.
For distribution of a random vector such that Ep(n)/oo I, then we have
VaPo (oppA((/aa)) (z)
AAT (pp((.
V.
V=Po,A(yl=))
V vv T VoVPaoo
Epo(eee) Ep(c)
(13)
v V''naoo (3/l))
Bppr)
V V
VaPo.A (1 22
We use the property that random vector independent of (ar Eq.
13 thus we can switch V.
and v.
F PROOF: SOFTPLUS IS THE EXPECTATION OF RELU
Softplus,(z) log + exp(Bx)),
(14)
where / is the shape-related hyper-parameter. The derivative of this function is
d
1
Softplus = exp(-3x)
(15)
dr
17

---

Published as a conference paper at ICLR 2022
When the pa(c) is implicitly defined, express Softplus as:
Softpluss(z) +o0 PeL()) -e)de.
e)de.
(16)
With respect to x, the differential of the equation is:
d Softplus,( = ps(c)0(x- - c)dc = [ (e(c)de,
(17)
dx
where ((x) {x > 0) is the Heaviside step function. From Eq. 15 the differential of Eq. 17
is:
8 ee/ 80/2/2 Pp(c).
(18)
Thus, the Softplus can be expressed as the expectation of ReLU in neighborhood as follows:
Softplus,(z) = Be-py RRLLU( +)],
(19)
where po(e) (e2//2te-
G THE INFLUENCE OF STRUCTURE PARAMETERS ON DENSENET-122 121
We first show the influence of the shape parameter when we use replace ReLU with Softplusp in Fig. II (Left). A proper 3 can improve the adversarial transfcrability. The general ResNet and four blocks. We illustrate the influence of applying X; on different blocks of
DenseNet consists ResNet-50 in paper. Here, illustrate the influence of applying on different blocks of DenseNet- 121 in Fig. [1 Right.. The target model is VGG19. X; applied to Block4 has great influence on adversarial transferability, and A;=0.4 can improve the attack success rate from 56.78% 67.70%. X applied to Block1, Block2, and Block3 can slightly improve the adversarial transferability. When apply the best 0₂ each block, the attack success rate can reach 85. 85.08%, which has exceeded statc-of-the-ar methods SGM (80.1%%) and IR (82.56%).
85
75
46,
98
BS,
SO 7B
Sucoese Rate
fane
45
Brook
VGG19
Blook 2
RN152
29. .4
87. 32
Blook 3
look
20
SE154
10
20
30
40
50
60
80
100
2.
70 0. 1.00
Figure 11: (Left) The attack success rate of adversarial examples generated by the pre-trained (DenseNet-121) modified with different Soflplusg. (Right) The influence of applying À on dif- ferent blocks. The source model is DenseNet-121 and the target model is VGG19. The green line with triangle mark shows attack success rate when applying À to Block4 (the last block), and À = 0.4 has the best effect. The red line with star mark shows attack success rate when applying to Block3, and À 0.8 has the best effect. The purple line with cross mark shows attack success rate when applying to Block2, and = 0.8 has the best effect. The blue line with circular mark shows attack success rate when applying the attack success rate when we respectively choose the best À for different blocks.
Block1, and À = 0.8 has the best effect. The horizontal line shows
H ADVERSARIAL EXAMPLES SHOW RECONIZABLE FETURES
We insert and fine-tune classifiers (fully connected layer) after each block of ResNet-50. Attack Block1 means attacking the classifier after block1. As show in Fig. 12 the perturbation generated
18

---

Published as a conference paper at ICLR 2022
Original Image
Attack Block
Attack Block2
Attack Block3
Attack Block4
We visualize the targeted adversarial examples by attacking the classifier deployed af- ter the different blocks of normally trained ResNet-50 and by attacking the robust ResNet-50. The perturbation radius is 64/255. The target label is tench. Attack Block1 means generating adver- sarial example by attacking the classifier deployed after the first block. The Block4 is the last block of ResNet-50 and Attack Block4 means generating adversarial example by attacking the normal ResNet-50.
Figure 12:
(a) Normal Block 1
(b) Normal
Block 2
(c) Normal Block 3
(d) Normal Block 4
Figure 13: We visualize the images generated by the classifiers fine-tuned after different blocks of a normally trained model. The target label is tench. The early blocks of the model has better generation capability than the last block, which also reflects that the early blocks has learned some distribution-relevant information.
by attacking normal ResNet-50 like unrecognizable noise. However the perturbation generated by attacking the classifiers deployed after block2 and block3 of normal ResNet-50 show recognizable features of tench. This shows that the normal model has also learned some distribution relevann information.
19

---

Published as a conference paper at ICLR 2022
I CONDITIONAL GENERATING EXPERIMENT
We also show the input-gradients in the normal model are somewhat aligned with the gradient of the ground truth data distribution Pdttt (rly) through a conditional generating experiment using the method proposed by Santurkar et al. (211)). The general ResNet consists of 4 blocks. We insert classifiers (fully connected layers) after each block of ResNet-50 to utilize information from differ- ent layers, then fine-tune these classifiers and use these classifiers generate images. As shown in Fig. 13 given the label tench), the classifier on Block4 (the last block) the model cannot generate images with recognizable semantic features. The images generated by the classifiers Block2 and Block3 are not good enough but can be recognized with high confidence. The generative capabil- ity of Blockl is poor maybe does not learn enough class-relevant information. This experiment intuitively shows that manipulating input-gradients can generate images with the features of target distribution (tench).
J OTHER METHODS TO ENHANCE THE IMPACT OF EARLY LAYERS
In our paper, IAA gives appealingly simple implementation ResNet-like neural networks. There are many other ways adjust the impact of different layers to improve the adversarial transferability. These methods are slightly complex to implement, but they also effective.
J.1 TAKING ADVANTAGE OF LOW-LEVEL INFORMATION
In our paper, we find that the later layers may reduce the adversarial transferability and apply nyper-parameters A; to reduce the influence of the later layers. We can also use the new loss to en- hance the influence of the early layers. We propose a loss take advantage low-level information during attacking feature Σ
N-1
KL [4, fea featureor(()).
(20)
where KT. is KI. divergence, is the number of blocks in source model, featureari is the feature of the natural examples extracted by the block and f featureado is the feature of the adversarial examples extracted by the block i. The general ResNet and DenseNet consists of four blocks. We optimize the classification loss and the low-level feature loss jointly during attacking:
marimize + 6). + feature), 1
(21)
where e is constant of the norm constraint and 7 is constant weight for the low-level feature loss. We call this method Enhancing the Early Blocks (EEB). As shown in Fig. [4] we show 7. Increasing the
the influence of the dyper-parameee low-level information) tends to improve ad adversarial transfcrability until it exceeds a threshold, e.g., 9=100. This may due y encourages the attack to utilize more transferable low-level information. However, the attack can be insufficient when just using low-level information and ignoring all high- level class-relevant information.
? (increasing the weight of
EEB can also improve the adversarial transferability of the deep neural networks without skip con- nection (VGG16, VGG19), As shown in Tab.
Table 6: Transfcrability against normal models: the success rates of black-box attacks crafted on VGG16 and VGG19 (DNNs without skip connection). The best results in bold.
Model
Attack
RN50
RN152
DNT21
DN200
SE154
IncV3
IncV4
IncRes
PGDD
30.76%
18.42%
32.14%
22.46%
22.22%
30.64%
36.80%
19.88%
VGG16
BEB(surs)
76.02%
59.14%
68.74%
57.82%
64.48%
60.74%
66.74%
41.86%
PGD
30.84%
18.40%
32.20%
22.46%
22.26%
17.36%
21.68%
11.74%
VGG19
BEB(s
70.26%
52.48%
70.48%
59.02%
57.98%
43.16%
51.66%
29.89%
20

---

Published as a conference paper at ICLR 2022
100
Target mecel:
50-
Target mecel:
lambdord
SE1GI Tambcuun)
60
40
10
100
101 10P
10"
Tarbda
Figure 14: The influence of the hyper-parameter for the low-level feature loss. The target models VGG19 and SE154. The source model is RN50. The horizontal lines show the success rate when using normal PGD attack (A=0) to generate adversarial examples, and the target models are respectively VGG19 (the purple line) and SE154 blue line). Optimizing the classification loss and the low-level feature loss jointly during attacking can improve the adversarial transferability.
J.2 ONLY ATTACK THE EARLY LAYERS
We insert a classifier after the Blocks3 (the penultimate block) of ResNet-50, then fine-tune this classifier, and attack this classifier generate adversarial examples. Such simple operation can compete with state-of-the-re methods (SGM). We call this method Dropping the Last Block (DLB).
Table Transferability against different models: the success rates black-bor attacks (untargeted) crafted on 50 and ResNet-152.
The best results are in bold.
Attack
VGG19
RNS0
DNN12
DN201
IncV4
PGD
45,96%
99.96%
57.44%
51.62%
45.7%
19.7%
18.96%
20.96%
14.30%
MI
64.86%
99.92%
73.22%
73.50%
64.33%
47.20%
39.08%
37.35%
25.26%
DI
75.06%
99.96%
81.65%
81.98%
74.80%
52.42%
42.58%
39.30%
27.12%
RN50
SGM
81.72%
99.92%
88 88.40%
83.56%
80.34%
57.30%
46.72%
43.83%
31.86%
(IB(uurs))
75.18%
99.94%
81.84%
80.42%
72.12%
46.06%
40.14%
41. 41.60%
28.08%
(lsB(ours)
92.34%
99.94%
92.05%
92.46%
88.26%
69.08%
64.35%
60.85%
47.77%
PGD
39.32%
69.72%
99.96%
48.44%
46.00%
22.20%
21.50%
22.58%
16.72%
MI
65.42%
83.40%
99.92%
77.60%
75.79%
53.00%
46.50%
43.32%
33.08%
DI 74.01%
88.18%
99.96%
79.46%
77.81%
57.49%
50.28%
47.16%
35.10%
RN152
SGM
80,40%
96.10%
99.92%
85.80%
82.76%
11.90%
53.16%
49. 49.24%
37.30%
(I.(() 73.94%
85.84%
99.94%
76.74%
64.30%
39.62%
43.72%
46.84%
31.56%
[l:BB(ous))
90.11%
98.46%
93.35%
87.26%
70.34%
63.96%
59.06%
47.30%
K IAA UTILIZE THE LOW-LEVEL INFORMATION
Centered kernel alignment (CKA) can reliably identify correspondences between representations in networks (Kornblith et al. 2019 Nguyen al., 2020). In order to show the difference between natural examples and adversarial examples, we show the CKA similarity between the features of natural examples and adversarial examples extracted by the same normal ResNet-50 (ImageNet). As shown in Fig. [5] (a), the adversarial examples generated by attacking the normal model using PGD show more difference from natural examples in features extracted by deep layers. Fig. [5] (b), the adversarial example generated by attacking the normal model using our IAA are different from the natural example in the shallow layers. For example, the shallow layers (layer 20 to layer 30), the CKA similarity between the adversarial examples generated by the normal model using PGD and the natural example is about 0.9, while the similarity of the adversarial example generated by IAA with the natural example is about 0.7. That is to say, IAA uses more levelevel information when generating adversarial examples and AAA enhances the influence the shallow layers on generating adversarial perturbation.
21

---

Published as a conference paper at ICLR 2022
1.0
0.8
0.4
0
0.0
10
20
30
40
(a) PGD Normal ResNet-50
LAA Normal ResNet-50
Figure 15: The CKA similarity between the features of natural image and adversarial images gen- erated by attacking different models. The horizontal axis represents the features of adversarial ex- amples extracted by different layers of ResNet-50, and the vertical axis represents the features of natural image extracted by different layers of the ResNet-50. (a) Adversarial examples generated by attacking normal ResNet-50 with PGD. (b) Adversarial examples are generated by attacking normal ResNet-50 with our IAA.
L OTHER CORRUPTION CASES FOR THE MOTIVATION EXAMPLE
As shown in Tab. we added other types of corruption like CIFAR-10-C and ImageNet-C (Hendrycks Dietterich 2019), including noise, blur, weather, etc. Consistent with the observations in Fig. the target models are more likely to misclassify corrupted LDD data. We hypothesize that these samples are the low-density region of the ground truth distribution where both source and target models are not well trained on. Moreover, we find that the adversarial counterparts of LDD have stronger adversarial transferability than that of HDD, which inspires us to match the direction of adversarial perturbation and the direction towards the low-density region to generate adversarial examples with high transferability.
Table 8: The attack success rate when applying different types of corruption on the LDD and HDD against different target models (VGG19, RN50, DN121). The best results are in bold.
Target
Caussian
Shet Impulse
Delocus blur
Glass
Zoom blur
Fog
Saturate
VGG19
HDD
6.2%
6.5%
6.0%
8.0%
7.2%
14.3%
6.3%
6.2%
L.D
48.1%
46.5%
47.5%
34.4%
31.5%
45.9%
37.9%
37.6%
RN50
4.1%
1.8%
6.0%%
6.0%
LDD
37.6%
36.1%
26.2%
24.8%
23.4%
33.8%
39.4%
34,9%
HDD
13%
1.7%
1.1%
3.8%
3.3%
7.6%
3.1%
1.7%
DN121
LDD
29.4%
27.2%
23.5%
21.7%
27.7%
24.5%
M COMPUTATION COST
Our proposed method includes two steps. The first step is adjusting the pre-trained source model to match the adversarial attack with the ground truth density decreasing direction by optimizing AAI, and the second step is generating adversarial samples by the modified model similar to the normal PGD attack.
Optimizing AAI to get strong source model by Bayesian optimization the pre-trained model needs some extra computation. As shown in Eq. 4 the computation cost for AAI can be reduced to (dd) from (dd²) by using ideas from sliced score matching (Song et al. 2020), where d is the dimension of data. The computation cost of AAI optimization is 10 hours on Tesla-V100 using ResNet-50.
However, the cost of generating adversarial examples is more important than the cost of adjusting the source model in practice. Once get the modified model, generating different adversarial
22

---

Published as a conference paper at ICLR 2022
Table 9: Top-1/Top- accuracy different defense models when predicting the black-box adver- sarial examples crafted by different methods on RN50. We consider two strengths of adversarial perturbation (c 16 and 24).
The best results are in bold.
Atiack
NRP
SIN
SIN-IN
AugMix 51.04788.00
c-005
0
c
c1.00
PGD
L32726648
61.66787.4
9.66756.60
66.06794.77
42/97.9
94.247 047799..8
95.82/99.84
SGM
0.20/15.86
41.50//7.06
1.6/31.62
15.28/61.20
22.68/71.76
42.76/84.38
82.32/97.45
90,82/98.76
TAA
0.00/0002
16.18/27.64
0.00/0.04
0.84/2.44
1,82/4.88
19.96
63.10/80.15
83.42/94.36
PGD
3.40735.78
65.56789.55 54
15.08 34
60,36/91.90
75.62797.00 D0
87.26798.88 80.80
96. W6.36/90 10/999.0 (0.0
97.12799.50
16
SGM
1.54/28.98
50.28/82.22
3.82/43.64
27.42/73.00
41.88/84.66 84.66
62, 02.98/93.10
90.66/99 666//99.200 24
94.60/99 999499
IAA
0,48/0,98
27.76/44.00
0.18/0.54
5.96/10.55 10.98
14.08/26.0
39.56/89.52
84.8/95.1
92.98/ /9.32
examples by the modified model shares the same computation with normal PGD adversarial attacks. That is say, it is very efficient to use our modified model to generate adversarial examples.
N ATTACK OTHER DEFENSE METHODS
In Tab. 4 we evaluate that our IAA can fool the robust model by ensemble adversarial training Tramer et al., 2018). Moreover, we show that the adversarial examples generated by IAA can also effectively penetrate many other defense methods in Tab. 9 We further add neural representation purifier (NRP) (Naseer et al. 2020) which is a state-of of-te-of-the-arr input processing defense method, and some other robust training methods, including training with Aug (Hendryeks et al., 2020), Styled ImageNet (SIN) (Geirhos et al. (2019), the mixture of Styled and natural ImageNet (SIN-IN), and adversarial examples (Salman et al. (020) as defense models. In Tab, g we show the top-1 accuracy and top-5 accuracy (ACCI/ACC5) on different defense models when predicting the black- box adversarial examples crafted by different methods. We use the ResNet-50 as the source model. IAA surpasses other attacks in different cases. Our IAA drives the examples to the low-density region by matching the adversarial attack and intrinsic attack. Thus the target models can hardly give correct prediction on IAA examples, and both the top-1 accuracy and the top-5 accuracy are lower than other methods.
O FURTHER DISCUSSION ON DIRECTLY OPTIMIZING THE TRANSFERABILITY.
In paper, used the method of optimizing AAI to choose hyperparameters for two reasons. On the one hand, we hope to verify the effectiveness of the metric we proposed. On the other hand, think that using the transferability for a certain target model to optimize may overfit the target model. Interestingly. 2ig.2 and Fig.3 show that the attack success rate among all the target models seems like aligns well under different structural hyperparameters. Maybe it won't 't be so over-fitted to the target model that it is difficult to select the proper hyperparameters. We conduct experiments on the source model ResNet-50. We use four methods choose the proper hyperparameters. including using the AAI as the optimization objective ("optimize- AAI"), using the transferability against the target model VGG19 as the optimization objective ("optimize-vgg the target model SE154 as the optimization objective ("optimize-senet 54""), and using transferabil- ity against the ensemble of VGG19 and SE154 as the optimization objective ("optimize-ensemble"" The following table represents the top-1 accuracy and top-5 accuracy (ACCI/ACC5) when predict- ing the adversarial examples crafted by different methods. Lower accuracy indicates better adversar- ial transferability. We found that directly using the transferability as the optimization objective can also find proper structural hyperparameters and outperforms the existing methods PGD and SGM. The ensemble-bar based strategy can slightly improve the adversarial transferability. The computation cost of directly optimizing transferability is half that of optimizing AAI.
using transferability against
Table 10: The attack success rate when using different optimizing methods to choose structural hyperparameters. The best results are in bold.
Vgg19
DN121
DN201
SE154
IncV3
IncV4
PGD
47.00/83.5 52
44.38/84.16 16
46 46.44/86.92
50.44/93.00
79. 79.14/96.66
78,04/5,66 688
SGM
17.28/58.68
16.44/63.52
19.66/68.46
38.7/75.62
46.28/84.28
50.17/85.95
optimize-AAl
3.60/5.67
2.54/4.37
3.36/6.60
15.26/18.68
23.72/29.36
28.62/33.72
optimize-senetl14 54
3.96/6.20
3,14/4.50
4.04/6.34
14.60/18.98
11/14/28.86
24.72/30.32
optimize-vgg19
3.02/4.70
2.68/3.94
3.78/5.56
16.64/20.36
23.16/31.36
28.28/32.45
optimize-ensemble
3.20/4.80
2.42/3.06
3.26/4.70
14.86/17.00
19.56/26.86
23.30/28.14
23


Reviews:
[{"review_id": "wp2rS0zBJqO", "reviewer": "Reviewer_f4GM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The proposed method is well-supported by theoretical analysis.\n- The experiment is thorough, different scenarios and multiple baseline methods have been covered.\n- The results is good. The proposed method outperforms other compared method.\n\nWeaknesses:\n- The process of generating adversarial examples includes a bayesian optimization and receptively evaluating the second derivative of the neural network, which could result in high computational cost. What is the computation time of generating adversarial examples? I expect it will slower than normal adversarial attacks but the authors should always report it (including the network structure, hardware information) so that other people could have more understanding on the overall usability of this method.\n- I feel there is a disconnection between the motivation and the actual proposed method. In the introduction, the authors show that small Gaussian noise could fool low density data. However, I did not see many connection between this observation and the proposed method. The only transition is that \"The most efficient direction towards the low-density region is...\". I think the author should elaborate more on this, since this statement is not very obvious and rigorous. Also, what is the formal definition of LDD and HDD?\n\nMinor Comments:\n- Since the proposed method can also be used for adversarial attack, I wonder what is the attack success rate of the proposed method? Is it also better than other attacks?\n- When citing a paper, one should cite its archival version first instead of non-archival version (e.g., Arxiv, CoRR). For example, citation for (Tramèr et al., 2020) should be \"Tramèr, Florian, et al. \"Ensemble Adversarial Training: Attacks and Defenses.\" International Conference on Learning Representations. 2018.\" instead of \"... CoRR, abs/1705.07204, 2020.\"\n", "summary_of_the_paper": "This paper presents a new adversarial attack method which could generate adversarial examples with higher transferability. The proposed method is based on the observation that low-density region of the training data is not well trained. To utilize this, the authors try to align the adversarial direction with the direction to decrease the ground truth density. The proposed method is theoretically support. Their experiments show that the proposed method outperforms multiple adversarial attack method in almost all evaluated scenario. I believe this could be a useful attack method for generating transferable adversarial examples, and providing a strong counterpart for future research on adversarial defense.", "summary_of_the_review": "Although there are some weaknesses of the paper, based on their experiments, I still believe the proposed method could be useful for generating strong adversarial examples and open the door for future research on defense. I also appreciate their theoretical analysis, although I haven't check the correctness. In general, I am feeling positive on this paper and would like to recommend for acceptance.\n\n==============================\nPost rebuttal:\n\nI appreciate the author's comments on my questions. My score will keep the same since it's already reflect my recommendation for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SHD8dTBk4xr", "reviewer": "Reviewer_cPB9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- Important topic\n\n- Somehow intuitive and interesting idea\n\n- Sound and feasible solution\n\n- Promising results\n\nWeakness:\n\n- The assumption is intuitive but limited evidence is provided. In particular, the authors only show cases of Gaussian noises, it is unclear to what extent the assumption could be generalized. It would be important to include other noises as well, e.g., corruption cases like CIFAR-10-C.\n\n- Besides robustly trained models, it is unclear how strong the proposed attack could penetrate SOTA defense methods. Please provide more evidence on this.\n\n- It is unclear what is the criterion to decide whether a layer is an early layer or late layer, please justify.", "summary_of_the_paper": "This paper proposes Intrinsic Adversarial Attack (IAA), a transfer attack method based by jointly matching data distribution. The key assumption of this paper is that the DNN might not be well trained on low-density regions (LDD). Therefore, taking data distribution into consideration during attack could potentially improve the transferability. Empirical evaluation on different model architectures, normal models, robustly trained models, and ensemble-based attack context, demonstrates the advantage of IAA.", "summary_of_the_review": "Overall, this paper is well written and works on an important problem. The assumption is simple and intuitive, the proposed method is also feasible and sound.\n\nEven though, there are still quite a few concerns. I am on the borderline of this paper and could be easily flipped to the other side based on the authors' response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D6iMPwZG_Z2", "reviewer": "Reviewer_wLmf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. The observation on the relationship between low-density data and the high transferability of both random noise and adversarial noise is interesting, likely to inspire future works.\n2. Provide theortical analysis on the AAI metric, provide a practical way of computing its value, and emperically shows its high correlation with the attack transferability.\n3. Extensive experiments showing the proposed method achieving significantly higher transfer attack performance than SOTA methods, showing the effectiveness of the proposed method. \n\n## Weaknesses\nMy major concern on this paper is on the practical significance of the proposed AAI metric and IAA algorithm.\n\n1. The necessity of utilizing the AAI metric to generate highly transferable attack is not justified. The paper introduces model hyperparameters $\\beta$ and $\\lambda$ utilizing the techniques of smoothing activation gradient and use early layer information, which are motivated by the previous works cited in Sec 3.2.1 and 3.2.2, and are not directly related to the AAI metric. As for the Baysian optimization process, since it's a blackbox optimizer assuming no knowledge of the relationship between the parameter and optimization objective, why not directly using the attack transferability as the optimization objective? An ablation study on this is needed to discuss why AAI is needed for this optimization.\n2. The cost of evaluating the AAI objective is not discussed. From the formulation in Equation (4) seems like the computation of AAI requires a second-order derivation with respect to x and a sampling process of Gaussian vector v. How is the computation cost of giving an accurate enough estimation of the AAI value comparing to generating an adversarial example with the model?\n\nBesides, there are some other issues in this paper\n\n3. Some steps in the proof of Theorem 1 are hard to understand. For example, how is the $\\log$ function got removed from the $\\frac{\\nabla_x \\log p(y|x)}{|| \\nabla_x \\log p(y|x) ||_2}$ in the first step of the derivation? As the author mentions the proof is similar to the ones from previous works, to make the paper self-contained I would suggest having some more detailed derivation or explaination in Appendix E so that the proof can be understood without referring to your cited papers.\n4. Citation should be provided for the baseline models mentioned in the experiment result tables.", "summary_of_the_paper": "This paper identifies that adversarial examples in the low-density region of the groud truth distribution have much stronger transferability. This observation leads to the AAI metric which evaluate the alignment of the model’s adversarial attack with intrinsic attack direction. The paper further identifies a set of model hyperparameters that can influence the AAI metric, and find the optimal hyperparameter choices to maximize AAI and generate more transferable adversarial examples.", "summary_of_the_review": "This paper provides interesting theortical insights on how to find highly transferable adversarial examples. However, from the practical side, the proposed training method seems to be inspired from other previous works rather than the observed insight, and whether the discovered AAI metric is practically useful for the optimization process is at doubt. Thus I would suggest a weak reject for now given the potentially low practical significance of the method.\n\n## After rebuttal\n\nAfter discussing with the author I have a better understanding on the significance of this work. The analysis of AAI motivates the idea of modifying structural hyperparameters of the source model to improve attack transferability, and AAI serves as a good indicator of the transferability towards all model. Although further experiments show it may not be the best objective for choosing the structural hyperparameters, I think it serves as a good theortical strating point for futher analysis of adversarial transferability. Thus I would like to suggest acceptance.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work studied an important issue, i.e., adversarial transferability, in adversarial examples. It provides a novel perspective that samples in  the low-density region of the ground truth distribution where models are not well trained have stronger transferability across different models. Based on that, it proposed a metric called Alignment between its Adversarial attack and the Intrinsic attack (AAI) to indicate transferability. Inspired by the connection between AAI and transferability, this work further proposed to replace the regular ReLU activation with some smooth activation functions, to enhance the transferability. 

Most reviewers appreciate that the observation is interesting, and the theoretical analysis and the proposed method are intuitive. The reviewers posed some important comments on experiments, and the relationship between the proposed method and the proposed metric. The authors provided satisfied responses to most of these concerns. Although there is one remaining concern that AAI may be not the best metric to choose the structural hyper-parameters, the reviewer still thought it is a good theoretical starting point to further analyze the adversarial transferability. 

After reading the submission, reviewers' comments and the discussions between reviewers and authors, I believe that this work has provided a valuable perspective, a reasonable theoretical analysis and an effective solution for adversarial transferability. It could inspire further studies on adversarial transferability.
Decision: Accept (Poster)