Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbation
Authors: Zeyu Qin
Yanbo Fan
Yi Liu
Li Shen
Yong Zhang
Jue Wang
Baoyuan Wu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Adversarial Transferability
Black-Box Attacks
Adversarial Examples

Abstract: Deep neural networks (DNNs) have been shown to be vulnerable to adversarial examples, which can produce erroneous predictions by injecting imperceptible perturbations. In this work, we study the transferability of adversarial examples, which is significant due to its threat to real-world applications where model architecture or parameters are usually unknown. Many existing works reveal that the adversarial examples are likely to overfit the surrogate model that they are generated from, limiting its transfer attack performance against different target models. To mitigate the overfitting of the surrogate model, we propose a novel attack method, dubbed reverse adversarial perturbation (RAP). Specifically, instead of minimizing the loss of a single adversarial point, we advocate seeking adversarial example located at a region with unified low loss value, by injecting the worst-case perturbation (the reverse adversarial perturbation) for each step of the optimization procedure. The adversarial attack with RAP is formulated as a min-max bi-level optimization problem.  By integrating RAP into the iterative process for attacks, our method can find more stable adversarial examples which are less sensitive to the changes of decision boundary, mitigating the overfitting of the surrogate model.  Comprehensive experimental comparisons demonstrate that RAP can significantly boost adversarial transferability. Furthermore, RAP can be naturally combined with many existing black-box attack techniques, to further boost the transferability. When attacking a real-world image recognition system, Google Cloud Vision API, we obtain 22% performance improvement of targeted attacks over the compared method. Our codes are available at https://github.com/SCLBD/Transfer_attack_RAP.

Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbation
Zeyu Qin', Yanbo Fan Yi Liu', Li Shen', Yong Zhang', Jue Wang', Baoyuan Wul
School of Data Science, Shenzhen Research Institute of Big Data,
The Chinese University of Hong Kong. Shenzhen
Tencent Al Lab
IJD Explore Academy
fzeyu6181136, fanyanbo0124, yiliuuh2000)@@mail... (mathshenli, zhangyong201303, arphid)8gmail com vubaoyuan@cuhk, edu.cn cn
Abstract
Deep neural networks (DNNs) have been shown to be vulnerable to adversarial examples, which can produce erroneous predictions by injecting imperceptible per- turbations. this work, we study the transferability of adversarial examples, which is significant due to its threat to real-world applications where model architecture parameters are usually unknown. Many existing works reveal that the adversarial examples are likely to overfit the surrogate model that they generated from, limiting its transfer attack performance against different target models. To mitigate the overfitting of the surrogate model, propose novel attack method, dubbed reverse adversarial perturbation (RAP). Specifically, instead minimizing the loss of single adversarial point, advocate seeking adversarial example located at a region with unified low loss value, by injecting the worst-case perturbation (i.e., the reverse adversarial perturbation) for each step of the optimization procedure. The adversarial attack with RAP is formulated as min-max bi-level optimization problem. By integrating RAP into the iterative process for attacks, our method can find more stable adversarial examples which are less sensitive to the changes of de- cision boundary. mitigating the overfitting of the surrogate model. Comprehensive experimental comparisons demonstrate that RAP can significantly boost adversarial transferability. Furthermore, RAP can be naturally combined with many existing black-box attack techniques, to further boost the transferability. When attacking real-world image recognition system, i.e., Google Cloud Vision API, we obtain 22% performance improvement of targeted attacks over the compared method. Our codes are available at: https: // thub. Com/SCLBD/Tramtt Transfer. wwwpsefrrattttc_Hee
Introduction
Deep neural networks (DNNs) have been successfully applicd in many safety-critical tasks, such autonomous driving, face recognition and verification, etc. However, it has been shown that DNN models are vulnerable to adversarial examples [9, 12, 27, 32, 35, 44, 45, 50], which are indistinguishable from natural examples but make model produce erroneous predictions. For real-world applications, the DNN models are often hidden from users. Therefore, the attackers need to generate the adversarial examples under black-box setting where they do not know any information of the target model [2, 3, 18, 32]. For black-box setting, the adversarial transferability matters since
"Equal contribution. 'Corresponding author.
This work is done when Zeyu Qin is research intern Tencent Lab.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

it can allow the attackers to attack target models by using adversarial examples generated on the surrogate models. Therefore, learning how to generate adversarial examples with high transferability has gained more attentions in the literature [5, 10, 14, 23, 26, 38, 48].
Under white-box setting where the complete information of the attacked model (e.g., architecture and parameters) is available, the gradient-based attacks such as PGD [27] have demonstrated good attack performance. However, they often exhibit the poor transferiability [5.48], 48], i.e., the adversarial example æadv generated from the surrogate model MM performs poorly against different target models M The previous works attribute that to the overfitting of adversarial examples to the surrogate models [5, 24, 48]. Figure (b) gives an illustration. The PGD attack aims to find an adversarial point and with minimal attack loss, while doesn't It consider the attack loss of the neighborhood regions round x Due to the highly non-convex of deep models, when xndd locates at a sharply local minimum, a slight change on model parameters of .MS could cause a large increase of the attack loss, making ap9d fail to attack the perturbed model.
Many techniques have been proposed to mitigate the overfitting and improve the transferability, including input transformation [6, 48], gradient calibration [14], feature-level attacks [17], and generative models [30], etc. However, there still exists large gap of attack performance between the transfer setting and the ideal white-box setting, especially for targeted attack, requiring more efforts for boosting the transferability.
In this work, we propose a novel attack method called reverse adversarial perturbation (RAP) to alleviate the overfitting of the surrogate model and boost the transferability of adversarial examples. We encourage that ad is not only of low attack loss but also locates at a local flat region, i.e., the points within the local neighborhood region around x ad should also be of low loss values. Figure 1 (b) illustrates the difference between the sharp local minimum and flat local minimum. When the model parameter of M has some slight changes, the variation of the attack loss w.r.f the flat local minimum is less than that of the sharp one. Therefore, the flat local minimum is less sensitive to the changes of decision boundary. To achieve this goal, we formulate a min-max bi-level optimization problem. The inner maximization aims to find the worst- perturbation (i.e., that with the largest attack loss, and this is why we call it reverse adversarial perturbation) within the local region around the current adversarial example, which can be solved by the projected gradient ascent algorithm. Then, the outer minimization will update the adversarial example to find new point added with the provided reverse perturbation that leads to lower attack loss. Figure 1 (a) provides an illustration of the optimization process. For 1-th iteration and x;' RAP first finds the point x.' + n rop with max attack loss within the neighborhood region of x². Then it updates x' with the gradient calculated by minimizing the attack loss w.r.f x' + n rap, Compared to directly adopting the gradient at x", RAP could help escape from the sharp local minimum and pursue relatively flat local minimum. Besides, we design a late-start variant of RAP (RAP-LS) to further boost the attack effectiveness and efficiency, which doesn't 't insert the reverse perturbation into the optimization procedure in the early stage. Moreover, from the technical perspective, since the proposed RAP method only introduces one specially designed perturbation onto adversarial attacks, one notable advantage of RAP is that it can be naturally combined with many existing black-box attack techniques to further boost the transferability. For example, when combined with different input transformations (e.g., the random resizing and padding in Diverse Input [48]), our RAP method consistently outperforms the counterparts by a clear margin.
Our main contributions are three-fold: 1) Based on a novel perspective, the flatness of loss landscape for adversarial examples, we propose novel adversarial attack method RAP that encourages both the adversarial example and its neighborhood region be of low loss value; 2) we present a vigorous experimental study and show that RAP can significantly boost the adversarial transferability on both untargeted and targeted attacks for various networks also containing defense models; 3) we demonstrate that RAP can be easily combined with existing transfer attack techniques and outperforms the ifateof--the-al performance by a large margin.
2 Related Work
The black-box attacks can be categorized into two categories: 1) query-bused attacks that conduct the attack based on the feedback of iterative queries to target models, and 2) transfer attacks that use the adversarial examples generated some surrogate models to attack the target models. In this work.

---

we focus on the transfer attacks. For surrogate models, existing attack algorithms such as FGSM [12] and I-FGSM [21] could achieve good attack performance. However, they often overfit the surrogate models and thus exhibit poor transferability. Recently, many works have been proposed to generate more transferable adversarial examples [5, 6, 13, 14, 17, 19, 20, 24, 39, 40, 41, 43, 48], which we
briefly summarize as below.
Data augmentation, which has been shown be effective in improving model generalization, has also been studied to boost the adversarial transferability, such as randomly resizing and padding [48], randomly scaling [24], and adversarial mixup [41]. In addition, the work of Dong et al. [6] uses set of translated images to compute gradient and get the better performance against defense models. Expectation of Transformation (EOT) method [1] synthesizes adversarial examples over a chosen distribution of transformations to enhance its adversarial transferability. Gradient modification: Instead of the I-FGSM, the work of Dong et al. [5] integrates momentum into the updating strategy. And Lin et al. [24] uses the Nesterov accelerated gradient to boost the transferability. The work of Wang and He [40] aims to find more stable gradient direction by tuning the variance of each gradient step. There are also some model-specific designs to boost the adversarial transferability. For example, Wu et al. [43] found that the gradient of skip connections is more crucial to generate more transferable attacks. The work of Guo et al. [14] proposed LinBP to utilize more gradient of skip connections during the back-pre procckppppggtiin. However, these methods tend to be specific to particular model architecture, such skip connection, and it is nontrivial to extend the findings to other architectures or modules. Intermediate feature attack: Meanwhile, Huang et al. [17], Inkawhich et al. [19, 20] proposed to exploit feature space constraints to generate more transferable attacks. Yet they need to identity the best performing intermediate layers or train one-vs-all binary classifies for all attacked classes. Recently, Zhao et al. [49] find iterative attacks with much more iterations and logit loss can achieve relatively high targeted transferabilityy and exceed the feature-based attacks. Generative models: In addition, there have been some methods utilizing the generative models to generate the adversarial perturbations [28, 30, 31J. For example, the work of Naseer et al. [30] proposed to train a generative model to match the distributions of source and target class, so as to increase the targeted transferability. However, the learning of the perturbation generator is nontrivial, especially on large-scale datasets.
Input transformation:
In summary, the current performance of transfer attacks is still unsatisfactory, especially for targeted attacks. In this work, we study adversarial transferability from the prespective of the flatness of adversarial examples. We find that adversarial examples located at flat local minimum will be more transferable than those at sharp local minimum and propose an novel algorithm to find adversarial example that locates at flat local minimum.
3 Methodology
3.1 Preliminaries of Transfer Adversarial Attack
Given an benign sample (x, y) E (X, the procedure of transfer adversarial attack is firstly constructing the adversarial example addd within the neighborhood region (x()) {a' : || ælIp I <) by attacking the white-box surrogate model (x; 0) XX + Y, then transferring zads to directly attack the black-box target model Mz (x; Φ): : X + ). The attack goal is to mislead the target model, i.e., M Φ) * y (untargeted attack), or M Ma Φ) = y/ (targeted attack) with Ve E ) indicting the target label. Taking the target attack as example, the general formulation of many existing transfer attack methods can be written as follows:
min gadveg, (a) (a)
L(M (((a)d)))))) 0).
The loss function L is often set as the cross entropy (CE) loss [48] or the logit loss [49], which will be specified in later experiments. Besides, the formulation of untargeted attack can be casily obtained by replacing the loss function L and yı by - and y, respectively.
Since M is white-box, if G()) is set as the identity function, then any off-the-shelf white-box box adversarial attack method can be adopted to solve Problem (1), such as I-FSGM [21], MI-FGSM [5], etc. Meanwhile, existing works have designed different G(-) functions and developed the corresponding optimization algorithms, to boost the adversarial transferability between surrogate and target models. For example, G(-) is specified as random resizing and padding (DI) [48], translation transformation (TI) [6], scale transformation (SI) [24], and adversarial mixup (Admix) [41].
3

---

sign( staphola m1
(MM(0)0)0 L(M (x:
L
(xM 0),
sign(Fy..CC +
minimum
Sharp misimam
6700
(a)
Figure I: These two plots schematic diagrams in 1D space. The x-axis means the value of input 2. The y-axis means the value of attack loss function L. (a) Illustration of our attack method and the original PGD attack. (b) Illustration of attack loss landscape of MS and MM .MS" denotes a slight change on the model parameters of M The blue and yellow dots correspond to attacks located at different local minima on M respectively. The gray and red points are their counterparts M
3.2 Reverse Adversarial Perturbation
As discussed above, although having good performance for the white-box setting, the adversarial examples generated from M exist poor adversarial transferability on MM, especially for targeted attacks. The previous works attribute this issue to the overfitting of adversarial attack to M [4, 5, 6, 38, 46]. As shown in Figure (b), when and locates at a sharp local minimum, it is not stable and is sensitive to changes of SM.. When having some changes on model parameters, agdd could results in high attack loss against M and lead to a failure attack.
To mitigating the overfitting M we advocate find wdvv located flat local region. That means we encourage that not only addv itself has low loss value, but also the points in the vicinity of adds have similarly low loss values.
To this end, we propose to minimize the maximal loss value within a local neighborhood region around the adversarial example ado The maximal loss is implemented by perturbing adde to maximize the attack loss, named Reverse Adversarial Perturbation (RAP). By inserting the RAP into the formulation (1), aim to solve the following problem,
min geade (a) (( + 0))
(2)
where
Tap
max L(M* , 3)), 1I
(3)
with TP indicating the RAP, and en defining its search region. The above formulations Equation (2) and Equation (3) correspond to the targeted attack, and the corresponding untargeted formulations can be easily obtained by replacing the loss function L and yı by -L and y. respectively.
It is a min-max bi-level optimization problem [25]. and can be solved by iteratively optimizing the inner maximization and the outer minimization problem. Specifically, in each iteration, given ado the inner maximization w.r.t. n rap solved by the projected gradient ascent algorithm:
n + n rap + ax sign Varop ((M + n0 ())
(4)
The above update is conducted by T steps, and w.r.f. add can be solved by any off-the-shelf algorithm that is developed for solving Equation (1). For example, it can be undated by one step projected gradient descent, as follows:
f.. Then, given ray the outer minimization
adv Clips, (w) [ar a sign(V gendo (G (g(
(2P 0), v.))],
(5)
with Clips (x)) (a) clipping a into the neighborhood region B. (a:). The overall optimization procedure summarized in Algorithm 1. Moreover, since the optimization w.r.f. ande can be implemented by any off-the-shelf algorithm for solving Problem Equation (1), one notable advantage of the proposed RAP is that it can be naturally combined with any one of them, such as the input transformation methods [6, 24, 41, 48].

---

Baselinc
Den-121 (M])
Den-121 (Admix)
VGG-16 (MI)
VGG-16 (Adrii))
Targeted Attack Success Date (85)
30
70
20-
15
155
43
30
30
10-
20
100
200 Iteration
300
100
200
300
Iteretion
Iteretion
Iteration
Figure 2: Targeted attack success rate (%) on Dense-121 and VGG-16. We take the Res-50 as the surrogate model and take MI and Admix as baseline methods.
Algorithm/ Reverse Adversarial Perturbation (RAP) Algorithm
Input: Surrogate model MM,, benign data (æ, v), target label V/, loss function L, transforma- tion G. the global iteration number K, the late-start iteration number KLs of RAP, as well as hyper-parameters in optimization (specified in later experiments) Output: the adversarial example add
1: Initialize x adv <
2: for k = 1..... K do
3: if h: > Kis then
4: Initialize rap
5: f
T do Update n rp using Equation (4)
6:
7:
Update wadv using Equation (5)
Late-Start (LS) Variant of RAP. In our preliminary experiments, we find that RAP requires more iterations to converge and the performance is slightly lower during the initial iterations, compared to its baseline attack methods. and adopt ResNet-50 the surrogate model. We take the evaluation
shown Figure 2, we combine MI [5] and Admix [41] with RAP,
1000 images from ImageNet (see Scc.4.1). 1). is observed that the method with RAP (see the orange curves) quickly surpasses its baseline method (see the blue curves) and finally achieves much higher success rate with more iterations, which verify the effect of RAP on enhancing the adversarial transferability, However, is also observed that the performance of RAP is slightly lower than its baseline method in the early stage. The possible reason is that the early-stage attack is of very weak attack performance to the surrogate model. In this case, it may be waste to pursue better transferable attacks by solving the min-max problem. better strategy may be only solving the minimization problem Equation (1) in the early stage quickly achieve the region of relatively high adversarial attack performance, then starting RAP to further enhance the attack performance and transferability simultaneously. This strategy denoted RAP with late-start (RAP-LS), whose effect preliminarily supported by the results shown in Figure (see the green curve) and will be evaluated extensively in later experiments.
3.3 A Closer Look at RAP
To verify whether RAP can help us find add located the local flat region or not, we use ResNet- 50 as surrogate model and conduct the untargeted attacks. We visualize the loss landscape around ado on M by plotting the loss variations when we move ado along a random direction with different magnitudes The details of the calculation are provided in Appendix. Figure 3 plots the visualizations. We take I-FGSM [21] (denoted as I), MI [5], DI [48], and MI-TI-DI (MTDI) as baselines attacks and combined them with RAP. We can see that comparing to the baselines, RAP could help find adv located at the flat region.
5

---

Beseline
Beseline+RAP
MILD
.40
40-
Foomess
02
80-
60-
20
20
20-
-20
Magnitude
Magnitude
Magnitude
Magnitude
Figure 3: The flatness visualization of untargeted adversarial examples on M The implementation details are shown in Section of Appendix.
Experiments
4.1 Experimental Settings
Dataset and Evaluated Models. We conduct the evaluation on the ImageNet-compatible dataset comprised of 1,000 images. For the surrogate models, consider the four widely used network architectures: Inception-v3 (Ine-v3) [36], ResNet-50 (Res-50) [15], DenseNet-121 (Dense- 121) [16], and VGG-16bn (VGG-16) [34]. For target models, apart from the above models, also utilize more diverse architectures: Inception-Resee
Lnc-Res-v2. [37], NASNet-Large (NASNet-L) [51]. and ViT-Base/ 16 (ViT-B/16) [8]. For defense models, adopt the two widely used ensemble adversarial training (AT) models: adv-Inc-v3 (Inc-v3ade) and Sis-adv--In-Res--e (IncRes-v2ens) [38]. Besides, also test multi-step AT model [33], imput transformation defense [46], feature denoising [47], and purification defense (NRP) [29].
Compared Methods. We adopt I-FGSM [21] (denoted D), MI [5], TI [6], DI [48], SI [24], Admix [41], VT [40], EMI [42], ILA [17], LinBP [14], Ghost Net [22], and the generative targeted attack method TTP [30]. We also consider the combination of baseline methods, including MI- TI-DI (MTDI), MI-TI-DI-SI (MTDSI), and MI-TI-DI-Admix (MTDAI). Besides, Expectation of Transformation (EOT) method [1] is also comparable baseline method. We also conduct the comparison of RAP and EOT.
Implementation Details. For untargeted attack, we adopt the Cross Entropy (CE) loss. For targeted attack, apart from CE, we also experiment with the logit loss, where Zhao al. [49] shows behaves better for targeted attack. The adversarial perturbation is restricted by Loo 16/255. The step size a set as 2/255 and number of iteration is set as 400 for all attacks. In the following, we mainly show the results at K 400 and the results at different value of K are shown in Appendix. For RAP, we set Kr, as 100 and as, as 2/255. We set en as 12/255 for and in untargeted attack and 16/255 for other attacks in all other settings. The computational cost is shown in Section B of Appendix.
Extra Experiments in Appendix. Due to the space limitation, we put extra experiment results in Appendix. The comparisons of RAP and EOT, VT, EMI, and Ghost Net methods are shown in Section C in Appendix. The evaluation of RAP on stronger defense model, multi-step AT models, NRP, and feature denoising, is shown in Section D. The evaluation of ensemble-mode attacks on these diverse network architectures is given in Section E.2.
4.2 The Evaluation of Untargeted Attacks
Baseline Methods. We first evaluate the performance RAP and RAP-LS with different baseline attacks, including I, MI, DI, TI, SI, and Admix. The results are shown in Table 1. For instance, the 'MI/ +RAP/ +RAP-LS' denotes the methods of baseline MI, MI+RAP, and MI+RAP-LS, respectively. RAP achieves the significant improvements for all methods each target model. For average attack success rate of all target models, RAP outperforms the and MI by 9.6% and 16.3%, respectively.
Publicly available from cleverhans_vv v3. 1..0 htw.a///e..aeeeaaaeerraasaaaaaoe

---

Table 1: The untargeted attack success rate (%) of baseline attacks with RAP. The results with CE loss are reported. The best results are bold and the second best results are underlined.
Attack
Dense- 121
RosNect50 VGG-16
Inc-v3
Res-50
DenseNet-121 VGG--6
Inc-v3
RF-RAP/ / GRAP-LS
79.2/ 91.591./7
99.9
15.1/91110
92.9
34.6 34.6/57.01
57.2
87.4 94.2 99,3
85.1 /1.1/91.7/ 192.8
46.5/5022
1122/663 661.1
MEI HRAPT SRAPLS
85.81 99.0/ 96.1
82.4/93.9
94.5
50.3/ 75.9/
90.3/ 9976 /9.9
87.5 99607 97.6
59.31
80.4 /8.8
TE/ SRAP RAPPLS
99.1 195.1
81.0/93.11
93.3
45.5/66.1
89.6
87.0/ 92.1
54.27 66.7
DU RRPP ARAPLS
99.87 99.6 99.7
99.6
57.2 92.9
98.2/ 99.6
98.1 999.9
67.67
ST ET/RRAP/ +RAP-LS
94.9/ 99.97 99.7
88.6/9577
7: 97.2
63.9 79.7 84.4
95.1
99.1/95699999 998.8
91.9/ 05.0/99.5
71.67 83.3
Admix / +RAP/ +RAP-S
97.9/9065
779/996/999 /9.9
95.8
99.77 99.0
77.7
87.4
00/4/9222 992.6
97.0/
00.0//90///0 /99.2
95.67 97.7 /98.6
82.07 89.8 933.
Attack
Res-50
VGG-16
Dense-12)
Inc-v3
Res 50
Inc-43
Dense: 121
VGG-16
17-RAP/ ARAPLSS
$3.2/530
54.2
49.1
/50////51 51.4
24.7/
51.5/ 66..0
60.8 60LD
65.0)
MI/ +RAP/ +RAP-LS
62.5/ 2/ 776.
60.57 /7.00 73.9
30.0/
42.7 /42.2
62.0 85.8 884.8
56.71 84.6 /8.6
/ 84.9 /4.66
TRA-RAP/ -RAP-LS
62.87 64.8 66.8
55.9 /637 (62
20. 10.1/5
36.22/37.1
49. 3/ 63.4 61.6
49.4163.4/ 63.8
58. 1/ 66 69.5
DU/ +RAP/+RAP-LS
72.2/ 56.0 58.8
68.8 $5.01 $7.4
29.9/ 16.6/ /516
68.4 $1.7 $1.88
71.91 /
76.1 85.2 /56.4
SU/ SU/+RAP/ +RAP-LS
80.0/
92.7 /94.7
22.1/9968:95 95.7
45.8/ 74.0/74.7
66.2 /6.88 72.8
65,9/ 74.9/ 77.3
66.0 69.2 7330
Admix/ HRAP/ +RAPLLS
87.57 94.0/ 96.8
88.2/96.4
75.91 /5499
85.7 /87.4
74.5/ 8335
Table 2: The untargeted attack success rate (%) of combinational methods with RAP. The results with CE loss are reported. The best results are bold and the second best results are underlined.
ResNet-50 VGG
NenseNet-121 VGG-16
Dense
Inc.v3
DRAD/+RAP// +RAP-LS
9 100 100
99.8/ 100 59.9
8
95.0/ 96.9
99.4/ /9.8// 100
99.2 /9.5 100
8 / /
MUDSI/ HTDDSI/RAA//AA RRAP/S
100 100 190
99.7 99.999999 19.88
97.0/
99.1 99.1
99.8
/99.9/999 99.9
99.2 /9.3 /9.7
95.1 3 /9894
MIDAL/ TRAP/+KAPAL
100 100 100
99.8/ 99.9 99.9
98.3 99.2 99.8
99.8 199.8 99.99
99.4/ 598. / 199.8
97.9/ /9.9
VGG-16
Inc-43
Res:50
Dense 121
Inc.v3
Res:50
Desse
VGG-16
MIDI/ +RAP/ / +RAP-LS
90.0 / 99.1
88.8 9T.0/ 97,3
56.8/ 82.6/ SL.4
62.9/918//
90.6
85-7 / 94,2 93.3
/
99.77/1..0
MUDSI/ -RAPLLS
97.6/ 1968 / 9994
98.1 992 2 19.4
85.0/ /95.
89.0 89.0/912/
/9.2/
92.0 99.22 2 99.6
87.6/
77.5/903//99 /9.22
MIDAIL /RAP/ +RAP-LS
97.8/ 20.2 / 99.6
98.9/ /9.5 99.6
89.3/ 95.0/95.5
91.5 /.../ /94.7
95.4/ 262 197.6
91.4 24.2//2./90 :94,1
For TI, DI, SI, and Admix, RAP gets the improvements by 10.2%, 10.9%, 9.3%, and 6.3%. With late-start, RAP-LS further enhance the transfer attack performance for almost all methods.
Combinational Methods. Prior works demonstrate the combination of baseline methods could largely boost the adversarial transfcrability [41, 49]. We also investigate of behavior of RAP when incorporated with the combinational attacks. The results are shown in Table 2. As shown in the table, there exist the clear improvements of the combinational attacks over all baseline attacks shown in Table 1. In addition, our RAP-LS further boosts the average attack success rate of the three combinational attacks by 6.9%, 2.6%, and 1.7% respectively. Combined with the three combinational attacks, RAP-LS achieves 95.4%. 97. 7.. and 98.3% average attack success rate, respectively. These results demonstrate RAP can significantly enhance the transferability.
4.3 The Evaluation of Targeted Attacks
We then evaluate the targeted attack performance of the different methods with RAP. The results with logit loss are presented and the results with CE loss are shown in Appendix.
Baseline Methods. The results of RAP with baseline attacks are shown in Table 3. From the results, RAP is also very effective in enhancing the transferability in targeted attacks. Taking ResNet- 50 and DenseNet- 121 as surrogate models for example, the average performance improvements induced by RAP are 5.0% (I), 8.1% (MI), 4.6% (TI), 10.4% (DI), 18.5% (SI), and 15.1% (Admix), respectively. Comparing to the ResNet-50 and DenseNet- 121, the baseline attacks generally achieve lower transferability when using the VGG-16 or Inception-v3 as the surrogate models, which has also been verified in existing works [48, 49]. However, for Inception-v3 and VGG-16 as the surrogate models, RAP also consistently boosts the transferability under all cases. With late-start, RAP-LS could further improve the transferability of RAP for most attacks. The average attack success rate under all attack cases of RAP-LS is 2.6% higher than that of RAP.
Combinational Methods. As did in the untargeted attacks, we also evaluate the performance of combinational methods. The results are shown in Table 4. Similar to the findings in untargeted attacks, the combinational methods obtain significantly improvements over baseline methods. The RAP-LS outperforms all combinational methods by significantly margin. For example, taking the average attack success rate of all target models as evaluation metric, RAP-LS obtains 14.2%, 11. 11.8%, 9.3% improvements over the MTDI, MTDSI and MTDAI, respectively.
7

---

Table 3: The targeted attack success rate (%) of baseline methods with RAP. The results with logit loss are reported. The best results are bold and the second best results are underlined.
ResNes-50 VGG-16
DenseNet-121 VGG--1
Alaakk
Dense-12 121
Inc-v3
Res-50
Inc-v3
1/ STAPY / +RAP-LS
4,5795 14.3
2,47
11.8
01/1.11 0.7
5.1 12.8 17.9
2.9/ 10LL 115.9
0.0 0.//0.// / 1,2
ML/ HRAP +RAPLS
6.3: 17.5 129.
2.2/
20.6
0.1/1.1 24
4.67 16.2 26.5
11//13.4/
0.5:
4RA-RAP/ GRAPLLS
7.2/ 11.07737
40/12.9/ 15.3
0.1/8.8/1.
847 13.5 20.8
1.2/124/10 2224/16.44
0.2/2.1/
DE GRAPAS
62.//499 73.9
57.27
15/2.7
10.1
50.2/526
32.17 /2.5
1478877 10.0
ST/-RAI SE/-RRP//RRON ARAPLS
30.0/ /3.0/332/60
3.5/32./364.
36.0
1.7993/ 10.5
14:2/41.5
/44.4
84/31.07 36.2
1.6/8.5 10.4
Admis / -RAPLES
146/680//44 74.6
26.07 /5./5166
58/12.1
12/1/22//44 19.6
29.37 53.0 55.2
21.57 42.7 21.5/42.77
48.2
30/12.1/1 17.6
VGG-16
Inc-43
Atlack
Res-50
Dense-121
Inc.33
Res-50
Dense 121
VGG-16
-F-RAP/ JRAPLL
0.1/07/14
02/14/1.7
0.070.146
70.5
ME/ +RAP/ -RAP-LS
0./13/199
0./23/330
0.0/0.0003
0.2/
0.1/
0.2 1.//0/
TE/ TR/-RAP/ -RAP-LS
07/12/32
0.8 1
0.0/0.1/4 /..4
0.2//0.5 /0.7
0.1/ 70.6
0.2/ 0
DE /
28/23/99/
3.81 8.8. 12.7
0.0/0.4/ 11
16/46//6.4
2.8/58/ 1558/7..
2.//5.3/.1
SL/-RAP/+RON +RAP-LS
3,3/ 9.8
7.2/163/ 17.8
0,2/1.7 18
0.//29/2.5
0.9 2.22332
0.5/ 1.5/33
Admix/ /RAP/
202
0.4
1.5/2//52
2./09/11
1,57 3.3.444
Table 4: The targeted attack success rate (%) of combinational methods with RAP. The results with logit loss are reported. The best results are bold and the second best results are underlined.
Attack
Dense 121
ResNet-50 = VGG-16
Inc.v3
Res-50
DenseNet-121 VGG-16
Inc.v3
MTDI/ &RAP-LS
74.9782 N8.5
62.8/ 72.//1.
18.0/28./.8
44.9/64./
38.5755.0
23,0/
MIDSI/ +RAP/ +RAP-S MTDAI/+RAP/-RAN
86.3/88.1/99 1/933
70.1 77.2 184.7
/
$8.0
55.07 72/2 775.8
22.0/5..1 62.3
19.8: 0/ 39.2
014//9.4/934
79.9/
00/090/880 86.3
50.87 52.1
/64.1
/9.1/742/ 74.2/22.1
$4.7/ /69.3
32.0//1133 49.3
Attack
Res-50
VGG-16 =
Dense-121
Inc-v3
Res-50
Inc-13=>
Desse 121
VGG-16
MTDI/ -RAPALS
11/8167//24
13.7/ 19 27,4
0.734/446
18/83/255
4.17 11/1488111
2.0/8.0/
MIDSI/ +RAP/ +RAPLS
31.0/35./360
41.7
(4A/ 49.6
9.6/132 12.2
56/1119/102
/ /2.99
4.2/89/8.6
MIDAI/ -RAAPLS
/ /9.0/43.1
16.0/ 45.1 55.22
11.6/ [2.1 17.6
96/116/166
17. /316
84/12.0/ 12.1
4.4 The Comparison with Other Types of Attacks
Apart from the baseline and the combina- tional methods, we also experiment with more diverse attack methods, including the model-specific attack LinBP [14], the
Table 5: The comparison with ILA and LinBP. We use ResNet-50 as MS The best results are bold.
Turgeted
feature-based attack ILA [17), and the generative targeted attack TTP [30]. The LinBP depends on the skip connection and the authors only provide the source code about ResNet-50. We use their released code and thus conduct experiments with ResNet-50 as MM.. The results of LibBP and ILA are shown
Dense 121
77.7
2.8
59.8
9.4
2.0
LiaBP-LAA-SE EiERP-MI-DI
91.1
13.3
7.2
2.8
80.3
26.1
16.5
LERRAAI-D--SMM
99.3
90.2
32.6
22.1
4.6
MI DI+RAP
99.9
100
93.7
75.1
69.7
13.9
in Table 5, where we also implement the variants of LinBP following Guo al. [14], inlcuding LinBP-ILA, LLBP--L,-SGMM LinBP-MI-DI, and LinBP-MI-DI-SGM, We observe that our MI-DI- RAP significantly outperforms the LinBP and ILA, especially for the targeted attacks. Compared with the second-best method (i.e., LinBP-MI-DI-GGM) we obtain a large improvement by 33.5% on average ASR of targeted attacks.
TTP [30] is the duct targeted attack. To compare with it, we adopt the gen- erators based on ResNet-50 provided by the authors. Since TTP needs to train the perturbation generator for each targeted class, we follow their " 10-Targets (all-source)" setting, as did in Zhao et al. [49]. The results are shown in Table 6, where our MTDSI+RAP-LE behaves best and outperforms TTP and MTDI by large margins of 14.9% and 25.7%, respectively.
generative method to con-
Table 6: The comparison with TTP on targeted attack. The best results are bold.
Attack
Dense-121
VGG-16
Inc-v3
TTP
79.6
78.6
40.3
MIDI
78.6
74,6
12.7
MIDI-RAP-LS
35.4
MTDSI
93.2
80.0
41.3
MIDSI-RAP-LS
95.7
88.1
59,3
4.5 The Evaluation on Diverse Network Architectures and Defense Models
To further demonstrate the efficacy of RAP, we evaluate our method on more diverse network architectures, including SoeptionnRRsNettaae NASNet-Large and ViT-Base/16. We adopt ResNet-50 as the surrogate model and the results are shown in Table 7, col 2-7. As shown in the table, the
8

---

Table The evaluation on diverse network architectures and defense models.
Allack
In:Res-v2
Untarged NASNet-I.
VIT-8/16
IncRes-v2
Targeted NASNei-I.
VIT-8/16
Inc-V3wir
Untarged
Inc IncRos-V2own
Inc-vSul-
Targeted
IncRes-1
MTDI
83.4
80.0
27.9
14.8
32.1
0.4
68.1
50.9
0.8
0.0
MTDI-RAP-LS
97.5
43.0
62.5
1.7
N6.5
72.3
9.7
4.1
90.0
79.6
12.77
MIDSIRRAPLLS 98.6
99.7
57.4
64.0
80.4
5.3
96.5
91.5
31.0
22.0
MIDAI
97.3
98.8
45.5
58.4
75.3
3.3
92.1
82.7
17.2
12.2
MIDAIHRAP-LS
99.2
99.8
60.2
70,4
82.6
7.4
96.7
91.6
34,4
26.0
proposed RAP-LS achieves significant improvements for all three combinational methods all target models, and MTDAI+RAP-LS achieves the best performance for diverse models. For MTDAI, the average performance improvements induced by RAP-LS 5.9% and 7.8% for untargeted and targeted attacks, respectively. Since ViT is based on the transformer architecture that totally being different from convolution models, the transfer attacks based Resnet-50 behave relatively poor on especially on targeted attacks. Yet our RAP-LS still obtains consistent improvements for all compared methods. We also consider the mneemble-mode attack on these diverse network architectures and the results are given in Appendix.
Furthermore, we evaluate RAP on attacking defense models. We choose ensemble adversarial training (AT) model, adv-Inc-v3 and ens-adv-Inc-Res-v2 [38], multi-step AT model [33], imput transformation defense [46], feature denoising [47], and purification defense (NRP) [29]. We only demonstrate the results of ensemble AT in main submission. The evaluations of other defense models are shown in Appendix. Following prior works [41, 48]. we adopt the ensemble-model attack by averaging the logits of different surrogate models, including ResNet-50, ResNet- 101, Inception-v3, and Dnception-Reseess col 8-/1. /. We can observe that our RAP-LS further boosts transferability of the baseline methods on both targeted and untargeted attacks. For untargeted attacks, RAP-LS achieves average performance improvements of 9.8% and 14.1% Inc-v3adv and IncRes-v2ens respectively. For targeted attacks, the average performance improvements RAP-LS are 14.8% and 11.1%, respectively.
The transfer attack success rate on defense models are shown in Table
4.6
Ablation Study
We conduct ablation study on the neighborhoods en, the iteration number of inner optimization T and late-start KEs. We adopt targeted attacks with ResNet-50 as the surrogate model.
of the proposed RAP, including the size of
We first evaluate the effect of c, and T. We consider different values of e,. including 2/255, 4/255, 8/255, 12/255, 16/255, and 20/255. In Figure (a), we plot the tendency curves of the targeted attack success rate under different values of 6n and T. Note that in Sec. 3.2, we set on = G,/T. Thus for fixed 6n. larger T indicating lower stepsize an- The minimum stepsize of a, is set to 2/255. We have the following observation from the plot: for fixed cn, the more iterations T, the better attack performance. Thus, we adopt relatively smaller a, = 2/255 our experiments. In Figure 4 (b-d), we further plot the results of different attack methods and target models w.rlf en. where ax, = 2/255. As shown in the plots, the larger 6, generally improves the attack performance. For Inception-v3 and DenseNet- 21, the improvements become mild for even larger 6n Overall, the value 12 or 16 could lead to satisfactory result under most cases.
Then we conduct the ablation study of K KLs. In Figure 5, we report the targeted attack success rate of MI. DI, and MI-TI-DI combined with RAP-LS with KLS 0, 25, 50, 100, 150, 200. Note that the RAP-LS with KLS 0 reduces to RAP. As shown in the plots, the proposed late-start strategy can further boost attack performance of RAP for most cases. In general, the performance improvements increase as Ki,s increases, and then become mild when K7.s is larger than 100. The suitable value of 1.s is relatively consistent among different methods and target models.
4.7 The Targeted Attack Against Google Cloud Vision API
Finally, we conduct the transfer attacks to attack a practical and widely used image recognition system, Google Cloud Vision API, and in the more challenging targeted attack scenario. MTDAI-RAP-LS behaves the best performance in above experiments, so we choose to conduct the attack. We take the evaluation randomly selected 500 images and use ResNet-50 surrogate model. As the API

---

Den 121 (M): RAP)
141 RAP
RAP
Dr RAP
Targeted Attack Success hete (%6
60
Isov3
Inc.va
Den-121
Inc-33
Den-121
60-
VGG 16
Den-121
VGS 15
VSG
60
4,72235
40-
1,122258
16255
10
6222 2125 12235 taz31 23/33 F. Perturbatior Size of Fa F. Porturbation Size of 6a L. Perturbation Size of
4215 sats 12251 16221 20221
.0255 a015 12233 16215 20235
Number of Inner Attack Sceps
[A]
(b)
(c)
(d)
Figure 4: Targeted attack success rate (%) with various T and en. Res-50 is set surrogate model.
Inciva
Dens: 121
VGG 15
Torgeted Attack Soccess Rate (8)
1+ 1+RAP.15
RAP
DI RA-.AALLL
MTDI RAP
35
20-
30
60-
15
25
10
1003
132
A00
0
50
1003
A02
40
100
152
.03
100
350
AM
Late-Start Itaration Number Kic
Late-Start Iteracion Number Kis
Late-Start Iteration Number Kis
Late-Start Iteracioo Number K.s
Figure 5: Targeted attack success rate (%) with various KLs. Res-50 is set surrogate model.
returns 10 predicted labels for each query, evaluate the attacking performance, we whether or not the target class appears in the returned predictions. Since the predicted label space of Google Cloud Vision API do not fully correspond to the 1000 ImageNet classes, we manually treat classes with similar semantics to be the same classes. In comparison, the baseline MTDAI successfully attacks 232 images against the Google API. Our RAP-LS achieves large improvement, successfully attacking 342 images, leading to 22.0% performance improvements. Theese demonstrates the high efficacy of our method improve transferability on real-world system.
Conclusion
In this work, we study the transferability of adversarial examples that is significant for black-box attacks. The transferability of adversarial examples generally influenced by the overfitting of surrogate models. To alleviate this, we propose to seeking adversarial examples that locate at flatter local regions. That is, instead optimizing the pinpoint attack loss, we aim to obtain a consistently low loss at the neighbor regions of the adversarial examples. We formulate this min-max bi-level optimization problem, where the inner maximization aims to inject the worse-case perturbation for the adversarial examples. We conduct rigorous experimental study, covering untargeted attack and targeted attack, standard and defense models, and real-worke Google Clould Vision API. The experimental results demonstrate that RAP can significantly boost the transferability of adversarial examples, which also demonstrates that transfer attacks have become serious threats. We need to consider how effectively defense against them.
Acknowledgments
We want to thank the anonymous reviewers for their valuable suggestions and comments. This work is supported by the National Natural Science Foundation China under grant No.62076213, Shenzhen Science and Technology Program under grant No. R0R.RC22221000 1030050000 and the university development fund of the Chinese University of Hong Kong, Shenzhen under grant No.01001810 and sponsored by CCF-Tensent Open Fund.
10

---

References
[1] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversar- ial examples. In International conference machine learning, pages PMLR, 2018.
2, 4.1,, C, C.3
[2] Minhao Cheng, Thong Le, Pin- Yu Chen, Huan Zhang, JinFeng Yi, and Cho-Jui Hsieh. Query- efficient hard-label black-box attack: An optimization- approach. International Confer- ence Learning Representations, 2019.
[3] Minhao Cheng, Simranjit Singh, Patrick H. Chen, Pin- Yu Chen, Sijia Liu, and Cho-Jui Hsieh. Sign-opt: A query-efficient hard-label adversarial attack. In International Conference on Learning Representations, 2020.
[4] Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio, Alina Oprea, Cristina Nita Sita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? explaining transferability of evasion and poisoning attacks. 28th USENIX security symposium (USENIX security 19), pages 321-338, 2019. 3.2
[5] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018. 2, 3.1, 3.2, 3.3, 4.1
[6] Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by wanslation-invarian attacks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4312-4321, 2019. 1, 2, 3.2, 4.1
[7] Yinpeng Dong, Qi-An Fu, Xiao Yang. Tianyu Pang. Hang Su, Zihao Xiao, and Jun Zhu. Benchmarking adversarial robustness on image classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 321-331, 2020. D
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. 4.1
[9] Yanbo Fan, Baoyuan Wu, Tuanhui Li, Yong Zhang, Mingyang Li, Zhifeng Li, and Yujiu Yang. Sparse adversarial attack via perturbation factorization. European conference on computer vision, pages 35 50. Springer, 2020.
[10] Yan Feng, Baoyuan Wu, Yanbo Fan, Li Liu, Zhifeng Li, and Shu-Tao Xia. Boosting black-box attack with partially transferred conditional adversarial distribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15095- 15104, June 2022.
[11] Lianli Gao, Qilong Zhang, Jingkuan Song, Xianglong Liu, and Heng Tao Shen. Patch-wise attack for fooling deep neural network. In European Conference on Computer Vision, pages 307-322. Springer, 2020. B
[12] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint
2014. 1,2
[13] Martin Gubri, Maxime Cordy, Mike Papadakis, Yves Le Traon, and Koushik Sen. Lgv: Boosting adversarial example transfcrability from large geometric vicinity. arXiv preprint arXiv:2207.13122 13/29, 2022.
[14] Yiwen Guo, Qizhang Li, and Hao Chen. Backpropagating linearly improves transferability of adversarial examples. Advances in Neural Information Processing Systems, 33:85-95, 2020. 1,
2, 4.1,, 4.4, B
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 4.1

---

[16] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017. 4.1
[17] Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019. 1, 2, 4.1, 4.4
[18) Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International Conference on Machine Learning, pages 2137-2146. PMLR, 2018. 1
[19] Nathan Inkawhich, Kevin Liang, Lawrence Carin, and Yiran Chen. Transferable perturbations of deep feature distributions. In International Conference on Learning Representations, 2020. 2
[20] Nathan Inkawhich, Kevin Liang, Binghui Wang, Matthew Inkawhich, Lawrence Carin, and Yiran Chen. Perturbing across the feature hierarchy improve standard and strict blackbox attack transferabilityy Advances in Neural Information Processing Systems, 33, 2020. 2
[21] Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial examples in the physical doi: 10.1201/
world. Artificial Intelligence Safety and Security, page 99-112, Jul 2018.
081351225189-88 2, 3.1, 3.3, 4.1
[22] Yingwei Li, Song Bai, Yuyin Zhou, Cihang Xie, Zhishuai Zhang, and Alan Yuille. Learning transferable adversarial examples via ghost networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11458-11466 -1485, 2020. 4.1, C
[23] Siyuan Liang, Baoyuan Wu, Yanbo Fan, Xingxing Wei, and Xiaochun Cao. Parallel rectangle flip attack: A query- based black-box attack against object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7697-7707, 2021.
[24] Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E. Hoperoft. Nesterov accelerated gradient and scale invariance for adversarial attacks. In International Conference on Learning Representations, 2020. 1, 2, 3.2, 4.1
[25] Risheng Liu, Jiaxin Gao, Jin Zhang, Deyu Meng, and Zhouchen Lin. Investigating bi-level optimization for learning and vision from a unified perspective: A survey and beyond. arXiv preprint arXiv:2101. 115177 2021. 3.2
[26] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. arXiv preprini arXiv: 16/i::6/1..022 1.02770, 2016. 1
[27] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[28] Muhammad Muzammal Naseer, Salman H Khan, Muhammad Haris Khan, Fahad Shahbaz Khan, and Fatih Porikli. Cross-domain transferability of adversarial perturbations. Advances in Neural Information Processing Systems, 32: 1905-1 2019. 2
[29] Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self-supe approach for adversarial robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 262-271, 2020. 4.5, D
[30] Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. On generating transferable targeted perturbations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7708-7717, 2021. 1, 2, 4.1, 4.4, 4.4, B
[31] Omid Poursaced, Isay Katsman, Bicheng Gao, and Serge Belongie. Generative adversarial per- turbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4422 4431, 2018. 2
[32] Zeyu Qin, Yanbo Fan, Hongyuan Zha, and Baoyuan Wu. Random noise defense against query- based black-box attacks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. 1
12

---

[33] Hadi Salman, Andrew Iyass, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In H. Larochelle, M. Ranzato, R. Had- sell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 3533-3545. Curran Associates, Inc., 2020. 4.1, 4.5, D
[34] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014. 4.1
[35] Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 1312.6199, 2013.
[36] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbignniw Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 28 1818,282,, 2016. 4.1
[37] Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-vé v4, inception-resnet and the impact of residual connections on learning. In Thirty-first AAAI conference on artificial intelligence, 2017. 4.1
[38] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations, 2018. 1, 3.2, 4.5
[39] Jingkang Wang, Tianyun Zhang. Sijia Liu, Pin-Yu Chen, Jiacen Xu, Makan Fardad, and Bo Li. Adversarial attack generation empowered by min-max optimization. Advances in Neural Information Processing Systems, 34, 2021. 2
[40] Xiaosen Wang and Kun He. Enhancing the transferability of adversarial attacks through variance tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1924-1933, June 2021. 2, 4.1,
[41] Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. Admix: Enhancing the transferability of adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16158-16167, 2021. 3.1, 3.2, 3.2, 4.1, 4.2,
[42] Xiaosen Wang, Jiadong Lin, Han Hu, Jingdong Wang, and Kun He. Boosting adversarial transferability through enhanced momentum. arXiv preprint arXiv:2103. 2021. 4.1, C
[43] Dongxian Wu, Yisen Wang. Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In International Conference Learning Representations, 2020. 2
[44] Jiancong Xiao, Yanbo Fan, Ruoyu Sun, and Zhi-Quan Luo. Adversarial rademacher complexity of deep neural networks, 2022.
[45] Jiancong Xiao, Yanbo Fan, Ruoyu Sun, Jue Wang, and Zhi-Quan Luo. Stability analysis and generalization bounds of adversarial training. arXiv preprint arXiv:2210.00960, 2022.
[46] Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In International Conference on Learning Representations, 2018.
3.2, 4.1, 4.5, D
[47] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 501-509, 2019. 4.1, 4.5, D,D
[48] Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang. Zhou Ren, and Alan Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2730-2739, 2019. 1, 3.1, 3.2, 3.3, 4.3, 4.5
[49] Zhengyu Zhao, Zhuoran Liu, and Martha Larson. On success and simplicity: A second look at transferable targeted attacks. arXiv preprint arXiv:2012. 1207, 2020. 2, 3.1, 4.1, 4.2, 4.3, 4.4, B, E.3
13

---

[50] Xin Zheng, Yanbo Fan, Baoyuan Wu, Yong Zhang, Jue Wang, and Shirui Pan. Robust physical- world attacks on face recognition. Pattern Recognition, page 109009, 2022.
[51] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8697-8710, 2018. 4.1
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See final part of Section 1.
(b) Did you describe the limitations of your work? [Yes] See Appendix
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1
(b) Did you mention the license of the assets? [Yes] See Appendix
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] No, all the used data well-known public benchmark datasets such as CIFAR1O 10, ImageNet.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] No, all the used data are well-known public benchmark datasets such CIFAR10 10, ImageNet.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] Not applicable.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] Not applicable.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] Not applicable.
14


Reviews:
[{"review_id": "zxjtPtUSwq9", "reviewer": "Reviewer_pUE8", "summary": "This paper focuses on generating transferable adversarial examples. Observing previous works overfits to the surrogate model, the authors propose a novel method to address this issue. According to the experiments, the proposed method generates stronger transferable adversarial examples.\n", "questions": "See weakness. If the concern in weakness can be well addressed, I am willing to increase the score.", "limitations": "1. It is not clear if the proposed method can attack strong defense models.\n2. It is not clear if the proposed method can be applied to generate other kinds of adversarial examples (such as patch attack)", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Strength\n1. The proposed method is interesting and insightful. Previous methods usually address the overfitting issues by borrowing the techniques from deep learning model training strategy, such as momentum term, data augmentation. In contrast, this paper proposes a method specifically designed for generating adversarial examples.\n2. The results are reasonably good. According to the results in the main paper, the proposed RAP and RAP-LS significently improve the transferability of adversarial examples.\n- Weakness\nI found the proposed method is less effective when attacking strong defense models (such as Feature Denoising), according to Table 8. Is there any explanation?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "r1O5naRNSGz", "reviewer": "Reviewer_HFVj", "summary": "The paper proposed to add reverse adversarial perturbation at each iteration in gradient-based attacks to compute more accurate gradient directions and escape sharp local minimum", "questions": "- How is. the performance of the attack wrt to PGD-based adversarial training\n\n- Recent works of improving gradient directions by looking ahead [a] or by accumulating the gradients of multiple data points around the current data point sampled [b] can be discussed to clearly highlight the contributions\n\n[a] Jang, Donggon, Sanghyeok Son, and Dae-Shik Kim. \"Strengthening the Transferability of Adversarial Examples Using Advanced Looking Ahead and Self-CutMix.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[b] Wang, Xiaosen, et al. \"Boosting adversarial transferability through enhanced momentum.\" arXiv preprint arXiv:2103.10609 (2021).\n\n- SOTA transferability technique   Variance tuning-based FGSM [c] as well as [a] should be reported since they refine the gradient direction as in the proposed method. I appreciate the authors for including another line of methods such as LinBP and ILA, and TTP.  \n\n[c] Wang, Xiaosen, and Kun He. \"Enhancing the transferability of adversarial attacks through variance tuning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nI'm giving the score of 4 because of no evaluation and discussion to  [a,b,c] which also find optimal gradient direction at each step. I'm very much open to increasing the scores given the comprehensive experiments already reported in the paper.   ", "limitations": "-  calculation of inner optimization increases the computational time of attack wrt baseline methods.\n- Adds more hyperparameters (S and mu) but the ablation studies show they are largely target model agnostic", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n- Paper is clear and the idea is simple.  \n- Experiments are comprehensive\n- Can be integrated easily into existing gradient-based attacks\n- Figure 3 is very insightful\n\n\n#### Weakness\n- The paper improves already an earlier line of works to find a more accurate gradient direction. The novelty of the method is not high but that said, the paper clearly provides detailed analyses from a loss landscape perspective and is validated through a battery of experiments in both targeted and untargeted persepctive.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Iceg-We4ALP", "reviewer": "Reviewer_Ra8U", "summary": "The authors claim that the adversarial examples should be in a flat local region for better transferability. To boost the transferability, they propose a simple yet effective method named Reverse Adversarial Perturbation (RAP). RAP adds an inner optimization to help the attack escape sharp local minima, which is general to other attacks. Experimental results demonstrate the high effectiveness of RAP.", "questions": "See weakness, especially 4 and 5.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n1. The idea is novel and interesting. The adversarial example in flat local region could boost the transferability, which seems to be reasonable.\n\n2. The method is simple yet effective. Adding an inner optimization could effectively boost the transferability, which is also general to existing attacks.\n\n3. Extensive evaluations on ImageNet demonstrate the effectiveness of the proposed method in most scenarios.\n\n# Weakness\n\n1. Figure 1 is not clear. What do the x-axis and y-axis indicate? How do you obtain the curve?\n\n2. Why does the flat local region indicate better transferability? It might be interesting if you could provide some analysis.\n\n3. What does one iteration mean? Does it indicate one forward and back propagation? \n\n4. RAP is general to existing attacks, not limited to input transformation based attacks. Hence, the authors should integrate RAP into various attacks, such as momentum-based attack [1], different loss function [2], model specific attack [3].\n\n5. The biggest concern is the huge computation cost due to the larger iteration and the inner optimization in Eq. (4). Existing attacks could exhibit superior attack performance in much lower iterations. For instance, MI-FGSM [4], VMI-FGSM [1], EMI-FGSM [5], and the corresponding input transformations only adopt 10 iterations.\n\n6. Minor typos:\n\nLine 42 “a slightly change…” -> “a slight change…”\n\nLine 65 “help to escape…” -> “help escape…”\n\nLine 76 “can significant boost…” -> “can significantly boost…”\n\n\n[1] Wang et al. Enhancing the transferability of adversarial attacks through variance tuning. CVPR 2021.\n\n[2] Wu et al. Boosting the Transferability of Adversarial Samples via Attention. CVPR 2020.\n\n[3] Li et al. Learning Transferable Adversarial Examples via Ghost Networks. AAAI 2020.\n\n[4] Dong et al. Boosting Adversarial Attacks with Momentum. CVPR 2018.\n\n[5] Wang et al. Boosting Adversarial Transferability through Enhanced Momentum. BMVC 2021.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Gygif0cheP", "reviewer": "Reviewer_VMp1", "summary": "In this paper, the authors propose a method (RAP) for generating adversarial examples that are transferable to other models. The authors identify that adversarial examples lay on a flat minimum will be more likely to be transferred to other models. The authors then propose the reverse adversarial perturbation guided by an additional inner maximization optimization process, to generate adversarial examples lay on a flat minimum.", "questions": "Though It seems that the theory of flat minimum is intuitive, is there any additional analysis besides the empirical experimental results that justify the theory? Additionally, why simple EOT method is not enough for generating adversarial examples at a flat minimum? It seems to the reviewer that no competitive baselines are provided in the paper.", "limitations": "The limited novelty of the paper. The theory of flat minimum is intuitive and not well justified by either experiment or theoretical proofs. Competitive baselines are not provided to show the effectiveness of the proposed method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The proposed method is straightforward and effective.\n+ The authors conduct a comprehensive evaluation of several models with various types of attacks.\n+ The authors conduct a case study with Google Cloud Vision API to validate the effectiveness of the proposed method.\n- The proposed method is not new.\n- The authors did not provide an analysis or theoretical proof for justifying the flat minimum hypothesis.\n- The evaluation does not have competitive baselines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Privacy and Security (e.g., consent)"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept