Rethinking Model Ensemble in Transfer-based Adversarial Attacks
Authors: Huanran Chen
Yichi Zhang
Yinpeng Dong
Xiao Yang
Hang Su
Jun Zhu
Venue: ICLR 2024 poster (2024)
Keywords: Adversarial attack
transferability
ensemble attack
robustness
Primary area: societal considerations including fairness, safety, privacy
TL;DR: We rethink model ensemble in transfer-based adversarial attacks and propose a Common Weakness Attack to generate more transferable adversarial examples.

Abstract: It is widely recognized that deep learning models lack robustness to adversarial examples. An intriguing property of adversarial examples is that they can transfer across different models, which enables black-box attacks without any knowledge of the victim model. An effective strategy to improve the transferability is attacking an ensemble of models. However, previous works simply average the outputs of different models, lacking an in-depth analysis on how and why model ensemble methods can strongly improve the transferability. In this paper, we rethink the ensemble in adversarial attacks and define the common weakness of model ensemble with two properties: 1) the flatness of loss landscape; and 2) the closeness to the local optimum of each model. We empirically and theoretically show that both properties are strongly correlated with the transferability and propose a Common Weakness Attack (CWA) to generate more transferable adversarial examples by promoting these two properties. Experimental results on both image classification and object detection tasks validate the effectiveness of our approach to improving the adversarial transferability, especially when attacking adversarially trained models. We also successfully apply our method to attack a black-box large vision-language model -- Google's Bard, showing the practical effectiveness. Code is available at \url{https://github.com/huanranchen/AdversarialAttacks}.

Published as a conference paper at ICLR 2024
RETHINKING MODEL ENSEMBLE IN TRANSFER-BASED ADVERSARIAL ATTACKS
Huanran Chen Yichi Zhang Yinpeng Dong Xiao Yang", Hang Su², Jun Zhu 1 School of Computer Science, Beijing Institute of Technology
Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bood Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University, Beijing, 100084, China RRealAI
baanranchen@bit... adu. cn, (zyc22, yangxiaol yemssls.tsinghus.e wsinghus.edu.com
fdongyinpeng, suhangss, deszj eeeamm.ttinnnhaeeeee.e.ee. wwwngnnn..neee.....e edu.cn
ABSTRACT
It is widely recognized that deep learning models lack robustness to adversarial examples. An intriguing property of adversarial examples is that they can transfer across different models, which enables black-box attacks without any knowledge of the victim model. An effective strategy to improve the transferability attacking an ensemble of models. However, previous works simply average the outputs of different models, lacking an in-depth analysis on how and why model ensemble methods can strongly improve the transferability. In this paper, we rethink the en- semble in adversarial attacks and define the common weakness of model ensemble with two properties: 1) the flatness of loss landscape; and 2) the closeness to the local optimum of each model. We empirically and theoretically show that both properties are strongly correlated with the transferability and propose Common Weakness Attack (CWA) to generate more transferable adversarial examples by pro- moting these two properties. Experimental results both image classification and object detection tasks validate the effectiveness of our approach to improving the adversarial transferability, especially when attacking adversarially trained models. We also successfully apply our method to attack black-bor large assion-language model - Google's 's Bard, showing the practical effectiveness. Code is available https: / // thub. com ranchen. wwt.///aaaaaaaaata.aaaaaaaaaaaaa
INTRODUCTION
Deep learning has achieved remarkable progress over the past decade and has been widely deployed in real-world applications (LeCun et al., 2015). However, deep neural networks (DNNs) are vulnerable to adversarial examples (Szegedy et al., 2014; Goodfellow et al., 2015), which are crafted by imposing human-impenn imperceippeee perturbations to natural examples. Adversarial examples can mislead the predic- tions of the victim model, posing great threats to the security of DNNs and their applications (Kurakin et al., 2018; Sharif et al., 2016). The study of generating adversarial examples, known as adversarial attack, has attracted tremendous attention since it can provide a better understanding of the working mechanism of DNNs (Dong et al., 2017), evaluate the robustness of different models (Carlini & Wagner, 2017), and help to design more robust and reliable algorithms (Madry et al., 2018).
Adversarial attacks can be generally categorized into white-bon and black-box attacks according to the adversary's 's knowledge of the victim model. With limited model information, black-box attacks either rely on query feedback (Chen et al., 2017) or leverage the transferability (Liu et al., 2017) to generate adversarial examples. Particularly, transfer-based attacks use adversarial examples generated for white-box surrogate models to mislead black-box models, which do not demand query access to the black-box models, making them more practical in numerous real-world scenarios (Liu et al., 2017: Dong et al., 2018). With the development of adversarial defenses (Madry al., 2018; Tramèr et al., 2018; Wong et al., 2020; Wei et al., 2023b) and diverse network architectures (Dosovitskiy al., 2020; Liu et al., 2021; 2022b), the transferability of existing methods can be largely degraded.
Y. Dong and J. Zhu corresponding authors. H. Chen has done this work Tsinghua University.

---

Published as a conference paper at ICLR 2024
Sorrig Tagus and
Bemoan m6-11 Teginendi
Torget woidd
Rupa anii
(a) Not flat and not close
(b) Flat and not close
(c) Not flat and close
(d) Flat and close
Figure| 1: Illustration of Common Weakness. The generalization error is strongly correlated with the flatness of loss landscape and the distance between the solution and the closest local optimum of each model. We define the common weakness of model ensemble as the solution that is at the flat landscape and close to local optima of training models, as shown in (d).
By making an analogy between the transferability of adversarial examples and the generalization of deep neural networks (Dong et al., 2018), many researchers improve the transferability by designing advanced optimization algorithms to avoid undesirable optimum (Dong et al., 2018; Lin et al., 2020; Wang & He, 2021; Wang et al., 2021) or leveraging data augmentation strategies to prevent overfitting (Xie et al., 2019 Dong al., 2019; Lin et al., 2020). Meanwhile, generating adversarial examples for multiple surrogate models can further improve the transferability, similar to training on more data to improve model generalization (Dong et al., 2018). Early approaches simply average the outputs of multiple models in loss (Liu et al., 2017) or in logits (Dong et al., 2018), but ignore their different properties. Xiong et al. (2022) introduce the SVRG optimizer to reduce the variance of gradients of different models during optimization. Nevertheless, recent studies ( (Yang et al., 2021) demonstrate that the variance of gradients does not always correlate with the generalization performance. From the model perspective, increasing the number of surrogate models can reduce the generalization error (Huang et al., 2023), and some studies (Li et al., 2020; Huang et al., 2023) propose to create surrogate models from existing ones. However, the number of surrogate models in adversarial attacks usually limited due to the high computational costs, making it necessary to develop new algorithms that can leverage fewer surrogate models while achieving improved attack success rates.
In this paper, we rethink the ensemble methods in adversarial attacks. By using a quadratic approxi- mation of the expected attack objective over the victim models, we observe that the second-order term involves the Hessian matrix of the loss function and the squared 6z distance to the local optimum of each model, both of which are strongly correlated with the adversarial transferability (as illustrated in Fig, 1), especially when there are only few models in the ensemble. Based these two terms, define the common weakness of an ensemble of models as the solution that is at the flat landscape and close to the models' local optima. To generate adversarial examples that exploit the common weakness of model ensemble, we propose a Common Weakness Attack (CWA) composed of two sub- mebboodsddd named Sharpness Aware Minimization (SAM) and Cosine Similarity Encourager (CSE), which are designed to optimize the two properties of common weakness separately. As our methods orthogonal to previous methods, e.g., MI (Dong et al., 2018), VMI (Wang & He, 2021), SSA (Long et al., 2022), our methods can be incorporated with them seamlessly for improved performance.
We conduct extensive experiments to confirm the superior transferability of adversarial examples generated by our methods. We first verify in image classification for 31 victim models with various architectures (e.g., CNNs (He al., 2016), Transformers (Dosovitskiy et al., 2020; Liu al., 2021)) and training settings (e.g., standard training, adversarial training (Salman et al., 2020; Wong et al., 2020), input purification (Naseer et al., 2020; Nie et al., 2022)). Impressively, when attacking the state-of-the-arr defense models, MI-CWA surpasses MI with the logits ensemble strategy (Dong et al., 2018) by 30% under the black-box setting. We extend our method to object detection by generating a universal adversarial patch that achieves averaged mAP of 9.85 over modern detectors, surpassing the recent work (Huang et al., 2023). Ablation studies also validate that our methods indeed flatten the loss landscape and increase the cosine similarity of gradients of different models. Moreover, demonstrate successful black-box attacks against a recent large vision- Languala model (i.e., Google's Bard) based on our method, outperforming the baseline method by a large margin.
RELATED WORK
In this section, we briefly review the existing cransferbbseed adversarial attack algorithms. Overall, these methods usually make an analogy between the transferability of adversarial examples and the generalization of deep learning models, and categorize them into 3 classes follows.
2

---

Published as a conference paper at ICLR 2024
Gradient-based optimization. Comparing the optimization of adversarial examples and the training of deep learning models, researchers introduce some optimization techniques to boost the transfer- abillity of adversarial examples. Momentum Iterative (MI) method (Dong al., 2018) and Nesterov Iterative (NI) method (Lin et al., 2020) introduce the momentum optimizer and Nesterov accelerated gradient prevent the adversarial examples from falling into undesired local optima. Enhanced Momentum Iterative (EMI) method (Wang et al., 2021) accumulates the average gradient of the data points sampled in the gradient direction of the previous iteration to stabilize the update direction and escape from poor local maxima. prriance-tuning Momentum Iterative (VMI) method (Wang & He, 2021) reduces the variance of the gradient during the optimization via tuning the current gradient with the gradient variance in the neighborhood of the previous data point.
Input transformation. These methods transform the input images before feeding to the classifier for diversity, similar to data augmentation techniques in deep learning. Diverse Inputs (DI) method (Xie et al., 2019) applies random resizing and padding to the input images. Cranslation-nvariiii (TI) attack (Dong al., 2019) derives an efficient algorithm for calculating the gradients w.r.t. the translated images, which is equivalent to applying translations to the input images. Scale-Invariant (SI) attack (Lin et al., 2020) scales the images with different factors based the observation that the model has similar performance on these scaled images.
Model ensemble methods. As mentioned in Huang et al. (2023), increasing the number of classifiers in adversarial attacks can reduce the generalization error upper bound in Empirical Risk Minimization (ERM), similar increasing the number of training examples in deep learning. Dong et al. (2018) propose to alternatively average over losses, predicted probabilities, or logits of surrogate models to form an ensemble. Li al. (2020) and Huang et al. (2023) also propose novel methods to generate massive variants of the surrogate models and then take the average value of losses or logits. Xiong et al. (2022) introduce the SVRG optimizer (Allen-Zhh & Yuan, 2016) into ensemble adversarial attack to reduce the variance of the gradients during optimization. However, the number of surrogate models in practice is usually small, which cannot guarantee satisfying generalization upper bound in the theory of ERM, and only focusing on variance does not necessarily lead to better generalization (Yang et al., 2021). Additionally, self-ensembled surrogate models typically exhibit lower effectiveness compared to standard surrogate models, resulting in limited performance.
3 METHODOLOGY
In this section, we present our formulation of common weakness based on second-order approxima- tion and propose the Common Weakness Attack (CWA) composed of Sharp Aware Minimization (SAM) and Cosine Similarity Encourager (CSE).
3.1 PRELIMINARIES
We F= { denote the set of possible image classifiers for a given task, where each classifier f: IR' - R outputs the logits over K classes for an input æ E IR." Given a natural image aot and the corresponding label transfer-based attacks aim to craft an adversarial example æ that could be misclassified by the models in F. It can be formulated
constrained optimization problem:
min FE (T( y)], s.t. || @nna al S
where L. is the loss function, e.g., negative cross-entropy loss as [(f(x), y) = log (f(x)) and we study the loo norm. Eq. (1) can be approximated using few "training" classifiers F,
(fi) C F (known as ensemble) as A > i-1 ((fi(a), y) (i.e., loss ensemble (Liu et al., 2017)) or
E. fz(z).)) y) (i.e., logits ensemble (Dong et al.. 2018)). Previous works (Dong et al., 2018; 2019; Xie et al., 2019; Wang & He, 2021) propose gradient computation or input transformation methods based on this empirical loss for better transferability. For example, the momentum iterative (MI) method (Dong et al., 2018) performs gradient update (as shown in Fig. 2(a)) as
VeL(2 E L E vil
m = f m
0t+1 = clip (x; + a sign(m)),
where m accumulates the gradients, j is the decay factor, a is the step size, clipa....e (x) would project to the l-x ball around natt with radius e, and we use the logits ensemble strätegy here.
3

---

Published as a conference paper at ICLR 2024
3.2 MOTIVATION OF COMMON WEAKNESS
Although the existing methods can improve the transferability to some extent, a recent work (Huang al., 2023) points out that the optimization of adversarial example conforms to Empirical Risk Minimization (ERM) (Vapnik, 1999) and the limited number of training models could lead to large generalization error. In this paper, we consider a quadratic approximation of the objective in Eq. (1).
Formally, we let pt denote the closest optimum of model fi E F to x and H; denote the Hessian matrix of (z)( y) at py. We employ the second-order Taylor expansion to approximate Eq. (1) at p; for each model fi as
Brer ((F(())) + 1 +/(10-
-2(0-pp)) - -p)) H2 -H, -p:)
(2)
In the following, we omit the subscript of the expectation in Eq. (2) for simplicity. Based on Eq. (2). we can see that a smaller value of (f y)] v)] and E[(x -P:)
means a smaller loss on testing models, i.e., better transferabilityy The first term represents the loss value of each model at its own optimum p.. Some previous works (Laurent & Brecht, 2018: Kawaguchi et al., 2019) have proven that the local optima have nearly the same value with the global optimum in neural networks. Consequently, there may be room for further improving this term, and we put our main focus on the second term to achieve better transferability.
In the following theorem, we derive upper bound for the second term, since directly optimizing it which requires third-order derivative is intractable.
Theorem 3.1. (Proof in Appendix A.I) Assume that the covariance between |Hlle and l|p: æ||2 is zero, we can get the upper bound of the second term as
B[(x= p:) H. p:)] - E H. E|I( 1:)
(3)
Intuitively, |H, represents the sharpness/latness of loss landscape (Tu et al., 2016; Tsuzuku et al., 2020; Liu et al., 2022a; Wei et al., 2023c) and [ p:) l describes the translation of landscape. The two terms can be assumed independent and therefore their covariance can be assumed as zero As shown in Theorem 3.1, smaller value of TE[ H, H|] and Eii((x p;) provides a smaller bound and further leads to a smaller loss on testing models. Previous works have pointed out that a smaller norm of the Hessian matrix indicates flatter landscape of the objective, which is strongly correlated with better generalization (Wu et al., 7017; Li et al., 2018; Chen et al., 2022). As for El P1ll representing the squared l distance from to the closest optimum p; of each model, we empirically show that it also has a tight connection with generalization and adversarial transferability in Appendix D.2 and theoretically prove this under simple case in Appendix A.2. Fig. 1 provides an illustration that the flatness of loss landscape and closeness between local optima of different models can help with adversarial transferability.
Based on the analysis above, the optimization of an adversarial example turns into looking for point near the optima of victim models (i.e., minimizing the original objective), while pursuing that the landscape at the point is flat and the distance from it to each optimum is close. The latter two targets are our main findings and we here define the concept of common weakness with these two terms as local point who has small value of Bi]l H [[]] and [ill P.) Note that there is clear boundary between common weakness and non-common weakness the smaller these two terms are, the more likely æ to be the common weakness. The ultimate goal is to find example that has the properties of common weakness. As Theorem 3.1 indicates, we can achieve this by optimizing these two terms separately.
3.3 SHARPNESS AWARE MINIMIZATION
To cope with the flatness of loss landscape, we minimize | / for each model in the ensemble. However, this requires third-order derivative w.r.t. æ, which is computationally expensive. There are some research to ease the sharpness of loss landscape in model training (Foret et al., 2020; Izmailov et al., 2018; Kwon et al., 2021). The Sharpness Aware Minimization (SAM) (Foret et al., 2020) is an effective algorithm problem. The inner maximization aims to find a direction along which the loss changes more rapidly; while the outer problem minimizes the loss at this direction to improve the flatness of loss landscape.
acquire a flatter landscape, which is formulated
min-max optimization
4

---

Published as a conference paper at ICLR 2024
4
7
2111
V.L.
Feil
Zt-1
(a) MI
(b) SAM
(c) MI-SAM
Figure 2: Illustration of MI, SAM, and MI-SAM. The symbols are introduced Eq. (4)-(6)
In the context of generating adversarial examples restricted by the &o norm as in Eq. (1), we aim to optimize the flatness of landscape in the space of & norm boost the transferability. which is different from the original SAM in the space of & norm. Therefore, we derive a modified SAM algorithm suitable for the Eo norm (in Appendix B.1). As shown Fig. 2, at the 1-th iteration of adversarial attacks, the SAM algorithm first performs an inner gradient ascent step at the current adversarial example 2e with a step size r as
me + sign (Vol (vo f(m))))) (())))))))))
/ = clip Pentss
(4)
and then performs an outer gradient descent step at ot with step size a as
x = clip anatic x a sign VoL 1 (((E).))) (
(5)
Note that in Eq. (4) and Eq. (5), we apply SAM on the model ensemble rather than each training model. Thus,
not only perform parallel computation during the backward pass improve efficiency, but also obtain better results using the logits ensemble strategy (Dong et al., 2018).
Moreover, we combine the reverse step and forward step of SAM as single update direction x! Tt, and integrate into existing attack algorithms. For example, the integration of MI and SAM yields the MI-SAM algorithm with the following update (as shown in Fig. 2(c))
m m + x! Rz; I++1 clip (x; + m))
(6)
where accumulates the gradients with decay factor μ.. By iteratively repeating this procedure, the adversarial example will converge into flatter loss landscape, leading to better transferability.
3.4 COSINE SIMILARITY ENCOURAGER
We then develop an algorithm to make the adversarial example converge to point which is close to the local optimum of each model. Instead of directly optimizing En [|(x p:))l with the training models the ensemble, since it is hard to compute the gradient w.r.t. æ, we derive an upper bound for this loss.
Theorem 3.2. (Proof in Appendix A.3) The upper bound of 1 E
I1 - p:)ll is proportional to
the dot product similarity between the gradients of all models:
1
c P.)
2M
Σ
1-1
9:9j.
where M max H and g; L ((((())),)) y) represents the gradient of the i-th model.
Based on Theorem 3.2, the minimization of the distance to the local optimum of each model turns into the maximization of the dot product between gradients different models. To solve this problem, Nichol et al. (2018) propose efficient algorithm via first-order derivative approximation. We apply this algorithm to generate adversarial examples by successively performing gradient updates using each model sampled from the ensemble F, with small step size B. The update process is
x clip e (x) 1-1 3 V (f.
(8)
where n²! = ær, Once the update for every model is complete, calculate the final update using larger step size a as
Lt++ clip (x, + a (x - æz)).
Although directly applying this algorithm can achieve good results, is incompatible with SAM due to the varying scales of gradient norm (Dong al., 2018). To solve this problem, we normalize the
5

---

Published as a conference paper at ICLR 2024
Algorithm 1 MI-CWA algorithm
Require: natural image &nat, label y, perturbation budget c, iterations 7' loss function L, model ensemble
Fr {ft)==1. decay factor μa, step sizes r. 3 and a.
1: Initialize: m = 0, inner momentum m = 0, æo = Bnati
2: fort= = =0=00 to 7 - do
3: Calculate g V L Ei= fa(x)),, v):
4: Update 2e hy m = clip clip (ate + sign(g)))
5: for:= = to n do
6: Calculate g = v. (. "),y);
Update inner momentum by m p.m. m.-mi Toll
8:
Update 2i by क; = clip (x) 1- m²); end for
9:
10: Calculate the update g
Te:
Update momentum m μ m g:
12: update w1+1 by 2++1 = clip clip=nat. nc (2 + a sign(m)))
13:
end for
14:
Return: øT.
gradient at each update by their & norm. We discover that the modified version actually maximizes the cosine similarity between gradients (proof in Appendix B.2). Thus, we call it Cosine Similarity Encourager (CSE), which can be further combined with MI as MI-CSE. MI-CSE involves an inner momentum to accumulate the gradients of each model. We provide the pseudocode in Appendix B.2.
3.5 COMMON WEAKNESS ATTACK
Given the respective algorithms to optimize the flatness loss landscape and the closeness between local optima of different models, we thus need to combine them unified Common Weakness Attack (CWA) to achieve better transferability. In consideration of the feasibility parallel gradient backpropagation and time complexity, we substitute the second step of SAM with CSE, and the resulting algorithm called CWA. We also combine CWA with MI to obtain MI-CWA, with the psuedocode shown in Algorithm 1. CWA can also be incorporated with other strong adversarial attack algorithms, including VMI (Wang Hc, 2021), SSA (Long et al., 2022), etc. The details of these algorithms provided in Appendix B.3.
4 EXPERIMENTS
In this section, we conduct comprehensive experiments to show the superiority of our methods. The tasks range from image classification to object detection and image description with the recent large vision-language models, demonstrating the universality of our methods.
4.1 ATTACK IN IMAGE CLASSIFICATION
Similar to previous works, adopt the NIPS2017 dataset which is comprised of 1000 images compatible with ImageNet (Russakovsky et al., 2015). All the images resized to 224 224. (2) Surrogate models: We choose four normally trained models ResNet-18, 18, ResNet-32, ResNet-50, ResNet-101 (He et al., 2016) from Torch (Marcel & Rodriguez, 2010), and two adversarially trained models ResNet-50 (Salman et al., 2020). XCIT-S12 (Debenedetti et al., 2023) from RobustBench (Croce et al., 2021). They contain both normal and robust models, which is effective assess the attacker's ability utilize diverse surrogate models. We further conduct experiments with other surrogate models and other settings (e.g., using less diverse surrogates, settings in Dong al. (2018), settings in Naseer et al. (2020), attacks under 4/255) in Appendix C. (3) Black-box models: We choose a model from each model type available in Torch Vision and RobustBench as the black-box model. We also include 7 other prominent defenses in our evaluation. In total, there are 31 models selected. For further details, please refer to Appendix C.I. (4) Compared methods: We integrate our methods into MI (Dong et al., 2018) as MI-SAM, MI-CSE and MI-CWA. We also integrate CWA into two tfte-offttheart methods VMI (Wang & He, 2021) and SSA (Long et al,, 2022) as VMI-CWA and SSA-CWA. We compare them with FGSM (Goodfellow
Experimental settings. (1) Dataset:

---

Published as a conference paper at ICLR 2024
Table 1: Black- -ox attack success rate (%, 1) on NIPS2017 dataset. While our methods lead the performance on 16 normally trained models with various architectures, they significantly empower the transfe-bbase attack with great margins on 8 adversarially trained models available on RobustBench (Croce et al., 2021). The model details are provided in Appendix C.I.
Meibod
Baackhoe
FGSM
BIM MI DI
TI
VMI SVRE:
PI SSA RAP
MESSAM
ML-CSE
MI-CWA
VMI-CWA
SSA-CW
AlexNet
7h.4
54.9
73.2 78.9
78.0
83.3
82.5
78.2
89:0
82.9
81.0
93.6
94.0
95.9
96.9
VGG-16
68.9
86.1
91.9 92.9
82.5
94.8
96.4
93.1
97.7
93.1
95.6
99.6
99.5
99.9
99.9
GoogleNet
54.4
76.6
89.1 92.0
77.8
94,2
95.7
91.0
97.2
90.4
94.4
98.8
99.0
99.8
99.8
Inception. V3
54.5
64.9
84.6 89.0
75.7
91.1
92.6
85.9
95.6
85.0
89.2
97.3
97.2
98.9
99.6
ResNet-152
54.5
96.0
96.6
99.0
97.2
97.6
95.3
97.9
99.9
99.8
100.0
DeaseNet-12 121
57.4
93.0
95.8 93.8 88.0 96.6
99.1
96.9
98.2
94.1
98.0
99.9
99.8
99.9
100.0
SqueezeNet
85.0
80.4
89.4 92.9 85.8
96.1
97.2
92.1
94.1
99.
99.5
99.6
99.8
Normal
MobileNet-V3
ShutfleNet-V2
58.9
81.2
55.6
65.3
79.9 85.7
71.8 78.6
78.2
74.5
89.9
87.3
90.3
80.6
71.1 91.4
85.8
93.9
81,1
89.3
87.9
80.7
97.2
94.6
95.7
97.3
97.8
98.7
98.8
98.1
BetiieenNet-B0
50.8
80.2
90.1 91.5
76.8
94.6
96.7
98.3
9699
91.4
95.2
98.5 8
98.9
99.7
99,9
MNasNett
64.1
80.8
88.8 91.5
75.5
94,1
94.2
90.3
97.2
92.5
94.3
99. 1
98.7
99.6
99.9
RegNarX-400MP
57.1
81.1
89.3 91.2
82.4
95.3
95.4
91.0
97.4 90.8
93.9
98.9 9
99.8
99.9
Con<NeXi-T
39.8
68.6
81.6 85.4 56.2
92.4
88.2
85.7
93.1
86.8
90.1
96.2
95.6
97.8
98.1
VIT-B/16
33.8
35.0
59.2
66.8
56.9
81.8
65.8
64.5
K3.0
66.7
68.9
89.6
89.6
92.3
90.0
Swin-S
34.0
48.2
66.0
74.2
40.9
84.2
73.4
69.1
85.2
72.2
75.1
88.6 6
87.6
91.6
88.4
MaTV
31.3
49.7
66.1
73.2
32.7
83.5
71.1
70.1
K5.2
69.7
75.6
85.8
85.9
S8.1
86.1
MAA
Incepeion.V
539
43.4
55.9
61.8
66.
66.8
61.1
84.3
09.6
644.5
89.6
91.3
92.7
LasAI
Inckes- V2
32.5
28.5
42.5
52.9
SK.5
66.4
46.8
45.1
76.1
48.6
47.9
78.2
79.1 1
83.2
84.1
FaSEAT
ResNer 50
45.6
41.6
45.7
47.1
49.3
51.4
51.0
33.1
14.7
56.5
50.6
75.0
74.6
73.5
70.4
PGDAT
30
36.3
30.9
37,4 38.0
43.9
47.1
43.9
23.0
25.3
$1.0
43.9
73.5
73.6
72.7
66.8
PGDAT
ResNet-18
46.8
41.0
45.7
47.7
50.7
48.9
48.5
39.0
41.1
55.5
48.0
68.4
69.5
69.2
65.9
WRN-50-2
27.7
20.9
27.8 31.3
37.0
36.2
33.0
17.9
18.7
41.2
33.4
64.4
64.8
63.1
55.6
PGDAT
XCI1-MI2
XCiT-L12
23.0
19.8
15.7
16.4
22.8
19.8
25.4
29.4
33.4
26.7
30.2
11.9
11.5
13.1
41.3
44.7
26.9
31.8
77.5
77.8
75.1
59.4
66.3
HGD
IncRes-V2
36
78.0
76.2 88.4
73.5
92.0
K5.5
79.2
93.9
79.0
87.9
95.6
95.6
98.2
98.7
R&P
ResNot-50
67.9
95.8
96.3 96.2
99.9
98.2
95,3
98.8
99.8
100.0
Bit
55
69.1
97.0
97.3 96.1
94. 99.0
99.9
98.8
99.5
97.1
99.4
100.0
100.0
100.0
100.0
JPEG
RosNet-50
68.5
96.0
96.3 95.4 93.5
98.6
99.5
97.6
99.2 96.0
99.4
99.8
99.9
100,0
100.0
RS
ResNet-50
60.9
96.1 95.6 95.6 89.9
96.9
99.3
96.4
98.
95.9
98.1
100.0
100.0
100.0
100.0
NRP
Re
36.6
88.7
72.4
63.1 71.7
89.0
91.2
81.3
92.8
33,3
87.3
88.1 1
86.8
33.1
85.4
DitPPure
|ResNet-50
50.9
68.5
76.0
82.0
87.1
87.7
93.4
79.6
85.6
97.5
et al., 2015), BIM (Kurakin et al., 2018), MI (Dong et al., 2018), DI (Xie al., 2019), TI (Dong et al., 2019). VMI (Wang& & He, 2021). PI (Gao et al., 2020), SSA (Long et al., 2022), RAP (Qin et al.. 2022) and SVRE (Xiong et al.. 2022). All these attacks adopt the logits ensemble strategy following Dong et al. (2018) for better performance. Except FGSM and BIM, all algorithms have been integrated with MI. (5) Hyper-parameters: We set the perturbation budget = 16/255, total iteration T = 10, decay factor μ = 1, step sizes 3 = 50, r = 16/255/12 15, and a = 16/255/5. For compared methods, we employ their optimal hyper-parameters as reported in their respective papers.
Results on normal models. The results of black-box attacks are shown in the upper part of Tab. 1. Both MI-SAM and MI-CSE greatly improve the attack success rate compared with MI. Note that MI-CSE improves the attack success rate significantly if the black-box model is similar to one of the surrogate models. For example, when attacking ViT-B/16, the attack success rate of MI-CSE is nearly 30% higher than MI. This is because MI-CSE attacks the common weakness of the surrogate models, so as long as there is any surrogate model similar to the black-bor box model, the attack success rate will be very high. While other methods cannot attack the common weakness of surrogate models, and the information of the only surrogate model which is similar to the black-box model will be overwhelmed by other surrogate models. Note that MI-SAM increases the attack success rate consistently, no matter whether the surrogate models are similar to the black-box model or not. This is because MI-SAM boosts the transferability of adversarial examples via making them converge into flatter optima.
When incorporating the MI-SAM into MI-CSE to form MI-CWA, the attack success rate is further improved. Because MI-SAM and MI-CSE aim to optimize two different training objectives, and they are compatible as shown in Theorem 3.2, the resultant MI-CWA can not only make adversarial examples converge into a flat region, but also close to the optimum of each surrogate model. Therefore, MI-CWA further boosts the transferability of adversarial examples.
By integrating our proposed CWA with recent state-of-the-arr attacks VMI and SSA, the resultant attacks VMI-CWA and SSA-CWA achieve a significant level of attacking performance. Typically, SSA-CWA achieves more than 99% attack success rates for most normal models. VMI-CWA and SSA-CWA can also outperform their vanilla versions VMI and SSA by large margin. The results not only demonstrate the effectiveness of our proposed method when integrating with other attacks, but also prove the vulnerability of existing image classifiers under strong cansfer-based attacks.
Results on adversarially trained models. As shown in Tab. 1, since our method attacks the common weakness of surrogate models, we can make full use of the information of XCIT-S12 (Debenedetti
7

---

Published as a conference paper at ICLR 2024
Table 2: mAP 1%, J of black-hox detectors under attacks INRTA dataset. The universal adversarial patch trained on YOLOv3 and YOLOv5 by Adam-CWA achieves the lowest mAPs on multiple modern detectors (9.85 on average) with large margins.
Method
Surmgate
YOLOv2 YOLOv3
YOLOvv-T YOL.0v4 YOLOv4-T
YOLOO5
FasterRCNN
Avg.
Single
YOLOv3
54.63
12.35
53,99
58.20
53.38
69.21
50.81
58.13
51.34
Single
YOLOv5
30.45
34.17
33,26
53.55
54.54
7.98
37.87
37.00
36.10
Loss Ensemble
YOLOv3+YOLOW
25.84
8.08
38,50
47.22
43.50
19.21
34.41
35.04
31.48
Adam CWA
YOLOv
6.59
2.32
8.44
11.07
8.33
2.06
14.41
25.56
9.85
Misdescription
Blindness
Refusal
imae
8
image.
8
Descrbe
8
The image you sent me a lone-up of pensan hand hokcing a hot dog. The hor dog in asft, betohup clean has
I canner desoriee be the inage becsuse I connet see
Serry, help withiinages people
topped
trages. proceee descrighion could
the snurkles The fiegens slightly curled around the but dog, as the persor is about to take a bite.
the may able help you.
Figure 3: Examples of transfer-based black-box attacks against Google's Bard.
et al., 2023) to attack other adversarially trained XCIT models, without being interfered by four normally trained ResNets in the ensemble. Because FastAT (Wong et al., 2020) and PGDAT (En- gstrom et al., 2019; Salman et al., 2020) are adversarially trained ResNets, our algorithm significantly improves the attack success rate by 40% over existing methods via making full use of the information of ResNets and defense models in the ensemble.
Results on other defense models. From Tab. 1, we observe that our algorithm significantly boosts the existing attacks against ofute-of-thh-ar defenses. For instance, when integrated with SSA (Long et al., 2022), our method attains a 100% attack success rate against randomized defenses, and 97.5% % against the recently proposed defense DiffPure (Nie et al., 2022). These findings underscore the remarkable efficacy of our proposed method against defense models.
4.2 ATTACK IN OBJECT DETECTION
Experimental settings. (1) Dataset: We generate universal adversarial patches on the training set of the INRIA dataset (Dalal & Triggs, 2005) and evaluate them on black-box models on its test set, presenting challenges for adversarial patches to transfer across different samples and various models. (2) Surrogate models: We choose YOLOv3 (Redmon & Farhadi, 2018) and YOLOv5 (Jocher, 2020) as our surrogate models. (3) Testing models: We evaluate adversarial patches on 2 white-box models mentioned above and black-box object detectors, including YOLOv2 (Redmon & Farhadi, 2017), YOLOv3-T (Redmon & Farhadi, 2018), YOLOv4, YOLOv4-T (Bochkovskiy et al., 2020), FasterRCNN (Ren al., 2015), and SSD (Liu et al., 2016). (4) Compared methods: Since the Adam method is often used to optimize adversarial patches for object detection (Thys et al., 2019; Zhang al., 2023), we first combine the proposed CWA with Adam to form Adam-CWA and compare its performance with patches trained by YOLOv3 only, YOLOv5 only, and Loss Ensemble of YOLOv3 and YOLOv5 (Enhanced Baseline Huang et al. (2023)). (5) Training Procedure: We follow the "Enhanced Baseline" settings in Huang et al. (2023) including the image size, detector weight, the way to paste the patch on the image, learning rate, etc.
Results. As shown in Tab. 2, our method exceeds the Loss Ensemble method by 20% average all models and has a larger margin compared to patches generated on one detector, is noticeable that the universal adversarial patch generated by Adam-CW: has even lower mAPs (2.32 on YOLOv3 and 2.06 on YOLOv5) on the two white-box box models compared with results of white-box attacks (12.35 on YOLOv3 and 7.98 on YOLOv5), which means that method boosts the transferability of the adversarial patch not only between different models, but also between different samples.
4.3 ATTACK IN LARGE VISION-LANGUAGE MODELS
Experimental settings. (1) Dataset: We randomly choose 100 images from the NIPS2017 dataset. (2) Surrogate models: We adopt the vision encoders of ViT-B/16 (Dosovitskiy al., 2020), CLIP
8

---

Published as a conference paper at ICLR 2024
de 291
ARtL d- SACCAAE DAAA (50)
12
22
3
10
10-
6.0
356
601
0.10
0.50
011
816
0.31
612
0160
0.15
3.64
16
25
15
Utnaass under Iafieity home
Dlense: ander tallway Norm
(a) MI
(b) MI-CWA
(c) T
Figure 4: Additional results. (a-b): The loss landscape around the convergence point optimized by
MI and MI-CWA respectively. (c): Attack success rate under different attack iterations of T.
(Radford et al., 2021), and BLIP-2 (Li et al., 2023) as surrogate models, using the attack objective as
max m - ) s.t. || xnnall S
(10)
where the distance between the features of ad adversarial example and the original image is maximized.
(3) Testing models: We test the attack success rate of our generated adversarial examples on Google's Bard?. (4) Hyper-parameters: We use exactly the same perrpaaamettrs as in Sec. 4.1.
Results. As shown in Tab. 3. SSA-CWA outperforms
Table 3: Black-box attack success rate (%,
SSA by large margin under both 16/255 and c= 32/ 32/255.. We observe that adversarial examples can lead to three wrong behaviors of Bard, including misdescription, blindness, and refusal, as shown in Fig. 3. SSA-CWA leads to 8% increase in mis- description compared to SSA. Moreover, SSA-CWA exhibits substantially higher rejection rate than SSA.
1) against Bard.
Attack
Misdescription Blindness Retusal
SSA
16/255
4
16
SSA-CWA
16/255
12
27
SSA
32/255
20
36
SSA-CWA
3222222
32/255
28
These results underscore the potent efficacy of our algorithm against cutting-edge commercial large models, emphasizing the imperative to develop defenses tailored to such models. More results be found in our follow-up work (Dong et al., 2023).
4.4 ANALYSIS AND DISCUSSION
In this section, we conduct some additional experiments to prove our claims about the effectiveness of our methods, regarding flattening the loss landscapes, encouraging the closeness between local optima, and computational efficiency compared to other algorithms.
Visualization of landscape. We first analyze the relationship between the loss landscape and the transferability of adversarial examples by visualizing the landscapes by different algorithms (detailed in Appendix D.2). As shown in Fig. 4(a) and 4(b), the landscape at the adversarial example crafted by MI-CWA is much flatter than that of MI. is also noticeable that the optima of the models in MI-CWA are much closer than those in MI. This supports claim that CWA encourages the flatness of the landscape and closeness between local optima, thus leading to better transferability.
Computational efficiency. When the total number of iterations T in Algorithm 1 remains the same, the proposed CWA algorithm requires twice the computational cost of MI, so it is unfair to compare with other algorithms directly. Therefore, we measure the average attack success rate among the 31 models in Tab. 1 of the adversarial examples crafted by different algorithms under different T. The result is shown in Fig. 4(c). Our algorithm outperforms other algorithms for any number of iterations, even if T = 5 of the CWA algorithm. This indicates that the CWA algorithm is effective not because of the large number of iterations, but because it captures the common weakness of different models.
5 CONCLUSION
In this paper, rethink model ensemble in black-box adversarial attacks. We engage in theoretical analysis, exploring the relationship among the transferability of adversarial samples, the Hessian matrix's F-norm, and the distance between the local optimum of each model and the convergence point. Stemming from these insights, we define common weaknesses and propose effective algorithms to find common weaknesses of the model ensemble. Through comprehensive experiments in both image classification and object detection, we empirically demonstrate that our algorithm excels in finding common weakness, thereby enhancing the transferability of adversarial examples.
Phttps://bar google..o.... gpogle.com

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
potential negative impact of our approach that malicious attackers could use our method to attack large commercial models, leading toxic content generation or privacy leakage. As people currently focus on improving big models due to their excellent performance, it's even more important to explore and address the vulnerability of deep learning models which could be targeted by black-box attacks without knowing specific details of the target models. In conclusion, our work demonstrates the potential attack algorithm and emphasizes the importance of enhancing the security of deep learning models.
ACKNOWLED
This work was supported by the National Natural Science Foundation of China (Nos. U2341228, 62276149, 22061136001, 62076147), BNRist (BNR2022RC0106), Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. Y. Dong was also supported by the China National Postdoctoral Program for Innovative Talents and Shuimu Tsinghua Scholar Program. J. Zhu was also supported by the XPlorer Prize.
REFERENCES
Zeyuan Alhu and Yang Yuan. Improved svrg for non-strongly-convex or sum-of sum-of-none of-non-convex objectives. In International Conference on Machine Learning, pp. 1080-1089, 2016.
Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy, 2017.
Huanran Chen, Shitong Shao, Ziyi Wang, Zirui Shang, Jin Chen, Xiaofeng Ji, and Xinxiao Wu. Bootstrap generalization ability from loss landscape perspective. In European Conference on Computer Vision, pp. 500-517, 2022.
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on Artificial Intelligence and Security, pp. 15-26, 2017.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference Machine Learning, pp. 1310-13200 1320, 2019.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 886-893, 2005.
Edoardo Debenedetti, Vikash Sehwag, and Prateck Mittal. A light recipe to train robust vision transformers. In 2023 IEEE Conference on Secure and Trustworthy Machine Learning, pp. 225-253, 2023.
Yinpeng Dong, Hang Su, Jun Zhu, and Fan Bao. Towards interpretable deep neural networks by leveraging adversarial examples. arXiv preprint arXiv: 05499, 2017.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 9185-9193, 2018.
Yinpeng Dong, Tianyu Pang. Hang Su, and Jun Zhu. Evading defenses transferable adversarial ex- amples by translation- innlattiooninvaiantt attacks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 4312-4321 2019.
10

---

Published as a conference paper at ICLR 2024
Yinpeng Dong, Huanran Chen, Jiawei Chen, Zhengwei Fang, Xiao Yang, Yichi Zhang, Yu Tian, Hang Su, and Jun Zhu. How robust is google's ' bard to adversarial image attacks? arXiv preprint htt.:2309.1175/// 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthis Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library). 2019. URL https: he///gttuuu com/ Saaddyylab/obbusuteee
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2020.
Lianli Gao, Qilong Zhang, Jingkuan Song. Xianglong Liu, and Heng Tao Shen. Patch-wis. attack for fooling deep neural network. In European Conference on Computer Vision, pp. 307-322, 2020.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamnsssing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. International Conference on Learning Representations, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenetv3. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 1314-1324, 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks, In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 4700-4708, 2017.
Hao Huang, Ziyan Chen, Huanran Chen, Yongtao Wang, and Kevin Zhang. T-sea: self-ensemble attack object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20514 20514-2000 1-5223,, 2023.
Transfer-base
Forrest N landola. Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and; 0.5 mb model size. arXiv preprint
2016.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Av- eraging weights leads to wider optima and better generalization. arXiv preprint arx.v: 2018.
Glenn Jocher. YOLOvS by Ultralytics, May 2020. ultralyticssyolovs.
URL https://github.cr
Kenji Kawaguchi, Jiaoyang Huang, and Leslie Pack Kaelbling. Every local minimum value is the global minimum value of induced model in nonconvex machine learning. Neural Computation, pp. 2293-2323, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, pp. 84-90, 2017.
Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In International Conference Learning Representations, 2016.
11

---

Published as a conference paper at ICLR 2024
Alexey Kurakin, Ian J Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial Intelligence Safety and Security, pp. 99-112, 2018.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pp. 5905-5914, 2021.
Thomas Laurent and James Brecht. Deep linear networks with arbitrary loss: All local minima are global. In International Conference on Machine Learning, pp. 2902-2907, 2018.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521((553)::436 444, 2015.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances Neural Information Processing Systems, 2018.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- pre- training with frozen image encoders and large language models. arXiv preprini 12X5: 2023.
Yingwei Li, Song Bai, Yuyin Zhou, Cihang Xie, Zhishuai Zhang, and Alan Yuille. Learning transferable adversarial examples via ghost networks. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 11458-11465. 2020.
Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against adversarial attacks using high-level representation guided denoiser. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1778-1787, 2018.
Jiadong Lin, Chuanbiao Song, Kun He, Liwei Wang, and John E Hoperoft. Nesterov accelerated gradient and scale invariance for adversarial attacks. In International Conference on Learning Representations, 2020.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, pp. 85 116, 2022a.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European Conference on Computer Vision, pp. 21 21-37, 2016.
Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10012-10022, 2021.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11976-1986,, 2022b.
Yuyang Long, Qilong Zhang, Boheng Zeng, Lianli Gao, Xianglong Liu, Jian Zhang, and Jingkuan Song. Frequency domain model augmentation for adversarial attack. In European Conference on Computer Vision, pp. 549-566 2022.
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings the European Conference Computer Vision, pp. 1/16-111,2 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
12

---

Published as a conference paper at ICLR 2024
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM International Conference on Multimedia, pp. 1485-1488, 2010.
Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self-supervised approach for adversarial robustness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 262-271, 2020.
Muzammal Naseer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. On generating transferable targeted perturbations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7708-7717, 2021.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint
2018.
Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Animashree Anandkumar. Diffusion models for adversarial purification. In International Conference on Machine Learning, pp. 16805-16827, 2022.
Zeyu Qin, Yanbo Fan, Yi Liu, Li Shen, Yong Zhang, Jue Wang, and Baoyuan Wu. Boosting the transferability of adversarial attacks with reverse adversarial perturbation. Advances in Neural Information Processing Systems, pp. (8845-29858, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763, 2021.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10428-10436, 2020.
Joseph Redmon and Ali Farhadi. Yolo9000:
better, faster, stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7263-7271, 2017.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint a8Xiv: 02767, 2018
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, pp. 211-252 2015.
Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adver- sarially robust imagenet models transfer better? In Advances in Neural Information Processing Systems, pp- 3533-3545, 2020.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Accessorize to a crime: Real and stealthy attacks state-of-the-art face recognition. In ACM Sigsac Conference Computer and Communications Security, pp. 1528-1540, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint arXiv:1409.1556 2014.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-9, 2015.
13

---

Published as a conference paper at ICLR 2024
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-28266 2016.
Mingxing Tan and Quoe Le. Efficientner: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 14, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and
Quoc Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2820-2828, 2019.
Simen Thys, Wiebe Van Ranst, and Toon Goedemé. Fooling automated surveillance cameras:
adversarial patches to attack person detection. In Proceedings of the IEEE/CVF conference on Computer Vision and Puttern Recognition Workshops, 2019.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. International Conference on Learning Representations, 2018.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Normalized flat minima: Exploring scale invariant definition of flat minima for neural networks using pac-bayesian analysis. In International Conference on Machine Learning, pp. 9636-9647, 2020.
Stephen Tu, Ross Boczar, Max Simchowitz, Mahdi Soltanolkotabi, and Ben Recht. Low-rank solutions of linear matrix equations via procrustes flow. International Conference on Machine Learning, pp. 964-973, 2016.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. In European Conference on Computer Vision, pp. 459-479, 2022.
Vladimir Vapnik. The nature of statistical learning theory. 1999.
Xiaosen Wang and Kun He. Enhancing the transferability of adversarial attacks through variance tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Xiaosen Wang, Jiadong Lin, Han Hu, Jingdong Wang, and Kun He. Boosting adversarial transferabil- ity through enhanced momentum. In British Machine Vision Conference, 2021.
Zeming Wei, Yifei Wang, Yiwen Guo, and Yisen Wang. Cfa: Class-wise calibrated fair adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8293-8201, 2023a.
Zeming Wei, Yifei Wang, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint arXiv:2310.0638 2023b.
Zeming Wei, Jingyu Zhu, and Yihao Zhang. Sharpness-aware minimization alone can improve adversarial robustness. In The Second Workshop on New Frontiers in Adversarial Machine Learning, 2023c.
Eric Wong, Leslie Rice, and Zico Kolter. Fast better than free: Revisiting adversarial training. In International Conference Learning Representations, 2020.
Lei Wu, Zhanxing Zhu, et al. Towards understanding generalization of deep learning: Perspective of loss landscapes. arXiv preprint arXiv: 7066. 10239, 2017.
Cihang Xie, Jianyu Wang, Zhishuai Zhang, Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In International Conference on Learning Representations, 2018.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferability adversarial examples with input diversity. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 2730-2739, 2019.
14

---

Published as a conference paper at ICLR 2024
Yifeng Xiong, Jiadong Lin, Min Zhang, John E Hopcroft, and Kun He. Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferabilityy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14983-14992, 2022.
Zhuolin Yang, Linyi Li, Xiaojun Xu, Shiliang Zuo, Qian Chen, Pan Zhou, Benjamin Rubinstein, Ce Zhang, and Bo Li. Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances Neural Information Processing Systems, pp. 17642-17655, 2021.
Hongkun Yu, Chen Chen, Xianzhi Du, Yeqing Li, Abdullah Rashwan, Le Hou, Pengchong Jin, Fan Yang. Frederick Liu, Jaeyoun Kim, et al. Tensorflow model garden. 2020. URL https://github. com/lensoofllow/modlls, 2020.
Yichi Zhang, Zijian Zhu, Hang Su, Jun Zhu, Shibao Zheng, Yuan He, and Hui Xue. To make yourself invisible with adversarial semantic contours. Computer Vision and Image Understanding, pp- 103659, 2023.
Yihao Zhang, Hangzhou He, Jingyu Zhu, Huanran Chen, Yifei Wang, and Zeming Wei. On the duality between sharpness-aware minimization and adversarial training. arXiv preprint Xiv:2402.15//5 2024.
15

---

Published as a conference paper at ICLR 2024
A PROOFS AND DERIVATIONS
A.1 PROOF OF THEOREM 3.1
With Hölder's inequality, we can prove this theorem:
Proof.
[[(x p.) TH H.(x P:)]
1 - 1
Ell pt) 1p| H p.)lla (where
1)
p q
< [[]l(( - pt) |p| H. -
E H; H, |
-p))lll
where || Ilr is an induced matrix norm.
Special case: When p q 2, we have
[ p.) H:(r (n - p.)] S R|| H. 22] p:)112),
where [HHl is the spectral norm of H,, As we also have [H; 1 S HHlle, we obtain
E[(x= p.) H P.) S E H Bill P.)
where H; is the Frobenius norm of H;
Note that both the spectral norm and Frobenius norm of the Hessian matrix correspond to the flatness of loss landscape, such that we can adopt sharpness aware minimization for optimization.
A.2 THE GENERALIZATION ABILITY OF COMMON WEAKNESS
Let c be the final optimum of the training objective ΣnEr, n (((((())))) y) that we converge to, and let the optimum of the testing objective I2 [, y) nearest to c. Let Os, be the set of closest optimum of model
Fe to C₁ and let O₂ be the of closest optimum of model f. F to c. Let the hessian matrices at P: E Og form a set He, and the hessian matrices at p: E Oc, form set He. To find the optimum with a smaller EH. [| ]] and Ep.eo.lll(e (c P:) at test time, need to optimize EH,eH,, | and Ep.eo., lI(c p.)1 2 at training time. In the following, we will show that both terms have strong relationship between the training objective and the test eиror. Therefore, optimizing the training objective will lead to a smaller test error, consequently, smaller generalization error.
Several studies (Li et al., 2018; Wu et al., 2017: Chen et al., 2022; Zhang et al., 2024) have shown that the || H F term corresponds the flatness of the landscape. flatter landscape during training has consistently been associated with improved generalization ability, which pertains to the transferability when training adversarial examples. The [|(c - p: term, which measures the distance between the local optima, also exhibits strong relationship between the training and testing phases. Intuitively, assuming that (c - P.) follows some distribution with the existing variance, the training objective is merely empirical observation from the testing objective. Thus, the testing objective can be probabilistically bounded by the training objective through a convergence bound like the Chebyshev inequality, i.e.,
Var C; Pt (n 1)282
P( n
c. - -p:113 Ep, -p) > 83
In other words, as long as Varl p exists, the smaller the training error, the higher the probability that the test error are smaller.
1p
In the following, we consider simple case (Assumption A.1) where the local optima are Gaussianly
distributed, the following theorem shows that smaller values 1 Ep. [I(,, - -p) tend to lead
to smaller values of Ep. -P:)]l3-
16

---

Published as a conference paper at ICLR 2024
Assumption A.1. The optimum of each model p; in Og follows Gaussian distribution with and unknown variance o². Meanwhile, the optimum of the ensemble training model, C₁ given by the mean of all p; within Oc. That is:
Vp p N(c,o²1);
C = E
PiE0s
Theorem A.2. Denote Fmm, n) F-distribution with parameter and Fg (m,n) as the upper alpha quantile, For any two different optimum of ensemble model cº and c² and corresponding
si 1 n Ep.o. (p: c) 22 Ep.c0 (p: c)) if Fa(n 1, 1), then Ep.co. [ll(e1 p:) E0 [ll(e2 P.)
(A.1)
holds with rype one evvor of a.
This theorem suggests that when the optima of surrogate models are closer, as indicated by a smaller value of n Dp.eo EO., II(c p:)ll the optima of the target models also tend to be closer, which is represented by a smaller value of Ep. - [p))))))
Proof. The training models F; can be viewed as sampling from the of all models F. Therefore, the sample variance x² is:
8² 71 P.EOa,
Σ cp)
Because that no
follows chi-square distribution, that is:
p: ct)
Σ
Pp:
Σ p:)a 5=1
~ no² X22 1).
1=1
Consequently, follows F distribution.
of 02
F(n 1,n-11 1).
We perform F-test here. We set the null hypothesis Hu {or 02). and the alternative hypothesis H1 (ar 2 02). Thus, if 2 Fa(n
1.n 1), with type one enror of a:
22 +
This indicates that smaller value of 1 Σm (p: c2 tends to indicate a smaller value of a. Next, we will demonstrate that smaller implies enhanced transferability of our adversarial examples.
Lemma A.3. Ep.Eo, (cc p:) is function that monotonically increases with a. That is:
Eo. [ 2
Do
Reparameterize p, oe, c, where e; N(O, 1). We can get:
Ep.eo. P:)
E Eo-No.n
d B B
Therefore 2 E de Ep.eo,
[](cc po) 2 Importantly, the expectation of any norm of c P; mono- tonically increases with the parameter not limited solely to the second norm.
Therefore, both |H and lpp phase, indicating that encouraging the flatness of the landscape and closeness local optima can result in improved generalization ability.
2l12 exhibit strong correlation between the training and testing
17

---

Published as a conference paper at ICLR 2024
Algorithm 2 MI-SAM
Require: natural image ænat, label y, perturbation budget c, iterations 7', loss function L, model ensemble Fr {ff) decay factor μe, step sizes y; / and a.
1: Initialize: m = 0, Ty Inati
2: fort= = 0 to 7
do
3: Calculate g = 2= Ei=1 f((w)),))) y):
4: Update æ; by xi clip anoee (at + r sign(g)):
5: Calculate g T E v.
6:
Update x by अ/ cliponatir (x)
sign(g));
7: Calculate the update y x;
Update momentum m P m g:
9: update 2c+1 by #++1 = clip clipanat.c (e: + a (m))):
10:
end for Return:
11:
A.3 PROOF OF THEOREM 3.2
In this section, aim prove that dot product similarity between gradient of each model is the upper bound of 1 n. - -p.2 p.)².
Using Cauchy-Swand theorem,
get:
n Σ (c PI)l = Σ (H,' g.) (H, 9:)= > II g.)11 S Σ H 19.1..
n
11
The treatment of H has already been discussed in Appendix B.1. In this section, set M as the maximum value of |H; l1,,, which allows obtain the following result:
Σ I(c p.) S M 9 0. 9: = M 9)))²
2
9:96
i=1
1=1
im j=
Since e is the optimal solution for the ensemble model, we have final training objective is:
(Ein 2, 0. Consequently, our
max
9:9
(A.2)
Which is the dot product similarity between the gradient of each model. However, maximizing this dot product similarity can lead to gradient explosion, making it incompatible with MI-SAM. Therefore, we opt to maximize the cosine similarity between the gradients of each model.
The equality sign holds when p. for all This condition implies that the local optimum for each model is identical, which results in dot product similarity of 0 between the gradients.
B ALGORITHMS
B. SHARPNESS AWARE MINIMIZATION ALGORITHM UNDER INFINITY NORM
In this section, for simplicity, represent E F, by L(x). Our training objective is slightly different from Foret et al. (2020). We emphasize only the flatness of the landscape, without simultancously minimizing L(x). This approach yields better results:
min max [ollieke
L(x + 0 - L(x)).
(B.1)
Although the Frobenius norm the Hessian matrix | could also be a potential choice, requires the computation of third-order derivatives with respect p, which is computationally expensive.
18

---

Published as a conference paper at ICLR 2024
Algorithm 3 MI-CSE algorithm
Reguire: natural image &nat, label y, perturbation budget c, iterations 7' loss function L, model ensemble Fi { t=1, decay factor μa, step sizes and a.
1: Initialize: m 0, inner momentum m = 0, æo = anati
2: for 0
7 do
3: for to do
4: Calculate y
[= )).):
Update inner momentum by = μ 11012
8
6:
Update a: by क = elipanat.e (ar 1- rm));
7:
end for
8:
Calculate the update g x/ 21;
Update momentum m μ m g:
10: update #1+1 by 2++1 = clipanat. anot (et a sign(m)))
11:
end for
12:
Return: TT.
To optimize objective Eq. (B.1), first, we need to compute 8. We use the Taylor expansion to get
arg max L(x + 6)
~ arg max L(x) + ō TT sign V (xx) [
arg max 6 sign (zL(())(()) 1
sign (sig((V[(<z))).
Then, get the derivative of Va 6):
VaL(w 8) VatoL(x 8) Vx 8) V ox 8 VetoL( 8) v. (B.2)
Following previous works (Foret al., 2020; Wei et al., 2023c), discard the second term of Eq. (B.2) as has negligible influence on the optimization of the SAM algorithm.
In general, the procedure of the SAM algorithm under the infinity norm involves first finding the worst case perturbation ö, followed by optimizing L(x + 6) L(x). The detail at Algorithm 2.
B.2 DERIVATION OF COSINE SIMILARITY ENCOURAGER
In this section, our objective is to develop an algorithm that maximizes the cosine similarity between the gradients of each model. Directly pursuing this goal necessitates a second-order derivative with respect to the adversarial examples. However, this is both time-intensive and memory-intensive. Inspired by Nichol al. (2018), we design our Cosine Similarity Encourager, as shown in Algorithm 3.
In the following sections, we will demonstrate that this algorithm effectively maximizes the cosine similarity of the gradient between each model.
Theorem B.1. Denote as (aL(f(()), y) and g, as (aL(ff(x).))
y). When 3 + O, and the 9:g,
llg lla
negligible, updating by Algorithm will increase the cosine similarity of the gradient between each model.
Proof. The derivative of cosine similarity between gradient of each model can be written as:
9:9;
H.
I
9:9.
g,
H,
I
9,9
gi
Fx
9: 1 9
1g
11g.11
19g,
l1
1g
l2
|9
1gg
1
Because Tgg
is negligible, which is verified in Appendix D.I, the gradient can be approximated as:
0
9:9
H.
9i
H,
gi
(B.3)
0x
9
12
1g
ll
Igg
12
1|g
21
l|g:l12
19

---

Published as a conference paper at ICLR 2024
Because i.j only represent the i-th model and j-th model in Fi, and can permute the set Fe. therefore we can get that:
H.
9j
H 9:
E
|1
1
=
E
|1g
l2
11
Therefore the expectation of the derivative is:
9:93
H
9j |1
E
2E
Ox 11g:
9 1l
[llg: ll
In the following, we will prove that the update of Algorithm contains the gradient of vanilla loss ((f( y) and the derivative of the cosine similarity between the gradients of each model, which is shown Eq. (B.3).
Denote x' as the adversarial example after the i-th iteration in the inner loop of the Algorithm 3. g the gradient at i-th iteration in the inner loop, we can represent g by gi using Taylor expansion:
g gi + H2 x²)
9
gi 3
| ||
j=1
i-1
9j + o(3)
9: HH; 1=1
99 0
9j + 0(8
gi 3H,
9 12
The update over the entire inner loop is:
g
x
8
1=1
Substitute gf with - 3H, Σ j=
9.
+ 0(32).
11aill2 with llg: + 0(8) =
119: we can get:
las la
x"'
E[B
9
3H,
g
0(33)1/119/112
| 2
= B
91
3
H
0(33)
1:
1|
1
1=1
1=1
f=1
t=1
82
i
9:9;
3E
91
1119:12
Σ
O(8*).
99
||
2
dæ
1,
1=1
Consequently, using Algorithm 3 will enhance the cosine similarity between the gradients of each model. For the gradient ascent algorithm, we have two options: either multiply the training objective by -1 and address it as gradient descent issue, or develop an algorithm tailored for gradient ascent.
B.3 GENERALIZED COMMON WEAKNESS ALGORITIM
The CWA algorithm can be incorporated with not only MI, but also other arbitrary optimizers and attackers, such as Adam, VMI, and SSA. This is outlined in Algorithm 4. For each optimizer, the step function takes the gradient with respect to z and performs gradient descent. For the generalized Common Weakness algorithm, we require a total three optimizers. It's important to note that these three optimizers either be of the same type different types.
20

---

Published as a conference paper at ICLR 2024
Algorithm 4 Generalized Common Weakness Algorithm (CWA)
Require:
image æo; label y; total iteration T;; loss function L, model set Fromn momentum weight
μı: inner optimizer 3, reverse optimizer r, outer optimizer
Calculate the number of models
for t = 1 to T do
0 - copy(())
# first step
calculate the gradient g -V ( f((x)
update by r.step(-9)
second step for j to do
pick the jth model fj
calculate the gradient g (=L(()()),
update æ by S.step( Talll
end for
calculate the update this iteration = = - update by a.step(g)
end for
return
C ADDITIONAL EXPERIMENTS
C. MAIN EXPERIMENT SETTINGS
Target Models:
We evaluate the performance of different attacks on 31 black-box models, including 16 normally trained models with different architectures AlexNet (Krizhevsky et al., 2017). VGG-16 (Simonyan & Zisserman, 2014). GoogleNet (Szegedy et al., 2015), Inception- V3 (Szegedy et al., 2016), ResNet-152 (He al., 2016), DenseNet-121 (Huang al., 2017), SqueezeNet (landola et al., 2016), ShuffleNet-V2 (Ma et al., 2018), MobileNer-V V3 (Howard et al., 2019), EfficientNet-B3 (Tan & Le, 2019), MNasNet (Tan al., 2019), ResNetXX4400MF (Radosavovic et al., 2020), Con (Liu et al., 2022b), ViT-B/16 (Dosovitskiy et al., 2020), Swin-S (Liu et al., 2021). Max ViT-T (Tu al., 2022), and adversarially trained models (Madry al., 2018; Wei al., 2023a) available on RobustBench (Croce et al., 2021) FGSMAT (Kurakin et al., 2016) with Inception- V3, Ensemble AT (EnsAT) (Tramèr et al., 2018) with ResNet-50, PGDAT (Engstrom et al., 2019; Salman et al., 2020) with ResNet-50, ResNet- 18, Wide- ResNet-50-2, a variant of PGDAT tuned by bag-of-tricks (PGDAT') (Debenedetti al., 2023) with XCIT-MI2 and XCIT-L12. Most defense models state-of-the-att 2021). Regarding defenses other than adversarial training, we consider defenses (i.e., HGD (Liao al., 2018), R&P (Xie et al., 2018), Bit BeiduepthRdduction in Guo al. (2018)), JPEG (Guo et al., 2018), RS (Cohen et al., 2019), NRP (Naseer et al., 2020), DiffPure (Nie et al., 2022)) that are robust against black-box attacks.
-eeptiionRReNeet-Wee V2, FastAT (Wong et al., 2020) with
RobustBench (Croce et al.,
C.2 COMPARISON WITH NASEER ET AL. (2021)
In order to show that method works well under different surrogate models and target models, supplement an experiment following the setting in Naseer et al. (2021):
Table C.I: Attack success rate (%,1).
Surrogates
Methods
VGG19
Dense121
Res50
Res152
WRN50-2
Naseer et al. (2021)
97.34
71.41
71.68
50.78
48.03
Vens
MI-CWA
100.00
94.50
94.30
81.50
89,20
Naseer et al. (2021)
76.96
96.25
88.81
83,48
81.85
Dens
MI-CWA
99.60
99,90
99.80
99.50
99.60
Naseer et al. (2021)
90.43
94.39
96.67
95.48
92.63
Rens
MI-CWA
99.50
99.70
100.00
100.00
99.90
21

---

Published as a conference paper at ICLR 2024
Table C.2: Black-box attack success rate (%, 1) on NIPS2017 dataset. Surrogates are ResNet18,
ResNet32, ResNer50 and ResNet101.
Method
Backbone
FGSM
BIM
MI
T1
VMI
SVRE
P1
SSA
RAP
MI-SAM
MI-CSE
MI-CWA
SSA-CWA
AlexNet
82.6
83.1
92.7
VGG-16
71.3
91.0
90.8
95.8
84.7
97.0
97.3
91.0
99.1
94.1
96.7
99.1
99.4
100.0
GoogleNet
58.8
88.4
87.7 95.4
79.6
96.7
96.5
88.3
98.8
90.5
94.2
98.3
98.4
99.9
Incuption-V3
58.7
79.0
81.6
91.8
77.1
94.1
92.2
80.9
97.0
84.0
88, 1
92.4
98.7
ResNet-152
60.2
96.6
96.1
96.8
89.9
98.5
99.5
96.4
99.1
96.2
99. 1
100.0
100.0
100.0
DenseNet-121
96.2
95.2
96.0
95.0
99.8
100.0
SqueezeNet
86.4
89.8
90.2
95.7
86.7
96.2
96.7
93.2
98.6
93.7
94.8
98.0
98.6
99.8
Normal
ShuffleNet-
78.8
95.1
91.5
97.8
MobileNet-V3
60.6
65.6
69.5 80.6
77.3
89.7
78.2
80.4
92.2
77.1
78.2
80,2
79.9
95.7
EfficienNNet BO
55.6
89.3
87.7
94.7
77.9
96.8
96.9
86.6
98.8
92.5
95.5
98.1
97.9
99.9
MNasNet
66.8
87.1
84.8
93.5
75.6
96.4
95.0
84.3
98.1
92.1
94.8
97.5
99.9
RegNetX-400MP
60.3
87.9
86.8
94,9
86.0
97.2
94,9
89.3
98.7
91.3
94,4
97.6
97.5
100.0
Con-NeXi-T
VTT B/16
37.0
51.7
52.9
64.3
53.8
81.7
60.5
55.8
81.4
50.2
61.9
48.8
46.6
71.7
Swin-S
62.3
72.4
39,4
70.4
48.9
61.5
70.8
59.5
80.7
Mix
34.1
63.0
61.6
73.1
31.9
85.1
68.6
43,9
86.7
58.9
70.5
56,7
54.5
79.0
FGSMAT
Inception- V3
55.3
51.2
55.2
59.2
65.6
73.9
61.1
56.2
84.4
39.6
58.6
GT.1
G0.3
78.2
EnsAT
IncRes
52.5
74.7
FastAT
ResNer-30 50
45.2
42.3
44,6
44,2
46.7
47.5
45.3
48.5
50.4
46.6
45.3
46,2
46.2
49.6
PGDAT
KesNct-50
31.2
34.1
35.1
38.4
41.3
35.7
42.0
43.7
36.5
36.1
35.7
40.3
PODAT
ResNet-18
47.1
42.0
45.4
45.3
47.6
47.2
45.6
50.4
50.6
47.8
45.8
46.8
46.9
50.7
WRN-50-2
28.1
22.7
25.6
27.4
31.2
32.6
27.3
33,9
35,3
27.7
28.0
27.6
26.7
31.7
PGDAT
XCT
21.7
16.8
18.5
19.9
22,9
25.0
20.7
25.3
28.1
22.0
21.3
21.2
26.5
XCIT-L12
18.7
14.9
16.3
17.3
20.9
21.6
18.6
22.3
26.0
18.4
18.4
18.5
18.2
22.6
HGGD
77.4
73.0
17.9
97.8
R&P
ReaNet-50
69.7
97.5
96.8
97.9
94.3
98.8
99.7
97.7
99.6
96,9
99.3
100.0
100.0
100.0
Bii
RexNet-50
73.2
98.4
98.4
97.7
96.6
99.1
99.9
99.2
99.7
98.7
99.6
100.0
100.0
100.0
JPFG
ResNet-50
70.5
98.1
97.8
97.2
96.3
98.9
98.5
99.7
98.1
99.5
100.0
100.0
100.0
RS
ResNet-50
66.7
97.2
96.7
97.3
93.1
98.7
99.5
97.0
99.5
96.6
99.4
100.0
100.0
100.0
NRP
ResNet-50
90.4
76.0
82.6
34.1
25.1
36.0
DiffPure
ResNei-50
57.7
68.6
75.0
84.1
88.6
95.9
85.8
89.4
96.0
76.4
84.9
81.2
79.5
95.0
Here, Vens represents the ensemble comprising VGG-11, VGG-13, VGG-16, and VGG-19. Rens denotes the ensemble of R8sNet-11, ResNet-50, ResNet-101, and ResNet-152. Similarly, Dens corresponds to the ensemble of DenseNet- 121, DenseNet-161, DenseNet-169, and DenseNet-201. In this configuration, surrogate models exhibit high similarity. Therefore, effective to assess the algorithm's 's capacity for generalization to unseen target models, particularly those that differ significantly from the surrogate models.
As shown in Tab. C.I, method surpasses previous techniques by average of approximately 20%, underscoring the efficacy of our approach attacking unseen target models.
C.3 ATTACKING USING LESS DIVERSE SURROGATE MODELS
To further illustrate the efficacy of the CWA algorithm even with a limited diversity of surrogate models, we conducted additional experiments in this section. Specifically, we limit our selection to models of the ResNet family, employing only R8sNet-18, ResNet-32, ResNet-50, and ResNet- 101 (He et al., 2016) as surrogate models. As shown in Tab. C.2, our SSA-CWA still achieves superior results than other attackers. These results suggest that even when the diversity surrogate models is limited, adversarial examples generated using the CWA attacker are less prone overfitting those surrogate models and continue to generalize effectively to previously unseen target models.
C.4 ATTACKING USING THE SURROGATE IN DONG ET AL. (2018)
We also evaluate our methods using surrogate models from Dong et (2018), which include Inc-v3, Inc-v4, IncRes-v2, and Res-152 from TensorFlow model garden (Yu et al., 2020). Note that these models are different from the corresponding models in TorchVision This configuration ensures that the majority of the target models are dissimilar to the surrogate models, providing robust assessment of our algorithm's 's capability to generate adversarial examples that effectively transfer to diverse and previously unseen models.
As demonstrated in Tab. C.3, SSA-CWA outperforms previous methods significantly when attacking both normally trained models and defended models. Notably, our SSA-CWA achieves remarkable
22

---

Published as a conference paper at ICLR 2024
Table C.3: Black-box attack success rate (%,1) on NIPS2017 dataset. Surrogate models are Inc-v3,
Inc-v4, IncRes-v2 and Res-152 from Dong et al. (2018).
Method
Backhone
FGSM
BIM
MI
DI
TT
VMI
SVRE
PT
SSA
RAP
MI-SAM
MI-CSE MI-CWA
SSA-CWA
AlexNet
75.8
64.2
71.3
13.3
73.5
77.6
74.8
80.2
80.4
73.7
76.8
77.8
86.0
VGG-16
79.8
90.8
89.5
85.2
94.1
88.9
96.3
99.2
GoogleNet
61.3
78.3
79.4
91.5
74.5
94.3
87.9
84.0
96.1
93.3
88.1
96.1
95.3
99.4
Inception.
91.5
97.5
99.7
ResNet-152
54.9
81.4
80.3
91.9
71.1
95.6
87.0
82.0
96.3
93.1
89.8
94.7
95.4
99.3
DenseNet-121
85.7
85.4
94.7
80.6
95.8
91.3
89.2
97.0
94,9
92.4
97.4
97.1
99.7
SqueezeNet
85.9
80.1
84.3
89.6
78.5
90.3
89.0
87.1
92.9
92.8
87.9
91.9
93.3
97.6
Normal
ShuflleNet- V2
81.1
71.3
76.5
81.3
69.8
82.8
81.9
81.2
84.4
85.7
80.2
84.9
84.7
91.4
MobileNet-3
70.1
91.1
EfiiieetNett BO
60.3
77.3
77.9
90,9
68.2
94.4
86,7
80.1
95.7
93.9
88.1
93.9
93.6
99.2
MNasNet
70.3
87.0
84.7
98.3
RogNetX-400MP
63.6
72.1
75.8
86.3
70.3
90.6
85.1
79.3
94.6
91.2
83.9
91.1
92.1
98.7
ConvNeXi-T
45.2
71.9
71.8
85.3
47.5
91.6
77.0
64.8
93.0
89.2
83.2
84.4
84.3
96.2
VIT-B/16
37.8
47.1
51.6
60.4
49.1
79.6
57.4
55.9
82.3
56.8
60.3
53.4
53.0
81.3
Swin-5
37.4
54.4
58.2
71.9
38.0
82.0
63.6
47.5
85.2
65.9
65.4
60.4
59.1
84.2
VixV
57.6
74.0
64.4
42.0
61.7
FGSMAI
Inception-V V3
61.2
54.8
57.1
62.4
68.7
73.8
61.9
68.1
79.9
63.3
60.9
63.2
63.4
80.2
EnsAT
IacRes-
36.0
39.8
40.7
57.4
61.0
72.5
45.4
55.7
79.9
39.1
44.1
43.0
44.5
74.3
FastAT
RexNet 50
44.8
41.4
43.2
43.6
44.5
44.6
45.0
46.9
47.6
46.2
44.9
46.7
45.1
48.2
EGDAT
ResNer-30
35.5
30.2
32.7
33.8
36.2
37.1
34.8
39.3
41.6
36.6
34.2
32.3
35.1
39.8
PGDAT
ResNet-50
40.8
43,9
45.1
45.1
47.3
47.6
47.3
45.2
49.1
WRN-50-2
27.5
22.3
25.2
25.9
29.2
29.7
26.8
31.4
33.1
28.8
26.8
26.7
26.8
31.5
MC1
PGDAT
XCTT-111
18.9
15.0
18.0
19.5
19.3
21.8
18.5
21.1
26.3
19.3
19.1
18.2
17.0
22.9
HGD
IncRes-V2
45.7
W2.3
81.9
93.2
78.4
96.1
84.3
86.4
9644
90.7
90.3
93.1
92.4
99.5
R&P
ResNet-50
74.9
87.5
99.1
Bit
ReaNet-00
64.8
78.0
81.3
92.0
72.6
94.7
87.1
83.1
96.4
93.6
88.9
94.4
94.8
99.3
92.1
99.2
RS
ResNet-50
61.3
81.1
80.8
91.8
74,3
94.2
88.6
83.1
95.8
93.5
89.8
94.6
94.9
99.0
NRP
ResNet-50
36.6
23.2
29.3
37.0
37.7
20.9
33.0
8.9
15.6
16.1
14.1
16.3
DiffPure
ResNet-50
52.3
51,9
62.0
72.9
73.5
86.2
67.5
76.8
89.9
70.2
68.8
67.1
67.9
94.6
94.6% attack success rate against challenging DifiPure defenses (Nie et al., 2022). This underscores the effectiveness of our approach in targeting state-of-the-aar defense mechanisms.
C.5 EXPERIMENTS ON 4/255
Many adversarially trained models and defenses are predominantly examined within the context of the < = 4/255 threat model. To remain consistent with these evaluations, we have also carried out an additional experiment under this specific perturbation budget, 4/255.
As demonstrated Tab. C.4, our methods still achieve superior results on most target models. Notably, when attacking the adversarially trained models, our methods outperform previous methods by about 5% on average. This demonstrates the strong efficacy our methods when attacking with small perturbation budget, especially adversarially trained models.
C.6 VISUALIZATION ADVERSARIAL PATCHES
(a) YOLOv3
(b) YOLOv5
(c) Loss Avg
(d) Adam-CWA
(c) Strongest
Figure C.I: Visualization of adversarial patches from different methods. The patch simply trained by loss ensemble looks like the fusion of those trained by YOLOv3 and YOLOv5. Adam-CWA captures the common weakness of YOLOv3 and YOLOv5, and therefore generates completely different patch.
As illustrated in Fig. C.I, we observe that YOLOY5 is more vulnerable to adversarial attacks compared YOLOv3. Consequently, the patch generated by the Loss Ensemble method resembles the one
23

---

Published as a conference paper at ICLR 2024
Table C.4: Black-box aftack success rate (%, 1) on NIPS2017 dataset. Settings are same as
Sec. 4.1, except for e, which is set to 4/255.
Method
Backhone
RGSM
BIM
MI
11
VMI SVRE
Pi
SSA
KAP
MI-SAM
MI-CSE
MI-CWA
VMI-CWA SSA-CWA
AlcaNet
43.4
44.2
45.8
50.3
48.9
47.8
55.5
47.7
49.5
54.9
59.8
VGG-16
19.0
62.2
67.0
78.7
50.5
76.4
71.5
60.4
72.1
55.6
76.8
79.3
76.5
83.1
77.4
GoogleNet
63.2
76.8
Inception-V3
35.1
43.3
47,5
62.
42.6
55.6
53.5
46.1
65.8
44.6
54.7
63.
60.3
63.9
71.1
ResNet-152
41.7
79.7
83.7
83.7
84.5
75.3
75.8
59.1
90.4
90.3
87.3
79.8
DemseNet- 121
42.2
72.1
78.7
82.4
62.1
86.5
812
71.6
71.2
57.4
87.7
87.5
84.5
90.8
81.7
SqueezeNet
49.2
63.3
67.5
76.1
56.7
74.4
73.6
63.3
76.4
60.7
73.1
80.3
79.3
83.7
84_1
Normal
MobileNet-V3
ShuffleNet-V2
33.3
39.9
42.7
52.2
46.4
49.5
44.5
46.8
56.6
43.3
56.7
47.9
55.2
$1.9
59.7
65.5
EfficienNet-B0
29.8
66.3
MNasNett
37.6
58.2
62.4
75.9
41.7
73.3
63.5
53.4
65.0
$1.9
71.9
73.3
70.5
77.6
71.4
RegNetX-1O0MP
37.0
56.6
63.4
75.3
51.7
74.3
67.1
59.0
71.7
51.7
72.5
79.9
75.9
83.7
82.4
NeX-N
19.2
34.6
40.4
55.3
27.8
34,7
31.3
48.7
35.8
32.0
VIT-B/16
16.0
16.2
20.5
24.5
21.6
26.5
18.0
23.1
21.8
20.0
24.0
25.6
24.6
26.8
23.4
Swin-S
16.5
23.9
32.1
29.1
MaxVTT-T
17.1
21.7
26.3
39.3
11.2
32.8
22.5
18.4
23.4
23.1
32.2
25.5
22.4
28.6
23.7
FGSMAT
Inception-V
35.5
39.3
40.0
56.3
EasAT
IncRes-V2
21.0
21.2
22.0
26.8
28.
24.7
23.1
24.9
12.4
23.5
23.8
31.0
31
30.8
36.7
PaStAT
Re
38.8
39.3
39.3
39.8
40.2
39.5
39.9
40.1
42.2
39,9
39.9
45.6
46.1
46.1
45.4
ResNet-50
25.7
26.7
27.4
27.9
27.9
27.9
27.9
33.7
PGDAT
RedNet-18 18
38.3
38.6
39.0
39.5
40.8
39.1
39.6
40.0
41.7 39.6
39.5
45.9
45.8
45.6
45.7
WRN-50-2
XCTTM12
13.3
13.5
14.0
15.0
15.5
14.9
15.1
15.2
15.4
15.2
14.8
25.5
25.6
24.9
19.9
PGDAT
XCYT-L12
12.5
13.4
14.0
14.0
14.8
15.0
15.2
15.2
14.6
23.3
22.3
18.8
HGD
Incket.
18.7
26.8
31.9
50.8
50.7
41.4
26.6
31.3
11
24.5
39.0
34.0
29.8
33.7
13.5
R&P
ResNet-50
51.9
86.3
89.7
91.4 01.470 76.6
93.1
93.5
85,5
85.4
67.8
93.0
94.8
93.8
95.9
89.0
ResNer-50
95.0
88.1
98.0
92.0
97.1
95.6
JPEG
50
52.4
82.3
88.3
82.4
83.0
93.9
88.5
86.9
8300
66.0
93.1
90.8
89.6
97.3
87.9
NRP
R5
47.0
78.7
81,9
76.1
65.6
87.3
86.0
72.0
72.6
60.7
88.3
78.0
76.7
83.6
69.6
DidPure
50
23.9
22.0
25.6
33.1
40.4
28.8
23.6
35.2
315
26.4
28.2
28.7
29.0
30.8
33.2
obtained by YOLOv5, resulting in similar performance for both patches. We hypothesize that the Loss Ensemble method does not attack the common weakness of YOLOv3 and YOLOv5, and instead nearly solely relies on information from YOLOv5. Contrarily, our proposed method, aims to exploit this common vulnerability and generates patch that differs significantly from both YOLOv3 and YOLOv5. As a result, our patch are more effective in attacking the object detectors.
In order to craft the strongest universal adversarial patch for object detectors, ensemble all the models in Huang et al. (2023) and craft an adversarial patch by our CWA algorithm. The patch is visualized in Fig. (e)). Our patch outperforms previous patches by large margin. Compared to the previous state-of-the-arl methods (i.e., the loss ensemble by enhanced baseline in Huang et al. (2023)), our approach improves by 4. 4.26%, achieving 4.69% mAP on eight testing models Tab. 2.
C.7 ABLATION STUDIES
In this section, we investigate the roles of the two additional hyperparameters: the reverse step size and the inner step size B. We use the same experimental settings as Sec. 4.1, and average the attack success rates across all target models.
26.5
32.
572
251
066
640
(a) 3
(b)
Ablation study inner step size 3. Our MI-CSE algorithm could be viewed optimizing the original loss using the learning rate 3 and the cosine similarity regularization term using the learning rate 2. As shown in Fig. 2(a), increases, the proportion of the regularization term gradually
g²
24

---

Published as a conference paper at ICLR 2024
Table C.5: Black-box attack success rate (%,
of our methods without incorporating with MI.
Method
Backbone
FGSM
BIM
MI
SAM
CSE
CWA
AlexNet
76,4
54.9
73.2
69.5
88.1
87.3
VGG-16
68.9
86.1
91.9
94.3
92.5
87.6
GoogleNet
54.4
76.6
89.1
92.5
92.8
88.5
Inception-Ve
54.5
64.9
84.6
84.4
90.4
88.4
ResNet-152
54.5
96.0
96.6
98.0
95.3
89.1
DenseNet-121
57.4
93.0
95.8
97.7
95.5
88.1
SqueezeNet
85.0
80.4
89,4
91.3
ShuffleNet-V2
81.2
65.3
79.9
82.4
91.9
90.9
Normal
MobileNet-V2.
58.9
55.6
71.8
73.1
90.6
89.7
EfficientNet-BO
50.8
80.2
90.1
93.6
93.3
87.6
MNasNet
80.8
88.8
93.0
87.2
RegNetX-400MP
81.1
89.3
92.9
93.6
ConvNeXI-T
39.8
68.6
81.6
86.7
83.0
72.5
ViT-B/16
33.8
35.0
59.2
55.8
84.5
83.2
Swin-S
34.0
48.2
66.0
69.6
80.2
72.0
MaxViT-1
49.7
71.1
FGSMAT
Inception- V3
53.9
43.4
55.9
55.7
88.5
89.8
EnsAT
IncRes-V2 V2
32.5
28.5
42.5
40.3
82.1
83.4
FastAT
ResNet-50
45.6
41.6
45.7
46.0
72.2
74.7
PGDAT
ResNet-50
36.3
30.9
72.9
ResNet-18
46.8
41.0
45.7
43.6
70.8
73.6
PGDAT
WRN-50-2
27.7
20.9
27.8
26.4
64.7
68.1
XCIT-M12
23.0
16.4
22.8
22.8
73.0
77.7
PGDAT
XCIT-LI2
19.8
15.7
19.8
20.5
67.1
72.0
HGD
IncRes-Ve V2
36.0
78.0
76.2
82.8
86.4
81.9
R&P
ResNet-50
67.9
95.8
96.3
98.3
95.0
88.7
Bit
ResNet-50
69.1
97.0
97.3
98.8
97.5
89.3
JPEG
ResNet-50
68.5
96.0
96.3
98.6
96.1
89.0
RS
ResNet-50
60.9
96.1
95.6
98.2
95.8
89.9
NRP
ResNet-50
36.6
88.7
72.4
92.4
82.9
62.7
DittPure
ResNet-50
50.9
68.5
76.0
67.9
83.3
82.8
increases, leading to an increase in the attack success rate. However, when 3 becomes too large, the error of our algorithm also increases, causing the attack success rate to plateau or decrease. Therefore, we need to choose appropriate value of 3 to balance between the effectiveneess of the regularization term and the overall performance the algorithm.
Ablation study reverse step size Fig. 2(b) shows that the attack success rate initially increases and then decreases as 16/255/r increases. This is because when the reverse step size is too large, the optimization direction is opposite to the forward step, leading to decrease in the attack success rate. Thus, decreasing the reverse step size initially increases the attack success rate. As the reverse step size continues decrease, MI-CWA gradually degrade MI-CSE, causing the attack success rate converge to MI-CSE.
Ablation study on step size We evaluate the average attack success rate for various a, ranging from 16/255/1 to 16/255/20. As depicted in Fig. 2(c), When a < 0.01, the reverse step size surpass the forward step size. This results in opposing optimization directions and subsequently causes sharp decline in the efficacy of our method. For a > 0.01, the attack success rate remains relatively consistent regardless of specific value of a. This showcases our method's resilience to hyper-parameter variations.
Ablation study on momentum. We also conduct an experiment where our methods are not combined with MI-FGSM. It is important to note that in other experiments, our methods and the baselines we compare against, except for BIM and FGSM, are combined with MI-FGSM. As shown in Tab, C.5, methods still achieve superior results than FGSM and BIM. However, it is notable that there a significant decrease in performance compared with the results that incorporate MI-FGSM. This experiment demonstrates that MI-FGSM has become popular houg-and-plar module capable of efficiently enhancing the performance of various attack algorithms, and it has even become fundamental and indivisible component in the development of advanced attack algorithms.
25

---

Published as a conference paper at ICLR 2024
D ANALYSIS AND DISCUSSIONS
In this section, we conduct some additional experiments to prove our claims about the effectiveness of our methods, regarding flattening the loss landscapes and maximizing the cosine similarity of gradients to boost the transferability.
D.1 VERIFICATION THE APPROXIMATION IN CSE
To verify whether 9.92 lig.lla holds, We first average the value of over NIPS17 dataset,
9:0
lg: la
and then downsample this matrix to (256, 256), finally we visualize this matrix in the form heatmap. The result is shown in Fig. D.1.
Figure D.1: Heatmap of the matrix.
As shown in the figure, the diagonal elements are close to 1, while the others are close to 0. As result, the approximation
9.9."
reasonable. This supports the Assumption A.I.
D.2 VISUALIZATION OF LANDSCAPE
We illustrate claim about the relationship between the loss landscape and the transferability of adversarial examples by visualizing the landscapes by different algorithms. In this part, we generate the adversarial example the ensemble of six surrogate models used in Sec. 4.1 and test on all the defense models in RobustBench (Croce et al., 2021).
For each algorithm, first craft an adversarial example via this algorithm, which expected to be the convergence point during training and near the optimum of each black-box model. Then, we use BIM (Kurakin et al., 2018) to fine-tune the adversarial example on each black-box model get the position of the optimum of each model (note that this process is white-box). To visualize the loss landscapes of test models around æ, we perturb æ along unit direction (p: - <>/lpp; x ||oc for each model and plot the loss curves in one plane. The results of MI and our method MI-CWA are shown Fig. 4(a) and Fig. 4(b) respectively.
As shown in Fig. 4(a) and 4(b), the landscape at the adversarial example crafted by MI-CWA is much flatter than that of MI. is also noticeable that the optima of the models in MI-CWA are much closer than those in MI. This supports claim that CWA encourage the flatness of the landscape and closeness between local optima, thus leading to better transferability.
D.3 COSINE SIMILARITY OF GRADIENTS
also measure the average cosine similarity of the gradients between training (surrogate) models and testing (black-box) models. The results shown in Tab. D.I. Our method improves the cosine similarity of gradients both between training models and testing models compared with MI. It shows that our method is likely find the common weakness of the training models, and the common weakness of the training models tends to generalize to testing models.
D.4 TIME COMPLEXITY ANALYSIS
To intuitively illustrate the time complexity of different methods in Sec. 4.1, we list their Number of Function Evaluations (NFEs) in Tab. D.2, where is the number of surrogate models. As shown,
26

---

Published as a conference paper at ICLR 2024
Table D.I:
Cosine Similarity of gradients. MI-CWA universally increases the cosine similarity of gradients among models, regardless of whether they are from Ftrain or F.
MI
MI-CWA
F
Fi
0.185
0.193
F x F
0.024
0.032
F
0.061
0.076
methods MI-SAM, MI-CSE and MI-CWA are quite efficient. When combined with previous attacks, they maintain their efficiency and enhance their efficacy, showcasing the potential integrating our approach with leading-edge attacks and optimizers.
We also test the real time cost of generating adversarial examples using different methods, utilizing the surrogate models mentioned in Sec. 4.1. As shown in Tab. D.2, method incurs only slight computational overhead compared to the baseline attacks. This demonstrates that our method can serve blug-and-play algorithm effectively and efficiently enhance transfer attack performance.
Table D.2: The number of function evaluations (NFEs) and real time cost of methods that in
Sec. 4.1.
Method
FGSM BIM
VMI
SVRE
SSA
RAP
MI-SAM
MICSEE MI-CWA VMI-CWA SSA-CWA
NEEs
10n.
10m
10n
10n
200g
30m
10m
200n
3400n
20n
10n
20n
210z
210n
Time (8)
0.2
0.8
0.8
0.8
0.9
2.1
3.0
0.7
18.5
168.2
1.7
1.0
1.8
21.2
26.3
27


Reviews:
[{"review_id": "rj1s0Mf5Zr", "reviewer": "Reviewer_s2Ct", "summary": "The paper proposes a novel adversarial attacks (Common Weakness Attack) against ensembles of model. It composes of two components, sharpness aware minimization (SAM) and cosine similarity encourager (CSE) that regularize the flatness of the loss landscape and the closeness to the optimum respectively. The author performs extensive experiments over extensive datasets and models.", "questions": "1. Can the author explain why the model is able to achieve significantly more effective attacks compared the existing methods for the adversarial trained model? I am mainly concerned about the fairness of the evaluation. Also, can the author explain how the AT model trained for ensembles in their experiments?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written in general and has very clear motivation and mathematical formulations. The derivation of attack based on the second order appropriation is intuitive. \n2. The author conducts very extensive empirical studies with many choices of architectures and datasets.", "weaknesses": "1. In table 1, the author only provides the results of CWA in combintation with other methods. Is there results for CWA only, and how well it performs. \n2. As an ablation study, the author might want to try different norm decomposition, e.g. the operator norm of H.\n3. The author should clarify the novelty compared to the previous methods. Specially, MI, VMI, SSA are all existing proposed attacks. The sharpness aware minimization techniques have been previously used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GaV21kBKdY", "reviewer": "Reviewer_JLTP", "summary": "This paper proposes a new method for constructing adversarial examples for black-box models, using an ensemble of surrogate models. Using a second order approximation of the adversarial loss function, the authors construct an upper bound using the Hessian on the adversarial loss, and establish approximate methods for minimizing the upper bound, yielding their proposed attack: Common Weakness Attack (CWA). The CWA algorithm consists of two steps: first a gradient ascent step on the averaged logits, inspired by sharpness-aware minimization, and a second gradient descent step using their proposed cosine similarity encourager (CSE). The authors evaluate their attack method on a variety of model architectures using the NIPS2017 dataset, and compare with a variety of different attack algorithms from the literature.", "questions": "- In Fig 1, it is not clear what is being plotted, and the different colors are not well contrasted. \n- In Sec 3.1, below eq (2), the authors write \" ... we can see that a smaller value of ... means a smaller test error ...\". What is the test error in this case? is it the error on the black-box model? the term test error is rather confusing.\n- In Theorem 3.1, there is a typo, should be $||H_i||_F$? \n- In Sec 3.2: \" There are some researches ...\" typo\n- In Fig 2, what are $x_t$, $x_t^f$, .... The symbols are not defined. \n- In Alg1, does the order of the classifiers in the for loop matter? if so how is it currently chosen?\n- In Table 1, some of the highlighted numbers are not the best performing numbers, for instance the FastAT RN50 row. \n- Can the proposed method be also applied in the white-box setting? i.e. assume $\\mathcal{F} = [f]$ and simply attack the classifier.  Have the authors experimented with something like this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper tackles an important subject, understanding how vulnerable can ML models be is crucial for safe-guarding them against possible adversaries\n\n- The proposed method, as far as I can tell, seems novel to me.\n\n- The strengths of this paper are primarily in the effectiveness of their proposed method, as it seems to combine very well with prior attacks (e.g, MI-CWA, SSA-CWA) and achieves superior results. The transfer-based black-box attacks against Bard is interesting.\n\n- The proposed algorithm seems reasonable to implement", "weaknesses": "- I felt that the paper was bit hard to follow. Section 3 mixes a lot of prior results with proposed ones. Thus it makes it a bit hard to distinguish original contributions vs reusing prior results. For instance, it would greatly improve the readability if SAM was properly explained prior to this section (or as a subsection). Figures 1 and 2 are not very informative (see Questions), they lack axis and/or labels. It would also improve the quality of the paper if the authors include a mathematical description of some of the prior ensemble attack methods (e.g. is it simply PGD on the averaged logits?), to better understand the contrast between them and the proposed method.\n\n- Despite the compactness of Algorithm 1, there is a lack of analysis on its complexity. How does it compare (in terms of wall-clock time) to prior attacks or vanilla PGD-style attacks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ytRIyfQmu3", "reviewer": "Reviewer_9Y8Q", "summary": "This paper studies the transfer-based attack towards the ensemble of models. The authors suggest that the ensemble model would have \"common weaknesses\" that are strongly correlated with adversarial examples' transferability and propose the common weaknesses attack (CWA). Theoretically, this paper provides clear intuition on why the models have common weaknesses. This paper also conducts extensive experiments to validate their theoretical findings.", "questions": "See the Weakness part.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This paper conducts extensive experiments on several datasets, demonstrating that CWA receives superior results than previous methods.\n2. The intuition of common weakness is strong and clear. This paper converts the task \"crafting adversarial examples on ensemble models\" to \"optimizing the second term in Equation (2)\". By Theorem 3.1, this term is further decomposed into a \"flatness term\" and a \"closeness term\". These two terms can be efficiently optimized by SAM and CSE.", "weaknesses": "1. The intuition behind SAM and CSE is not clear enough. Equation (4) and (5) is confusing for those readers not familiar with this area.\n2. In Section 3.1, the authors mentioned that \"the goal of transfer-based attacks is to craft an adversarial example $x$ that is misclassified by all models in $\\mathcal{F}$\". However, fooling all target models seems to be an impossible job in practice. Besides, the experiments in this paper cannot support this claim.\n3. The perturbation in Figure 3 is not imperceptible to humans.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AbpVUQZG9u", "reviewer": "Reviewer_GTKq", "summary": "In this paper, the authors rethink model ensemble in black-box adversarial attacks: exploring the relationship among the transferability of adversarial samples, the Hessian matrix’s F-norm, and the distance between the local optimum of each model and the convergence\npoint. Based on the above theoretical analysis, the authors define common weaknesses and propose effective algorithms\nto find common weaknesses of the model ensemble.", "questions": "- How to explain the significant performance degradation of VMI-CWA compared to VMI in Tab 1when the target models are Swin-S and Max ViT-T?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The experiments are solid and compresive: the author consider image classification, object detection and large vision-language models, and the results demonstrate the effectivenss of the proposed method.\n- The authors introduce extensive mathmatical proofs, which makes the proposed method more convincing.", "weaknesses": "- The description of the experimental part could be improved. Take \"Results on normal models\" as an example. In the experiment setup, the authors choose MI, VMI and SSA to validate the effectiveness of proposed CWA. However, in \"Results on normal models\", the authors only discuss MI, and ignore the VMI and SSA. I think the authors should pay more attention on the recent SOTA (e.g., SSA), since MI was proposed in 2018 (5 years ago).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)