Understanding Model Ensemble in Transferable Adversarial Attack
Authors: Wei Yao
Zeliang Zhang
Huayi Tang
Yong Liu
Venue: ICML 2025 poster (2025)
Keywords: adversarial transferability
model ensemble attack
Primary area: social_aspects->robustness
TL;DR: We establish a theoretical framework for transferable model ensemble adversarial attacks through statistical learning theory.

Abstract: Model ensemble adversarial attack has become a powerful method for generating transferable adversarial examples that can target even unknown models, but its theoretical foundation remains underexplored. To address this gap, we provide early theoretical insights that serve as a roadmap for advancing model ensemble adversarial attack. We first define transferability error to measure the error in adversarial transferability, alongside concepts of diversity and empirical model ensemble Rademacher complexity. We then decompose the transferability error into vulnerability, diversity, and a constant, which rigidly explains the origin of transferability error in model ensemble attack: the vulnerability of an adversarial example to ensemble components, and the diversity of ensemble components. Furthermore, we apply the latest mathematical tools in information theory to bound the transferability error using complexity and generalization terms, validating three practical guidelines for reducing transferability error: (1) incorporating more surrogate models, (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting. Finally, extensive experiments with 54 models validate our theoretical framework, representing a significant step forward in understanding transferable model ensemble adversarial attacks.

Understanding Model Ensemble in Transferable Adversarial Attack
Wei Yao 123
Zeliang Zhang
Huayi Tang
123 Yong Liu 123
Abstract
1
Introduction
Neural networks are highly vulnerable to adversarial exam- ples (Szegedy et al., 2013; Goodfellow et al., 2014) per- turbations that closely resemble the original data but can severely compromise safety-critical applications (Zhang & Li, 2019; Kong et al., 2020; Bortsova et al., 2021). Even more concerning is the phenomenon of adversarial transfer- ability (Papernot et al., 2016; Liu et al., 2017): adversarial examples crafted to deceive one model often succeed in attacking others. This property enables attacks without re- quiring any knowledge of the target model, significantly complicating efforts to ensure the robustness of neural net- works (Dong et al., 2019; Silva & Najafirad, 2020).
Model ensemble adversarial attack has become a powerful method for generating transferable ad- versarial examples that can target even unknown models, but its theoretical foundation remains un- derexplored. To address this gap, we provide early theoretical insights that serve as roadmap for advancing model ensemble adversarial attack. We first define transferability error to measure the error in adversarial transferability, alongside concepts of diversity and empirical model ensem- ble Rademacher complexity. We then decompose the transferability error into vulnerability, diver- sity, and a constant, which rigidly explains the origin of transferability error in model ensemble attack: the vulnerability of an adversarial exam- ple to ensemble components, and the diversity of ensemble components. Furthermore, we apply the latest mathematical tools in information theory to bound the transferability error using complexity and generalization terms, validating three practi- cal guidelines for reducing transferability error: (I) incorporating more surrogate models, (2) in- creasing their diversity, and (3) reducing their complexity in cases of overfitting. Finally, ex- tensive experiments with 54 models validate our thcoretical framework, representing significant step forward in understanding transferable model ensemble adversarial attacks.
To enhance adversarial transfcrability, researchers have pro- posed a range of algorithms that fall into three main cate- gories: input transformation (Xie et al., 2019; Wang et al., 2021), gradient-basee optimization (Gao et al., 2020; Xiong et al., 2022), and model ensemble attacks (Li al., 2020; Chen et al., 2024b). Among these, model ensemble attacks have proven especially powerful, as they leverage multiple models to simultaneously generate adversarial examples that exploit the strengths of each individual model (Dong et al., 2018). Moreover, these attacks can be combined with input transformation and gradient- based optimization meth- ods to further improve their effectiveness (Tang al., 2024). However, despite the success of such attacks, their theoreti- cal foundation remains poorly understood. This prompts an important question: Can we establish theoretical frame- work for transferable model ensemble adversarial attacks to shape the evolution of future algorithms?
To conduct preliminary exploration of this profound ques- tion, we propose three novel definitions prerequisite of our theoretical framework. Firstly, we define transferability error as the gap in expected loss between adversarial example and the one with the highest loss within a feasi- ble region of the input space. It captures the ability of an adversarial example to generalize across unseen models, representing its transferability. Secondly, we introduce pre- diction variance across the ensemble classifiers. It offers a novel perspective quantifying diversity in model ensem- ble attacks, providing fresh approach to guide the selection of ensemble components. Finally, we also introduce the em- pirical model ensemble Rademacher complexity, inspired
"Equal contribution !Gaoling School of Artificial Intelligence, Remmin University of China, Beijing, China BBijiing Key Lab- oratory of Research on Large Models and Intelligent Gover- nance 'Enginerring Research Center of Next-Generation Intelli- gent Search and Recommendation, MOE 'Independent Researcher. Contributed ideas during the author's B.S. studies at Huazhong University Science and Technology, Wuhan, China. Correspon- dence to: Yong Liu <liuyonggsai @ www.uuu..cco edu.en>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Understanding Model Ensemble in Transferable Adversarial Attack
by Rademacher complexity (Bartlett & Mendelson, 2002), as measure of the flexibility of ensemble components.
versarial transferability (Li et al., 2023; Wu et al., 2024; Chen et al., 2024b). An extended and detailed summary of related work is in Appendix A.
With these three definitions, we offer two key theoretical insights. First, we show the vulnerabiity-diivrrity decom- position of transferability error (Figure 1), highlighting the preference for ensemble components that are powerful at- tackers and induce greater prediction variance among them- selves. However, this also uncovers a fundamental trade-oft off between vulnerability and diversity, making it challenging to maximize both simultaneously. To mitigate this issue and provide more practical guidelines, we present an up- per bound for transferability error, incorporating empirical model ensemble Rademacher complexity and a generaliza- tion term. The primary challenge in proof lies in the appli- cation of cutting-edge mathematical tools from information theory (Esposito& & Mondelli, 2024), which are crucial for addressing the complex issue of relaxing the independence assumption among surrogate classifiers. Our theoretical analysis leads to a crucial takeaway for practitioners: In- cluding more and diverse surrogate models with reduced model complexity in cases of overfitting helps tighten the transferability error bound, thereby improving adversarial transferability. Finally, the experimental results support the soundness of our theoretical framework, highlighting a key step forward in the deeper understanding of transferable model ensemble adversarial attacks.
Within the extensive body of research on model ensemble attacks, two notable and intriguing observations stand out. First, increasing the number of models in an ensemble im- proves adversarial transferability (Liu al., 2017; Dong et al., 2018; Lin al., 2019; Gubri et al., 2022b; Liu et al., 2024). Second, using more diverse surrogate models with varying architectures and mack-propagated gradients (Tang et al., 2024) further enhances transferability. However, to our best knowledge, these intriguing phenomena have yet to be fully understood from a theoretical perspective. In this paper, we present the first theoretical framework to explain these phenomena, providing actionable insight that pave the way for future algorithm design.
2.2 Theoretical Understanding of Adversarial Transferability
In contrast the wealth of empirical and intuitive stud- ies, research on the theoretical understanding of adversarial transfcrability remains limited. Recent efforts have primar- ily focused on aspects such as data (Tramèr et al., 2017), surrogate model (Wang & Farnia, 2023), optimization (Yang et al., 2021; Zhang et al., 2024a; Chen et al., 2024b; Fan et al., 2024) and target model (Zhao et al., 2023). Tramèr et al. (2017) investigates the space of transferable adversarial examples and establishes conditions on the data distribu- tion that suggest transferability for some basic models. In terms of the surrogate model generalization, Wang & Farnia (2023) builds the generalization gap to show that a surro- gate model with a smaller generalization error leads to more transferable adversarial examples. From an optimization perspective, Yang et al. (2021); Zhang al. (2024a) estab- lish upper and lower bounds adversarial transfcrability, linking it to model smoothness and gradient similarity. They suggest that increased surrogate model smoothness and less loss gradient similarity improve transferability. Chen et al. (2024b) provide theoretical evidence connecting transfer- ability to loss landscape flatness and closeness to local op- tima. Fan et al. (2024) decompose adversarial transferability into local effectiveness and transfer-related loss, suggesting that flatness alone is insufficient to determine the whole picture of adversarial transferabilityy. Regarding the target model, Zhao et al. (2023) theoretically reveal that reducing the discrepancy between the surrogate and target models can limit adversarial transferability.
2 Related Work
2.1 Transferable Adversarial Attack
Researchers have developed various algorithms to enhance adversial transferability. Most of them fall into three cate- gories: input transformation, gradient- based optimization, and model ensemble attack. Input transformation tech- niques apply data augmentation strategies to prevent overfit- ting to the surrogate model. For instance, random resizing and padding (Xie et al., 2019), downscaling (Lin et al., 2019), and mixing (Wang et al., 2021). Gradient-based op- timization optimizes the generation of adversarial examples to achieve better transferability. Some popular ideas include applying momentum (Dong et al., 2018), Nesterov acceler- ated gradient (Lin al., 2019), scheduled step size (Gao et al., 2020) and gradient variance reduction (Xiong al., 2022). Model ensemble attack combine outputs from sur- rogate models to create an ensemble loss, increasing the likelihood to deceive various models simultancously. It can be applicd collectively with both input transformation and gradient-based optimization algorithms (Tang et al., 2024). Some popular ensemble paradigms include loss-based en- semble (Dong et al., 2018), prediction-based (Liu et al., 2017), logit-based ensemble (Dong et al., 2018), and lon- gitudinal strategy (Li et al., 2020). Moreover, advanced ensemble algorithms have been created to ensure better ad-
Despite these theoretical advances, to the best of our knowl- edge, transferable model ensemble adversarial attacks re- main unexplored. To address this gap, we take pioneering step by presenting the first theoretical analysis of such at- tacks. Our work not only offers theoretical insights into
2

---

Understanding Model Ensemble in Transferable Adversarial Attack
Clean data
Adversarial example
Optimal adversarial example
Decision boundary (attacker)
Decision boundary (expected ensemble attacker)
Decision boundary region
Tiger
2.0
r=
Adversarial region
(a) Transferability Error
(b) Vulnerability
(c) Diversity
Figure 1. snmterabiittyddivessi decomposition transferability error. (a) The transferability error defined the difference expected loss value between given adversarial example and the most transferable one. (b) Vulnerability is the loss value of the expected ensemble classifier on the adversarial example. (c) Diversity the variance in model ensemble predictions that correspond to correct class.
these attacks but also incorporates recent advancements in information theory, laying the groundwork for future theo- retical investigations into adversarial transferability.
the parameter sets (01,..., 0N) from the distribution Pg*. For a clean data å = (2. y) E Z, an adversarial example (zz, y) E Z, and N classifiers for model ensemble attack, define the population risk Lp(2) and the empirical risk LE(z) of the adversarial example as
3 Key Definitions: Transferability Error, Diversity, and Ensemble Complexity
Lp(x) = Ee-Pe P ((f((0=).
(I)
In this section, we first highlight the fundamental goal of model ensemble adversarial attack (Section 3.1). Then we define the transferability error (Section 3.2), diversity in
1 and LF(2) N Σ (f(002z),))
(2)
transferable model ensemble attack (Section 3.3) and empir- ical model ensemble Rademacher complexity (Section 3.4).
Intuitively, a transferable adversarial example leads large T.(2)) because it can attack many classifiers with pa- rameter 0 E θ. Therefore, the most transferable adversarial
3.1 Model Ensemble Adversarial Attack
example 2 = (xx, y) around 2 is defined as
Given the input space X' C IRd and the output space y C R, we have a joint distribution Pz over the input space Z = Xx). The training set Zirain = {2,/2, = (I,, v1) 2, y4 E (-1,1), 1,... K). which consists of K examples drawn independently from Pz. We denote the hypothesis space by H. : X x y and the parameter space by O. Let -(::)) E H be a classifier parameterized by f E θ, trained for a classification task using a loss function f y x y + RØ. Let Pe represent the distribution over the parameter space @. Define PeN as the joint distribution over the product space N which denotes the space of N such sets of parameters. We use Zurin train N surrogate models f(01;)), f(xx; ff):: for model ensemble. The training process of these N classifiers can be viewed as sampling
x² = arg max Lp(2), EB).(2)
where B. fx || a l2 S c) is adversarial region centered at 2 with radius > O. However, the expectation in Lp(2) cannot be computed directly. Thus, when generating adversarial examples, the empirical version Eq. (2) is used in practice, such as loss-based ensemble attack (Dong et al., 2018). Therefore, the adversarial example = (x, y) obtained from the following equation
x arg max LE(Z).
(4)
2E,
There is a gap between the adversarial example & we find and the most transferable one =". is due to the fact that the
3

---

Understanding Model Ensemble in Transferable Adversarial Attack
ensemble classifiers cannot cover the whole parameter space of the classifier, i.e., there is difference between Lr(=) and (z).. Accordingly, the core objective of transferable model ensemble uttack is to design approaches that approxi- mate (=) to [r(2), thereby increasing the transferability of adversarial examples.
Definition 3.3 (Diversity of Model Ensemble Attack), The diversity of model ensemble attack across 0 ~ Pe for specific adversarial example (x, is defined as the variance of model prediction:
Vara~Pe (f(0; x)) = EB [f(0:z) - Pe f(0; x))
3.2 Transferability Error
It indicates the degree of dispersion in the predictions of different ensemble classifiers for the same adversarial exam- ple. The diversity of model ensemble attack is a measure of ensemble member disagreement, independent of the label. From an intuitive perspective, the disagreement among the ensemble components helps prevent the adversarial example from overfitting to the classifiers in the ensemble, thereby enhancing adversarial transferability to some extent.
Considering the difference between z and =", the transfer- ability of z can be characterized as the difference in popula- tion risk between and the optimal one.
Definition 3.1 (Transferability Error). The transferability error of 2 with radius is defined as:
TE e) Lp(x))
(5)
There always holds T ez) 2 aas Lp(x") Lp(z). The closer TE(,, e) is to 0, the better the transferability of 2. Therefore, in principle, the essential goal of various model ensemble attack algorithms is to make transferability error T as small possible. Moreover, if the distribution over the parameter space Pe, adversarial region B, (x) and loss function É are fixed, then Lx(²*) becomes a constant, which means that the goal of minimizing T TEz e) becomes maximizing [z)..
To calculate the diversity explicitly metric, con- sider a dataset of adversarial examples Zattack { (I,, MA), é = 1, ... ....... .f} and N classifiers in the ensemble. The diversity is computed as the average sample variance of predictions for all adversarial examples in the dataset:
M
[(0):a.)) j=1
N
Σ
N
)
27
M
N 3=1
Remark. For multi-class classification problems, f(0; x is replaced with the logit corresponding to the correct class prediction made by the classifier.
In the following lemma, we will show how the difference between empirical risk and population risk affects the trans- ferability error of z. The proof is in Appendix C.I.
Lemma 3.2. The transferability error defined by Eq. (5) is bounded by the largest absolute difference between Lp(z)
3.4 Empirical Model Ensemble Rademacher Complexity
and (z),,, i.e.
We define the empirical Rademacher complexity for model ensemble by analogy to the original empirical Rademacher complexity (Koltchinski & Panchenko, 2000; Bartlett & Mendelson, 2002).
TE(z,)) < 2 sup =E2
(6)
The lemma strictly states that if can bound the difference between Lp(a and (z),,, the transferability error can be constrained to a small value, thereby enhancing adversarial transferability. This indicates that we can develop strate- gies to make Lz(²) closely approximate (2),,, ultimately improving the transferability of adversarial examples.
Definition 3.4 (Empirical Model Ensemble Rademacher Complexity). Given the input space Z X x ) and N classifiers f(01;-). ((00::))..., (0w::)). Let = {o. be collection of independent Rademacher variables, which are random variables taking values uniformly in (+1, -1). We define the empirical model ensemble Rademacher complex- ity (Z) as follows:
3.3 Quantifying Diversity in Model Ensemble Attack
Before the advent of model ensemble attacks, the formal definition of diversity in ensemble learning had remained a long-standing challenge for decades (Wood al., 2024). While diverse intuitive definitions of diversity exist in the model ensemble attack literature (Li et al., 2020; Yang et al., 2021; Tang et al., 2024), we bridge the gap between trans- ferable model ensemble attacks and recent advancements in ensemble learning theory (Ortega et al., 2022; Wood et al., 2024). Specifically, we propose measuring diversity among ensemble attack classifiers through prediction variance.
RN = E sup or ol((((()))
(8)
In conventional settings of machine learning, the empirical Rademacher complexity captures how well models from function class can fit a dataset with random noisy la- bels SShalev-Swwrrz & Ben-David, 2014). A sufficiently complex function class includes functions that can effec- tively arbitrary label assignments, thereby maximizing

---

Understanding Model Ensemble in Transferable Adversarial Attack
the complexity term (Mohri et al., 2018). Likewise, in model ensemble attack, Eq. (8) is expected to measure the complexity of the input space Z relative to the N classifiers. Some extreme cases are analyzed in Appendix E.1.
ble attacks, similar to the well-known bias-variance trade- off (Geman al., 1992). This means that, in practice, it is not feasible to maximize both "Vulnerability" and "Di- versity" simultaneously. Recognizing this limitation, we proceed with further theoretical analysis to propose more guidelines for practitioners in the following section.
4 Theoretically Reduce Transferability Error
4.1 Vulnerability Decomposition of Transferability Error
4.2 Upper Bound of Transferability Error
We develop an upper bound of transferability error in this section. We begin by taking Multi-Layer Perceptron (MLP) as an example of deep neural network and derive the upper bound of R2 (Z). The proof is in Appendix B.4.
Inspired by the bias-variance decomposition (Geman et al., 1992; Domingos, 2000) in learning theory, we provide the corresponding theoretical support for prediction variance by decomposing the transferability error into vulnerability, diversity and constants. The proof and the empirical version of it is in Appendix C.2.
Lemma 4.2 (Ensemble Complexity of MLP). Let H. {ar - Wi (Wi-101-2 (... (W1æ)))} be the class of real-valued networks of depth I, where E RRd W E Rd:1 Given N classifiers from H. where the puram- eter matrix is Wey, i E (1,..,,n), E {l,-.-, ,I} and T = II supse|m) ||W p. Let ||>|| B. With 1- Lipschitz activation functions .i,... , and 1-Lipschitz loss function ((y((x)), there holds:
Theorem 4.1 V'llneaablityydiierii Decomposition), For (x. v). we consider the squared er- for loss (O((0=)),)) = ()(0;))-u] y)² Let x(0;))) = Ee-Pa f (0; x) be the expectation of prediction over the dis- tribution on the parameter space. Then there holds
data point 2
cE(,,c) = =bp(="- - -1[(0;,).,) (0)
Vara-PeJ
(Vi2log log 2)1 + +1) BT
Rx)
(10)
Vulnerrbilitt
Diversity
VN
Remark. We also derive the upper bound of (2 for the cases of linear model (Appendix B.2) and two-layer neural network (Appendix B.3). These results are special cases of the above theorem.
Remark. A similar formulation also applies to the KL diver- gence loss in the multi-class classification setting, which is proved in Appendix C.3.
The "Vulnerability" term measures the risk of a data point z being compromised by the model ensemble. If the model ensemble is sufficiently strong to fit the direction opposite to the target label, the resulting high loss theoretically re- duces the transferability error. This insight suggests that selecting strong attackers as ensemble components leads to lower transferability error. The "Diversity" term implies that selecting diverse attackers in a model ensemble attack theo- retically contributing to a reduction in transferability error. In conclusion, Theorem 4.1 provides the following guideline for reducing transferability error in model ensemble attack: are supposed to choose ensemble components that are both strong and diverse. Theorem 4.1 connects the existing body of work and clarifies how each algorithm strengthens adversarial transferability. For instance, some approaches tend to optimizing the attack process (Xiong et al., 2022; Chen et al., 2023) to improve "Vulnerability", while others aim to diversify surrogate models (Li et al., 2020; 2023: Wang et al., 2024) to enhance "Diversity". Also, there are other definitions of diversity based on gradient in previous literature (Yang et al., 2021; Kariyappa & Qureshi, 2019). A more detailed discussion is presented in Appendix E.2.
In particular, a larger N and smaller T will give Rv (2) a tighter bound. Notice that T contains the norm of weight matrices, which is related to model complexity (Bartlett et al., 2017: Neyshabur et al., 2018). And smaller model complexity corresponds to smaller T (Loshchilov & Hut- ter, 2019). In summary, Lemma 4.2 mathematically shows that increasing the number of surrogate models and reducing the model complexity of them can limit R (2).
We now provide the upper bound of transferability error, and the proof is in Appendix C.4.
Theorem 4.3 (Upper bound of Transferability Error). Given the transferability error defined by Eq. (5) and general rademacher complexity defined by Eq. (8). Let P be the joint measure induced by the product of the marginals. If the loss function l is bounded by B E R.. and PoNx is absolutely continuous with respect to P P@N foro any func- tion fo. then for a > 1 and n = a-1 with probability at least 6, there holds
TE(2,)) S (Z)+
2 Ha
(Pax || ( e)
18y32 N
However, due to the mathematical nature of Eq. (9), there remains a bineerabbiiyydiiversi trade-off in model ensem-
In
(11)
5
5

---

Understanding Model Ensemble in Transferable Adversarial Attack
where Ha (-11-) is the Hellinger integrals (Hellinger. 1909) with parameter o, which measures the divergence between two probability distributions if > 1 (Liese & Vajda, 2006).
ity is lower, but diversity may also be limited due to the simpler structure of the ensemble attackers. On the other hand, attacking simpler models with stronger, more di- verse ensemble may increase diversity but also raise model complexity. In this scenario, reducing complexity can help prevent overfitting and lead to a tighter transferability error bound, albeit with a slight reduction in ensemble diversity. In summary, striking a balance between model complexity and diversity is crucial in practice.
Our proposed setting where both the surrogate model and the target model adopt the same parameter space aligns with many realistic scenarios, as demonstrated in (Wu et al., 2024; Tang et al., 2024; Li et al., 2023; Xiong et al., 2022; Lin et al., 2019). However, Theorem 4.3 can be also extended to scenarios where the parameter distributions of surrogate model and target model differ. It is discussed in Appendix C.5 via redefinition of the model space.
Remark I.
From generalization error to transferability error. The mathematical form of Eq. (11) is in line with the generaliza- tion error bound (Bartlett & Mendelson, 2002). However, we note that a key distinction between transferability error and generalization error lies in the independence assump- tion. Conventional generalization error analysis relies on assumption: each data point from the dataset is inde- pendently sampled (Zou & Liu, 2023; Hu et al., 2023). By contrast, the surrogate models for ensemble attack are usu- ally trained on the datasets with similar tasks, e.g., image classification. In this case, we cannot assume these surro- gate models behave independently for a solid theoretical analysis. To build the gap between generalization error and transferability error, our proof introduces the latest tech- niques in information theory (Esposito & Mondelli, 2024). And refer to Appendix E.4 for detailed discussion about it. Thus, equipped with Theorem 1 from Esposito & Mondelli (2024), we swap the role of the model and data in learn- ing theory literature (Geman et al., 1992; Golowich et al., 2018; Bartlett & Mendelson, 2002; Ortega et al., 2022) with analogical proof steps and prove the results.
Remark 2.
We provide further explanation of the Hellinger integral term (P || &N,, e) in Appendix C.6.
Theorem 4.3 is grounded in the empirical model ensemble Rademacher complexity defined in Eq. (8). How- ever, it can be extended to ieformation-theoreele analysis with similar conclusions, as demonstrated in Appendix C.7.
Remark 3.
The first term in Eq. (11) suggests that incorporating more surrogate models with less model complexity in ensem- ble attack will constrain R, (Z) and enhances adversarial transferability. Intuitively, incorporating more models helps prevent any single model from overfitting to a specific adver- sarial example. Such theoretical heuristic also supported by experimental results (Liu et al., 2017; Dong et al., 2018; Lin et al., 2019; Li et al., 2020; Gubri et al., 2022b; Chen et al., 2023; Liu et al., 2024), which also stress the advan- tage of more surrogate models to obtain transferable attack. Additionally, when there is an overfitting issue, models with reduced complexity will mitigate it.
The second term also suggests that a large N (using more models) can lead tighter bound. Furthermore, it mo- tivates the idea that reducing the interdependence among the parameters in ensemble components (i.e., increasing their diversity) results in tighter upper bound for T E(2, c). Recall that (P IiP@N e) represents the divergence between the joint distribution PaN and the product of marginals Pø,, e- The joint distribution captures depen- dencies, while the product of marginals does not. Therefore, Ha P e) measures the degree of dependency among the parameters from N classifiers. As a result, in- creasing the diversity of parameters in surrogate models and reducing their interdependence enhances adversarial transferability. This theoretical conclusion is also supported by empirical results (Li et al., 2020; Tang et al., 2024), which also advocate for generating adversarial examples from diverse models.
4.3 The Analogy between Generalization and Adversarial Transferability
In addition to providing inspiration for model ensemble attacks, the theoretical evidence in this paper also offers new insights into another fascinating idea. Within the ex- tensive body of research on transferable adversarial attack algorithms accumulated over the years (Gu et al., 2024), we revisit a foundational analogy that is universally applicable in the adversarial transferability literature: The transfer- ability of an adversarial example is an analogue to the generalizability of the model (Dong et al., 2018). In other words, the ideas that enhance model generalization in deep learning may also improve adversarial transferability (Lin et al., 2019). Over the past few years, this analogy has sig- nificantly inspired the development of numerous effective algorithms, which directly reference it in their papers (Lin et al., 2019; Wang et al., 2021; Wang & He, 2021: Xiong et al., 2022; Chen et al., 2024b). And some recent papers are also inspired by it (Chen et al., 2023; Wu et al., 2024; Wang et al., 2024; Tang et al., 2024). Thus, validating this influential analogy is indispensable for defining the future landscape of adversarial transferability. Interestingly, our
The trade-off between complexity and diversity. Reduc- ing model complexity may conflict with increasing diver- sity. We discuss this issue from two angles. On one hand, when generating adversarial examples from simpler models to attack more complex ones, the overall model complex-
6

---

Understanding Model Ensemble in Transferable Adversarial Attack
0.0003
0.001
-.03
130
330
030
P. stpps
879p4
staps
MLP
(b) CNN
Figure 2. Evaluation of ensemble attacks with increasing the number of steps using MLPs and CNNs the MNIST dataset.
2.01
0.003
2000
9tsppss
53005
Step
5tops
(a) MLP
(h) CNN
Figure 3. Evaluation of ensemble attacks with increasing the number of steps using MLPs and CNNs the Fashion-MNIST dataset.
paper sheds light on this insight in several ways.
perspective of a theoretical researcher, the extensive and advanced generalization theory may yield enlightening in- sights in the field of adversarial transferability. On the other hand, from an practitioner's point of view, ideas from deep learning algorithms can also be leveraged to develop more effective transferable attack algorithms.
First, the mathematical formulations in Lemma 3.2 is simi- lar to generalization error (Vapnik, 1998; Bousquet & Elis- differ-
seeff, 2002), which also derives objective ence between the population risk and the empirical risk. Such similarity between transferability error and general- ization error suggests the possible validity of the analogy. Also, Lemma 4.2 is similar to the bound of the original Rademacher complexity (Golowich et al., 2018), which also suggests that obtaining a larger training set as well as a less complex model contribute tighter bound of Rademacher complexity. Such similarities between transfcrability error and generalization error suggests the possible validity of the analogy. More importantly, if the analogy is correct, then recall that in the conventional framework of learning theory: (1) increasing the size of training set typically leads to a better generalization of the model (Bousquet & Elis- seeff, 2002); (2) improving the diversity among ensemble classifiers makes more advantageous for better general- ization (Ortega et al., 2022); and (3) reducing the model complexity (Cherkassky, 2002) benefits the generalization ability. It is natural to ask: in model ensemble attack, do (1) incorporating more surrogate models, (2) making them more diverse, and (3) reducing their model complexity theo- retically result in better adversarial transferability?
5 Experiments
We conduct our experiments on three datasets, including the MNIST (LeCun, 1998). Fashion-MNISS (Xiao et al., 2017). and CIFAR- 10 (Krizhevsky al., 2009) datasets. We use these datasets to empirically validate our theory and build powerful ensemble adversarial attack in practice.
We build six deep neural networks for image classifica- tion, including three MLPs with one to three hidden layers followed by a linear classification layer, and three convo- lutional neural networks (CNNs) with one to three con- volutional layers followed by linear classification layer. To ensure diversity among the models, we apply three different types of transformations during training. Addi- tionally, we set the weight decay under the L2 norm to 4 10 33 1002, respectively. This results in a total of models. establish a gold standard for
x x 3 adversarial transferability evaluation, we additionally train ResNet-18 (He et al., 2016) from seratch on three datasets (MNIST, Fashion-MNIST. and CIFAR- respectively. We will leverage the models at hand to attack this ResNet-18 for reliable evaluation. For models trained on MNIST, Fashion-MNIST, we set the number of epochs as 10. For
In this section, theoretical framework provides consis- tently affirmative responses to the above question as well as the analogy. Considering a higher perspective, the theory is also instructive in two ways. On the one hand, from the

---

Understanding Model Ensemble in Transferable Adversarial Attack
3,0023
-.003
5..0
3,0023
0.005
0.00
601-
600
atep4
sTepp
(a) MLP
(b) CNN
Figure 4. Evaluation of ensemble attacks with increasing the number of steps using MLPs and CNNs the CIFAR- 10 dataset.
models trained on CIFAR-10, we set the number of epochs as 30. We use the Adam optimizer with setting the learning rate as 10-3, 3 We set the batch size as 64.
Section 4.2, but also to the complex behavior of variance. In the vias-variance trade-off literature (Yang et al., 2020; Lin & Dobriban, 2021; Derumigny & Schmidt-Hieher 2023; Chen et al., 2024c), different trends in variance have been observed. For example, Yang et al. (2020) suggests that variance may follow a bell-shaped curve, rising initially and then falling as network width expands. While a full investi- gation of variance behavior is beyond the scope of this work, more discussion is provided in Appendix E.5.
5.1 Evaluation on the Attack Dynamics
For each dataset (MNIST & Rishion-MNIST & CIFAR- 10), we record the attack success rate (ASR), loss value, and the variance of model predictions with increasing the number of steps for attack. We use MI-FGSM (Dong et al., 2018) to craft the adversarial example and use the cross-entropy as the loss function to optimize the adversarial perturbation. Generally, the number of steps for the transferable adver- sarial attack is set as 10 (Zhang et al., 2024b), but to study the attack dynamics more comprehensively, we perform 20- step attack. In our plots, we use the leaan--uureederse to validate our theory, which indicates the vulnerability from the theory perspective better. The first metric exhibits inverse relationship with transferability error. And the latter two metries correspond to the vulnerability and diversity components in the decomposition in Section 4.1. The num- ber of steps for attack is indicated by the z-axis. And we denote À as the weight decay. We respectively report the results on three datasets in Figure 2. Figure 3, and Figure 4.
The potential trade-off between diversity and complex- ity. Our experimental results (specifically the "variance" sub-figure), indicate the potential trade-off between diver- sity and complexity. Consider two distinct phases in the attack dynamics: 1) Initial phase of the attack (first few steps): During this phase, the adversarial example strug- gles to attack the model ensemble effectively (a low loss). Consequently, both the loss and variance increase, aligning with the cuneerablity-diversiii decomposition. 2) Potential "over-fitting" phase of the attack (subsequent steps): In this phase, the adversarial example can effectively attack the model ensemble, achieving high loss. Here, the trade-off between diversity and complexity becomes evident, partic- ularly at the final step of the attack. As the regularization term X increases (i.e., lower model complexity), the vari- ance of the model ensemble may increase. For instance, in the variance sub-figure, the red curve may exceed one of the other curves, indicating this potential trade-off.
'ulnerability-diverstt decomposition. Across all three datasets, we observe consistent pattern: as the number of steps increases, both ASR and loss values improve steadily, meaning that transferability error decreases while vulner- ability increases. Notably, the magnitude of variance is approximately ten times smaller than that of the loss value, indicating a much smaller impact on transferability error. Thus, "vulnerability"" predominantly drives the vulnerability- diversity decomposition, and the upward trend in vulnera- bility aligns with the reduction in transfcrability error.
Additional experiments. Firstly, in Appendix D.1, we present additional experimental results on the CIFAR- 100 (Krizhevsky et al., 2009) to reinforce the validity of decomposition. Secondly, in Ap- pendix D.2, we introduce weight norm constraints and in- vestigate how model complexity influences ensemble com- plexity to support Lemma 4.2. Finally, in Appendix D.3, we use the ImageNet dataset (Russakovsky et al., 2015) to provide a straightforward demonstration of how controlling model complexity enhances adversarial transferability.
vulnerability-diverse
The trend of variance. On the MNIST and Fashion- MNIST datasets, diversity initially increases but later de- clines. In contrast, on the CIFAR dataset, the variance for MLP consistently increases, whereas for CNNs, it decreases with small regularization term but increases with a larger one. This intriguing phenomenon is tied not only to the trade-off between complexity and diversity discussed in
8

---

Understanding Model Ensemble in Transferable Adversarial Attack
0.0001
0,001
3.92
6
Conclusion
08
This paper establishes a theoretical foundation for trans- ferable model ensemble adversarial attacks. We introduce three key concepts: transferability error, prediction variance, and empirical model ensemble Rademacher complexity. By decomposing transferability error into vulnerability and di- versity, we reveal a fundamental trade-off between them. Leveraging recent mathematical tools, we derive an upper bound on transferability error, validating practical insights for enhancing adversarial transferability. Extensive experi- ments support our findings, advancing the understanding of transferable model ensemble adversarial attacks.
models
models
(a) MNIST
1.2
POR
Acknowledgement
0:5
We are deeply grateful to Bowei Zhu, Xiaolin Hu, Shaojie Li, anonymous reviewers, area chair and senior area chair for their valuable suggestions and detailed discussion. Wei Yao, Huayi Tang and Yong Liu were supported by National Nat- ural Science Foundation of China (No.62476277), National Key Research and Development Program of China(NO. 2024YFE0203200), CCF-ALIMAMA TECH Kangaroo Fund(No.CCFAALIMAMA OF 2024008), and Huawei- Renmin University joint program on Information Retrieval. We also acknowledge the support provided by the fund for building worldelass universities (disciplines) of Ren- min University of China and by the funds from Beijing Key Laboratory of Big Data Management and Analysis Methods, Gaoling School of Artificial Intelligence, Renmin University of China, from Engineering Research Center of Next-Generation Intelligent Search and Recommendation, Ministry of Education, from Intelligent Social Governance Interdisciplinary Platform, Major Innovation Planning Interdisciplinary Platform for the "DoubleFirst Class" Ini- tiative, Renmin University of China, from Public Policy and Decision-making Research Lab of Renmin University of China, and from Public Computing Cloud, Renmin Univer- sity of China.
models
models
models
(b) MaSHion-MNIST
1-- 0.5001
A -0005
0.00
Vartunce
model
(c) CIFAR-10
Figure 5. Evaluation of ensemble attacks with increasing the num- ber of models using MLPs and CNNs on the three datasets.
5.2 Evaluation on the Ensemble Framework
We further validate the effectiveness of the vulnerability- diversity decomposition within the ensemble framework. Specifically, instead of focusing solely on the training dy- namics, we progressively increase the number of models in the ensemble attack evaluate the decomposition's impact. We begin by incorporating MIPs with different architec- tures and regularization terms, followed by CNNs. In total, up to 18 models are included in single attack. We depicted the results in Figure 5.
Impact Statement
We recognize the potential socictal impact of our work on transferable adversarial attacks and emphasize its contribu- tion to improving the robustness and security of machine learning models. Our study upholds high standards of sci- entific excellence through transparency, rigor, and repro- ducibility. No human subjects were involved, and no privacy or confidentiality concerns arise from the data used. We have also ensured that work does not introduce discrimi- natory biases, and we are committed to the fair and inclusive participation of all individuals in the research community. While our research focuses on theoretical advancements, we are aware of the potential risks associated with adversarial
We can consistently observe that increasing the number of ensemble models improves the attack success rate, i.e., re- duces the transferability error. On the MNIST and Fashion- MNIST datasets, both vulnerability and diversity also in- crease as the number of models grows. Although the diver- sity sometimes shows a decreasing trend on the CIFAR-10 dataset, its magnitude is approximately 100 times smaller than vulnerability, thus having a minimal impact on ASR.
9

---

Understanding Model Ensemble in Transferable Adversarial Attack
attack techniques. We encourage responsible of these insights build more secure Al systems and minimize any unintended harm.
Chen, H., Zhang, Y., Dong, Y., Yang. X., Su, H., and Zhu, J. Rethinking model ensemble in transfer-based adver- sarial attacks. In International Conference on Learning Representations, 2024b.
References
Chen, L., Lukasik, M., Jitkrittum, W., You, C., and Ku- mar, S. On bias-variance alignment in deep models. In International Conference on Learning Representations, 2024c.
Abe, T., Buchanan, E. K., Pleiss, G., and Cunningham, Pathologies of predictive diversity in deep ensembles. arXiv preprint wwXiv:2302.0770. 2023.
Bartlett, P. L. and Mendelson, S. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463 4822334422222 2002.
Chen, Z., Xie, L.,, Niu, J., Liu, X., Wei, L., and Tian, Q. Vis- former: The liioonfrriedly transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 589-598, 2021.
Bartlett, P. L.,, Foster, J., and Telgarsky, M. J. Spectrally- normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Cherkassky, V. Model complexity control and statistical learning theory. Nutural computing, 1:109-133, 2002.
Bartlett, P. L.,, Montanari, A., and Rakhlin, A. Deep learning: statistical viewpoint. Acta numerica, 30:87-201, 2021.
Cortes, C. and Vapnik, V. Support-vector networks. Ma- chine learning, 20:273-297, 1995.
Belkin, M., Hsu, D., Ma, S., and Mandal, S. Reconciling modern machine-learnin practice and the classical bias- variance trade-off. Proceedings of the National Academy of Sciences, 116(32): 15841-18849444 2019.
Deng, Y. and Mu, T. Understanding and improving ensem- ble adversarial defense. Advances in Neural Information Processing Systems, 36, 2023.
Derumigny, A. and Schmidt- Hiebbr,, J. On lower bounds for the bias varianar trade-off. The Annals of Statistics, 51 (4):1510-1533, 2023.
Bengio, Y., Louradour, J., Collobert, R., and Weston, Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Domingos, A unified bias-variance decomposition for zero-one and squared loss. AAAI/IAAI, 2000:564-569, 2000.
Bortsova, G., González-Gonzalo, C., Wetstein, S. C., Du- bost, F., Katramados, I., Hogeweg, L.,, Liefers, B., Ginneken, B., Pluim, J. P., Veta, M., et al. Adversarial attack vulnerability of medical image analysis systems: Unexplored factors. Medical Image Analysis, 73:102141, 2021.
Dong, Y., Liao, F., Pang, T., Su, H., Zhu, J., Hu, X., and Li, Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9199 2018.
Boucheron, S., Lugosi, G., and Massart, P. Concentration Inequalities: A Nonasymptotic Theory of Independence. Oxford university press, 2013.
Dong. Y., Pang. T., Su, H., and Zhu, Evading defenses to transferable adversarial examples by inniaatiooninvariant attacks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2019.
Bousquet, O. and Elisseeff, A. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
Du, K.-L. and Swamy, M. N. Neural networks and statistical learning. Springer Science & Business Media, 2013.
Chen, B., Yin, J., Chen, S., Chen, B., and Liu, X. An adap- tive model ensemble adversarial attack for boosting ad- versarial transferability, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4489- 4498, 2023.
Esposito, A. R. and Mondelli, M. Concentration without independence via information measures. IEEE Transac- tions on Information Theory, 2024.
Fan, M., Li, X., Chen, C., Zhou, W., and Li, Y. Trans- ferability bound theory: Exploring relationship between adversarial transferability and flatness. Advances in Neu- ral Information Processing Systems, 37:41882-4199 2024.
Chen, C., Liu, K., Chen, Z., Gu, Y., WU, Y., Tao, M., Fu, Z., and Ye, J. Inside: Llms' internal states retain the power of hallucination detection, arXiv preprint arXiv:2402.03744, 2024a.
10

---

Understanding Model Ensemble in Transferable Adversarial Attack
Friedman, D. and Dieng, A. B. The vendi score: A diversity evaluation metric for machine learning. arXiv preprint arXiv:2210.02410, 2022.
Kong, Z., Guo, J., Li, A., and Liu, C. Physgan: Gener- adversarial examples for
ating thysical-world-resillee autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14254-14263, 2020.
Gao, L., Zhang, Q.. Song, J., Liu, X., and Shen, H. T. Patch- wise attack for fooling deep neural network. In European Conference on Computer Vision, pp. 307-322, 2020.
Kontorovich, L. and Ramanan, K. Concentration inequal- ities for dependent random variables via the martingale method. The Annals of Probability, 2008.
Geman, S., Bienenstock, E., and Doursat, R. Neural net- works and the bias/variance dilemma. Neural computa- lion, 4(1):1-58, 1992.
Krizhevsky, A., Hinton, G., al. Learning multiple layers of features from tiny images. 2009.
Golowich, N., Rakhlin, A., and Shamir, O. Size- Size-independent sample complexity of neural networks. In Conference On Learning Theory, 2018.
Lampert, C. H., Ralaivola, L.., and Zimin, A. Dependency- dependent bounds for sums of dependent random vari- ables. arXiv preprint arXiv: 1.01404, 2018.
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explain- ing and harnessing adversarial examples, arXiv preprint arXiv: 1412.66412.6722 2014.
Lancaster, H. Correlation and complete dependence of random variables. The Annals of Mathematical Statistics, 34(4):1315-1321, 1963.
Gu, J., Jia, X., de Jorge, P., Yu, W., Liu, X., Ma, A., Xun, Y., Hu, A., Khakzar, A., Li, Z., et al. A survey transferabil- ity of adversarial examples across deep neural networks. Transactions Machine Learning Research, 2024.
LeCun, Y. The mnist database of handwritten digits, 1998. URL http://yann. lecun. com/emcoom/exii mnist/.
Gubri, M., Cordy, M., Papadakis, M., Le Traon, Y., and Sen, K. Efficient and transferable adversarial examples from bayesian neural networks. In Uncertainty in Artificial Intelligence, pp. 738-748. 2022a.
Lei, Y., Dogan, Ü., Zhou, D.-.., and Kloft, M. Data- dependent generalization bounds for multi-class classifi- cation. IEEE Transactions on Information Theory, 65(5): 2995-3022 2019.
Gubri, M., Cordy, M., Papadakis, M., Traon, Y. L., and Sen, K. Lgv: Boosting adversarial example transferability from large geometric vicinity. In European Conference on Computer Vision, pp. 603-618, 2022b.
Li, Q. Guo, Y., Zuo, W., and Chen, H. Making substi- tute models more bayesian can enhance transfcrability of adversarial examples. In International Conference on Learning Representations, 2023.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn ing for image recogmition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Li, Q., Guo, Y., Zuo, W., and Chen, H. Improving adver- sarial transferability via intermediate-llveel perturbation decay. Advances in Neural Information Processing Sys- lems, 36, 2024.
Hellinger, E. Neue begründung der theorie quadratischer formen von unendlichvielen veränderlichen. (in german). Journal für die reine und angewandte Mathematik, pp. 210-271, 1909.
Li, S. and Liu, Y. Towards sharper generalization bounds for structured prediction. Advances in Neural Information Processing Systems, 02226444268575 2021.
Li, Y., Bai, S., Zhou, Y., Xie, C., Zhang, Z., and Yuille, A. Learning transferable adversarial examples via ghost networks. In Proceedings of the AAAI conference on artificial intelligence, pp. 11458-11465, 2020.
Hu, X., Li, S., and Liu, Y. Generalization bounds for fed- erated learning: Fast rates, unparticipating clients and unbounded losses. In International Conference on Learn- ing Representations, 2023.
Liese, and Vajda, I. On divergences and informations in statistics and information theory. IEEE Transactions Information Theory, 52(10) 44122 2006.
Kariyappa, S. and Qureshi, M. K. Improving adversarial robustness of ensembles with diversity training. arXiv preprint arXiv:1901. 09981, 2019.
Lin, J., Song, C., He, K., Wang, L., and Hopcroft, J. E. Nesterov accelerated gradient and scale invariance for adversarial attacks. arXiv preprint arXiv:1908.06288 2019.
Koltchinskii, V. and Panchenko, D. Rademacher processes and bounding the risk of function learning. In High di- mensional probability II, pp. 443-457. Springer, 2000.

---

Understanding Model Ensemble in Transferable Adversarial Attack
Lin, L. and Dobriban, E. What causes the test error? going beyond bias- variance via anova. Journal of Machine Learning Research, 22(155):1-22, 2021.
Papernot, N., McDaniel, P., and Goodfellow, I. Transfer- ability in machine learning: from phenomena black- box attacks using adversarial samples. arXiv preprint arXiv: arXiv: (ee::0700007 2016.
Liu, C., Chen, H., Zhang. Y., Dong, Y., and Zhu, J. Scaling laws for black box adversarial attacks. arXiv preprinl arXiv:2411.16722 2024.
Garrado-eernnnder E.,, Ambroladze, A., Shawe-Taylor, J., and Sun, S. Pac-bayes bounds with data dependent priors. The Journal of Machine Learning Research, 13(1):3507-
Liu, Y. Chen, X., Liu, C., and Song. D. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning Representations, 2017.
3531, 2012.
Rice, L.., Wong, E., and Kolter, Z. Overfitting in adversari- ally robust deep learning. In International conference on machine learning, pp. 8093-8100 8104, 2020.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115: 211-252, 2015.
Ma, W., Li, Y.. Jia, X., and Xu, W. Transferable adversar- ial attack for both vision transformers and convolutional networks via momentum integrated gradients. In Pro- ceedings of the IEEF/CVF International Conference Computer Vision, pp. 4630-4639, 2023.
Sason, I. and Verdú, S. Upper bounds the relative en- tropy and rényi divergence as function of total variation distance for finite alphabets. In 2015 IEEE Information Theory Workshop-Fond (ITW), pp. 214-218 IEEE, 2015.
Martins, A. and Astudillo, R. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International conference on machine learning, pp. 1614 1623. PMLR, 2016.
Schmidt, L., Santurkar, S., Tsipras, D., Talwar, K., and Madry, A. Adversarially robust generalization requires more data. Advances in neural information processing systems, 31, 2018.
Mohri, M. and Rostamizadeh, A. Rademacher complex- ity bounds for non-iid processes. Advances in neural information processing systems, 21, 2008.
Shalev-Shwartz, S. and Ben-David, S. Understanding ma- chine learning: From theory to algorithms. Cambridge university press, 2014.
Mohri, M., Rostamizadeh, A., and Talwalkar, A. Founda- tions of machine learning. MIT press, 2018.
Shalev-Shwartz, S., Shamir, O., Srebro, N., and Sridharan, K. Learnability, stability and uniform convergence. The Journal of Machine Learning Research, 1 11:2635-2670, 2010.
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechan- ics: Theory and Experiment, 2021 124003, 2021.
J. and Cristianini, N. Kernel methods for pattern analysis. Cambridge university press, 2004.
Neal, B. On the virianer tradeoff: Textbooks need update. arXiv preprint arXiv: 19/2.08280 2019.
Shiryaev, A. N. Probability-1, volume 95. Springer, 2016.
Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. arXiv preprint arXiv: 100.. 00000000000.... 2017.
Neal, B., Mittal, S., Baratin, A., Tantia, V., Scicluna, M., Lacoste-Julien, S., and Mitliagkas, I. A modern take on the bias variance tradeoff neural networks. arXiv preprint 1.Xiv:8800.08555 2018.
Silva, S. H. and Najafirad, Opportunities and challenges in deep learning adversarial robustness: A survey. arXiv preprint arXiv:2007.0075: 2020.
Neyshabur, B., Li, Z., Bhojanapalli, S., LeCun, Y., and Srebro, N. Towards understanding the role of over- iroorettizzttion in generalization of neural networks. arXiv preprint arXiv: 1805. 018805/2776 /076 2018
parametrization
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint arXiv: 111.1144955555 2014.
Ortega, L. A., Cabañas, R., and Masegosa, A. Diversity and generalization in neural network ensembles. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 11111-111111 11700, PMLR, 2022.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D.., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks, arXiv preprint orXiv: 13/2,6199 2013.
12

---

Understanding Model Ensemble in Transferable Adversarial Attack
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR). 2016.
Wang, Z.,, Zhang, Z., Liang, S., and Wang, X. Diversifying the high-level features for better adversarial transferabil- ity, arXiv preprint 2rXiv:2304.10130 2023b.
Wood, D., Mu, T., Webb, A. M., Reeve, H. W., Lujan, M., and Brown, G. A unified theory of diversity in ensemble learning. Journal of Machine Learning Research, 24 (359);1-49, 2024.
Tang. B., Wang. Z., Bin, Y., Dou, Q. Yang. Y. and Shen, H. T. Ensemble diversity facilitates adversarial transfer- ability. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24377- 24386, 2024.
Wu, H., Ou, G., Wu, W., and Zheng, Z. Improving transferable targeted adversarial attacks with model self- enhancement. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 24615-24624 2024.
Tramèr, F., Papernot, N., Goodfellow, I., Boneh, D., and Me- Daniel, The space of transferable adversarial examples. arXiv preprint arXiv: 170400345, 2017.
Vapnik, V. Estimation of dependences based on empirical data. Springer Science & Business Media, 2006.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist. novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 708.07747,
Vapnik, V.N. Statistical learning theory. Wille--nnerscceenc,, 1998.
Xiaosen, W., Tong, K., and He, K. Rethinking the backward propagation for adversarial transferability. Advances in Neural Information Processing Systems, 16:1905-1922 2023.
Vapnik, V. An overview of statistical learning theory. IEEE transactions on neural networks, 10(5):988 999, 1999.
Vapnik, V. N. and Chervonenkis, A. Y. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and Its Applications, 16(2):264 280, 1971.
Xie, C., Zhang, Z., Zhou, Y., Bai, S., Wang. J., Ren, Z., and Yuille, A. L. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2730-2739, 2019.
Wang, K., He, X., Wang, W., and Wang, X. Boosting adversarial transferability by block shuffle and rotation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 24336-24346, 2024.
Xiong, Y., Lin. J., Zhang, M., Hopcroft, J. E., and He, K. Stochastic variance reduced ensemble adversarial attack for boosting the adversarial transferability. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pp. 14983-14992, 2022.
Wang, X. and He, K. Enhancing the transferability of adver- sarial attacks through variance tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1924-1933, 1933, 2021.
Xu, A. and Raginsky, M. Information ceemmattoo--hhoretti analy- sis of generalization capability of learning algorithms. Advances in neural information processing systems, 30, 2017.
Wang, X., He, X., Wang. J., and He, K. Admix: Enhancing the transferability of adversarial attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16158-16167, 2021.
Yan, C. W., Cheung, T.-H., and Yeung, D.-Y. Ila-da: Improv- ing transferability of intermediate level attack with data augmentation. In International Conference on Learning Representations, 2023.
Wang, X., Zhang, Z., Tong, K., Gong, D., He, K., Li, Z., and Liu, W. Triangle attack: A query-efficient decision-basee based adversarial attack. In European conference computer vision, pp. 156-174. Springer, 2022.
Yang. Z., Yu, Y., You, C., Steinhardt, J., and Ma, Y. Rethink- ing bias-variance trade-off for generalization of neural networks. In International Conference Machine Learn- ing. pp. 10767-10777, 2020.
Wang, X., Zhang, Z., and Zhang, J. Structure invariant transformation for better adversarial transferability. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 00.4607-455 2023a.
Yang. Z., Li, L., Xu, X., Zuo, S., Chen, Q. Zhou, P., Ru- binstein, B., Zhang, C., and Li, B. Trs: Transfcrability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Pro- cessing Systems, 34:17642-17653 2021.
Wang, Y. and Farnia, F. On the role of generalization in transfcrability of adversarial examples. In Uncertainty in Artificial Intelligence, pp. 2259-22277 2023.
13

---

Understanding Model Ensemble in Transferable Adversarial Attack
Yin, D., Kannan, R., and Bartlett, P. Rademacher complexity for adversarially robust generalization. In International conference machine learning, pp. 7085-7094, PMLR, 2019.
Yu, C., Han, B., Shen, L., Yu, J., Gong, C., Gong, M., and Liu, T. Understanding robust overfitting of adversarial training and beyond. In International Conference Machine Learning, pp. 1-595-25610, 2022.
Zhang, and Li, C. Adversarial examples: Opportunities and challenges. IEEE transactions on neural networks and learning systems, 31(7):2578-2555 2019.
Zhang, R.-R. and Amini, M.-R. Generalization bounds for learning under graph-dependence: survey. Machine Learning, 1137))3929-3959, 2024.
Zhang, R. R., Liu, X., Wang, Y., and Wang, L. Mediarmid- type inequalities for graph-dependent variables and stabil- ity bounds. Advances in Neural Information Processing Systems, 32, 2019.
Zhang. Y., Hu, S., Zhang, I.. Y., Shi, J., Li, M., Liu, X., and Jin, H. Why does little robustness help? further step towards understanding adversarial transferability. In Proceedings of the 45th IEEE Symposium on Security and Privacy (S&P') (4), volume 2, 2024a.
Zhang, Z., Zhu, R., Yao, W., Wang. X., and Xu, C. Bag of tricks to boost adversarial transferability. arXiv preprint arXiv:2401.088 04.344 2024b.
Zhao, A., Chu, T., Liu, Y., Li, W., Li, J., and Duan, L. Minimizing maximum model discrepancy for transfer- able black-box targeted attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8153-8162. 2023.
Zhu, R., Zhang, Z., Liang, S., Liu, Z., and Xu, C. Learning to transform dynamically for better adversarial transfer- ability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 24273- 24283, 2024.
Zou, J., Pan, Z., Qiu, J., Liu, X., Rui, T., and Li, W. Im- proving the transferability of adversarial examples with resized-diivers--innuts, diverslty-ensemble and region fit- ting. In European Conference on Computer Vision, pp. 563-579, 2020.
Zou, X. and Liu, W. Generalization bounds for adversar- ial contrastive learning. Journal of Machine Learning Research, 24(114):1-5 2023.
14

---

Understanding Model Ensemble in Transferable Adversarial Attack
A More Related Work
A.1 Transferable Adversarial Attack
Input transformation. Input transformation-basee based attacks have shown great effectiveness in improving transferability and can be combined with gradient-based attacks. Most input transformation techniques rely on the fundamental idea of applying data augmentation strategies to prevent overfitting to the surrogate model (Gu et al., 2024). Such methods adopt various input transformations to further improve the transferability of adversarial examples (Wang et al., 2023b;a). For instance, random resizing and padding (Xie et al., 2019), downscaling (Lin et al., 2019), mixing (Wang et al., 2021), automated data augmentation (Yan et al., 2023), block shuffle and rotation (Wang et al., 2024), and dynamical transformation (Zhu et al., 2024).
Gradient-based optimization. The central concept of these methods is to develop optimization techniques in the generation of adversarial examples to achieve better transferability. Dong et al. (2018): Lin et al. (2019); Wang & He (2021) draw an analogy between generating adversarial examples and the model training process. Therefore, conventional optimization methods that improve model generalization can also benefit adversarial transferability. In gradient- based optimization methods, adversarial perturbations are directly optimized based on one or more surrogate models during inference. Some popular ideas include applying momentum (Dong et al., 2018), Nesterov accelerated gradient (Lin et al., 2019), scheduled step size (Gao et al., 2020) and gradient variance reduction (Wang & He, 2021; Xiong et al., 2022). There also other elegantly designed techniques in recent years (Gubri al., 2022b; Wang et al., 2022; Xiaosen al., 2023; Li et al., 2024; Wu et al., 2024; Zhang et al., 2024b), such as collecting weights (Gubri et al., 2022b), modifying gradient calculation (Xiaosen et al., 2023) and applying integrated gradients (Ma et al., 2023).
Model ensemble attack. Motivated by the use of model ensembles in machine learning. researchers have developed diverse ensemble attack strategies to obtain transferable adversarial examples (Gu et al., 2024). It is a powerful attack that employs an ensemble of models to simultaneously generate adversarial samples. It can not only integrate with advanced gradient-based optimization methods, but also harness the unique strengths of each individual model (Tang et al., 2024). Some popular ensemble paradigms include loss- ensemble (Dong et al., 2018), prediction-basek (Liu al., 2017), logit-based ensemble (Dong et al., 2018), and longitudinal strategy (Li et al., 2020). There is also some deep analysis to compare these ensemble paradigms (Zhang et al., 2024b). Moreover, advanced ensemble algorithms have been created to ensure better adversarial transferability (Zou et al., 2020; Gubri et al., 2022a; Xiong et al., 2022; Chen et al., 2023; Li et al., 2023; Wu et al., 2024; Chen et al., 2024b).
A.2 Statistical Learning Theory
Statistical learning theory forms the theoretical backhone of modern machine learning by providing rigorous frameworks for understanding model generalization (Vapnik, 1999). It introduces foundational concepts such as Rademacher complex- ity (Bartlett& & Mendelson, 2002), VC dimension (Vapnik & Chervonenkis, 1971), structural risk minimization (Vapnik, 1998). It has also been instrumental in the development of Support Vector Machines (Cortes & Vapnik, 1995) and kernel methods (Shawe-Taylor & Cristianini, 2004), which remain pivotal in supervised learning tasks. Recent advances extend statistical learning theory to deep learning, addressing challenges of high-dimensional data and model complexity (Bartlett et al., 2021). These contributions have significantly enhanced the capability to design robust learning algorithms that generalize well across diverse applications (Du & Swamy, 2013). In addition, there are also some other novel theoretical frameworks, such as information-theereiicc analysis (Xu & Raginsky, 2017), PAC-Bayes bounds (ParradoHHeráánde et al., 2012), transductive learning (Vapnik, 2006), and stability analysis (Bousquet & Elisseeff, 2002: 2010). Most of them derive bound of the order O VM while some others derive sharper bound of generalization (Li
et al.,
Liu, 2021) of the order O ( ). Such theoretical analysis suggests that with the increase of the dataset volume, the model generalization will become better.
15

---

Understanding Model Ensemble in Transferable Adversarial Attack
B Proof of Generalized Rademacher Complexity
B.1 Preliminary
For simplicity. denote a(0;;z) as fx)... For 1-Lipschitz loss function (x)) (for example, hinge loss {(f()),)) max (0, 1 - (1)),) there holds:
(2)
sup #EZ
{((((())))))
$ E8
sup
())
xE2 N
N
E
sup
o:f((z)
R2).
:E2
N
1=1
So can bound Ry (2) instead of R. (Z).
B.2 Linear Model
Given Section B.1, provide the bound below.
Lemma B.1 (Linear Model). Let fo + w where x, Rd. Given N classifiers from H. assume that || S B and || 2 & C. Then Rx < BC
VN
Proof.
Rx (2)
sup ||a||a B N
sf((())
(f(x) wf(x) x)
sup
||a||| N
T
(a=b= b"
sup
og Wi;
|||||2B
024
(a b < ||0|2/666
N
B
NV Σ
E
02W0
(Jensen inequality: Ex S <VEa*
N
1=1
B
E
o u
Σ
o
N
B
E
w
T
W
Σ
o.omw wy
N
j= ii
B
wi
N
N max | (12)
N
16

---

Understanding Model Ensemble in Transferable Adversarial Attack
BC
(|w||2 S C)
V N
B.3 Two-layer Neural Network
Given Section B.1, we provide the bound below.
Lemma B.2 (Two-layer Neural Network). Let = far + w (UUx), where a E ER4, U RRmxd, w E R m the number of the hidden layer, and (xx) max (0, x) is the element-wise ReLU function. Given classifiers from H. assume that | - B, 11012 I B', and |Ui1|2 S C, where Ug is the j-th row of U. Then Rx < MMBB'C
VN
Proof.
Rx Rx(2)
sup
orli(z)
N
||r|aa
E
sup
awwo ((U())
(f((a) w/ (U(x))
N
||alIaBB
i=1
E
sup
(UU))
(1ww12 S B')
N
||=|||<
B3
i=1
B'
E
sup
Σ oiVi Vi i=1
(Denote V;
E ))
N
||>|||2B
((Ummo))
B'
N Σ V,V i=1
N
E
sup
||
B'
N Σ
sup
V T Vi Σ Σ
o,o,VTVV
N
||
114
B'
N
sup
Σ
V.TV
Vi
||
i=1
S
sup
[N max [Vel12)
N
|
B'
mmax (max//Vill \|Vill))
sup
o(Unx))
For Vi
E RR we have
((Um))))
(UU))))
max [Vall2)
sup
=
sup
max
|
(UUmi)))
17

---

Understanding Model Ensemble in Transferable Adversarial Attack
U
sup l
max
(1((z) S (x))
Univ
m
sup
max max ||U
|| B
I m
sup
maxmax ||/Eell |201|
|
MBC
(|>||2 S B and ||Ug,|12 S ()
Finally,
B
VmBB'C N
RN Rx v
sup ||
max lVVilla
<
VN
The proof is complete.
B.4 Proof of Lemma 4.2
For simplicity, denote [(;;;::)) as fi(w) and i (1,-.. N] as i E [N].
First, we begin with lemma, which is a similar version of Lemma from (Golowich et al., 2018).
Lemma B.3, Let & be a I-Lipschitz, hhatoveeoo activation activationfunnciion which is applied element-wise (such as the ReLU), Then for any class of vector-valued functions F, any convex and monotonically increasing function R - [0, oo) and R.,, there holds:
( 1)
Σ as (W)))
E.
sup JEJ, WET ER
9
2 E sup g fEF
R Σ
(12)
Proof. Let w1.""" wh be the rows of W, we have
N do
2
Σ o ( fo (x))
j=1
(1w0))1
w
)
)}²
Σ
||wl 1|
Σ a.
TTugl fc (or)
(6(ar) aι(x:)
Therefore, the supremum of this over all w1,- ||u,||| R for some j and ||we|] = for all i + j. So we have
such that | Σラ ||05
< R² must be attained when
E.
sup
Σ o, 2
(W (+))
= E.
sup
Σ 1=1
o, (w (w (mTT((oo)))
JEF SEF.W.I W.F.W:
Since g(lz1) s g(2) + 2), this can be upper bounded by
N
(- N (w
N Σ o. (w fo (c))
Eg sup g
Σ
o. (w fo ((0) + E. sup g
(wTFt
(w fo (( = E. sup g
where the equality follows from the symmetry in the distribution of the a, random variables. The right hand side in turn be upper bounded by
N N
N Σ i=1
2 Eg
oow fc fx(
2 E.
sup JEF w: =
||m||
o:fi (z)
i=1
18

---

Understanding Model Ensemble in Transferable Adversarial Attack
= 2. E sup SEF
R
Σ i=
o (x)
With this lemma in hand, we can prove Lemma 4.2:
Proof. For À > the rademacher complexity can be upper bounded as
NRv (2) = Eg sup
(osf((z)
f1r" In
N
< log Eo exp
exp
A
fff((z))
(Jensen's 's inequality)
==1
N
$ log E.r supexp
sup (E(m)
A
Jobi-1 101--161-2(.-- Φi (W2,1z)))
J.. 1())
n
We write this last expression as
N
0 (fia-1(z)) ))
ā log E. sup exp
Ti A
Σ
i=1
1
N
2 1))
< log
2 2.E supexp
7i 17-7i-1
Σ
(Lemma B.3)
...
(Repeatedly apply Lemma B.3)
1-1
log
2/-2. Eg saa exp
A
T
Σ o, (
1=1
1)
log
21-1. Eo dns dxa
IIII
T
Σ
W 1112
i=1
Assume that i E [N] maximizes
1-1
N
W.. )
supexp
A
T;
Σ
o.
[=1
Therefore,
Σ o, W 12 )
N
X log
2/-1 EE sup exp
A
T;
1=1
1-1
N
X
log
2/ Eg exp
A
T
σ.
W
1,1
= log (2'-1 1 EE exp (AZ))
19

---

Understanding Model Ensemble in Transferable Adversarial Attack
(I - 1) 1og(2)
log (Eg exp (AZ))
X A
(1 1) log(2) (2)
+ log Exp A(2 - -EZ)] + EZ
For EZ, we have
1-1
N
EZ-
T;
Eg
Σ
o
W
1
1-1
N
T;
Eg
Σ
010 (Wizz)² 7
(W))1z)
T.
(RBVV]
=B T
Note that a deterministic function of the i..dd random variables "1,". NN and satisfies
Z(o ".. - ((01,... -o. , - 2B T
This means that Z satisfies bounded-differenee condition. According to Theorem 6.2 in Boucheron al. (2013), Z is Sab-Gaussian with variance factor
(2BT)2 NB?T3,
and satisfies
1
log (Eexp - -EZ)} <
X² NB222
- 2
X
NB
log (2)
Choosing À
and using the above, we get that
BTVN
(1 - 1) l2g
Ee - + EZ < (V(2log +1) BTVN
+
Finally, we get
(2 2)( 1) BT
RN (2)
C Main Proof
C.1 Transferability Error and Generalization Error
For (z,v), there holds
TF()) -r(=")) - -Tr(2) S -r((z)) - I.(2)) (Ix(+)) Lx(("))
20

---

Understanding Model Ensemble in Transferable Adversarial Attack
(Lp(=*) - Lg(=")) -((=")))))))))) (Ix(2) Lp(z))
s sup (Lp(z)- Lg(()) + sup (LE(Z) Lp(z))
=B).
=EB,(x)
S sup(Lp(z) - Le(x)) + sup(Le(2) -p(2)).
s62
=E2
S 2 sup [D((z) LF(z)].
262
C.2 Proof of Theorem 4.1
We prove a general version of the theorem as follows:
Theorem C.I. Consider the squared error loss 100,2,01 y) (5(0;z) vi² for data point = (x, Assume that the data is generated by a function g(x) such that g(x) + P. where the zero-mean noise has variance of rf and is independent of Then there holds
TE(z,e) Lpp("") 1/² ((0;a) -g((x) - JO;
(13)
Diversity
Artack
Remark. The irreducible error n² is constant because it arises from inherent noise and randomness in the data (Geman et al., 1992).
Proof. Given Eq. (5), it is equivalent to prove
Lp(2) Varo + [g(x) [(; + +
(14)
Note that
Lp(z) = Ea-po [f(0;.) y1²
EgNPo [f(0:z) g(x) + g(z) y)²
= EgNPo [(f(6:z) - g(x))² + (g(x) - yy²² + 2(g(z) - ()((==))) 9(x))]
Recall that y = g(x) + p with (pp) and Vor(p) n we have
Ba-po (g(x)) - y)² =1
and
Ea-Po -))(()) g(x))] -2E(p)EoaPP [f(0;)) - g(x) = 0.
Therefore,
Lp(z) = BaNpo (8(0xx) o(x))² -12
(15)
Likewise, we decompose the first term as
Es (f) g(x))²
-Eo (f(0;)) Eoʃ x) + Ea,((;(00)) - -xx)
E. [([(0:=) Eos(0;; 2); + (0;2)) ((x))²
-2([(0;=)) EE x) (B). - -x()))
=E,(((0;x) - (e](0;))) EB (0; 2(x))2
Vara (10xx)
(o(()))
2. [f(0;=)) Eg (O:)) g(x))
with the derivations for the second and third term:
(f; (o, x)); (0; 2a +) gx())
21

---

Understanding Model Ensemble in Transferable Adversarial Attack
(g(x) (F(;x)))
and
Eg [f(0;z) - Eef(0; (B, g(x)
(Eo ((0;=))) (()))oo(( (Bof(0;z))² g(z) ((aa)
0.
As result,
Eg [f - -xx)]² =
- Po x0;
(16)
Combining the above results and complete the proof.
To prove Theorem 4.1. we just set p = O in the above general version of theorem.
Similarly, consider the empirical version of Theorem 4.1, we decompose Lg(2) as follows:
Theorem C.2 (Vulnerability-divernn
Decomposition (empirical version)). Consider the squared error loss (f x),9)
(f(0;z) - vi* for data point z = (xt, v)- Let f(0;x) = N E f(0;;;) x) be the expectation of prediction over the distribution on the parameter space. Then there holds
N
Læ(²)
= (ff(O:z).v)
1
Σ
N
1=1
= =1(f(0=)).))
((0:))) E((n)))
Valnerabilitt
Diversity
The proof is similar to the above:
LE(2)= N (f(0;:)- - 3)
N
N
1 N
2
[(0;;a)- - NV.
Σ
[(0xz)
Σ
[(0 -1
N
N
i=1
:=1
N
N
2
((0::x)
1
Σ
[(0:))))
+
(0(;z)) -1)
1=1
1=1
N
N
2
[(0:;00) Σ
[(0;;)) -B
N
1=1
2
-(((0x)),))
[(0xz))
N ((0ria))
+
Vulnerability
Diversity
N
N Σ
N
[(O(z))
f(0f:z))
f(0;: 23) -
N
N
N
i=1
The last terms equals to because
N
N f(0;:))
(
1 N
Σ f(0;;a))
f(0;: E50;;)))
N i=1
N
im1
22

---

Understanding Model Ensemble in Transferable Adversarial Attack
2
N
N
2
N
Σ
f(0(:
Σ
f(O(:))
1
Σ
f((::z)
+
Σ f(0
N
N
2=1
The proof is complete.
C.3 Proof of Theorem 4.1 (KL Divergence Loss)
In this section, we consider different problem setting and show how to extend Theorem 4.1 KL divergence loss. We first define multi-class classification in the context of transferable model ensemble adversarial attack.
Multi-class classification. Consider a k-classification problem. Given the input space X' C IR and the output space y' C TR,, we have joint distribution Pz over the input space Z = A'' V The training set Zurin = [z, lz = (x,, y;) e y, E {0, 1} ||y:|l = 1, 1, , .M.. which consists of M examples drawn independently from Pz. We denote the hypothesis space by H. : + y and the parameter space by O. Let f(0:-) E H. be a classifier parameterized by .'
O, trained for a classification task using a loss function E y y - RR Let Pe represent the distribution over the parameter space O. Define PAN as the joint distribution over the product space ON which denotes the space of N such sets of parameters. We use Zunin to train N surrogate models ((0 /(00 for model ensemble. The training process of these N classifiers can be viewed as sampling the parameter sets 0² (0I..... 0N) from the distribution PaN, i.e., 0 ~ PaN. For a data point z = (x E Z and N classifiers for model ensemble attack, let the model output be normalized (i.e., (0; x]|| = 1). Define the empirical risk (z( and the population risk Lz(z) of the adversarial example z as
1 Lr(z) N Σ (((0g)),)))
(17)
and Lp(z)= = Eg NP Lg(z).
(18)
Intuitively, transferable adversarial example leads large 1.(22) because it can attack many classifiers with parameter 0 E O Therefore, the most transferable adversarial example zt (x", y) around is defined
x arg max (z),,
(19)
xEB, (x)
where Bx(x) = (x: - - S c) is adversarial region centered at & with radius 0. However, the expectation in (z)) cannot be computed directly. Thus, when generating adversarial examples, the empirical version Eq, (2) is used in practice, such as loss-based ensemble attack (Dong et al., 2018). So the adversarial example z = (x,y) is obtained from
= arg max LE( (x).(x)
(20)
There is a gap between the adversarial example z we find and the most transferable one z". It is due to the fact that the ensemble classifiers cannot cover the whole parameter space the classifier, i.e., there is a difference between Lr(z) and Lz)z). Accordingly, the core objective of transferable model ensemble attack is to design approaches that approximate Lg (z) to (z),, thereby increasing the transferability of adversarial examples.
Note that the training process of N classifiers can be viewed as sampling the parameter sets 0N (01,...,00 0x) from the distribution PÆN, i.e., 0N ~ PÆN. We generate a transferable adversarial example using these N models and evaluate its performance on another N models gN (fI,..., .,.), which is an independent copy of V For data 2 (a, y) E Z and the parameter set @N our aim is bound the difference of attack performance between the given N models 7 and N unknown models 0.,, In other words, if
An adversarial example can effectively attack the given model ensemble, i.e., a large LE Lz(z).
There is guarantee for the difference of attack performance between known and unknown models, i.e., small Ey ~P,
PP. an [L((z) [=(2)]
23

---

Understanding Model Ensemble in Transferable Adversarial Attack
Then there is adversarial transferability guarantee for z. We perform the decomposition to analyze L (g) in this section.
While provide information-theon analysis to deal with B 2..0%~2..e [Lp(z) [=() in Appendix C.7.
Now we decompose (z) into vulnerability, diversity and constants. It is similar version of Theorem 4.1 using KL divergence loss.
Proposition C.3 (Vulnerability-ddveeisty Decomposition), Consider KI. divergence as the loss function, i.e,, ((S x) = Σド=1 f(0,; x) log f(0.x), Let f(0:x) ())x be the normalized geometric mean of ensembles { ( Then there holds
N
(z) = =(,,T[(0:)))
{((;0x),((0;xx))
(21)
N
Vuluenabiliey
1=1
Diversity
The "Vulnerability" term measures the risk of a data point being compromised by the model ensemble. If the model ensemble is sufficiently strong to fit the direction opposite to the target label, the resulting high loss theoretically improves (z). This insight suggests that selecting strong attackers as ensemble components leads to lower Lz(z). The "Diversity" term implies that selecting diverse attackers in a model ensemble attack theoretically contributing to a increase Læ(z). In conclusion, it provides similar guideline comparing to Theorem 4.1: we are supposed to choose ensemble components that both strong and diverse.
Proof. We first introduce Bregman divergence.
Definition C.4 (Bregman divergence). Let Φ Ω + R be function that is: a) strictly convex, b) continuously differentiable, c) defined on closed convex set Ω. Then the Bregman divergence is defined
yx = ((x) - <ly) (Vo(y) x y), Vx,y Ω
That is, the difference between the value of á at x and the first order Taylor expansion of ó around y evaluated at point Notice that let Ω y and KL divergence can be special case of Bregman divergence if ((x) (x; log æ; x;) 6(x) = E z log F;, where I; (i 1.... k) are the components of x.
Now we start the proof. It follows the Bregman ambiguity decomposition in Wood et al. (2024).
Denote f, = f(0,; x) E IR and
N Σ Vo (E)
=
Vo
(22)
which is the Bregman Centroid Combiner (Wood et al., 2024) of ensembles ( Therefore, we have
VO
(f:),
so that
f y, Vo(t)) -f-y, Vo())
In other words
Boly, f] f) = <by) ((f) (y- - vd(l)
1 N
= =(yy) o(f) + N
Σ
f - - vo(f))
1=1
((y) 1 N
1 N
1 N
1 N
Σ
(ff.)
Σ
- f. vo(a))
+
- (f) +
Σ f f. vo(())
N 1=1
N
N
1=1
1=1
24

---

Understanding Model Ensemble in Transferable Adversarial Attack
B (y, f.)
Σ B (f,f.)
(23)
N
Let ((x) = (x: log - Iz)) in Eq. (23) and we have
N D
1 N N
D
(y,
Σ
DKL f.).
1=1
(x)
Replace DKL with l and we can prove the result.
C.4 Proof of Theorem 4.3
We first define a divergence measure taken into account. Given a measurable space and two measures which render it measure space, we denote // < if v is absolutely continuous with respect to μ. Hellinger integrals are defined below:
Definition C.5 (Hellinger integrals (Hellinger, 1909)). Let v, p be two probability measures on (R) and satisfy v < and Pa : IR+ >> IR be defined as (x((x) = xa. Then the Hellinger integral of order a given by
Ha(viln)= =
/
dv dp
du
It can be seen as d-Divergence with specific parametrised choice of (Liese & Vajda, 2006). For a > 1, the Hellinger integral measures the divergence between two probability distributions (Liese & Vajda, 2006). There holds (vll)))) E [1, +oo), a 1, and it equals to 1 if the two measures coincide (Shiryaev, 2016). Given such divergence measure, we now provide the proof.
Proof. From Section C.I, we know that
TE(2) = Lp(2*) Lp(z) < Lp(z*) - (LE()) Lx(²"))
(Lp(z*) - Lg(3")) + (Læ(z) - Lp(z))
s sup
(Lp(z) LE(3)) + sup (LE(Z) - Lp(x))
=B).(x)
=EB,(x)
sup(Tpp(2) Lg(2)) sup((=(2)) Ip((2))
sEZ
xE2
Let 0N = ON),
0N),0' ON = that satisfy 0N ,N, ~ PeN, and the m-th member is different, i.e., 0' + 0m.
We define
Lx 1
(((0)=)))))
N
and
(E = sup (Lp(=) - Læ(z))
zEZ
$ (F')) sup (bp(z) (x))
We have
(i(E) 4 (1(E)) = sup (Lp(z) - Lg(2)) - sup (Lp(z) (2)
2E2
=E2
25

---

Understanding Model Ensemble in Transferable Adversarial Attack
sup xE2
(Lp(2) Lg(2) (Lp(z) (z)))
sup
(Te - (
=EZ
N
N
1
Σ ((((0)2),))
Σ (f((0z)).)))
=
sup N
ZEZ
1=1
By assuming that loss function é is bounded by 3, have
(1 - (F() 1 N
According to Theorem in Esposito & Mondelli (2024), for all 6 E 1) and a > 1, with probability at least
6. we
have
-d
H
PaN || 8
ag
(E) S EaN [E) +
In
(24)
2(ax
I)N
6
Denote z(0zz))) as f((x) and x0);;z) as (1). Then we estimate the upper bound of BON~ ~ ~ (I(EE) as follows:
Eg [B, = BgN
Lm)())
=EEZ
BaN
sup B - "009,---6a000 P2PNbN (L) Le(²))
1=E2
BBN a'
sup (Z Lx())) LEEZ
(Jensen inequality)
1
EgN 8i
sup xEZ
Σ ((f(0(=)).))
' ((((0))))))
N
Eg gex
sup
1
Σ a.
-(f(())))
=EZ N
Σ 010((1(0)000 }
1
1
< Ear
sup 2E2
a.((f(()).))
+ EEOEoN
sup 262 N
N
1
2 EgN
sup #EZ N
a(((((()))))
2EEN [RN(F).
(25)
Since changing one element in A changes Rr (F) by at most (2024) and obtain that for all 8 E (0, 1), with probability at least
N we again apply Theorem in Esposito & Mondelli 1. we have
0g82
- H
PON || |P
EaN [RN(F)) & (N) +
In
(26)
2(a - 1)N
1
Likewise, if we define
2₂(E) = sup (L)(z) - Lp(2)},
ZE2
(2 = sup (Z) Lp(2)}
26

---

Understanding Model Ensemble in Transferable Adversarial Attack
then have
D2(E) 22(EE) = sup (Lx(2) - Lp(2)) - sup (
xE2
2E2
S sup (LE(Z) Lp(2) (L - -
xE2
sup =EZ
(L - LE<(2))
N
N
= N
1
sup sE2
Σ
((((02)))))
Σ ((((O))))v)
According to the assumption that loss function e is bounded by 8, we have
6 N
(2(E)) - 22( -
According to Theorem in Esposito & Mondelli (2024), for all 8 (0, 1) and a > 1, with probability at least 1 0. we have
- Ha
Pex 1o 0.)
a
(E) < Ega [E)] +
In
(27)
2(a-1)N 1)N
We estimate the upper bound of Egg (E)) follows:
Eg (B2(E)] Eyx
(up(Le(z)- Lp(())
L=EZ
BBN
sup Beeg.-. -00 (Lx(2)) bbx()))
=Eez
S BgN ai
sup (LE(2) - LE(()) LEEZ
(Jensen inequality)
sup 1 ZEZ N
Egn 8.N
Σ ((((0:)).))) Σ ((((0())))))
Egg EgN gex
sup
Σ a, ((((((x))))) - - ((F(()),,
#EZ N
1 N
}
< BBgE
sup =EZ N
Σ a.l(f((z).)) v)
+ EoEy
sup
Σ o,((((()),))
#EZ N
1
2 Bax
sup xE2
((((((()),))
N
2EAM [RN(F))
(28)
Likewise, we again apply Theorem in Esposito & Mondelli (2024) and obtain that for all 5 E (0, 1), with probability at least 1 - 18, we have
a,
2d
H
Pon IP
EaN [N((F]] S RN(FF +
In
(29)
2(a
1)N
10
Therefore, combining Eq. (24), Eq. (25). Eq. (26), Eq. (27), Eq. (28) and Eq. (29) with union bound, we obtain that, with probability least 1 5, there holds
27

---

Understanding Model Ensemble in Transferable Adversarial Attack
18a =² Ha (Px | x.)
( (2 S 4Rx(F)+
In
(a- 1)N
6
The proof is complete.
C.5 Extension of Theorem 4.3
fN trained to generate adversarial examples. Let D be the distribution over the We consider N surrogate classifiers fi.... . surrogate models (for instance, the distribution of the low-risk models), and f, D, E [N] The low-risk claim is in line with Lemma 5 in (Yang al., 2021), which assumes that the risk of surrogate model and target model low (have risk most e). Therefore, the surrogate model and target model can be seen as drawing from the same distribution (such distribution of all the low-risk models). For data point (z, E Z and N classifiers for model ensemble attack, define the population risk (2) and the empirical risk LD( as
Lp(z) EE>D|(((()),)))
(D(2)= = N E {(f(().
KE(N),EED
Now here is extension of Theorem 4.3 based the above definition.
Theorem C.6 (Extension of Theorem 4.3), Let D be the joint distribution of f1,.... fN. and P Pox. be the joint measure induced by the product of the marginals. the loss function e is bounded by R4, and P Da < Poi for
any function fú, then for and y a-I. with probability at least 8, there holds
2 Hº (PONIIOON 18y N
ln
(30)
The proof almost the same as Appendix C.4, but the definition distribution is different. The first term answers the question that more surrogate models and smaller complexity will lead smaller R2( and contributes tighter bound of 7 (=, e). The second term motivates us that if we reduce the interdependency among the ensemble components, then the upper bound of TF <) will be tighter. Recall that H. DP | D) quantifies the divergence between the joint distribution Pa' and product of marginals P&x D. The joint distribution captures dependencies while the product := of marginals does not. So the divergence between them measures the degree of dependency among the N classifiers f,. SN. As a result, improving the diversity of f1,... f'v and reduce the interdependence among them is beneficial to adversarial transferability.
C.6 Further Explanation of the Hellinger Integral Term
We provide two examples of the Hellinger integral term in Theorem 4.3.
Example (Independent case). Suppose that the N surrogate models are independent. In this case, the hellinger integral achieves its minimum 1. Therefore, let a = 2 and Theorem 4.3 becomes
3632 N
4
TE(2,)) (Z) V
This theoretical result similar the generalization error bound in the literature on statistical learning theory (Bartlett & Mendelson, 2002) with different constant coefficients. The difference arises because (Bartlett & Mendelson, 2002) applies the concentration inequality once, but our proof applies it several times.
Example (Dependent case). For a more general case, the N surrogate models are interdependent to each other. While is hard to model the behavior of each model and the whole parameter space, we simplify the problem to make clear
28

---

Understanding Model Ensemble in Transferable Adversarial Attack
understand the hellinger integral Ha || P &. e). In partieular, let P= PaN and and = . model parameters for given precision that P and Q are discrete distributions. Firstly, Equation (8) from (Esposito & Mondelli, 2024) tells that H. (PIIQ) ((a.1))..(P.0) where (P,, Q) is the Rényi divergence. Secondly, let Q(a) be defined Equation (8) from (Sason Verdú, 2015), i.e., the minimum of the ratio of the probability density function of distributions Q and P. Now we approximate $1. Consider there are ł parameter configurations for each model. For simplicity, we assume that part of the models (f(N) models) play a key role in adversarial transferability, and the other N ((V) models are random sampled from these f(N) models.
88 e. We consider the
3 = m P(a)
For the product of marginal distribution Q, the parameters from each model are random. Consider the case of uniform distribution, where every parameter in the N models share the same probability, i.e., Q(a) gx
For the joint distribution P. we also consider the case of uniform distribution, where S(N) models are fixed and N f(N) models are randomly sampled, i.e., P(a) -V-NNN.
Therefore, B1
Q(a) (-((N), which is less than Substitute the above into Theorem from (Sason & Verdú, 2015), P(a)
have
( s 1+ (8T--1 -
H. (PIIQ) S 11+
S 33T t1
1-31
31
Let a = and substitute the above into Theorem 4.3 our paper, we have
TF(2,6) S 4R) + V 1882 S(N)
4v2 1
+ 36,²2 In
N
8 N
Here several cases:
1. S(N) ON*),, where E (0, 1),
2. f(N) OIn
3. f(N) = sN, where s E (0,
For Cases and 2, the above term asymptotically converges to zero as N becomes large. Notably, the true Hellinger term may be smaller than our derived upper bound above. Quantifying the core subset of models f(V) that dominate the performance of the ensemble attack presents theoretically profound and practically significant research direction. This problem is particularly well-suited for future exploration, could fundamentally advance our understanding of transferable adversarial model ensemble attacks.
C.7 Information-theoree Analysis
This section follows the multi-classification setting in Appendix C.3. Note that while we a different theoretical framework comparing to Theorem 4.3, the conclusion is consistent with it.
Firstly, we define the KL divergence, mutual information and TV distance.
Definition C.7 Becllback-Leibbee Divergence). Given two probability distributions P and Q, the Kullback-Leiller (KL) divergence between P and Q is
(kI()) [
P(x) dx. / P(x) log Q(x)
We know that DKL. (PII) E [0, +oo], and DRL. (PI() = 0 if and only if P= Q.
Definition C.8 (Mutual Information). For continuous random variables and Y with joint probability density function p(z, y) and marginal probability density functions p(x) and p(y), the mutual information is defined as:
1(x;8) // p(z, y) log P(I,3) dady. p(x))(y)
29

---

Understanding Model Ensemble in Transferable Adversarial Attack
We know that [(x; Y) E [0, +x],, and Y(X 0 if and only if X and Y are independent to each other. Definition C.9 (Total Variation Distance), Given two probability distributions and Q, the Total Variation (TV) distance between P and Q is
Drv(PIIQ) = f ,EX P((x) - Qa()) dr.
We know that (PI E [0, Also, DTv 0 if and only if P and Q coincides, and Drv((PlQQ 1 if and only if P and Q are disjoint.
Here we provide further analysis from the perspective of information (Shwartz-Ziv & Tishby, 2017; Xu & Raginsky, 2017).
2017))
Theorem C.10. Given N surrogate models ON ~ PaN as the ensemble components. Let 0 = (O...... 0N) ~ PaN be the target models, which is an independent copy of 0%. Assume the loss function is bounded by 3 E R+ and PaN is absolutely continuous with respect to PoN. AN(0,2) = Lp(z)= - Lz(z). Then there holds
No er For a > and adversarial example = (x,y) ~ Pz, let
E Pr.0"~PZ,0' .P, An( 2B Drv (Pew 11 1//B,)))
0r32
(I( (I(O" :z) log Ha (Pex IP )))
1
2(a 1)N
where (-v(- I(-II-) and H.(-ll) denotes TV distance, mutual information and Hellinger integrals, respectively.
In Theorem C.10: Av (0 z) quantifies how effectively the surrogate models represent all possible target models. Taking the expectation of A (0, 2) over z and 0 accounts for the inherent randomness in both adversarial examples and surrogate models. The mutual information I (ex; :z) quantifies how much information about the surrogate models is retained in the adversarial example. Intuitively, higher mutual information indicates that the adversarial example is overly tailored to the surrogate models, capturing specific features of these models. This overfitting reduces its ability to generalize and transfer effectively to other target models. By controlling the complexity of the surrogate models, the specific information captured by the adversarial example can be limited, encouraging it to rely on broader, more transferable patterns rather than model- specific details. This reduction in overfitting enhances the adversarial example's transferability to diverse target models. The TV distance Drv (PeN | e) and the Hellinger integral Ha (Pex iPo. the surrogate models.
capture the interdependence among
Theorem C.10 reveals that the following strategies contribute to a tighter bound: 1) Increasing the number of surrogate models, i.e., increasing N:2) 2) Reducing the model complexity of surrogate models, i.e., reducing / (oN; :z): 3) Making the
surrogate models more diverse, i.e., reducing DTv (Paw | | e) and Ha (Pew ll e). tighter bound ensures
that an adversarial example maximizing the loss function on the surrogate models will also lead to high loss on the target models, thereby enhancing transfcrability.
Proof. According to Donsker and Varadhan's variational formula, for any À E R, there holds:
(Pz PP @ PaN)= 2 AB 20 ~PN (N - log E EaN.Pr .Pwx [cndw(Ox2]
(31)
Fix E Z,
EaN~PaN (2
) dPa
dPax
dP
8 v
s <(1 للمي x 0 2)
a-1
dPON
a :)
dP
dP 0 N
JP
30

---

Understanding Model Ensemble in Transferable Adversarial Attack
- (( ((eestAAwees dP &
Hä (Pox P
(32)
The third line uses Hölder's inequality, while the last line follows Definition C.5. Now we deal with the first term. Denote
A
E T
1
(((:)))))))
10
((((,::),,))
Pe
N
N
1 N
1
R
(((Ox))),))
((((0;))))))
0
P
N
Notice that
A //
N
N (((00;:),,)
(PoN(O,,... (0,,... ,0,) oo "0 ,0x)
d0, doN
S 8 /f / PPN (0),... 0n) -P P0.,e 00,,,... 0N) d0, ... doN
3 / |Po (8") -Poy (0") dā
/
s 23 Drv (PoN 1P P
(33)
Also,
/ ( dP 8,O o =Fa
Np
ad
BBinPe 1 N [((10(xx),)) - [[[(0x))))))
E.g
Pe
exp
i=1
N
a²
8(a - 1)2N2 12g22)
<
exp
a²
<exp
8(a - 1)2N 3232
(34)
The third line is due to Hoeffding's Lemma (using for each 9.). Therefore, recall the fact that (0, 2) A + A2, have
8 0 = / /0/e//erre - -
JP
dP &
S exp
2203
DTV (Pen (Pen (PoN
e
M5z dP
8 NN
(Using (33))
a a
2\a a- DrV
a²
< exp
+
8(a 1)2 3232
(Using (34))
With the above results, we obtain the following:
log Ez~Pz ~PN [22A)(00) 22B Drv (Pox (PoNIPoN,))+ 8(a - 1)N X2 + log H (Pow (PoN IP e).
Substitute the above into Eq. (31), we have
8(a- - 1)N 82X2 + (23 Drv (Paw (PoNN 0)
Ez PNONNPZNNAN(0:. At
(P. @ Par) log H (Pav | e) 20.
31

---

Understanding Model Ensemble in Transferable Adversarial Attack
0.001
0.01
0.001
1.00
010.3
220
$05.00
1000
330
0350
P steps
staps
Mtx24
MLP
(b) CNN
Figure 6. Evaluation of ensemble attacks with increasing the number of steps using MLPs and CNNs the CIFAR- 100 datasct.
Let the discriminant of the quadratic function with respect be less than or equal 0, leading to:
23 DTV (Pex IP o)
Bz, ON ~P
NONND,,NN 2,0N (N)
a2
1
log Ha (Po* PPaNIP8N10 IPor,))) (35)
DKL (Pz,aw s0 Por)
V 2(a - 1)N
a
In other words,
0N N ~D 2.08 (n() 28 Drv (PeN Por 8)+
a,32
(DKL. (P2,ax 8 Pav) log H. (Pox IP IP
1
V 2(a - 1)N
Finally, substitute / (0":2) = -DL (Pz,aN|PP) 6 Pax) into above and we can get the desired result.
D Further Experiments
D.1 Evaluation CIFAR-100
Following the same setting in our experiments, we further validate the vulnerability-din diveriitty decomposition the CIFAR 100 (Krizhevsky et al., 2009) dataset. The results are shown in Figure 6. As the model becomes stronger (i.e., smaller A), the three metrics (ASR, loss and variance) increases, validating the soundness of vulnerability-divrriii decomposition.
D.2 Further Investigation into Model Complexity
We conduct deeper investigation into the role of model complexity by applying max norm constraint to the model parameters. Specifically, we constrain the L2 norm of each weight vector to a predefined threshold, effectively limiting the model's capacity. Empirically, larger max norm values allow for more expressive feature representations but may increase the risk of overfitting. In contrast, smaller max norms encourage simpler models and reduce overfitting but may also lead to underfitting due to restricted representational power. The validation of this trade-off illustrated Table 1, which shows the classification accuracy across range of max norm values for both MLP and CNN architectures with varying depths. Lower accuracy values indicate stronger adversarial attack performance.
The results reveal a consistent trend: the max norm constraint is relaxed from a highly restrictive value (e.g., 0.1) to moderate level (e.g.. 5.0), the effectiveness of adversarial attacks improves and then declines. This observation indicates that overly tight constraints can impair model expressiveness, while moderately relaxed constraints can achieve a better trade-off' between simplicity and capacity. These findings empirically support our thcoretical claim that weight regularization, e.g., via weight decay or norm bounds, directly influences model complexity and, consequently, adversarial transferability.
D.3 Experiments on ImageNet
To further investigate how appropriately controlling the complexity of surrogate models contributes to effective adversarial attack algorithms- line with our theoretical insights we conduct additional experiments on ImageNet (Russakovsky
32

---

Understanding Model Ensemble in Transferable Adversarial Attack
Table Effect varying max constraints adversarial attack performance, measured by classification accuracy (%, lower better). FC and CNN denote fully connected and convolutional networks with increasing layers.
Max Norm
FC1
FC2
FC3
CNN1
CNN2
CNN3
Avg
0.1
84.66
87.80
85.39
97.57
98.31
98.59
92.05
0.5
59.37
68.31
74.05
96.50
97.66
98.34
82.37
1.0
64.31
55.27
57.12
95.37
97.08
97.93
77.85
2.0
68.00
57.40
57.86
95.41
97.04
97.87
78.93
4.0
68.19
57.94
58.12
95.53
97.00
97.85
79.11
5.0
69.68
59.40
59.26
97.48
98.02
98.87
80.45
et al., 2015). For model ensemble attacks, we fine-tune several surrogate models, including VGG16 (Simonyan & Zisserman, 2014), Inception-V3 (Szegedy et al., 2016), and Visformer (Chen et al., 2021), using sparse Softmax cross-entropy loss (Martins & Astudillo, 2016). This modification encourages sparsity in the model's output distribution. As shown in Table this approach leads a reduction model complexity, as indicated by the decreased L2 norm of the weights.
Table 2. Comparison of model complexity between original and sparse Sofimax loss variants different backbones. Lower values indicate reduced L2 norm weights.
VGG16
Visformer
Inception
Original Sparse Softmax Loss
37.37
25.94
49.24
33.12
20.60
48.53
We then leverage these sparsified models for ensemble attacks by applying MI-FGSM (Dong et al., 2018), SVRE (Xiong et al., 2022), and SIA (Wang et al., 2023a) to both the original and sparsified versions, resulting in MI-FGSM-S, SVRE-S, and SIA-S, respectively. The transferability of these attacks is evaluated on a range of target models, and the results are presented Table 3. As observed, these sparsified variants consistently outperform their standard counterparts most cases, validating the advantage of model complexity control for enhancing adversarial transferability. This improvement holds across both CNN-based and transformer-bee architectures. Beyond this example, our findings may inspire the design of stronger adversarial attack strategies that systematically exploit sparsity and simplicity in surrogate modeling.
Table Transferability results of different attack methods across various target models. Bold entries indicate improved top-performing variants.
ResNet50
VGG16
MobileNetV2
Inception
VIT-B16
PÎT-B
Visformer
Swin-T
MI-FGSM
66.0
99,9
76.8
97.5
37.3
53.8
88.9
66.7
MI-FGSM-S
68.9
99.7
79.2
99.1
39.0
54.5
90.6
68.1
SVRE
65.2
99.9
79.0
98.6
32.4
49.2
90.3
64.3
SVRE-S
66.9
99.9
81.2
98.9
34.2
51.3
93.0
65.9
SIA
97.2
100.0
98.4
99.7
75.9
91.9
90.0
96.1
SIA-S
98.1
100.0
98.2
99.6
79.2
93.2
99.5
97.5
E
Further Discussion
E.1
Analyze Empirical Model Ensemble Rademacher Complexity
In particular, we present detailed analysis for the simple and complex cases below, within the context of transferable model ensemble attack.
The simple input space. Firstly, consider the trivial case where the input space contains too simple examples so that all classifiers correctly classify (x, v) Z. Then there holds R (Z) E v) E. 0. In this case, Z simple enough for f1,... . Such corresponds R. (2) close However, it important to note that an
33

---

Understanding Model Ensemble in Transferable Adversarial Attack
overly simplistic space 2 may be impractical for model ensemble attack: the adversarial examples in such a space may not successfully attack the models from D, leading small value of Lr(=*)) In other words, the existence of transferable adversarial examples implicitly imposes constraints on the minimum complexity of 2.
The complex input space. Secondly, we consider the complex case. In particular, given arbitrarily NN models in H. and any assignment of a, a sufficiently complex 2 contains all kinds of examples that make R (2) large: (1) If 0; = +1, there are adversarial examples that can successfully attack fi and leads to large (fL(f(()), v); (2) If a; = -1, there exists some examples that can be correctly classified by f.. leading to of((f((x).. y) = O. However, such a large Rv is also not appropriate for transferable model ensemble attack. may include adversarial examples that perform well against fi,".. .NN but are merely overfitted to the current N surrogate models (Rice et al., 2020; Yu et al., 2022). In other words, these examples might not effectively attack other models in H. thereby limiting their adversarial transferability. The above analysis suggests that an excessively large or small Rv (Z) is not suitable for adversarial transferability. So we are curious to investigate the correlation between Rx (2) and adversarial transferabilityy which comes to the analysis about the general case in Section 3.4.
Explain robust overfitting, After a certain point in adversarial training, continued training significantly reduces the robust training loss of the classifier while increasing the robust test loss, a phenomenon known as robust overfitting (Rice et al., 2020; Yu et al., 2022) (also linked to robust generalization (Schmidt et al., 2018; Yin et al., 2019)). From the perspective in Section 3.4, the cause of this overfitting is the limited complexity of the input space relative to the classifier used to generate adversarial examples during training. The adversarial examples become too simple for the model, leading to overfitting. To mitigate this, we could consider generating more "hard" and "generalizable" adversarial examples to improve the model's generalization in adversarial training. For a less transferable adversarial example (z, y), it is associated with a small Lp(z), which in turn makes T E22 c) large.
E.2 Other Opinions on "Diversity"
E.2.1
OTHER DEFINITIONS
There are other definitions of "Diversity" in transferable model ensemble adversarial attack. For example, in Yang et al. (2021), gradient diversity is defined using the cosine similarity of gradients between different models, and instance-level transferability is introduced, along with a bound for transferability. They use Taylor expansion to establish a theoretical connection between the success probability of attacking single sample and the gradients of the models. In Kariyappa & Qureshi (2019), inspired by the concept of adversarial subspace (Tramèr et al., 2017), diversity is defined based on the cosine similarity of gradients across different models. The authors aim to encourage models to become more diverse, thereby achieving "no overlap in the adversarial subspaces," and provide intuitive insights to readers. Both papers define gradient diversity and explain its impact.
In contrast, our definition of diversity stems from the unified theoretical framework proposed in this paper. Specifically: (I) We draw inspiration from statistical learning theory (Shalev-Shwartz et al., 2010; Bartlett & Mendelson, 2002) on generalization, defining transferability error accordingly. (2) Additionally, we motivated by ensemble learning (Abe et al., 2023; Wood et al., 2024), where we define diversity as the variation in outputs among different ensemble models. (3) Intuitively, when different models exhibit significant differences in their outputs for the same sample, their gradient differences during training are likely substantial as well. This suggests a potential connection between our output-based definition of diversity and the gradient-basse definitions in previous work, which is worth exploring in future research.
E.2.2 CONFLICTING OPINIONS
We observe a significant and intriguing disagreement within the academic community concerning the role of "diversity" in transferable model ensemble attacks: Some studies advocate for enhancing model diversity to produce more transferable adversarial examples. For instance, Li et al. (2020) applies feature-level perturbations to an existing model to potentially create a huge set of diverse "Ghost "etworks". Li et al. (2023) emphasizes the importance of diversity in surrogate models and promotes attacking a Bayesian model to achieve desirable transferability. Tang et al. (2024) supports the notion of improved diversity, suggesting the generation of adversarial examples independently from individual models. In contrast, other researchers adopt a diversity-rednction strategy to enhance adversarial transferability. For example, Xiong et al. (2022) focuses on minimizing gradient variance among ensemble models to improve transferability. Meanwhile, Chen al. (2023) introduces disparity- reduuedd filter designed to decrease gradient variances among surrogate models in ensemble attacks. Although all these studies reference "diversity," their perspectives appear to diverge. In this paper, we advocate for increasing
34

---

Understanding Model Ensemble in Transferable Adversarial Attack
the diversity of surrogate models. However, we also recognize that diversity-reduction approaches have their merits. For instance, consider the vulnerabiliyy-diversttt decomposition of transferability error presented in Theorem 4.1. It suggests the presence of citterrability-ivvesiiii trade-off' in transferable model ensemble attacks. In other words, we may need to prioritize either vulnerability or diversity to effectively reduce transferability error. diversity-reductin approaches aim to stabilize the training process, thereby increasing the "bias." " In contrast, diversity-promotie methods directly enhance "diversity." This analysis, framed within our unified theoretical framework, provides insight into the differing opinions regarding adversarial transferability in the academic community.
E.3 Compare with A Previous Bound
Lemma 5 in Yang et al. (2021) offer complementary perspectives in the analysis of transferable adversarial attack. We first restate Lemma in Yang et al. (2021) and our Theorem 4.1. Our theoretical results and theirs offer complementary perspectives in the analysis of transferable adversarial attack.
Lemma 5 (Yang et al. (2021)). Let f.g X y be classifiers, 6,p,c (0,1) be constants, and A(:) be attack strategy. Suppose that f 9 have risk at most e. Then
Pr(F(A(E)) G(A())) S 2$ p,
for a given random instance x and A(-) is p-conservative (TV distance between the adversarial example distribution and clean data distribution less than p. which defined as Definition and 8 in Yang et al. (2021)).
Lemma 5 states an intriguing conclusion: two models exhibit low risk on the original data distribution and the distributional discrepancy between adversarial examples and the original data is small, the predictions of the two models on the same input will be close. In other words, for two well well-performing models, if an attack strategy successfully targets one model, it is highly likely to succeed on the other. Lemma 5 thus describes the success rate of transferring an attack from one model to another. In contrast, Theorem 4. demonstrates that if the ensemble models exhibit significant output differences on the same input, the resulting diverse ensemble more effective at generating adversarial examples with reduced transferability.
To better clarify, let A denote the ensemble models generating adversarial examples and B the model being attacked. Comparing Lemma 5 and our work leads to the following reasoning: Suppose A and B both fit the original data distribution well (i.e., the risk of A and B is bounded by as in Lemma 5). As shown in our work, increasing ensemble diversity while keeping vulnerability constant reduces the transferability error of adversarial examples generated by the ensemble. Many models in parameter space, such as A and B, are vulnerable to these adversarial examples. However, fitting both the original data distribution and the adversarial example distribution simultaneously becomes challenging, leading to large distributional discrepancy. This discrepancy enlarges p in Lemma 5, thereby loosening its "conservative condition" and weakening its theoretical guarantee of successful transferability. Consequently, adversarial transferability decreases, which could be interpreted potential contradiction.
No actual contradiction exists between Lemma 5 and our work. Instead, they provide complementary analyses. Lemma 5 provides an upper bound rather than an equality or lower bound. While an increase in p loosens this upper bound, it does not necessarily imply that the left-hand side (i.e., transfcrability success) will increase. The significance of an upper bound lies in the fact that tighter right-hand side suggests the potential for a smaller left-hand side. However, a looser upper bound does not necessarily imply that the left-hand side will increase. Therefore, while increasing ensemble diversity may loosen the upper bound in Lemma 5, it does not contradict the fundamental interpretation of it. While Lemma 5 analyzes the trade-off between (model fit to the original data) and o (distributional discrepancy), our work focuses on the trade-off between vulnerability and ensemble diversity. Together, they provide a comprehensive understanding of the factors influencing adversarial transferability.
We now further elucidate the relationship between our results and Lemma 5. To minimize transferability error (as in our work), the adversarial transferability described by Lemma 5 may have stronger theoretical guarantees, requiring its upper bound be tighter. To tighten the bound in Lemma 5, either e or p must decrease. However, the two exhibit a trade-off:
If e decreases, A and fit the original data distribution better. However, beyond certain point, the adversarial examples generated by A diverge significantly from the original data distribution, increasing p.
If p decreases, the adversarial example distribution becomes closer to the original data distribution. However, beyond a certain point, A exhibits similar losses on both distributions, resulting a higher c.
35

---

Understanding Model Ensemble in Transferable Adversarial Attack
Therefore, Lemma indicates the potential trade-off between and p in adversarial transferability, while our Theorem I emphasizes the trade-off between vulnerability and diversity. By integrating the perspectives from both Lemma 5 and our findings, these results illuminate different facets of adversarial transferabilityy, offering complementary theoretical insights.
E.4 Compare with Generalization Error Bound
We note that key distinction between transferability error and generalization error lies in the independence assumption. Conventional generalization error analysis relies
assumption: each data point from the dataset is independently sampled (Zou & Liu, 2023; Hu et al., 2023). By contrast, the surrogate models fi. .N for ensemble attack are usually trained on the datasets with similar tasks, e.g., image classification. In this case, such models tend to correctly classify easy examples while misclassify difficult examples (Bengio et al., 2009). Consequently, such correlation indicates dependency (Lancaster, 1963), suggesting that we cannot assume these surrogate models behave independently for a solid theoretical analysis. Additionally, there are alternative methods for analyzing concentration inequality in generalization error analysis that do not rely on the independence assumption (Kontorovich & Ramanan, 2008; Mohri & Rostamizadeh, 2008; Lei et al., 2019; Zhang et al., 2019). However, such data-dependent analysis is either too loose (Lampert et al., 2018) (because it includes an additional additive factor that grows with the number of samples (Esposito & Mondelli, 2024)) or requires specific independence structure of data (Zhang & Amini, 2024) that may not align well with model ensemble attacks. Therefore, we uses the latest techniques of information theory (Esposito & Mondelli, 2024) about concentration inequality regarding dependency.
E.5 -uideeabilityddiverittt Trade-off Curve
The relationship between vulnerability and diversity, as discussed in Section 5, merits deeper exploration. Drawing on the parallels between the uunerability-diveritty trade-off and the bias-variance trade-off (Geman et al., 1992), we find that insights from the latter may prove valuable for understanding the former, and warrant further investigation. The classical bias-variance trade-off suggests that model complexity increases, bias decreases while variance rises, resulting in U-shaped test error curve. However, recent studies have revealed additional phenomena and provided deeper analysis (Neal et al., 2018 88 Neal, 2019; Derumigny& & Schmidt-Hieber, 2023), such the double descent (Belkin et al., 2019; Nakkiran et al., 2021). Our experiments indicate that diversity does not follow the same pattern as variance in classical bias- varianc trade-off. Nonetheless, there are indications within the birs-variance trade-off literature that suggest similar behavior might occur. For instance, Yang et al. (2020) proposes that variance may exhibit a bell-shaped curve, initially increasing and then decreasing as network width grows. Additionally, Lin & Dobriban (2021) offers a meticulous understanding of variance through detailed decomposition, highlighting the influence of factors such as initialization, label noise, and training data. Recent studies have even revealed that bias and variance can exhibit concurrent relationship in deep learning models (Chen et al., 2024c). Overall, the trend of variance in model ensemble attack remains valuable area for future research. We may borrow insights from machine learning literature (see the above papers and the references therein) to get better understanding of this in future work.
E.6 Insight for Model Ensemble Defense
While our paper primarily focuses on analyzing model ensemble attacks, our theoretical findings can also provide valuable insights for model ensemble defenses: (1) From a theoretical perspective, the vulnerability-diveritt decomposition introduced for model ensemble attacks can likewise be extended to model ensemble defenses. Mathematically, this results in a decomposition similar to conclusions in ensemble learning (see Proposition 3 in Wood et al. (2024) and Theorem in Ortega et al. (2022)), which shows that within the adversarial perturbation region, Expected loss S Empirical ensemble loss Diversity. Thus, to improve model robustness (reduce the expected loss within the perturbation region), the core strategy involves minimizing the ensemble defender's loss or increasing diversity. However, there is also an inherent trade-off between these two objectives: when the ensemble loss is sufficiently small, the model may overfit to the adversarial region, potentially reducing diversity; conversely, when diversity is maximized, the model may underfit the adversarial region, potentially increasing the ensemble loss. Therefore, from this perspective, our work provides meaningful insights for adversarial defense that warrant further analysis. (2) From an algorithmic perspective, we can consider recently proposed diversity metries, such as Vendi score (Friedman & Dieng, 2022) and EigenScore (Chen et al., 2024a). Following the methodology outlined in Deng & Mu (2023), diversity can be incorporated into the defense optimization objective strike a balance between diversity and ensemble loss. By finding an appropriate trade-off between these two factors, the effectiveness of ensemble defense may be enhanced.
36


Reviews:
[{"review_id": "qSW4FQ2KCT", "reviewer": "Reviewer_uk2i", "summary": "This paper proposes novel definitions for theoretically analyzing the adversarial transferability of adversarial attacks with a model ensemble; then, it provides three practical guidelines to improve the transferability of the model ensemble attacks. Specifically, the paper first defines the transferability error, the gap between the adversarial risks between the most transferable example $z^*$ and the adversarial example $z$ that the model ensemble outputs. The paper analyzes this transferability error in two different ways.\n\nIn the first analysis, the paper decomposes the population risk on $z$ into the vulnerability term (which measures the attack power of the ensemble attack) and the diversity term (which measures the diversity of the ensemble attack). This suggests two guidelines: improving the ensemble attack’s power and diversity are both beneficial to the transferability of the model ensemble attack.\n\nIn the second analysis, the paper provides an upper bound of transferability. This upper bound contains two terms. The first term represents the complexity of the surrogate models, and the second term decreases as the number of surrogate models increases. This upper bound gives us another guideline: having more surrogate models with less model complexity is beneficial to the transferability of the model ensemble attack.\n\nWith a set of experiments, the paper experimentally supports the theoretical findings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful review of our work!\n\n>**Q1**: ...Comparing another attack method (either extremely strong or weak) would give us more insights about the vulnerability-variance tradeoffs.\n\n**A1**: We sincerely appreciate the reviewer's constructive suggestion. We have conducted additional experiments using the VMI-FGSM attack [1] on MNIST.\n\n$\\lambda=10^{-4}$\n\n| Steps | 1     | 3     | 6     | 9     | 12    | 16    | 19    | 22    | 25    |\n|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| ASR   | 2.1   | 6.9   | 8.4   | 17.1  | 24.6  | 29.5  | 37.3  | 38.4  | 40.2  |\n| loss  | 0.012 | 0.044 | 0.225 | 0.351 | 0.378 | 0.390 | 0.392 | 0.397 | 0.401 |\n| Variance | 0.007 | 0.025 | 0.033 | 0.019 | 0.008 | 0.006 | 0.004 | 0.003 | 0.003 |\n\n$\\lambda=10^{-3}$\n\n| Steps    | 1     | 3     | 6     | 9     | 12    | 16    | 19    | 22    | 25    |\n|-----------|-------|-------|-------|-------|-------|-------|-------|-------|-------|\n| ASR  | 2.2   | 7.5   | 8.3   | 16.9  | 25.4  | 30.1  | 38.2  | 38.9  | 40.6  |\n| loss | 0.011 | 0.039 | 0.214 | 0.337 | 0.365 | 0.381 | 0.385 | 0.392 | 0.399 |\n| Variance | 0.004 | 0.017 | 0.029 | 0.015 | 0.006 | 0.005 | 0.003 | 0.003 | 0.003 |\n\nThe observed vulnerability-variance tradeoffs demonstrate consistent alignment with our paper. We will include the complete experimental details and analysis on other datasets in our final manuscript.\n\n[1] Enhancing the transferability of adversarial attacks through variance tuning. CVPR 2021.\n\n>**Q2**: Please clarify the effect of increasing $\\lambda$ in the paper and explain how it relates to the model complexity...\n\n**A2**: Thank you for your insightful question. Firstly, as suggested in Lemma 4.2 and Appendix A, the empirical model ensemble Rademacher complexity can be upper bounded by model complexity and the number of ensemble components (for instance, the analysis in Section 4.2 states that reducing the weight norm of the model and increasing the number of the models will reduce the empirical model ensemble Rademacher complexity). These two kinds of effect have been reported in our experiments: \n- In Section 5.1, we adjust the weight decay factor $\\lambda$ to change the model complexity and investigate the trends in how complexity interacts with other factors.\n- In Section 5.2, we improve the number of ensemble components and observe an increasing trend of attack success rate.\n\nFollowing the reviewer’s suggestion, we conduct a deeper investigation into the impact of model complexity by applying a max norm constraint to the model parameters. This technique limits the L2 norm of each weight vector to a predefined threshold to control the model complexity.\n- Larger max norms enable richer feature representations at the cost of potential overfitting.\n- Smaller max norms promote simpler models and may induce underfitting by limiting model capacity.\n\nAs illustrated in the table below, this trade-off manifests across architectures (MLPs and CNNs with 1–3 layers) and varying max norm values. We measure attack performance via accuracy, where lower accuracy corresponds to stronger attack effectiveness.\n\n| Max norm | FC1    | FC2    | FC3    | CNN1  | CNN2  | CNN3  | Avg   |\n|----------|--------|--------|--------|-------|-------|-------|-------|\n| 0.1      | 84.66  | 87.80  | 85.39  | 97.57 | 98.31 | 98.59 | 92.05 |\n| 0.5      | 59.37  | 68.31  | 74.05  | 96.50 | 97.66 | 98.34 | 82.37 |\n| 1.0      | 64.31  | 55.27  | 57.12  | 95.37 | 97.08 | 97.93 | 77.85 |\n| 2.0      | 68.00  | 57.40  | 57.86  | 95.41 | 97.04 | 97.87 | 78.93 |\n| 4.0      | 68.19  | 57.94  | 58.12  | 95.53 | 97.00 | 97.85 | 79.11 |\n| 5.0      | 69.68  | 59.40  | 59.26  | 97.48 | 98.02 | 98.87 | 80.45 |\n\nThe results demonstrate a clear trend: as the max norm constraint is relaxed from very small values (e.g., 0.1) to moderate levels (e.g., 5.0), the attack effectiveness first increases and then decreases. This pattern indicates that excessively restrictive constraints can impair model expressiveness, whereas an optimally tuned max norm effectively balances model complexity and representational capacity. More importantly, these findings support our paper's claim regarding the influence of the weight decay factor $\\lambda$ on model complexity.\n\n>**Q3**: A comparison between CIFAR-10 and CIFAR-100 could also be interesting...\n\n**A3**: We sincerely appreciate your insightful question. As shown in our paper, the attack success rate and loss exhibit an initial increase followed by stabilization with growing steps, while the variance first rises and then declines. We fully agree with the reviewer's insightful comment regarding dataset-dependent characteristics; indeed, the inflection points vary across datasets due to their distinct characteristics. Due to the space limitations of this rebuttal, we will extend the step range and include a comparative analysis of CIFAR-10 and CIFAR-100 in Appendix E in our final version.", "claims_and_evidence": "The paper supports all of its claims well with proof and experimental results.", "methods_and_evaluation_criteria": "I checked the evaluation methods and criteria. Although the evaluation can be improved further, the experiments are well-designed to support the theoretical findings.", "theoretical_claims": "I did not have enough time to check the proofs in the appendices.", "experimental_designs_or_analyses": "The experimental designs are overall well-designed, and the analyses make sense. In particular, the analysis of the different behaviors on the CIFAR dataset seems interesting.", "relation_to_broader_scientific_literature": "This paper makes many contributions to adversarial machine learning. This paper provides theoretical understandings of model ensemble attacks and practical guidelines for ML practitioners.", "essential_references_not_discussed": "The paper cited the needed references well.", "other_strengths_and_weaknesses": "### Strengths\n1. To the best of my knowledge, all the proposed concepts (transferability error, prediction variance, and model ensemble Rademacher complexity) are novel.\n2. The decomposition of the transferability error into two interpretable terms seems impressive.\n3. The paper presents all of the theoretical analysis, practical discussions, and experimental validation of the proposed idea.\n4. The experiments support the findings well.\n\n### Weaknesses\n1. Only one attack method is used for the evaluation. Comparing another attack method (either extremely strong or weak) would give us more insights about the vulnerability-variance tradeoffs.\n2. I’m not very convinced whether or not the experiments are enough to explore the model complexity. Only three values of $\\lambda$ are used, and no other quantity controls the model complexity in the experiments.\n3. The effect of increasing the $\\lambda$ parameter is not clearly explained, and it is unclear whether it definitely lowers the model complexity or not. In my opinion, the empirical model ensemble Rademacher complexity is computable for low values of $N$, so the model complexity can be quantified during the experiments.", "other_comments_or_suggestions": "1. Please clarify the effect of increasing $\\lambda$ in the paper and explain how it relates to the model complexity.\n2. If there are other factors that can quantitatively control model complexity, consider adding more experiments involving them.\n3. Please consider experiments with other attack methods.\n4. A comparison between CIFAR-10 and CIFAR-100 could also be interesting. They contain the same set of images, only more labels in CIFAR-100, but this changed the variance behaviors. Does variance on CIFAR-100 experiments decrease for lower $\\lambda$?", "questions_for_authors": "* Could you explain what factors would increase/decrease the model complexity in practice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lwoI4Cc1Ps", "reviewer": "Reviewer_kypp", "summary": "The paper provides a theoretical study on transferability of model ensemble adversarial attacks. The authors formulate the problem by considering the expected value of the attacked loss over the distribution of model ensemble (equation 1) and the averaged attacked loss over the set of considered models (equation 2). The goal is to use concentration and uniform convergence analysis to bound the transferability error in Definition 3.1 through bounding the gap between (1) and (2) over all input samples (Lemma 3.2). The authors define the model ensemble Rademacher complexity in (8) and bound it in Lemma 4.2. In Theorem 4.3, they connect the bound on the defined Rademacher complexity to bound the transferability error. Section 5 includes experimental results supporting the theorems.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful review of our work!\n\n>**Q1**: ...However, in the standard model ensemble attack scenario, the models may have been trained with fully or partially identical training data, and therefore the models could be quite correlated. Therefore, it seems to me that the term $H\\_\\alpha^{\\frac{1}{\\alpha}}\\left(\\mathcal{P}\\_{\\Theta^N} \\\\| \\mathcal{P}\\_{\\bigotimes\\_{i=1}^N \\Theta} \\right)$ could be quite large and make the bound vaccuous in practice.\n\n**A1**: To make it clear and easy to understand, we provide an intuitive approximation below. We choose $\\alpha=10$, $\\delta=0.01$, $\\beta=1$ in our Theorem 4.3. Let $P=\\mathcal{P}\\_{\\Theta^N}$ and $Q=\\mathcal{P}\\_{\\bigotimes\\_{i=1}^N \\Theta}$. We consider the model parameters for a given precision so that $P$ and $Q$ are discrete distributions.\n- Equation (8) from [1] tells us that $H\\_\\alpha(P \\\\| Q)=e^{(\\alpha-1)D\\_\\alpha(P,Q)}$, where $D\\_\\alpha(P,Q)$ is the Rényi divergence. \n- Let $\\delta \\in [0,1]$ be the TV distance between $Q$ and $P$, and $\\beta_1=\\min\\_{a \\in \\mathcal{A}} \\frac{Q(a)}{P(a)}$ be defined in Equation (8) from [2], i.e., the minimum of the ratio of the probability density function of distributions $Q$ and $P$. \n- Now we approximate $\\beta\\_1$. Consider there are $t$ parameter configurations for each model. For simplicity, we assume that part of the models ($f(N)$ models) play a key role in adversarial transferability, and the other $N-f(N)$ models are random sampled from these $f(N)$ models.\n    - For the product of marginal distribution $Q$, the parameters from each model are random. Consider the case of uniform distribution, where every parameter in the $N$ models share the same probability, i.e., $Q(a)=\\frac{1}{t^N}$.\n    - For the joint distribution $P$, we also consider the case of uniform distribution, where $f(N)$ models are fixed and $N-f(N)$ models are randomly sampled, i.e., $P(a)=\\frac{1}{t^{N-f(N)}}$.\n    - Therefore, $\\beta\\_1 \\approx\\frac{Q(a)}{P(a)}=t^{-f(N)}$, which is less than 1.\n- Substitute the above into Theorem 3 from [2], we have\n$$H\\_\\alpha(P \\\\| Q) \\le 1+\\frac{\\delta\\left(\\beta\\_1^{-1}-1\\right)}{1-\\beta\\_1} \\le \\beta\\_1^{-1} \\approx t^{f(N)}$$\n- Substitute the above into Theorem 4.3 in our paper, we have\n$$\\sqrt{\\frac{18 \\gamma \\beta^2}{N} \\ln \\frac{2^{2+\\frac{1}{\\gamma}} H\\_\\alpha^{\\frac{1}{\\alpha}}\\left(\\mathcal{P}\\_{\\Theta^N} \\\\| \\mathcal{P}\\_{\\Theta\\_{i=1}^N \\Theta}\\right)}{\\delta}} \\le \\sqrt{\\frac{20}{N} \\ln \\left(800 \\cdot t^{\\frac{f(N)}{10}}\\right)} \\approx \\sqrt{\\frac{140}{N} + \\frac{2\nf(N)}{N} \\ln t}$$\n\nHere are several cases:\n1. $f(N)=\\mathcal{O}(N^{s})$, where $s \\in (0,1)$\n2. $f(N)=\\mathcal{O}(\\ln N)$\n3. $f(N)=s N$, where $s \\in (0,1)$\n\nFor Cases 1 and 2, the above term asymptotically converges to zero as N becomes large. Notably, the true Hellinger term may be smaller than our derived upper bound above. Quantifying the core subset of models $f(N)$ that dominate ensemble attack performance presents a theoretically profound and practically significant research direction. This problem is particularly well-suited for future exploration, as it could fundamentally advance our understanding of transferable adversarial model ensemble attacks.\n\n[1] https://arxiv.org/pdf/2303.07245. TIT 2024.\n\n[2] https://arxiv.org/pdf/1503.03417. arXiv preprint.\n\n>**Q2**: ...Based on the theoretical framework, using disjoint training sets should reduce the correlation term, thereby improving transferability. I encourage the authors to conduct and report such an experiment.\n\n**A2**: Thank you for your valuable insight! \n\nAs suggested by the reviewer, we evaluate three settings:\n- Full: All models are trained on the full training set.\n- Split: Models are trained on disjoint partitions of the training data.\n- Split-FT: Models are first trained on disjoint data partitions and then fine-tuned on the full dataset.\n\nWe conduct experiments on MLP across two datasets. We measure attack performance via accuracy, where lower accuracy corresponds to stronger attack effectiveness.\n\n|            | MNIST |        |        | Fashion MNIST |        |        |\n|:----------:|:-----:|--------|--------|---------------|--------|--------|\n| $\\epsilon$ | 8/255 | 16/255 | 32/255 |     8/255     | 16/255 | 32/255 |\n|    Split   |   **80.44**   | **55.12**      | **12.04**      | 61.59      | 36.33      | **11.83**      |\n|  Split-FT  |   81.99   | 59.31      | 15.15      | **61.57**      | **36.21**      | 12.27      |\n|    Full    |   84.26\t   | 68.05      | 23.65      | 64.35      | 41.49      | 15.65      |\n\nThe results in the table demonstrate that employing disjoint training sets (Split) indeed enhances adversarial transferability and reducing model accuracy, consistent with our theory. Also, if we consider larger model architectures, the limited training data in each split subset may result in model underfitting, which may adversely impact attack effectiveness. We will incorporate the full results in our final version.", "claims_and_evidence": "Mostly yes. First, let me say that I like the authors' idea on how to extend the mathematics of uniform convergence analysis in statistical learning theory to the transferability on model ensemble attacks. The theorems look correct and make sense to me.\n\nThe only gap that I can find in the authors' analysis is the term $H_{\\alpha}^{\\frac{1}{\\alpha}}(P_{\\theta^N}\\Vert P_{\\otimes_{i=1}^n \\theta})$. The main difference between uniform convergence analysis in statistical learning theory and the authors' formulation is that in generalization analysis, we commonly assume the samples are drawn independently from a distribution, and so there is no term $H_{\\alpha}^{\\frac{1}{\\alpha}}(P_{\\theta^N}\\Vert P_{\\otimes_{i=1}^n \\theta})$. However, in the standard model ensemble attack scenario, the models may have been trained with fully or partially identical training data, and therefore the models could be quite correlated. Therefore, it seems to me that the term $H_{\\alpha}^{\\frac{1}{\\alpha}}(P_{\\theta^N}\\Vert P_{\\otimes_{i=1}^n \\theta})$ could be quite large and make the bound vaccuous in practice.\n\nI suggest the authors to clearly discuss the above point in the paper, because I find it a major difference between the two problem settings. However, I still tend to rate the work positively, as I find the authors' idea very interesting.", "methods_and_evaluation_criteria": "The experimental methodology appears well-structured and aligned with the theoretical claims.", "theoretical_claims": "While I have not verified every derivation in full detail, the results appear correct and consistent with existing theoretical techniques.", "experimental_designs_or_analyses": "One key aspect of the experimental design that needs discussion is the correlation between ensemble models due to shared training data. Have the authors examined how the results change if disjoint training sets are used for the ensemble? Based on the theoretical framework, using disjoint training sets should reduce the correlation term, thereby improving transferability. I encourage the authors to conduct and report such an experiment.", "relation_to_broader_scientific_literature": "The work extends the mathematics of uniform convergence analysis to the setting of adversarial transferability in model ensembles. The results parallel standard generalization bounds and adapt them to this new context.", "essential_references_not_discussed": "No missing key reference as far as I can tell", "other_strengths_and_weaknesses": "See my previous comments.", "other_comments_or_suggestions": "See my previous comments.", "questions_for_authors": "See my previous comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jkZyVidG1e", "reviewer": "Reviewer_Ejwu", "summary": "The paper presents a theoretical framework for model ensemble adversarial attacks, focusing on transferable adversarial examples. It defines transferability error, diversity, and Rademacher complexity, and decomposes transferability error into vulnerability and diversity. The authors apply information theory to derive bounds on transferability error and suggest practical strategies for improving adversarial transferability. The framework is validated through experiments on multiple datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your insightful review of our work!\n\n>**Q1**: The authors should include direct comparisons with state-of-the-art adversarial attack methods in the experiments.\n\n**A1**: We sincerely appreciate the reviewer’s constructive feedback. In direct response to Reviewer uk2i’s Question 1, we have significantly expanded our evaluation to incorporate additional attack algorithms as suggested.\n\n>**Q2**: The mathematical derivations in the paper are highly complex and lack intuitive explanations, which may pose difficulties for readers, especially those without a deep background in information theory.\n\n**A2**: Thank you for your constructive comments. We will provide more step-by-step explanations about information-theoretic analysis in Appendix B.5 in the final version.\n\n>**Q3**: The paper could benefit from a more detailed discussion of the practical implications of the theoretical results, particularly in terms of designing more effective adversarial attack algorithms.\n\n**A3**:  We thank the reviewer for the constructive suggestion. We show an example here to investigate how properly controlling the model complexity of surrogate models can contribute to more effective adversarial attack algorithms, which is in line with our theory. We use ImageNet as the dataset here. To conduct model ensemble attack, we fine-tune surrogate models—VGG16, InceptionV3, and Visformer—using a sparse Softmax cross-entropy loss [1]. This modification encourages sparsity in the model’s output distribution, and we observe in our experiments that the model complexity (L2 norm of the weight matrix) are reduced after using such a loss:\n\n|                     | VGG16 | Visformer | InceptionV3 |\n|:-------------------:|:-----:|-----------|:-----------:|\n|       Original      | 37.37 | 25.94     |    49.24    |\n| Sparse Softmax Loss | 33.12 | 20.6      |    48.53    |\n\nBuilding upon three advanced transfer attack methods—MI-FGSM [2], SVRE [3], and SIA [4]—we propose their sparsity-enhanced variants (MI-FGSM-S, SVRE-S, and SIA-S) through the integration of sparse Softmax loss during surrogate model training. We consider eight kinds of model architectures and measure attack performance via attack success rate, where higher value corresponds to stronger attack effectiveness. \n\n|   | ResNet50 | VGG16  | MobileNetV2 | InceptionV3 | ViT-B16 | PiT-B  | Visformer | Swin-T |\n|-----------|----------|--------|-------------|-------------|---------|--------|-----------|--------|\n| MI-FGSM   | 66.0     | **99.9**   | 76.8        | 97.5        | 37.3    | 53.8   | 88.9      | 66.7   |\n| **MI-FGSM-S** | **68.9**     | 99.7   | **79.2**        | **99.1**        | **39.0**    | **54.5**   | **90.6**      | **68.1**   |\n| SVRE      | 65.2     | **99.9**   | 79.0        | 98.6        | 32.4    | 49.2   | 92.3      | 64.3   |\n| **SVRE-S**    | **66.9**     | **99.9**   | **81.2**        | **98.9**        | **34.2**    | **51.3**   | **93.0**      | **65.9**   |\n| SIA       | 97.2     | **100.0**  | **98.4**        | **99.7**        | 75.9    | 91.9   | 99.0      | 96.1   |\n| **SIA-S**     | **98.1**     | **100.0**  | 98.2        | 99.6        | **79.2**    | **93.2**   | **99.5**      | **97.5**   |\n\nAs can be seen in the table, these variants outperform their standard counterparts in most cases, demonstrating the benefit of controlling model complexity in both CNNs and visual transformer settings to improve adversarial transferability. Beyond this example shown above, we believe that our work can also inspire the development of more stronger attack algorithms in the future.\n\n[1] From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification. ICML 2016\n\n[2] Boosting Adversarial Attacks with Momentum. CVPR 2018.\n\n[3] Stochastic Variance Reduced Ensemble Adversarial Attack\nfor Boosting the Adversarial Transferability. CVPR 2022.\n\n[4] Structure Invariant Transformation for better Adversarial Transferability. ICCV 2023.\n\n>**Q4**: To improve the paper's usability, the authors should consider releasing the code and providing practical examples to help other researchers implement and test the framework.\n\n**A4**: We appreciate the reviewer's valuable suggestion. We are happy to release the code and provide implementation examples to facilitate reproducibility and adoption by the research community.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The key contributions of the paper are well-related to the broader scientific literature. The authors build on existing work in adversarial attacks and model ensemble methods, providing a theoretical foundation for transferable model ensemble attacks. They reference relevant studies on adversarial transferability, model ensemble diversity, and generalization in machine learning. The paper also discusses the connection between adversarial transferability and model generalization, drawing insights from statistical learning theory.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths：\n\n1. The paper introduces a new theoretical framework to understand the role of model ensembles in transferable adversarial attacks. This framework combines Rademacher complexity and information-theoretic tools, which adds some theoretical novelty.\n2. The authors provide a mathematical decomposition of the transferability error in model ensemble adversarial attacks, highlighting the trade-offs between vulnerability and diversity. This new perspective contributes to the understanding of adversarial transferability.\n3. The experimental results are comprehensive and validate the theoretical claims across multiple datasets and model architectures.\n\nWeaknesses：\n\n1. While the framework is theoretically innovative, its practical effectiveness remains unclear. The paper does not provide a sufficient comparison with existing state-of-the-art adversarial attack methods, nor does it demonstrate the practical performance of the method. \n2. The paper could benefit from a more detailed discussion of the practical implications of the theoretical results, particularly in terms of designing more effective adversarial attack algorithms.\n3. The experiments could be expanded to include a wider range of model architectures and datasets to further strengthen the empirical validation.\n4. The mathematical derivations in the paper are highly complex and lack intuitive explanations, which may pose difficulties for readers, especially those without a deep background in information theory. To improve its practical applicability, the paper should provide a clearer connection between the theoretical framework and real-world use cases.\n5. Although the paper presents a new theoretical framework, it does not sufficiently compare it to current adversarial attack methods, such as gradient-based attacks, input transformation techniques, or other model ensemble methods. Without demonstrating that the proposed framework performs better than existing approaches, its contribution remains unclear.\n6. Despite the theoretical contributions, the paper fails to demonstrate significant practical benefits or applications. The framework's real-world impact is not well established.", "other_comments_or_suggestions": "Suggestions：\n\n1. The authors should include direct comparisons with state-of-the-art adversarial attack methods in the experiments, especially focusing on black-box attacks and model ensemble techniques, to highlight the advantages of the proposed approach.\n2. The authors should consider simplifying the mathematical derivations and providing more intuitive explanations or examples, making the theoretical framework more accessible to a broader audience.\n3. The authors should provide more experimental validation of the method’s performance in real-world adversarial tasks, and compare it directly with other methods to establish its practical value.\n4. To improve the paper's usability, the authors should consider releasing the code and providing practical examples to help other researchers implement and test the framework.", "questions_for_authors": "See weaknesses and suggestions above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bDAIyplZuv", "reviewer": "Reviewer_L4RB", "summary": "The authors investigated the issue of transfer attacks based on ensembles. They provided a theoretical framework for the transferability of adversarial examples, which can be controlled by theor loss and variance among models. The authors conducted some experiments to validate their theoretical findings.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive comments! We address all your questions and concerns in the following responses.\n\n>**Q1**: In transfer attacks, the surrogate model and the target model have different model architectures and parameter numbers. However, the authors implicitly assume that the surrogate and target models share the same parameter space (right column, Line 111-112), which significantly limits the generalizability of their theoretical results. \n\n**A1**: Thank you for your comments. \n- Firstly, in Section 4.2 (Remark 2), we explicitly discuss how Theorem 4.3 can be generalized to cases where the surrogate and target models have distinct parameter distributions. This addresses part of the concern regarding parameter space assumptions.\n- Secondly, as the first theoretical work to formally link adversarial transferability with generalization theory, our primary contribution is a foundational framework rather than an exhaustive treatment of all scenarios. To our best knowledge, transferable model ensemble adversarial attacks remain less theoretically explored. The field still lacks papers that understand it theoretically. We hope our work inspires follow-up studies to fully address the problems you mention in the future.\n\n>**Q2**: I suggest revising some of the notation. It is strange that (x, y) represents an adversarial example (right column, Line 121-122).\n\n**A2**: Thank you for your constructive comments. We agree with your suggestion and will use $(x^{\\text{adv}}, y)$ instead to represent an adversarial example in our final version.\n\n>**Q3**: The theoretical results do not seem particularly novel, as they represent a widely accepted conclusion.\n\n**A3**: We sincerely appreciate the reviewer's comments. Firstly, we're encouraged that multiple other reviewers (uk2i, kypp, and Ejwu) have explicitly recognized the novelty of our theoretical framework in their comments. It suggests our approach does offer fresh insights to the field. Secondly, we theoretically analyzes and validates three important practical guidelines to improve adversarial transferability, including (1) incorporating more surrogate models, and (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting. Our work provides the first theoretical framework to explain these empirical observations in adversarial transferability, while offering practical insights for algorithm design. We hope this foundation inspires deeper understanding and future advances in the field.\n\n>**Q4**: The experiments are limited. The authors' experimental results do not differ significantly from those in previous empirical studies. Additionally, the authors mentioned conducting experiments in ImageNet, yet I could not find any results in ImageNet. The model architectures and attack methods used in the experiments are also limited. The authors could include visual transformers and more transfer attack methods.\n\n**A4**: We appreciate the reviewer's constructive feedback, which has helped us strengthen our work. Our experimental design was carefully crafted to validate our novel theoretical contributions previously unexplored in the literature. In response to the reviewer's valuable suggestions, we have expanded our experiments to include additional attack methods, diverse model architectures (including transformers) and larger dataset (ImageNet). These new experiments, detailed in our response to Reviewer Ejwu's Question 3, provide important insights for future research in transfer adversarial attacks. Furthermore, in our response to Reviewer uk2i's Question 1, we have also expanded the experimental validation by employing another effective attack methodology. This additional analysis not only confirms our theoretical findings but also broadens the scope and strengthens the practical implications of our research.\n\n>**Q5**: Some theoretical papers have been overlooked, e.g., [1].\n>\n>[1] Transferability Bound Theory: Exploring Relationship between Adversarial Transferability and Flatness, NeurIPS 2024.\n\n**A5**: We appreciate you bringing this work to our attention. We will ensure proper citation of [1] in our final version. While [1] examines flatness and transferability, we establish the theoretical link between statistical learning theory and adversarial transferability. Our framework provides new insights into transferability through the lens of generalization theory, offering complementary (rather than overlapping) perspectives to [1]. We’re happy to further clarify these differences if needed.", "claims_and_evidence": "Most of the claims are clear and reasonable.", "methods_and_evaluation_criteria": "This paper does not introduce new methods or datasets.", "theoretical_claims": "There are some flaws in the authors' theory. Firstly, in transfer attacks, the surrogate model and the target model have different model architectures and parameter numbers. However, the authors implicitly assume that the surrogate and target models share the same parameter space (right column, Line 111-112), which significantly limits the generalizability of their theoretical results. Secondly, I suggest revising some of the notation. It is strange that (x, y) represents an adversarial example (right column, Line 121-122). Finally, the theoretical results do not seem particularly novel, as they represent a widely accepted conclusion.", "experimental_designs_or_analyses": "The experiments are limited. The authors' experimental results do not differ significantly from those in previous empirical studies. Additionally, the authors mentioned conducting experiments in ImageNet, yet I could not find any results in ImageNet. The model architectures and attack methods used in the experiments are also limited. The authors could include visual transformers and more transfer attack methods.", "relation_to_broader_scientific_literature": "The authors establish a theory regarding ensemble transfer adversarial attacks.", "essential_references_not_discussed": "Some theoretical papers have been overlooked, e.g., [1].\n\n[1] Transferability Bound Theory: Exploring Relationship between Adversarial Transferability and Flatness, NeurIPS 2024.", "other_strengths_and_weaknesses": "The authors' theory appears to offer little benefit for the future development of transfer adversarial attacks. It is also unclear how tight the suggested bound is.", "other_comments_or_suggestions": "I have no further comments.", "questions_for_authors": "Please refer to the above points.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Three of the four reviewers have given positive scores with detailed reviews. The authors have put a lot of effort into addressing the reviewers' questions. In particular, the only reviewer who gave a "2: Weak Reject" said during the discussion phase that "the major concern is the applicability of the proposed theory to transfer-based attacks where the surrogate and the target models differ in architecture, and the authors acknowledge that the theory cannot be extended to this scenario for now." Considering that in the context of model ensemble-based transfer attacks, theoretical analyses are very rare, the AC thought this is not a big problem that would lead to rejection.

Suggestions from the AC and SAC for improving the paper:
(1) The theoretical analyses can be significantly shortened. Specifically, the theory is a direct application of usual uniform convergence techniques, with the setting that the learned models are the random variables and the data points define the loss functions for these random variables. Given these definitions, the analysis becomes a straightforward application of standard theory (or a more advanced version of it, coming from Esposito & Mondelli, 2024). This should be made explicit in the paper. As such, a lot of the proofs could be shortened instead of rederiving well-known parts of existing proofs with the current notation. For example, Theorem 4.3 is a direct consequence of Theorem 1 of Esposito and Mondelli (2024) and the standard way of going from tail bounds to Rademacher bounds. 

(2) The paper does not discuss the dependence of the models other than the Hellinger term should be made small by having as independent models as possible. For example, a simple scenario is that if all models are trained on the same dataset then -- conditioned on the dataset -- they are independent (assuming independent initialization), so we immediately get a much stronger transfer guarantee for all models which are trained on the same training data. Exploring how the random choice of the training data affects the bound would also be an interesting addition.

(3) In addition to validating the three well-known practical guidelines for reducing transferability error, the authors should discuss what new insights the proposed theory can provide into further improving transferability.
Decision: Accept (poster)