Understanding Model Ensemble in Transferable Adversarial Attack

## Abstract

Abstract {NoHyper} † † ⋆ ⋆ \star ⋆ indicates equal contribution. {NoHyper} † † † † \dagger † indicates the corresponding author. Model ensemble adversarial attack has become a powerful method for generating transferable adversarial examples that can target even unknown models, but its theoretical foundation remains underexplored.
To address this gap, we provide early theoretical insights that serve as a roadmap for advancing model ensemble adversarial attack.
We first define transferability error to measure the error in adversarial transferability, alongside concepts of diversity and empirical model ensemble Rademacher complexity.
We then decompose the transferability error into vulnerability, diversity, and a constant, which rigidly explains the origin of transferability error in model ensemble attack: the vulnerability of an adversarial example to ensemble components, and the diversity of ensemble components.
Furthermore, we apply the latest mathematical tools in information theory to bound the transferability error using complexity and generalization terms, contributing to three practical guidelines for reducing transferability error: (1) incorporating more surrogate models, (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting.
Finally, extensive experiments with 54 models validate our theoretical framework, representing a significant step forward in understanding transferable model ensemble adversarial attacks.

## 1 Introduction

Neural networks are highly vulnerable to adversarial examples [Szegedy et al., , Goodfellow et al., ]—perturbations that closely resemble the original data but can severely compromise safety-critical applications [Zhang and Li, , Kong et al., , Bortsova et al., , Zhang et al., , Tang et al., ].
Even more concerning is the phenomenon of adversarial transferability [Papernot et al., , Liu et al., ]: adversarial examples crafted to deceive one model often succeed in attacking others.
This property enables attacks without requiring any knowledge of the target model, significantly complicating efforts to ensure the robustness of neural networks [Dong et al., , Silva and Najafirad, ].

To enhance adversarial transferability, researchers have proposed a range of algorithms that fall into three main categories: input transformation [Xie et al., , Wang et al., ], gradient-based optimization [Gao et al., , Xiong et al., ], and model ensemble attacks [Li et al., , Chen et al., ].
Among these, model ensemble attacks have proven especially powerful, as they leverage multiple models to simultaneously generate adversarial examples that exploit the strengths of each individual model [Dong et al., ]. Moreover, these attacks can be combined with input transformation and gradient-based optimization methods to further improve their effectiveness [Tang et al., ].
However,
despite the success of such attacks, their theoretical foundation remains poorly understood.
This prompts an important question:
Can we establish a theoretical framework for transferable model ensemble adversarial attacks to shape the evolution of future algorithms?

Figure: Figure 1: Vulnerability-diversity decomposition of transferability error. (a) The transferability error of a given adversarial example is defined as the difference in expected loss value between that example and the most transferable one. (b) Vulnerability is the loss value of the expected ensemble classifier on the adversarial example. (c) Diversity represents the variance in model ensemble predictions that correspond to the correct class.
Refer to caption: x1.png

To conduct a preliminary exploration of this profound question,
we propose three novel definitions as a prerequisite of our theoretical framework.
Firstly, we define transferability error as the gap in expected loss between an adversarial example and the one with the highest loss within a feasible region of the input space.
It captures the ability of an adversarial example to generalize across unseen models, representing its transferability.
Secondly, we introduce prediction variance across the ensemble classifiers.
It address an open problem in model ensemble attack about how to to quantify diversity and assists in selecting ensemble components.
Finally, we also introduce the empirical model ensemble Rademacher complexity, inspired by Rademacher complexity [Bartlett and Mendelson, ], as a measure of the flexibility of ensemble components in an attack.

With these three definitions, we offer two key theoretical insights.
First, we show the vulnerability-diversity decomposition of transferability error (Figure 1), highlighting the preference for ensemble components that are powerful attackers and induce greater prediction variance among themselves.
However, this also uncovers a fundamental trade-off between vulnerability and diversity, making it challenging to maximize both simultaneously.
To mitigate this issue and provide more practical guidelines, we present an upper bound for transferability error, incorporating empirical model ensemble Rademacher complexity and a generalization term.
The primary challenge in proof lies in the application of cutting-edge mathematical tools from information theory [Esposito and Mondelli, ], which are crucial for addressing the complex issue of relaxing the independence assumption among surrogate classifiers.
Our theoretical analysis leads to a crucial takeaway for practitioners: Including more and diverse surrogate models with reduced model complexity in cases of overfitting helps tighten the transferability error bound, thereby improving adversarial transferability.
Finally, the experimental results support the soundness of our theoretical framework, highlighting a key step forward in the deeper understanding of transferable model ensemble adversarial attacks.

## 2 Related Work

### 2.1 Transferable Adversarial Attack

Researchers have developed various algorithms to enhance adversial transferability.
Most of them fall into three categories: input transformation, gradient-based optimization, and model ensemble attack.
Input transformation techniques apply data augmentation strategies to prevent overfitting to the surrogate model.
For instance, random resizing and padding [Xie et al., ], downscaling [Lin et al., ],
and mixing [Wang et al., ].
Gradient-based optimization optimizes the generation of adversarial examples to achieve better transferability.
Some popular ideas include applying momentum [Dong et al., ], Nesterov accelerated gradient [Lin et al., ], scheduled step size [Gao et al., ] and gradient variance reduction [Xiong et al., ].
Model ensemble attack combine outputs from surrogate models to create an ensemble loss, increasing the likelihood to deceive various models simultaneously.
It can be applied collectively with both input transformation and gradient-based optimization algorithms [Tang et al., ].
Some popular ensemble paradigms include loss-based ensemble [Dong et al., ], prediction-based [Liu et al., ], logit-based ensemble [Dong et al., ], and longitudinal strategy [Li et al., ].
Moreover, advanced ensemble algorithms have been created to ensure better adversarial transferability [Li et al., , Wu et al., , Chen et al., ].
An extended and detailed summary of related work is in Appendix C.

Within the extensive body of research on model ensemble attacks, two notable observations stand out.
First, increasing the number of models in an ensemble improves adversarial transferability [Liu et al., , Dong et al., , Lin et al., , Gubri et al., ].
Second, using more diverse surrogate models with varying architectures and back-propagated gradients [Tang et al., ] further enhances transferability.
However, to our best knowledge, these intriguing phenomena have yet to be fully understood from a theoretical perspective.
In this paper, we are the first to provide a theoretical explanation for them, offering insights that can guide future algorithm design.

### 2.2 Theoretical Understanding of Adversarial Transferability

In contrast to the wealth of empirical and intuitive studies, research on the theoretical understanding of adversarial transferability remains limited.
Recent efforts have primarily focused on aspects such as data [Tramèr et al., ], surrogate model [Wang and Farnia, ], optimization [Yang et al., , Zhang et al., , Chen et al., ] and target model [Zhao et al., ].
Tramèr et al. [] investigates the space of transferable adversarial examples and establishes conditions on the data distribution that suggest transferability for some basic models.
In terms of the surrogate model generalization, Wang and Farnia [] builds the generalization gap to show that a surrogate model with a smaller generalization error leads to more transferable adversarial examples.
From an optimization perspective, Yang et al. [], Zhang et al. [] establish upper and lower bounds on adversarial transferability, linking it to model smoothness and gradient similarity,
while Chen et al. [] provides theoretical evidence connecting transferability to loss landscape flatness and closeness to local optima.
Regarding the target model, Zhao et al. [] theoretically reveals that reducing the discrepancy between the surrogate and target models can limit adversarial transferability.

Despite these theoretical advances, to the best of our knowledge, transferable model ensemble adversarial attacks remain unexplored.
To address this gap, we take a pioneering step by presenting the first theoretical analysis of such attacks.
Our work not only offers theoretical insights into these attacks but also incorporates recent advancements in learning theory, laying the groundwork for future theoretical investigations into adversarial transferability.

## 3 Key Definitions: Transferability Error, Diversity, and Ensemble Complexity

In this section, we first highlight the fundamental goal of model ensemble adversarial attack (Section 3.1).
Then we define the transferability error (Section 3.2), diversity in transferable model ensemble attack (Section 3.3) and empirical model ensemble Rademacher complexity (Section 3.4).

### 3.1 Model Ensemble Adversarial Attack

Given a feature space $\mathcal{X}\subset\mathbb{R}^{d}$ and a label space $\mathcal{Y}=\{-1,1\}$, we have a joint distribution $\mathcal{P}_{\mathcal{Z}}$ over the input space $\mathcal{Z}=\mathcal{X}\times\mathcal{Y}$.
The training set $Z_{\text{train}}=\{z_{i}|z_{i}=(x_{i},y_{i})\in\mathcal{Z},i=1,\cdots,K\}$, which consists of $K$ examples drawn independently from $\mathcal{P}_{\mathcal{Z}}$.
We denote the hypothesis space by $\mathcal{H}:\mathcal{X}\mapsto\mathcal{Y}$ and the parameter space by $\Theta$. Let $f(\theta;\cdot)\in\mathcal{H}$ be a classifier parameterized by $\theta\in\Theta$, trained for a classification task using a loss function $\ell:\mathcal{Y}\times\mathcal{Y}\mapsto\mathbb{R}_{0}^{+}$.
Let $\mathcal{P}_{\Theta}$ represent the distribution over the parameter space $\Theta$. Define $\mathcal{P}_{\Theta^{N}}$ as the joint distribution over the product space $\Theta^{N}$, which denotes the space of $N$ such sets of parameters.
We use $Z_{\text{train}}$ to train $N$ surrogate models $f(\theta_{1};\cdot),\cdots,f(\theta_{N};\cdot)$ for model ensemble.
The training process of these $N$ classifiers can be viewed as sampling the parameter sets $(\theta_{1},\ldots,\theta_{N})$ from the distribution $\mathcal{P}_{\Theta^{N}}$.
For a clean data $\hat{z}=(\hat{x},y)\in\mathcal{Z}$, an adversarial example $z=(x,y)\in\mathcal{Z}$, and $N$ classifiers for model ensemble attack, define the population risk $L_{P}(z)$ and the empirical risk $L_{E}(z)$ of the adversarial example $z$ as

$$ $\displaystyle L_{P}(z)=\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}[\ell(f( \theta;x),y)],$ (1) and $\displaystyle L_{E}(z)=\frac{1}{N}\sum_{i=1}^{N}\ell(f(\theta_{i};x),y).$ (2) $$

Intuitively, a transferable adversarial example leads to a large $L_{P}(z)$ because it can attack many classifiers with parameter $\theta\in\Theta$.
Therefore, the most transferable adversarial example $z^{*}=(x^{*},y)$ around $z$ is defined as

$$ $\displaystyle x^{*}=\arg\max_{x\in\mathcal{B}_{\epsilon}(\hat{x})}L_{P}(z),$ (3) $$

where $\mathcal{B}_{\epsilon}(\hat{x})=\{x:\|x-\hat{x}\|_{2}\leq\epsilon\}$ is an adversarial region centered at $\hat{x}$ with radius $\epsilon>0$.
However, the expectation in $L_{P}(z)$ cannot be computed directly. Thus, when generating adversarial examples, the empirical version Eq. (2) is used in practice, such as loss-based ensemble attack [Dong et al., ].
So the adversarial example $z=(x,y)$ is obtained from

$$ $\displaystyle x=\arg\max_{x\in\mathcal{B}_{\epsilon}(x)}L_{E}(z).$ (4) $$

There is a gap between the adversarial example $z$ we find and the most transferable one $z^{*}$. It is due to the fact that the ensemble classifiers cannot cover the whole parameter space of the classifier, i.e., there is a difference between $L_{P}(z)$ and $L_{E}(z)$.
Accordingly,
the core objective of transferable model ensemble attack is to design approaches that approximate $L_{E}(z)$ to $L_{P}(z)$, thereby increasing the transferability of adversarial examples.

### 3.2 Transferability Error

Considering the difference between $z$ and $z^{*}$, the transferability of an adversarial example $z$ can be characterized as the difference in population risk between it and the optimal one.

###### Definition 1 (Transferability Error) .

The transferability error of $z$ with radius $\epsilon$ is defined as:

$$ $TE(z,\epsilon)=L_{P}(z^{*})-L_{P}(z).$ (5) $$

There always holds $TE(z,\epsilon)\geq 0$ as $L_{P}(z^{*})\geq L_{P}(z)$. The closer $TE(z,\epsilon)$ is to 0 0, the better the transferability of $z$.
Therefore, in principle, the essential goal of various model ensemble attack algorithms is to make transferability error $TE(z,\epsilon)$ as small as possible.
Moreover, if the distribution over the parameter space $\mathcal{P}_{\Theta}$, adversarial region $\mathcal{B}_{\epsilon}(x)$ and loss function $\ell$ are fixed, then $L_{P}(z^{*})$ becomes a constant, which means that the goal of minimizing $TE(z,\epsilon)$ becomes maximizing $L_{P}(z)$.

In the following lemma, we will show how the difference between empirical risk and population risk affects the transferability error of $z$.
The proof is in Appendix B.1.

###### Lemma 1 .

The transferability error defined by Eq. (5) is bounded by the largest absolute difference between $L_{P}(z)$ and $L_{E}(z)$, i.e.,

$$ $\displaystyle TE(z,\epsilon)\leq 2\sup_{z\in\mathcal{Z}}\left|L_{P}(z)-L_{E}(z )\right|.$ (6) $$

The lemma strictly states that if we can bound the difference between $L_{P}(z)$ and $L_{E}(z)$, the transferability error can be constrained to a small value, thereby enhancing adversarial transferability.
This indicates that we can develop strategies to make $L_{E}(z)$ closely approximate $L_{P}(z)$, ultimately improving the transferability of adversarial examples.

###### Definition 1 (Transferability Error) .

###### Lemma 1 .

### 3.3 Quantifying Diversity in Model Ensemble Attack

Before the advent of model ensemble attacks, the formal definition of diversity in ensemble learning had remained a long-standing challenge for decades [Wood et al., ].
Unfortunately, this challenge continues to persist in the context of transferable model ensemble attacks.
Despite various intuitive approaches [Li et al., , Tang et al., ], there is still no widely accepted method for rigorously quantifying diversity.
In the following definition, we propose measuring diversity among ensemble attack classifiers through prediction variance, building on recent advances in ensemble learning theory [Ortega et al., , Wood et al., ].

###### Definition 2 (Diversity of Model Ensemble Attack) .

The diversity of model ensemble attack across $\theta\sim\mathcal{P}_{\Theta}$ for a specific adversarial example $z=(x,y)$ is defined as the variance of model prediction:

$$ $\displaystyle\textit{Var}_{\theta\sim\mathcal{P}_{\Theta}}\left(f(\theta;x) \right)=\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}\left[f(\theta;x)-\mathbb{E }_{\theta\sim\mathcal{P}_{\Theta}}f(\theta;x)\right]^{2}.$ (7) $$

It indicates the degree of dispersion in the predictions of different ensemble classifiers for the same adversarial example.
The diversity of model ensemble attack is a measure of ensemble member disagreement, independent of the label.
From an intuitive perspective,
the disagreement among the ensemble components helps prevent the adversarial example from overfitting to the classifiers in the ensemble, thereby enhancing adversarial transferability to some extent.

To calculate the diversity explicitly as a metric, we consider a dataset of adversarial examples $Z_{\text{attack}}=\{z_{i}|z_{i}=(x_{i},y_{i}),i=1,\cdots,M\}$ and $N$ classifiers in the ensemble.
The diversity is computed as the average sample variance of predictions for all adversarial examples in the dataset:

$$ $\displaystyle\text{Diversity}=\frac{1}{M}\sum_{i=1}^{M}\Bigg{[}\frac{1}{N}\sum _{j=1}^{N}\bigg{(}f(\theta_{j};x_{i})-\frac{1}{N}\sum_{j=1}^{N}f(\theta_{j};x_ {i})\bigg{)}^{2}\Bigg{]}.$ (8) $$

###### Remark .

For multi-class classification problems, $f(\theta;x)$ is replaced with the logit corresponding to the correct class prediction made by the classifier.

###### Definition 2 (Diversity of Model Ensemble Attack) .

###### Remark .

### 3.4 Empirical Model Ensemble Rademacher Complexity

We define the empirical Rademacher complexity for model ensemble by analogy to the original empirical Rademacher complexity [Koltchinskii and Panchenko, , Bartlett and Mendelson, ].

###### Definition 3 (Empirical Model Ensemble Rademacher Complexity) .

Given the input space $\mathcal{Z}=\mathcal{X}\times\mathcal{Y}$ and
$N$ classifiers $f(\theta_{1};\cdot),\cdots,f(\theta_{N};\cdot)$.
Let $\boldsymbol{\sigma}=\{\sigma_{i}\}_{i\in[N]}$ be a collection of independent Rademacher variables, which are random variables taking values uniformly in $\{+1,-1\}$.
We define the empirical model ensemble Rademacher complexity $\mathcal{R}_{N}(\mathcal{Z})$ as follows:

$$ $\mathcal{R}_{N}(\mathcal{Z})=\mathop{\mathbb{E}}\limits_{\boldsymbol{\sigma}} \left[\sup_{z\in\mathcal{Z}}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}\ell(f(\theta_{ i};x),y)\right].$ (9) $$

In conventional settings of machine learning, the empirical Rademacher complexity captures how well models from a function class can fit a dataset with random noisy labels [Shalev-Shwartz and Ben-David, ].
A sufficiently complex function class includes functions that can effectively fit arbitrary label assignments, thereby maximizing the complexity term [Mohri et al., ].
Likewise, in model ensemble attack, Eq. (9) is expected to measure the complexity of the input space $\mathcal{Z}$ relative to the $N$ classifiers.
Some extreme cases are analyzed in Appendix D.1.

###### Definition 3 (Empirical Model Ensemble Rademacher Complexity) .

## 4 Theoretically Reduce Transferability Error

### 4.1 Vulnerability-diversity Decomposition of Transferability Error

Inspired by the bias-variance decomposition [Geman et al., , Domingos, ] in learning theory,
we provide the corresponding theoretical support for prediction variance by decomposing the transferability error into vulnerability, diversity and constants.

###### Theorem 1 (Vulnerability-diversity Decomposition) .

Consider the squared error loss $l(f(\theta;x),y)=\left[f(\theta;x)-y\right]^{2}$ for a data point $z=(x,y)$.
Let $\tilde{f}(\theta;x)=\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}f(\theta;x)$ be the expectation of prediction over the distribution on the parameter space.
Then there holds

$$ $\displaystyle TE(z,\epsilon)=L_{P}(z^{*})-\underbrace{l(\tilde{f}(\theta;x),y) }_{\text{Vulnerability}}-\underbrace{\textit{Var}_{\theta\sim\mathcal{P}_{ \Theta}}f(\theta;x)}_{\text{Diversity}}.$ (10) $$

The proof and the empirical version of it is in Appendix B.2.
The “Vulnerability” term measures the risk of a data point $z$ being compromised by the model ensemble.
If the model ensemble is sufficiently strong to fit the direction opposite to the target label, the resulting high loss theoretically reduces the transferability error.
This insight suggests that selecting strong attackers as ensemble components leads to lower transferability error.
The “Diversity” term implies that selecting diverse attackers in a model ensemble attack theoretically contributing to a reduction in transferability error.
In conclusion,
Theorem 1 provides the following guideline for reducing transferability error in model ensemble attack: we are supposed to choose ensemble components that are both strong and diverse.

###### Remark .

This theoretical result connects the existing body of work and clarifies how each algorithm strengthens adversarial transferability.
For instance,
some approaches tend to optimizing the attack process [Xiong et al., , Chen et al., ] to improve “Vulnerability”, while others aim to diversify surrogate models [Li et al., , , Wang et al., ] to enhance “Diversity”.
Further discussion of these algorithms is provided in Appendix is provided in the Appendix D.4.

However, due to the mathematical nature of Eq. (10), there remains a vulnerability-diversity trade-off in model ensemble attacks,
similar to the well-known bias-variance trade-off [Geman et al., ].
This means that, in practice, it is not feasible to maximize both “Vulnerability” and “Diversity” simultaneously.
Recognizing this limitation,
we proceed with further theoretical analysis to propose more guidelines for practitioners in the following section.

###### Theorem 1 (Vulnerability-diversity Decomposition) .

###### Remark .

### 4.2 Upper Bound of Transferability Error

We develop an upper bound of transferability error in this section.
We begin by taking Multi-Layer Perceptron (MLP) as an example of deep neural network and derive the upper bound of $\mathcal{R}_{N}(\mathcal{Z})$.
The proof is in Appendix A.4.

###### Lemma 2 .

Let $\mathcal{H}=\{x\mapsto W_{l}\phi_{l-1}\left(W_{l-1}\phi_{l-2}\left(\ldots\phi_
{1}\left(W_{1}x\right)\right)\right)\}$ be the class of real-valued networks of depth $l$, where $x\in\mathbb{R}^{d_{1}}$, $W_{i}\in\mathbb{R}^{d_{i+1}\times d_{i}}$.
Given $N$ classifiers from $\mathcal{H}$, where the parameter matrix is $W_{ij},i\in\{1,\cdots,n\},j\in\{1,\cdots,l\}$ and $T=\prod_{j=1}^{l}\sup_{i\in[n]}\|W_{i,j}\|_{F}$. Let $\|x\|_{F}\leq B$.
With 1-Lipschitz activation functions $\phi_{1},\cdots,\phi_{l-1}$ and $1$-Lipschitz loss function $\ell(yf(x))$, there holds:

$$ $\displaystyle\mathcal{R}_{N}(\mathcal{Z})\leq\frac{\left(\sqrt{(2\log 2)l}+1 \right)BT}{\sqrt{N}}.$ (11) $$

###### Remark .

We also derive the upper bound of $\mathcal{R}_{N}(\mathcal{Z})$ for the cases of linear model (Appendix A.2) and two-layer neural network (Appendix A.3). These results are special cases of the above theorem.

In particular,
a larger $N$ and smaller $T$ will give $\mathcal{R}_{N}(\mathcal{Z})$ a tighter bound.
Notice that $T$ contains the norm of weight matrices, which is related to model complexity [Bartlett et al., , Neyshabur et al., ].
And a smaller model complexity corresponds to a smaller $T$ [Loshchilov and Hutter, ].
In summary,
Lemma 2 mathematically shows that increasing the number of surrogate models and reducing the model complexity of them can limit the value of $\mathcal{R}_{N}(\mathcal{Z})$.

We now provide the upper bound of transferability error, and the proof is in Appendix B.3.

###### Theorem 2 (Upper bound of Transferability Error) .

Given the transferability error defined by Eq. (5) and general rademacher complexity defined by Eq. (9).
Let
$\mathcal{P}_{\bigotimes_{i=1}^{N}\Theta}$ be the joint measure induced by the product of the marginals.
If the loss function $\ell$ is bounded by $\beta\in R_{+}$ and $\mathcal{P}_{\Theta^{N}}$ is absolutely continuous with respect to $\mathcal{P}_{\bigotimes_{i=1}^{N}\Theta}$ for any function $f_{i}$, then for $\alpha>1$ and $\gamma=\frac{\alpha}{\alpha-1}$, with probability at least $1-\delta$, there holds

$$ $TE(z,\epsilon)\leq 4\mathcal{R}_{N}(\mathcal{Z})+\sqrt{\frac{2\gamma\beta^{2}} {N}\ln{\frac{2^{\frac{1}{\gamma}}H_{\alpha}^{\frac{1}{\alpha}}\left(\mathcal{P }_{\Theta^{N}}\|\mathcal{P}_{\bigotimes_{i=1}^{N}\Theta}\right)}{\delta}}},$ (12) $$

where $H_{\alpha}\left(\mathcal{P}_{\Theta^{N}}\|\mathcal{P}_{\bigotimes_{i=1}^{N}
\Theta}\right)$ is the Hellinger integrals [Hellinger, ] with parameter $\alpha$, which measures the divergence between two probability distributions if $\alpha>1$ [Liese and Vajda, ].

The first term in Eq. (12) suggests that incorporating more surrogate models with less model complexity
in ensemble attack will constrain $\mathcal{R}_{N}(\mathcal{Z})$ and enhances adversarial transferability.
Intuitively, incorporating more models helps prevent any single model from overfitting to a specific adversarial example.
Such theoretical heuristic is also supported by experimental results [Liu et al., , Dong et al., , Lin et al., , Li et al., , Gubri et al., , Chen et al., ], which also stress the advantage of more surrogate models to obtain transferable attack.
Additionally,
when there is an overfitting issue,
models with reduced complexity will mitigate it.

The second term also suggests that a large
$N$ (using more models) can lead to a tighter bound.
Furthermore, it motivates the idea that reducing the interdependence among the parameters in ensemble components (i.e., increasing their diversity) results in a tighter upper bound for $TE(z,\epsilon)$.
Recall that $H_{\alpha}(\mathcal{P}_{\Theta^{N}}\|\mathcal{P}_{\bigotimes_{i=1}^{N}\Theta})$ represents the divergence between the joint distribution $\mathcal{P}_{\Theta^{N}}$ and the product of marginals $\mathcal{P}_{\bigotimes_{i=1}^{N}\Theta}$.
The joint distribution captures dependencies, while the product of marginals does not.
Therefore, $H_{\alpha}(\mathcal{P}_{\Theta^{N}}\|\mathcal{P}_{\bigotimes_{i=1}^{N}\Theta})$ measures the degree of dependency among the parameters from $N$ classifiers.
As a result, increasing the diversity of parameters in surrogate models and reducing their interdependence enhances adversarial transferability.
This theoretical conclusion is also supported by empirical algorithms [Li et al., , Tang et al., ], which also advocate for generating adversarial examples from diverse models.

###### Lemma 2 .

###### Remark .

###### Theorem 2 (Upper bound of Transferability Error) .

#### The trade-off between complexity and diversity.

Reducing model complexity may conflict with increasing diversity.
We discuss this issue from two angles.
On one hand, when generating adversarial examples from simpler models to attack more complex ones, the overall model complexity is lower, but diversity may also be limited due to the simpler structure of the ensemble attackers.
On the other hand, attacking simpler models with a stronger, more diverse ensemble may increase diversity but also raise model complexity. In this scenario, reducing complexity can help prevent overfitting and lead to a tighter transferability error bound, albeit with a slight reduction in ensemble diversity.
In summary, striking a balance between model complexity and diversity is crucial in practice.

#### From generalization error to transferability error.

The mathematical form of Eq. (12) is in line with the generalization error bound [Bartlett and Mendelson, ].
However, we note that a key distinction between transferability error and generalization error lies in the independence assumption.
Conventional generalization error analysis relies on an assumption: each data point from the dataset is independently sampled [Zou and Liu, , Hu et al., ].
By contrast, the surrogate models for ensemble attack are usually trained on the datasets with similar tasks, e.g., image classification.
In this case, we cannot assume these surrogate models behave independently for a solid theoretical analysis, which increases the difficulty of proof in our paper.
To build the gap between generalization error and transferability error, our proof introduces the latest techniques in information theory [Esposito and Mondelli, ].
And Refer to Appendix D.2 for a detailed discussion.

### 4.3 Generalization and Adversarial Transferability

We offer new insights into a foundational and popular analogy in the literature:
The transferability of an adversarial example is an analogue to the generalizability of the model [Dong et al., , Lin et al., , Wang et al., , Wang and He, , Xiong et al., , Chen et al., ]. Interestingly, our theory also sheds light on this insight in several ways.

First,
the mathematical formulations in Lemma 1 is similar to generalization error [Vapnik, , Bousquet and Elisseeff, ].
Also,
Lemma 2 is similar to the bound of the original Rademacher complexity [Golowich et al., ].
More importantly, recall that in the conventional framework of learning theory:
(1) increasing the size of training set typically leads to a better generalization of the model [Bousquet and Elisseeff, ];
(2) improving the diversity among ensemble classifiers makes it more advantageous for better generalization [Ortega et al., ];
and (3) reducing the model complexity [Cherkassky, ] may mitigate overfitting and benefit the generalization ability.
These ideas correspond to each of our theoretical understanding in Section 4.
Overall, we support the analogy from a theoretical perspective.
And further detailed discussion is in Appendix D.3.

## 5 Experiments

We conduct our experiments on four datasets, including the MNIST [LeCun, ], Fashion-MNIST [Xiao et al., ], CIFAR-10 [Krizhevsky et al., ], and ImageNet-1K [Russakovsky et al., ] datasets. We use the first three datasets to empirically validate our theoretical contribution, and leave the experiments on ImageNet to build a powerful ensemble adversarial attack in practice.

We build six deep neural networks for image classification, including three MLPs with one to three hidden layers followed by a linear classification layer, and three convolutional neural networks (CNNs) with one to three convolutional layers followed by a linear classification layer. To ensure diversity among the models, we apply three different types of transformations during training. Additionally, we set the weight decay under the $L_{2}$ norm to $10^{-4},10^{-3},10^{-2}$, respectively. This results in a total of $6\times 3\times 3=54$ models. To establish a gold standard for adversarial transferability evaluation, we additionally train a ResNet-18 [He et al., ] from scratch on three datasets (MNIST, Fashion-MNIST, and CIFAR-10), respectively. We will leverage the models at hand to attack this ResNet-18 for a reliable evaluation.

### 5.1 Evaluation on the Attack Dynamics

For each dataset (MNIST & Fashion-MNIST & CIFAR-10), we record the attack success rate (ASR), loss value, and the variance of model predictions.
The first metric exhibits an inverse relationship with transferability error.
And the latter two metrics correspond to the vulnerability and diversity components in the decomposition in Section 4.1.
We respectively report the results on three datasets in fig. 2, fig. 3, and fig. 4.

Figure: (a) MLP
Refer to caption: x2.png

Figure: (a) MLP
Refer to caption: x4.png

Figure: (a) MLP
Refer to caption: x6.png

#### Validation of vulnerability-diversity decomposition.

Across all three datasets, we observe a consistent pattern: as the number of steps increases, both ASR and loss values improve steadily, meaning that transferability error decreases while vulnerability increases.
Notably, the magnitude of variance is approximately ten times smaller than that of the loss value, indicating a much smaller impact on transferability error.
Thus, “vulnerability” predominantly drives the vulnerability-diversity decomposition, and the upward trend in vulnerability aligns with the reduction in transferability error.

#### The trend of variance.

On the MNIST and Fashion-MNIST datasets, diversity initially increases but later declines. In contrast, on the CIFAR dataset, the variance for MLP consistently increases, whereas for CNNs, it decreases with a small regularization term but increases with a larger one.
This intriguing phenomenon is tied not only to the trade-off between complexity and diversity discussed in Section 4.2, but also to the complex behavior of variance.
In the bias-variance trade-off literature, different trends in variance have been observed. For example, Yang et al. [] suggests that variance follows a bell-shaped curve, rising initially and then falling as network width expands. Similarly, Lin and Dobriban [] provides a detailed decomposition of variance, illustrating the influence of factors like initialization, label noise, and training data.
While a full investigation of variance behavior is beyond the scope of this work, we provide additional discussion in Appendix D.5.

### 5.2 Evaluation on the Ensemble Framework

We further validate the effectiveness of the vulnerability-diversity decomposition within the ensemble framework. Specifically, instead of focusing solely on the training dynamics, we progressively increase the number of models in the ensemble attack to evaluate the decomposition’s impact. We begin by incorporating MLPs with different architectures and regularization terms, followed by CNNs. In total, up to 18 models are included in a single attack.
We depicted the results in fig. 5.

Figure: (a) MNIST
Refer to caption: x8.png

We can consistently observe that increasing the number of ensemble models improves the attack success rate, i.e., reduces the transferability error.
On the MNIST and Fashion-MNIST datasets, both vulnerability and diversity also increase as the number of models grows.
Although the diversity sometimes shows a decreasing trend on the CIFAR-10 dataset,
its magnitude is approximately 100 times smaller than vulnerability, thus having a minimal impact on the attack success rate.

## 6 Conclusion

In this paper, we address the underdeveloped theoretical foundation of transferable model ensemble adversarial attacks.
We introduce three key definitions: transferability error, prediction variance, and empirical model ensemble Rademacher complexity.
Through the vulnerability-diversity decomposition of transferability error, we identify a crucial trade-off between vulnerability and diversity in ensemble components, presenting the challenge of optimizing both simultaneously.
To overcome this, we introduce recent mathematical tools and derive an upper bound on transferability error, offering practical guidelines for improving adversarial transferability.
Our extensive experiments validate these insights, marking a significant advancement in the understanding and development of transferable model ensemble adversarial attacks.

## Ethics Statement

We recognize the potential societal impact of our work on transferable adversarial attacks and emphasize its contribution to improving the robustness and security of machine learning models. We have been careful to ensure that our findings advance the public good by providing insights that can enhance defenses against malicious use of adversarial attacks, rather than contributing to harmful applications.
Our study upholds high standards of scientific excellence through transparency, rigor, and reproducibility. No human subjects were involved, and no privacy or confidentiality concerns arise from the data used. We have also ensured that our work does not introduce discriminatory biases, and we are committed to the fair and inclusive participation of all individuals in the research community.
While our research focuses on theoretical advancements, we are aware of the potential risks associated with adversarial attack techniques. We encourage responsible use of these insights to build more secure AI systems and minimize any unintended harm. Finally, any potential conflicts of interest have been disclosed.

## Appendix

## Appendix A Proof of Generalized Rademacher Complexity

### A.1 Preliminary

For simplicity, denote $f(\theta_{i};x)$ as $f_{i}(x)$.
For $1$-Lipschitz loss function $\ell(yf(x))$ (for example, hinge loss $\ell(f(x),y)=\max{(0,1-yf(x)}$), there holds:

$$ $\displaystyle\mathcal{R}_{N}(\mathcal{Z})$ $\displaystyle=\mathop{\mathbb{E}}\limits_{\boldsymbol{\sigma}}\left[\sup_{z\in \mathcal{Z}}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}\ell(f_{i}(x),y)\right]$ $\displaystyle\leq\mathop{\mathbb{E}}\limits_{\boldsymbol{\sigma}}\left[\sup_{z \in\mathcal{Z}}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}yf_{i}(x)\right]$ $\displaystyle=\mathop{\mathbb{E}}\limits_{\boldsymbol{\sigma}}\left[\sup_{z\in \mathcal{Z}}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}f_{i}(x)\right]:=\Re_{N}( \mathcal{Z}).$ $$

So we can bound $\Re_{N}(\mathcal{Z})$ instead of $\mathcal{R}_{N}(\mathcal{Z})$.

### A.2 Linear Model

Given Section A.1, we provide the bound below.

###### Lemma 3 (Linear Model) .

Let $\mathcal{H}=\left\{x\mapsto w^{T}x\right\}$, where $x,w\in\mathbb{R}^{d}$.
Given $N$ classifiers from $\mathcal{H}$,
assume that $\|x\|_{2}\leq B$ and $\|w\|_{2}\leq C$. Then

$$ $\Re_{N}(\mathcal{Z})\leq\frac{BC}{\sqrt{N}}.$ $$

###### Proof.

We have

$$ $\displaystyle\Re_{N}(\mathcal{Z})$ $\displaystyle=\mathop{\mathbb{E}}\limits_{\boldsymbol{\sigma}}\left[\sup_{\|x \|_{2}\leq B}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}f_{i}(x)\right]$ $\displaystyle=\mathop{\mathbb{E}}\limits_{\boldsymbol{\sigma}}\left[\sup_{\|x \|_{2}\leq B}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}w_{i}^{T}x\right]$ ( $f_{i}(x)=w_{i}^{T}x$ ) $\displaystyle=\underset{\boldsymbol{\sigma}}{\mathbb{E}}\left[\sup_{\|x\|_{2} \leq B}x^{T}\left(\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}w_{i}\right)\right]$ ( $a^{T}b=b^{T}a$ ) $\displaystyle=\frac{B}{N}\underset{\boldsymbol{\sigma}}{\mathbb{E}}\left\|\sum _{i=1}^{N}\sigma_{i}w_{i}\right\|_{2}$ ( $a^{T}b\leq\|a\|_{2}\|b\|_{2}$ ) $\displaystyle\leq\frac{B}{N}\left(\underset{\boldsymbol{\sigma}}{\mathbb{E}} \left\|\sum_{i=1}^{N}\sigma_{i}w_{i}\right\|_{2}^{2}\right)^{\frac{1}{2}}$ (Jensen inequality: $\mathbb{E}x\leq\sqrt{\mathbb{E}x^{2}}$ ) $\displaystyle=\frac{B}{N}\left\{\underset{\boldsymbol{\sigma}}{\mathbb{E}} \left[\left(\sum_{i=1}^{N}\sigma_{i}w_{i}^{T}\right)\left(\sum_{i=1}^{N}\sigma _{i}w_{i}\right)\right]\right\}^{\frac{1}{2}}$ $\displaystyle=\frac{B}{N}\left[\underset{\boldsymbol{\sigma}}{\mathbb{E}}\left (\sum_{i=1}^{N}\underbrace{\sigma_{i}^{2}}_{1}w_{i}^{T}w_{i}+\underbrace{\sum_ {i=1}^{N}\sum_{j=1,j\neq i}^{N}\sigma_{i}\sigma_{j}w_{i}^{T}w_{j}}_{0}\right) \right]^{\frac{1}{2}}$ $\displaystyle=\frac{B}{N}\left(\sum_{i=1}^{N}w_{i}^{T}w_{i}\right)^{\frac{1}{2}}$ $\displaystyle\leq\frac{B}{N}\left(N\max{\left\|w\right\|_{2}^{2}}\right)^{ \frac{1}{2}}$ $\displaystyle\leq\frac{BC}{\sqrt{N}}.$ ( $\left\|w\right\|_{2}\leq C$ ) $$

The proof is complete.

∎

###### Lemma 3 (Linear Model) .

###### Proof.

### A.3 Two-layer Neural Network

Given Section A.1, we provide the bound below.

###### Lemma 4 (Two-layer Neural Network) .

Let $\mathcal{H}=\{x\mapsto w^{T}\phi(Ux)\}$, where $x\in\mathbb{R}^{d}$, $U\in\mathbb{R}^{m\times d}$, $w\in\mathbb{R}^{m}$, $m$ is the number of the hidden layer, and $\phi(x)=\max{(0,x)}$ is the element-wise ReLU function.
Given $N$ classifiers from $\mathcal{H}$,
assume that $\|x\|_{2}\leq B$, $\|w\|_{2}\leq B^{\prime}$, and $\|U_{i}\|_{2}\leq C$, where $U_{j}$ is the j-th row of $U$. Then

$$ $\Re_{N}(\mathcal{Z})\leq\frac{\sqrt{m}BB^{\prime}C}{\sqrt{N}}.$ $$

###### Proof.

We have

$$ $\displaystyle\Re_{N}(\mathcal{Z})$ $\displaystyle=\mathop{\mathbb{E}}\limits_{\boldsymbol{\sigma}}\left[\sup_{\|x \|_{2}\leq B}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}f_{i}(x)\right]$ $\displaystyle=\mathop{\mathbb{E}}\limits_{\boldsymbol{\sigma}}\left[\sup_{\|x \|_{2}\leq B}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}w_{i}^{T}\phi(U_{i}x)\right]$ ( $f_{i}(x)=w_{i}^{T}\phi(U_{i}x)$ ) $\displaystyle=\frac{B^{\prime}}{N}\mathop{\mathbb{E}}\limits_{\boldsymbol{ \sigma}}\left[\sup_{\|x\|_{2}\leq B}\left\|\sum_{i=1}^{N}\sigma_{i}\phi(U_{i}x )\right\|_{2}\right]$ ( $\|w\|_{2}\leq B^{\prime}$ ) $\displaystyle=\frac{B^{\prime}}{N}\mathop{\mathbb{E}}\limits_{\boldsymbol{ \sigma}}\left[\sup_{\|x\|_{2}\leq B}\left\|\sum_{i=1}^{N}\sigma_{i}V_{i}\right \|_{2}\right]$ (Denote $V_{i}=\begin{bmatrix}\phi(U_{1i}x)\\ \vdots\\ \phi(U_{mi}x)\end{bmatrix}\in\mathbb{R}^{m}$ ) $\displaystyle=\frac{B^{\prime}}{N}\mathop{\mathbb{E}}\limits_{\boldsymbol{ \sigma}}\left[\sup_{\|x\|_{2}\leq B}\sqrt{\left(\sum_{i=1}^{N}\sigma_{i}V_{i}^ {T}\right)\left(\sum_{i=1}^{N}\sigma_{i}V_{i}\right)}\right]$ $\displaystyle=\frac{B^{\prime}}{N}\mathop{\mathbb{E}}\limits_{\boldsymbol{ \sigma}}\left[\sup_{\|x\|_{2}\leq B}\left(\sum_{i=1}^{N}\underbrace{\sigma_{i} ^{2}}_{1}V_{i}^{T}V_{i}+\underbrace{\sum_{i=1}^{N}\sum_{j=1,j\neq i}^{N}\sigma _{i}\sigma_{j}V_{i}^{T}V_{j}}_{0}\right)^{\frac{1}{2}}\right]$ $\displaystyle=\frac{B^{\prime}}{N}\sup_{\|x\|_{2}\leq B}\left(\sum_{i=1}^{N}V_ {i}^{T}V_{i}\right)^{\frac{1}{2}}$ $\displaystyle\leq\frac{B^{\prime}}{N}\sup_{\|x\|_{2}\leq B}\left(N\max_{i}{ \left\|V_{i}\right\|_{2}^{2}}\right)^{\frac{1}{2}}$ $\displaystyle\leq\frac{B^{\prime}}{\sqrt{N}}\sup_{\|x\|_{2}\leq B}\left(\max_{ i}{\|V_{i}\|_{2}}\right)$ $$

For $V_{i}=\begin{bmatrix}\phi(U_{1i}x)\\
\vdots\\
\phi(U_{mi}x)\end{bmatrix}\in\mathbb{R}^{m}$, we have

$$ $\displaystyle\sup_{\|x\|_{2}\leq B}\left(\max_{i}{\|V_{i}\|_{2}}\right)$ $\displaystyle=\sup_{\|x\|_{2}\leq B}\left(\max_{i}{\left\|\begin{bmatrix}\phi( U_{1i}x)\\ \vdots\\ \phi(U_{mi}x)\end{bmatrix}\right\|_{2}}\right)$ $\displaystyle\leq\sup_{\|x\|_{2}\leq B}\left(\max_{i}{\left\|\begin{bmatrix}U_ {1i}x\\ \vdots\\ U_{mi}x\end{bmatrix}\right\|_{2}}\right)$ ( $|\phi(x)|\leq|x|$ ) $\displaystyle=\sqrt{m}\sup_{\|x\|_{2}\leq B}\left(\max_{i}{\max_{j}{\left\|U_{ ji}x\right\|_{2}}}\right)$ $\displaystyle\leq\sqrt{m}\sup_{\|x\|_{2}\leq B}\left(\max_{i}{\max_{j}{\left\| U_{ji}\right\|_{2}\left\|x\right\|_{2}}}\right)$ $\displaystyle=\sqrt{m}BC$ ( $\|x\|_{2}\leq B$ and $\|U_{ji}\|_{2}\leq C$ ) $$

Finally,

$$ $\Re_{N}(\mathcal{Z})\leq\frac{B^{\prime}}{\sqrt{N}}\sup_{\|x\|_{2}\leq B}\left (\max_{i}{\|V_{i}\|_{2}}\right)\leq\frac{\sqrt{m}BB^{\prime}C}{\sqrt{N}}$ $$

The proof is complete.

∎

###### Lemma 4 (Two-layer Neural Network) .

###### Proof.

### A.4 Proof of Lemma 2

For simplicity, denote $f(\theta_{i};x)$ as $f_{i}(x)$ and $i\in\{1,\cdots,N\}$ as $i\in[N]$.

First, we begin with a lemma, which is a similar version of Lemma 1 from [Golowich et al., ].

###### Lemma 5 .

Let $\phi$ be a 1-Lipschitz, positive-homogeneous activation function which is applied element-wise (such as the ReLU). Then for any class of vector-valued functions $\mathcal{F}$ and any convex and monotonically increasing function $g:\mathbb{R}\rightarrow[0,\infty)$, there holds:

$$ $\mathbb{E}_{\boldsymbol{\sigma}}\sup_{f\in\mathcal{F},W:\|W\|_{F}\leq R}g\left (\left\|\sum_{i=1}^{N}\sigma_{i}\phi\left(Wf_{i}\left(x\right)\right)\right\| \right)\leq 2\cdot\mathbb{E}_{\boldsymbol{\sigma}}\sup_{f\in\mathcal{F}}g\left (R\cdot\left\|\sum_{i=1}^{N}\sigma_{i}f_{i}\left(x\right)\right\|\right)$ (13) $$

###### Proof.

Let $w_{1},\cdots,w_{h}$ be the rows of $W$, we have

$$ $\displaystyle\left\|\sum_{i=1}^{N}\sigma_{i}\phi\left(Wf_{i}\left(x\right) \right)\right\|^{2}=$ $\displaystyle\sum_{j=1}^{h}\left[\sum_{i=1}^{N}\sigma_{i}\phi(w_{j}f_{i}(x)) \right]^{2}$ $\displaystyle=$ $\displaystyle\sum_{j=1}^{h}\left\|w_{j}\right\|^{2}\left[\sum_{i=1}^{N}\sigma_ {i}\phi\left(\frac{w_{j}^{\top}}{\left\|w_{j}\right\|}f_{i}\left(x\right) \right)\right]^{2}$ ( $\phi(ax)=a\phi(x)$ ) $$

Therefore, the supremum of this over all $w_{1},\cdots,w_{h}$ such that $\|W\|_{F}^{2}=\sum_{j=1}^{h}\left\|w_{j}\right\|^{2}\leq R^{2}$ must be attained when $\left\|w_{j}\right\|=R$ for some $j$ and $\left\|w_{i}\right\|=0$ for all $i\neq j$.
So we have

$$ $\mathbb{E}_{\boldsymbol{\sigma}}\sup_{f\in\mathcal{F},W:\|W\|_{F}\leq R}g\left (\left\|\sum_{i=1}^{N}\sigma_{i}\phi\left(Wf_{i}\left(x\right)\right)\right\| \right)=\mathbb{E}_{\boldsymbol{\sigma}}\sup_{f\in\mathcal{F},w:\|w\|=R}g\left (\left|\sum_{i=1}^{N}\sigma_{i}\phi\left(w^{\top}f_{i}\left(x\right)\right) \right|\right).$ $$

Since $g(|z|)\leq g(z)+g(-z)$, this can be upper bounded by

$$ $\mathbb{E}_{\boldsymbol{\sigma}}\sup g\left(\sum_{i=1}^{N}\sigma_{i}\phi\left( w^{\top}f_{i}\left(x\right)\right)\right)+\mathbb{E}_{\boldsymbol{\sigma}}\sup g \left(-\sum_{i=1}^{N}\sigma_{i}\phi\left(w^{\top}f_{i}\left(x\right)\right) \right)\\ =2\cdot\mathbb{E}_{\boldsymbol{\sigma}}\sup g\left(\sum_{i=1}^{N}\sigma_{i} \phi\left(w^{\top}f_{i}\left(x\right)\right)\right),$ (14) $$

where the equality follows from the symmetry in the distribution of the $\sigma_{i}$ random variables. The right hand side in turn can be upper bounded by

$$ $\displaystyle 2\cdot\mathbb{E}_{\boldsymbol{\sigma}}\sup_{f\in\mathcal{F},w:\| w\|=R}g\left(\sum_{i=1}^{N}\sigma_{i}w^{\top}f_{i}\left(x\right)\right)$ $\displaystyle\leq 2\cdot\mathbb{E}_{\boldsymbol{\sigma}}\sup_{f\in\mathcal{F}, w:\|w\|=R}g\left(\|w\|\left\|\sum_{i=1}^{N}\sigma_{i}f_{i}\left(x\right)\right \|\right)$ $\displaystyle=2\cdot\mathbb{E}_{\boldsymbol{\sigma}}\sup_{f\in\mathcal{F}}g \left(R\cdot\left\|\sum_{i=1}^{N}\sigma_{i}f_{i}\left(x\right)\right\|\right).$ $$

∎

With this lemma in hand, we can prove lemma 2:

###### Proof.

For $\lambda>0$, the rademacher complexity can be upper bounded as

$$ $\displaystyle N\Re_{N}\left(\mathcal{Z}\right)$ $\displaystyle=\mathbb{E}_{\boldsymbol{\sigma}}\sup_{f_{1},\cdots,f_{n}}\sum_{i =1}^{N}\sigma_{i}f_{i}(x)$ $\displaystyle\leq\frac{1}{\lambda}\log\mathbb{E}_{\boldsymbol{\sigma}}\sup\exp \left(\lambda\sum_{i=1}^{N}\sigma_{i}f_{i}(x)\right)$ (Jensen’s inequality) $\displaystyle\leq\frac{1}{\lambda}\log\mathbb{E}_{\boldsymbol{\sigma}}\sup\exp \left(\underbrace{\sup_{i\in[n]}\|W_{i,l}\|_{F}}_{T_{l}}\left\|\lambda\sum_{i= 1}^{N}\sigma_{i}\phi_{l-1}\underbrace{\left(W_{i,l-1}\phi_{l-2}\left(\ldots \phi_{1}\left(W_{i,1}x\right)\right)\right)}_{f_{i,l-1}(x)}\right\|\right)$ $$

We write this last expression as

$$ $\displaystyle\frac{1}{\lambda}\log\mathbb{E}_{\boldsymbol{\sigma}}\sup\exp \left(T_{l}\cdot\lambda\left\|\sum_{i=1}^{N}\sigma_{i}\phi_{l-1}\left(f_{i,l-1 }(x)\right)\right\|\right)$ $\displaystyle\leq$ $\displaystyle\frac{1}{\lambda}\log\left(2\cdot\mathbb{E}_{\boldsymbol{\sigma}} \sup\exp\left(T_{l}\cdot T_{l-1}\cdot\lambda\left\|\sum_{i=1}^{N}\sigma_{i}f_{ i,l-2}\left(x\right)\right\|\right)\right)$ (Lemma 5 ) $\displaystyle\leq$ $\displaystyle\cdots$ (Repeatedly apply Lemma 5 ) $\displaystyle\leq$ $\displaystyle\frac{1}{\lambda}\log\left(2^{l-2}\cdot\mathbb{E}_{\boldsymbol{ \sigma}}\sup\exp\left(\lambda\cdot\prod_{i=1}^{l-1}T_{i}\cdot\left\|\sum_{i=1} ^{N}\sigma_{i}\phi_{1}(W_{i,1}x)\right\|\right)\right)$ $\displaystyle\leq$ $\displaystyle\frac{1}{\lambda}\log\left(2^{l-1}\cdot\mathbb{E}_{\boldsymbol{ \sigma}}\sup\exp\left(\lambda\cdot\prod_{i=1}^{l-1}T_{i}\cdot\left\|\sum_{i=1} ^{N}\sigma_{i}W_{i,1}x\right\|\right)\right)$ $$

Assume that $W_{i,1}^{*},i\in[N]$ maximizes

$$ $\sup\exp\left(\lambda\cdot\prod_{i=1}^{l-1}T_{i}\cdot\left\|\sum_{i=1}^{N} \sigma_{i}W_{i,1}x\right\|\right).$ $$

Therefore,

$$ $\displaystyle\frac{1}{\lambda}\log\left(2^{l-1}\cdot\mathbb{E}_{\boldsymbol{ \sigma}}\sup\exp\left(\lambda\cdot\prod_{i=1}^{l-1}T_{i}\cdot\left\|\sum_{i=1} ^{N}\sigma_{i}W_{i,1}x\right\|\right)\right)$ $\displaystyle=$ $\displaystyle\frac{1}{\lambda}\log\left(2^{l-1}\cdot\mathbb{E}_{\boldsymbol{ \sigma}}\exp\left(\lambda\cdot\underbrace{\prod_{i=1}^{l-1}T_{i}\cdot\left\| \sum_{i=1}^{N}\sigma_{i}W_{i,1}^{*}x\right\|}_{Z}\right)\right)$ $\displaystyle=$ $\displaystyle\frac{1}{\lambda}\log\left(2^{l-1}\cdot\mathbb{E}_{\boldsymbol{ \sigma}}\exp\left(\lambda Z\right)\right)$ $\displaystyle=$ $\displaystyle\frac{(l-1)\log(2)}{\lambda}+\frac{1}{\lambda}\log\left\{\mathbb{ E}_{\boldsymbol{\sigma}}\exp\left(\lambda Z\right)\right\}$ $\displaystyle=$ $\displaystyle\frac{(l-1)\log(2)}{\lambda}+\frac{1}{\lambda}\log\{\mathbb{E} \exp\lambda(Z-\mathbb{E}Z)\}+\mathbb{E}Z$ $$

For $\mathbb{E}Z$, we have

$$ $\displaystyle\mathbb{E}Z=$ $\displaystyle\prod_{i=1}^{l-1}T_{i}\sqrt{\mathbb{E}_{\boldsymbol{\sigma}}\left [\left\|\sum_{i=1}^{N}\sigma_{i}W_{i,1}^{*}x\right\|^{2}\right]}$ $\displaystyle=$ $\displaystyle\prod_{i=1}^{l-1}T_{i}\sqrt{\mathbb{E}_{\boldsymbol{\sigma}}\left [\sum_{i=j}^{N}\sigma_{i}\sigma_{j}\left(W_{i,1}^{*}x\right)^{T}\left(W_{j,1}^ {*}x\right)\right]}$ $\displaystyle\leq$ $\displaystyle\prod_{i=1}^{l-1}T_{i}\left(T_{1}B\sqrt{N}\right)$ $\displaystyle=$ $\displaystyle B\sqrt{N}\prod_{i=1}^{l}T_{i}$ $$

Note that $Z$ is a deterministic function of the i.i.d. random variables $\sigma_{1},\cdots,\sigma_{N}$, and satisfies

$$ $Z(\sigma_{1},\cdots,\sigma_{i},\cdots,\sigma_{N})-Z(\sigma_{1},\cdots,-\sigma_ {i},\cdots,\sigma_{N})\leq 2B\underbrace{\prod_{i=1}^{l}T_{i}}_{T}.$ $$

This means that $Z$ satisfies a bounded-difference condition. According to Theorem 6.2 in Boucheron et al. [], $Z$ is sub-Gaussian with variance factor

$$ $\frac{1}{4}\sum_{i=1}^{N}(2BT)^{2}=NB^{2}T^{2},$ $$

and satisfies

$$ $\frac{1}{\lambda}\log\{\mathbb{E}\exp\lambda(Z-\mathbb{E}Z)\}\leq\frac{1}{ \lambda}\cdot\frac{\lambda^{2}}{2}NB^{2}T^{2}=\frac{\lambda}{2}NB^{2}T^{2}.$ $$

Choosing $\lambda=\frac{\sqrt{2\log(2)l}}{BT\sqrt{N}}$ and using the above, we get that

$$ $\frac{(l-1)\log(2)}{\lambda}+\frac{1}{\lambda}\log\{\mathbb{E}\exp\lambda(Z- \mathbb{E}Z)\}+\mathbb{E}Z\leq\left(\sqrt{(2\log 2)l}+1\right)BT\sqrt{N}$ $$

Finally, we get

$$ $\Re_{N}\left(\mathcal{Z}\right)\leq\frac{\left(\sqrt{(2\log 2)l}+1\right)BT}{ \sqrt{N}}$ $$

The proof is complete.

∎

###### Lemma 5 .

###### Proof.

###### Proof.

## Appendix B Proof of Transferability Error

### B.1 Transferability Error and Generalization Error

For $z=(x,y)$, there holds

$$ $\displaystyle TE(z)=L_{P}(z^{*})-L_{P}(z)$ $\displaystyle\leq L_{P}(z^{*})-L_{P}(z)+(L_{E}(z)-L_{E}(z^{*}))$ $\displaystyle=(L_{P}(z^{*})-L_{E}(z^{*}))+(L_{E}(z)-L_{P}(z))$ $\displaystyle\leq\sup_{x\in\mathcal{B}_{\epsilon}(x)}(L_{P}(z)-L_{E}(z))+\sup_ {x\in\mathcal{B}_{\epsilon}(x)}(L_{E}(z)-L_{P}(z))$ $\displaystyle\leq\sup_{z\in\mathcal{Z}}(L_{P}(z)-L_{E}(z))+\sup_{z\in\mathcal{ Z}}(L_{E}(z)-L_{P}(z)).$ $\displaystyle\leq 2\sup_{z\in\mathcal{Z}}\left|L_{P}(z)-L_{E}(z)\right|.$ $$

### B.2 Proof of Theorem 1

We prove a general version of the theorem as follows:

###### Theorem 3 .

Consider the squared error loss $l(\theta,x,y)=\left[f(\theta;x)-y\right]^{2}$ for a data point $z=(x,y)$.
Assume that the data is generated by a function $g(x)$ such that $y=g(x)+\rho$, where the zero-mean noise $\rho$ has a variance of $\eta^{2}$ and is independent of $x$.
Then there holds

$$ $\displaystyle TE(z,\epsilon)=L_{P}(z^{*})-\eta^{2}-\underbrace{\textit{Var}_{ \theta\sim\mathcal{P}_{\Theta}}f(\theta;x)}_{\text{Diversity}}-\underbrace{ \left[g(x)-\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}f(\theta;x)\right]^{2}}_ {\text{Attack}}.$ (15) $$

###### Remark .

The irreducible error $\eta^{2}$ is constant because it arises from inherent noise and randomness in the data [Geman et al., ].

Now we start our proof of it.

###### Proof.

Given Eq. (5), it is equivalent to prove

$$ $L_{P}(z)=\textit{Var}_{\theta}f(\theta;x)+\left[g(x)-\mathbb{E}_{\theta\sim \mathcal{P}_{\Theta}}f(\theta;x)\right]^{2}+\eta^{2}.$ (16) $$

Note that

$$ $\displaystyle L_{P}(z)$ $\displaystyle=\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}\left[f(\theta;x)-y \right]^{2}$ $\displaystyle=\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}\left[f(\theta;x)-g(x )+g(x)-y\right]^{2}$ $\displaystyle=\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}\left[(f(\theta;x)-g( x))^{2}+(g(x)-y)^{2}+2(g(x)-y)(f(\theta;x)-g(x))\right].$ $$

Recall that $y=g(x)+\rho$ with $\mathbb{E}(\rho)=0$ and $\textit{Var}(\rho)=\eta^{2}$, we have

$$ $\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}(g(x)-y)^{2}=\eta^{2},$ $$

and

$$ $\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}\left[2(g(x)-y)(f(\theta;x)-g(x)) \right]=-2\mathbb{E}(\rho)\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}\left[f( \theta;x)-g(x)\right]=0.$ $$

Therefore,

$$ $L_{P}(z)=\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}\left[f(\theta;x)-g(x) \right]^{2}+\eta^{2}.$ (17) $$

Likewise, we decompose the first term as

$$ $\displaystyle\mathbb{E}_{\theta}\left[f(\theta;x)-g(x)\right]^{2}$ $\displaystyle=$ $\displaystyle\mathbb{E}_{\theta}\left[f(\theta;x)-\mathbb{E}_{\theta}f(\theta; x)+\mathbb{E}_{\theta}f(\theta;x)-g(x)\right]^{2}$ $\displaystyle=$ $\displaystyle\mathbb{E}_{\theta}\left[(f(\theta;x)-\mathbb{E}_{\theta}f(\theta ;x))^{2}+(\mathbb{E}_{\theta}f(\theta;x)-g(x))^{2}\right.$ $\displaystyle-\left.2(f(\theta;x)-\mathbb{E}_{\theta}f(\theta;x))(\mathbb{E}_{ \theta}f(\theta;x)-g(x))\right]$ $\displaystyle=$ $\displaystyle\underbrace{\mathbb{E}_{\theta}(f(\theta;x)-\mathbb{E}_{\theta}f( \theta;x))^{2}}_{\textit{Var}_{\theta}f(\theta;x)}+\underbrace{\mathbb{E}_{ \theta}(\mathbb{E}_{\theta}f(\theta;x)-g(x))^{2}}_{\left(g(x)-\mathbb{E}_{ \theta}(f(\theta;x)\right)^{2}}$ $\displaystyle-2\underbrace{\mathbb{E}_{\theta}\left[f(\theta;x)-\mathbb{E}_{ \theta}f(\theta;x))(\mathbb{E}_{\theta}f(\theta;x)-g(x)\right]}_{0},$ $$

with the derivations for the second and third term:

$$ $\displaystyle\mathbb{E}_{\theta}(f(\theta;x)-\mathbb{E}_{\theta}f(\theta;x))^{2}$ $\displaystyle=\left(\mathbb{E}_{\theta}f(\theta;x)\right)^{2}-2g(x)\mathbb{E}_ {\theta}f(\theta;x)+g^{2}(x)$ $\displaystyle=\left(g(x)-\mathbb{E}_{\theta}(f(\theta;x)\right)^{2},$ $$

and

$$ $\displaystyle\mathbb{E}_{\theta}\left[f(\theta;x)-\mathbb{E}_{\theta}f(\theta; x))(\mathbb{E}_{\theta}f(\theta;x)-g(x)\right]$ $\displaystyle=$ $\displaystyle(\mathbb{E}_{\theta}f(\theta;x))^{2}-g(x)\mathbb{E}_{\theta}f( \theta;x)-(\mathbb{E}_{\theta}f(\theta;x))^{2}+g(x)\mathbb{E}_{\theta}f(\theta ;x)$ $\displaystyle=$ $\displaystyle 0.$ $$

As a result,

$$ $\mathbb{E}_{\theta}\left[f(\theta;x)-g(x)\right]^{2}=\textit{Var}_{\theta}f( \theta;x)+\left[g(x)-\mathbb{E}_{\theta\sim\mathcal{P}_{\Theta}}f(\theta;x) \right]^{2}.$ (18) $$

Combining the above results and we complete the proof.

∎

To prove Theorem 1, we just set $\rho=0$ in the above general version of theorem.

Similarly,
consider the empirical version of Theorem 1,
we decompose $L_{E}(z)$ as follows:

###### Theorem 4 (Vulnerability-diversity Decomposition (empirical version)) .

Consider the squared error loss $l(f(\theta;x),y)=\left[f(\theta;x)-y\right]^{2}$ for a data point $z=(x,y)$.
Let $\hat{f}(\theta;x)=\frac{1}{N}\sum_{i=1}^{N}f(\theta_{i};x)$ be the expectation of prediction over the distribution on the parameter space.
Then there holds

$$ $\displaystyle L_{E}(z)$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\ell(f(\theta_{i};x),y)$ $\displaystyle=\underbrace{l(\hat{f}(\theta;x),y)}_{\text{Vulnerability}}+ \underbrace{\frac{1}{N}\sum_{j=1}^{N}\bigg{(}f(\theta_{i};x)-\frac{1}{N}\sum_{ j=1}^{N}f(\theta_{i};x)\bigg{)}^{2}}_{\text{Diversity}}.$ $$

The proof is similar to the above:

$$ $\displaystyle L_{E}(z)$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\left(f(\theta_{i};x)-y\right)^{2}$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\left(f(\theta_{i};x)-\sum_{i=1}^{N}f( \theta_{i};x)+\sum_{i=1}^{N}f(\theta_{i};x)-y\right)^{2}$ $\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\Bigg{[}\left(f(\theta_{i};x)-\sum_{i=1 }^{N}f(\theta_{i};x)\right)^{2}+\left(\sum_{i=1}^{N}f(\theta_{i};x)-y\right)^{ 2}+$ $\displaystyle 2\left(f(\theta_{i};x)-\sum_{i=1}^{N}f(\theta_{i};x)\right)\left (\sum_{i=1}^{N}f(\theta_{i};x)-y\right)\Bigg{]}$ $\displaystyle=\underbrace{l(\hat{f}(\theta;x),y)}_{\text{Vulnerability}}+ \underbrace{\frac{1}{N}\sum_{j=1}^{N}\bigg{(}f(\theta_{i};x)-\frac{1}{N}\sum_{ j=1}^{N}f(\theta_{i};x)\bigg{)}^{2}}_{\text{Diversity}}+$ $\displaystyle\frac{2}{N}\sum_{i=1}^{N}\left(f(\theta_{i};x)-\frac{1}{N}\sum_{i =1}^{N}f(\theta_{i};x)\right)\left(\frac{1}{N}\sum_{i=1}^{N}f(\theta_{i};x)-y \right).$ $$

The last terms equals to 0 0 because

$$ $\displaystyle\sum_{i=1}^{N}\left(f(\theta_{i};x)-\frac{1}{N}\sum_{i=1}^{N}f( \theta_{i};x)\right)\left(\frac{1}{N}\sum_{i=1}^{N}f(\theta_{i};x)-y\right)$ $\displaystyle=$ $\displaystyle\frac{1}{N}\left(\sum_{i=1}^{N}f(\theta_{i};x)\right)^{2}-y\sum_{ i=1}^{N}f(\theta_{i};x)-\frac{1}{N}\left(\sum_{i=1}^{N}f(\theta_{i};x)\right)^ {2}+y\sum_{i=1}^{N}f(\theta_{i};x)$ $\displaystyle=$ $\displaystyle 0.$ $$

The proof is complete.

###### Theorem 3 .

###### Remark .

###### Proof.

###### Theorem 4 (Vulnerability-diversity Decomposition (empirical version)) .

### B.3 Proof of Theorem 2

We first define a divergence measure taken into account.
Given a measurable space and two measures $\mu,\nu$ which render it a measure space,
we denote $\nu\ll\mu$ if $\nu$ is absolutely continuous with respect to $\mu$. Hellinger integrals are defined below:

###### Definition 4 (Hellinger integrals [Hellinger, 1909 ] ) .

Let $\nu,\mu$ be two probability measures on $(\Omega,\mathcal{F})$ and satisfy $\nu\ll\mu$, and $\varphi_{\alpha}:\mathbb{R}^{+}\rightarrow\mathbb{R}$ be defined as $\varphi_{\alpha}(x)=x^{\alpha}$. Then the Hellinger integral of order $\alpha$ is given by

$$ $H_{\alpha}(\nu\|\mu)=\int\left(\frac{d\nu}{d\mu}\right)^{\alpha}\mathrm{d}\mu.$ $$

It can be seen as a $\phi$-Divergence with a specific parametrised choice of $\phi$ [Liese and Vajda, ].
For $\alpha>1$, the Hellinger integral measures the divergence between two probability distributions [Liese and Vajda, ].
There holds $H_{\alpha}(\nu\|\mu)\in[1,+\infty),\alpha>1$, and it equals to $1$ if the two measures coincide [Shiryaev, ].
Given such a divergence measure, we now provide the proof.

###### Proof.

From Section B.1, we know that

$$ $\displaystyle TE(z)=L_{P}(z^{*})-L_{P}(z)$ $\displaystyle\leq L_{P}(z^{*})-L_{P}(z)+(L_{E}(z)-L_{E}(z^{*}))$ $\displaystyle=(L_{P}(z^{*})-L_{E}(z^{*}))+(L_{E}(z)-L_{P}(z))$ $\displaystyle\leq\sup_{x\in\mathcal{B}_{\epsilon}(x)}(L_{P}(z)-L_{E}(z))+\sup_ {x\in\mathcal{B}_{\epsilon}(x)}(L_{E}(z)-L_{P}(z))$ $\displaystyle\leq\sup_{z\in\mathcal{Z}}(L_{P}(z)-L_{E}(z))+\sup_{z\in\mathcal{ Z}}(L_{E}(z)-L_{P}(z)).$ $$

Let $(\theta^{\prime}_{1},\ldots,\theta^{\prime}_{N})\sim\mathcal{P}^{\prime}_{
\Theta^{N}}$, where $\mathcal{P}^{\prime}_{\Theta^{N}}$ be a distribution over the product space, and the m-th member is different from $\mathcal{P}_{\Theta^{N}}$, i.e., $(\theta^{\prime}_{1},\ldots,\theta^{\prime}_{m},\cdots,\theta^{\prime}_{N})=(
\theta_{1},\ldots,\theta^{\prime}_{m},\cdots,\theta_{N})$, where $\theta^{\prime}_{m}\neq\theta_{m}$.
The training process of $N$ surrogate models $f(\theta^{\prime}_{1}),\cdots,f(\theta^{\prime}_{N})$ can be viewed as sampling the parameter sets $(\theta^{\prime}_{1},\ldots,\theta^{\prime}_{N})$ from the distribution $\mathcal{P}^{\prime}_{\Theta^{N}}$.

We define

$$ $L_{E^{\prime}}(z)=\frac{1}{N}\sum_{i=1}^{N}\ell(f(\theta^{\prime}_{i};x),y),$ $$

and

$$ $\displaystyle\Phi_{1}(E)=\sup_{z\in\mathcal{Z}}\left\{L_{P}(z)-L_{E}(z)\right\},$ $\displaystyle\Phi_{1}(E^{\prime})=\sup_{z\in\mathcal{Z}}\left\{L_{P}(z)-L_{E^{ \prime}}(z)\right\}.$ $$

We have

$$ $\displaystyle\Phi_{1}(E)-\Phi_{1}(E^{\prime})$ $\displaystyle=\sup_{z\in\mathcal{Z}}\left\{L_{P}(z)-L_{E}(z)\right\}-\sup_{z \in\mathcal{Z}}\left\{L_{P}(z)-L_{E^{\prime}}(z)\right\}$ $\displaystyle\leq\sup_{z\in\mathcal{Z}}\left\{L_{P}(z)-L_{E}(z)-(L_{P}(z)-L_{E ^{\prime}}(z))\right\}$ $\displaystyle=\sup_{z\in\mathcal{Z}}\left\{L_{E^{\prime}}(z)-L_{E}(z)\right\}$ $\displaystyle=\frac{1}{N}\sup_{z\in\mathcal{Z}}\left[\sum_{i=1}^{N}\ell(f( \theta^{\prime}_{i};x),y)-\sum_{i=1}^{N}\ell(f(\theta_{i};x),y)\right].$ $$

By assuming that loss function $\ell$ is bounded by $\beta$, we have

$$ $\left|\Phi_{1}(E)-\Phi_{1}(E^{\prime})\right|\leq\frac{\beta}{N}.$ $$

According to Theorem 1 in Esposito and Mondelli [], for all $\delta\in(0,1)$ and $\alpha>1$, with probability at least $1-\delta$, we have

$$ $\Phi_{1}(E)\leq\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}[\Phi_{1}(E)]+\sqrt{\frac{ \alpha\beta^{2}}{2(\alpha-1)N}\ln{\frac{2^{\frac{\alpha-1}{\alpha}}H_{\alpha}^ {\frac{1}{\alpha}}\left(\mathcal{P}_{\Theta^{N}}\|\mathcal{P}_{\bigotimes_{i=1 }^{N}\Theta}\right)}{\delta}}}.$ (19) $$

Denote $f(\theta_{i};x)$ as $f_{i}(x)$ and $f(\theta^{\prime}_{i};x)$ as $f_{i}^{\prime}(x)$.
Then we estimate the upper bound of $\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}[\Phi_{1}(E)]$ as follows:

$$ $\displaystyle\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}[\Phi_{1}(E)]$ $\displaystyle=\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}\left[\sup_{z\in\mathcal{Z} }(L_{P}(z)-L_{E}(z))\right]$ $\displaystyle=\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}\left[\sup_{z\in\mathcal{Z} }\mathbb{E}_{(\theta_{1}^{\prime},\cdots,\theta_{N}^{\prime})\sim\mathcal{P}^{ \prime}_{\Theta^{N}}}\left(L_{E^{\prime}}(z)-L_{E}(z)\right)\right]$ $\displaystyle\leq\mathbb{E}_{\mathcal{P}_{\Theta^{N}},\mathcal{P}^{\prime}_{ \Theta^{N}}}\left[\sup_{z\in\mathcal{Z}}\left(L_{E^{\prime}}(z)-L_{E}(z)\right )\right]$ (Jensen inequality) $\displaystyle=\mathbb{E}_{\mathcal{P}_{\Theta^{N}},\mathcal{P}^{\prime}_{ \Theta^{N}}}\left\{\sup_{z\in\mathcal{Z}}\frac{1}{N}\left[\sum_{i=1}^{N}\ell(f (\theta^{\prime}_{i};x),y)-\sum_{i=1}^{N}\ell(f(\theta_{i};x),y)\right]\right\}$ $\displaystyle=\mathbb{E}_{\boldsymbol{\sigma}}\mathbb{E}_{\mathcal{P}_{\Theta^ {N}},\mathcal{P}^{\prime}_{\Theta^{N}}}\left\{\sup_{z\in\mathcal{Z}}\frac{1}{N }\left[\sum_{i=1}^{N}\sigma_{i}\left[\ell(f^{\prime}_{i}(x),y)-\ell(f_{i}(x),y )\right]\right]\right\}$ $\displaystyle\leq\mathbb{E}_{\boldsymbol{\sigma}}\mathbb{E}_{\mathcal{P}^{ \prime}_{\Theta^{N}}}\left\{\sup_{z\in\mathcal{Z}}\frac{1}{N}\left[\sum_{i=1}^ {N}\sigma_{i}\ell(f^{\prime}_{i}(x),y)\right]\right\}+\mathbb{E}_{\boldsymbol{ \sigma}}\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}\left\{\sup_{z\in\mathcal{Z}} \frac{1}{N}\left[\sum_{i=1}^{N}\sigma_{i}\ell(f_{i}(x),y)\right]\right\}$ $\displaystyle=2\cdot\mathbb{E}_{\boldsymbol{\sigma}}\mathbb{E}_{\mathcal{P}_{ \Theta^{N}}}\left\{\sup_{z\in\mathcal{Z}}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i} \ell(f_{i}(x),y)\right\}$ $\displaystyle=2\cdot\mathbb{E}_{\boldsymbol{\sigma}}\left\{\sup_{z\in\mathcal{ Z}}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}\ell(f_{i}(x),y)\right\}$ $\displaystyle=2\mathcal{R}_{N}(\mathcal{F}).$ $$

Likewise, if we define

$$ $\displaystyle\Phi_{2}(E)=\sup_{z\in\mathcal{Z}}\left\{L_{E}(z)-L_{P}(z)\right\},$ $\displaystyle\Phi_{2}(E^{\prime})=\sup_{z\in\mathcal{Z}}\left\{L_{E^{\prime}}( z)-L_{P}(z)\right\},$ $$

then we have

$$ $\displaystyle\Phi_{2}(E)-\Phi_{2}(E^{\prime})$ $\displaystyle=\sup_{z\in\mathcal{Z}}\left\{L_{E}(z)-L_{P}(z)\right\}-\sup_{z \in\mathcal{Z}}\left\{L_{E^{\prime}}(z)-L_{P}(z)\right\}$ $\displaystyle\leq\sup_{z\in\mathcal{Z}}\left\{L_{E}(z)-L_{P}(z)-(L_{E^{\prime} }(z)-L_{P}(z))\right\}$ $\displaystyle=\sup_{z\in\mathcal{Z}}\left\{L_{E}(z)-L_{E^{\prime}}(z)\right\}$ $\displaystyle=\frac{1}{N}\sup_{z\in\mathcal{Z}}\left[\sum_{i=1}^{N}\ell(f( \theta_{i};x),y)-\sum_{i=1}^{N}\ell(f(\theta^{\prime}_{i};x),y)\right].$ $$

According to the assumption that loss function $\ell$ is bounded by $\beta$, we have

$$ $\left|\Phi_{2}(E)-\Phi_{2}(E^{\prime})\right|\leq\frac{\beta}{N}.$ $$

According to Theorem 1 in Esposito and Mondelli [], for all $\delta\in(0,1)$ and $\alpha>1$, with probability at least $1-\delta$, we have

$$ $\Phi_{2}(E)\leq\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}[\Phi_{2}(E)]+\sqrt{\frac{ \alpha\beta^{2}}{2(\alpha-1)N}\ln{\frac{2^{\frac{\alpha-1}{\alpha}}H_{\alpha}^ {\frac{1}{\alpha}}\left(\mathcal{P}_{\Theta^{N}}\|\mathcal{P}_{\bigotimes_{i=1 }^{N}\Theta_{i}}\right)}{\delta}}}.$ (20) $$

We estimate the upper bound of $\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}[\Phi_{2}(E)]$ as follows:

$$ $\displaystyle\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}[\Phi_{2}(E)]$ $\displaystyle=\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}\left[\sup_{z\in\mathcal{Z} }(L_{E}(z)-L_{P}(z))\right]$ $\displaystyle=\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}\left[\sup_{z\in\mathcal{Z} }\mathbb{E}_{(\theta_{1}^{\prime},\cdots,\theta_{N}^{\prime})\sim\mathcal{P}^{ \prime}_{\Theta^{N}}}\left(L_{E}(z)-L_{E^{\prime}}(z)\right)\right]$ $\displaystyle\leq\mathbb{E}_{\mathcal{P}_{\Theta^{N}},\mathcal{P}^{\prime}_{ \Theta^{N}}}\left[\sup_{z\in\mathcal{Z}}\left(L_{E}(z)-L_{E^{\prime}}(z)\right )\right]$ (Jensen inequality) $\displaystyle=\mathbb{E}_{\mathcal{P}_{\Theta^{N}},\mathcal{P}^{\prime}_{ \Theta^{N}}}\left\{\sup_{z\in\mathcal{Z}}\frac{1}{N}\left[\sum_{i=1}^{N}\ell(f (\theta_{i};x),y)-\sum_{i=1}^{N}\ell(f(\theta^{\prime}_{i};x),y)\right]\right\}$ $\displaystyle=\mathbb{E}_{\boldsymbol{\sigma}}\mathbb{E}_{\mathcal{P}_{\Theta^ {N}},\mathcal{P}^{\prime}_{\Theta^{N}}}\left\{\sup_{z\in\mathcal{Z}}\frac{1}{N }\left[\sum_{i=1}^{N}\sigma_{i}\left[\ell(f_{i}(x),y)-\ell(f^{\prime}_{i}(x),y )\right]\right]\right\}$ $\displaystyle\leq\mathbb{E}_{\boldsymbol{\sigma}}\mathbb{E}_{\mathcal{P}^{ \prime}_{\Theta^{N}}}\left\{\sup_{z\in\mathcal{Z}}\frac{1}{N}\left[\sum_{i=1}^ {N}\sigma_{i}\ell(f^{\prime}_{i}(x),y)\right]\right\}+\mathbb{E}_{\boldsymbol{ \sigma}}\mathbb{E}_{\mathcal{P}_{\Theta^{N}}}\left\{\sup_{z\in\mathcal{Z}} \frac{1}{N}\left[\sum_{i=1}^{N}\sigma_{i}\ell(f_{i}(x),y)\right]\right\}$ $\displaystyle=2\cdot\mathbb{E}_{\boldsymbol{\sigma}}\mathbb{E}_{\mathcal{P}_{ \Theta^{N}}}\left\{\sup_{z\in\mathcal{Z}}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i} \ell(f_{i}(x),y)\right\}$ $\displaystyle=2\cdot\mathbb{E}_{\boldsymbol{\sigma}}\left\{\sup_{z\in\mathcal{ Z}}\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}\ell(f_{i}(x),y)\right\}$ $\displaystyle=2\mathcal{R}_{N}(\mathcal{F}).$ $$

Therefore, with probability at least $1-\delta$, there holds

$$ $TE(z,\epsilon)=\Phi_{1}(E)+\Phi_{2}(E)\leq 4\mathcal{R}_{N}(\mathcal{F})+\sqrt {\frac{2\alpha\beta^{2}}{(\alpha-1)N}\ln{\frac{2^{\frac{\alpha-1}{\alpha}}H_{ \alpha}^{\frac{1}{\alpha}}\left(\mathcal{P}_{X^{n}}\|\mathcal{P}_{\bigotimes_{ i=1}^{n}X_{i}}\right)}{\delta}}}.$ $$

The proof is complete.

∎

###### Definition 4 (Hellinger integrals [Hellinger, 1909 ] ) .

###### Proof.

## Appendix C More Related Work

### C.1 Transferable Adversarial Attack

#### Input transformation.

Input transformation-based attacks have shown great effectiveness in improving transferability and can be combined with gradient-based attacks.
Most input transformation techniques rely on the fundamental idea of applying data augmentation strategies to prevent overfitting to the surrogate model [Gu et al., ].
Such methods adopt various input transformations to further improve the transferability of adversarial examples [Wang et al., , ].
For instance, random resizing and padding [Xie et al., ], downscaling [Lin et al., ], mixing [Wang et al., ], automated data augmentation [Yan et al., ], block shuffle and rotation [Wang et al., ], and dynamical transformation [Zhu et al., ].

#### Gradient-based optimization.

The central concept of these methods is to develop optimization techniques in the generation of adversarial examples to achieve better transferability.
Dong et al. [], Lin et al. [], Wang and He [] draw an analogy between generating adversarial examples and the model training process.
Therefore, conventional optimization methods that improve model generalization can also benefit adversarial transferability.
In gradient-based optimization methods, adversarial perturbations are directly optimized based on one or more surrogate models during inference.
Some popular ideas include applying momentum [Dong et al., ], Nesterov accelerated gradient [Lin et al., ], scheduled step size [Gao et al., ] and gradient variance reduction [Wang and He, , Xiong et al., ].
There are also other elegantly designed techniques in recent years [Gubri et al., , Wang et al., , Xiaosen et al., , Li et al., , Wu et al., , Zhang et al., ], such as collecting weights [Gubri et al., ], modifying gradient calculation [Xiaosen et al., ] and applying integrated gradients [Ma et al., ].

#### Model ensemble attack.

Motivated by the use of model ensembles in machine learning, researchers have developed diverse ensemble attack strategies to obtain transferable adversarial examples [Gu et al., ].
It is a powerful attack that employs an ensemble of models to simultaneously generate adversarial samples.
It can not only integrate with advanced gradient-based optimization methods, but also harness the unique strengths of each individual model [Tang et al., ].
Some popular ensemble paradigms include loss-based ensemble [Dong et al., ], prediction-based [Liu et al., ], logit-based ensemble [Dong et al., ], and longitudinal strategy [Li et al., ].
There is also some deep analysis to compare these ensemble paradigms [Zhang et al., ].
Moreover, advanced ensemble algorithms have been created to ensure better adversarial transferability [Zou et al., , Gubri et al., , Xiong et al., , Chen et al., , Li et al., , Wu et al., , Chen et al., ].

### C.2 Statistical Learning Theory

Statistical learning theory forms the theoretical backbone of modern machine learning by providing rigorous frameworks for understanding model generalization [Vapnik, ].
It introduces foundational concepts such as Rademacher complexity [Bartlett and Mendelson, ], VC dimension [Vapnik and Chervonenkis, ], structural risk minimization [Vapnik, ] .
It has also been instrumental in the development of Support Vector Machines [Cortes and Vapnik, ] and kernel methods [Shawe-Taylor and Cristianini, ], which remain pivotal in supervised learning tasks.
Recent advances extend statistical learning theory to deep learning, addressing challenges of high-dimensional data and model complexity [Bartlett et al., ].
These contributions have significantly enhanced the capability to design robust learning algorithms that generalize well across diverse applications [Du and Swamy, ].
In addition, there are also some other novel theoretical frameworks,
such as information-theoretic analysis [Xu and Raginsky, ],
PAC-Bayes bounds [Parrado-Hernández et al., ],
transductive learning [Vapnik, ],
and stability analysis [Bousquet and Elisseeff, , Shalev-Shwartz et al., ].
Most of them derive a bound of the order $\mathcal{O}(\frac{1}{\sqrt{M}})$, while some others derive sharper bound of generalization [Li and Liu, ] of the order $\mathcal{O}(\frac{1}{M})$.
Such theoretical analysis suggests that with the increase of the dataset volume, the model generalization will become better.

## Appendix D Further Discussion

### D.1 Analyze Empirical Model Ensemble Rademacher Complexity

In particular, we present detailed analysis for the simple and complex cases below, within the context of transferable model ensemble attack.

#### The simple input space.

Firstly, consider the trivial case where the input space contains too simple examples so that all classifiers correctly classify $(x,y)\in\mathcal{Z}$. Then there holds

$$ $\mathcal{R}_{N}(\mathcal{Z})=\ell(y,y)\mathop{\mathbb{E}}\limits_{\boldsymbol{ \sigma}}\left[\frac{1}{N}\sum_{i=1}^{N}\sigma_{i}\right]=0.$ $$

In this case, $\mathcal{Z}$ is simple enough for $f_{1},\cdots,f_{N}$. Such $\mathcal{Z}$ corresponds to a $\mathcal{R}_{N}(\mathcal{Z})$ close to 0 0.
However, it is important to note that an overly simplistic space $\mathcal{Z}$ may be impractical for model ensemble attack:
the adversarial examples in such a space may not successfully attack the models from $D$, leading to a small value of $L_{P}(z^{*})$.
In other words, the existence of transferable adversarial examples implicitly imposes constraints on the minimum complexity of $\mathcal{Z}$.

#### The complex input space.

Secondly, we consider the complex case.
In particular, given arbitrarily $N$ models in $\mathcal{H}$ and any assignment of $\boldsymbol{\sigma}$,
a sufficiently complex $\mathcal{Z}$ contains all kinds of examples that make $\mathcal{R}_{N}(\mathcal{Z})$ large: (1) If $\sigma_{i}=+1$, there are adversarial examples that can successfully attack $f_{i}$ and leads to a large $\sigma_{i}\ell(f_{i}(x),y)$; (2) If $\sigma_{i}=-1$, there exists some examples that can be correctly classified by $f_{i}$, leading to $\sigma_{i}\ell(f_{i}(x),y)=0$.
However, such a large $\mathcal{R}_{N}(\mathcal{Z})$ is also not appropriate for transferable model ensemble attack. It may include adversarial examples that perform well against $f_{1},\cdots,f_{N}$ but are merely overfitted to the current $N$ surrogate models [Rice et al., , Yu et al., ].
In other words, these examples might not effectively attack other models in $\mathcal{H}$, thereby limiting their adversarial transferability.

The above analysis suggests that an excessively large or small $\mathcal{R}_{N}(\mathcal{Z})$ is not suitable for adversarial transferability.
So we are curious to investigate the correlation between $\mathcal{R}_{N}(\mathcal{Z})$ and adversarial transferability, which comes to the analysis about the general case in Section 3.4.

#### Explain robust overfitting.

After a certain point in adversarial training, continued training significantly reduces the robust training loss of the classifier while increasing the robust test loss, a phenomenon known as robust overfitting [Rice et al., , Yu et al., ] (also linked to robust generalization [Schmidt et al., , Yin et al., , Zhang et al., ]).
From the perspective in Section 3.4, the cause of this overfitting is the limited complexity of the input space relative to the classifier used to generate adversarial examples during training. The adversarial examples become too simple for the model, leading to overfitting.
To mitigate this, we could consider generating more “hard” and “generalizable” adversarial examples to improve the model’s generalization in adversarial training.
For a less transferable adversarial example $(x,y)$, it is associated with a small $L_{P}(z)$, which in turn makes $TE(z,\epsilon)$ large.

### D.2 Compare with Generalization Error Bound.

We note that a key distinction between transferability error and generalization error lies in the independence assumption.
Conventional generalization error analysis relies on an assumption: each data point from the dataset is independently sampled [Zou and Liu, , Hu et al., ].
By contrast, the surrogate models $f_{1},\cdots,f_{N}$ for ensemble attack are usually trained on the datasets with similar tasks, e.g., image classification.
In this case, such models tend to correctly classify easy examples while misclassify difficult examples [Bengio et al., ].
Consequently, such correlation indicates dependency [Lancaster, ], suggesting that we cannot assume these surrogate models behave independently for a solid theoretical analysis.
Additionally, there are alternative methods for analyzing concentration inequality in generalization error analysis that do not rely on the independence assumption [Kontorovich and Ramanan, , Mohri and Rostamizadeh, , Lei et al., , Zhang et al., ].
However, such data-dependent analysis is either too loose [Lampert et al., ] (because it includes an additional additive factor that grows with the number of samples [Esposito and Mondelli, ]) or requires specific independence structure of data [Zhang and Amini, ] that may not align well with model ensemble attacks.
Therefore, we uses the latest techniques of information theory [Esposito and Mondelli, ] about concentration inequality regarding dependency.
To our best knowledge, it is the first mathematical tool in concentration inequality that fits our needs.

### D.3 The Analogy between Generalization and Adversarial Transferability

Besides providing inspiration for model ensemble attacks, the theoretical evidence in this paper also offers new insights into another fascinating idea.
Within the extensive body of research on transferable adversarial attack algorithms accumulated over the years [Gu et al., ],
we revisit a foundational analogy that is universally applicable in the adversarial transferability literature:
The transferability of an adversarial example is an analogue to the generalizability of the model [Dong et al., ].
In other words, the ideas that enhance model generalization in deep learning may also improve adversarial transferability [Lin et al., ].
Over the past few years, this analogy has significantly inspired the development of numerous effective algorithms, which directly reference it in their papers [Lin et al., , Wang et al., , Wang and He, , Xiong et al., , Chen et al., ].
And some recent papers are also inspired by it [Chen et al., , Wu et al., , Wang et al., , Tang et al., ].
Thus, validating this influential analogy is indispensable for defining the future landscape of research in adversarial transferability.
Interestingly, our paper sheds light on this insight in several ways.

First,
the mathematical formulations in Lemma 1 is similar to generalization error [Vapnik, , Bousquet and Elisseeff, ]
, which also derives an objective as a difference between the population risk and the empirical risk.
Such similarity between transferability error and generalization error suggests the possible validity of the analogy.
Also,
Lemma 2 is similar to the bound of the original Rademacher complexity [Golowich et al., ], which also suggests that obtaining a larger training set as well as a less complex model contribute a tighter bound of Rademacher complexity.
Such similarities between transferability error and generalization error suggests the possible validity of the analogy.
More importantly, if the analogy is correct, then recall that in the conventional framework of learning theory:
(1) increasing the size of training set typically leads to a better generalization of the model [Bousquet and Elisseeff, ];
(2) improving the diversity among ensemble classifiers makes it more advantageous for better generalization [Ortega et al., ];
and (3) reducing the model complexity [Cherkassky, ] benefits the generalization ability.
It is natural to ask:
In model ensemble attack, do
(1) incorporating more surrogate models,
(2) making them more diverse,
and (3) reducing their model complexity
theoretically result in better adversarial transferability?

In Section 4, our theoretical framework provides consistently affirmative responses to the above question as well as the analogy.
Considering a higher perspective, the theory is also instructive in two ways.
On the one hand, from the perspective of a theoretical researcher, the extensive and advanced generalization theory may yield enlightening insights in the field of adversarial transferability.
On the other hand, from an practitioner’s point of view, ideas from deep learning algorithms can also be leveraged to develop more effective transferable attack algorithms.

### D.4 Conflicting Opinions on “Diversity”

We observe a significant and intriguing disagreement within the academic community concerning the role of “diversity” in transferable model ensemble attacks:

- •
Some studies advocate for enhancing model diversity to produce more transferable adversarial examples. For instance, Li et al. [] applies feature-level perturbations to an existing model to potentially create a huge set of diverse “Ghost Networks”.
Li et al. [] emphasizes the importance of diversity in surrogate models and promotes attacking a Bayesian model to achieve desirable transferability. Tang et al. [] supports the notion of improved diversity, suggesting the generation of adversarial examples independently from individual models.
- •
In contrast, other researchers adopt a diversity-reduction strategy to enhance adversarial transferability. For example, Xiong et al. [] focuses on minimizing gradient variance among ensemble models to improve transferability. Meanwhile, Chen et al. [] introduces a disparity-reduced filter designed to decrease gradient variances among surrogate models in ensemble attacks.

Although all these studies reference “diversity,” their perspectives appear to diverge.
In this paper, we advocate for increasing the diversity of surrogate models.
However, we also recognize that diversity-reduction approaches have their merits.

Consider the vulnerability-diversity decomposition of transferability error presented in Theorem 1. It suggests the presence of a vulnerability-diversity trade-off in transferable model ensemble attacks.
In other words, we may need to prioritize either vulnerability or diversity to effectively reduce transferability error.
Diversity-reduction approaches aim to stabilize the training process, thereby increasing the “bias.” In contrast, diversity-promoting methods directly enhance “diversity.” This analysis, framed within our unified theoretical framework, provides insight into the differing opinions regarding adversarial transferability in the academic community.

### D.5 Vulnerability-diversity Trade-off Curve

The relationship between vulnerability and diversity, as discussed in Section 5, merits deeper exploration.
Drawing on the parallels between the vulnerability-diversity trade-off and the bias-variance trade-off [Geman et al., ],
we find that insights from the latter may prove valuable for understanding the former, and warrant further investigation.

The classical bias-variance trade-off suggests that as model complexity increases, bias decreases while variance rises, resulting in a U-shaped test error curve.
However, recent studies have revealed additional phenomena and provided deeper analysis [Neal et al., , Neal, , Derumigny and Schmidt-Hieber, ], such as the double descent [Belkin et al., , Nakkiran et al., ].

Our experiments indicate that diversity does not follow the same pattern as variance in classical bias-variance trade-off.
Nonetheless, there are indications within the bias-variance trade-off literature that suggest similar behavior might occur.
For instance, Yang et al. [] proposes that variance exhibits a bell-shaped curve, initially increasing and then decreasing as network width grows.
Additionally, Lin and Dobriban [] offers a meticulous understanding of variance through detailed decomposition, highlighting the influence of factors such as initialization, label noise, and training data.
Overall, the trend of variance in model ensemble attack remains a valuable area for future research.
We may borrow insights from machine learning literature to get a better understanding of this.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Why is Line 1035 equal to Line 1038?
2. Why is Line 1378 equal to Line 1381?
soundness: 3
strengths: - The theoretical results are solid and novel. Specifically, it is interesting to see the transferability error can be connected with the empirical Rademacher complexity in a similar form with generalization bound, and the Hellinger distance can be used to quantify the inter-dependency across surrogate models.

- The theoretical results can have a broader impact, as the analysis tools, such as those for bounding dependent random variables and the empirical Rademacher complexity for ensemble, can be applied elsewhere.

- The writing is clear and easy to follow in general.
summary: This submission theoretically studies the transferability error - the chance of being successful if the attack is generated by transferring from an ensemble of models. The core result is an upper bound of transferability error involving a vulnerability term and a diversity term, which further boils down to empirical ensemble Rademacher complexity and the Hellinger distance between joint model distributions and i.i.d. model distribution. The key insight is that the transfer attack needs to involve both more and diverse models and reduce model complexity to be powerful. Results are empirically verified on multiple datasets.
weaknesses: - The studied problem and practical implications may be limited. The analysis is only applicable for ensemble-based transfer attacks and it can only directly guide the design of more powerful attacks of this kind. How to leverage the analysis to better defend the model, or how to generalize the results beyond L2-bounded attacks, are worth further exploration.

- Some insights from the theory may not be justified enough. For example, in Line 333-335, the paper mentioned that we need to increase the diversity of parameters in surrogate models to reduce $H_\alpha(\cdot)$. It seems that surrogate models need to be independently trained to achieve a minimal $H_\alpha(\cdot)$. However, in practice, encouraging model diversity, e.g., introducing some diversity-promoting regularizers, can sometimes further improve the attack efficiency. As a result, encouraging model diversity introduces model-level dependency and increases $H_\alpha(\cdot)$ but reduces transferability error. From this point of view, the theory may not reflect the full picture of adversarial transferability.

- The experiment part is relatively not clear. For example, in Figure 2, good to mention that $\lambda$ is the weight decay, explain what the $x$-axis is, and discuss detail training parameters in the main text. 


Minor:
1. Line 106: combine -> combines
2. Line 153: the hypothesis space maps to a discrete label space, and then the loss function $\ell: \mathcal{Y} \times \mathcal{Y} \mapsto \mathbb{R}_0^+$ has a discrete domain $\\{-1,1\\} \times \\{-1,1\\}$ which is weird, may need some fix.
3. Line 279: the redundant phrase "provided in Appendix"
4. Line 1061: please define $R$ beforehand.
5. Line 1281 - 1290: seems that there is a missing $1/N$ coefficient before all $\sum_{i=1}^N f(\theta_i; x)$.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Other than the concerns pointed out in the weaknesses I have some additional questions for the authors:

1. I have some confusion about the presented plots in the experiments which are not well-explained. Regarding the experiments, are you using mini-batch SGD as the optimizer? By "# step" on the x-axis do you mean the number of epochs? For loss value, is this the loss value of the expectation of logits on a training sample or test sample? Isn't that supposed to be decreasing as all the models are being trained?

2. In figure 4, the variance of the logits from the models in the ensemble is shown to be increasing for CIFAR-10, but the number of epochs is too small and it is not clear whether the same trend continues. Could authors plot them with a higher number of epochs?

3. The plots with increasing values of the variance of the logits from the models of the ensemble seem contradictory to Lemma 5 of [1]. The authors also mention for some datasets they see a decreasing trend similar to what is expected from Lemma 5 of [1]. Could the authors comment on the potential reasons for their different observations for other datasets?
soundness: 3
strengths: The paper is well-written and well motivated with a good survey of related works.

By defining the transferability error, authors make a good analogy to generalization error and derive some corresponding results to provide a better understanding of model ensemble attacks.

Authors avoid the independence assumption that is used for studying generalization and derive an upper-bound is based on the divergence of the joint distribution of the parameters of the models from the case where they are independent.
summary: The authors propose a theoretical framework to explain the observations by prior empirical methods on increasing the effectiveness of model ensemble attacks. They define transferability error to measure the effectiveness of an adversarial example which is basically analogous to the generalization of the adversarial example to unseen trained models belonging to a specific function class. They also define an empirical version of Rademacher complexity as a measure of complexity for the input space for an ensemble of N classifiers and show that the transferability error is upper-bounded by a combination of this measure of input space complexity and the divergence of joint distribution of the model parameters of the ensemble from the product of their marginals which accounts for non-independence of the models of an ensemble.
weaknesses: 1) Authors connect their theoretical results with empirical observations in prior work regarding the diversity of the models; however, their definition of diversity does not match with many of these prior works. For example in [1] and [2] the diversity is defined as having gradient vectors (with respect to the inputs) with low cosine similarity. What authors consider as diversity here actually is supposed to decrease naturally according to Lemma 5 in [1]. Could authors clarify how their definition of diversity relates to these previous definitions in the literature, particularly those based on gradient similarity. 

[1] Yang, Z., Li, L., Xu, X., Zuo, S., Chen, Q., Zhou, P., ... & Li, B. (2021). Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems, 34, 17642-17655.

[2] Kariyappa, S., & Qureshi, M. K. (2019). Improving adversarial robustness of ensembles with diversity training. arXiv preprint arXiv:1901.09981.


2) The complexity of the models in the ensemble and the complexity of the input space seem to be used interchangeably sometimes. Equation 12 shows the complexity of input space defined by the authors, but in the follow-up discussion (line 342) it is mentioned that the model complexity has to be controlled when using stronger and more diverse ensembles.

3) The interpretation of input space Rademacher complexity defined by the authors does not seem clear! The presented results suggest decreasing this complexity to achieve a tighter upper bound on the transferability error. However, decreasing this complexity means achieving a state where the sample in the input space is not able to achieve a high loss value for the models in the ensemble. This basically means that the optimization in equation 3 will achieve a lower value for equation 1 which authors are seeking to increase. This seems contradictory and it would be great if authors could clarify that. 

4) The experiments do not seem to be comprehensive in evaluating the presented theoretical results. For example, there is no analysis with respect to the complexity of the input space or the trade-off of diversity and complexity.

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: - Given the identified trade-off between vulnerability and diversity, could the authors suggest any criteria or metrics for balancing these components during ensemble model selection?

- The experiments use standard datasets like MNIST and CIFAR-10, which may not fully represent the complexity encountered in real-world applications. Have the authors considered testing on more complex datasets (e.g. CIFAR-100, SVHN, ImageNet, etc.)

- Can the author give the specific method of generating adversarial samples in the experiment and the specific meaning of "steps" in fig. 2, 3 and 4.
soundness: 4
strengths: The paper demonstrates strong originality by addressing the theoretical gaps in model ensemble-based adversarial attacks, introducing the novel concepts of transferability error, vulnerability-diversity decomposition, providing well-founded upper bounds for transferability error.
summary: This paper provides a theoretical foundation for model ensemble methods used to generate transferable adversarial examples. The authors introduce three new concepts: transferability error, diversity, and empirical Rademacher complexity, which together decompose transferability error into two primary components: vulnerability and diversity. Futhermore, the authors establish a bound on transferability error and propose practical guidelines to reduce it, such as increasing model diversity and managing complexity to prevent overfitting. Extensive experiments validate these findings​.
weaknesses: Although this paper provides a strong theoretical foundation, some limitations affect its overall impact. 

While the experiments are broad in scope, they can be enhanced by testing on a wider range of real-world scenarios or datasets outside of standard benchmarks such as MNIST and CIFAR-10 to verify applicability in more diverse contexts (e.g. CIFAR-100, SVHN, etc.).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: My main concerns are the implicit assumptions in Eq. (3), making the derivations much less interesting. Besides, the concluded practical guidelines are already widely applied in the literature, and there is also a lack of empirical comparisons to previous baselines for ensemble-based transfer attacks.
soundness: 2
strengths: The Weaknesses of this paper include:

- The writing is clear with intuitive explanations are in Figure 1.
- Notations are clearly defined with neat formulations, the derivations are self-consistent.
- The concluded practical guidelines are correct and already used in the literature.
summary: This paper presents theoretical insights into model ensemble adversarial attacks. The authors define transferability error, which measures the error in adversarial transferability. They also discuss diversity and empirical model ensemble Rademacher complexity. The authors then decompose the transferability error to explain how it originated in the model ensemble attack. Furthermore, they derive bounds on the transferability error using complexity and generalization terms, and conclude three practical guidelines for reducing transferability error: (1) incorporating more surrogate models, (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting. Some empirical evaluations are done on MNIST, CIFAR-10, and ImageNet.
weaknesses: The Weaknesses of this paper include:

- **Implicit assumptions in Eq. (3).** The authors define the most transferable adversarial example $z^*$ in Eq. (3) as $z^*=\textrm{argmax} L_P$, where $L_P$ in Eq. (1) is defined by taking expectation over $\theta\sim P_\Theta$. This formulation has implicit assumptions that **1)** the target model share the same parameter space $\Theta$ with the surrogate models, i.e., they have the same architectures; **2)** the target model follow the same distribution $P_\Theta$ with the surrogate models, i.e., they apply the same (or same distribution) of training configurations. Both of these assumptions make the transfer problem overly simplistic, because in practice, the target model typically employs different model architectures and training configurations (including different datasets) than the surrogate models.

- **Using Rademacher Complexity in deep cases.** First, I personally don't believe that Rademacher Complexity can convey reliable information when we are talking about deep networks. Second, Rademacher Complexity is more useful for asymptotic analysis, otherwise a lower upper bound of TE (i.e., Eq. (12)) does not indicate a lower value of TE.

- **The three practical guidelines are already well-known.** While the authors demonstrated some theoretical bounds, the three guidelines they concluded—(1) incorporating more surrogate models, (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting—are all well-known in literature. There is also a lack of empirical comparisons to previous baselines for ensemble-based transfer attacks.

Meta-review: This paper introduces a theoretical framework for understanding transferable adversarial attacks, proposing concepts such as transferability error and a vulnerability-diversity decomposition, with practical guidelines supported by experiments on standard datasets. While the theoretical insights are novel and well-presented, the paper suffers from limited empirical scope, with experiments confined to standard datasets like MNIST and CIFAR-10, and a lack of testing on more complex, real-world scenarios. The definition of diversity diverges from established gradient-based metrics, raising questions about its alignment with prior work. Additionally, the paper offers limited practical utility for defense mechanisms or broader adversarial settings. Despite revisions, unresolved concerns around experimental clarity, trade-offs between complexity and diversity, and the applicability of findings suggest that the work, though promising, is not yet ready for publication.

Decision: Reject