Understanding Model Ensemble in Transferable Adversarial Attack

Authors: Wei Yao, Zeliang Zhang, Huayi Tang, Yong Liu
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: adversarial examples, adversarial transferability, model ensemble attack

Abstract:
Model ensemble adversarial attack has become a powerful method for generating transferable adversarial examples that can target even unknown models, but its theoretical foundation remains underexplored. To address this gap, we provide early theoretical insights that serve as a roadmap for advancing model ensemble adversarial attack.We first define transferability error to measure the error in adversarial transferability, alongside concepts of diversity and empirical model ensemble Rademacher complexity. We then decompose the transferability error into vulnerability, diversity, and a constant, which rigidly explains the origin of transferability error in model ensemble attack: the vulnerability of an adversarial example to ensemble components, and the diversity of ensemble components.Furthermore, we apply the latest mathematical tools in information theory to bound the transferability error using complexity and generalization terms, contributing to three practical guidelines for reducing transferability error: (1) incorporating more surrogate models, (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting. Finally, extensive experiments with 54 models validate our theoretical framework, representing a significant step forward in understanding transferable model ensemble adversarial attacks.

Decision: reject
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1729445956672

thread_type:
official_review

<thread_object>
creator: reviewer_w7xy

<content>
summary:
This paper presents theoretical insights into model ensemble adversarial attacks. The authors define transferability error, which measures the error in adversarial transferability. They also discuss diversity and empirical model ensemble Rademacher complexity. The authors then decompose the transferability error to explain how it originated in the model ensemble attack. Furthermore, they derive bounds on the transferability error using complexity and generalization terms, and conclude three practical guidelines for reducing transferability error: (1) incorporating more surrogate models, (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting. Some empirical evaluations are done on MNIST, CIFAR-10, and ImageNet.

strengths:
The Weaknesses of this paper include:

- The writing is clear with intuitive explanations are in Figure 1.
- Notations are clearly defined with neat formulations, the derivations are self-consistent.
- The concluded practical guidelines are correct and already used in the literature.

weaknesses:
The Weaknesses of this paper include:

- **Implicit assumptions in Eq. (3).** The authors define the most transferable adversarial example $z^*$ in Eq. (3) as $z^*=\textrm{argmax} L_P$, where $L_P$ in Eq. (1) is defined by taking expectation over $\theta\sim P_\Theta$. This formulation has implicit assumptions that **1)** the target model share the same parameter space $\Theta$ with the surrogate models, i.e., they have the same architectures; **2)** the target model follow the same distribution $P_\Theta$ with the surrogate models, i.e., they apply the same (or same distribution) of training configurations. Both of these assumptions make the transfer problem overly simplistic, because in practice, the target model typically employs different model architectures and training configurations (including different datasets) than the surrogate models.

- **Using Rademacher Complexity in deep cases.** First, I personally don't believe that Rademacher Complexity can convey reliable information when we are talking about deep networks. Second, Rademacher Complexity is more useful for asymptotic analysis, otherwise a lower upper bound of TE (i.e., Eq. (12)) does not indicate a lower value of TE.

- **The three practical guidelines are already well-known.** While the authors demonstrated some theoretical bounds, the three guidelines they concluded—(1) incorporating more surrogate models, (2) increasing their diversity, and (3) reducing their complexity in cases of overfitting—are all well-known in literature. There is also a lack of empirical comparisons to previous baselines for ensemble-based transfer attacks.

questions:
My main concerns are the implicit assumptions in Eq. (3), making the derivations much less interesting. Besides, the concluded practical guidelines are already widely applied in the literature, and there is also a lack of empirical comparisons to previous baselines for ensemble-based transfer attacks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692332213

thread_type:
official_review

<thread_object>
creator: reviewer_secg

<content>
summary:
This paper provides a theoretical foundation for model ensemble methods used to generate transferable adversarial examples. The authors introduce three new concepts: transferability error, diversity, and empirical Rademacher complexity, which together decompose transferability error into two primary components: vulnerability and diversity. Futhermore, the authors establish a bound on transferability error and propose practical guidelines to reduce it, such as increasing model diversity and managing complexity to prevent overfitting. Extensive experiments validate these findings​.

strengths:
The paper demonstrates strong originality by addressing the theoretical gaps in model ensemble-based adversarial attacks, introducing the novel concepts of transferability error, vulnerability-diversity decomposition, providing well-founded upper bounds for transferability error.

weaknesses:
Although this paper provides a strong theoretical foundation, some limitations affect its overall impact. 

While the experiments are broad in scope, they can be enhanced by testing on a wider range of real-world scenarios or datasets outside of standard benchmarks such as MNIST and CIFAR-10 to verify applicability in more diverse contexts (e.g. CIFAR-100, SVHN, etc.).

questions:
- Given the identified trade-off between vulnerability and diversity, could the authors suggest any criteria or metrics for balancing these components during ensemble model selection?

- The experiments use standard datasets like MNIST and CIFAR-10, which may not fully represent the complexity encountered in real-world applications. Have the authors considered testing on more complex datasets (e.g. CIFAR-100, SVHN, ImageNet, etc.)

- Can the author give the specific method of generating adversarial samples in the experiment and the specific meaning of "steps" in fig. 2, 3 and 4.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706024429

thread_type:
official_review

<thread_object>
creator: reviewer_ignu

<content>
summary:
The authors propose a theoretical framework to explain the observations by prior empirical methods on increasing the effectiveness of model ensemble attacks. They define transferability error to measure the effectiveness of an adversarial example which is basically analogous to the generalization of the adversarial example to unseen trained models belonging to a specific function class. They also define an empirical version of Rademacher complexity as a measure of complexity for the input space for an ensemble of N classifiers and show that the transferability error is upper-bounded by a combination of this measure of input space complexity and the divergence of joint distribution of the model parameters of the ensemble from the product of their marginals which accounts for non-independence of the models of an ensemble.

strengths:
The paper is well-written and well motivated with a good survey of related works.

By defining the transferability error, authors make a good analogy to generalization error and derive some corresponding results to provide a better understanding of model ensemble attacks.

Authors avoid the independence assumption that is used for studying generalization and derive an upper-bound is based on the divergence of the joint distribution of the parameters of the models from the case where they are independent.

weaknesses:
1) Authors connect their theoretical results with empirical observations in prior work regarding the diversity of the models; however, their definition of diversity does not match with many of these prior works. For example in [1] and [2] the diversity is defined as having gradient vectors (with respect to the inputs) with low cosine similarity. What authors consider as diversity here actually is supposed to decrease naturally according to Lemma 5 in [1]. Could authors clarify how their definition of diversity relates to these previous definitions in the literature, particularly those based on gradient similarity. 

[1] Yang, Z., Li, L., Xu, X., Zuo, S., Chen, Q., Zhou, P., ... & Li, B. (2021). Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems, 34, 17642-17655.

[2] Kariyappa, S., & Qureshi, M. K. (2019). Improving adversarial robustness of ensembles with diversity training. arXiv preprint arXiv:1901.09981.


2) The complexity of the models in the ensemble and the complexity of the input space seem to be used interchangeably sometimes. Equation 12 shows the complexity of input space defined by the authors, but in the follow-up discussion (line 342) it is mentioned that the model complexity has to be controlled when using stronger and more diverse ensembles.

3) The interpretation of input space Rademacher complexity defined by the authors does not seem clear! The presented results suggest decreasing this complexity to achieve a tighter upper bound on the transferability error. However, decreasing this complexity means achieving a state where the sample in the input space is not able to achieve a high loss value for the models in the ensemble. This basically means that the optimization in equation 3 will achieve a lower value for equation 1 which authors are seeking to increase. This seems contradictory and it would be great if authors could clarify that. 

4) The experiments do not seem to be comprehensive in evaluating the presented theoretical results. For example, there is no analysis with respect to the complexity of the input space or the trade-off of diversity and complexity.

questions:
Other than the concerns pointed out in the weaknesses I have some additional questions for the authors:

1. I have some confusion about the presented plots in the experiments which are not well-explained. Regarding the experiments, are you using mini-batch SGD as the optimizer? By "# step" on the x-axis do you mean the number of epochs? For loss value, is this the loss value of the expectation of logits on a training sample or test sample? Isn't that supposed to be decreasing as all the models are being trained?

2. In figure 4, the variance of the logits from the models in the ensemble is shown to be increasing for CIFAR-10, but the number of epochs is too small and it is not clear whether the same trend continues. Could authors plot them with a higher number of epochs?

3. The plots with increasing values of the variance of the logits from the models of the ensemble seem contradictory to Lemma 5 of [1]. The authors also mention for some datasets they see a decreasing trend similar to what is expected from Lemma 5 of [1]. Could the authors comment on the potential reasons for their different observations for other datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731004005888

thread_type:
official_review

<thread_object>
creator: reviewer_6nca

<content>
summary:
This submission theoretically studies the transferability error - the chance of being successful if the attack is generated by transferring from an ensemble of models. The core result is an upper bound of transferability error involving a vulnerability term and a diversity term, which further boils down to empirical ensemble Rademacher complexity and the Hellinger distance between joint model distributions and i.i.d. model distribution. The key insight is that the transfer attack needs to involve both more and diverse models and reduce model complexity to be powerful. Results are empirically verified on multiple datasets.

strengths:
- The theoretical results are solid and novel. Specifically, it is interesting to see the transferability error can be connected with the empirical Rademacher complexity in a similar form with generalization bound, and the Hellinger distance can be used to quantify the inter-dependency across surrogate models.

- The theoretical results can have a broader impact, as the analysis tools, such as those for bounding dependent random variables and the empirical Rademacher complexity for ensemble, can be applied elsewhere.

- The writing is clear and easy to follow in general.

weaknesses:
- The studied problem and practical implications may be limited. The analysis is only applicable for ensemble-based transfer attacks and it can only directly guide the design of more powerful attacks of this kind. How to leverage the analysis to better defend the model, or how to generalize the results beyond L2-bounded attacks, are worth further exploration.

- Some insights from the theory may not be justified enough. For example, in Line 333-335, the paper mentioned that we need to increase the diversity of parameters in surrogate models to reduce $H_\alpha(\cdot)$. It seems that surrogate models need to be independently trained to achieve a minimal $H_\alpha(\cdot)$. However, in practice, encouraging model diversity, e.g., introducing some diversity-promoting regularizers, can sometimes further improve the attack efficiency. As a result, encouraging model diversity introduces model-level dependency and increases $H_\alpha(\cdot)$ but reduces transferability error. From this point of view, the theory may not reflect the full picture of adversarial transferability.

- The experiment part is relatively not clear. For example, in Figure 2, good to mention that $\lambda$ is the weight decay, explain what the $x$-axis is, and discuss detail training parameters in the main text. 


Minor:
1. Line 106: combine -> combines
2. Line 153: the hypothesis space maps to a discrete label space, and then the loss function $\ell: \mathcal{Y} \times \mathcal{Y} \mapsto \mathbb{R}_0^+$ has a discrete domain $\\{-1,1\\} \times \\{-1,1\\}$ which is weird, may need some fix.
3. Line 279: the redundant phrase "provided in Appendix"
4. Line 1061: please define $R$ beforehand.
5. Line 1281 - 1290: seems that there is a missing $1/N$ coefficient before all $\sum_{i=1}^N f(\theta_i; x)$.

questions:
1. Why is Line 1035 equal to Line 1038?
2. Why is Line 1378 equal to Line 1381?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732175769685

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We sincerely thank all the reviewers for their detailed and constructive feedback, which we deeply value and appreciate. We are especially encouraged by the reviewers’ recognition of our work and their thoughtful comments:
- Reviewer 6NcA: "The theoretical results are **solid and novel**. **The theoretical results can have a broader impact, as the analysis tools**, such as those for bounding dependent random variables and the empirical Rademacher complexity for ensemble, can be applied elsewhere."
- Reviewer ignu: "By defining the transferability error, authors **make a good analogy to generalization error** and derive some corresponding results to **provide a better understanding of model ensemble attacks**."
- Reviewer secg: "**The paper demonstrates strong originality by addressing the theoretical gaps** in model ensemble-based adversarial attacks, **introducing the novel concepts** of transferability error, vulnerability-diversity decomposition, providing well-founded upper bounds for transferability error."
- Reviewer w7Xy: "The writing is **clear with intuitive explanations**. The derivations are **self-consistent**. The concluded practical guidelines are **correct**."

We work diligently and have carefully addressed all the reviewers' concerns through very detailed clarifications, providing individual responses to each reviewer. Additionally, we have incorporated the following modifications into the revised manuscript:

- Lines 278-280: Discussed alternative definitions of diversity (Reviewer ignu).
- Lines 281-283: Compared Theorem 1 with Lemma 5 in Yang et al. (2021), highlighting their complementary perspectives in analyzing transferable adversarial attacks (Reviewer ignu).
- Line 320: Changed some constant terms in Theorem 2 to make it more solid (Authors).
- Lines 324-326: Explained how Theorem 2 can be naturally extended to scenarios where the surrogate and target model distributions differ (Reviewer w7Xy).
- Lines 327-330: Introduced another version of our theoretical framework using information-theoretic bound (Reviewer w7Xy).
- Line 418-420, 426-430, 463: Provided more details of experiments, including how the surrogate models are trained, the meaning of "# steps", the adversarial example generation algorithm (Reviewer 6NcA, ignu, secg).
- Line 481-490: Provided a discussion on the potential trade-off between diversity and complexity (Reviewer ignu).
- Appendix B.3 (especially Line 1385-1386): Changed two notations to make the proof more clear (Reviewer 6NcA).
- Appendix D.2.1: Provided a detailed discussion on previous definitions of diversity (Reviewer ignu).
- Appendix D.3: Examined Lemma 5 in Yang et al. (2021) in depth (Reviewer ignu).
- Appendix D.4: Proposed two approaches to generalize our theoretical framework: redefining the model space (Appendix D.4.1) and drawing insights from domain adaptation theory (Appendix D.4.2) (Reviewer w7Xy).
- Appendix D.5: Provided an information-theoretic analysis as a natural extension to the theoretical framework established in this paper (Reviewer w7Xy).
- Appendix D.9: Offered insights into model ensemble defense strategies (Reviewer 6NcA).
- Appendix E: Provided the experimental results on CIFAR-100 dataset (Reviewer secg).

The above modifications are all highlighted in blue for the reviewers' convenience for now.
We hope these revisions address the reviewers' concerns comprehensively and improve the overall clarity of our work.

Thanks again to all reviewers. We are glad to continue the discussion if there are any further questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a theoretical framework for understanding transferable adversarial attacks, proposing concepts such as transferability error and a vulnerability-diversity decomposition, with practical guidelines supported by experiments on standard datasets. While the theoretical insights are novel and well-presented, the paper suffers from limited empirical scope, with experiments confined to standard datasets like MNIST and CIFAR-10, and a lack of testing on more complex, real-world scenarios. The definition of diversity diverges from established gradient-based metrics, raising questions about its alignment with prior work. Additionally, the paper offers limited practical utility for defense mechanisms or broader adversarial settings. Despite revisions, unresolved concerns around experimental clarity, trade-offs between complexity and diversity, and the applicability of findings suggest that the work, though promising, is not yet ready for publication.