Training Robust Ensembles Requires Rethinking Lipschitz Continuity

Authors: Ali Ebrahimpour-Boroojeny, Hari Sundaram, Varun Chandrasekaran
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: robustness, lipschitzness, ensembles, convolutions, transferability, black-box attacks

Abstract:
Transferability of adversarial examples is a well-known property that endangers all classification models, even those that are only accessible through black-box queries. Prior work has shown that an ensemble of models is more resilient to transferability: the probability that an adversarial example is effective against most models of the ensemble is low. Thus, most ongoing research focuses on improving ensemble diversity. Another line of prior work has shown that Lipschitz continuity of the models can make models more robust since it limits how a model's output changes with small input perturbations. {\em In this paper, we study the effect of Lipschitz continuity on transferability rates.} We show that although a lower Lipschitz constant increases the robustness of a single model, it is not as beneficial in training robust ensembles as it increases the transferability rate of adversarial examples across models in the ensemble. Therefore, we introduce LOTOS, a new training paradigm for ensembles, which counteracts this adverse effect. It does so by promoting orthogonality among the top-$k$ sub-spaces of the transformations of the corresponding affine layers of any pair of models in the ensemble. We theoretically show that $k$ does not need to be large for convolutional layers, which makes the computational overhead negligible. Through various experiments, we show LOTOS increases the robust accuracy of ensembles of ResNet-18 models by $6$ percentage points (p.p) against black-box attacks on CIFAR-10. It is also capable of combining with the robustness of prior state-of-the-art methods for training robust ensembles to enhance their robust accuracy by $10.7$ p.p.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729223910372

thread_type:
official_review

<thread_object>
creator: reviewer_hkex

<content>
summary:
The paper studies how Lipschitz continuity affects the transferability of adversarial examples in model ensembles. It finds that while lower Lipschitz constants improve individual model robustness, they increase transferability in ensembles. To address this, the authors propose LOTOS, a training method that promotes subspace orthogonality between models, enhancing ensemble robustness. Experiments show that LOTOS improves the robust accuracy of ResNet-18 ensembles and boosts the performance of existing robust ensemble methods.

strengths:
- It's interesting to observe the trade-off between the robustness of individual models and the transferability of adversarial examples when changing Lipschitz constant. It highlights the importance of addressing the trade-off, an issue that has been overlooked in prior research.
- The proposed LOTOS method is presented as a solution that uniquely focuses on orthogonalizing affine layers, setting it apart from existing methods that target different aspects of model diversity. To my best knowledge, such contribution is also novel.
- The empirical results are shown clearly and shed light on so much understanding of the proposed solution.

weaknesses:
- **The soundness of the claim in this paper**: The authors suggest that "although a lower Lipschitz constant increases the robustness of a single model, it is not as beneficial in training robust ensembles as it increases the transferability rate of adversarial examples across models in the ensemble." While I believe that the first part is sound, I want to challenge the statement "a lower Lipschitz constant is not as beneficial in training robust ensembles as it increases the transferability rate of adversarial examples across models in the ensemble". There are too many aspects that may influence the result, for instance, the generalization of the models (because a lower Lipschitz constant may make the models more difficult to train). In other words, on a broad view, it's not just a trade-off between two aspects mentioned in this paper. I encourage the authors to incorporate deeper understanding of this issue (see below).
- **Missing related work and discussion**: As far as I know, there may be some papers that is relevant to yours. For example, some theoretical works [1,2,3] discuss the smoothness & Lipschitz constant in transferability of adversarial examples (see the bounds therein), which may be somehow relevant to Lipschitz continuity emphasized in this paper. Some of them may not discuss the model ensemble issue. However, these insights on individual models is theoretically connected with the analysis in this paper. Furthermore, the vulnerability-diversity decomposition [4] also sheds light on the relationship among transferability, single model robustness and diversity, which may be in line with your claim "We observe that while decreasing the Lipschitz constant makes each model of the ensemble individually more robust, it makes them less diverse and consequently increases the transferability rate among them which in turn hinders the overall ensemble robustness". In addition, the model ensemble literature [5] also suggests the trade-off between diversity and individual model performance. Therefore, maybe you could give a brief discussion of them to help us better understand your theoretical insights. You are also encouraged to choose to discuss other work that I haven't mentioned but is relevant to your paper.
- **Experiments**: Based on the above, I encourage the authors to incoporate more experiments. Since the model generalization / optimization process may also change the trade-off mentioned in this paper, you may add more experiments regarding more algorithms to show the soundness and consistency of your claim. However, in the current version, the experimental parts only consider their own method as well as TRS [1].

If the author's answers address my concerns, I will consider raising the score.

[1] Yang Z, Li L, Xu X, et al. Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness[J]. Advances in Neural Information Processing Systems, 2021, 34: 17642-17655.

[2] Zhang Y, Hu S, Zhang L Y, et al. Why does little robustness help? a further step towards understanding adversarial transferability[C]//2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024: 3365-3384.

[3] Wang Y, Farnia F. On the role of generalization in transferability of adversarial examples[C]//Uncertainty in Artificial Intelligence. PMLR, 2023: 2259-2270.

[4] Yao W, Zhang Z, Tang H, et al. Understanding Model Ensemble in Transferable Adversarial Attack[J]. arXiv preprint arXiv:2410.06851, 2024.

[5] Abe T, Buchanan E K, Pleiss G, et al. Pathologies of predictive diversity in deep ensembles[J]. arXiv preprint arXiv:2302.00704, 2023.

questions:
- What is the definition of $\Re$ in the appendix (page 14)?
- Typos: line 333: "ensebmels"
- Do you think the method in this paper requires significant work on hyperparameter tuning?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730170733611

thread_type:
official_review

<thread_object>
creator: reviewer_au68

<content>
summary:
The paper analyzes the relationship between model Lipschitzness and the transferability of adversarial examples. Under the setting of TRS model ensemble, decreasing the model's Lipschitz constant improves standalone robustness but also increases transferability, which hurts ensemble robustness. To this end, the authors propose "LOTOS", which promotes the orthogonality of the singular vectors corresponding to the largest singular values of the transformation matrices of the layers of the ensemble candidate models. LOTOS can decrease transferability while only having a slight effect on individual robustness, thereby improving the ensemble robustness. Furthermore, LOTOS can work with heterogeneous model architectures in conjunction with adversarial training.

strengths:
The paper analyzes an interesting problem, and uses the conclusions to propose a method that seems to work.

weaknesses:
First of all, I would like to disclose that I am not an expert on black-box attacks or diversity-promoting robust ensembles. I am more familiar with the literature about the white-box scenario.

- The proposed method seems promising in the black-box setting, but how about the white-box case? I think this question is important because it will disentangle robustness from attack difficulty.
- What would happen if the "source model" in the black-box transfer attack is a LOTOS model itself (trained separately from the target model)?
- All definitions in this paper seem to build on the $\ell_2$ norm. Is the analyses and the proposed method only applicable to $\ell_2$ attacks? How about other attack budgets?
- Just to clarify, are all models in the ensemble trained simultaneously, or can they be trained sequentially? Asking for clarification here because the proposed loss function for each model depends on the weights of all other models.
- Line 285 says "for the orthogonalization to be effective in Equation (4), it is necessary to increase the value of $k$ (dimension of orthogonal sub-spaces)". Then, Line 294 says "$k = 1$ can be effective in orthogonalization with respect to the remaining singular vectors for convolutional layers." Are these not contradictory?
- In Theorem 4, the right-hand-side of Equation 5 has two terms under the square root. Which one is dominant? $\epsilon^2$ or the other terms?
- Line 333: typo "ensebmels".

questions:
- The experiments in the paper focuses on ResNet and DLA. Do you think the method will extend to other popular architectures such as vision transformer, which operates differently and can be shallower?
- Line 215 (using the difference in the population loss of the two models on these adversarial examples as a proxy) -- How good is this approximation? If we use some alternative loss functions other than cross-entropy, is there any chance for this approximation become exact?
- In Equation (3), how would we determine $w_i$? Why do we need this weight, because intuitively we could perhaps just use the singular values here?
- Line 260 ($\\| \sum_{i=1}^d \sigma_i u_i v_i^\top v_i' \\|_2 = 0$). Here we only try to make the singular vectors orthogonal for each $i$, but the singular vectors for different $i$'s are not accounted. Would it possible to have the case where, for example, the first singular vector of $f^{(j)}$ is orthogonal to the first singular vector of $g^{(j)}$, but highly similar to the second singular vector of $g^{(j)}$? In this case, is the transferability between $f$ and $g$ still low?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730724849772

thread_type:
official_review

<thread_object>
creator: reviewer_ulen

<content>
summary:
This paper finds that while Lipschitz continuity improves individual model robustness, it unexpectedly raises adversarial transferability in model ensembles. To counter this, the authors introduce LOTOS to boost ensemble robustness by promoting orthogonality between models.

strengths:
This paper links decreasing Lipschitz continuity with increasing transferrability. This is an interesting point of exploration.

weaknesses:
- What are the white-box attacks that you used in this paper? Some attacks may not be reliable in assessing ensemble robustness.
- The method appears to be an improvement upon TRS. However, TRS was shown to have an overestimated robustness [a], and it is not well-explained why the LOTOS loss is an improvement.
- It is not clear to me why DVERGE was not considered in the baseline comparisons, as it is known to be an effective robust baseline for ensemble defenses.
- Section 4 is poorly written. I am having trouble understanding your method. For eq. (3):
  - Where is $w_i$ introduced?
  - Why does this equation measure similarity?
  - What are $A$ and $B$ matrices, are they model parameters?
  - Also given that $A$ and $B$ are trainable, do you backprop through the SVD of them?
- For eq. (4): There is no $k$ in the LHS.
- Theorem 4.1:
  - "In Theorem 4.1, we prove even k = 1 can be effective in orthogonalization with respect to the remaining singular vectors for convolutional layers." It is not clear to me how it is proven for $k = 1$.
  - Why circular padding?
  - It is not clear to me how this theorem links with eq. (4).

questions:
- Could you compare your methods with other ensemble defenses, especially with DVERGE?
- Could you consider stronger attacks such as AutoAttack [b] and MORA [a] which specifically target ensemble models?

[a]: Yu et al., MORA: Improving Ensemble Robustness Evaluation with Model-Reweighing Attack. NeurIPS 2022. https://arxiv.org/abs/2211.08008
[b]: Croce et al., Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. ICML 2020. https://arxiv.org/abs/2003.01690
</content>
</thread_object>

</thread>
<thread>
creation date:
1731415409418

thread_type:
official_review

<thread_object>
creator: reviewer_olsw

<content>
summary:
This paper makes 3 contributions to the field of robust ensemble training: 

1. It identifies a trade-off in neural network ensembles: while decreasing the Lipschitz constant of individual models improves their robustness, it unfortunately also increases the transferability of adversarial examples between models in the ensemble, potentially undermining the ensemble's overall robustness. 

2. It introduces LOTOS (Layer-wise Orthogonalization for Training rObust enSembles), a new training method that promotes orthogonality between corresponding layers of different models in an ensemble. LOTOS is computationally efficient and pretty architecturally agnostic. 

3. Through experiments on CIFAR-10 and CIFAR-100, the authors demonstrate that LOTOS improves ensemble robustness against black-box attacks by 6 percentage points, and when combined with existing robust ensemble methods, enhances their performance by up to 10.7 percentage points.

strengths:
Overall I'm a fan of establishing (through published literature) empirical observations of interesting deep learning phenomena. This is an example of such a thing. The fact that we can both a) make an individual model more robust via Lipschitz constant changes, but at the same time b) make an ensemble of such models more similar in a way that reduces the overall robustness of the ensemble, is a very curious piece of empirical evidence.

I think the evaluation is pretty strong and despite some obvious additions (that I mention later) I think it is sufficient to establish this as a phenomenon that might be real and that should be investigated further.

weaknesses:
There are a few weakness that I believe could be relatively easily addressed.

### Limitation 1: Testing only against a single type of adversarial attack
As far as I understand it, the authors only test the robustness against one type of an adversarial attack -- the PGD-50 = Projected Gradient Descent with 50 iterations. I think it is important to 1) add more types of attacks (including black box ones), and 2) possibly look at some standard attack suites such as RobustBench (https://github.com/RobustBench/robustbench)

### Limitation 2: Adversarial robustness vs other kinds of robustness
What about other kinds of robustness such as robustness to noise, sheer, etc. Do you see similar phenomena to what you describe under adversarial attacks?

### Limitation 3: Attack strength norms and their effect on the phenomenon and LOTOS
I would be very curious what the effect of the attack norm (L_infty vs L_2 etc) and attack size (8/255, ...) is on the observed phenomenon and the efficacy of LOTOS. I can imagine that different attack size norms might lead to different qualitative tradeoffs. Could you please be more specific with what exactly you do and how does this relate to your theory & experiments?

### Limitation 4: Weak models?
The results and improvements you present are (I think) pretty non-robust to begin with. Am I wrong about this? How do robust model perform under the same attack? If the model you are using does not have a high robustness to begin with, I believe this might lead to misleading results on what helps and what doesn't. This is because many things tend to help when we're not close to the maximum robustness, but might not generalize. **Addressing this is crucial for me to raise your score.**

questions:
I addressed the questions in the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732490923320

thread_type:
official_comment

<thread_object>
creator: area_chair_je6t

<content>
title:
Reminder - Public Discussion Phase Ending Soon

comment:
Dear PC memebers, 

Thank you for your valuable comments during the review period, which raised many interesting and insightful questions. I am also glad to see that we have lots of discussions for this paper. Now the discussion period is coming to a close, please take a moment to review the authors’ responses if you haven’t done so already. Even if you decide not to update your evaluation, kindly confirm that you have reviewed the responses and that they do not change your assessment. 

Timeline: As a reminder, the review timeline is as follows:

November 26: Last day for reviewers to ask questions to authors.
November 27: Last day for authors to respond to reviewers.
November 28 - December 10: Reviewer and area chair discussion phase.
December 20: Meta reviews and initial decisions are due.

Thank you for your time and effort!

Best regards,
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732654302896

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Revision

comment:
We thank all the reviewers for providing insightful comments and all the encouragement we received for our work. We believe addressing these concerns has made our work stronger and further showcases its effectiveness. The reviewers' comments focused on making connections to prior work, clarifying our proposed method and discussions, using more ensemble robustness methods in our experiments, and exploring other attack algorithms, including white-box attacks specifically designed for evaluating ensemble robustness. We addressed these concerns by:

- Making detailed clarification on the parts of our manuscript that raised confusion for the reviewers. In particular, we further clarified the explanation of our methods and the experiments of our work (see line 259, line 856, and line 870 of the revised version).

- Making connections to other theoretical results by prior work and explaining how the trade-off observed by us fits the theoretical framework in these works. In particular, we show the recent theoretical results in [1] regarding the trade-off of complexity and diversity of the ensemble models for transferability attacks aligns with our finding about the trade-off caused by Lipschitz continuity as this leads to less complex models which in turn lowers the diversity needed for defense against transferability attacks (see Appendix C.1 in the revised manuscript for more details).

- Adding additional experiments on other attack methods, attack sizes, and perturbation norms. Our results show the effectiveness of our proposed method in all different cases and the effectiveness becomes even clearer as the attack becomes stronger (see Appendix B.10.1 and Appendix B.10.2 in the revised version).

- Evaluating our methods against benchmarks [2] for other types of noise (e.g., Gaussian noise, spatter, etc.). Our results show that LOTOS leads to more robustness against most of the noise types and leads to the highest robust accuracy on average (see Appendix B.11 of the revised version).

- Adding the results for white-box attacks including the ones suggested by the reviewers (AutoAttack [3] and MORA [4]) to evaluate the effectiveness of our methods. These attacks are stronger attacks and the higher robust accuracy of LOTOS further showcases its effectiveness (see Appendix B.10.4 of the revised version).

- Adding new models (ResNet50 that are shown to be more robust [5]) to showcase the generality of our proposed method for models with different levels of robustness (see Appendix B.10.3 of the revised version). 

- Adding a new SOTA ensemble robustness method (DVERGE [6]) to evaluate the effectiveness of LOTOS in improving its robustness. These ensembles are much stronger than TRS against MORA attacks, but the results show that adding LOTOS can further improve their robustness by 9.5 percentage points (see appendix B.9 of the revised version).

We believe we have addressed all the concerns raised by the reviewers in their individual responses and have incorporated the updates into the revised version of our manuscript before the discussion deadline. We would appreciate it if they could review their responses to identify any remaining concerns.

[1] Yao W, Zhang Z, Tang H, et al. Understanding Model Ensemble in Transferable Adversarial Attack[J]. arXiv preprint arXiv:2410.06851, 2024.

[2] Hendrycks, D., & Dietterich, T. (2018, September). Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In International Conference on Learning Representations.

[3] Croce et al., Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. ICML 2020.

[4] Yu et al., MORA: Improving Ensemble Robustness Evaluation with Model-Reweighing Attack. NeurIPS 2022.

[5] Peng, S., Xu, W., Cornelius, C., Li, K., Duggal, R., Chau, D. H., & Martin, J. (2023). Robarch: Designing robust architectures against adversarial attacks. arXiv preprint arXiv:2301.03110.

[6] Yang, Huanrui, et al. "Dverge: diversifying vulnerabilities for enhanced robust generation of ensembles." Advances in Neural Information Processing Systems 33 (2020): 5505-5515.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper discusses robust ensemble training based on the observation that "decreasing Lipschitz continuity corresponds to increasing transferability." While the idea is not particularly surprising, and the analysis lacks novelty (a sentiment shared by Reviewer HKek and myself), one reviewer noted that this is the first time its significance has been explicitly emphasized. Moreover, the authors proposed a new method, "LOTOTS," to promote orthogonality, which has been recognized by the reviewers as a notable technical contribution. Despite some weaknesses in the theoretical analysis, the method shows promise and adds value to the field.

Overall, I believe this is a borderline paper, but I am inclined to recommend acceptance.