Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the problem of robustness evaluation for ensemble-based defense methods. Authors argue that existing SOTA white-box attack methods may overestimate the robustness of ensemble defenses because of two reasons, i.e., (1) the ensemble-forming strategy used in ensemble defenses could lead to gradient obfuscation and (2) attacking would fool majority sub-models, while the ensemble model's output may still correct. In order to provide a more effective and reliable way to evaluate ensemble defenses, this paper proposes a novel attack method that adaptively assigns different weights for different sub-models based on their contributions to the attack gradient. To evaluate the effectiveness of the proposed attack method, authors compare it's attack performance with several SOTA white-box attack methods on four ensemble defenses under three ensemble-forming strategies. Experimental results verify that the proposed method outperforms baseline methods with large margin and converges much faster.
strengths_and_weaknesses: Strengths:

1. This paper investigates the robustness of ensemble defenses under multiple ensemble-forming strategies and provides extensive empirical results. 
2. The proposed method achieves much improved success rate on ensemble defense methods and convergence rate than SOTA attack methods among different experimental settings, which sufficiently demonstrate the superiority of the proposed method.
3. Authors make the source code available and also maintain a leaderboard of ensemble defenses under various attack strategies, which can boost the research on this direction.


Weaknesses:

Although experimental results verify the effectiveness of the proposed attack method, the intuition/motivation of the designing of this method is not quite clear. Specifically, it's not clear why the sub-model reweighing technique is able to produce a stronger attack method than SOTA white-box attack methods for ensemble defenses. For more detailed weaknesses, see the following questions.
questions: 1. Is it possible to verify the two reasons that cause the existing SOTA white-box attack methods overestimate the robustness of ensemble defenses through either empirical study or theoretical analysis? Although the two reasons authors pointed out in this paper sound reasonable, it would be better if more convincing evidence can be provided to further verify these two reasons.

2. Some observations from the experimental results are very interesting. (1) As shown in Table 2, larger ensembles lead to reduced robustness when sub-models are adversarially trained facing with any attack methods listed in the table. However, ideally the adverasrially trained sub-models could be more robust against attacks and, hence, the larger ensemble of robust sub-models should provide improved robustness performance, comparing with small number of sub-models. Can authors provide some understanding about this observation? Does this observation reveal some possible drawbacks/limitations of applying the ensemble idea to improve model's robustness? (2) Since the majority votes ensemble strategy requires attacks to break at least half of sub-models, intuitively the robustness of ensemble defense methods based on the majority votes strategy should be higher than other two ensemble strategies. However, the experimental results give an opposite observation. Can authors give some discussion about this observation? Does this observation caused by the simplicity of the majority votes strategy (e.g. comparing with other two strategies, the majority votes only focuses on the final outputs and ignore some useful information such as logits)? 

3. Some notations are not clear. (1) Does the three star signs with pick color in Figure 1 and 2 represent the attack performance of the MORA method on some specific ensemble defense method under three different ensemble-forming strategies? If so, it would be better if each star can be denoted more specifically. (2) Does $\text{MORA}^{mt}$ represent the multi-targeted version of MORA? If so, it would be better if some introduction about this variant of MORA can be provided in the beginning of the experiment part. (3) What's the meaning of "Nominal" appeared in both Figure 1&2 and Table 1&2? Is it the robust accuracy reported in corresponding original papers? 

limitations: The limitations and potential societal impacts are discussed in the appendix of this paper. As pointed by the authors, although the attack method proposed in this paper may be used by a malicious party, an effective attack method can boost the development of robust DNN models in the future.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes an attack targetting the model ensemble defenses. The main idea is to assign different weights to different models. The values of the weights depend on the models' contribution to outputs. The results demonstrate that the proposed attack outperforms existing ones.
strengths_and_weaknesses: Strengths: The idea is simple and easy to understand. The experiments show 

Weaknesses: Overall, the contributions are incremental and the application scenario restricts to white-box ensemble defenses.
questions: 1. Can MORA be extended to black-box attack scenarios or be used to attack other defenses? The application scenario is limited. It would be more exciting to see some extensions.

2. There are some typos in the existing version. For example, should it be 'A successful attack happens when $k^{[E]} < 0$'?
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces MORA, a novel adversarial attack against ensembles. The attack is motivated by the fact that existing white-box attacks over estimate the robustness of ensembles, either due to gradient obfuscation or the difficulty of simultaneously fooling diverse gradients. MORA is designed to adapt its focus to the models in the ensemble. Experiments are performed on CIFAR-10 and -100 with ResNet20 in white-box setting. MORA is compared against other attacks and multiple ensemble defenses.
strengths_and_weaknesses: Strengths:
- The design of the loss function used for MORA seems sound and well-motivated by the causes of attack failures on ensembles.
MORA incorporates state-of-the-art community practices for thorough robust evaluation.
- Under small perturbation budgets, MORA seems to find adversarial examples that other attacks do not, thus providing a better estimate of robustness for the given ensembles.
- The experiments are extensive, comparing the proposed attack against state-of-the-art white-box attacks and ensemble defenses, under different aggregation rules and ensemble sizes. An ablation analysis is also provided.
- Very well written paper.

Weaknesses:
- MORA seems to lose its advantage over other attacks under certain regimes. Results are almost on par with A$^3$ and the nominal values in Tab. 2.
- It is standard to use perturbation budget $\epsilon = 8 / 255$ on CIFAR-10. However, the main results table in the paper (Tab. 1) uses $\epsilon=0.01$, just under a third of the standard value. Results for $\epsilon=0.03$ are included in the appendix, where we see that the gap in ensemble robustness estimation significantly narrows between general-purpose white-box attacks and the proposed MORA. The numbers are on par for most rows, diminishing the claims of the paper.
- Despite the extensive experiments, some aspects of the evaluation are limited. Particularly, focusing only on ResNet20 architecture, CIFAR data and $L_{\infty}$ robustness does not convince beyond a doubt that MORA is the new adversarial standard for ensemble evaluation.

Minor:
- It would be a plus to start by stating the threat model. It become apparent later in the paper that the assumption is white-box access althroughout.
- L135: there exists -> there exist
- L186: Base on this property -> based on this property
- L216: attackss -> attacks
questions: - In Alg. 1, it seems the adversarial sample obtained after a maximum of $I$ iterations is used. What are the odds that it is the strongest sample? Should the best sample out of the $I$ iterations not be used instead (see also [1])?
- Tab. 2 presents results of adversarially trained models ensembled using Dverge defense. What happens when attacking with MORA an ensemble of (individually) adversarially trained models without ensemble defenses?
- Does it makes sense to compare attacks convergence in terms of number of iterations (Fig. 4)? Do they perform the same computations in each iterations? Does it also make sense to compare convergence over computation time?
- Are the results presented for CW attack also under $L_\infty$ norm?

References:
- [1] Pintor M. et al. Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples. https://arxiv.org/abs/2106.09947
limitations: The limitations of the threat model are addressed in the appendix.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors propose an adversarial attack (MORA) against model-ensemble-based defense by reweighing the importance of sub-models. The authors first investigate the factors which cause existing white-box attacks to fail against several state-of-art model-ensemble defenses, that is gradient obfuscation. Then, the authors propose a strategy for crafting adversarial samples by conducting optimization on several 'important' sub-models within ensembled models.  Motivated by such a strategy, the authors propose a MORA loss, which incorporates a sum of the weighted variant of sub-model logits, in order to
expose sub-model logits with adaptive reweighing. Through extensive empirical evaluation, MORA is proved to be able to bypass existing SOTA model ensemble defense approaches. The authors further compare MORA with several state-of-art white-box attacks(PGD, C&W attacks) in both the dimensions of efficiency(convergence rate) and attack efficacy (attack success rate).  The results show that MORA can significantly outperform existing white-box attacks against model ensemble defense.
strengths_and_weaknesses: =====Strengths=====
I. This work is well-structured and easy to follow.

II. This work proposes an effective method MORA and solves an interesting problem.

III. The evaluation is comprehensive and convincing. 


=====Weakness=====

I have two main concerns about this work.

I. Limited Novelty and Practicality. This work seems to be a kind of 'defense-aware attacks' approach which built upon the existing defense approach(i.e., model-ensemble defense). The proposed attack can only bypass a specific category of defense, which limits its practicality. Moreover, the proposed attack is evaluated under the white-box settings, however, as mentioned in the abstract and introduction, the ensemble-based defense approach is mainly used against black-box attacks (i.e., transferable attacks).  In general, there's no evidence to show that ensemble-based defense is or performs close to the state-of-art defense against white-box adversarial attacks. Therefore, the motivation for developing a white-box attack bypassing ensemble-based defenses that are mainly used against black-box attacks is unclear to me. 

II. Evaluation. Indeed, I am convinced that MORA can outperform existing white-box attacks against ensembled-based defense through empirical evaluation. However, I think selecting current white-box attacks for comparison is not enough. This is because existing white-box attacks are prone to gradient obfuscations, which is also claimed in your introduction section and other related work.  In contrast, I think several decision-based attacks (hop skip jump attack, Sign-Flip attack.) is more suitable for this scenario.  Even though these decision-based attacks are built under black-box, they can also be applied in the white-box settings. I have expertise in such decision-based attacks and observe that decision-based attacks can bypass gradient-obfuscation defense. I think you should also compare MORA with them. 






questions: I have several questions:

I. What's the motivation for developing a white-box attack against SOTA defense methods against black-box attacks?

II. Can you compare MORA with several decision-based attacks?


limitations: See above.
review_title: null

Meta-review: metareview: This paper introduces a new way to generate adversarial examples for defenses that work by ensembling over many different independent predictors. The reviewers all liked this paper and believe that the results were a useful improvement on prior work. While the margin of improvement is not massive (and it is a smaller gap as perturbations get larger) but the improvement appears correct and is well explained.
confidence: Certain
Decision: Accept