MORA: Improving Ensemble Robustness Evaluation with Model Reweighing Attack
Authors: Yunrui Yu
Xitong Gao
Cheng-zhong Xu
Venue: NeurIPS 2022 Accept (2022)
Keywords: adversarial attack
ensemble adversarial defense
model robustness

Abstract: Adversarial attacks can deceive neural networks by adding tiny perturbations to their input data.  Ensemble defenses, which are trained to minimize attack transferability among sub-models, offer a promising research direction to improve robustness against such attacks while maintaining a high accuracy on natural inputs.  We discover, however, that recent state-of-the-art (SOTA) adversarial attack strategies cannot reliably evaluate ensemble defenses, sizeably overestimating their robustness.  This paper identifies the two factors that contribute to this behavior.  First, these defenses form ensembles that are notably difficult for existing gradient-based method to attack, due to gradient obfuscation.  Second, ensemble defenses diversify sub-model gradients, presenting a challenge to defeat all sub-models simultaneously, simply summing their contributions may counteract the overall attack objective; yet, we observe that ensemble may still be fooled despite most sub-models being correct.  We therefore introduce MORA, a model-reweighing attack to steer adversarial example synthesis by reweighing the importance of sub-model gradients.  MORA finds that recent ensemble defenses all exhibit varying degrees of overestimated robustness.  Comparing it against recent SOTA white-box attacks, it can converge orders of magnitude faster while achieving higher attack success rates across all ensemble models examined with three different ensemble modes (i.e, ensembling by either softmax, voting or logits).  In particular, most ensemble defenses exhibit near or exactly $0\%$ robustness against MORA with $\ell^\infty$ perturbation within $0.02$ on CIFAR-10, and $0.01$ on CIFAR-100.  We make MORA open source with reproducible results and pre-trained models; and provide a leaderboard of ensemble defenses under various attack strategies.

MORA: Improving Ensemble Robustness Evaluation with Model-Reweighing Attack
Yunrui Yu"
Xitong Gao'
Cheng-Zhong Xu
State Key Lab of IOTSC, University Macau
Shenzhen Institute of Advanced Technology.
State Key Lab of IOTSC, University of Macau
Chinese Academy of Sciences
Macau SAR, China
Shenzhen, China
Macau SAR, China
yb974 74744455 um. www.mmm.uu.......
xt gt..galsiit.com ac.cn
czxu8 um, edu, mo
Abstract
Adversarial attacks can deceive neural networks by adding tiny perturbations to their input data. Ensemble defenses, which are trained to minimize attack transferability among sub-models, offer promising research direction to improve robustness against such attacks while maintaining high accuracy on natural inputs. We discover, however, that recent state-of-the-an (SOTA) adversarial attack strategies cannot reliably evaluate ensemble defenses, sizeably overestimating their robustness. This paper identifies the two factors that contribute this behavior. First, these defenses form ensembles that are notably difficult for existing gradient- based method to attack, due gradient obfuscation. Second, ensemble defenses diversify sub-model gradients, presenting a challenge to defeat all sub-models simultaneously, simply summing their contributions may counteract the overall attack objective; yet, we observe that ensemble may still be fooled despite most sub-models being correct. We therefore introduce MORA, a rewighin attack to steer adversarial example synthesis by reweighing the importance of sub- model gradients. MORA finds that recent ensemble defenses exhibit varying degrees of overestimated robustness. Comparing it against recent SOTA white-box attacks, it can converge orders of magnitude faster while achieving higher attack success rates across all ensemble models examined with three different ensemble modes (i.e., ensembling by either softmax, voting or logits). In particular, most ensemble defenses exhibit near exactly 0% robustness against MORA with go perturbation within 0.02 on CIFAR-10 10, and 0.01 on 1IFAR-100. We make MORA open source with reproducible results and pre-trained models; and provide a leaderboard of ensemble defenses under various attack strategies
1 Introduction
Many safety-critical applications, such as autonomous robots 34 self-driving S search en- gines [24], etc. are becoming increasingly powerful and reliant on deep neural networks (DNNs). Despite the monumental success of DNNs on these applications, they are highly susceptible to adversarial examples-
attacker can add tiny delibrate perturbations the input data, misleading the model into giving incorrect results 23 9. Such adversarial attacks could pose a significant threat to the safety and reliability of deep learning applications.
To mitigate this threat, many defense strategies [7 33 based on adversarial training 17 have been proposed to improve model robustness. Adversarial training, however, gains robustness at the
Xitong Gao and Yunrui Yu contributed equally to this work.
Conespondence to: Cheng-Zhong Xu (c2. xuê um. edu.mo).
https://githuut com www.lllaaata//aaaaa /mora
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Nominal
CAA
Nominal
CAA 8''
100
40
PGD
PGD
Robust Aocuracy (%5
CReW
MORA
Robust Accuracy (%6
CRWW
MORA
Attack sueress rate (%6
40-
A.A
AA
80-
30-
16-
46-
40
PGD-ADD
22-
SEORA
MORI
Pon--Deeg
20-
x2-
MORA
SCORA
20
MORA-ADF
MORA (CAL
40-
MORMA-
15
# Iterations (k)
# Iterations (k)
Remmining zobust sub-models
(a) GAL
(b) Dverge.
(c) Robust sub-models vs. attacks.
16 with strong baselines are neither efficient in the number of model forward/backward passes, nor reliable in the estimation of ensemble robustness when compared with MORA. GAL [4] and Dverge [30] defenses are trained on CIFAR- 10 with 8 sub-models. We used f attacks within = 0.01, "Nominal" is self-reported. @ MORA can successfully fool logit-based ensembles (ADP 19 GAL and Dverge) even with the majority of their sub-models giving correct outputs A*AA- > B" means using A to attack B for up to 100 iterations).
Figure 1: a Existing attacks 17r
18
expense of model accuracy on clean natural images 2
Ensemble defenses 19
30 3 have
thus emerged combine multiple predictions from independent sub-models. The intuition is that an ensemble of models can often lead to higher accuracy, while learning to stop adversarial example transfer among sub-models may improve robustness against adversarial attacks. This approach could potentially offer a promising research direction to improve model robustness while preserving high accuracy on natural inputs.
Yet surprisingly, under the white-box threat model, existing state-of-the-art (SOTA) adversarial attacks with strong performance conventional DNN models performed poorly on ensemble models, sizeably overestimating their robustness (Figures La and This also suggests, to some extent, that ensemble defenses may rely on two potential design flaws below that cause obfuscated gradients M. i.e., they are either deliberately non-differentiable, or give no useful gradients, thus inducing overestimated robustness:
(a) Gradient obfuscation via ensemble-forming strategy. They typically form ensembles by averaging probability vectors (softmax) of sub-models, and softmax operations can easily cause gradient obfuscation. While the model's actual robustness is pertinent to the strategy used to form an ensemble, this indicates that gradient- gradieet-based attacks have to also leverage this effectively.
(b) Gradient diversificatio. Motivated by the reasoning that a majority of sub-models may need be fooled for successful attacks, they learn to reduce adversarial transferability among sub-models, often via gradient diversification. This intuitively causes sub-models to counteract each other, averaging to small or inaccurate overall gradient. Attacking only the ensemble loss would fool most sub-models, but the ensemble may remain still correct; conversely, it is actually possible to fool an ensemble, despite the majority of its sub-models giving correct predictions (Figure Ic
perceivable that the practical evaluation of ensemble robustness cannot be solely done by treating such models holistically. To this end, this paper introduces MORA, reweiging attack, adaptively adjust the importance of sub-models in attack iterations. Sub-models are reweighed according to their respective "ease of attack", which is in turn evaluated by the gradient of the difference of ensemble classification outputs w.E.t. the ones of individual sub-models. Pushing the limits of the current SOTA in ensemble robustness evaluation, it draws inspiration from recent effective attack tactics, e.g., momentum 7 5 step size schedule [6] loss
From the above observations,
normalization
32 and multiple targets 5. 25
We summarize our contributions:
This paper presents the first extensive study on the robustness of ensemble defenses under multiple mnsemble-forming strategics.
* By reweighing the importance weights of sub-models to steer adversarial example synthesis, we show that gradient-based attacks on ensemble defenses can often be orders of magnitude faster, while enjoying higher success rate.

---

Empirical results on a wide variety of different ensemble defenses show that MORA out- performs competing attacks in both performance and convergence rate. Finally, this paper provides extensive ablation of its components and sensitivity analyses of hyperparameters.
To our best knowledge, MORA is currently the strongest attack against a wide range of ensemble defenses. We make MORA open source with reproducible results and pre- pre-trained models; moreover, we maintain a leaderboard of ensemble defenses under various attack strategies.
2 Preliminaries & Related Work
2.1 Adversarial Attacks
An adversarial example adds small perturbation, typically bounded small value with l norms, to
the original image, such that the model under attack can be deceived into giving incorrect results. The
advent of adversarial attacks [ has piqued the interest of deep learning practitioners, and revealed
security concerns of deep learning [26] 21]. improved GAN training [2] transfer learning (28 [
and DNN interpretability (22). etc. Formally, assuming a defending classifier f I - R taking
an input image x E I 00,
CxXXW as input, and arg max f(x) predicts the correct class label
E C, then an attacker attempts to find an adversarial example & in the set:
( Ar,x : arg max [(x # y)-
Here, X E .Aex constrains the adversarial example x to be within both the input space I and small c-ball of e (i-distance from the original image x, or equivalently ||x- - S c. Satisfying the condition arg max f(x) & y means that fx) fails to give the expected correct classification y. We focus on the 8or white-box threat model commonly considered by the defenses examined this paper, which grants the attacker completely access to the internals of the defender, including, for instance, its model architecture, parameters, training algorithms, etc.
One of the popular and effective white-box attacks used by many defenders to evaluate their robustness projected gradient descent (PGD) [17], which finds adversarial examples by maximizing the classification loss with gradient descent:
x (X; + a sign V L(f(x,), v))).
(2)
where C is typically the softmax cross-entropy (SCE) loss used to train the model, or is the step size, and we let the initial &o & (x + (4). The projection function P.,x(v) constrains its input v to be within the feasible region Ar x. and finally μ ~ U4 (-e,e) is uniformly distributed noise bounded by [-e, Besides PGD, C&W [ is also a gradient-bbrr -bssed attack which, instead of projection, indirectly constrains the search space by regularization.
As PGD gains popularity, many defense mechanisms rely on it to evaluate their robustness. Unfortu- nately, Auto/Atack (AA) (3] finds that many of the defenses may inadvertently break "GD-based attacks, which result in drastic overestimation of their robustness, and proposes to combine an ensemble of diverse attacks minimize robustness overestimation. LAFEAT 32) learns to leverage intermediate layers of the DNN, and shows that attacking multiple layers can produce stronger attacks, but unfortunately it cannot be applied ensemble defenses. Adaptive Auto Attack (A3) [16] improves attack success rates by using the gradient directions to preseribe a more effective initial random perturbation. As defenders may design mechanisms to circumvent existing attacks, Adap- tive attacks (25) manually tailor specific attack strategies for an extensive set of defenses. Finally, Composite Adversarial Attacks (CAA) [ further combine large selection of attack methods, and use a genetic algorithm to learn an optimal attacking sequence. In comparison, MORA is a unified approach which uses only one attack algorithm, does not require a compute-intensiin learning procedure, and yet it still achieves fast and SOTA estimation of ensemble robustness.
2.2 Defending Against Adversarial Attacks
Defending against adversarial attacks can be defined as a saddle-point problem to minimize the training loss on adversarial examples [77 with samples (x, y) drawn from the training set:
ming E(x,v) [maxxE C(f()) v)]
(3)
3

---

where L is the training loss, e.g., the SCE loss. A direct optimization- approach to approximately solving the above problem adversarial training 17,, i.e., to train the DNN model with its own adversarial examples. Training DNNs to be robust is, however, a challenging endeavor. First, it may be much more compute intensive as training examples are typically generated with PGD [7], requiring a few forward/backwart passes of the DNN. Second, to avoid overfitting, it requires stopping training early, a much larger size of the training set A. and using improved data augmentation [20] or generated data [TO]. Thirdly, as noted by other literatures S [2] currently other design choices can rival the robustness provided by adversarial training, and notably, many defense strategies are considered harmful to model robustness 25 Finally, the resulting models often cannot achieve high clean accuracy
2.3 Ensemble Defenses & Ensemble-forming Strategies
Ensemble-based defense techniques may pave an alternative path to address the challenges of adversarial robustness, as they could potentially work around the above limitations of adversarial training. Adopting the theme of minimizing adversarial example transferability across sub-models, each ensemble defense proposed unique solutions. ADP 19 increases the orthogonality of non- maximal class logits among sub-models to encourage diversity. GAL [14] minimizes a gradient alignment loss, which directly reduces the posine-similarii between sub-models. Building on top of this, TRS 31 further regularizes the smoothness of the loss function, gradient orthogonality with smoothness may further diversify sub-models. Dverge KC instead uses the adversarial examples of a sub-model to train another sub-model, thus lowering transferability. Ensemble defenses also particularly interesting, as they are the last line of defense against even the strongest existing white-box attacks without resorting to adversarial training, showing a certain degree of robustness.
Besides the above mechanisms for training a successful ensemble defense, there exists different ways to combine sub-model predictions. Let us assume that an ensemble defense trains M uubmmoeels, fm I + RK for m E [1 MM, an ensemble fF. I - IR K thus forms final classification result by combining individual decisions from the sub-models, namely:
fπ(x) N3 Eme[i:a)] ens(Jm(x)))
(4)
where ens is the ensemble-forming operator. In this paper, we investigate ens E {softmax, wta, id}, where the potential choices respectively denoting forming an ensemble from sub-model outputs fm (x) by either summing predicted probabilities (evaluated with softmax), or majority votes (using wta, the winner-take-all all operator), or simply summing the logits (with id, the identity operator). Defending ensemble methods [19 [4] 30, 31] tested in this paper all employed the softmax based strategy to report their robustness. Methods that are exceptions to these options exist, for instance. ECOC (29 allows sub-models to produce binary predictions, and use error correcting codes based on the Hamming distance to combine the predictions into classification outputs. This approach is unfortunately not robust, and the added complexity is error-prone and may harm robustness 25
Moreover, as the voting (wta) strategy is non-differentiable, an attacker can soften it approximately using a softmax operation with temperature where we used T = 0.1 universally:
softwta, (x) - =softmxxxx/ (x/T).
3 The Model-Rewweighnng Attack (MORA)
3.1 Problem Formulation & High-Level Overview
As discussed in Section T existing ensemble defenses may obfuscate gradients with the ensemble- forming mode and gradient diversification, such that the final loss of the ensemble model can no longer provide effective signals for gradient descent. It is therefore desirable to find alternative L. to the original SCE loss _ on the ensemble, such that for a given number of iterations 7, the original Lace loss can be maximized:
max c L0 (&1,y) where Xn = Pxxxxxx>> + 11),
(6)
x PGD(C(FE(x.), (t))C((((()),,,/)x(x,.),,,.)),,)
and PGD(-) denotes PGD step along the gradient of loss function L(fe(&)), fl:wa] (xx), v), which not only takes the ensemble predictions fx( input, but can further utilize sub-model predictions

---

fi:mM] to guide the PGD iterations. The challenge at hand is, therefore, to find suitable L which can generate attacks on ensemble defenses efficiently and effectively.
fi
MORA
norm
B
Sub-model Reweighing
fz
x/2
Amora 3,7
x
1 - 3
fM
ens
morm
Sub-models
Ensemble- formingg
strategy
Figure 2: high-level overview of MORA. Existing attack methods focus on maximizing the Lace of the ensemble predictions z MORA further introduces a model-reweighing mechanism, which takes as input sub-models predictions z(,M0, and the ensemble output z²",, forming combined loss Cmora. MORA generates adversarial examples by maximizing the resulting loss.
MORA aims to provide a potential optimization route towards the above problem formulation. Namely, in addition to the original output of the ensemble fF(X), we leverage the sub-model predictions (&) to facilitate the optimization. By way of illustration, Figure shows a high- level overview of the model-reweighing attack, where we compliment the ensemble loss, with newly added sub-model reweighing loss 4 as an auxiliary attack vector alongside the original objective. Not only can the new loss bypass the ensemble-forming strategy to work around its obfuscated gradients, but it further exploits information present in the individual sub-model and ensemble predictions to steer the direction of adversarial example synthesis.
3.2 Adaptive Sub-model Importance
Before we begin, assume that (!m) A [m) represents the mh sub-model output, and let z!!" denote the corresponding logit label I. We define the difference of logits (DL) kimil A 2/11) 9 which the difference between the predictions of the ground truth 2|n" and the maximum of the remaining classes z|!" A maX zi"",, where C/y y is the set of all class labels except y. Similarly, we let (M] A fr(x), and z,th and [ill be the respective variants of the ensemble prediction. It is notable that a successful attack happens when kin < O, and similarly klm) < 0 means the m sub-model is producing incorrect classification.
Im]
Ensemble defenses tend to diversify sub-model gradients, for instance, ADP minimizes the aesine-similarity (Vl(z!")), V((z(")) among each loss function gradient pairs of sub-models (() and (() Their intuition is that it may lower transferability among these sub-models, such that attacks with the overall gradient of the ensemble, i.e., V "े Eme(l:M) V((z!")). are becoming less effective in misleading all sub-models simultaneously. as individual gradients in V((z!")) are encouraged to be orthogonal to each other. To this end, we propose to reweigh the importance of sub-models, by instead considering the modified gradient:
Vi(ZIE) MT Eme(i:M (z (m) ((z))))
where X!!|l assigns weights to important sub-models contribute more heavily to the attack gradient.
While the adversarial examples of the ensemble could present a challenge to discover, individual bob-models are weak defenders which can be easily defeated. Based on this property, we propose to i.e., weigh sub-models importance with the rate of change in k w.n.t that of i.e,, sub-models would be given higher weights if attacking it would bring significant change to the ensemble's 's prediction. Following this idea, for ensemble-forming strategies (softmax, voting, logits), we rewrite |IB) as function of k!!",,, where the term below can become a function of k(m):
[ ens( (m))) ens(z (m)) ens(z mml 2/ ens(z!") (m) z/("))g
(8)
ens(k!ml) ... - ens . 2 hm (k!ml))
5

---

Algorithm The MORA white-box robust evaluation for ensemble defenses.
1: function MORA_ Attack(/..a) x, 3,
2: Xa + Pe, (x + =u) where u ~ ~ (-e,e)
Random init
3: μ40 <-0 0)
4:
for i E 10:I-1 1-1 do
5:
z!'l f- fm((x: for all m [1 M]
Sub-model predictions b Ensemble prediction
6:
= +- 1
M7 m] ens(z (z!
7:
A <- // z
8:
if m < 0 then return &
Successful attack Sign-gradiog with the MORA loss
9:
9i+1 - -iig( v 3.T (z(IM) 2111
10:
a + <(1 + cos(ts/T))
Cosine step-size schedule D Iterative update .... with momentum
11:
Mi+1 + Pex (1:: + +09++1)
12:
X,+1 + Pex x (x; + (Mi+1 Xx) + (1 v) (x: x4-1))
13:
end for
14: return X
Give up after iterations
15:
end function
The weights are thus defined follows: [B] (pefmd)
Bl., (A (klmn))
Bh.., (R(m))
Alril (z ml)
M Σ
(9)
5kim
8klm
M
8klm)
While it is possible compute the weights using gradient back-proyy back-propagation. we can simply derive the following closed-form solution of the weights for each of the three ensemble-formin strategies. For wta, we use the softened version of wta as defined in (5) and can derive the weights as follows:
Xpmll (zl"") 1 [k > 0] (letach(1)yss
Sy 8z)): where
softmax (x) (10)
Here 1[k]m] > 0] is the indicator function that equals 1 if k/m) > 0, or 0 otherwise, effectively stopping the attack the mhh sub-model upon success, and the detach operator admits backward propagation to its input. In the case of using sums sub-model softmax outputs to form an ensemble softmax, it is special case of softwta, where the temperature coefficient can be fixed at r = 1. Finally, when ens = id, i.e., forming ensembles by summing logits, (z!")) simply reduces to 1 | >> 0) for the m sub-model.
decision, i.e,,
3.3 The MORA Loss
For reference, defenses mechanisms we examine in this paper aim find which maximizes the SCE loss (z (9) yy. to evaluate the ensemble robustness. The MORA loss improves this further by proposing two additional modifications to the untargeted loss function used attack ensembles:
Cmora B,T
2 1:M
z [s] y) A Ccce norm ( Eme[uMM (z!m() mm) + (1 B) (F)),
(11)
First, it additionally introduces sum of the weighted variant of sub-model logits, in order to expose sub-model logits with adaptive reweighing described in Section 3.2 Second, ,3 interpolates the importance of the newly added auxiliary logits and the original ensemble logits. Finally, inspired by the effective surrogate loss in 32] it further normalizes the logits by their respective DL using:
norm 2
1 1/z 2g > 0) z/ /eteach (z., -zg)
(12)
Finally, the targeted variant of the MORA loss simply replaces with t where is the intended target.
3.4 Improving the State-of-the-art
While the new Cmora loss is highly effective against ensemble defenses test in this paper, we strive for further advances in MORA's ability to generate faster and better adversarial examples. Inspired by recent publications, we borrow ideas from related adversarial attack tactics, which includes adopting a cosine step-size schedule [16], momentum 7 5. random restarts 25 and multiple target
attacks (5] 25). We provide the overall algorithm Algorithm D. which computes an adversarial image , as return, by taking input the sub-models /[:://- natural image x, ground truth label interpolate between the auxiliary logits and the original, controls the temperature, momentum 32 [5), perturbation bound, and finally the maximum number of iterations I.
0.75 following

---

Table 1: Comparing accuracies among iterative methods 17 learned attacks (CAA [B] AutoAt- tack (AA) [ adaptive auto attack (A") 16. and MORA across various ensemble defense strategies under ensembling modes (softmax, voting and logits), and = 0.01. The "Complexity" row shows the worst-case complexity in iteration counts. The " " column shows the accuracy overestimation from rerteppprrcc/eeproduueedddd "Nominal" values to MORA Baselines with are reproduced with
source code. All results re-run 5 times and within =0.05% standard deviation.
Defense Complexity
Clean
Nominal
PGD
CW
MORA
A
AA
CAA
MORA unt
#
A
500
500
500
4.9k
1.8k
1.4k
Softmax
3
92.88
29.12
5.98
7.72
0.59
2.12
0.98
3.34
0.34
28.78
ADP
93.34
25.14
7.10
8.70
0.97
3.62
2.18
4.25
0.67
24.47
8
93.48
20.20
9.22
9.59
1.70
4.84
3.94
6.04
1.32
18.88
91.99
47 /242
44.49
40. 17
25.77
33.36
30.58
32,98
25.26
22.16
Dverge
92.38
55.72
54.61
52.83
40.02
48.41
43.29
46.65
39.50
16.22
91.65
59.63
59.13
58.25
55.68
57.29
56.71
56.89
55.57
4.06
3
89.41
19.48
8.13
11.57
0.67
0.70
0.85
1.00
0.51
18.97
GAL
90.93
41.38
37.59
35.52
17.45
26.94
23.90
25.11
16,05
25.33
8
92.45
56.31
53.39
52.56
28.71
36.51
37.46
35.30
27.44
28.87
70.02
19.71
14.01
10.87
8.11
8.72
8.46
9.75
7.60
12.11
TRS
69.00
23.17
15.91
15.28
12.67
13.22
13.20
13.78
12.47
10.70
73.01
23.64
18.02
17.59
16.22
16.51
16.73
15.64
8.00
Voting
91.84
41.62
11.84
0.64
3.06
6.13
8.29
0.29
41.33
ADP
93.13
40.291
12.42
12.05
1.17
10.13
0.67
39.67
8
93.28
30.10*
12.53
10.50
3.16
6.11
9.21
1.69
1.65
28.45
91.72
39.05!
31.48
28.00
23.57
24.95
24.98
27.65
22.91
16.14
Dverge
92.18
49.367
44.28
42.28
35.06
39.15
39.20
40.85
34.46
14.90
8
91.58
56.851
53.72
52.35
47.12
50.58
50.04
51.15
46.10
10.75
3
89.09
21.481
5.85
7.64
0.87
0.71
0.56
0.78
0.35
21.13
GAL
90.77
37.32
29.33
27.62
12.96
18.55
20.82
22.17
12.25
25.07
92.37
55.397
49.56
48.02
21.66
30.35
31.39
30.93
20.16
35.23
3
68.95
13.79
10.19
8.71
5.73
11.89
6.69
8.08
5.44
8.35
TRS
5
68.31
15.36
12.71
11.88
8.82
10.08
10.30
11.21
8.38
6.98
8
72.05
17.00
14.57
13.48
11.39
11.99
11.85
12.80
10.69
6.31
Logits
3 92.86
3,44
0.87
2.05
0.48
0.25
0.22
0.31
0.21
3,23
ADP
5
93.48
4.57!
1.97
4.24
1.12
1.00
0.97
1.09
0.89
3.68
93.38
5.397
3.57
4.77
2.13
2.20
2.05
2.11
3.46
3 92.19
38.31
37.99
38.60
36.89
36.94
36.96
37.07
36.84
1.47
Dverge
5
92.28
50.771
50.57
51.28
49.65
49.72
49.66
49.75
49.59
1.18
8
91.73
61.06*
60.95
61.51
60.52
60.59
60.52
60.55
60.49
0.57
3 89.50
15.471
10.01
10.53
0.52
0.02
0.02
0.08
0.03
15.44
GAL
5
90.93
36,36*
33.97
35.14
22.24
33.43
20.24
21.66
19.40
16.96
8
92.54
56.08
53.67
54.69
31.52
40.90
30.89
31.17
30.66
25.42
69.72
13.31
13.06
13.80
12.11
12.13
12.16
12.21
12.07
1.24
TRS
68.90
16.89
17.34
15.88
15.86
15.90
15.95
15.82
1.07
72.24
19.40
19.20
19.67
18.20
18.18
18.27
18.34
18.17
1.23
Experimental Results
We compare MORA against SOTA attacks for wide range of ensemble defenses under three ensemble-forming strategies (softmax, voting, and logits). We use pre- treitreined ResNet-20 [12] models from ADP 19 Dverge 30), GAL [14] and reproduced TRS Bi] using the same architecture with official source code, as pre-trained models were unavailable. Our robustness evaluation considers the 8or white-box attacks on the CIFAR-10 test set [15]. with perturbation
0.01 unless specified. The full comparison results can found in Table I larger e comparisons, and similar results on CIFAR- 100 models are in Appendix A We provide our key observations below.

---

Traditional attacks may fail to break through gradient obfuscation. We reproduce two traditional white-box attacks, i.e., projected gradient descent (PGD) [7 and C&W [3] with 5 random restarts, each with maximum of 100 iterations, giving a total of 500 iterations. PGD uses fixed step size of c/4. For fair comparison, MORA with 500 iterations sweeps 3 E (0. 0.25, 0.5, 0.75, 1), with each A up to 100 iterations. Even with 500 iteration budget, it is clear that PGD and C&W may substantially overestimate robustness, especially when tested under the softmax and voting ensemble- forming options, and MORA can work around this obstacle thanks to its attacks sub-models.
Diversified gradients can hamper even integrated attacks with large arsenals. Moreover, we test the defenses against recent integrated attacks with SOTA baselines on robustness evaluation, namely Adaptive Auto Attack (A (ió], AutoAttack (AA) [5], and Composite Adversarial Attacks (CAA) T8 which comprise large arsenals of various attack strategies. We reproduce CAA fol- lowing
T to search for the attack policy before evaluating the defending models. Note that its computational complexity is thus much higher than the other attacks, but we only report its test-time complexity. In particular, while they enjoyed much higher success rates than PGD and C&W some defenses render their tacties ineffective. We observe, e.g., sizeable robustness overestimation on ADP 19 under softmax and voting, which explicitly diversifies sub-model gradients. As MORA can dynamically re-adjust sub-model importance w.r.f. their "case-of-attack"" it performs substantially better with much fewer iterations. In addition to the earlier 500 iterations, the multi talti-targeted MORA" mi targets the remaining 9 class labels with 100 iterations for each label and / fixed at 0.5. Others also use multi targeet attacks along with respective tactics.
Robustness of most sub-models vs. robustness of ensemble. We find that robustness of a majority of sub-models (fooling 3/s for softmax and 2/8 for logits) usually do not translate to the overall robustness of the ensemble (Figures IC and Bap. As voting requires breaking 1/2 sub-models simultaneously (Figure 3b), it is perceivable that using voting may give rise to a higher overall robustness. Yet surprisingly, for most defending ensembles, voting performs worse than softmax and logits.
Ensemble-forming strategies may give false sense of security. On one hand, softmax and voting strategies exhibit substantially larger overestimated robustness (up to 40%) than logits. On the other hand, in stark contrast to the proposed use of softmax from [4 10 30 31, we find summing by logits can form ensembles that are notably more robust than the other two (Figure Bc), while attackers only needs to successfully deceive a few sub-models (referring back to Figure Ic).
Up to 60x faster convergence under 500 iterations. Figure 4 compares the convergence speed of MORA against AA losses, C&W, and PGD on defending ensembles. MORA converges substantially faster than the other attacks, using only up to 31 steps to match AA losses with 500 iterations.
Ensemble defense mechanisms may be at odds with robustness. In Table 2
we compare respective attacks on adversarially trained Dverge models. To our surprise, forming larger ensembles is actually harmful to the robustness of ensemble.
100
100
Solimax
Votisg
Attack suc0nss rate (%6)
Attack 5H000000 ratle (56)
MI
Logits
60
60
Accuary
30
PGD-+AD2
PGD--ADP
20
PGD :GAL
PCD CCAL
MORRA-ADP
MORLA ADP
MORA-GAL
MORA-GAL
MORA -Dvegge
MORA Deerge
0.010
0.020
0.025
0.000
Romaining robust sub morlels
# Remaining robust sub- models
8 perturbation (e)
(a) Softmax.
(b) Voting.
(c) Dverge w.n.t bound.
Figure 3:
by MORA can successfully fool ensemble-forming methods (ADP 19 GAL 14 and
Dverge 30 even with the majority of their sub-models giving correct outputs under softmax and
logits (Figure Tcp. While voting requires 2 1/2 sub-models to be incorrect, it is unfortunately the least robust option in all defenses. "A- + 8" means using A to attack B for up to 100 iterations. (c)
Dverge with 5 sub-models w.r.f. the bound on lR perturbation. Contrary to existing literatures, find logits to be the most robust option of the three ensemble-forming strategies.
8

---

90
PGB
FGD
PGD
CLWW
CLW
CSEW
80
APGI-DER
85
APGI-DUR
80
APGI-DUR
APGI-CE
APGD-CE
APGD-CE
Accurary (56)
Accuracy (%)
Accuracy (%)
No zewrigh
No rewcigh
No zesicigh
60
60
50
70
65
30
10
10a
10
10'
10P
10:
102
Ieratiins
Iteaations
Iterations
(a) Dverge 3.
(b) Dverge 8.
(c) GAL 8.
Figure 4: a Comparing the convergence speed of MORA against variant of MORA without sub-model reweighing, C&W, PGD and AA losses (APGD- (DL,,CE) on Dverge and GAL with softmax. While "No reweigh" converge faster initially, struggles to improve after 10 iterations; in contrast, adaptive reweighing (MORA) continues to converge at much faster rate. The horizontal and vertical axes respectively show the iteration count used, and the percentage of unsuccessful images remaining. We annotated the number of iterations for MORA to overtake all with 500 iterations.
Table 2: Attacking udversarially trained Dverge 60
models under the same setting as Table ! except let 0.03. Notably, forming larger ensembles actually be detrimental robustness.
Dverge Complexity
Clean
Nominal
PGD
CW
MORA
A
AA
CAA
MORA
A
500
500
500
12k
4.9k
1.8k
1.4k
3
83.78
45.09
44.85
44,21
42.91
42.66
42.70
42,69
42.65
2.44
Softmax
5
86.09
42.57
42.40
42.51
41.05
40.74
40.85
40.84
40.75
1.79
86.69 8
40.80
40.58
40.94
39.49
39.33
39.40
39.39
39.35
1.41
3
83.67
59.131
55.35
55.85
38.59
38.78
39.97
40.31
38.24
20.89
Voting
5
86.05
47.72*
44.32
45.18
36.19
36.42
37.29
37.89
36.01
11.71
8
86.54
38.93'
37.16
38.17
34.20
34.89
35.54
36.19
34,03
4,90
83.74
44.831
44.69
44.24
42.83
42.66
42,70
42.70
42.63
2.20
Logits
5
86.03
42.47
42.22
42.63
41.00
40.91
40.92
40.92
40.87
1.60
8
86.65
40.53
40.33
41.14
39.53
39.43
39.50
39.47
39.43
1.09
Additional results, ablation, and sensitivity analyses. Due the page limit, we provide full results of relevant figures in Appendix A note that the above key observations still hold true for all ensemble defenses we test under different ensemble-forming strategies and e perturbation bounds. In addition, we provide extensive ablation study on the design choices we made, and sensitivity analysis on the temperature constant T.
5 Conclusions
This paper identifies severe robustness overestimation in many ensemble defense techniques, and further investigates problem the robustness evaluation under three ensemble-forming strategies. To efficiently and accurately evaluate the robustness of ensembles, we introduce MORA, a new attack technique which reweighs sub-model importance adaptively by their respective "case-of-attace during attack iterations. MORA enjoys a much improved success rate and convergence rate compared with other SOTA attacks. Moreover, we found several surprising observations related to ensemble defenses, for instance, (1) misleading minority of sub-models is sufficient fool the ensemble. (2) summing by logits is the simplest yet most robust way to form ensembles, (3) with adversarial training. ensemble defenses may actually harm robustness, etc. We hope the above observations may help to guide future avenue on ensemble defenses, and provide strong attack baseline for potential approaches. Finally, MORA is open source with reproducible results and pre-trained models; and we continually update leaderboard of ensemble defenses under various attack strategics.

---

Acknowledgements
This work is supported in part by National Key R&D Program of China (No. 2019YFB21002000, Key-Area Research and Development Program of Guangdong Province (No. 002080101640000 Science and Technology Development Fund of Macao S.A.R (FDCT) under No. 0015/2019/KKP Shenzhen Science and Technology Innovation Commission (No. 1XX1200001121100003 19). and Shenzhen Industrial Application Projects of undertaking the National Key R&D Program of China 1440009 10000022)) 600002). This work was carried out in part at SICC which is supported by SKL-IOTSC, University of Macau.
References
[1] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference on Machine Learning. JCML 2018, July 2018.
[2] Dina Bashkirova, Ben Usman, and Kate Saenko. Adversarial self-defense for cycle-consistent GANs. Advances in Neural Information Processing Systems, 32, 2019.
[3] N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pages 39-57, 2017.
IEEE
[4] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, volume 32, pages 11192-11203 2019.
[5] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. ICML, 2020.
[6] Zhun Deng, Linjun Zhang, Kailas Vodrahalli, Kenji Kawaguchi, and James Y Zou. Adversarial training helps transfer learning via better representations. Advances Neural Information Processing Systems, 34, 2021.
[7] Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li. Boosting adversarial attacks with momentum. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9185-9193, 2018.
[8] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmmti,, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical world attacks on deep learning visual classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1625-1634, 2018.
[9] lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
[10] Sven Gowal, Sylvestre-Alie Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy Mann. Improving robustness using generated data, In Advances in Neural Information Processing Systems, volume 34, 2021.
[11] Sven Gowal, Jonathan Uesato, Chongli Qin, Po-Sen Huang, Timothy Mann, and Pushmeet Kohli, An alternative surrogate loss for PGD-based adversarial testing. arXiv 1910.09338, 2019.
[12] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.0335 2015.
[13] Huang. Zhiwei Ke, Yi Wang, Wei Wang, Linlin Shen, and Feng Liu, Adversarial defence by diversified simultancous training of deep ensembles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages
2021.
[14] Sanjay Kariyappa and Moinuddin K Qureshi. Improving adversarial robustness of ensembles with diversity training. arXiv preprint arviv: M00iv:/901.0000 1901,09981, 2019.
[15] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The CIFAR-10 and CIFAR-100 datasets, 2014. Available at: http::/ / /www///wwwwk// www.ctt.... toronto................. adu/~kri pariz/c. ci chmml
[16] Liu, Yaya Cheng, Lianli Gao, Xianglong Liu, Qilong Zhang, and Jingkuan Song. Practical evaluation of adversarial robustness via adaptive auto attack. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[17] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
[18] Xiaofeng Mao, Yuefeng Chen, Shuhui Wang, Hang Su, Yuan He, and Hui Xue. Composite adversarial attacks. Association for the Advancement of Artificial Intelligence (AAAI), 2021.
10

---

[19] Tianyu Pang. Kun Xu. Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning, pages 4970-4979. PMLR, 2019.
[20] Sylvestre-Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg. Olivia Wiles, and Timothy A Mann. Data augmentation can improve robustness. Advances in Neural Information Processing Systems, 34, 2021.
[21] Ishai Rosenberg, Asaf Shabtai, Yuval Elovici, and Lior Rokach. Adversarial machine learning attacks and defense methods in the cyber security domain. ACM Computing Surveys (CSUR), 54(5)1136,, 2021.
[22] Andrew Ross and Finale Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[23] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. International Conference on Learning Representations, 2014.
[24] Giorgos Tolias, Filip Radenovic, and Ondrej Chum. Targeted mismatch adversarial attack: Query with flower to retrieve the lower. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5037-5046, 2019.
[25] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses, arXiv preprint arXiv:2002. 083477 2020.
[26] Florian Tramer, Pascal Dupré, Gili Rusak, Giancarlo Pellegrino, and Dan Boneh. AdVersarial: Perceptual Ad blocking meets adversarial machine learning. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 2005-2021, 2019.
[27] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robust- may be at odds with accuracy. In International Conference on Learning Representations, 2019.
[28] Francisco Utrera, Evan Kravitz, N. Benjamin Frichson, Rajiv Khanna, and Michael W. Mahoney. Adversarially-trainee deep nets transfer better: Illustration on image classification. In International Conference on Learning Representations, 2021.
[29] Gunjan Verma and Ananthram Swami. Error correcting output codes improve probability estimation and adversarial robustness of deep neural networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
[30] Huanrui Yang. Jingyang Zhang, Hongliang Dong, Nathan Inkawhich. Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. DVERGE: Diversifying vulnerabilities for enhanced robust generation of ensembles. In H. Larochelle, M. Ranzato, R. Hadsell. M.F. Balcan, and H. Lin. editors. Advances in Neural Information Processing Systems, volume 33, pages 5505-5515, 2020.
[31] Zhuolin Yang, Linyi Li, Xiaojun Xu, Shiliang Zuo, Qian Chen, Pan Zhou, Benjamin Rubinstein, Ce Zhang, and Bo Li. TRS: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems, 34, 2021.
[32] Yunrui Yu, Xitong Gao, and Cheng-Zhong Xu. LAFEAT: Piercing through adversarial defenses with latent features. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5735-5745, June 2021.
[33] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoreti- cally principled trade-off between robustness and accuracy. In ICML, 2019.
[34] Yi Zhu, Chenglin Miao, Tianhang Zheng, Foad Hajiaghajani, Lu Su, and Chumming Qiao. Can we arbitrary objects to attack lidar perception autonomous driving? In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 1945-1960, 2021.
11


Reviews:
[{"review_id": "nWbv6DLaw8", "reviewer": "Reviewer_9Ruy", "summary": "This paper proposes an attack targetting the model ensemble defenses. The main idea is to assign different weights to different models. The values of the weights depend on the models' contribution to outputs. The results demonstrate that the proposed attack outperforms existing ones.", "questions": "1. Can MORA be extended to black-box attack scenarios or be used to attack other defenses? The application scenario is limited. It would be more exciting to see some extensions.\n\n2. There are some typos in the existing version. For example, should it be 'A successful attack happens when $k^{[E]} < 0$'?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The idea is simple and easy to understand. The experiments show \n\nWeaknesses: Overall, the contributions are incremental and the application scenario restricts to white-box ensemble defenses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cmLp--p1TZ0", "reviewer": "Reviewer_GcPa", "summary": "This paper focuses on the problem of robustness evaluation for ensemble-based defense methods. Authors argue that existing SOTA white-box attack methods may overestimate the robustness of ensemble defenses because of two reasons, i.e., (1) the ensemble-forming strategy used in ensemble defenses could lead to gradient obfuscation and (2) attacking would fool majority sub-models, while the ensemble model's output may still correct. In order to provide a more effective and reliable way to evaluate ensemble defenses, this paper proposes a novel attack method that adaptively assigns different weights for different sub-models based on their contributions to the attack gradient. To evaluate the effectiveness of the proposed attack method, authors compare it's attack performance with several SOTA white-box attack methods on four ensemble defenses under three ensemble-forming strategies. Experimental results verify that the proposed method outperforms baseline methods with large margin and converges much faster.", "questions": "1. Is it possible to verify the two reasons that cause the existing SOTA white-box attack methods overestimate the robustness of ensemble defenses through either empirical study or theoretical analysis? Although the two reasons authors pointed out in this paper sound reasonable, it would be better if more convincing evidence can be provided to further verify these two reasons.\n\n2. Some observations from the experimental results are very interesting. (1) As shown in Table 2, larger ensembles lead to reduced robustness when sub-models are adversarially trained facing with any attack methods listed in the table. However, ideally the adverasrially trained sub-models could be more robust against attacks and, hence, the larger ensemble of robust sub-models should provide improved robustness performance, comparing with small number of sub-models. Can authors provide some understanding about this observation? Does this observation reveal some possible drawbacks/limitations of applying the ensemble idea to improve model's robustness? (2) Since the majority votes ensemble strategy requires attacks to break at least half of sub-models, intuitively the robustness of ensemble defense methods based on the majority votes strategy should be higher than other two ensemble strategies. However, the experimental results give an opposite observation. Can authors give some discussion about this observation? Does this observation caused by the simplicity of the majority votes strategy (e.g. comparing with other two strategies, the majority votes only focuses on the final outputs and ignore some useful information such as logits)? \n\n3. Some notations are not clear. (1) Does the three star signs with pick color in Figure 1 and 2 represent the attack performance of the MORA method on some specific ensemble defense method under three different ensemble-forming strategies? If so, it would be better if each star can be denoted more specifically. (2) Does $\\text{MORA}^{mt}$ represent the multi-targeted version of MORA? If so, it would be better if some introduction about this variant of MORA can be provided in the beginning of the experiment part. (3) What's the meaning of \"Nominal\" appeared in both Figure 1&2 and Table 1&2? Is it the robust accuracy reported in corresponding original papers? \n", "limitations": "The limitations and potential societal impacts are discussed in the appendix of this paper. As pointed by the authors, although the attack method proposed in this paper may be used by a malicious party, an effective attack method can boost the development of robust DNN models in the future.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper investigates the robustness of ensemble defenses under multiple ensemble-forming strategies and provides extensive empirical results. \n2. The proposed method achieves much improved success rate on ensemble defense methods and convergence rate than SOTA attack methods among different experimental settings, which sufficiently demonstrate the superiority of the proposed method.\n3. Authors make the source code available and also maintain a leaderboard of ensemble defenses under various attack strategies, which can boost the research on this direction.\n\n\nWeaknesses:\n\nAlthough experimental results verify the effectiveness of the proposed attack method, the intuition/motivation of the designing of this method is not quite clear. Specifically, it's not clear why the sub-model reweighing technique is able to produce a stronger attack method than SOTA white-box attack methods for ensemble defenses. For more detailed weaknesses, see the following questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Nb5dGlD9trU", "reviewer": "Reviewer_qYQF", "summary": "This paper introduces MORA, a novel adversarial attack against ensembles. The attack is motivated by the fact that existing white-box attacks over estimate the robustness of ensembles, either due to gradient obfuscation or the difficulty of simultaneously fooling diverse gradients. MORA is designed to adapt its focus to the models in the ensemble. Experiments are performed on CIFAR-10 and -100 with ResNet20 in white-box setting. MORA is compared against other attacks and multiple ensemble defenses.", "questions": "- In Alg. 1, it seems the adversarial sample obtained after a maximum of $I$ iterations is used. What are the odds that it is the strongest sample? Should the best sample out of the $I$ iterations not be used instead (see also [1])?\n- Tab. 2 presents results of adversarially trained models ensembled using Dverge defense. What happens when attacking with MORA an ensemble of (individually) adversarially trained models without ensemble defenses?\n- Does it makes sense to compare attacks convergence in terms of number of iterations (Fig. 4)? Do they perform the same computations in each iterations? Does it also make sense to compare convergence over computation time?\n- Are the results presented for CW attack also under $L_\\infty$ norm?\n\nReferences:\n- [1] Pintor M. et al. Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examples. https://arxiv.org/abs/2106.09947", "limitations": "The limitations of the threat model are addressed in the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The design of the loss function used for MORA seems sound and well-motivated by the causes of attack failures on ensembles.\nMORA incorporates state-of-the-art community practices for thorough robust evaluation.\n- Under small perturbation budgets, MORA seems to find adversarial examples that other attacks do not, thus providing a better estimate of robustness for the given ensembles.\n- The experiments are extensive, comparing the proposed attack against state-of-the-art white-box attacks and ensemble defenses, under different aggregation rules and ensemble sizes. An ablation analysis is also provided.\n- Very well written paper.\n\nWeaknesses:\n- MORA seems to lose its advantage over other attacks under certain regimes. Results are almost on par with A$^3$ and the nominal values in Tab. 2.\n- It is standard to use perturbation budget $\\epsilon = 8 / 255$ on CIFAR-10. However, the main results table in the paper (Tab. 1) uses $\\epsilon=0.01$, just under a third of the standard value. Results for $\\epsilon=0.03$ are included in the appendix, where we see that the gap in ensemble robustness estimation significantly narrows between general-purpose white-box attacks and the proposed MORA. The numbers are on par for most rows, diminishing the claims of the paper.\n- Despite the extensive experiments, some aspects of the evaluation are limited. Particularly, focusing only on ResNet20 architecture, CIFAR data and $L_{\\infty}$ robustness does not convince beyond a doubt that MORA is the new adversarial standard for ensemble evaluation.\n\nMinor:\n- It would be a plus to start by stating the threat model. It become apparent later in the paper that the assumption is white-box access althroughout.\n- L135: there exists -> there exist\n- L186: Base on this property -> based on this property\n- L216: attackss -> attacks", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Isgd4zy776z", "reviewer": "Reviewer_8pbN", "summary": "In this paper, the authors propose an adversarial attack (MORA) against model-ensemble-based defense by reweighing the importance of sub-models. The authors first investigate the factors which cause existing white-box attacks to fail against several state-of-art model-ensemble defenses, that is gradient obfuscation. Then, the authors propose a strategy for crafting adversarial samples by conducting optimization on several 'important' sub-models within ensembled models.  Motivated by such a strategy, the authors propose a MORA loss, which incorporates a sum of the weighted variant of sub-model logits, in order to\nexpose sub-model logits with adaptive reweighing. Through extensive empirical evaluation, MORA is proved to be able to bypass existing SOTA model ensemble defense approaches. The authors further compare MORA with several state-of-art white-box attacks(PGD, C&W attacks) in both the dimensions of efficiency(convergence rate) and attack efficacy (attack success rate).  The results show that MORA can significantly outperform existing white-box attacks against model ensemble defense.", "questions": "I have several questions:\n\nI. What's the motivation for developing a white-box attack against SOTA defense methods against black-box attacks?\n\nII. Can you compare MORA with several decision-based attacks?\n\n", "limitations": "See above.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=====Strengths=====\nI. This work is well-structured and easy to follow.\n\nII. This work proposes an effective method MORA and solves an interesting problem.\n\nIII. The evaluation is comprehensive and convincing. \n\n\n=====Weakness=====\n\nI have two main concerns about this work.\n\nI. Limited Novelty and Practicality. This work seems to be a kind of 'defense-aware attacks' approach which built upon the existing defense approach(i.e., model-ensemble defense). The proposed attack can only bypass a specific category of defense, which limits its practicality. Moreover, the proposed attack is evaluated under the white-box settings, however, as mentioned in the abstract and introduction, the ensemble-based defense approach is mainly used against black-box attacks (i.e., transferable attacks).  In general, there's no evidence to show that ensemble-based defense is or performs close to the state-of-art defense against white-box adversarial attacks. Therefore, the motivation for developing a white-box attack bypassing ensemble-based defenses that are mainly used against black-box attacks is unclear to me. \n\nII. Evaluation. Indeed, I am convinced that MORA can outperform existing white-box attacks against ensembled-based defense through empirical evaluation. However, I think selecting current white-box attacks for comparison is not enough. This is because existing white-box attacks are prone to gradient obfuscations, which is also claimed in your introduction section and other related work.  In contrast, I think several decision-based attacks (hop skip jump attack, Sign-Flip attack.) is more suitable for this scenario.  Even though these decision-based attacks are built under black-box, they can also be applied in the white-box settings. I have expertise in such decision-based attacks and observe that decision-based attacks can bypass gradient-obfuscation defense. I think you should also compare MORA with them. \n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept