Synergy-of-Experts: Collaborate to Improve Adversarial Robustness
Authors: Sen Cui
Jingfeng Zhang
Jian Liang
Bo Han
Masashi Sugiyama
Changshui Zhang
Venue: NeurIPS 2022 Accept (2022)
Keywords: adversarial defense
collaboration
model ensemble.

Abstract: Learning adversarially robust models require invariant predictions to a small neighborhood of its natural inputs, often encountering insufficient model capacity. There is research showing that learning multiple sub-models in an ensemble could mitigate this insufficiency, further improving the generalization and the robustness. However, the ensemble's voting-based strategy excludes the possibility that the true predictions remain with the minority. Therefore, this paper further improves the ensemble through a collaboration scheme---Synergy-of-Experts (SoE). Compared with the voting-based strategy, the SoE enables the possibility of correct predictions even if there exists a single correct sub-model. In SoE, every sub-model fits its specific vulnerability area and reserves the rest of the sub-models to fit other vulnerability areas, which effectively optimizes the utilization of the model capacity. Empirical experiments verify that SoE outperforms various ensemble methods against white-box and transfer-based adversarial attacks.

Synergy-of-Experts: Collaborate to Improve Adversarial Robustness
Sen Cui" Jingfeng Zhang", Jian Liang', Bo Han', Masashi Sugiyama*5, Changshui Zhang'
Institute for Artificial Intelligence, Tsinghua University (THUAI),
Beijing National Research Center for Information Science and Technology (BNRist),
Department of Automation,Tsinn Tsinghua University, Beijing, P.R.Chime
2 RIKEN Center for Advanced Intelligence Project, Tokyo, Japan Alibaba Group. China
Hong Kong Baptist University, Hong Kong SAR. China
The University of Tokyo, Tokyo, Japan jingfeng. zhang@riken bhanml@comp. hkbu. edu. hib...edu... hk zca@mail acatmail.tsinnhuu. edu, cn
www.nnguuaeuuu..... quis19@mails tainghua. edu. en xuelang- babaaiibbb--innc...... augiōk. u-tokyo.ace ac.
Abstract
Learning adversarially robust models requires invariant predictions a small neighborhood of its natural inputs, often encountering insufficient model capacity. There is research showing that learning multiple sub-models in an ensemble could mitigate this insufficiency, further improving the generalization and the robustness. However, the ensemble's 's voting-based strategy excludes the possibility that the true predictions remain with the minority. Therefore, this paper further improves the ensemble through a collaboration scheme scheme-Synergy of-Experts (SoE). Compared with the voting-basee strategy, the SoE enables the possibility of correct predictions even if there exists a single correct sub-model. In SoE, every sub-model fits its specific vulnerability area and reserves the rest of the sub-models to fit other vulnerability areas, which effectively optimizes the utilization of the model capacity. Empirical experiments verify that SoE outperforms various ensemble methods against white-box and transfer- based adversarial attacks. The source codes are available hittt:t:////
com/ cuis15/ 1e/1///////pppe synergy- of exfeexesst
Introduction
Deep models have been widely applied in various real-world applications including high-stakes scenarios (such as in healthcare, finance, and autonomous driving). An increasing concern whether these models make adversarially robust decisions [ 2] Recently, there are research revealing that an adversarially robust method requires invariant predictions to a small neighborhood of its natural inputs, thus often encountering insufficient model capacity 43.4 This limits the further improvement of robustness and has the undesirable degradation generalization [5].
Learning multiple sub- models in an ensemble 6 can mitigate this insufficiency. Remarkably. there are research 8 I0] proposing to minimize the vulnerability overlaps between each pair of sub-models and improving both robustness and generalization over single model. However, the voting-based ensemble may waste the limited capacity of multiple models.
In the example of three sub-models (see Figure [(b)), the adversarial input that lies in the black areas can fool the ensemble successfully, i.c., more than half of sub-models must correctly classify the adversarial input. Therefore, the ensemble's voting-basee strategy excludes the possibility that true
The first two authors made equal contributions.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

vulnerable area
Single model
(b) Ensemble
(c) Collaboration
Figure Illustrations of the vulnerability area of (a) Single model (b) Ensemble, and (c) Collaboration. The black area represents the vulnerability area in which the model is undoubtedly fooled.
predictions remain with the minority. other words, learning an ensemble requires more than half of the sub-modell to fit the same vulnerability areas, which leaves the following question unanswered whether we could only leverage a single sub-model fit a vulnerability area and reserve the rest of the sub-models to fit other vulnerability areas.
Inspired by mixture-of-experts (MoE) [[]] we propose learn a collaboration among multiple sub- models to optimally utilize the limited capacity. As shown in Figure [[c] the adversarial input that lies in the vulnerability overlaps of all sub-models can undoubtedly fool the collaboration. Compared with the ensemble in Figure l(b)). collaboration enables the possibility of correct predictions even there exists single correct sub-model. Besides, learning a collaboration could enable every sub-model to fit its vulnerability area, which could collectively fix broader vulnerability areas. Then, sub-models could collaboratively choose trustworthy one make the final predictions.
Classic MoE methods assume that the problem space is separable, and the separability is irrelevant the learned model [11]. However, the non-i.i.d adversarial inputs [12] depend on the learned models and are hard to be classified by learned gate in MoE. To tackle the above challenge, we propose Coneery-of-Expereee (SoE), which explicitly builds the relationship between learned models and adversarial inputs. Specifically, each sub-model has dual heads: one outputs a vector of predicted probability fo another outputs scalar that measures the confidence of the prediction. In the adversarial training phase, given an adversarial input each sub-model chooses an easy one(s) to feed itself. The other head is meanwhile updated by comparing the predicted probability on the true (-) (a scalar). In the inference phase, given an input, SoE chooses sub-model with the largest confidence as the representative to output the overall prediction.
label (-)
We highlight our key contributions as follows.
We provide a new perspective on learning multiple sub- sub-models for defending against adver- sarial attacks. We show that the collaboration could make better decisions than the ensemble (Proposition T. which implies collaboration may fix broader vulnerability areas.
We propose novel collaboration framework SoE (see Section 3.2 In the training phase, SoE minimizes the vulnerability overlap of all sub-models; In the inference phase, SoE could effectively choose representative sub-m sub-model make correct predictions. We also provide a comprehensive analysis illustrating the rationale SoE in Appendix.
Empirical experiments corroborate the SoE outperforms various ensemble methods 10 against white-box and transfer attacks.
13
2 Related Work
Adversarial attack. Adversarial attacks aim to craft the human- -imeerceimmrre adversarial input to fool the deep models. Adversarial attacks could be roughly divided into white-bor attacks in which the adversary is fully aware of the model's structures [ 14
21
23 24
and black-box attacks in which the deep models are treated as black boxes to the 35 36). This paper focuses on building effective defense and select both white-bon and black-box attack methods as robustness evaluation metrics.
25
26
adversary
k3

---

Adversarial defense. Defending adversarial attacks is challenging task and researchers have proposed various solutions. Certified defense tries learn provably robust deep models against norm- bounded (e.g., f2 and )xx) perturbations 37
38
39
40
42
43
44
45
46
47
48,
49]
Empirical
defense. leverages adversarial data to build effective defense such as adversary detection [50] [1 64,
52 53,
54
55 56 57 58, 59 [60] [1] 62,
63]
65 66)
67
[8] and adversarial training (AT), in
which AT stands out as the most effective defense. Researchers have investigated various aspects
of AT, such as improving AT's robustness
generalization [5] 69 70 71 72 73 74,
75 76 77
78 79
80
ST
82
83
85
86
87
88
89
31, fixing AT's undesirable robust overfitting [90, 9
92
improving AT's training efficiency 193
94
95
96
97
98 imderstandin/interreeinn AT's
unique traits 199. 100,
[10], applying AT into applications III 112 etc. Besides, researchers have also actively investigated robust-structured [1g 119. Nevertheless, the above research thoroughly investigated single model; this paper focuses on the collaboration among multiple models for adversarial defense.
[01
102
103,
104 105
106,
107
108, 49.
109,
models [11 114 115
117
Ensemble methods for adversarial robustness. The most relevant studies are the ensemble methods. Ensemble methods such as bagging [6] and boosting [7] have been investigated for significantly improving the model's generalization. Motivated by the benefits of ensemble methods in improving generalization, researchers introduced an ensemble to improve the model robustness [T0] 2 120]. . Tramèr er al. [120] proposed to reduce the adversarial transferability by training a single model with adversarial examples from multiple pretrained sub-models. Pang er al. [8] introduce a regularization method ADP to encourage high diversity in the non-maximal predictions of
sub-models. Kariyappa et al. [9] improved the ensemble diversity by maximizing the introduced cosine distance between the gradients of sub-models with respect to the input. Yang et al. [TO] proposed to distill non-robust features in the input and diversify the adversarial vulnerability. These methods reduced overlaps of vulnerability areas between sub-models [IO] Compared with voting strategy in ensemble, mixture-of-expertt (MoE) assumes that the problem space can be divided into multiple sub- sub-problems through gate module [2I] However, in adversarial training, the adversarial samples, which depend on the learned models, are not i.id. A vanilla MoE is hard to identify the best performing sub-models for each adversarial sample without the information about the learned models.
3
Collaboration to Defend Against Adversarial Attacks
3.1 Superiority of Collaboration
This section shows a collaboration, in theory, could make better decisions than an ensemble.
Ensemble. Suppose that there are M learned sub-models (fe:> feesJee Jea.-.., Je>r }. given an input x, M sub-models make predictions {fo, (x), (x),... ..., ( The ensemble outputs a final prediction ensemble(æ, So, ..., by the voting-based strategy:
ce foxc) arg max (I......
Σ 1
(1)
where is the indicator function and K denotes the number of classes. Note that the ensemble outputs the predicted label y that agrees with the majority predictions of the sub-modek
Definition 1 (best- peeestperformin sub-model). Given input and its label y. the best best-performing sub-model achieves the lowest objective loss on the data (x, y) among all M sub-models:
Sonca (x) = arg
min fo,E fo
((),()))))
Note that the best- pest-performing sub-model is w.r.t. the input data (x, v), i.c., different input data correspond to different perfforrrnn sub-models.
Collaboration. Suppose that there are M learned sub-models (Ja Jog. soxc }. Given an input I, sub-models make predictions {fe, (z)) fe, .... (x) The collaboration tries to output final prediction collaboration fer .... fex. by the best- pest-peerorming sub-model: ...,
collaboration(2; fo,, So ) = Sorms (x))
3

---

I
positive sample
negative sample
confidence
head
eub-model
(scalar)
X
sub-model
feature exactor
P head
p(x)
correctly clsseified
misclaselfid
(K-dim vector)
fe
Sample index
(a) Assign data their best-performing sub-models
(b) Dual-head structured sub-models
Figure 2: (a) The blue and pink lines denote the decision boundaries of two sub-models, Each sub-model makes negative predictions (-) its left and makes positive predictions its right (+). The given data will be assigned to the sub-model that has the lowest objective loss. The arrows represent the data assignment. (b) Each sub-mode has two heads head that outputs the predicted probability (vector) and E head approximates the predicted probability of the true label in the prediction (scalar).
jo.
fer (1
PGD
PGD
:0sess!
Wifer (2 119!
Toas
fex (5
fin
(a2) collaboration training
for (J00e)
yl (1x
fu
H. 1,91
Po (are)
BCE loss
foxr (r
Py(a)
(al) optimizing the module fa (sub-model training)
(b) optimizing.R0
Figure 3: Optimization process of the M sub-models in a collaboration
Proposition 1. Given M learned sub-models the predicted accuracy of the collaboration is upper- bounded that of the ensemble, i.e.,
((..)))D [1 [ )=9] 2 E(a.))D [FreeeD[ooneeeee Lonsemble(e..oo.....o ]my]
(4)
Proof. Given an (a. 3) E D, if the ensemble's prediction is correct, at least one sub-model makes correct prediction, i.e., fo (x)=y holds; therefore, the collaboration' prediction is correct. If the collaboration's 's prediction is correct, there exists case that the majority of sub-models make consistent but wrong predictions, while single sub-model's prediction correct; then, ensemble's prediction is wrong. Therefore, Proposition
| holds.
From Proposition D collaboration could achieve equal or higher performance than ensemble. Compared with the ensemble, the collaboration requires the identification of the best-performing sub-models using label information. Next, will introduce realization of our collaboration framework.
3.2 Realization of collaboration for defending against adversarial attacks
Notation. We firstly introduce the needed notations. Suppose X' and y denote input space and output space, where y = {1,... K} } for K K-class classification problem. There are N samples the dataset = {(x, v) where x E and y V. Let dint (x, x') = || x'|l oo denotes the infinity distance metric, and [x] (x' E X dinf (I,x²') < { the closed ball of of radius > 0 centered at æ. To search for adversarial data within norm ball B, [zr]. (5) proposed a projected gradient descent (PGD) method that iteratively searches for adversarial data ă (a refers to natural data). fo outputs K-dimensional predicted probability, i.c., p(x) = p (x),
Goal of collaboration. 1) ensure the correct prediction of the best- pest-performing aub-model for given input, and 2) select the best-performing sub-model among all sub-models to make predictions.
First, intuitively, every sub-model in collaboration should maximize its expertise to fit its areas and leave the remaining areas fitted by others. As result, the collaboration can minimize the vulnerability overlaps of all sub-models. Section 4 shows "minimizing the vulnerability overlap of all sub-models"

---

Algorithm 1 training phase I: the sub-model training
Input: the sub-mode uub-models with dual heads {fe, ) and {go, where fo, outputs the label predic- tion and 9s. outputs the approximated confidence, the training dataset D, and the hyperparameter
1: for each data (z.y) E do
2: for each sub-model ʃo,, - 1, 2, , M do
3: Obtain the adversarial data 5 of the sub-model fo, using the PGD method;
4: for each sub-model ja 2, M do
5: Calculate the approximated confidence, i.e. I0, (3"
6:
Minimize BCE loss Es = BCE(ge, (ā ). Py (i }) to update the module gos
7: Collect sub-model i's cross entropy (CE) loss on data ;".; e (So, (z" :U);
8:
end for
9: Calculate surrogate loss data j0:; -m= Σソ= exp (==(
10: Update (Jo, J by minimizing é //choose the best-performing sub-model to fit ;0,
end for
12:
end for
Output: the learned sub-models with dual heads {fe, )i= and {go, M.M
is "minimizing the objective loss of the best-performing sub-models". Therefore, during the training phase, the given data should always be allocated to the sub-model that has the lowest objective loss. In other words, the sub-models always choose the easiest data to learn. In the example of Figure 2(a) Data@) misclassified by both sub-models. The blue sub-model is near Data@) and has the lowest objective loss. We assign the blue sub-model to fit Data(). it) Data(2 is correctly classified by the pink model but wrongly classified by the blue model; for ease of effort, we assign the pink model to fit Data2), because the collaboration can correctly be classified Data()) by selecting the pink model as the representative. tit) Data() is correctly classified by both models. The blue model is far from Data() and takes the lowest effort on fitting it; therefore, we assign the blue model to fit Data()).
Second, select the best-perfonming sub-model, we construct dual-head structured sub-models. As shown in Figure 2(b) our sub-model has dual heads: 1) predictor (P) head: Sa predicts the label probability (x( ((z) ) = | (z)...., ..., (z)] vector); 2) evaluator (E) head: go approximates
the confidence of the prediction p(x).
Note that of given prediction p(z). Meanwhile, the largest confidence corresponds to the lowest objective loss, and vice versa (see theoretical proof in Proposition 2). Therefore, the rest-performing could be identified using the approximated confidence (ga (z)) by the E head.
9o to approximate the true label probability (x)), which denotes the true confidence
To defend against adversarial attacks, the collaboration needs to learn from adversarial data. Algorithm [[2along with Figure 3 articulates how to learn such the collaboration. In particular, the training of our framework SoE shown in Figure is as follows:
the sub-model training
Alg.
1. the sub-model training: the sub-models fit adversarial samples from their own or other sub-models;
the collaboration training
Alg. 2
2. the collaboration training: the sub-models fit adversarial samples from the collaboration.
The sub-model training. As shown in Figure 3 aall,, given natural training data æ and its label y. we use the PGD method 1.M L.M to obtain M adversarial variants (5 Jizı of .M sub-models as existing baselines do. For each adversarial variant 3d, we assign the best-performing sub-model to learn and update its feature extractor and label head. This process corresponds to Lines 1-8 and Lines 9-10 in Algorithm 1 Note that in Line 9, we use surrogate loss to approximate the sub-model assignment process (reasons see Eq. and 6). As shown in Figure given an adversarial variant 26, we propose to use the binary-cross
the learned SoE
Figure 4: Training process of SoE.
(BCE) loss between
5

---

Algorithm 2 training phase II: the collaboration training
Input: sub-models with dual heads {ff and (9s, J==1 where fe, outputs the label
M
prediction and Ié, outputs the approximated confidence, training dataset D, hyperparameter
0;
1: for each data (x. E do
2: for each sub-model Se,, 2, M do
Calculate the approximated confidence, i.e. Io. go. (z); 4: Calculate the prediction i.c., fe. (r);
3:
5: end for
6: Output the prediction (a) with the highest confidence;
7:
Obtain by perturbing x to worsen the prediction; generate the adversarial samples of the collaboration
8: Minimize BCE loss &o (Z) to update the module Io of all sub-models;
9: Update {ffe to fit ă by minimizing the surrogate loss (#);
end for
11: Output: the learned sub-models (fo, ( with {gou 1am
the predicted label probability on the true label (i.e., Py (a%)) and the approximated confidence (i.e.. (i to update each sub-mode model's''s E head. This process corresponds to Lines 5-6 in Algorithm |
The collaboration training. During the sub-model training, we learn the most adversarial data from each sub-model using the sub-models performing best. The most adversarial samples cannot attack the collaboration successfully. However, there may exist harmful samples (which may not be the most adversarial for any sub-model) that are unexplored and can attack all sub-models. Therefore, the sub- model training may converge without a full exploration of the adversarial samples. (The experimental verification about this could be found in Appendix.) To defend these potential adversarial samples, Firstly, we propose to generate the adversarial samples that can worsen the outputs of the collaboration. particular, for each data sample x, we output the prediction p(x) whose confidence 9x(x) is the highest. We perturb x to fool the prediction using PGD method. Then we minimize a surrogate loss to fit this adversarial data Z. This process is detailly shown in Algorithm 2 We use Algorithm T and Algorithm 2 in sequence to train our collaboration. Algorithm 2 is proposed to explore the adversarial samples which could be not the most adversarial samples of any sub-model, but could fool all sub-models In Algorithm2) 2 we attack the collaboration iteratively to obtain the adversarial samples. However, in each iteration
we propose the collaboration training shown in Figure (a2) and Algorithm 2
update the adversarial sample ã, the best-performing sub-model could be different. For example, in the first iteration, given the input x, the best- best-performing sub-model is fo. We obtain an adversarial sample H' by attacking fe,- However, in the second iteration, given the input H', the best-performin sub-model is another sub-model (e.g., fo,). We attack the best- pest-performing sub- model fer obtain the adversarial sample i". Therefore, by attacking the collaboration following Algorithm 2 we could obtain the adversarial sample r² which is not the most adversarial samples but could fool all sub-models and unseen in Algorithm
During the inference phase shown in Algorithm 3 in Appendix, once M sub-models are properly learned, SoE chooses representative sub-model whose confidence is the highest among all sub- models, and then outputs this sub-model's prediction.
3.3 Analyses of SoE
Optimizing the best-performing sub-models. We firstly show that minimizing the vulnerability overlap of all sub-models is equal to minimizing the objective loss of the best-performing sub-models. For ease of optimization of the best-performing sub-model, we provide a surrogate loss.
The vulnerability overlap of all sub-models refers to the set of adversarial data (ū, 1) that are misclassified by all sub-models, i.e., all sub-models' objective loss is higher than a certain degree 8: ming, Osr) E(Jo, (ū, x)) > 6, E δ, where ò denotes the vulnerability overlap of all uub-models.

---

To reduce the vulnerability overlap of all sub-models, we only need to reduce objective loss of single model, which is equal to minimizing the loss of the best-perforning sub-model, i.c.
min E(D.)DDD (Ess((12.....M)) (best(#"
(0i O2r
(5)
min e (Jo, (, where ( JE(1,2...,,M)
where : is the adversarial data generated by the sub-model fe..
While directly performing the outer minimization in Eq. may cause trivial solution (e.g., there is only one optimized sub-model), for ease of the optimization of Eq.(5) (Corresponding to Lines 9-10 in Algorithm[] [). we provide surrogate objective as follows.
min (01,02.....0af Bas L(x,))E (n
(6)
(19) and o is pre- pre-de arper-paramete.
where l (1"; - Σソ= exp
In 66. we approximate the objective ming, 010,, Ox) e (fe, (in:).)) using a smooth surro-
gated maximum function due to
min e (Jo, ()) 8. In(M) s JE.1,....,,F)
s
min e (Jo,( je(1......,Af)
(7)
The proof of (q.(7) is in Appendix.
The best-performing sub-model has the highest confidence. We show that sub-model with the highest confidence achieves the minimum of the objective loss among all sub-models, i.e., the pest-performing sub-model.
Proposition 2. Given imput x, the sub-model that has the highest confidence corresponds the best best-performing sub-model, i.e.,
arg max confidence(fo,(e)) JE(1......M)
arg min (1)o) jE(1.2...M)
(8)
where confidence corresponds the probability of the true label in given ((x)). To learn the E head to approximate the confidence of p(x), we could simply compare its output with the predicted probability on the true label (i.e.,,(zz)) then update the E head by gradient descent (corresponds to Lines 6-7 in Algorithm|!] D). Given an input to the collaboration, our dual-head structured sub-models can collaboratively decide the best-performing sub-model by comparing the values of the E head (corresponds to Algorithm 3 in Appendix).
Note that the E head may be susceptible adversarial attacks in the white-box setting. In implementation, we use a simple linear structure to regress the confidence. Experimental results on adaptive attacks demonstrate the reliability of our framework.
Experiments
In this section, we conduct experiments on benchmark dataset to verify the effectiveness of our method in defending against white-box and transfer attacks. Then we provide ablation studies to demonstrate the significance the collaboration training described in Algorithm 2 The experimental results about black-bor box attacks and more discussions could be found in Appendix.
4.1 Experimental Setup
Following the work in [10], we compare our method with various related methods, including ADP [8). GAL 19]. DVERGE [10] and MoRE [3]. We use ResNet-20 [122] sub-models in all methods for
fair comparisons, and we use CIFARIO as the data set, a classical image dataset [123) that has 50,000 training images and 10,000 test images.
4.2 Performance on White-box attack As there are mainly two threat modes in the adversarial attack setting: white-box attack and black- box attack. White-box attack refers to that attackers know the information about the models, including training data, model architectures, and parameters, while black-box attackers have no access to the information about the model's 's structures and parameters and rely on surrogate models to generate transferable adversarial examples.
7

---

6-0.01
02
C-0.06
48-
35.0-
8 72.6
33
30
12.5-
8 71.6
30
44-
11.2
56
58
46
42-
40
13
24
27.5-
22.5
0,008.500500
75.05
7500500.7533
0.000
Figure 5: SoE robustness with varying under adaptive attacks with e E 0.02..0......... 0..0,,
We compare our method with baselines on defending against white-box attacks using subset of CIFARIO. We use 50-step PGD with five random starts and the step size of e/5 to attack all methods as in [10]. We learn the models in adversarial training and evaluate the robustness under various attacks with the same For example, evaluate the robustness under white-box attack with
c <= 0.01, we first learn the models in adversarial training with 6 = 0.01. We evaluate all methods following the setting in [TO]. In particular, randomly select 1000 samples under different For the PGD attack, we select the cross-entropy loss to update the perturbations to search for adversarial samples. In addition to the robustness, we also report the performance of all methods on clean data with the adversarial training under different e.
Table I: Robustness and clean data accuracy (%) under white-box attack.
r(robusticleam)
0.01
0.02
0.03
0.04
0.05
0.06
0.07
GAL
49,5/87.8
31.4/85.4
25.4781.2
22.7/78.7
18:4/77.3
13,4/66.2
9.0/6.00
DVERGE
67.3/85.4
52.3/8.0
29.9/77.6
22.5/76.7
14.2/75.8
10.0755.3
MoRE
67.9/88.0
49.9/85.3
37,8/82.
31.3/79.5
24.0/78.2
15.6/77
12.3/77.8
ADP
67.7/89.0
52.9/86.8
40.8/85.4
30.8/83.3
25.8/76.0
23.4/66.4
20.3/63.0
72.0/88.8
57.5085.6
47,8/80.2
38.7/80.0
30,4/79.1
24.0/74.1
SoE (adaptive)
70.9/88.8
56.2/88.6
45.6/80.2
38.7/80.0
28.9/79.1
22.1/76.7
21.8/4.11
From Table SoE achieves better robustness performance under white-bon attack. The results verify that collaboration significantly improves the utilization of the limited model capacity. Therefore, SoE can fit more adversarial data and have relatively smaller vulnerable area.
Performance on Adaptive Attacks. We follow the suggestions in 124 and conduct two different adaptive attacks to fool the dual heads simultaneously. For the first adaptive at- tack, we attack the E head to minimize the confidence of the best-performing sub-model. In particular, we maximize I the best pest-performing sub-model. For the second adaptive at- tack, we try to achieve a mismatch between the correct predictions and the highest confi- dence. Specifically, maximize l2
j-th sub-model is identified BCF( (x), 1), where j arg ma 9s, (x) means the
log So, (I), Jo, (x)) + gos (s)) where j
arg ma - log Jo. (I) * Do: (x) +1 go: (x) We conduct experiments by maximizing
the weighted loss padp = ((fe(z).)) + A- e1 and Egd ((fe(x),)) v) + X lg with varying A. The robustness with respect to À the stronger adaptive attack (i.e., the attack achieves higher success rate) is shown in Figure 5 Though both adaptive attacks could attack the predictor and the evaluator
Table Transfer attack with 3 adversarial variants (%)
0.01
0.02
0.03
0.04
0.05
0.06
0.07
methods
GAL
64.2+4.22
48.74
30.243.5
4..9
52.344.5
48.7+3.2
42.2++4.1
ADP
85.6
82.9..2
78.3
73.2
69.6
60.4+.2
5744.1
MoRE
84.8 84.8
82.1
78.41
74.32.1
73.2
70.3
69.1
DVERGE
83.4 a
80.1
77.3
72.4
71.94.2
68.8
66.24.2
SaF:
85.2
83.4
78.8
76.61.2
74.6
72.3
70.21.2
simultaneously, the evaluator in our method is robust the adversarial samples because of its sample structure. As seen in Figure 5| Table our method is slightly degraded by adaptive attacks and still outperforms baseline models.
4.3 Visualization of the our Collaboration Scheme
To intuitively understand the collaboration mechanism of our proposed SoE, we show the decision boundaries of the ensemble and the collaboration in Figure 6 In particular, we learn the ensemble and the collaboration with 3 ResNet-20 sub-model CIFAR10 dataset with adversarial training.

---

Table 3: Transfer attack with 30 adversarial variants (%).
0.01
0.02
0.03
0.04
0.05
0.06
0.07
methois
GAL
57.8
64.1
46.3
56.013.0
43.9
44.5
4144g.2
ADP
84.22.2
80.2
73.74.11
69.42.2
65.24.2
26.7
54.4.2
MoRE
83.74.8
79.64.2
74.44-2
70,24.a
67.64.1
63.84.a
59.34.2
DVERGE
81.5+.a
78.1
73.5+.3
68.4
67.24.2
63.8
57.1
SoE
83.1+.3
75.1+.8
70.8+.2
69.0+.2
3
sub-model 1
sub-model
sub-model 3
Ensemble
sub-moder 1 sub-model 2
sub-model 3
Collaboration
Figure 6: The visualization of the decision boundaries the ensemble and the collaboration. The same color areas denotes the same predicted label. The black regions means that the models predict correctly while other color regions means that the models are fooled and predict incorrectly. The vertical axis is along the adversarial direction and the horizontal axis is along random Rademacher vector.
The same color areas in Figure denotes the same predicted label. The black regions means that the models predict correctly while other color regions means that the models are fooled and predict incorrectly. The ensemble defending against attacks requires more than half of the sub-models fit the same vulnerability areas. From the top of Figure all sub-models similar vulnerability areas and there is boarder vulnerability overlap between sub-models. Our proposed collaboration proposes to minimize the overlap of all sub-models. From the bottem of Figure 6 different sub-models defend against different vulnerability areas collaboratively and our collaboration achieves smaller vulnerability areas.
4.4 Performance on Transfer Attack
Due to the transferability of adversarial examples, transfer adversaries can craft adversarial examples based on surrogate models and perform an attack on the target model. In our experiments, we follow the transfer attack setting in [T0] and select 1000 test samples randomly. We use hold-out baseline ensembles with three ResNet-20 sub-models as the surrogate models to generate adversarial samples. partieular, we use three attack methodologies: PGD with momentum [35], SGM (2) which adds
weight to the gradient through the skip connections of the model, and M-FGSM [36 which randomly augments the input images in each step. For each sample, three adversarial variants are using the three attack methods. Only when the model can classify all kinds of adversarial variants the model successfully defend against adversarial attacks. We show the results of all methods Table 2 Furthermore, also use more challenging setting following the work in [T]] We use hold-out baseline models with 3, 5, and ResNet-20 sub-models as the surrogate models. Meanwhile, we generate adversarial samples with cross- cross-entropy loss and CW loss [15]. For each sample, generate 30 adversarial variants, and only if the model classifies all the 30 variants can the model defend the transfer attack successfully. The results are shown in Table
In our experiments, GAL hard to learn stably adversarial training. In Table 2 when 0.01 S 6, SoE achieves better performance compared with baselines. With the increase of the volume

---

of e-ball increases exponentially. The performances of all methods get worse significantly because of insufficient model capacity. Similar results can also be found in Table 3 Since SoE addresses more adversarial data using a collaboration mechanism, it achieves relatively better robustness performance as increases.
4.5 The Robustness of SoE. under Different Number of Sub-models
To explore the robustness of the collaboration under different with different number of sub-models, we conduct experiments under transfer attacks with different number of sub-models (1 S N S 5). The detailed information could be found in Section 4.3 in Appendix. In summary, we have the following findings. For different more sub-models could achieve a more significant robustness improvement with the increase of e. For different number of sub-models, more sub-models are more likely to achieve a higher robustness, but the margin gain decreases with more sub-models.
5 Conclusion
In this paper, we study an essential question in the field of adversarial attacks that when we should collaborate. (i) If a single model handle everything, there is no need for multiple models. (ii) If a single model can only handle a part of the whole, collaboration among multiple models makes sense. Adversarial defense is typical task that falls into the circumstance (ii) because a single model hardly fits adversarial data. We provided collaboration framework- -oo--ooeea as the defense strategy over ensemble methods, and empirical experiments indeed verified the efficacy of SoE. Future work includes applying our collaboration framework to other areas such as kernel methods, fairness, and federated model, etc.
Acknowledgments
We would like to thank the anonymous reviewers of NeurlPS 2022 for their constructive com- ments. Sen Cui and Changshui Zhang would like to acknowledge the funding by the Natural Science Fundation of China(NSFC. No. 62176132 ). Bo Han was supported by the RGC ECS No. 22200720, NSFC YSF No. 62006202, Guangdong Basic and Applied Basic Research Foundation No. 2022A151501 1652, and RIKEN Collaborative Research Fund. Jingfeng Zhang was supported by JST ACT-X Grant Number JPMIAX21AF and JSPS KAKENHI Grant Number 22K 7955, Japan. Masashi Sugiyama was supported by JST AIP Acceleration Research Grant Number JPMJCR20U3 and the Institute for AI and Beyond, UTokyo,, Japan.
References
[1] lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
[2] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. ICLR, 2014.
[3] Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry- aware instance-rowcighted adversarial training. In ICLR, 2021.
[4] Yaodong Yu, Zitong Yang, Edgar Dobriban, Jacob Steinhardt, and Yi Ma. Understanding gencralization in adversarial training via the bias-variance decomposition. arXiv preprint arXiv:2 2021.
[5] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ICLR, 2018.
[6] Leo Breiman. Bagging predictors. Machine learning, 24(2): 140, 1996.
[7] Yoav Freund, Robert E Schapire, et al. Experiments with a new boosting algorithm. In ICML, 1996.
[8] Tianyu Pang. Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In ICML, 2019.
[9] Sanjay Kariyappa and Moinuddin K Qureshi. Improving adversarial robustness of ensembles with diversity training. arXiv preprint arXiv:1901.09900 2019.
[10] Huanrui Yang, Jingyang Zhang, Hongliang Dong, Nathan Inkawhich. Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry. and Hai Li. Dverge: Diversifying vulnerabilities for enhanced robust generation of ensembles. In NeurIPS, 2020.
10

---

[11] Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
[12] Ruize Gao, Feng Liu, Jingfeng Zhang, Bo Han, Tongliang Liu, Gang Niu, and Masashi Sugiyama. Maximum mean discrepancy test aware of adversarial attacks. In ICMI, 2021.
[13] Kaidi Xu, Chenan Wang, Hao Cheng, Bhavya Kailkhura, Xue Lin, and Ryan Goldhahn. Mixture of robust experts (more); robust denoising method towards multiple perturbations. arXiv preprint arXiv:2104. 10586, 2021.
[14] Seyed-Mohsen Moosavi-Dezzoon Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In CVPR, 2016.
[15] Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In JEEE Symposium on Security and Privacy (SP), 2017.
[16] Pin- Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich, EAD: clastic-net attacks to deep neural networks via adversarial examples. In AAAI, 2018.
[17] Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018
[18] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In ICLR, 2018.
[19] Tianhang Zheng, Changyou Chen, and Kui Ren. Distributionally adversarial attack. In AAAI, 2019.
[20] Eric Wong, Frank R. Schmidt, and J. Zico Kolter. Wasserstein adversarial examples via projected sinkhorn iterations. In /CML, 2019.
[21] Konda Reddy Mopuri, Aditya Ganeshan, and R. Venkatesh Babu. Generalizable data-free objective for crafting universal adversaria perturbations. IEEE Trans. Pattern Anal. Mach. Iniell,, 41(10):2452-2465 2019.
[22] Rima Alaifari, Giovanni S. Alberti, and Tandri Gauksson. Adef: an iterative algorithm construct adversarial deformations. In ICLR, 2019.
[23] Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj, and Venkatesh Babu R. Guided adversarial attack for evaluating and enhancing adversarial defenses. NeurIPS, 2020.
[24] Kaiwen Wu, Allen Houze Wang, and Yaoliang Yu. Stronger and faster wasserstein adversarial attacks. In ICML, 2020.
[25] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
[26] Yunrui Yu, Xitong Gao, and Cheng-Zhone Xu. LAFEAT: piercing through adversarial defenses with latent features. CVPR, 2021.
[27] Minhao Cheng, Thong Le, Pin-Yu Chen, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Query-efficient hard-label black-box attack: An optimization bisizattiio---eeddddde approach. In /CLR, 2019.
[28] Minhao Cheng, Simranjit Singh, Patrick H. Chen, Pin- Chen, Sijia Liu, and Cho-Jui Hsich. Sign-opt:
A query-efficient hard-label adversarial attack. In JCLR, 2020.
[29] Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In ICLR, 2020.
[30] Jianbo Chen, Michael I. Jordan, and Martin J. Wainwright. Hopskipjumpattack: A query-efficient decision-base attack. In IEEE Symposium on Security and Privacy (SP), 2020.
[31] Huichen Li, Xiaojun Xu, Xiaolu Zhang, Shuang Yang, and Bo Li. QEBA: query-efficient boundary-based blackbox attack. In CVPR, 2020.
[32] Ali Rahmati, Seyed-Mohsen Moosavi-Dees -Dzzooli, Pascal Frossard, and Huaiyu Dai. Geoda: A geometric framework for black-box adversarial attacks. In CVPR, 2020.
[33] Ziang Yan, Yiwen Guo, Jian Liang, and Changshui Zhang. Policy-driven attack: Learning to query for label black-box adversarial examples. In ICLR, 2021.

---

[34] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
[35] Yinpeng Dong, Fangzhou Liao, Tiayyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In CVPR, 2018
[36] Cihang Xic, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In CVPR, 2019.
[37] Eric Wong and J. Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In ICML, 2018
[38] Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-Marge training: Scalable certification of perturbation invariance for deep neural networks. In NewrIPS, 2018
[39] Tsui- Wei Weng. Huan Zhang, Pin-Yu Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsich, and Luca Daniel. Evaluating the robustness of neural networks: An extreme value theory approach. In ICLR, 2018.
[40] Matthew Mirman, Timon Gehr, and Martin T. Vechev. Differentiable abstract interpretation for provably robust neural networks. In ICML, 2018.
[41] Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness a classifier against adversarial manipulation. In NeurlPS, 2017.
[42] Mathias Lécuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In Symposium on Security and Privacy (SP), 2019.
[43] Kai Yuanqing Xiao, Vincent Tjeng, Nur Muhammad (Mahi) Shafiullah, and Aleksander Madry. Training for faster adversarial robustness verification via inducing relu stability. In ICLR, 2019.
[44] Jeremy M. Cohen, Flan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. In JCML, 2019.
[45] Mislav Balunovic and Martin Vechev. Adversarial training and provable defenses: Bridging the gap. In ICLR, 2020.
[46] Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsich. Towards stable and efficient training of verifiably robust neural networks. In ICLR, 2020.
[47] Sahil Singla and Soheil Feizi. Second-order provable defenses against adversarial attacks. In ICMI., 2020.
[48] Mislav Balunovic and Martin T. Vechev. Adversarial training and provable defenses: Bridging the gap. In ICLR, 2020.
[49] Difan Zou, Spencer Frei, and Quanquan Gu. Provable robustness of adversarial training for learning halfspaces with noise. In ICML, 2021.
[50] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In ICLR, 2017.
[51] Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. In ICCV, 2017.
[52] Nicholas Carlini and David A. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec@CCS. 2017.
[53] Shixin Tian, Guolei Yang, and Ying Cai. Detecting adversarial examples through image transformatton. In AAAI, 2018
[54] Xingjun Ma, Bo Li, Yisen Wang. Sarah M. Erfani, Sudanthi N. R. Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E. Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018
[55] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NewrIPS, 2018
[56] Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial examples. In NewrlPS, 2018
12

---

[57] Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. In UAI, 2018.
[58] Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting adversarial examples. In ICML, 2019.
[59] Jiayang Liu, Weiming Zhang, Yiwei Zhang, Dongdong Hou, Yujia Liu, Hongyue Zha, and Nenghai Yu. Detection based defense against adversarial examples from the steganalysis point of view. In CVPR, 2019.
[60] Xuwang Yin and Soheil Kolouri Gustavo K. Rohde. GAT: generative adversarial training for adversarial example detection and robust classification. In ICLR, 2020.
[61] Philip Sperl, Ching- Yu Kao, Peng Chen. Xiao Lei, and Konstantin Böttinge. DLA: dense-layer-an lans--layers for adversarial example detection. In IEEE European Symposiun on Security and Privacy, EuroS&P, 2020.
[62] Gilad Cohen, Guillermo Sapiro, and Raja Giryes. Detecting adversarial samples using influence functions and nearest neighbors. In CVPR, 2020.
[63] Fatemeh Sheikholeslami, Ali Lotfi, and J. Zico Kolter. Provably robust classification of adversarial examples with detection. In ICLR, 2021.
[64] Kejjang Chen, Yuefeng Chen, Hang Zhou, Chuan Qin, Xiaofeng Mao, Weiming Zhang, and Nenghai Yu. Adversarial examples detection beyond image space. In ICASSP, 2021.
[65] Puyudi Yang, Jianbo Chen, Cho-Jui Hsich, Jane-Ling Wang, and Michael I. Jordan. ML-LOO: detecting adversarial examples with feature attribution. In AAAI, 2020.
[66] Yao Qin, Nicholas Frosst, Sara Sabour, Colin Raffel, Garrison W. Cottrell, and Geoffrey E. Hinton. Detecting and diagnosing adversarial images with class-conditional capsule reconstructions. In ICLR, 2020.
[67] Jinyu Tian, Jiantao Zhou. Yuanman Li. and Jia Duan. Detecting adversarial examples from sensitivity inconsistency of spatial tratiill-tanffomm domain. In AAAI, 2021.
[68] Yuhang Wu, Sunpreet S. Arora, Yanhong Wu, and Hao Yang. Beating attackers at their own games: Adversarial example detection using adversarial gradient directions. In AAAI, 2021.
[69] Ziang Yan, Yiwen Guo, and Changshui Zhang. Deep defense: Training duns with improved adversarial robustness. In NeurlPS, 2018.
[70] Xi Wu, Uyeong Jang, Jiefeng Chen, Lingjaa Chen, and Somesh Jha. Reinforcing adversarial robustness using model confidence induced by adversarial training. In ICML, 2018.
[71] Qi-Zhi Cai, Chang Liu, and Dawn Song. Curriculum adversarial training. In IJCAI, 2018.
[72] Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversarial perturbations learning from incomplete data. In NeurlPS, 2019.
[73] Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli, Are labels required for improving adversarial robustness? NeurlPS, 2019.
[74] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled data improves adversarial robustness. In NeurlPS, 2019.
[75] Farzan Farnia, Jesse M. Zhang, and David Tse. Generalizable adversarial training via spectral normaliza- tion. In ICLR, 2019.
[76] Chuanhiao Song, Kun He, Liwei Wang, and John John F. Hopcroft. Improving the generalization of adversarial training with domain adaptation. In ICI.R, 2019.
[77] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
[78] Yisen Wang. Xingjun Ma, James Bailey. Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML, 2019.
[79] Florian Tramèr and Dan Boneh Adversarial training and robustness for multiple perturbations. In NewrIPS, 2019.
13

---

[80] Haichao Zhang and Jianyu Wang. Defense against adversarial attacks using feature scattering-base adversarial training. In NewrlPS, 2019.
[81] David Stutz, Matthias Hein, and Bernt Schiele. Confidence--cllibraeed adversarial training: Generalizing to unseen attacks. In ICML, 2020.
[82] Tianyu Pang. Xiao Yang. Yinpeng Dong, Taufik Xu, Jun Zhu, and Hang Su, Boosting adversarial training with hypersphere embedding. In NeurIPS, 2020.
[83] Zhe Gan, Ven-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for ridion-and-o aannn-nnd--aggggn language representation learning. In NerIPS, 2020.
[84] Yinpeng Dong, Zhijic Deng, Tianyu Pang, Jun Zhu, and Hang Su. Adversarial distributional training for robust deep learning. In NewrlPS, 2020.
[85] Jingfeng Zhang, Xilie Xu, Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan S. Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In ICML. 2020.
[86] Tianlong Chen, Sijia Liu, Shiyu Chang. Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning. CVPR, 2020.
[87] Chuanbiao Song, Kun He, Jiadong Lin, Liwei Wang, and John Hopcroft. Robust local features for improving the generalization of adversarial training. In ICLR, 2020.
[88] Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. In ICLR, 2020.
[89] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
[90] Leslie Rice, Eric Wong, and J Zico Kolter. Overfitting in adversarially robust deep learning. In ICML, 2020.
[91] Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening. In ICLR, 2021.
[92] Jingfeng Zhang. Xilie Xu, Bo Han, Tongliang Liu, Lizhen Cui, Gang Niu, and Masashi Sugiyama. Noilin: Improving adversarial training and correcting stcreotype of noisy lahels. Transactions on Machine Learning Research, 2022.
[93] Dinghuai Zhang. Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. In NeurlPS, 2019.
[94] Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In NeurlPS, 2019.
[95] Haizhong Zheng, Ziqi Zhang, Juncheng Gu, Honglak Lee, and Atul Prakash, Efficient adversarial training with transferable adversarial examples. In CVPR, 2020.
[96] Vivek B.S. and R. Venkatesh Babu. Single-step adversarial training with dropout scheduling. In CVPR, 2020.
[97] Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. In NewrIPS, 2020.
[98] Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. In ICLR, 2020.
[99] Preetum Nakkiran. Adversarial robustness may odds with simplicity. arXiv:1901.00532. 2019.
[100] Dong Yin, Kannan Ramchandran, and Peter L. Bartlett, Rademacher complexity for adversarially robust generalization. In ICML, 2019.
[101] Ruiqi Gao, Tianle Cai, Haochuan Li, Cho-Jui Hsieh, Liwei Wang, and Jason D. Lee. Convergence of adversarial training in overparametrized neural networks. In NewriPS, 2019.
[102] Zac Cranko, Aditya Krishna Menon, Richard Nock, Cheng Soon Ong. Zhan Shi, and Christian J. Walder. Monge blunts bayes: Hardness results for adversarial training. In ICML, 2019.
14

---

[103] Huan Zhang, Hongge Chen, Zhao Song, Duane S. Boning, Inderjit S. Dhillon, and Cho-Jui Hsieh. The limitations of adversarial training and the blind-spot attack. In ICLR. 2019.
[104] Chen Liu, Mathieu Salzmann, Tao Lin, Ryota Tomioka, and Sabine Süsstrunk. On the loss landscape of adversarial training: Identifying challenges and how to overcome them. In NeurIPS, 2020.
[105] Kevin Roth, Yannic Kilcher, and Thomas Hofmann. Adversarial training is a form of data-dependent operator norm regularization. In NewrlPS, 2020.
[106] Haotao Wang. Tianlong Chen, Shupeng Gui, Ting-Kuei Hu, Ji Liu, and Zhangyang Wang. Once-for-all adversarial training: In-situ tradeoff between robustness and accuracy for free. In NeurlPS, 2020.
[107] Yi Zhang, Orestis Plevrakis, Simon S. Du, Xingguo Li, Zhao Song, and Sanjeev Arora. Over- parameterized adversarial training: An analysis overcoming the curse of dimensionality. In NeurIPS, 2020.
[108] Yan Li, Ethan X. Fang, Huan Xu, and Tuo Zhao. Implicit bias of gradient descent based adversarial training on separable data. In ICLR, 2020.
[109] Mohammad Mehrabi, Adel Javanmard, Ryan A. Rossi, Anup B. Rao, and Tung Mai. Fundamental tradeoff's in distributionally adversarial training. In ICML, 2021.
[110] Han Xu, Xiaorui Liu, Yaxin Li. Anil K. Jain, and Jiliang Tang. To be robust be fair: Towards faimess adversarial training. /CML, 2021.
[111] Hanshu Yan, Jingfeng Zhang. Jiashi Feng, Masashi Sugiyama, and Vincent F. Tan. Towards adversari- ally robust deep image denoising. In IJCAI, 2022.
[112] Xilie Xu, Jingfeng Zhang, Feng Liu, Masashi Sugiyama, and Mohan Kankanhalli. Adversarial attack and defense for non-parametric two-sample tests. In ICML, 2022.
[113] Moustapha Cisse. Piotr Bojanowski, Edouard Grave Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In ICML, 2017.
[114] Cihang Xie, Mingxing Tan, Boging Gong, Alan Yuille, and Quoc V Lc. Smooth adversarial training. arXiv preprint arXiv:2006.14536 2020.
[115] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robustness via curvature regularization, and vice versa. In CVPR, 2019.
[116] Cihang Xie and Alan L. Yuille. Intriguing properties of adversarial training at scale. In ICLR, 2020.
[117] Hanshu Yan. Jingfeng Zhang, Gang Niu, Jiashi Feng. Vincent Tan, and Masashi Sugiyama. Cifs: Improving adversarial robustness of enns via channel-wise aeportancc-basee feature selection. In ICML, 2021.
[118] Xuefeng Du, Jingfeng Zhang. Bo Han, Tongliang Liu, Rong, Gang Niu, Junzhou Huang, and Masashi Sugiyama. Learning diverse-structured networks for adversarial robustness. In ICMI,, 2021.
[119] Tianyu Pang, Huishuai Zhang, Di He, Yinpeng Dong, Hang Su, Wei Chen, Jun Zhu, and Tie-Yan Liu. Two coupled rejection metries can tell adversarial examples apart. In CVPR, 2022.
[120] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. In JCLR, 2018.
[121] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts In SIGKDD, 2018.
[122] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016.
[123] Alex Krizhevsky, Geolfrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[124] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras. lan Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint A902: 2019.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes] We discuss Appendix
(b) Did you describe the limitations your work? [Yes] The limitations are Appendix
(c) Did you discuss any potential negative societal impacts of your work? [Yes] The societal impacts
are Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] The proof' in Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in supplemental material or as a URL)? [Yes] The source codes are made publicly available.
(b) Did you specify the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The training details are in Appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The devices of computing are in Appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're us-
ing/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [Yes] We discuss our used data in Scction and Appendix.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "xKEnJ3g_3o", "reviewer": "Reviewer_1A7M", "summary": "This work introduces a model collaboration scheme to tackle the insufficient model capacity against adversarial examples instead of model ensemble. Through replacing voting-based strategy with selecting the best-performing sub-model, each sub-model only fits its specific adversarial areas, which enables the models with limited capacity to achieve better adversarial robustness. An auxiliary head which outputs the confidence is introduced to identify the best-performing sub-models. Experiments on CIFAR-10 and ResNet-20 demonstrate the effectiveness of proposed algorithm.", "questions": "Overall, I think this paper is interesting. I have several suggestions for the authors:\n\n1. Please provide some empirical evidence that sub-model handles its corresponding adversarial subspace via collaboration scheme.\n2. Please include some stronger attacks for comparison, such as autoattack.\n", "limitations": "The authors mentioned that they have discussed the limitations in the Appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n* This work is well-organized with clear motivation and generally easy to read.\n* The authors provide the source code with instructions, which makes the algorithm easy to follow.\n* The designing of collaboration scheme seems natural and reasonable with its corresponding theoretical analysis.\n* The comparison with other defense baselines on CIFAR-10 demonstrate the superiority of proposed algorithm.\n\nCons:\n\n* The motivation of this paper is that each sub-model can handle its corresponding adversarial subspace via collaboration scheme so that the adversarial robustness with limited model capacity can be improved. However, it is difficult to the empirical evidence of it in experimental section. For example, some visualization of adversarial examples with decision boundaries of collaborated models are recommended.\n* The performance under different attack settings includes PGD and adaptive attacks is reported. However, some popular stronger attacks are not included for comparison, such as autoattack.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zk_6DhGXNAD", "reviewer": "Reviewer_UDEK", "summary": "This paper proposes a method to improve the adversarial robustness of an ensemble. Instead of using the voting strategy, the proposed method aims to identify the model with the highest confidence on the ground-truth label. In this way, it can improve the model performance when the correct prediction remains with the minority. The authors conduct experiments on both wight-box and black-box attacks to demonstrate the effectiveness of their method.", "questions": "- Given that $g_{\\phi}$ and $f_{\\theta}$ are learned alternatively, how stable is the learning?\n- Which loss function is used in Table 1 and Figure 4? Why does the robustness increase when the weight $\\lambda$ increases? Why is the robustness higher when $\\epsilon=0.07$ than $\\epsilon=0.06$?\n- Have you measured the robustness when the ensemble contained different numbers of sub-models? When using SoE, can you use fewer sub-models to achieve higher robustness than previous voting-based methods?\n", "limitations": "The authors do not address the limitations of their work. ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n+ The authors rethink the classic voting-based strategy and note the problem with the case when correct predictions remain with the minority. Then, they propose a new “collaboration” strategy, which is very novel and interesting.\n+ This paper is well-structured and easy to follow.\n\n[Weaknesses]\n- The formulation of the evaluator head in the SoE is questionable. In SoE, the evaluator head is designed to estimate the predicted probability $\\hat p_y(x)$ through a BCE loss. However, if $g_{\\phi}$ has successfully fitted the probability $\\hat p_y(x)$, then we can directly identify which dimension in $\\hat p(x)$ is the same as $g_{\\phi}$, thereby obtaining the ground-truth label. Therefore, the ensemble seems useless.\n- To learn the collaboration between sub-models, why not directly construct a one-hot label for the best-performing sub-model on each input, and use the cross-entropy to learn the probability of each sub-model being the best-performing model?\n- The utility of the proposed method in minimizing the vulnerability overlap of all sub-models is not directly proven or verified. The main purpose of the proposed SoE is to minimize the vulnerability overlap of all sub-models. However, its effectiveness is not directly demonstrated. I suggest the authors provide some visual demonstrations or quantitative results to compare the vulnerability lap between SoE and other methods.\n- In Algorithm 1, a surrogate loss is used to approximate the loss on the best-performing sub-model. Why not directly use the loss of the sub-model with the largest confidence $g_{\\phi}(x)$?\n- Algorithm 2 is somewhat confusing. The adversarial example is generated to worsen $\\hat{p}(x)$, which is the output of the sub-model with the highest confidence. Therefore, this adversarial example is generated only based on the best-performing sub-model. Actually, such adversarial examples have been used in Algorithm 1, so it is confusing why we need Algorithm 2.\n- The attacking methods used in experiments are not sufficient. First, the proposed method should be at least compared with the PGD/C&W attack on the best-performing sub-model. Second, the attack that simultaneously destroys the estimator output and the predictor head should be considered. Third, the adaptive attack used in the paper is confusing. There should be more discussions about $l_2$ and the selection of $j$. Besides, in $l_2$, $y$ is usually not known by the attacker, so it is unclear how to compute $l_2$.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "77jTnko3Kgx", "reviewer": "Reviewer_t6Y8", "summary": "This paper proposes an adversarial defense method that can better utilize the ensemble of multiple experts. In contrast to previous ensembling methods that require more than half methods give the correct prediction, for each sample, the proposed method employs a router module that routes the sample to the best model for prediction. In this way, the proposed method can yield the correct result if the selected model can predict right. The proposed method demonstrates better performance compared to previous ensemble methods.", "questions": "A more careful and convincing white attack method should be employed to evaluate the proposed method.", "limitations": "No social negative impact is found.", "rating": 7, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The writing is good and easy to follow\n2. There are some theoretical proofs\n\nWeaknesses:\nMy main concern lies in the fairness of evaluation. For white-box attacks, I don't think the non-adaptive SoE can be a fair comparison: all adversarial methods assume the visibility of the whole model, which should include the E head. However, in non-adaptive SoE, E head is not visible. \nThe author also gives a setting with an adaptive adversarial attack that can attack the E head, which is fairer than the previous setting. However, I don't think the adaptive setting is fair enough. For example, besides attacking the best model to be less confident, all worse models should also be encouraged to predict higher confidence. I think this adversarial setting can be more challenging given the E head of worse models can be easier to be fooled for this image (since its backbone is easier to be fooled). \nThis example is one evaluation method. If the author can give a more systematic evaluation setting besides this, I would be more convincing.\n\nAlso, what's the adversarial sample employed in transfer adversarial attack? Is it adaptive or non-adaptive?\n\nSome minor typos:\n1. Algorithm 2, \"duel\" heads should be \"dual\" heads", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept