Collaborate to Defend Against Adversarial Attacks
Authors: Sen Cui
Jingfeng Zhang
Jian Liang
Masashi Sugiyama
Changshui Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: adversarial defense
collaboration
ensemble.

Abstract: Adversarially robust learning methods require invariant predictions to a small neighborhood of its natural inputs, thus often encountering insufficient model capacity. Learning multiple models in an ensemble can mitigate this insufficiency, further improving both generalization and robustness. However, an ensemble still wastes the limited capacity of multiple models. To optimally utilizing the limited capacity, this paper proposes to learn a collaboration among multiple sub-models. Compared with the ensemble, the collaboration enables the possibility of correct predictions even if there exists a single correct sub-model. Besides, learning a collaboration could enable every sub-model to fit its own vulnerability area and reserve the rest of the sub-models to fit other vulnerability areas. To implement the idea, we propose a collaboration framework---CDA$^2$ the abbreviation for Collaborate to Defend against Adversarial Attacks. CDA$^2$ could effectively minimize the vulnerability overlap of all sub-models and then choose a representative sub-model to make correct predictions. Empirical experiments verify that CDA$^2$ outperforms various ensemble methods against black-box and white-box adversarial attacks.

Under review as a conference paper at ICLR 2022
COLLABORATE TO DEFEND AGAINST ADVERSARIAL ATTACKS
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarially robust learning methods require invariant predictions to small neighborhood of its natural inputs, thus often encountering insufficient model ca- pacity. Learning multiple sub-models in ensemble can mitigate this insufficiency, further improving both generalization and robustness. However, an ensemble still wastes the limited capacity of multiple models. To optimally utilize the limited capacity, this paper proposes to learn a collaboration among multiple sub-models. Compared with the ensemble, the collaboration enables the possibility of correct predictions even if there exists single correct sub-model. Besides, learning a collaboration could enable every sub-model to fit its vulnerability area and reserve the rest of the sub-models to fit other vulnerability areas. To implement the idea, we propose collaboration framework CDA the abbreviation for Collaborate to Defend against Adversarial Attacks. CDA² could effectively minimize the vul- nerability overlap of all sub-models and then choose a representative sub-model to make correct predictions. Empirical experiments verify that CDA² outperforms various ensemble methods against black-box and white-box adversarial attacks.
INTRODUCTION
Safety-critices applications (such as in medicine and finance) require the adversarial robustness of deep models (Goodfellow et al., 2015; Szegedy et al., 2014). An adversarially robust learning method requires invariant predictions to a small neighborhood of its natural inputs, thus often encountering insufficient model capacity (Zhang et al., 2021; Yu et al., 2021a). This limits the further improvement of robustness and has the undesirable degradation generalization (Madry et al., 2018).
Learning multiple sub-models in ensemble (Breiman, 1996; Freund et al., 1996) can mitigate this insufficiency (Pang et al., 2019; Kariyappa & Qureshi, 2019; Yang et al., 2020a). Remarkably, Pang et al. (2019), Kariyappa & Qureshi (2019) and Yang al. (2020a) minimized the vulnerability overlaps between each pair of sub-models and improved both robustness and generalization over a single model.
However, an ensemble wastes the limited capacity of multiple models. In the example of three sub-models (see Figure 1(b)), the adversarial input that lies in the black areas can fool the ensemble successfully, i.c., more than half of sub-models must correctly classify the adversarial input. Therefore, the ensemble's 's voting-based strategy excludes the possibility that true predictions remain with the minority. Besides, learning ensemble requires more than half of the sub- sub-models to fit the same vulnerability areas, which leaves the following question unanswered whether we could only leverage single sub-model to fit vulnerability area and reserve the rest of the sub-models to fit other vulnerability areas.
To optimally utilize the limited capacity, this paper proposes to learn a collaboration among multiple sub-models. As shown in Figure I(c), the adversarial input that lies in the vulnerability overlaps of all sub-models can undoubtedly fool the collaboration. Compared with the ensemble in Figure 1(b)), collaboration enables the possibility of correct predictions even if there exists a single correct sub- model merely. Besides, learning collaboration could enable every sub-model to fit its vulnerability area, which could collectively fix broader vulnerability areas than the ensemble does. Then, sub- models could collaboratively choose trustworthy ones to make the final predictions.

---

Under review as a conference paper at ICLR 2022
valuorabli
(a) Single model
(b) Ensemble
(c) Collaboration
Figure 1: Illustrations of the vulnerability of (a) Single model (b) Ensemble, and (c) Collaboration. The black area represents the vulnerability area which the model is undoubtedly fooled.
To realize the idea, we propose a collaboration framework- Collaborate to Defend against Adversarial Attacks (CDA²) (Algorithms and 2). CDA², each sub-model has dual heads: one outputs vector of predicted probability fo()); another outputs scalar that measures posterior probability density (PPD) of the prediction. In the training phase, given a natural or adversarial input æ, each sub-mat mooemmdel chooses easy one(s) to feed itself. The PPD head is meanwhile updated by comparing the predicted probability on the true
(-) (a scalar). In the inference phase, given an input, CDA chooses a sub-model with the largest PPD value as the representative to output the prediction.
We highlight our key contributions as follows.
We provide a new perspective on learning multiple sub-models for defending against adversarial attacks. We theoretically show the collaboration makes better decisions than the ensemble, which implies collaboration may fix broader vulnerability areas.
We propose a novel collaboration framework-CDAA (see Section 3.2). In the training phase, CDA2could effectively minimize the vulnerability overlap of all sub-models; In the inference phase, CDA could effectively choose representative sub-mode to make correct predictions. We also provide comprehensive analysis illustrating the rationale of CDA?.
Empirical experiments verify that CDA? outperforms various ensemble methods against black-box and white-box adversarial attacks.
RELATED WORKS
Adversarial attack Adversarial attacks aim craft the human-imperccetibble adversarial input fool the deep models. Adversarial attacks could be roughly divided into white-box attacks in which the adversary is fully aware of the model's structures (Goodfellow al., 2015; Moosavi-Dezfooli et al., 2016; Carlini & Wagner, 2017b; Chen et al., 2018; Athalye et al., 2018; Xiao et al., 2018; Zheng et al., 2019; Wong et al., 2019; Mopuri et al., 2019; Alaifari al., 2019; Sriramanan al., 2020; Wu et al., 2020b; Croce & Hein, 2020; Yu et al., 2021b) and black-box attacks in which the deep models are treated as black boxes to adversary (Cheng et al., 2019; 2020; Wu et al., 2020a; Chen et al., 2020a; Li al., 2020a; Rahmati et al., 2020; Yan et al., 2021b; Hendrycks al., 2021; Dong et al., 2018; Xie et al., 2019). This paper focuses on building effective defense and select both white-box and black-box attack methods our robustness evaluation metrics.
Adversarial defense Defending adversarial attacks is a challenging task and researchers have proposed various solutions. Certified defense tries learn provably robust deep models against norm-bounded (e.g., l, and fxe) perturbations (Wong & Kolter, 2018; Tsuzuku et al., 2018; Weng al., 2018; Mirman al., 2018; Hein & Andriushchenko, 2017: Lécuyer al., 2019; Xiao et al.. 2019; Cohen et al., 2019; Balunovic & Vechev, 2020a; Zhang et al., 2020a; Singla & Feizi, 2020; Balunovic & Vechev, 2020b; Zou et al., 2021). Empirical defense leverages adversarial data to build effective defense such as adversary detection (Metzen et al., 2017: Li & Li, 2017; Carlini & Wagner, 2017a; Tian et al., 2018; 10 Ma et al., 2018b; Lee et al., 2018; Pang et al., 2018; Smith & Gal, 2018; Roth et al., 2019; Liu et al., 2019; Yin & Rohde, 2020: Sperl et al., 2020; Cohen et al., 2020; Sheikholeslami et al., 2021; Chen et al., 202 la; Yang et al., 2020b; Qin et al., 2020; Tian et al., 2021; Wu et al., 2021) and adversarial training (AT), in which AT stands out as the most effective defense. Researchers have investigated various aspects of AT, such as improving AT's robustness or generalization (Madry et al., 2018; Yan et al., 2018: Wu et al., 2018: Cai al, 2018; Najafi et al.,
2

---

Under review as a conference paper at ICLR 2022
2019; Alayrac et al., 2019; Carmon al., 2019; Farnia et al., 2019; Song al., 2019; Zhang et al.,
2019b; Wang et al., 2019; Tramèr & Boneh, 2019; Zhang & Wang, 2019; Stutz et al., 2020; Pang et al., 2020; Gan et al., 2020: Dong et al., 2020; Zhang et al., 2020b; Chen et al., 2020b; Song et al., 2020; Ding et al., 2020; Wang et al., 2020b; Zhang et al., ), fixing AT"s undesirable robust overfitting (Rice et al., 2020; Chen et al., 2021b), improving AT"s training efficiency (Zhang et al., 2019a; Shafahi et al., 2019; Zheng et al., 2020; B.S. & Babu, 2020; Andriushchenko & Flammarion,
2020; Wong al., 2020), understandingiiieeppreing AT's unique traits (Nakkiran, 2019; Yin et al.,
2019; Gao et al., 2019; Cranko et al., 2019; Zhang et al., 2019c; Liu et al., 2020; Roth et al., 2020; Wang et al., 2020a; Zhang al., 2020c; Li et al., 2020b; Zou al., 2021; Mehrabi al., 2021; Xu et al., 2021), etc. Besides, researchers have alao actively investigated robust-structured models (Cisse al., 2019; Xie & Yuille, 2020; Yan et al., 2021 la; Du et al., 2021; Pang et al., 2021). Nevertheless, the above research thoroughly investigated single model; this paper focuses on the collaboration among multiple models for adversarial defense.
et al., 2017; Xie et al., 2020; Moosavi-Dezfooli et
Ensemble methods for adversarial robustness The most relevant studies are the ensemble meth- ods. Ensemble methods such as bagging (Breiman, 1996) and boosting (Freund et al., 1996) have been investigated for significantly improving the model's generalization. Motivated by the benefits of ensemble methods in improving generalization, researchers introduced an ensemble to improve the model robustness (Yang et al., 2020a; Kariyappa & Qureshi, 2019; Pang al., 2019; Tramèr et al., 2018). Tramèr et al. (2018) proposed to reduce the adversarial transferability by training a single model with adversarial examples from multiple pretrained sub-models. Pang et al. (2019) introduce regularization method-A ADP--AD--6 encourage high diversity in the non-maximal predictions of sub-models. Kariyappa Qureshi (2019) improved the ensemble diversity by maximizing the introduced cosine distance between the gradients of sub-models with respect to the input. Yang et al. (2020a) proposed to distill non-robust features in the input and diversify the adversarial vulnerability. These methods reduced overlaps of vulnerability areas between sub-models (Yang et al., 2020a).
To further improve the ensembles, mixture-of ex-ertsss (MOE) assume that the problem space can be divided into multiple sub- sub-problems through a gate module; the gate module specifies each sub-model on a specific sub-problem (Jacobs et al., 1991; Ma et al., 2018a).
Nevertheless, the best of our knowledge, MOE-based methods have been not applied help adversarial robustness. Inspired by MOE, we propose the collaboration framework to defend against adversary attacks.
COLLABORATION TO DEFEND AGAINST ADVERSARIAL ATTACK
3.1
SUPERIORITY OF COLLABORATION
This section shows collaboration, in theory, could make better decisions than an ensemble.
Ensemble Suppose that there are M learned sub-models (Jor Jeg...., fox. ). given an input x. M sub-models make predictions fo (z), for (z), (u)) }. The ensemble outputs final prediction ensemble(x,, fo., Soxz) by the voting-based strategy:
M Σ
ensemble(z, fo.. Fow.) )= = arg max (1....,,
(I)
where 1 is the indicator function. Note that the ensemble outputs the predicted label y that agrees with the majority predictions of the sub-models.
Definition 1 (best-performing sub-model). Given input and its label y, the best pest-performing sub-model achieves the lowest objective loss on the data (I, u) among all M sub-models:
foreen (x)
min
e(fo, (x).
f0, Se
Note that the best- pest-performin sub-model w.r.t. the input data (x,, y), i.c., different input data correspond to different test-performing sub-models.
3

---

Under review as a conference paper at ICLR 2022
Collaboration Suppose that there are M learned sub- {fes fez..... }- Given input 2,
sub-models make predictions {Jo, (x)) Jo, (2)), (w))(w)) }. 'hee collaboration tries to output a final prediction collaboration (z, for. fess) by the best rest-performing sub-model:
collaboration(r, fo.,... fe = ferme (z).
(3)
Given M learned sub-models, the predicted accuracy of the collaboration is upper- bounded that of the ensemble, i.e., (z)g)) f
Proposition 1.
)=y] D( Lemsemble(s.foo....... ]=y]
(4)
Proof. Given (x. v) D, the ensemble's prediction is correct, at least one sub-mode makes correct prediction, i.e., f (x)my holds; therefore, the collaboration' prediction is correct. the collaboration's prediction is correct, there exists a case that the majority of sub-models make consistent but wrong predictions, while single sub-model's prediction is correct; then, ensemble's prediction is wrong. Therefore, Proposition holds.
From Proposition 1, a collaboration can theoretically achieve an equal or higher performance than an ensemble. Next, we will introduce realization of our collaboration framework.
3.2 REALIZATION OF COLLABORATION FOR DEFENDING AGAINST ADVERSARIAL. ATTACK
This section realizes the framework of Collaboration for Defending against Adversarial Attack (CDA3).
Notation We firstly introduce the needed notations. Suppose X' and y denote input space and output space, where y {1,.... K} for K-class classification problem. There are N samples in the dataset D ((x, y) where x E X and y V. Let dnn x') || x oc denotes the infinity distance metric, and [x] = {x' E X diat (x,x²') ) S c is the closed ball of of radius e >> 0 centered To search for adversarial data within norm ball [.[x], Madry et al. (2018) proposed projected gradient descent (PGD) method that iteratively searches for adversarial data ă (z refers to natural data). f1) outputs K-dimensional predicted probability, i.e., p(x) [pz ... (x))..
9
positive sample negative sample sub-model a sub-modiel b
PPD head
PPD
X
(scalar)
feature exactor
label head
p(x)
correctlyy classified
misclasalfied
(K-dim vector)
fe
Sample
(a) Assign data to their best- performiff sub-models
(b) Dual-head structured sub-models
Figure 2: (a) The blue and pink lines denote two sub-models. Each sub-model makes negative predictions (-) on its left and makes positive predictions on its right (+). The given data will be assigned to the sub-model that has the lowest objective loss. The arrows represent the data assignment. (b) Each sub-model has two heads-lubel head that outputs the predicted probability (vector) and PPD head that approximates the value of posterior probability density (scalar) of the predicted probability.
Goal of collaboration 1) ensure the correct prediction of the best best-performing sub-model for given input, and 2) select the best-performing sub-model among all sub-models make prediction.
First, intuitively, every sub-model in a collaboration should maximize its expertise to fit its areas and leave the remaining areas fitted by others. As result, the collaboration can minimize the vulnerability overlaps of all sub-models. Section 4 shows "minimizing the vulnerability overlap of all sub-models" "minimizing the objective loss the best-performing sub-models". Therefore, during the training phase, the given data should always be allocated to the sub-model that has the lowest objective loss. In other words, the sub-models always choose the easiest data to learn. In the example of Figure 2(a), i) Data() is misclassified by both sub-modek The blue sub-model is near Data() and has the lowest objective loss. We assign the blue sub-model to fit Data@). ii) Data is correctly classified by the pink model but wrongly classified by the blue model; for ease of effort, assign the pink model to
4

---

Under review as a conference paper at ICLR 2022
PGD
Jo
foz (#01)
y
Were (E9) 1.u)
PPD
f0r (5
sub-model
Pa
BOE loss
Lor (a
y
(ioo (3900)
(x(
fer (i"Me)
Jo
(a) optimizing the module fo Figure 3: Optimization process of the M sub-models in collaboration
(b) optimizing 9o
fit Data@), because the collaboration correctly be classified Data²) through selecting the pink model as the representative. dii) Data() is correctly classified by both models. The blue model is far from D and takes the lowest effort on fitting it: therefore, assign the blue model fit Data().
Second, to select the best-performing sub-model, we construct dual-head structured sub-models. As shown in Figure 2(b), our sub-model has dual heads: 1) fa contains the feature extractor module and the label head and predicts the label probability fa(æ)) = p(x) = (x)..... pr (I)] (a vector); 2) the PPD head 9s approximates the posterior probability density (PPD) (a scalar) of each prediction p(x).
Note that the PPD is the likelihood that a given prediction p(x) equals the true label distribution p(z). Thus, the PPD value measures the quality of the sub-model's s predictions. In other words, the best persforerfrring sub-models can be decided by the PPD values because the largest PPD value corresponds to the lowest objective loss, and vice versa (see theoretical proof in Proposition 2).
Collaborate to defend against adversarial attack To defend against adversarial attack, the col- laboration needs to learn from adversarial data. Algorithm along with Figure 3 articulates how to learn such the collaboration, inculding the natural data training and the adversarial data training of the collaboration. Algorithm 2 articulates how use the collaboration make predictions. For the natural data training of the collaboration, as shown in Figure 3(a), given natural training data v and its label y. we use the PGD method to obtain M adversarial variants {ā" L.M. of M sub-models as existing baselines do. For each adversarial variant , we assign the best-performing sub-model to learn and update its feature extractor and label head. This process corresponds to Lines 2-5 and Lines 9-10 Algorithm 1. Note that in Line 8, we use a surrogate loss to approximate the sub-model assignment process (reasons see Eq. 6 and 7). As shown in Figure 3(b), given an adversarial variant i": we propose to use the binary-cross- cross-erroeett (BCE) loss between the predicted label probability on the true label (i.e., py (z%) )) and the approximated PPD (i.e., (5 )) to update each sub-model's 's PPD head. This process corresponds to Lines 6-7 Algorithm 1.
The natural data training of the collaboration using the most adversarial data of each sub-model may converge without full exploration of the adversarial samples. For the adversarial data training of the collaboration, the collaboration. In particular, for each data sample x, we output the prediction p(xx whose PPD value (x) is the highest. We perturb x to T to worsen the prediction using PGD method. Then we minimize a surrogated loss to fit this adversarial data ă. This process corresponds to Lines 11-18. In our implementation, the natural data training of the collaboration is not necessary. Because the adversarial data training of the collaboration also captures the most adversarial samples of sub-models and can provide more other adversarial samples for the collaboration.
propose to generate the adversarial samples that can worsen the outputs of
During the inference phase shown in Algorithm 2, once M sub-models are properly learned, CDA chooses representative sub-model whose PPD value is highest among all sub-models, and then outputs this sub-model's prediction.
3.3 ANALYSES OF CDA
Optimizing the best-performing sub-modeis We firstly show that minimizing the vulnerability overlap of all sub-models is equal to minimizing the objective loss of the best perfformno sub-models. For ease of optimization of the best-performing sub-model, provide surrogate loss.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Collaboration for Defending against Adversarial Attack (CDA?) (training phase)
Input: sub-models with duel heads {fo, ;=1 and (go, | where fo, outputs the label prediction and fé, outputs the approximated PPD), training dataset D, hyperparameter o;
the natural data training of the collaboration
2:
for each data (z, y) E D do for each sub-model Se, = =1 2..,....
3:
4:
Obtain the adversarial data n of the sub-model So, using PGD method;
5:
for each sub-model fe, j = 1, 2, M do
6:
Calculate the approximated PPD, i.e. Poy (T%) Minimize BCE loss l = BCE (i")), (y (z":)) to update module g0,
7:
8:
Collect sub-model i's cross entropy (CE) loss data ;0:; 8 (So, (5 .):
Calculate surrogate loss data ;0:: ém a In Σド= exp (- (*% ).9)))))
9:
Update (fo,) by minimizing lmi // choose the best-performing sub-model to fit jº: 11: the adversarial data training of the collaboration
10:
12:
for each data {2r, y) E do for each sub-model fa. 1, .... .M do
14:
Calculate the approximated PPD, i.c., 96. (æ);
Calculate the prediction i.e., fo. (I:);
16:
Output the prediction (xx) with the highest PPD value;
17:
Obtain & by perturbing x to
the prediction; // generate the adversarial samples of the
collaboration
Minimize BCE loss (T( update the module gø of all sub-models;
19:
Update { to fit E by minimizing the surrogate loss &. (7);
20:
return the learned sub-models ] with {go.)
The vulnerability overlap of all sub-models refers the set of adversarial data (i, that misclassified by all sub-models, i.e., all sub-models' objective loss is higher than a certain degree
min E(fo, (ă,0)) y)) > 6, where 2, E D,
(5)
where ò denotes the vulnerability overlap of all sub-models.
To reduce the vulnerability overlap of all sub-models, we only need to reduce objective loss of a single model, which is equal to minimizing the loss of the best-performing sub-model, i.c.,
min
L(Wa)D
min [E. 02....0 (0,.00.0.... .2.....0s1 Bmr) (fo, (F%)))
(6)
(e1.0a...,.M.) 0m)
where f0: is the adversarial data generated by the sub-model fe.-
While directly performing the outer minimization in (6) may cause a trivial solution (c.g.. there is only one optimized sub-model), for ease of the optimization of (6) (Corresponding Lines 8-9 in Al Algorithm 1), we provide a surrogate objective as follows.
min (01.0.....0a)) E(x.w)e
(3 .1)).))).. (3)
where ém (i": -o En exp
- -e(1o, and o > is pre-defined hyper-parameter.
In (7), we approximate the objective ming, co0, Px)) e (Fo, ( using smooth surro- gated maximum function due
min 0, (6 (61.00.....0s))
é (Ja, (z - In(M) s (z a, 22...0
min (Jo, (,)) (8)
The proof of (8) in Appendix.

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Collaboration for Defending against Adversarial Attack (CDA?) (inference phase)
Input: the learned sub-models {fo. A with (9s. 1 test input æ.
I: for all sub-models fo, i = ,,.,, M (in parallel) do
2: make label predictions f(z) fe. (4)) and output approximated PPD value Iø, (a));
3:
return prediction ((x) whose PPD value (x) is the highest among M sub-models.
The berf-performing sub-model has the highest PPD. We show that a sub-model with the highest PPD achieves the minimum of the objective loss among all sub-models, i.e., the rest-performing sub-model.
According to Bayesian theory, for a given input æ in K-class classification problem, the true probability of the corresponding label p(z) = [pr (z)..... Px (2)] px(2)) ] is agnostic. Usually we assume p(x) comes from a prior probability distribution, e.g., p(x) ~ Dir((x), where Dir(a)) is the Dirichlet distribution with the pre-defined parameter vector [ai, 012, .... ax). The prior probability density function of Dir( is
022,..,
II ((())a11 where B(a)= = 1 ((s)) ((ao)
K
K
f(p(.), - B(a)
OK
OK
=
1=1
where (()) denotes the Gamma function. According to the Bayesian views, when we learn the model in supervised manner, the label y happens times after observations given the data (I, D. Therefore, the posterior probability distribution is Dir(a') where a' = [al, Oty + (.k. and the posterior probability density (PPD) function is
PPD(p(z), B(a')
II p(=)a--1 P py(x) i1,,iv
(10)
where PPD corresponds to the likelihood of given p(x) being the probability of the true label p(r). In other words, the predictions p(x:) from the sub-model fo, with higher PPD is more likely to be the true label probability. Therefore, we propose to use PPD to measure the quality of predictions. In (10), when n (i.c., the number of posterior observations) is sufficiently large, Py (y ( becomes a dominant part; therefore, the value of PPD monotonically increases w.r.t. the value of (z).. Consequently, we have the following proposition.
Proposition 2. Given an input I, the sub-model that has the highest PPD corresponds to the best-performing sub-model, ie..
arg max jE( {
(2). min E(fo, (z).v).
(11)
JE(1.2...
From Proposition 2. the PPD value can decide the best-performing sub-model. Given an input to the collaboration, dual-head structured sub-models can collaboratively decide the best-performing sub-model by just comparing the values of the PPD head (corresponds to Algorithm 2). To learn the PPD head, we could simply compare its output with the predicted probability the true label (i.e..f, (x)), then update the PPD head by gradient descent (corresponds to Lines 6-7 in Algorithm 1).
Note that the PPD head may be susceptible adversarial attacks in the white-box setting. In our implementation, we use a simple linear structure to regress the PPD value.
4 EXPERIMENTS
In this section, we provide a synthetic experiment to illustrate the behavior of our method compared with prior ensemble methods. Then, we provide series of experiments on a benchmark dataset to verify the effectiveness of our method in defending against adversarial attacks.
4.1 EXPERIMENTAL SETUP
Following the work in (Yang et al., 2020a), we compare our method with various related methods, including ADP (Pang et al., 2019), GAL (Kariyappa & Qureshi, 2019), DVERGE (Yang et al., 2020a). We use ResNet-20 (He et al., 2016) sub-models in all methods for fair comparisons, and CIFAR 10 as the data set, a classical image dataset (Krizhevsky et al., 2009) that has 50,000 training images and 10,000 test images.
7

---

Under review as a conference paper at ICLR 2022
4.2 SYNTHETIC DATA EXPERIMENTS
To demonstrate the behavior of collaboration, we apply our collaboration framework on the well- known XOR problem as an example.
In XOR problem, there is binary training set D = {zy, W.) where y/s E {=1}. For the samples with the label y/ = the input feature is independently sampled from the two Gaussian distributions x E N(μ
[1, 1]. o 0.1 I2) and x E WN [-1, 1] = 0.1 I2), where
N(p = [1. 1], o = 0.1 [2] denotes 2-d Gaussian distribution with the mean vector μ4 [1. 1] and
covariance matrix a 0.1 Iz. For the samples with the label y; -1, we sample the input feature
independently x; EN - 1-, = 0.11 I2) and x; E NN = [1, -1, = 0.1 0.1.12).
Suppose there are two linear sub-models (xx a x + b and f2(x) 02 x: b2. Ensemble methods output prediction pf(x) by a voting-based strategy e.g., averaging the predictions as the output, i... p(x) M7
Σ fu, (x).
Collaboration aims to 1). minimize the mean square error (MSE) of the best-performing sub-models; 2). optimize the module 91 and 92 to measure the quality of the multiple predictions during inference. From Figure 4(a), learning multiple linear sub-models and averaging the predictions (ensemble)
averaging linear sub-models
collaboration
mob-medd/; f
sub roddd fz
0
-1
-2
1
-2
-1
0
X1
(a) ensemble
(b) collaboration
Figure 4: Illustration of the behavior of the ensemble and the collaboration.
is still a linear model, so it cannot tackle XOR problem. Collaboration can address XOR which classifies each sample by learning to specify the sub-tasks to different sub-models. From Figure 4(b), the samples are assigned to sub-models, in which the blue samples are assigned to fi and the red samples are assigned to fz. Finally, output the prediction by identifying the best-performing sub-models
4.3 PERFORMANCE ON WHITE-BOX ATTACK
As there are mainly two threat modes in the adversarial attack setting: white-box attack and black-box attack. White-box attack refers to that attackers know all the information about the models, including training data, model architectures, and parameters, while black-box attackers have no access to the information about the model's 's structures and parameters and rely on surrogate models to generate transferable adversarial examples.
We compare method with baselines (GAL (Kariyappa & Qureshi, 2019), DVERGE (Yang et al., 2020a) and ADP (Pang et al., 2019)) on defending white-box attacks using a subset of CIFAR10. Following in the setting in Yang et al. (2020a), the step size of e/5 to attack all methods. In particular, we randomly select 1000 samples under different e. For the PGD attack, we select the cross-entropy loss update the perturbations search for adversarial samples. In addition to the accuracy under different e. we also report the performance of all methods on clean data with the adversarial training which e 0.03.
use 50-step PGD with five random starts and
Table 1: Robustness results (%) under white-box attack.
clean
0.01
0.02
0.03
0.04
0.05
0.06
0.07
methods
GAL
81.2
49.5
31.4
25.4
22.7
18.4
13.4
9.0
DVERGE
79.7
67.3
52.3
41.1
29.9
22.5
14.2
10.0
ADP
85.4
67.7
52.9
40.8
30.8
25.8
23.4
20.3
CDA
80.2
72.0
57.5
47.8
38.7
30.4
24.3
24.0
8

---

Under review as a conference paper at ICLR 2022
From Table 1, CDA achieves a better robustness performance under white-box attack. The results verify that collaboration significantly improves the utilization of the limited model capacity. Therefore, CDA² can fit more adversarial data and has relatively smaller vulnerable area.
4.4 PERFORMANCE ON BLACK-BOR BOX ATTACK
Due to the transferability of adversarial examples, transfer adversaries can craft adversarial examples based on surrogate models and perform an attack on the target model. In our experiments, we follow the transfer attack setting in (Yang et al., 2020a) and select 1000 test samples randomly. We select 1000 test samples randomly and use hold-out baseline ensembles with three ResNet-20 sub-models as the surrogate models to generate adversarial samples. In particular, we use three attack methodologies: PGD with momentum (Dong et al., 2018), SGM (Wu et al., 2020a) which adds weight to the gradient through the skip connections of the model, and M-FGSM (Xie et al., 2019) which randomly augments the input images in each step. For each sample, three adversarial variants using the three attack methods. Only when the model can classify all kinds of adversarial variants can the model successfully defend against adversarial attacks. We show the results of all methods in Table 2. In our experiments, GAL is hard to optimize in adversarial training. From Table 2,
Table 2: Robustness results (%) under transfer attack.
e
0.01
0.02
0.03
0.04
0.05
0.06
0.07
methods
GAL
65.1
49.9
49.7
47.3
53.4
51.1
42.2
ADP
85.6
83.0
79.3
79.0
69.6
60.4
57.4
DVERGE
83.4
80.1
77.3
72.4
71.9
68.8
66.2
CDA
85.4
83.4
79.3
77.0
74.2
72.3
70.2
we show the transfer attack robustness ensemble methods across a wide range of attack radius 0.01 S 0.07. When 0.01 I S 0.04, CDA? achieves a comparable performance compared with the SOTA method. With the increase of c, the volume of c-ball increases exponentially, the performances of all methods get worse significantly because of insufficient model capacity. Since CDA addresses more adversarial data using a collaboration mechanism, it achieves a relatively better robustness performance 0.05 S 0.07.
In addition to transfer attack, query attack can also craft adversarial samples hased on the predicted scores of the model. To evaluate the robustness of of CDA² under query attack, we use Square Attack method (Andriushchenko et al., 2020) to attack all method. Square Attack selects localized square-shaped updates random positions each step (Andriushchenko al., 2020). In particular, we set e = 0.03 and learn the models for each method with adversarial training. Then we evaluate the robustness of each method under Square Attack with 5000 iterations and the results are presented in Table 3. From Table 3, CDA² outperforms the baselines under query attack. CDA? optimizes the
Table 3: Robustness results (%) under Square Attack.
method GAL ADP DVERGE Acc(%) 25.0
CDA
47.2
48.5
53.0
utilization of the model capacity by specifying each sub- sub-models to handle the "specific" adversarial attacks, which defends against query attacks more efficiently.
CONCLUSION
In this paper, we study an essential question in the field of adversarial attacks that when we should collaborate. (i) If a single model can handle everything, there is no need for multiple models. (ii) If a single model can only handle a part of the whole, collaboration among multiple models makes sense. Adversarial defense is a typical task that falls into the circumstance (ti) because a single model hardly fits adversarial data. We provided a collaboration strategy ensemble methods, and empirical experiments indeed verified the efficacy of CDA?. Future work includes applying our collaboration framework to other areas such as kernel methods, fairness, and global federated model, etc.
as the defense

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
In this work, we investigate the robustness issue in deep neural networks (DNNs), We propose collaboration mechanism for reducing the vulnerability of existing DNNs to enhance their trustwor- thiness. Having reading the codes of ethics, we make sure that our work conforms to them. In our experiments, we use a public data which complies with requirements.
REPRODUCIBILITY STATEMENT
Our instructions and experimental settings are illustrated the maintext. For the theoretical results, provide rigorous analysis and detailed proofs in maintext and the appendix. For more experimental results and the discussions about CDA? please refer to the Appendix. The source code of our method attached in Appendix.
REFERENCES
Rima Alaifari, Giovanni S. Alberti, and Tandri Gauksson. Adef: an iterative algorithm construct adversarial deformations. In ICLR, 2019.
Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In NeurIPS, 2019.
Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. In NeurIPS, 2020.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pp. 484-501. Springer, 2020.
Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In ICML, 2018.
Mislav Balunovic and Martin Vechev. Adversarial training and provable defenses: Bridging the gap. In ICLR, 2020a.
Mislav Balunovic and Martin T. Vechev. Adversarial training and provable defenses: Bridging the gap. ICLR, 2020b.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
Vivek B.S. and R. Venkatesh Babu. Single-step adversarial training with dropout scheduling. In CVPR, 2020.
Qi-Zhi Cai, Chang Liu, and Dawn Song. Curriculum adversarial training. In IJCAI, 2018.
Nicholas Carlini and David A. Wagner. Adversarial examples not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AlSec@ CCS, 2017a.
Nicholas Carlini and David A. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium Security and Privacy, SP, 2017b.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C. Duchi. Unlabeled data improves adversarial robustness. In NewrlPS, 2019.
Jianbo Chen, Michael I. Jordan, and Martin J. Wainwright. Hopskipjumpattack: A query-efficient decision- based attack. In IEEE Symposium on Security and Privacy, SP 2020, 2020a.
Kejiang Chen, Yuefeng Chen, Hang Zhou, Chuan Qin, Xiaofeng Mao. Weiming Zhang, and Nenghai Yu. Adversarial examples detection beyond image space. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2021a.
10

---

Under review as a conference paper at ICLR 2022
Pin- Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsich. EAD: elastic-net attacks to deep neural networks via adversarial examples. In AAAI, 2018.
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning. In CVPR, 2020b.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may mitigated by properly learned smoothening. ICLR, 2021b.
Minhao Cheng, Thong Le, Pin- Yu Chen, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Query-efficient hard-label black-box attack: An optimization-baset approach. In ICLR, 2019.
Minhao Cheng. Simranjit Singh, Patrick H. Chen, Pin-Yo Yu Chen, Sijia Liu, and Cho-Jui Hsieh. Sign-opt: A query-efficient hard-label adversarial attack. In ICLR, 2020.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In ICML, 2017.
Gilad Cohen, Guillermo Sapiro, and Raja Giryes. Detecting adversarial samples using influence functions and nearest neighbors. In CVPR, 2020.
Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. ICML, 2019.
Zac Cranko, Aditya Krishna Menon, Richard Nock, Cheng Soon Ong, Zhan Shi, and Christian J. Walder. Monge blunts bayes: Hardness results for adversarial training. In ICML, 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. In ICLR, 2020.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018.
Yinpeng Dong, Zhijie Deng, Tianyu Pang. Jun Zhu, and Hang Su. Adversarial distributional training for robust deep learning. In NeurlPS, 2020.
Xuefeng Du, Jingfeng Zhang, Bo Han, Tongliang Liu, Yu Rong, Gang Niu, Junzhou Huang, and Masashi Sugiyama. Learning diverse-structured networks for adversarial robustness. In Proceed- ings of the 38th International Conference Machine Learning, 2021.
Farzan Farnia, Jesse M. Zhang, and David Tse. Generalizable adversarial training via spectral normalization. In ICLR, 2019.
Yoav Freund, Robert E Schapire, et al. Experiments with a new boosting algorithm. In ICML, 1996.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for lision-and-language representation learning. In NerIPS, 2020.
Ruiqi Gao, Tianle Cai, Haochuan Li, Cho-Jui Hsich, Liwei Wang, and Jason D. Lec. Convergence of adversarial training overparametrized neural networks. In NewrIPS, 2019.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In ICLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In NeurlPS, 2017.

---

Under review as a conference paper at ICLR 2022
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021.
Robert A Jacobs, Michael I Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Sanjay Kariyappa and Moinuddin Qureshi. Improving adversarial robustness of ensembles with diversity training. arXiv preprint arXiv:1901.09990 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009,
Mathias Lécuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness to adversarial examples with differential privacy. In Symposium on Security and Privacy (SP), 2019.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurlPS, 2018.
Huichen Li, Xiaojun Xu, Xiaolu Zhang, Shuang Yang, and Bo Li. QEBA: query-efficient boundary- based blackbox attack. In CVPR, 2020a.
Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. In ICCV, 2017.
Yan Li, Ethan X. Fang, Huan Xu, and Tuo Zhao. Implicit bias of gradient descent based adversarial training on separable data. In ICLR, 2020b.
Chen Liu, Mathieu Salzmann, Tao Lin, Ryota Tomioka, and Sabine Süsstrunk. On the loss landscape of adversarial training: Identifying challenges and how to overcome them. In NeurIPS, 2020.
Jiayang Liu, Weiming Zhang, Yiwei Zhang, Dongdong Hou, Yujia Liu, Hongyue Zha, and Nenghai Yu. Detection based defense against adversarial examples from the steganalysis point of view. In CVPR, 2019.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-expprts. In SIGKDD, 20.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi N. R. Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E. Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In ICLR, 2018b.
Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. ICLR, 2018
Mohammad Mehrabi, Adel Javanmard, Ryan A. Rossi, Anup B. Rao, and Tung Mai. Fundamental tradeoffs in distributionally adversarial training. In ICML, 2021.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In ICIR, 2017.
Matthew Mirman, Timon Gehr, and Martin T. Vechev. Differentiable abstract interpretation for provably robust neural networks. In ICML, 2018.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: A simple and accurate method to fool deep neural networks. In CVPR, 2016.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Robust- ness via curvature regularization, and vice versa. In CVPR, 2019.
Konda Reddy Mopuri, Aditya Ganeshan, and R. Venkatesh Babu. Generalizable data-free objective for crafting universal adversarial perturbations. IEEE Trans. Pattern Anal. Mach. Intell., 41(10): 2452-2465, 2019. doi: 10.1109/PAAM..21188266...
Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversarial perturbations in learning from incomplete data. In NeurlPS, 2019.
12

---

Under review as a conference paper at ICLR 2022
2019. Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection adversarial examples. In NeurIPS, 2018.
Preetum Nakkiran. Adversarial robustness may be at odds with simplicity.
Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In ICML, 2019.
Tianyu Pang. Xiao Yang, Yinpeng Dong, Taufik Xu, Jun Zhu, and Hang Su. Boosting adversarial training with hypersphere embedding. In NeurIPS, 2020.
Tianyu Pang, Huishuai Zhang, Di He, Yinpeng Dong, Hang Su, Wei Chen, Jun Zhu, and Tie- Yan Liu. 2021.
Adversarial training with rectified rejection. arXiv preprint
Yao Qin, Nicholas Frosst, Sara Sabour, Colin Raffel, Garrison W. Cottrell, and Geoffrey E. Hinton. Detecting and diagnosing adversarial images with class-conditional capsule reconstructions. In ICLR, 2020.
Ali Rahmati, Seyed-Mohsen Moosavi-Dezfooli, Pascal Frossard, and Huaiyu Dai. Geoda: A geometric framework for black-box adversarial attacks. In CVPR, 2020.
Leslie Rice, Eric Wong, and J Zico Kolter. Overfitting in adversarially robust deep learning, In ICML, 2020.
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: statistical test for detecting adversarial examples. In ICML, 2019.
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. Adversarial training a form of data-dependent operator regularization. In NeurlPS, 2020.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In NeurIPS, 2019.
Fatemeh Sheikholeslami, Ali Lotfi, and J. Zico Kolter. Provably robust classification of adversarial examples with detection. In ICLR, 2021.
Sahil Singla and Soheil Feizi. Second-order provable defenses against adversarial attacks. In ICMI., 2020.
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. In UAI, 2018
Chuanbiao Song, Kun He, Liwei Wang, and John E. Hoperoft. Improving the generalization of adversarial training with domain adaptation. In ICLR, 2019.
Chuanbiao Song. Kun He, Jiadong Lin, Liwei Wang, and John E. Hopcroft. Robust local features for improving the generalization of adversarial training. In ICLR, 2020.
Philip Sperl, Ching Yu Kao, Peng Chen, Xiao Lei, and Konstantin Böttinger. DLA: dense-layer, analysis for adversarial example detection. In IEEE European Symposium on Security and Privacy, Pu 2020.
Gaurang Sriramanan, Sravanti Addepalli, Arya Baburaj. and Venkatesh Babu R. Guided adversarial attack for evaluating and enhancing adversarial defenses. NeurIPS, 2020.
David Stutz, Matthias Hein, and Bernt Schiele. Confidence-calibrated adversarial training: Generaliz- ing to unseen attacks. In ICML, 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICIR, 2014.
Jinyu Tian, Jiantao Zhou, Yuanman Li, and Jia Duan. Detecting adversarial examples from sensitivity inconsistency of -ratial-trasfoomm domain. In AAAI, 2021.
13

---

Under review as a conference paper at ICLR 2022
Shixin Tian, Guolei Yang, and Ying Cai. Detecting adversarial examples through image transforma- tion. In AAAI, 2018.
F Tramèr, D Boneh, A Kurakin, I Goodfellow, N Papernot, and P McDaniel. Ensemble adversarial training: Attacks and defenses. In ICLR, 2018.
Florian Tramèr and Dan Boneh. Adversarial training and robustness for multiple perturbations. In NeurlPS, 2019.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-Margin training: Scalable certification of perturbation invariance for deep neural networks. In NeurlPS, 2018
Haotao Wang, Tianlong Chen, Shupeng Gui, Ting-Kuei Hu, Ji Liu, and Zhangyang Wang. Once- for-all adversarial training: In-situ tradeoff between robustness and accuracy for free. In NewrIPS, 2020a.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML, 2019.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020b.
Tsui-Wei Weng, Huan Zhang, Pin- Yu Chen, Jinfeng Yi, Dong Su, Yupeng Gao, Cho-Jui Hsieh, and Luca Daniel. Evaluating the robustness of neural networks: An extreme value theory approach. In ICLR, 2018.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In ICML, 2018.
Eric Wong, Frank R. Schmidt, and J. Zico Kolter. Wasserstein adversarial examples via projected sinkhorn iterations. In ICML, 2019.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training. In ICLR, 2020.
Dongxian Wu, Yisen Wang, Shu-Tao Xia, James Bailey, and Xingjun Ma. Skip connections matter: On the transferability of adversarial examples generated with resnets. In ICLR, 2020a.
Kaiwen Wu, Allen Houze Wang, and Yaoliang Yu. Stronger and faster wasserstein adversarial attacks. In ICML, 2020b.
Xi Wu, Uyeong Jang, Jiefeng Chen, Lingjiao Chen, and Somesh Jha. Reinforcing adversarial robustness using model confidence induced by adversarial training. In ICML, 2018.
Yuhang Wu, Sunpreet S. Arora, Yanhong Wu, and Hao Yang. Beating attackers at their own games: Adversarial example detection using adversarial gradient directions. In AAAI, 2021.
Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. Spatially transformed adversarial examples. In ICLR, 2018.
Kai Yuanqing Xiao, Vincent Tjeng, Nur Muhammad (Mahi) Shafiullah, and Aleksander Madry. Training for faster adversarial robustness verification via inducing relu stability. In ICLR, 2019.
Cihang Xie and Alan L. Yuille. Intriguing properties of adversarial training scale. In ICLR, 2020.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan Yuille. Improving transferability of adversarial examples with input diversity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2730-2739, 2019.
Cihang Xie. Mingxing Tan, Boqing Gong. Alan Yuille, and Quoc V Le. Smooth adversarial training. arXiv preprint orXiv:2006. 14536, 2020.
Han Xu, Xiaorui Liu, Yaxin Li, Anil K. Jain, and Jiliang Tang. To be robust or to be fair: Towards fairness in adversarial training. In ICML, 2021.
14

---

Under review as a conference paper at ICLR 2022
Hanshu Yan, Jingfeng Zhang, Gang Niu, Jiashi Feng, Vincent Tan, and Masashi Sugiyama. Cifs: Improving adversarial robustness of cnns via channel-wist importance based feature selection. In ICML, 2021a.
Ziang Yan, Yiwen Guo, and Changshui Zhang. Deep defense: Training dans with improved adversarial robustness. In NeurIPS, 2018.
Ziang Yan, Yiwen Guo, Jian Liang, and Changshui Zhang. Policy-driven attack: Learning to query for habe-label black-box adversarial examples. In ICLR, 2021b.
Huanrui Yang. Jingyang Zhang, Hongliang Dong, Nathan Inkawhich, Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. Dverge: Diversifying vulnerabilities for enhanced robust generation of ensembles. In NeurIPS, 2020a.
Puyudi Yang, Jianbo Chen, Cho-Jui Hsieh, Jane-Ling Wang, and Michael Jordan. MIL-LOO: detecting adversarial examples with feature attribution. In AAAI, 2020b.
Dong Yin, Kannan Ramchandran, and Peter L. Bartlett. Rademacher complexity for adversarially robust generalization. In ICMI,, 2019.
Xuwang Yin and Soheil Kolouri Gustavo K. Rohde. GAT: generative adversarial training for adversarial example detection and robust classification. In ICLR, 2020.
Yaodong Yu, Zitong Yang, Edgar Dobriban, Jacob Steinhardt, and Yi Ma. Understanding generaliza- tion adversarial training via the bias-variance decomposition. arXiv preprint a7319:2103.09947, 2021a.
Yunrui Yu, Xitong Gao, and Cheng-Zhong Xu. LAFEAT: piercing through adversarial defenses with latent features. In CVPR, 2021b.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. In NeurIPS, 2019a.
Haichao Zhang and Jianyu Wang. Defense against adversarial attacks using feature scattering- based adversarial training. In NeurlPS, pp. 1829-1839, 2019.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019b.
Huan Zhang, Hongge Chen, Zhao Song, Duane Boning, Inderjit S. Dhillon, and Cho-Jui Hsieh. The limitations of adversarial training and the blind-spot attack. In ICLR, 2019c.
Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsieh. Towards stable and efficient training of verifiably robust neural networks. In ICLR, 2020a.
Jingfeng Zhang, Xilie Xu. Bo Han, Gang Niu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankanhalli. Attacks which do not kill training make adversarial learning stronger. In ICML, 2020b.
Jingfeng Zhang, Jianing Zhu, Gang Niu, Bo Han, Masashi Sugiyama, and Mohan Kankanhalli. Geometry-aware instance-reweighted adversarial training. In ICLR, 2021.
Yi Zhang, Orestis Plevrakis, Simon S. Du, Xingguo Li, Zhao Song, and Sanjeev Arora. Over- parameterized adversarial training: An analysis overcoming the curse of dimensionality. In NewrIPS, 2020c.
Haizhong Zheng, Ziqi Zhang, Juncheng Gu, Honglak Lee, and Atul Prakash. Efficient adversarial training with transferable adversarial examples. In CVPR, 2020.
Tianhang Zheng, Changyou Chen, and Kui Ren. Distributionally adversarial attack. In AAAI, 2019.
Difan Zou, Spencer Frei, and Quanquan Gu. Provable robustness of adversarial training for learning halfspaces with noise. ICML, 2021.
15

---

Under review as a conference paper at ICLR 2022
A THEORETICAL PROOF
The proof of (q.(8) as follows.
Proof. Since we have
exp -l (3%)
M
-l (fo S M- M-max exp
-e (fo,
max exp
==1
(12)
considering that -8lm(x) monotonically decreases w.r.t 2, have
-e (fo,(( ), {100 )) s ( (7t - max exp
- (fo ).9))
-5ln M max exp
a
(13)
- - (F0, (1((o,(("))))) ( )) (00,0, (fo, (z²)),2)
where -d 8l
. max, exp
8 In(M)
-E [e (1%)).)))
and -61mexp
max,
E e (fo, ).)).. so (q) holds.
The proof of Proposition 2 is follows.
Proof. The objective loss depends on the label probability (e.g., cross-entropy loss = py (x)), and the loss monotonically decreases with respect to (2),, the best-performing sub-model has the highest label probability. Combined the fact that PPD(p(z)) monotonically increases w.r.t. Py (z), the best- pers-merforing sub-model corresponds to the highest PPD; therefore, Eq.(1 holds.
B IMPLEMENTATION DETAILS
the design of the PPD head In our experiments, we use ResNet20 as our backbone model. For additional PPD head, we propose to use one-layer MLP model the relationship between the prediction and its confidence. Our design of the PPD head is based on the following aspects: 1). convenient optimization; simple model structure can optimized easily and will not bring significant computation cost 2). robustness; complex PPD head may be vulnerable to the adversarial attack. Therefore, in our implementation, we choose to learn a simple PPD head for each sub-model to improve its robustness.
Training devices
We conduct all experiments on the device GeForce RTX 2080Ti.
C
TRAINING COST ANALYSIS
Training cost is notable issue. We compare the training cost of all methods from the two aspects; 1). parameters and GFLOPs: all methods have the same model architecture (ResNet20), so all methods have similar number of parameters and GFLOps. Compared with baselines, our method has an additional head (PPD head), which one-layer MIP with 128 parameters and has negligible computation cost; 2). training manner, all methods except DVERGE achieve adversarial training by generating adversarial samples using PGD attack. The time consumption of all methods using the device Geforce 2080Ti (100 epochs) is in Table 4.
Table 4: time consumption of all methods (100 epoches).
methods time
GAL h 36min
DEVRGE 11 h 40 min
ADP h 35 min 7 23 min
CDA²
DVERGE distills non-robust features by computing transferable adversarial samples, which have O(N?) time complexity in which N is the number of sub-models, so it has relatively large time consumption. Our method outperforms baselines by training an additional PPD head and could cause additional small time consumption as shown in the above table.
16

---

Under review as a conference paper at ICLR 2022
D MORE EXPERIMENTAL RESULTS
For the transfer attack, also more challenging setting following the work in (Yang et al., 2020a). For the attack methods, we use PGD with momentum (Dong et al., 2018) with three random starts, M-FGSM (Xie al., 2019) and SGM (Wu et al., 2020a). We hold-out baseline models with 3, 5, and 8 ResNet-20 sub-models as the surrogate models. Meanwhile, we generate adversarial samples with cross-entropy loss and CW loss (Carlini & Wagner, 2017b). For each sample, we generate 30 adversarial variants, and only if the model classifies all the 30 variants can the model defend the transfer attack successfully. The results are shown the following Table. From Table CDA² outperforms baselines as 0.01 C.
Table 5: Robustness results (%) under transfer attack with 30 adversarial variants.
0.01
0.02
0.03
0.04
0.05
0.06
0.07
methods
GAL
57.8
64.1
46.3
56.0
43.9
44.5
41.4
ADP
84.2
80.1
73.9
69.6
65.3
56.0
60.7
DVERGE
81.5
78.1
73.5
68.4
67.2
63.8
57.1
CDA
83.2
80.4
75.0
71.0
69.1
62.8
61.4
E MORE DISCUSSION ABOUT THE COLLABORATION
E. MODEL CAPACITY AND COLLABORATION
model with sufficient capacity to cover all cases does not need to collaborate with others. To verify this claim, conduct experiments using the ResNet model with different depths and show the clean accuracy (%) of single/multiple models in the following table.
Table 6: the accuracy using different model structures
depth
2
14
20
single model
65.0
88.3
90.5
91.9
collaboration
67.0
89.5
91.6
92.5
gain
2.0
1.2
0.9
0.6
From the Table 6, with the depth 2, the model has the insufficient model capacity to learn the feature extractor, collaboration have relatively large improvement (2.0). As the depth of the model is 20, the model has sufficient model capacity to fit all data samples. Collaboration achieves slight improvement compared with a single model (0.6).
Compared with standard training, adversarial data are adaptively changed based on the current model smooth the natural data's local neighborhoods. The volume these surroundings exponentially large. The model often encounters insufficient model capacity especially when there is relatively large ball. Therefore, urgent to improve the utilization of the capacity for adversarial training.
E.2 COLLABORATION WITH SINGLE BIG MODEI.
advantages of the collaboration Compared with collaboration, single big model may be difficult to fit the adversarial data. For a big single model with deeper structure, it may face gradient vanish gradient explosion during optimization. Without well-designed optimization method, it could be more vulnerable to an imperceptible perturbation compared to a simple model. The collaboration alleviates this problem by learning multiple relatively small models.
vintavan of the collaboration A single big model may be a direct solution for address- ing insufficient capacity in adversarial training. Compared with learning single big model, the collaboration needs to design a complicated mechanism to improve the utilization of the model capacity.
17

---

Under review as a conference paper at ICLR 2022
E.3 COLLABORATION IN ADVERSARIAL TRAINING CAN AVOID TRIVIAL OPTIMIZATION
As our proposed collaboration mechanism enhances the performance of the best-performing sub- model. One may wonder whether obtains trivial case, e.g., only one sub-model is properly trained. In fact, our collaboration will not bring such a trivial case. Fitting the adversarial data consumes tremendous model capacity so the model usually cannot fit all adversarial samples. For a learned sub-model, there still exist adversarial samples that cannot handle. From our proposed collaboration mechanism, these adversarial samples generated from learned sub-model are more likely to be assigned to other sub-models which perform better. Therefore, in our collaboration, all sub- mobemodels will be trained.
Table 7: the accuracy (%) on the clean data
CDA 85.6
sub-model 83.9
sub-model B
sub-model
84.2
83.9
To experimentally verify this claim, we present the accuracies on clean data of all three sub-models with adversarial training (e = 0.02) in the Table 7. From the Table 7, all sub-models have a similar performance on the clean data.
18


Reviews:
[{"review_id": "y3zxosTKjxD", "reviewer": "Reviewer_g3Dc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of adversarial training on each other's adversarial examples is new. However, it is flawed, at least in its current form.\nThe issue is that such adversarial training may not provide enough coverage and, after the training converges, there may exist adversarial examples that can attack all sub-models.\n\nConsider a hypothetical situation with two sub-models. Both sub-models classify clean images well. However, sub-model A is easily attacked by adding a faint cross pattern near the upper left corner, and sub-model B is easily attacked by adding a faint cross pattern near the lower right corner.\n\nDuring the proposed training procedure, we'll only encounter these two types of adversarial examples because they are respectively the best attack with lowest Lp epsilon on the two sub-models. Then the two sub-models learn to solve each other's adversarial examples: sub-model A will become robust against faint cross patterns near the lower right corner and sub-model B will become robust against faint cross patterns near the upper left corner. The training procedure therefore converges quickly.\n\nHowever, after the adversarial training has converged, a vast space of adversarial examples has not been explored at all. There may exist adversarial examples that, although their epsilon is slightly higher than the faint cross patterns, they can attack both models.\nThe above discussion can be easily extended to the general case of M sub-models. The point is that the proposed adv training procedure can be self-limiting and may not provide enough coverage. \n\n \nThe experimental results are not sufficient to support the claims.\n* Standard accuracies on clean images are not reported in Table 1. They are critical missing information.\n* The CIFAR-10 model in (Madry et al. 2018) has robust accuracy of 45.8% against L_inf epsilon of 8/255=0.031 with PGD-20. Table 1 shows that the proposed method has robust accuracy of 44.5% against L_inf epsilon of 0.03 with PGD-10. It's unclear that the proposed method has an advantage.\n* This paper seems to confuse black-box attacks and transfer attacks. Tables 2 and 3 are transfer attacks. There are no black-box results in this paper.\n\nThe role of the PPD head is not explained well.\nAccording to Figure 3(b) and Algorithm 1 line 6, the PPD head is trained by minimizing binary cross entropy between it and the true-label logit from the normal output.\nIt seems that because the PPD head is trained to also track wrong predictions of a sub-model, it can serve as a confidence score at inference time.\nEquations (11)(12)(13) and the surrounding text do not help and they seem irrelevant to the actual implementation.\n\nThe philosophical idea claimed in the conclusion section is not new. What's new here is the adversarial training scheme where sub-models train on each other's adversarial examples.\n", "summary_of_the_paper": "This paper proposes an ensemble or mixture-of-experts method to defend against adversarial examples. Though the authors prefer to use the term collaboration method to highlight its difference from vanilla ensemble.\nThe main idea is that, during adversarial training, the sub-models are trained on each other's adversarial examples.\n\nSpecifically:\n* for each training image, one adversarial example is generated per sub-model by carrying out an attack on each sub-model.\n* each adversarial example is (softly) assigned to the sub-model that has the lowest loss on it as a training image.\n* each sub-model has a second output called PPD that quantifies its confidence.\n* at inference time, for each input, the sub-model with the highest PPD produces the output.\n\nThe rationale is that, because each sub-model only needs to cover part of the adversarial example space, they can do a better job.\nExperiments on CIFAR-10 with L_inf attacks are reported.", "summary_of_the_review": "1) Unfortunately the main idea is flawed and the proposed adversarial training may converge without providing enough coverage.\n2) The experimental results do not show advantage over previous non-ensemble method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "a1MfGozjOk6", "reviewer": "Reviewer_qeVf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The proposed method is clearly motivated and defined. And it has been demonstrated effective by quantitative experimental results.\n2. A comprehensive overview of related works is provided.\n\nWeaknesses\n1. Training cost is also a notable issue for adversarial training methods. It would be better if the training cost of the proposed method is compared with that of the previous methods.", "summary_of_the_paper": "This paper presents a new paradigm for defending against adversarial attacks with multiple sub-models.\nDifferent from ensemble, the proposed collaboration paradigm, a representative sub-model is chosen to make the decision, instead of letting all sub-models vote.\nThe proposed method has been validated on CIFAR-10 dataset, against both white-box and transferrability-based black-box attacks.", "summary_of_the_review": "This paper presents a new paradigm involving multiple sub-models which has several advantages compared to ensemble.\nIt has been validated effective by the quantiative results in terms of robustness, but comparison in other aspects such as training cost could be also helpful.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QkS4VGkoGYh", "reviewer": "Reviewer_Koh8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strength:\n\n1. This work considers the adversarial defense problem. It focuses on the insufficient model capacity of adversarial training and presents a completely fresh perspective on learning multiple models to improve the robustness. The proposed framework is technically solid. The idea of collaboration to minimize the vulnerable area is innovative compared with baselines. In summary, I assess a high novelty of this work;\n\n2. The proposed framework CDA^{2} makes sense for the goal of minimizing the vulnerable area of the overlap. A dual-head model is used for predicting and providing information about how to assign the adversarial sample. Optimizing the best-performing sub-model is to minimize the vulnerability overlap of all sub-models. The framework has contributed to the research problem and may enlighten other research problems. It could be a good paper for the ICLR community.\n\n3. This work is well-written. Figs. 1 and 2 show the motivation of the proposed method clearly, which helps understand it. \n\n# Weakness:\n\n1. the author states that the insufficient model capacity can hurt its performance in adversarial training. What if the model has sufficient model capacity? Are there other scenarios that have insufficient model capacity? Whether CDA^{2} can be useful for such scenarios if any? Compared to a big single model, can the author discuss more pros and cons of collaboration?\n\n2. To achieve collaboration, the author proposes to assign the samples to the sub-models that perform best. Does it obtain a trivial case? For example, only one sub-model has been trained. Can the author discuss more? \n\n3. For the framework CDA^{2}, ppd head is for evaluating the performance of the other head. Can the author discuss the impact on the quality of this head more?\n\n4. The experiment on the XOR problem is a little bit weak. Can the author provide more experiments to verify the effectiveness of CDA^{2}?\n\n5. For the experimental results on the white-box, the author gives the results in Table 1. The experimental setting is a little vague. I’m wondering about the robustness performance of PGD-50 as in [1];\n\n6. For the black-box experiments, the author uses M-FGSM and PGD to generate transferable adversarial samples. Can the author provide more experimental results to validate its claims?\n\nOverall, there are still some issues stated above. I will increase my score if they are well addressed.\n\n[1]. Dverge: Diversifying vulnerabilities for enhanced robust generation of ensembles. In NeurIPS, 2020a.\n\n========\n\n# Post rebuttal responses:\n\nThanks for the authors' responses. They address my concerns on the concept of collaboration, and the newly added experiments are convincing. Especially, I like the idea of collaboration in adversarial training, which is a new paradigm to defend against adversarial attacks. \n\nBesides robustness, the new paradigm may be helpful to other domains.\n\nAfter reading the review from other reviewers and the corresponding responses, I vote for acceptance and increase my scores further.", "summary_of_the_paper": "This paper firstly analyzes prior adversarial defense methods using ensemble strategy and claims that this method could cause a waste of model capacity. To improve the utilization of (multiple ) model capacity, the author proposes an interesting collaboration strategy $CDA^2$ to defend against adversarial attacks. Specifically, the author develop a dual-head model structure: one is for making a prediction and the other is for predicting the posterior probability of the input. During training, each model can address the adversarial attacks of other sub-models so that it improves the robustness of the collaboration. The experimental results partially verify the superiority of the proposed methods.", "summary_of_the_review": "A novel and interesting collaboration method for advancing the robustness of multiple sub-models.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel ensemble method, CDA^2,  in which base models collaborate to defend against adversarial attacks. To do so the base models have two heads: the label head for predicting the label and the posterior probability density (PPD) head that is trained by minimizing binary cross entropy between it and the true-label logit given by the label head. During inference the base model with the highest PPD value is chosen to make the prediction. During training base models learn from the adversarial examples produced by other base models. 

The evaluation of the manuscript of different reviewers was very diverse, resulting in final scores ranging between 3 and 8 after the discussion period. While the rebuttal clearly addressed the concerns of one reviewer and several additional experimental results were added for different adversarial attacks, it did not fully addressed the concerns of another reviewer, who rated his confidence higher. He was also not convinced by the update in the revised version of the manuscript, in which crucial changes in the pseudocode describing the proposed algorithm were made, which contradicted some statements in the first version. Therefore, the paper can unfortunately not be accepted in its current version. In a future version of the manuscript, the description of the algorithm and of he role of the PPD head should be improved and experiments on another dataset next to CIFAR-10 could be added.
Decision: Reject