Understanding and Improving Ensemble Adversarial Defense
Authors: Yian Deng
Tingting Mu
Venue: NeurIPS 2023 poster (2023)
Keywords: adversarial defense
ensemble diversity
robustness
curvature

Abstract: The strategy of ensemble has become popular in adversarial defense, which trains multiple base classifiers to defend against adversarial attacks in a cooperative manner. Despite the empirical success, theoretical explanations on why an ensemble of adversarially trained classifiers is more robust than single ones remain unclear. To fill in this gap, we develop a new error theory dedicated to understanding ensemble adversarial defense,  demonstrating a provable 0-1 loss reduction on challenging sample sets in adversarial defense scenarios. Guided by this theory, we propose an effective approach to improve ensemble adversarial defense, named interactive global adversarial training (iGAT). The proposal includes (1) a probabilistic distributing rule that selectively allocates to different base classifiers adversarial examples that are globally challenging to the ensemble, and (2) a regularization term to rescue the severest weaknesses of the base classifiers. Being tested over various existing ensemble adversarial defense techniques,  iGAT is capable of boosting their performance by up to 17\%  evaluated using  CIFAR10 and CIFAR100 datasets under both white-box and black-box attacks.

Understanding and Improving Ensemble Adversarial Defense
Yian Deng
Tingting Mu
Department of Computer Science
Department of Computer Science
The University of Manchester Manchester, UK, M13 9PI.
The University of Manchester Manchester, UK, M13 9PI. tingt.iigg.memmnccheess.
yian. yian..deng@manchestrr.coo uk
Abstract
The strategy of ensemble has become popular in adversarial defense, which trains multiple base classifiers to defend against adversarial attacks in a cooperative man- ner. Despite the empirical success, theoretical explanations on why an ensemble of adversarially trained classifiers is more robust than single ones remain unclear. To fill this gap, we develop a new error theory dedicated to understanding ensemble adversarial defense, demonstrating provable 0-1 loss reduction on challenging sample sets in adversarial defense scenarios. Guided by this theory, we propose effective approach to improve ensemble adversarial defense, named interac- tive global adversarial training (iGAT). The proposal includes (1) a probabilistic distributing rule that selectively allocates to different base classifiers adversarial examples that are globally challenging to the ensemble, and (2) a regularization term to rescue the severest weaknesses of the base classifiers. Being tested over var- ious existing ensemble adversarial defense techniques, iGAT is capable of boosting their performance by up 17% evaluated using CIFAR10 10 and CIFAR 100 datasets under both white-box and black-box attacks.
Introduction
Many contemporary machine learning models, particularly the end-to-end ones based deep neural networks, admit vulnerabilities to small perturbations in the input feature space. For instance, in computer vision applications, minor change of image pixels computed by algorithm can manipulate the classification results to produce undesired predictions, but these pixel changes can be imperceptible to human eyes. Such malicious perturbations referred to adversarial attacks. These can result in severe incidents, e.g., medical misdiagnose caused by unauthorized perturbations in medical imaging (43), and wrong actions taken by autonomous vehicles caused crafted traffic images 34
The capability of machine learning model to defend adversarial attacks is referred to adversarial robustness. A formal way to quantify such robustness is through an adversarial risk, which can be intuitively understood the expectation of the worst-scenario loss computed within a local neighborhood region around a naturally sampled data example [40]. It is formulated as below:
Rade (f) (x max (ū (f(z)) BEB(x)
(I)
where B(x) denotes a local neighborhood region around the example x sampled from a natural data distribution D. The neighbourhood usually defined based on selected norm, e.g., region containing any satisfying Iz x|| < e for a constant >> O. In general, {(), y) can be any loss function quantifying the difference between the predicted output ğ and the ground-truth output y. A
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

learning process that minimizes the adversarial risk is referred to as adversarially robust learning. In the classification context, a way to empirically approximate the adversarial robustness is through a classification error computed using set of adversarial examples generated by applying adversarial attacks to the model [4]. Here, an adversarial attack refers to an algorithm that usually builds on an optimization strategy, and it produces examples perturbed by certain strength so that a machine learning model returns the most erroneous output for these examples [10 24].
On the defense side, there is a rich amount of techniques proposed to improve the model robust- ness [10 I. However, robust neural network enhanced by a defense technique would mostly result in reduced accuracy for classifying the natural examples [52]. A relation between the adversarial robustness and the standard accuracy has been formally proved by Tsipras al. [38]: "Any classifier that attains at least 1 standard accuracy on dataset DD has robust accuracy at most óp/ (1 -p)) against an -osundddd adversary with & 2 2n,,", where e is the Toe bound indicating the attack strength, while 1) is sufficiently large and 2 0.5 and they both are used for data generation. This statement presents a tradeoff between the adversarial robustness and natural accuracy. There has been consistent effort invested to mitigate such tradeoff by developing defense techniques to improve robust accuracy without sacrificing much the natural accuracy.
Ensemble defense has recently arisen as a new state of the art 25 The core idea is to train strategically multiple base classifiers to defend the attack, with the underlying motivation of improving the statistical stability and cooperation between base models. Existing effort on ensemble defense has mostly been focused on demonstrating performance success for different algorithmic approaches. It is assumed that training and combining multiple base models can defend better adversarial attacks as compared to training single model. Although being supported by empirical success, there is few research that provides rigorous understanding to why this is the case in general. Existing results on analyzing generalized error for ensemble estimators mostly compare the error of the ensemble and the averaged error of the base models, through, for instance, decomposition strategies that divide the error term into bias, variance, co-variance, noise and/or diversity terms 45] 7 39]. J. is not straightforward to extend such results to compare the error of an ensemble model and the error of a model without an ensemble structure.
To address this gap, we develop a new error theory (Theorem dedicated to understanding ensemble adversarial defense. The main challenge in mitigating the tradeoff between the adversarial robustness and natural accuracy comes from the fact that adversarial defence techniques can reduce the classifier's capacity of handling weakly separable example pairs that are close to each other but from different classes. To analyse how ensemble helps address this particular challenge, we derive provable error reduction by changing from using one neural network to an ensemble of two neural networks through either the average or max combiner of their prediction outputs.
Although ensemble defense can improve the overall adversarial robustness, its base models can still be fooled individually in some input subspaces, due to the nature of the collaborative design. Another contribution we make is the proposal of a simple but effective way to improve each base model by considering the adversarial examples generated by their ensemble accompanied by a regularization term that is designed to recuse the worst base model. We experiment with the proposed enhancement by applying it to improve four state-of-fe -ftt---rr- ensemble adversarial defense techniques. Satisfactory performance improvement has been observed when being evaluated using CIFAR-10 10 and CIFAR-100 datasets [5I] under both white-box and black-box attacks.
2 Related Work
Adversarial Attack: Typical attack techniques include white-box and black-box attacks. The white-box adversary has access to the model, c.g., the model parameters, gradients and formulation, etc, while the black-box adversary has limited knowledge of the model, e.g., knowing only the model output, and therefore is closer to the real-world attack setting [5]. Representative white-box attacks include Deepfool (27] and the projected gradient descent (PGD) (26. The fast gradient sign method (FGSM) [18] is simplified one-iteration version of PGD. The momentum iterative method (MIM) [15] improves the FGSM attack by introducing the momentum term. Another two commonly-
used and cutting-edge white-box attacks are the Carlini- Wagner (CW) attack [ that computes the perturbation by optimizing the perturbation scales, and the Jacobian-basee saliency map attack (JSMA) [31] that perturbs only one pixel in each iteration. Representative black-box attacks include

---

the square attack (SA) [1, the SignHunter attack [2] and the simple black-box attack (SimBA) [9) among which SA can generate stronger attacks but is more computationally expensive. A thorough survey on adversarial attacks is provided by Akhtar et al. []]. Atto [12] encapsulates a selected set of strong white and black-box attacks and is considered as the state-of-the-an attack tool. In general, when evaluating the adversarial robustness of a deep learning model, it should be tested
under both the white and black-box attacks.
Adversarial Defense: Adversarial training is the most straightforward and commonly used defense technique for improving adversarial robustness. works by simply expanding the training data with additional adversarial examples. The initial idea was firstly adopted by Christian et al. [ to train robust neural networks for classification, later on, Madry al. [26] used adversarial examples generated by the PGD attack for improvement. Overall, this strategy is very adaptive and can be used to defend any attack, but it is on the expense of consuming more training examples. has been empirically observed that adversarial training can reduce loss curvatures in both the input and model parameter spaces 17 [6], and this reduces the adversarial robustness gap between the training and testing data 152 7 These findings motivate the development of a series of regularization techniques for adversarial defense, by explicitly regularizing the curvature or other relevant geometric character- istics of the loss function. Many of these techniques can avoid augmenting the training set and are computationally cheaper. Typical regularization techniques that attempt to flatten the loss surface in the input space include the curvature regularization (CURE) [28]. local linearity regularization (LLR) [32], input gradient regularization [29] and Lipschitz regularization [4]. There are also tech- niques to flatten the loss in the model parameter space, such TRADES [52]. misclassification aware adversarial training (MART) [44), robust self-training (RST) 9 3 adversarial weight perturbation (AWP) 47, and HAT
446 etc. Alternative adversarial training and regularization, other defense
strategies include pruning [35, pre-training [22] feature denoising
[48] domain adaptation [6) and
ensemble defense [25] 37 etc. Croce et al. 13 has provided a summary of recent advances.
Ensemble Adversarial Defense: Recently, ensemble has been actively used in adversarial defense, showing promising results. One core spirit behind ensemble is to encourage diversity between base models in order to achieve an improved ensemble prediction [45] 7 33]. Therefore, the advances are mostly focused on designing effective ensemble diversity losses in training improve adversarial robustness. For instance, the adaptive diversity promoting (ADP) method 300 uses Shannon entropy for uncertainty regularization and geometric diversity for measuring the difference between the predictions made by base classifiers. The transferability reduced smooth (TRS) method (50) for- mulates the diversity term based cosine similarities between the loss gradients of base models, meanwhile increases the model smoothness via an /z-regularization of these gradients during training. The similar idea of exploiting loss gradients was proposed earlier in the gradient alignment loss (GAL) method 23 The conditional label dependency learning (CLDL) method [42] is a latest improvement over GAL and TRS, which measures the diversity using both the predictions and loss gradients of the base models. However, the ensemble nature of encouraging diversity can cause vulnerability for some base models over certain subsets of adversarial examples (28). In practice, this can limit the overall robustness of the ensemble when the other base models are not strong enough to correct the weak ones. To address this, the diversifying vulnerabilities for enhanced robust generation of ensembles (DVERGE) [9 proposes vulnerability diversity to encourage each base model to be robust particularly to the other base models' weaknesses. The latest development for improving ensemble defense, known synergy-of offereess (SoE) [14, follows a different research path. For each input, it adaptively selects a base model with the largest confidence to make the final prediction instead of combining all, for which the supporting algorithm and theory have been developed. Some surveys on ensemble adversarial attacks and defense can be found in He et al. (21) Lu et al. (25]
3 Notations and Preliminaries
Bold capital and lower-case letters, e.g., X and x, denote matrices and vectors, respectively, while lower-case letters, e.g., 2, denote scalars. The i-th row and column of matrix X are denoted by x; and (0, respectively, while æijj and z; the elements of X and A classification dataset D {(x;, y:) >>> includes n examples, which are referred to as natural examples, with x; E X C Bd (feature vector) and yi E [C] = 21,2,... , C} (class label). We sometimes express the label of an example as y(x) or yx- Storing x; as a row of X E Raad and y an element of y E R", we also denote this dataset by D = yX The classifier f X + [0, S outputs class probabilities
3

---

usually computed by softmax function. Given the computed probability fo for the e-th class, Dr (r(x) = arg [C] fx(x) predicts the class. For neural network, we denote by w the weight matrix connecting the I-th and the (1 - 1) layers and by w/ its ij-th element. The L2 norm || ||2 is used to compute the vector length, while the -nom || || to generate adversarial attacks.
Concatenation of two sets is denoted by the symbol U.
We focus on classification by minimizing a classification loss {f(x), 1/x) and adapt it to ((((X), y) for the whole dataset. Also, we use fo to emphasize the cross-entropy loss. The loss gradient is (x), 3xx)
A cheap way to estimate the loss curvature is by finite difference approximation 28 e.g., the following curvature measure based on L2-norm:
Dx
[ (x - 8x) (x))(
8x
(2)
|
where 8 E d is a perturbation. It measures how a surface bends at a point by different amounts in different directions. An adversarial example x Φff, x, A) is generated by attacking the classifier f using attack algorithm A on a natural example It is further adapted to X of X, A) for the set of adversarial examples each generated from a natural example in X. The quantity 5(f, x; A) off,,x, x, A) x is referred to the adversarial perturbation of x, simplified to 8x x To control the perturbation strength, we restrict ||xx S E for some e > O, which results in the
following adversarial example formulation, as
de (f, x, A) (einnmaxx6(ff)) x, A), x E), +)),
where both min(.-) and max(:-)) are element-wise operators comparing their inputs.
4 An Error Theory for Adversarial Ensemble Defense
In adversarial ensemble defense, a widely accepted research hypothesis is that training and combining multiple base classifiers improve adversarial defense as compared to training a single classifier. However, this hypothesis is mostly supported by empirical successes and there is a lack of formal theoretical justification. In this work, we seek theoretical evidence, proving that, when using multi- layer perceptrons (MLPs) for classification, classification error reduces when applying adversarial defence the base MLPs of ensemble as compared to single MLP, under assumptions feasible in practice. The following theorem formalizes our main result.
Theorem 4.1. Suppose h, h² h² H X + [0, 1Jc are C-class L-layer MLPs satisfying Assump- Given a dataset D = {(xx,)))) vt) construct an ambiguous pair set A(D) by Definition
tion 4.2
4.3
Assume h, ho h are acceptable classifiers for A(D) by Assumption 4.4 Given a classifier
H X CRC and dataset D, assess its classification error by 0-1 loss, as
Roy (D.
Σ XED
for (x)
max fc(x)
D
c/
(0.1) of two base MLPs h2 and h- through where 1 [irue] = 1 while [false] = O. For an ensemble h
either an average or a max combiner, i.e., (0.1)) = (hh + hh) or (0.1) max(h" h²), it has lower empirical 0-1 loss than a single MLP for classifying ambiguous examples, such as
Eno, [Ro/ (a. ]] ELEN [Ro/1 ha.
(5)
We prove the result for MLPs satisfying the following assumption. Assumption 4.2 (MIP Requirement). Suppose a C-class L-layer MLP h Bd + [0, 11c expressed iteratively by
a (x) = =x, a (x) (w()g (x)) ,1 1,,, 1,
(6)
(7)
a (x))) (()) (4 (1-1) (x)) =((x),
(8)
h(x) softmax softmax(z(x)),
(9)

---

where a-)) is the activation function applied element-wise the representation vector z(x) E R returned by the L-th layer is fed into the prediction layer building upon the softmax function. Let (1)
W8 denote the network weight connecting the st-th neuron in the (-th layer and the 51+1-th neuron in the (1 + 1-tth layer for / E (1,2..., L.). Define a column vector ((A) with its i-th element computed from the neural network weights and activation derivatives, as p: Es.
Bag, (xx) (L)
for k = 1,,,... d and = 1, C, also matrix Pa = Ei= p ((k) and its factorization Ph = M,M, with full-rank factor matrix Mn. For constants X, B > O, suppose the following holds for h:
1. Its cross-entropy loss curvature measured by Eq.
satisfies A 8) S X.
2. The factor matrix satisfies ||Mall2 S Bo and M S B, where || ||2 denotes the vector
induced 12-norm for matrix.
We explain the feasibility of the above MI.P assumptions in the end of this section.
Although adversarial defense techniques can improve adversarial robustness, new challenges arise in classifying examples that are close each other but from different classes, due to the flattened loss curvature for reducing the adversarial risk. We refer to pair of such challenging examples as an ambiguous pair. Our strategy of proving improved performance for adversarial defense to (1) firstly construct a challenging dataset A(D) comprising samples from these pairs, which is referred to as an ambiguous pair set, and then (2) prove error reduction over A(D). To start, we provide formal definitions for the ambiguous pair and set.
Definition 4.3 (Ambiguous Pair). Given a dataset D {(x,, Vi) where x; E X and ys E [C], an ambiguous pair contains two examples ((x,, 31), (x,, U3)) satisfying y1 # y and
||x; - x,112 S
(10)
JB / (x2 (5--5)
where J > 2 is an adjustable control variable, X, B and & S X²2 are constants associated with the MLP under Assumption 4.2 The ambiguous pair set A(D) contains all the ambiguous pairs existing in D, for which J is adjusted such that A(D) + 0.
we are only interested in classifiers that do not fail too badly on A(D), e.g., having an accuracy level above 42.5%. Comparing poorly performed classifiers is not very meaningful, also the studied situation is closer to practical setups where the starting classifiers for improvement are somewhat acceptable. Such preference formalized by the following assumption:
In Theorem 4.1,
Assumption 4.4 (Acceptable Classifier). Suppose acceptable classifier f Rd - [0. 1] does not perform poorly on the ambiguous pair set A(D) associated with a control variable This means that, for any pair = ((x, W.), (xx; v2)) E A(D) and for any example (x,,). from the pair, the following holds:
1. With a probability p 2 42.5%, the classifier correctly classifies (x; y.) by a sufficiently large predicted score, i.e., fy. (x:) 2 0.5 J. while wrongly classifies the other example Xj to yi by a less score, i.e., fy. (x,) S 0.5 + }.
2. When the classifier predicts (x,, v1) to class D.. the predicted scores for the other classes excluding y; are sufficiently small, i.e., fe(x,) 1-f9, (x:) for + yi. St.
C-1
Proof for Theorem 4. together with a toy illustration example is provided in supplementary material.
Assumption Discussion. Assumption 4.2 is feasible in practice. Reduced loss curvature is natural result from adversarial defense, particularly for adversarial training and regularization based methods [16 [7] as mentioned in Section 2 Regarding its second part determined by neural network weights and activation derivatives, common training practices like weight regularization and normalization help prevent from obtaining overly inflated elements in Pn. and thus bound ||Mull2
and [M]]] Following Definition 4.3 the ambiguous pair a ((x,, y1), (xx, My)) is constructed to
5

---

let the classifier struggle with classifying the neighbouring example, e.g. (xj. Vj), when it is able to classify successfully, e.g.. (x). Consequently, the success of classifying (x, yi) mostly accompanied with failure of classifying (xj. M2) into yi, and vice versa. In Assumption 4.4 for an acceptable classifier, the first part assumes its failure is fairly mild, while the second part assumes its struggle is between Y/4 and Vj. As shown in our proof of Theorem 4.1 in order for Eq. (5) to hold, a polynomial inequality of the probability p needs to be solved, providing a sufficient condition on achieving reduced ensemble risk, i.e., p 42.5%. Later, we conduct experiments to examine how well some assumptions are met by adversarially trained classifiers and report the results in supplementary material.
5 iGAT: Improving Ensemble Mechanism
Existing ensemble adversarial defense techniques mostly base their design on a framework of combining classification loss and diversity for training. The output of each base classifier contains the probabilities an example belonging to the C' classes. For an input example x E X, we denote its output from the i-th base classifier by h'(x)= the number of used base classifiers. Typical practice for combining base predictions includes the averaging, i.e., hxx) = N Σ (xx), or the max operation, i.c., (x) = maX (h3 (x)). Without loss of generality, denote the combiner by h (h' :-.. h To train the base classifiers, exemplify ensemble loss function using one training example (x, yx), as below
hi (x).... (x)] for i [N], where N denotes
N
LE(x, Uxx Σ classilication loss
E(h' (x), Ux) +wReg (h(x)) + (iversity(h h²))......................)) ............... (x). (11)
where w, 7 2 0 are hyperparameters. An example choice for regularization is the Shannon entropy of the ensemble h(x) (30) Significant research effort has been invested to diversity design, for which it is optional whether to use the class information in diversity calculation. In the first section of supplementary material, we briefly explain four ensemble adversarial defense techniques highlighting
their loss design strategies. These include ADP 30). CLDL 42), DVERGE 49 and SoE [1], and they are used later in Section to to test our proposed enhancing approach.
Despite the effort in diversity design that encourages better collaboration between hase classifiers, it is unavoidable for some base classifiers struggle with classifying examples from certain input subspaces. There are intersected subspaces that all the base classifiers are not good at classifying. To address this, we propose an interactive global adversarial training (iGAT) approach. seeks support from adversarial examples globally generated by the ensemble and distributes these examples to base classifiers with a probabilistic strategy empirically proven effective. Additionally, it introduces another regularization term to improve over the severest weakness of the base classifiers. Below we describe our proposal in detail.
5.1 Distributing Global Adversarial Examples
We aim at improving adversarial robustness over intersected feature subspaces which are hard for all base classifiers classify. These regions be approximated by global adversarial examples generated by applying adversarial attacks to the ensemble, which are
(x = (bb (h',...,.'' .h) X, A).))
(12)
where rows of X store the feature vectors of the generated adversarial examples. For instance, the FGSM attack can be used A. Instead of feeding the same full set of adversarial examples to train each base classifier, we distribute different examples to different base classifiers, to improve performance and reduce training time. The generated examples are divided into N groups according
to their predicted class probabilities. The i-th group [ 5' is used to train the i-th base classifier, contributing to its classification loss.
Our core distributing strategy is to encourage each base classifier to keep improving over regions that they are relatively good at classifying. This design is motivated by our theoretical result. We have proved in Theory 4.1 error reduction achieved by the ensemble of base MLPs that satisfy

---

the acceptability Assumption 4.4 returns sufficiently high prediction score for the correct class low scores for most the incorrect classes for some challenging examples. By keeping assigning each base classifier new challenging examples that they are relatively good at classifying. encourages Assumption to continue hold. In Section 6.3] we perform ablation studies to compare our proposal with few other distributing strategies, and the empirical results also verify our design. Driven by this strategy, we propose one hard and one soft distributing rule.
This assumption is partially examined by whether the classifier
Hard Distributing Rule: Given generated adversarial example (x, y) = ((x), be V1)). the following rule determines which base classifier to assign it:
If (x)
>
max 2 (x). assign (x, to j+i
X (*)
(13)
We refer to it as hard distributing rule simply assigns examples in deterministic way. The example is assigned to the base classifier that returns the highest predicted probability on its ground truth class.
Soft Distributing Rule: A hard assignment like the above can be sensitive to errors. Alternatively, propose soft distributing rule that utilizes the ranking of the base classifiers based their prediction performance meanwhile introduces uncertainty. It builds upon roulette wheel selection [6], which is commonly used genetic operator in genetic algorithms for selecting promising candidate solutions. Firstly, we rank in descending order the predicted probabilities (x) by all the base
classifiers for the ground truth class, and let (h') E [N] denote the obtained ranking for the i-th base classifier. Then, we formulate a ranking-basse based score for each base classifier as
2 2
P:
EiElNy 2:-1
(14)
and it satisfies ΣiE[N] P; = 1. A more top ranked base classifier has higher score. Next, according to {px) DN apply roulette wheel selection and distribute the example the selected base classifier. Specifically, the selection algorithm constructs N intervals {[a;. b:)] where 0i = 0. b Pi-
also a, b and b2 = a, + Pt for 2.3,.... N. After sampling number E (0, 1] following
uniform distribution example is used to train the i-th base classifier. This enables to assign examples based on ranking but
U (0, 1). check which interval q belongs to. If q S 6;. then the
in probabilistic manner in order be more robust to errors.
5.2 Regularization Against Misclassification
We introduce another regularization term to address the severest weakness, by minimizing the probability score of the most incorrectly predicted class by the most erroneous base classifier. Given an input example (x, yx), the proposed term is formulated
LR (x, Mx) = -ō0/1 (c (h (x), h (x)) Uxx log (1- C N
mammax max max h (x)
(15)
i=1 j=1
Here, (f, E 10 is an error function, where if the input classifier can predict the correct label y, it returns O, otherwise 1. This design is also motivated by Assumption 4.4 to encourage weak base classifier to perform less poorly on challenging examples so that its chance of satisfying the acceptability assumption can be increased.
5.3 Enhanced Training and Implementation
The proposed enhancement approach iGAT, supported by (1) the global adversarial examples gener- ated and distributed following Section|5.1 and (2) the regularization term proposed in Section 5.. can be applied to any given ensemble adversarial defense method. We use Lg to denote the original
7

---

ensemble loss as in Eq. 11 the enhanced loss for training the base classifiers become
N
min (h J
E (xxx))(XX)) (x yx))
+ a
E (xxx=) (Cc 8x)
(16)
original ensemble loss
added global adversarial loss
BE (x. (xxz)) (())))))((X)(()(.)))))) [L (x, [x)],
added misclassilication regularization
where a, 2 hyper--aramettes. In practice, the base classifiers are firstly trained using an existing ensemble adversarial defense technique of interest, i.e., setting a = 3 = 0. If some pre-trained base classifiers are available, they can be directly used instead, and fine-tuned with the complete loss. In our implementation, we employ the PGD attack to generate adversarial training examples, is the most commonly used in existing literature and in practice.
6 Experiments and Results of iGAT
In the experiments, we compare with six state-of-the-an arr ensemble adversarial defense techniques including ADP [30], CLDL [42.. DVERGE [49]. SoE (14). GAL [30] and TRS [50]. The CIFAR-10 and CIEAR- 100 datasets are used for evaluation, both containing 50,000 training and 10,000 test images [11] Overall, ADP, CLDL, DVERGE and SoE appear to be the top performing methods, and
we apply iGAT" to enhance them. The enhanced, referred to as iGAT ADP. iGAT iGAT DVERRE
and iGAT SoE, compared with their original versions, and additionally GAL (30 and TRS (50)
6.1 Experiment Setting
We test against white-box attacks including PGD with 20 inner optimization iterations and CW with
Lo loss implemented by Wu et al. (47). and the black-bor box SignHunter (SH) attack [2] with 500 maximum loss queries. In accordance with Carmon al. [9], the CW attack is applied equidistantly sampled testing examples. We also test against the strongest AutoAttack (AA) [12].
1,000
which encapsulates variants of the PGD attack and the black-box square attack (3) All attack methods use the perturbation strength & = 8/255.
For all the compared methods, an ensemble of N = 8 base classifiers with ResNet-20 20 backbone experimented, for which results of both the average and max output combiners are reported. To implement the iGAT enhancement, the soft distributing rule from Eq. (14) is used. The two hyper-parameters are set as a 0.25 and = 0.5 for SoE, while a = 5 and = 10 for ADP, CLDL and DVERGE, found by grid search. Here SOE uses different parameter setting because its loss construction differs from the others, thus it requires different scale the parameter range for tuning and 3. In practice, minor adjustments to pperr-prraetters have little impact the results. The iGAT training uses a batch size of 512, and multi-step leaning rates of (0.01, 0.002] for CIFAR10 and 10.1, 0.02, 0.004) for CIFAR 100. Implementation of existing methods uses either their pre-trained models or their source code for training that are publicly available. Each experimental used one NVIDIA V100 GPU plus 8 CPU cores.
6.2 Result Comparison and Analysis
We compare different defense approaches by reporting their classification accuracies computed using natural images and adversarial examples generated by different attack algorithms, and report the results in Table The proposed enhancement has lifted the performance of ADP and DVERGE to state-of-thee -o-ttee-rttttttt level for CIFAR- 10 under most of the examined attacks, including both the white-bon and black-box ones. The enhanced DVERGE by iGAT has outperformed all the compared methods in most cases for CIFAR-100. In addition, we report in Table the the accuracy improvement obtained by iGAT for the studied ensemble defense algorithms, computed as their accuracy difference normalised by the accuracy of the original algorithm. It can be seen that iGAT has positively improved the baseline methods in almost all cases. In many cases, it has achieved an accuracy boost over 10%.
'The source codes and pre- trained models can be found https:/ggituubboo///q////AAT

---

Table I: Comparison of classification accuracies in percentage reported natural images and adversarial examples generated by different attack algorithms under Loe -norm perturbation strength e = 8/255. The results are averaged over five independent runs. The best performance is highlighted in bold, the 2nd best underlined.
Average Combiner (%)
Max Combiner (5)
Natural
PGD
CW
SH
AA
Natural
PGD
CW
SH
AA
TRS
83.15
12.32
10.32
39.21
9.10
82.67
11.89
10.78
37.12
7.66
CIFAR 10
GAL
80.85
41.72
41.20
54.94
36.76
80.65
31.95
27.80
50.68
9.26
SoE
82.19
38.54
37.59
59.69
32.68
82.36
32.51
23.88
41.04
18.37
iGAT
81.05
40.58
39.65
57.91
34.50
81.19
31.98
24.01
40.67
19.65
CLDL
84.15
45.32
41.81
55.90
37.04
83.69
39.34
32.80
51.63
15.30
cG.T
85.05
45.45
42.00
58.22
37.14
83.73
40.84
34.55
51.70
17.03
DVERGE
85.12
41.39
43.40
57.33
39.20
84.89
41.13
39.70
54.90
35.15
DGATDE
85,48
42.53
44.50
57.77
39.48
85.27
42.04
40.70
54.79
35.71
ADP
82.14
39.63
38.90
52.93
35.53
80.08
36.62
34.60
47.
27.72
iGATaDr
84.96
46.27
44.90
58.90
40.36
80.72
39.37
35.00
48.36
29.83
TRS
58.18
10.32
10.12
15.78
6.32
57.21
9.98
9.23
14:21
4.34
CIFAR 100
GAL
61.72
22.04
21.60
31.97
18.01
59.39
19.30
13.60
24.73
10.36
CLDL
58.09
18.47
18.01
29.33
15.32
55.51
18.89
13.07
22.14
4.51
iG
59.63
18.78
18.20
29.49
14.36
56.91
20.76
14.09
20.43
5.20
SoE
62.60
20.54
19.60
36.35
15.90
62.62
16.00
11.40
24.25
8.62
iG
63.19
21.89
19.70
35.60
16.16
63.02
16.02
11.45
23.77
8.95
ADP
60.46
20.97
20.55
30,26
17.37
56,20
17.86
13.70
21.40
10.03
iGATade
60.17
22.23
20.75
30.46
17.88
56.29
17.89
14.10
21.47
10.09
DVERGE
63.09
20.04
20.01
32,74
17.27
61.20
20.08
15.30
27.18
12.09
iGATnveaaee
63.14
23.20
22.50
33.56
18.59
61.54
20.38
17.80
27.88
13.89
Table Accuracy improvement in percentage by iGAT, i.c. XGAT original original
x 100%, reported on natural images and
adversarial examples generated by different attack algorithms under Lo. norm perturbation strength 8//255.
Average Combiner
Max Combiner
Natural
PGD
CW
SH
AA
Natural
PGD
CW
AA
CIFAR1
ADP
+3.43
+16.75
+15.42
+11.28
+13.59
+0.80
+7.51
+1.16
+1.40
+7.61
DVERGE
+0.42
+2.75
+2.53
+0.77
+0.71
+0.45
+2.21
+2.52
-0.20
+1.59
SoE
-1.39
+5.29
+5.48
-2.98
+5.37
-1.12
-1.63
+0.51
-0.90
+6.97
CLDL
+1.07
+0.29
+0.45
+4.15
+0.27
+0.05
+3.81
+5.34
+0.14
+11.31
CIFRR10
ADP
-0.48
+6.01
+0.97
+0.66
+2.94
+0.16
+0.17
+2.92
+0.33
+0060
DVERGE
+0.08
+15.77
+ +12.44
+.2.50
+7.64
+0.56
+1.49
+16.34
+2.58
+14.89
SoF.
+0.94
+6.57
+0.51
-2.06
+1.64
+0.64
+0.13
+0.44
-1.98
+3.83
CLDL
+2.65
+1.68
+1.05
+0.55
77.47
+2.52
+9.90
+7.80
-7.72
+15.30
Here are some further discussions on the performance. We observe that DVERGE and its iGAT enhancement perform proficiently both CIFARIO and CIFAR100. while ADP and its enhancement are less robust on CIFAR100. We attempt to explain this by delving into the algorithm nature of DVERGE and ADP. The ADP design encourages prediction disparities among the base models. As a result, each base model becomes proficient in classifying a subset of classes that the other base models may struggle with. However, side effect of this is discourage base models from becoming good at overlapping classes, which may become ineffective when having to handle a larger number of classes. The reasonably good improvement achieved by iGAT for ADP in Table indicates that an addition of global adversarial examples is able to rescue such situation to a certain extent. On the other hand, in addition to encouraging adversarial diversity among the base models, DVERGE also aims at stable classification so that each example is learned by multiple base models. This potentially makes suitable for handling both large and small numbers of classes. Moreover, we also observe that the average combiner provides better performance than the max combiner in general.

---

Table 3: Results of ablation studies based on iGAT ADP using CIFAR-10 under the PGD attack. The results averaged over five independent runs. The best performance is highlighted bold.
Opposite Distrihuting
Random Distributing
Hard Distributing
8
iGATADI ADP
Natural (%) PGD (%)
82.45
83.05
83.51
83.45 45
84.96
41.31
42.60
44.21
42.32
46.25
The reason can be that an aggregated prediction from multiple well-trained base classifiers is more statistically stable.
6.3
Ablation Studies
The key designs of iGAT include its distributing rule and the regularization term. We perform ablation studies to examine their effectiveness. Firstly, we compare the used soft distributing rule with three alternative distributing rules, including (1) a distributing rule opposite the proposed. which allocates the adversarial examples to the base models that produce the lowest prediction score, (2) a random distributing rule by replacing Eq. (4p by uniform distribution, and (3) the hard distributing rule in Eq. (33). Then, we compare with the setting of 3 = 0 while keeping the others unchanged. This change removes the proposed regularization term. Results are reported in Table Jssing ADP with the average combiner, evaluated by CIFAR- 10 under the PGD attack. It can be seen that a change or removal of key design results in obvious performance drop, which verifies the effectiveness of the design.
Conclusion, Limitation and Future Work
We investigate the challenging and crucial problem of defending against adversarial attacks in the input space of neural network, with the goal of enhancing ensemble robustness against such attacks while without sacrificing much the natural accuracy. We have provided formal justification of the advantage of ensemble adversarial defence and proposed an effective algorithmic improvement, bridging the gap between theoretical and practical studies. Specifically, we have proven a decrease in empirical 0-1 loss calculated on data samples challenging to classify, which constructed to simulate the adversarial attack and defence scenario, under neural network assumptions that are feasible in practice. Also, we have proposed the iGAT approach, applicable to any ensemble adversarial defense technique for improvement. It is supported by (1) probabilistic distributing rule for selectively allocating global adversarial examples to train base classifiers, and (2) a regularization penalty for addressing vulnerabilities across all base classifiers. We have conducted thorough evaluations and ablation studies using the CIFAR- 10 and CIFAR-100 datasets, demonstrating effectiveness of the key designs of iGAT. Satisfactory performance improvements up 17% have been achieved by iGAT.
However, there is limitation in our work. For instance, our theoretical result is developed for only two base MLPs. We are in progress of broadening the scope of Theorem 4.1 by further relaxing the neural network assumptions, researching model architectures beyond MI.Ps and beyond the average/max combiners, and more importantly generalizing the theory to more than two base classifiers. Additionally, we are keen to enrich our evaluations using large-scale datasets, e.g., ImageNet. So far, we focus exploiting curvature information of the loss landscapes to understand adversarial robustness. In the future, it would be interesting to explore richer geometric information to improve the understanding. Despite the research success, potential negative societal impact of our work is that it may prompt illegal attackers to develop new attack methods once they become aware of the underlying mechanism behind the ensemble cooperation.
Acknowledgments
We thank the five NeurlPS reviewers for their very insightful and useful comments that help improve the paper draft. We also thank "The University of Manchester China Scholarship Council Joint Scholarship" for funding Yian's PhD research.
10

---

References
[1] Naveed Akhtar, Ajmal Mian, Navid Kardan, and Mubarak Shah. Advances in adversarial attacks and defenses in computer vision: A survey. IEEE Access, 9:155161-155199 2021.
[2] Abdullah Al-Dujaili and Una-May O' R'Relly. Sign bits are all you need for black-box attacks. In International Conference Learning Representations (ICLR), 2020.
[3] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision (ECCV), 2020.
[4] Tao Bai, Jinqi Luo, Jun Zhao, Bihan Wen, and Qian Wang. Recent advances in adversarial training for ad adversarial robustness. In International Joint Conference on Artificial Intelligence (IJCAI), pages 4312 2021.
[5] Siddhant Bhambri, Sumanyu Muku, Avinash Tulasi, and Arun Balaji Buduru. A survey of black-box adversarial attacks on computer vision models. arXiv preprint, 2019.
[6] Tobias Blickle and Lothar Thiele. A comparison of selection schemes used in evolutionary algorithms. Evolutionary Computation, 4(4):361-394, 1996.
[7] Gavin Brown, Jeremy Wyatt, Peter Tino, and Yoshua Bengio. Managing diversity regression ensembles. Journal of Machine Learning Research (JMLR), 6(9), 2005.
[8] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium Security and Privacy (SP), 2017.
(9] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. Advances in neural information processing systems (NeurIPS), 32, 2019.
[10] Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. A survey on adversarial attacks and defences. CAAI Transactions on In- telligence Technology, 6(1):25-455 2,,5-45,0211 2021.
[11) Szegedy Christian, Zaremba Wojciech, Sutskever Ilya, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. International Conference Learning Representations (ICLR), 2014.
[12] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International Conference Machine Learning (ICML), 2020.
[13] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint, 2020.
[14] Sen Cui, Jingfeng Zhang, Jian Liang, Bo Han, Masashi Sugiyama, and Changshui Zhang. Synergy-of-experts: Collaborate to improve adversarial robustness. Advances in Neural Infor- mation Processing Systems (NeurIPS), 35:32552-3323
2022.
[15] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In International Conference Computer Vision and Pattern Recognition (CVPR), pages 9185-9193, 2018
[16] Alhussein Fawzi, Scyed-Mohsen Meosavi-Dezfooli, Pascal Frossard, and Stefano Soatto. Em- pirical study of the topology and geometry of deep networks. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[17] C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimiza- tion. In International Conference on Learning Representations (ICLR), 2017.
[18] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. International Conference on Learning Representations (ICLR), 2015.
[19] Chuan Guo, Jacob Gardner, Yurong You. Andrew Gordon Wilson, and Kilian Weinberger. Simple black-box adversarial attacks. In International Conference on Machine Learning (ICML), pages 2484-2493, 2019.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In International Conference on Computer Vision and Patern Recognition (CVPR), 2016.
11

---

[21] Ziwen He, Wei Wang, Jing Dong, and Tieniu Tan. Revisiting ensemble adversarial attack. Signal Processing: Image Communication. 107:116747, 2022.
[22] Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. International Conference on Machine Learning (ICML), 2019.
[23] Sanjay Kariyappa and Moinuddin K Qureshi. Improving adversarial robustness of ensembles with diversity training. arXiv preprint arXiv: 10009998, 2019.
[24] Daniel Lowd and Christopher Meek. Adversarial learning. International Conference on Knowledge Discovery and Data Mining (KDD), 2005.
[25] Zhiping Lu, Hongchao Hu, Shumin Huo, and Shuyi Li. Ensemble learning methods of adver- sarial attacks and defenses in computer vision: Recent progress. International Conference on Advanced Computing and Endogenous Security (ACES), pages 1-10, 2022.
[26] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
[27] Seyed-Mohsen Moosavi-Dezfoolli Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In International Conference on Computer Vision and Pattern Recognition CVPR, 2016.
[28] Seyed-Mohsen Moosavi- -aeaaaiiiieoooi,,, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[29] Amir Najafi, Shin-ichi Maeda, Masanori Koyama, and Takeru Miyato. Robustness to adversarial perturbations in learning from incomplete data. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[30] Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning (ICML), pages 4970 097000-999..000 2019.
[31] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In IEEE European Symposium on Security and Privacy (EuroSP), 2016.
[32] Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhus- sein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
[33] Aditi Raghunathan, Sang M. Xie, Fanny Yang, John C. Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In International Conference on Machine Learning (ICML). 2020.
[34] Kui Ren, Qian Wang. Cong Wang, Zhan Qin, and Xiaodong Lin. The security of autonomous driving: Threats, defenses, and future directions. Proceedings of the IEEE, 108(2):357-372, 2019.
[35] Vikash Sehwag, Shiqi Wang, Prateck Mittal, and Suman Jana. Hydra: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems (NeurIPS), 33: 19655-19666, 2020.
[36] Chuanbiao Song, Kun He, Liwei Wang, and John E. Hopcroft. Improving the generalization of adversarial training with domain adaptation. In International Conference on Learning Representations (ICLR), 2019.
[37] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations (ICLR), 2018
[38] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations (ICI.R), 2019.
[39] Naonori Ueda and Ryohei Nakano. Generalization error of ensemble estimators. In International Conference Neural Networks (ICNN), volume pages 90-95, 1996.
12

---

[40] Jonathan Uesato, Brendan Odonoghue, Pushmeet Kohli, and Aaron Oord. Adversarial risk and the dangers of evaluating against weak attacks. In International Conference on Machine Learning (ICML), 2018.
[41] Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems (NeurlPS), 31, 2018.
[42] Lele Wang and Bin Liu. Adversarial ensemble training by jointly learning label dependencies and member models. In International Conference on Intelligent Computing (ICIC), pages 3-20, 2023.
[43] Xianmin Wang, Jing Li, Xiaohui Kuang, Yu-an Tan, and Jin Li. The security of machine learning an adversarial setting: survey. The Journal of Parallel and Distributed Computing (JPDC), 130:12-2:: 23, 2019.
[44] Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. International Conference on Learning Representations (ICLR), 2019.
[45] Danny Wood, Tingting Mu, Andrew Webb, Henry Reeve, Mikel Lujan, and Gavin Brown. A unified theory of diversity in ensemble learning. arXiv preprint arXiv:2301.03962 2023.
[46] Boxi Wu, Jinghui Chen, Deng Cai, Xiaofei He, and Quanquan Gu. Do wider neural networks really help adversarial robustness? Advances in Neural Information Processing Systems (NeurIPS), 34:7054-7067, 2021.
[47] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances Neural Information Processing Systems (NeurIPS), 08-2958-2966 2020.
[48] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improving adversarial robustness. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[49] Huanrui Yang. Jingyang Zhang, Hongliang Dong, Nathan Inkawhich, Andrew Gardner, Andrew Touchet, Wesley Wilkes, Heath Berry, and Hai Li. Dverge: diversifying vulnerabilities for enhanced robust generation ensembles. Advances in Neural Information Processing Systems (NeurIPS), 33:5505 5515,, 2020.
[50] Zhuolin Yang, Linyi Li, Xiaojun Xu, Shiliang Zuo, Qian Chen, Pan Zhou, Benjamin Rubinstein, Ce Zhang, and Bo Li. Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[51] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. British Machine Vision Conference (BMVC). 2016.
[52] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning (ICML), 2019.
13


Reviews:
[{"review_id": "NBhD44ZmWU", "reviewer": "Reviewer_Xydx", "summary": "The ensemble strategy has gained popularity in the field of adversarial defense, wherein multiple base classifiers are trained cooperatively to protect against adversarial attacks. While empirical evidence supports its effectiveness, the underlying theoretical explanations for why an ensemble of adversarially trained classifiers outperforms individual classifiers remain elusive. This paper aims to bridge this gap by presenting a novel error theory dedicated to comprehending ensemble adversarial defense, providing provable reductions in the 0-1 loss on challenging example sets encountered in adversarial defense scenarios.\n\nBuilding upon this theory, the authors propose an innovative approach called interactive global adversarial training (iGAT) to further enhance ensemble adversarial defense. iGAT incorporates two key components: (1) a probabilistic distributing rule that selectively assigns globally challenging adversarial examples to different base classifiers, and (2) a regularization term designed to address the most severe weaknesses of the base classifiers.\n\nEvaluations conducted on CIFAR10 and CIFAR100 datasets demonstrate that iGAT significantly improves the performance of existing ensemble adversarial defense techniques. Under both white-box and black-box attacks, iGAT achieves performance gains ranging from 1% to 17%. These results highlight the efficacy of iGAT in bolstering the resilience of ensemble classifiers against adversarial attacks.", "questions": "Please provide results on ImageNet, and explain  the differentiability of Hard Distributing Rule", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel Error Analysis: The paper presents a new error theory specifically dedicated to understanding the advantages of ensemble adversarial defense. This analysis fills a crucial gap in the literature by providing theoretical explanations for why an ensemble of adversarially trained classifiers is more robust than individual classifiers. By offering a deeper understanding of ensemble defense mechanisms, the paper contributes to the advancement of the field.\n\n2. Targeted Solution Algorithm: In response to the theoretical insights gained from the error analysis, the paper proposes an innovative approach known as interactive global adversarial training (iGAT). This algorithmic solution addresses the shortcomings of existing ensemble adversarial defense techniques. It introduces a probabilistic distributing rule that selectively allocates globally challenging adversarial examples to different base classifiers, along with a regularization term that targets the most severe weaknesses of these classifiers. The proposed iGAT algorithm is tailored to enhance the performance and robustness of ensemble classifiers.", "weaknesses": "1. Limited Validation on Large-Scale Datasets: The paper's experimental evaluation focuses on the CIFAR10 and CIFAR100 datasets, which are relatively smaller in scale compared to datasets like ImageNet. The absence of validation on larger datasets limits the generalizability of the proposed iGAT algorithm and its effectiveness in real-world scenarios.\n\n2. Limited Range of Attack Algorithms: The evaluation of the proposed defense technique primarily considers a limited range of attack algorithms without exploring a wider variety, including black-box attacks. While the performance improvements demonstrated against the evaluated attack algorithms are promising, it is important to assess the effectiveness of iGAT against a broader set of attack strategies to ensure its robustness and applicability in different adversarial settings.\n\n3. Lack of Clear Explanation on the Differentiability of Hard Distributing Rule: The paper introduces a probabilistic distributing rule, referred to as the hard distributing rule, to assign adversarial examples to base classifiers. However, the paper does not provide a clear explanation regarding the differentiability of this rule. This omission raises concerns about the potential presence of obfuscated gradients, which could hinder the effective training of the ensemble classifiers and impact the overall performance of iGAT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the insightful comments and address them regarding the following three aspects.\n# Answers\n* **Q1. On ImageNet Results:** \n\nWe thank the reviewer for the suggestion. If allowed by time and the computing resource available to us, we would very much like to report the ImageNet results before the end of the rebuttal period, but unfortunately, we are not able to do so.  Using our computing resource of only one NVIDIA V100 GPU plus 8 CPU cores, it takes around 10-16 days to train one normal ResNet20 using the ImageNet data, and it takes even longer time to train a robust RestNet20 enhanced by an adversarial defense technique. Then, to train 8 such base ResNet20 and fine-tune them by the proposed adversarial ensemble enhancement, it will take more than 10 days $\\\\times$ 8 $\\\\times$ 1.2 > 3  months.  We are currently investigating computing resource availability and setting experiments for ImageNet, and hoping to obtain some ImageNet results for an ensemble with less than 8 ResNet20 in a shorter time.\n\nIn general,  most of the state-of-the-art works on ensemble adversarial defense (e.g., those listed below) validate their models using CIFAR10 and CIFAR100 datasets, and some use simpler or even smaller datasets for digit recognition like the MNIST and SVHN datasets. This is why we have followed a similar evaluation setup.\n\n[A] Pang T et al. Improving adversarial robustness via promoting ensemble diversity, ICML, 2019.\n\n[B] Yang H et al. DVERGE: diversifying vulnerabilities for enhanced robust generation of ensembles, NeurIPS, 2020.\t\t\t\n\n[C] Yang Z  et al. Trs: Transferability reduced ensemble via promoting gradient diversity and model smoothness, NeurIPS, 2021.\n\n[D] Gao X and Xu C Z. MORA: Improving Ensemble Robustness Evaluation with Model Reweighing Attack, NeurIPS, 2022.\n\n[E] Cui S et al. Synergy-of-Experts: Collaborate to Improve Adversarial Robustness. NeurIPS, 2022.\n\n[F] Dbouk H and  Shanbhag N. On the Robustness of Randomized Ensembles to Adversarial Perturbations, ICML, 2023.\t\n\n* **Q2. On Differentiability of Hard Distributing Rule:** \n\n We use the distributing rule to allocate different training samples to different base classifiers. The fine-tuning process by the proposed approach along with the classification loss and its gradient calculation start after the sample distributing process is completed. Thus, the training data used for fine-tuning each robust base classifier is fixed. There is no differentiation performed over these distributing rules.\n\n* **Q3. On Attack Algorithms:**\n\nEncouraged by the reviewer's comment regarding the limited range of attack algorithms, we have performed more experiments to test the effectiveness of iGAT against two more black-box attack strategies, using the CIFAR-10 dataset. These two attacks are two different black-box attacks that are time-efficient and commonly used, and we expect to use these two attacks to further compare the performance among ensemble defense models. Results show that in most cases the enhanced ensemble defense model has a better robustness against various black-box attacks. \n\n---\n|Table 1. Results on extra black-box attacks.| \n---\n---\n||Simple Attack (\\%)| Bandit Attack (\\%)|\n|---|---|---|\n|ADP|75.91|59.21| \n|iGAT$_{ADP}$|**79.43**|**64.55**| \n|DVERGE|79.43|63.80| \n|iGAT$_{DVERGE}$|**79.61**|**64.89**| \n|CLDL|76.82|63.80| \n|iGAT$_{CLDL}$|**78.84**|**65.25**| \n|SoE|**76.22**|66.10| \n|iGAT$_{SoE}$|75.18|**66.50**| \n---\n"}, {"review_id": "FPyakKtY9N", "reviewer": "Reviewer_cK43", "summary": "This paper investigates enhancing the performance of ensemble-based adversarial defense methods against adversarial attacks. Considering the effectiveness of ensemble-based adversarial defense methods have been proven empirically while the understanding of their mechanism is rather limited, authors first propose a theory model to demonstrate the advantages of an ensemble of adversarially trained classifiers compared to individual ones. Guided by theoretical analysis, a novel method is developed to enhance the performance of existing ensemble-based defense methods through distributing different examples to different base classifiers. Experiments conducted on various datasets verify the superiority of the proposed method.", "questions": "1. Based on both Table 1 and Table 2, the improvement brought by the proposed method is not consistent among different ensemble-based defense methods. For example, for average combiner, the most significant performance improvement under AA is brought by the proposed method with ADP on CIFAR10 while with DVERGE on CIFAR100. Can authors provide more discussion about this phenomenon? Or, under which scenarios, the proposed method can bring most significant improvement?\n2. Can authors provide the performance of all four methods in Table 1?", "limitations": "If possible, authors can add some discussion about the limitations of the proposed method.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes theoretical analysis on understanding the advantages of ensemble-based adversarial defense methods, which provides a new perspective to evaluate ensemble-based defense methods and hence could be helpful for future works\n2. Based on theoretical understanding, a novel method is proposed to boost the performance of existing ensemble-based defense methods with reduced training time\n3. Experiment results demonstrate the effectiveness of the proposed method", "weaknesses": "1. In experiments, authors apply the proposed method to enhance four state-of-the-art ensemble adversarial defense methods. However, in Table 1, only TOP 3 most enhanced performance are provided. It would be better if the performance related to all four ensemble defense methods can be provided\n2. One feature of the proposed method is distributing different examples to different base classifiers. As author mentioned, this feature can make the proposed method bring reduced training time. It would be better if some experiments about training efficiency can be conducted", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the very positive comments and the accurate summary of our work. We address the comments by grouping them into three categories on questions, weaknesses, and limitations.\n\n# Answers\n* **Q1.** DVERGE performs proficiently on both CIFAR10 and CIFAR100. This can be credited to its approach of not only achieving adversarial diversity among base models but also ensuring that each sample is learned by multiple base models for stable classifications. Conversely, ADP's performance on CIFAR100 is not as robust as on CIFAR10. We aim to explain this by delving into its algorithm design. The novel aspect of ADP is its effort to maximize the prediction disparities among base models, thereby ensuring ensemble diversity. This implies that each model becomes proficient in classifying a subset of classes that other base models may struggle with. When the class count rises from 10 (in CIFAR10) to 100 (in CIFAR100), the inherent limitations of a single base model's capacity to efficiently classify a multitude of classes become evident. Consequently, the ADP approach, which discourages base models from learning overlapping classes, may not perform as effectively on datasets with a larger number of classes.\n\n* **Q2.** We have added results for all methods using the average combiner. See our reply to weakness 1.\n\n# Discussion about weaknesses\n\n* 1.To follow the suggestion, we have now collected results for the worst performing method (the 4th), which are iGAT$\\_{SoE}$ for CIFAR10 and iGAT$\\_{CLDL}$ for CIFAR100 (see below), using the average combiner. We will include these to the performance Table 1 in the revised draft, and will also collect results for these two methods using the max combiner.\t\t\n\n---\n|Table 1. Added results of iGAT$\\_{SoE}$ for CIFAR10 and iGAT$\\_{CLDL}$ for CIFAR100.| \n---\n---\n|Average Combiner (\\%)|||||||\n|---|---|---|---|---|---|---|\n|CIFAR10|SoE|**82.19**|38.54|37.59|**59.69**|32.68|\n||iGAT$\\_{SoE}$|81.05|**40.58**|**39.65**|57.91|**34.50**|\n|CIFAR100|CLDL|58.09|18.47|18.01|29.33|**15.52**|\n||iGAT$\\_{CLDL}$|**59.63**|**18.78**|**18.20**|**29.49**|14.36|\n---\n\n* 2. The distributing feature reduces the training data size to $\\frac{1}{N}$ for training each base classifier, where  $N$ is the number of base classifiers. This has the direct effect of improving training time.   We have added a simple experiment by comparing the training time required by a full version of iGAT$\\_{ADP}$ and that by a modified version with this distributing feature removed. CIFAR10 data are used for Evaluation. The table below shows that using the distributing feature can slightly reduce the training time.\n\n---\n|Table 2. Evaluation of training efficiency averagely at an iteration.| \n---\n---\n||Time elapsed (s)|\n|---|---|\n|Without the distributing feature|5.63|\n|With the distributing feature|5.42|\n---"}, {"review_id": "THQ48A2gsJ", "reviewer": "Reviewer_hPTz", "summary": "1. This work aims at creating a theoretical framework to explain why an ensemble of MLPs can be more robust than the single components.\n2. The assumptions under which the ensemble of MLPs have more robustness are given. The first set of conditions are on each individual model itself. The second set of assumption state that the performance of each classifier is acceptable in data points that are somehow close to points of other classes.\n3. Under these assumptions, Theorem 4.1 states that an ensemble of two acceptable models always performs better than the individual ones.\n4. Authors propose Interactive Global Adversarial Training to boost the robustness of an ensemble.\n", "questions": "1. In Equation (5), what is the expectation over $h \\in \\mathcal{H}$? Are you computing the average error rate over all the family of hyothesis?\n2. In Assumption 4.2 for MLPs, assumption 1 says \"each activation output is i.i.d with expectation $\\mu$\". Is this reasonable ? What does this mean for the MLP ? Does this restrict the family of activation functions one can consider ?\n3. In Assumption 4.2 for MLPs, what are the orders of magnitude of the constants $\\tilde{\\lambda}$ and $B$ ? How does this impact the bound on Equation (6) ?\n4. Is Assumption 4.4 about \"Acceptable Classifiers\" always reasonable in practice ?\n5. Why do you think that for CIFAR-10, the ensemble diversity framework that obtained the largest improvement from your proposed technique was ADP ? And why for CIFAR-100 it seems to be DVERGE ?\n6. Why do you think that the iGAT parameters $\\alpha$ and $\\beta$ are so different for SoE and the rest of baselines ?\n7. Which attack was used for the iGAT training added global adversarial loss?\n8. How much time did training take ? Your implementation shows 2000 epochs were used, which seems huge for me. Does your method require more epochs than the baselines ?\nTypos:\n      Line 254: Theory 4.1  --> Theorem 4.1\n      Model size column in Table 2, Appendix 3 seems to be wrong.\n\n", "limitations": "Yes, they have done.\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Experiments involve a wide variety of ensemble defense baselines and attacks.\n2. Ablation studies are performed to test the effect of the different proposed mechanisms.\n3. Assumptions in theoretical part are enumerated thoroughly.\n4. Experimental setup is well described\n5. The motivation to understand and enhance the adversarial robustness of ensembles is an interesting research direction.\n6. Experimental results suggest that the proposed technique can indeed increase the robustness of existing ensemble training frameworks.\n", "weaknesses": "1. Theory and practice feel a bit disconnected. Theory works for 2 models, and only MLPs, and experiments are made with 8 Resnets.\n2. Some assumptions in the theoretical part could be discussed in more detail to let the reader know if the requirements and assumptions could actually hold in practice or not.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the insightful comments and questions and address them below.\n# Weaknesses\n* **On Weakness 1.** We appreciate the reviewer for pointing this out.  By the time we submitted this paper, we were only able to develop the error theory for two base MLPs under those reported assumptions. But we have made very promising progress to improve it since. Now, we have managed to extend the theory to an arbitrary number of base MLPs with more relaxed assumptions and plan to submit the new results to a suitable venue soon.\n\n* **On Weakness 2.** We thank the reviewer for the very useful comment. We will include more discussions on our assumptions, also their practical feasibility and indication in the revised draft, based on what has been addressed in our reply to the questions below.\n\n# Questions\n* **Q1.** Yes, you are right.  \n\n* **Q2.** We acknowledge that the assumption of independently having the same expectation $\\mu$ for all the activation outputs is somewhat restricted.  But  one can consider the commonly used batch normalization techniques that independently control the mean and variance of an activation output (or input) so that different activators can present outputs of the same mean and variance.  Therefore, for MLPs with batch normalization, our assumption is reasonable. In addition, if we consider fixed input for the neural network but allow i.i.d neural network weights, the i.i.d assumption is reasonable. We include a related paper concerning the i.i.d weights and layer outputs/inputs [*].\n\n[*] Apollonio N  et al. Normal approximation of Random Gaussian Neural Networks. arXiv preprint arXiv:2307.04486, 2023.\n\n* **Q3.** We have investigated empirically the values of the network curvature $\\tilde{\\lambda}$ and the weights-related quantity $B$ using neural networks trained by the ADP defense techniques, and recorded the values below. Smaller values of $\\tilde{\\lambda}$  and $B$ indicate a looser upper bound in Eq. (6).  According to our Definition 4.3, a looser upper bound allows us to define an ambiguous pair containing two intra-class examples that are less close to each other, thus less challenging to classify.   \n\n---\n|Table 1. Magnitude of $\\tilde{\\lambda}$ and $B$| \n---\n---\n|$\\tilde{\\lambda}$| $B$|\n|---|---|\n|0.06|<0.01| \n---\n\n* **Q4.**  The assumption of being an acceptable classifier is reasonable, and we believe that it can be met fairly easily by base classifiers of an adversarial ensemble model that is fairly well-trained. In our submitted supplementary material, we have already included some results to examine whether the base classifiers from the iGAT ensemble are acceptable (e.g., Tables 4 and 6).  To state some observations here,  the percentages for correctly classifying examples with part one of Assumption 4.4 satisfied are  $56\\\\%$ for iGAT$\\_{ADP}$, $79\\\\%$ for iGAT$\\_{DVERGE}$ and $50\\\\%$ for iGAT$\\_{CLDL}$, where each percentage is averaged over the 8 base classifiers. These are greater than $42.5\\\\%$. Correspondingly, the averaged percentages for incorrectly classifying examples with part two of Assumption 4.4 satisfied are $73.12\\\\%$ for iGAT$\\_{ADP}$, $80.19\\\\%$ for iGAT$\\_{DVERGE}$ and $78.39\\\\%$ for iGAT$\\_{CLDL}$, which are fairly high. Overall, for these examined cases, Assumption 4.4 is satisfied with a fairly high chance. We will enrich our discussion on this in the revised draft.\n\n* **Q5.** DVERGE performs proficiently on both CIFAR10 and CIFAR100. This can be credited to its approach of not only achieving adversarial diversity among base models but also ensuring that each sample is learned by multiple base models for stable classifications. Conversely, ADP's performance on CIFAR100 is not as robust as on CIFAR10. We aim to explain this by delving into its algorithm design. The novel aspect of ADP is its effort to maximize the prediction disparities among base models, thereby ensuring ensemble diversity. This implies that each model becomes proficient in classifying a subset of classes that other base models may struggle with. When the class count rises from 10 (in CIFAR10) to 100 (in CIFAR100), the inherent limitations of a single base model's capacity to efficiently classify a multitude of classes become evident. Consequently, the ADP approach, which discourages base models from learning overlapping classes, may not perform as effectively on datasets with a larger number of classes.\n\n* **Q6.** Both $\\alpha$ and $\\beta$ were tuned using grid search,  and their searching ranges were determined by the magnitude of losses used in the original ensemble methods. The loss construction of SoE differs from other methods quite a lot, which requires different searching ranges to match its magnitude and thus results in quite different values of $\\alpha$ and $\\beta$.\n\n* **Q7.** We used the PGD attack to generate adversarial training examples and to simulate global adversarial attacks, as it is the most commonly used by existing works.\n\n* **Q8.** We exemplify the training time by comparing the  ADP defense and its enhancement iGAT$\\_{ADP}$.  Firstly, we have observed the training time required by updating the weights of ADP and iGAT$\\_{ADP}$ using 1024 training examples 1000 times, which are  $50.431$ and $50.088$ in seconds. We can say that they take a similar time to train for the same number of iterations using the same amount of data.  Then, we have recorded the total number of training epochs required by both, which is $691$ epochs for ADP   while  $163$ epochs for  iGAT$\\_{ADP}$. Based on these, we can conclude that  iGAT$\\_{ADP}$ trains faster than ADP. In practice, the user will train their ensemble model using ADP first, and fine-tune it using iGAT$\\_{ADP}$, which will require around $691 +193 <900$ epochs in total to obtain a good model. The epoch number 2000 is the maximum allowed epoch number that we set. The training terminates a lot earlier than that."}, {"review_id": "jOVVBRseLT", "reviewer": "Reviewer_XCW5", "summary": "This paper derives provable error reduction when shifting from employing one neural network to an ensemble of two neural networks, which provides new insights into ensemble adversarial defenses. Based on the theoretical findings, the authors propose iGAT to narrow the subspace where all base classifiers misclassify inputs. ", "questions": "Why is the robustness of ensemble defenses significantly lower than the robustness of state-of-the-art non-ensemble defenses? For example, the proposed method achieves 40.36% robustness under AutoAttack on the CIFAR10 dataset with a perturbation budget of 8/255. However, state-of-the-art non-ensemble adversarial training achieves 70.69% robustness under the same conditions (See RobustBench for detail).", "limitations": "The core theorem only addresses the simple case where there are only two MLP classifiers.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes a new understanding of ensemble adversarial defenses, providing new insights into why ensemble adversarial defenses are more effective.\n2. Based on the theoretical analysis, the authors propose iGAT, which aims to narrow the subspace where all base classifiers misclassify inputs by allocating global adversarial examples to appropriate base classifiers and designing a regularization term for misclassified inputs.\n3. The detailed experimental results demonstrate that the proposed methods significantly improve the adversarial robustness of ensemble adversarial defenses.\n", "weaknesses": "There is a lack of experiments with a small number of base classifiers. Will the proposed method still be effective when there are only a few base classifiers?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the positive comments and the accurate summary of our work and address the comments by grouping them into three categories: weaknesses, questions, and limitations.\n\n# Weaknesses\n* **On Weakness 1.** Encouraged by your comment, we have conducted a new experiment to examine performance improvement of iGAT over the two baselines of ADP and DVERGE, for an ensemble with only two base classifiers and the average combiner. The results are shown below, which will be included in the revised draft. The improvement is still satisfactory when using only two base classifiers. \n\n---\n|Table 1. Results on ensembles of two base classifiers using the average combiner.| \n---\n---\n|||Natural (\\%)|PGD (\\%)|SH (\\%)|AA (\\%)|\n|---|---|---|---|---|---|\n|CIFAR10|ADP|73.02|31.94|33.17|25.72|\n||iGAT$\\_{ADP}$|**73.92**|**35.88**|**37.78**|**27.77**|\n|CIFAR100|DVERGE|55.72|21.39|28.26|16.97|\n||iGAT$\\_{DVERGE}$|**56.09**|**21.67**|**29.41**|**17.98**|\n---\n\n# Questions\n* **Q1.** The main reason for this is the model size. Obviously, the bigger the model is the better their performance is when being trained with sufficient data. For instance, the top-performing models in   RobustBench [A] are WRN-70-16 of 267M, WRN-106-16 of at least 250M, and WRN-34-10 of 184M  [B]. According to the leaderboard of CIFAR10 data [C], the top-performing models are over 600M.  But the size of our ensemble model experimented in the paper is only 9M, containing 8 base ResNet-20 each of 1.1M.  We chose to experiment with such modest-sized models because the state-of-the-art adversarial defense works (see [14,23,30,42,50,51] in the submitted manuscript) that we compare with have all used the same. We adopted the same model architecture to focus on examining the effectiveness of the proposed enhancement approach.\n\n[A] Croce F et al. RobustBench: a standardized adversarial robustness benchmark, NeurIPS, 2021}\n\t\t\t\t\n[B] https://github.com/yaodongyu/TRADES}\n\t\t\t\t\n[C] https://paperswithcode.com/sota/image-classification-on-cifar-10\t\n\n# Limitations\n\nWe appreciate the reviewer for pointing out this limitation.  We have actually made very promising progress after we submitted the paper. We have now managed to extend the theory to an arbitrary number of base MLPs, and plan to submit the new results to a suitable venue soon. Only for a matter of interest,  we briefly explain our extended proof sketch here. Suppose there are $M$ base models that can classify one sample, (e.g., $\\\\mathbf{x}\\_i$) well and $N$ base models that classify the other sample (e.g., $\\\\mathbf{x}\\_j$) well. For the max combiner, let $I\\_M := \\\\{i_m\\\\}\\_{m=1}^M$ be the indices of the $M$ base models and $I\\_N := \\\\{i_n\\\\}\\_{n=1}^N$ be the indices of the $N$ base models. Then, according to Eq. (62), given an index $i\\_m \\\\in I\\_M$, it has $\\\\max\\_{c\\\\ne y\\_i} \\\\max (h^{i\\_m}\\_c(\\\\mathbf{x}\\_i), h^{i\\_n}\\_c(\\\\mathbf{x}\\_i)) < h\\_{y\\_i}^{i\\_m}(\\\\mathbf{x}\\_i)$ for any $i\\_n\\\\in I\\_N$. This would lead to $0$ loss on $\\\\mathbf{x}\\_i$. Likewise, given an index $i\\_n\\\\in I\\_N$, it has $\\\\max\\_{c\\\\ne y\\_j} \\\\max (h^{i\\_n}\\_c(\\\\mathbf{x}\\_j), h^{i\\_m}\\_c(\\\\mathbf{x}\\_j)) < h\\_{y\\_j}^{i\\_n}(\\\\mathbf{x}\\_j)$ for any $i\\_m\\\\in I\\_M$, which then give $0$ loss on $\\\\mathbf{x}\\_j$. As our current proof shows one single classifier can have at least $1$ loss on the two samples $\\\\mathbf{x}\\_i$ and $\\mathbf{x}\\_j$, the combination of more than two base models can perform better. For the average combiner, we will show that when $M= N$, the combination of more than two base models performs at least the same as single-branch models, and when $M\\\\ne N$, the ensemble model performs better."}, {"review_id": "V3NH3K9ueg", "reviewer": "Reviewer_d7H2", "summary": "This paper first propose to understand ensemble adversarial defense from a theoretical perspective. Then the authors develop iGAT, including a distributing strategy is to encourage each base classifier to keep improving over regions that they are relatively good at classifying. This leads to a hard distributing rule that assigning in a deterministic way the example to the base classifier that returns the highest predicted probability on its ground truth class, and a soft distributing rule based on a ranking-based score. Besides, a  regularization term is introduced to address the severest weakness,  by minimizing the probability score of the most incorrectly predicted class by the most erroneous base classifier. Empirical evaluations are done on enhancing previous ensemble defenses.", "questions": "My questions are listed in Weaknesses.", "limitations": "Limitations are not discussed in the paper. As to the potential negative societal impact, there is a sentence in Conclusion.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper provides a theoretical analysis on why an ensemble of adversarially trained classifiers is more robust than single ones and develop a new error theory dedicated to understanding ensemble adversarial defense. This theoretical analysis provides some new insights on the effectiveness of ensemble defense.\n\nInspired from the theory, the authors propose iGAT, including probabilistic distributing rule and a regularization term for rescuing the severest weaknesses of the base classifiers. Empirically, iGAT is shown to consistently enhance previous ensemble defense strategies such as ADP, CLDL, DVERGE and SoE.", "weaknesses": "I have some concerns:\n\n- The designs of distributing rule in Section 5.1 and regularization in Section 5.2 are somewhat heuristic. For example, it is not well-justified on why ranking-based probability in Equation (10) is preferred compared to other soft distribution.\n\n- Does the hyperparameters \\alpha and \\beta have to be the same in Equation (7) and Equation (12)? If yes, please elaborate on why they have to be the same? If not, then there are too many hyperparameters (e.g., four) to tune.\n\n- In Table 1 (CIFAR-10), the empirical improvements of iGAT_DVERGE vs. DVERGE and iGAT_CLDL vs. CLDL are quite marginal under AA. \n\n- Besides, I observe that in Table 1, iGAT is combined with CLDL, DVERGE, and ADP on CIFAR-10; while iGAT is combined with SoE, ADP, DVERGE on CIFAR-100. Is there any special reason for this experimental setting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for the insightful comments, and address them below.\n\n# Weaknesses\n\n* **On Weakness 1.** Both our ranking and regularization designs are motivated by Assumption 4.4 (``Acceptable Classifier'') which identifies sufficient conditions that should be satisfied by a base classifier in order to guarantee a reduced 0/1 risk after ensemble. Part two of the assumption expects the classifier to have small prediction scores on wrong classes. An effective way to achieve this is by the proposed regularization. Part one of the assumption expects the base classifiers to perform well on ambiguous pairs. To promote this, we thus distribute each training example to its best-performing base classifier. The most straightforward way to find the best base classifier is through ranking their prediction scores, and Eq. (10) is an effective way of using such ranking information while being robust to prediction errors.  There can be other ways to design the distributing rules with similar goals, but we have found our design sufficient through empirical evaluations. Our ablation study (2) in Section 6.3 supports the validity of Eq. (10) design.\n\n* **On Weakness 2.** Indeed there are four hyper-parameters in total, and we should have used different notations for the two in either Eq. (6) or Eq. (12), instead of $\\alpha$ and $\\beta$. However,  the two hyper-parameters in Eq. (6) do not need to be tuned, for which any default or recommended setting fixed in the adopted adversarial ensemble baseline can be used. We tune only the two iGAT hyper-parameters in Eq. (12).\n\n* **On Weakness 3.**  Although the improvement is marginal, it is still positive.  At this stage, it is hard to form a theory to predict for which method iGAT will perform better, but we will gather more insight on this as our research progresses further in the future.\n\n* **On Weakness 4.** We compared using the top 3 performing methods for each dataset. Encouraged by this comment and also another comment from another reviewer, we have conducted experiments to include comparisons for all the four baseline methods using the average combiner. Please see the added results in our reply to Question 2 of Reviewer cK43 (Reviewer 2)."}]

Author rebuttal:

Meta-review: This paper develops an error based theory for explaining ensemble adversarial defenses. Based on the theory, the authors propose interactive global adversarial training (iGAT) to further enhance the ensemble adversarial defense. The proposed approach obtains improvements on CIFAR10 and CIFAR100. Although some reviewers were concerned about missing evaluations on large scale datasets, all the reviewers are leaning towards accepting the paper. The authors should consider the reviewers' comments and prepare the camera-ready paper.
Decision: Accept (poster)