Resilience to Multiple Attacks via Adversarially Trained MIMO Ensembles
Authors: Ruqi Bai
David I. Inouye
Saurabh Bagchi
Venue: ICLR 2022 Submitted (2022)
Keywords: Adversarial Example
Adversarial Attack
Evasion Attack
Defense
Adversarial Training
Security in Machine Learning

Abstract: While ensemble methods have been widely used for robustness against random perturbations (\ie the average case), ensemble approaches for robustness against adversarial perturbations (\ie the worst case) have remained elusive despite multiple prior attempts. We show that ensemble methods can improve adversarial robustness to multiple attacks if the ensemble is \emph{adversarially diverse}, which is defined by two properties: 1) the sub-models are adversarially robust themselves and yet 2) adversarial attacks do not transfer easily between sub-models. While at first glance, creating such an ensemble would seem computationally expensive, we demonstrate that an adversarially diverse ensemble can be trained with minimal computational overhead via a Multiple-Input Multiple-Output (MIMO) model. Specifically, we propose to train a MIMO model with adversarial training ({\emph{MAT}}), where each sub-model can be trained on a different attack type. When computing gradients for generating adversarial examples during training, we use the gradient with respect to the ensemble objective. This has a two-fold benefit: 1) it only requires 1 backward pass and 2) the cross-gradient information between the models promotes robustness against transferable attacks. We empirically demonstrate that {\emph{MAT}} produces an ensemble of models that is adversarially diverse and significantly improves performance over single models or vanilla ensembles while being comparable to previous state-of-the-art methods. On MNIST, we obtain $99.5\%$ clean accuracy and ($88.6\%, 57.1\%,71.6\%$) against $(\ell_\infty, \ell_2, \ell_1)$ attacks, and on CIFAR10, we achieve $79.7\%$ clean accuracy and ($47.9\%, 61.8\%,47.6\%$) against $(\ell_\infty, \ell_2, \ell_1)$ attacks, which are comparable to previous state-of-the-art methods.

Under review conference paper at ICLR 2022
RESILIENCE TO MULTIPLE ATTACKS VIA ADVERSARIALLY TRAINED MIMO ENSEMBLES
Anonymous authors Paper under double-blind review
ABSTRACT
While ensemble methods have been widely used for robustness against random pertur- bations (i.e., the average case), ensemble approaches for robustness against adversarial perturbations (i.e., the worst case) have remained clusive despite multiple prior attempts. We show that ensemble methods improve adversarial robustness to multiple attacks if the ensemble is adversarially diverse, which is defined by two properties: 1) the sub-models are adversarially robust themselves against multiple attacks, and yet sub-models are diverse measuring by adversarial transferability. While at first glance, creating such an ensemble would seem computationally expensive, we demonstrate that an adversarially diverse ensemble can be trained with minimal computational overhead via a Multiple-Input Multiple- Output (MIMO) model. Specifically, we propose to train MIMO model with adversarial training (MA7), where each sub-model can be trained on different attack type. When computing gradients for generating adversarial examples during training, we use the gradient with respect to the ensemble objective. The benefits of this scheme are twofold: 1) it only requires backward pass and 2) the cross- goss-ggrddeet information between the models promotes robustness against transferable attacks. We empirically demonstrate that MAT produces an ensemble of models that adversarially diverse and significantly improves performance over single models vanilla ensembles while being comparable to previous state-of-the-ar methods. On MNIST, we obtain 99. 99.5% clean accu- racy and (82.3%, 57.1%, 71.6%) against (Loo, l2 attacks, and on CIFAR1O, we achieve 79.7% 7% clean accuracy and (47.9%, 61.8%, 47.6%) against (Loo, l2, &1) attacks, which are comparable to previous state-of-thee thrttrr methods.
INTRODUCTION
The arms race between attacks and defenses against DNNs has created diversity of attack types. These include black-box vs white-box attacks and gradient-based vs gradient-free attacks. Ideally, adversarially robust models would be robust against all types of attack. Many defenses have been developed protect DNNs from adversarial attacks. For instance, gradient masking (21), input denoising [33]I5], and hidden layer randomization [6]. are three techniques that in their time were heralded as strong defense techniques. show that all of the above methods fail against adaptive adversaries, i.e., perturbations crafted with full knowledge of the defense measure, Among all attempts to make the deep model resilient, Adversarial Training (AT) 17 has stood the test of time and is regarded as the gold standard for its resilience white-box adversaries 18]. AT trains the deep models with adversarial examples instead of with benign training examples. Due to its empirical robustness, this method remains state-of-the art against a wide range of attacks. However, AT suffers from two fundamental drawbacks. First, it is computationally expensive, specifically the method to generate effective AEs. Second, the choice of attack type used during training significantly affects the type of robustness that is observed. For example, previous work [27 [2] shows that the model adversarial trained using the lo attack is robust against é and &g PGD attacks but vulnerable
However, [1] 3 28

---

Under review as conference paper at ICLR 2022
against gradient-free attacks. This phenomenon reveals that the robustness of adversarial training implicitly masks out the gradient information for & and lz attacks. Previous works 227 [8] try to revise adversarial training by training the model using the worst case of PGD adversarial examples among different attack metrics and achieve better results. MSD [8]] revises the Tramèr and Boneh's method [27] by calculating L,, f2, fox attacks for each step of PGD iteration and updates with the one that achieves the largest loss. This method achieved better performance against f1, l2 foo attacks. However, though MSD is faster than [27], it still requires multiple forward passes for each step of the PGD attack, which makes it more expensive to train comparing to the standard AT.
As ensemble methods have proven to be effective in robustness to random perturbations (i.e.. the average case), one natural idea to improve adversarial robustness is to exploit ensemble methods (or model averaging). Ensemble methods combine the predictions of sub-models based on some vote functions (like averaging or majority vote). Previous attempts of using ensemble methods for adversarial robustness have been shown be vulnerable under strong adversaries [5 19 23
29 [O]. They mainly failed for two reasons [34]. First, if the ensemble diverse but the models are weak (i.e.. not adversarially robust), an adaptive attack only needs to find a single AE that lies in the intersection of the AE space of the sub-models (i.e., a single worst case), which is easy [TO]. Intuitively, this finds a single perturbation that is a "common mistake" across sub-models. Second, adversarial examples tends to transfer between similar models, i.e., the adversarial perturbation generated from one sub-model may easily mislead another sub-model especially when they share the similar training set architecture [20]: and the transferability of attacks remains true even if the sub-models are adversarially robust themselves (see vanilla ensemble models in 4.2 of the experiments). Thus, neither diversity of the models nor adversarial robustness of each of the sub-models by themselves improve adversarial robustness.
To overcome these two prior failures of ensemble methods, we propose to encourage adversarial diversity which is defined to have two key features simultaneously: 1) adversarial robustness of each sub-model and 2) robustness against transfer attacks between sub-models. Additionally, we would like to do this with minimal computational overhead because AT is already very computationally expensive. Our approach is inspired by recent work called multi-input-muttiouueee simultaneously to form a diverse ensemble, and crucially the training time of the ensemble is almost the same as that of a single model. In this work, we show that by combining MIMO configuration with AT, it possible to attain robustness against multiple attack types with an ad adversarially diverse ensemble, i.e., we can achieve adversarial diversity. The results in 4 shows that by sharing model parameters, the sub-models act as an implicit way of bringing adversarial diversity while also remaining robust. Then in $ $4.22 we present that our MAT adversarially trained with each sub-model bestowing resilience to a particular attack metric leads to robustness against multiple types of perturbations. We achieve robustness accuracy (82.3%, 57.1%, 71.6%) against (Ca., l2. (1) attack with epsilon (10, 2, 0.3) on MNIST. On CIFARIO 10 dataset, our model achieves robustness accuracy (47.9%, 61.8%, 47.6%) against (41, (x)) attack with radius (12, 0.5. 0.03). Finally, in $ we provide an alternative application that one can utilize our MIMO+AT (MA7) by training the MAT with all sub-models trained with the same attack to strengthen previous AT methods to achieve better adversarial robustness without loss of clean accuracy. We provide three examples in this section as Ex AT, &, AT, and MSD. We summarize our contributions as follows:
(MIMO) training (8) MIMO can train M sub-models
1. We shed light on why prior ensemble methods may fail and hypothesize that ensemble methods can improve adversarial robustness if they are adversarially diverse, which is defined by two properties: 1) the sub-models are adversarially robust themselves and yet 2) adversarial attacks do not transfer easily between sub-models. 2. We propose use MIMO AT (MAT) to train such an ensemble in a computationally efficient way and discuss how the cross- gross-gradient information between the sub-models during training can increase the robustness to transfer attacks between models. 3. We empirically demonstrate that MAT is adversarially diverse and improves significantly over AT models and vanilla ensembles while being comparable or better than prior state-of- of-therrrt methods for multi-attack robustness.
2

---

Under review conference paper at ICLR 2022
2 PRELIMINARY
In this section, review some key concepts related to adversarial attacks, adversarial training (AT), and MIMO structure, all the notations defined here will be used throughout the paper.
PGD Adversarial Attacks. PGD attacks seek point x' in neighborhood of the benign example z with perturbation 6, which maximizes the loss L. x' is generated as: x' = I + arg m (( (I+ (), 6),"") where denotes the attack radius, fo is the DNN with parameter 0,, y is the true label. Denote the perturbation 5 = x' - x. A general PGD attack adopts gradient-type algorithm to update as
g(t+1) = =IB (e) (500 + a Vp (Vo((o(t)))")) with V2(0) = arg max V T
(1)
where p is the attack metric (usually lp norm), a is the step size, 11 is the operator projecting the gradient onto the Lp ball with radius e (denoted by B) -1, (e)). V, is the operator mapping the gradient to the steepest descent direction
the Ep ball with radius L. Note that V, is fundamentally different operator than 11, and in the case of the = oo, Vo reduces to the sign of the gradient (as in the FGSM attack). Intuitively, PGD adversarial attack uses iterate with fixed step size along the steepest direction Vp (V (.f(fo()), FGSM 77, BIM 13 and can all be treated as variants of the PGD Attack.
Adversarial Training, AT is regarded as one of the most practical yet effective methods for defending against adversarial attacks. This has stood the "test of time" through multiple iterations of proposed defenses and attacks that discover vulnerabilities in them. AT seeks to build a robust model against adversarial attacks by producing adversarial examples and injecting them into training data by solving saddle point optimization problem
0" arg min
Σ max C (fo(I, + +.).. (:EBBI1,(()
(2)
0
Here denote the i-th batch number of training set (I,,, y,). The inner maximization seeks to find the adversarial directions ó; given the current classifier fo, and the outer minimization tries to find the best 0 to minimize the summation of all batches' loss. PGD AT is widely accepted as a standard approximation of inner maximization methods since it is a universal first-order attack algorithm (17). Though it roughly needs K + 1 times more computation compared to standard training. PGD-AT remains the de facto standard for AT since walks the balance between strength of attack and computational cost of generating adversarial examples. The drawback of AT is that solving the inner maximization requires a pre-defined attack metric p. which only contributes robustness under lp attacks. (27]. In this paper, we show how the adversarial robustness can be achieved through AT but crucially, one can generalize the defense to multiple attack types and achieve this at about the exact cost AT.
MIMO Ensemble. Ensemble methods have been widely used to improve a model's calibration and robustness over out-of-distributtin samples by averaging over multiple neural network predictions 32) However, ensemble over M deep models (sub-models) is usually M times more expensive from computational and a memory standpoint. Furthermore, to improve uncertainty estimation, sub-models often sacrifice individual performance achieve diversified predictions.
To address the computational drawbacks of vanilla ensembles, g proposed an ensemble structure called MIMO Ensemble. It showed surprising result: the benefits of using multiple predictions can be achieved 'for free' under a single model's forward pass. In particular, they showed that, using a multi-input multi- output (MIMO) configuration, one can utilize a single model's capacity to train multiple sub-models that independently learn the task at hand. Since all the layers are shared except for the input layer and the output layer, this architecture uses only about 1 1% more parameters. Training MIMO model has almost
3

---

Under review conference paper at ICLR 2022
the same time complexity as training standard model while keeping the sub-models diverse without any special regularization and parameter tuning. During training, MIMO uses gradient method to solve following optimization problem for M sub-models. M
minimize
ΣEO m=1
C(fo(er)).... my Fi,m),
where ( are i.i.d sampled from the training set," With Ti,m denoting the input of the i-th batch for the m-th subnetwork, Vt Vi, represent the outputs from the model and the ground truth respectively, and C is the cross entropy loss. Notice that the overall objective the summation of each sub-model's loss. The separation structure promotes the sub-models' prediction to be less correlated because the output yj is independent to zi when * j. During testing, the same inputs are copied and fed to all the M sub-models, then MIMO decides the prediction by averaging yi.m.
3 OUR APPROACH: MAT
Given the weaknesses of prior ensemble methods, we seek an algorithm that produces an adversorially diverse ensemble, where the sub-models are robust to the attacks transferred from other sub-models so that the ensemble can have better robustness comparing to single sub-model. To achieve this, we propose to combine MIMO and AT, an approach we call MAT.
There are two difficulties to overcome in adversarially training an ensemble model: 1) For the outer minimiza- tion, how should one choose the loss function to update the sub-model parameters 0? To be specific, choosing a single sub-model loss or the ensemble loss? 2) For the inner maximization, how should the adversarial examples x', ................................ 'tMs be created during training? To solve (1), [8] demonstrates that taking the gradient with respect to the ensemble loss contributes to higher clean accuracy and is more computationally efficient. Therefore, we choose to update parameter 9 using SGD, where the gradient is taken with respect the ensemble loss instead of the loss of each sub-model. To tackle (2), one natural choice is taking gradient with respect each sub-model's loss in standard AT on each sub-model. However, we show in Sec. 3.2 that taking gradients with respect to the ensemble loss produces sub-models that re more robust to transfer attacks and, furthermore, it is more computationally efficient. Therefore, in the inner maximization, we generate adversarial examples by algorithms that compute gradients with respect to the ensemble loss instead of each sub-model's loss. We explore these two choices in more detail in the next two sections.
3.1 NAIVE MIMO ADVERSARIAL TRAINING
The foundation of MAT' is MIMO adversarial training method. The goal is to train an ensemble that is adversarially diverse. To encourage diversity, we use different attack types for each sub-model. We adopt the common choice in the literature as three PGD attacks E,, (2, and fx to instantiate our network. We aim to solve the following problem, 1r
min ΣΣ m=1 EL. Ipm (Em)
max
C(fo(re, + 3L., + dr,M) vi,m)
(4)
where for each batch i, and each sub-model m, the inner maxifiization seeks the worst-case adversarial perturbation 6t,m E Blp- (em) under the current classifier fe. Notice that dt might be different for different sub-model m. As a naive approach, we use K K-step PGD to update of.,m as the following
((++1) II (sm))
(o/t) am -pm (Ve., Clyh. F))
(5)
Notice the difference between equation [ and equation [ for differentt m. equation 5 allows for different
projections B1--1lpp stepsize O'm well as steep descent direction V/ Vpm. which results potential robustness
over diverse attacks; however, equation [ can only be seen as special case of equation 5 with m = 1, therefore
it lacks multiple robustness after the training.
4

---

Under review conference paper at ICLR 2022
3.2 MIMO ADVERSARIAL TRAINING (MAT)
While the naive MIMO adversarial training may produce models that adversarially robust by themselves, it does not prevent adversarial examples that would transfer across multiple sub-models and, additionally, it computationally expensive since it requires M backward passes for each ó computation. Therefore, to further improve the adversarial diversity of MAT, and also reduce the computational burden, we propose to replace equation by the following
(1+1) 8,m
IT 1lm (Em)
amm Vpm
ma,m Σ (lukg:
(6)
where for each tuple (i, m), we used the approximate gradient V a..m Σe= C(yb,j Vi,y) instead of the true gradient V va flyh W. where the difference between the is defined the cross gradient
cross gradient (i, m)
V Zi.m Σ C(Vl,, jifm
V Σ (iss j=1
V Pxim Ciy.. Vo
We include the pseudo-code MAT in appendix Sec. A.I, where the cross loss related to equation allows the exchange of information between the networks during the attack stage which may help in reducing the
transferability of Adversarial examples. also plays an essential role in reducing the computation burden as well as improving the adversarial diversity.
To obtain all the perturbations 6i,m, m = 1, .......,. M using the true gradient update equation [5] we need M times backward propagation (time complexity analysis is in the appendix (B). However, we only need backward propagation when we adopt the approximate gradient
sub-model
sub sub-madel
step equation [6. Therefore equation G reduces the computational
burden dramatically.
Next, we explore two properties of the cross gradient which con- tribute to adversarial diversity: 1) Reduced adversarial examples transferability between sub-models: In Section we showed that minimization weakens the correlation between the sub-models due MIMO structure. However, for problem equation [ there's 's still possibility that the adversarial perturbation 6,,m could flip the output yj of the j-th sub-model. Due to minimax structure of the problem equation [5] the flipping phenomena can effectively reduced by in- troducing the cross loss during maximization training process, where the cross gradient (i, m) describes the change of cross loss Σjtm L. w.r. the change of xı,m locally. Therefore, introducing cross loss can improve the robustness of MAT against different attacks. 2) Cross-gradients become small but non-zero during training: This can be validated by Fig. [ where the cross gradients become very small during the training but the final cross gradient does not become exactly 0, showing that there is still weak correlation between sub-models.
2
8
1.0
12
14
epachs
Figure 1: Training: validating the cross gradients norm approaching to zero: the number of epochs versus the ratio between fg-norm of the cross gradient term and the fn-norm of the true gradient.
4 EVALUATION
4. we demonstrate that the sub-models trained by our MAT are more robust to adversarial examples generated from other sub-models (reduced adversarial transferability). Furthermore, show that each sub-model robust against multiple adversarial examples. In $ 4.22 we present experimental results on MAT versus sfateoofthe-ar of-t
defenses for variety of attacks.
5

---

Under review conference paper at ICLR 2022
4.1 DIVERSITY AND ROBUSTNESS
Diversity of adversarial transferability among sub-models A group of classifiers can benefit from the ensemble if and only if each member diverse enough. There are several different measures trying to quantify the property 122 B5). In this section, observe how the ensemble of sub-models perform against diverse adversarial attacks. To explore the diversity among sub-models of manila--nnsemle (resp. MA7), the sub-models 1, 2, and have been trained against respectively é,, f2, and &oo PGD attacks; then, evaluate vanilla-ensemble (resp. MA7) against -ddversaaaial attacks and report their clean accuracy (the results corresponds to { and lg are similar and summarized in appendix [ We study two kinds of aadversrrria examples. The first kind is -adersarall attack generated from sub-model 1, from which we can study the transferability of fox -dveerrraiall attack generated from sub-model by counting the successful attacks sub-model 2, sub-model 3, and vanilla-ensemble (resp. MAT). In particular, we denote
No _ #ssuceessful attacks to sub-model i out of 1000 -ttacks generated from sub-model i}, Ni - #suuccessful attacks to sub-model j out of 1000 -attacks generated from sub-model i).
Then, we define the transferability of adversarial examples from jo j as raj - Nij/NN. Furthermore, we use -dveeraralllal examples generated from vanilla-ensemble (resp. MAT) attack each sub-model and their vanilla-ensemble (resp. MAT) to evaluate the overall performance of vanilla-ensemble (resp. MA7). The results are summarized in second row of Table. (resp. Table. 2 for vanilla-ensemble (resp. MAT).
Table 1: Manilla-Ensemble diversity testing: successful loo adversarall attack. Each column shows the misclassification samples out of 1000. The first (resp. second) row corresponds to Loo adversarial attacks generated from model (resp. vanilla-ensemble). The last row relates to clean accuracy with no attack.
Sub-model 1
Sub-model 2
Sub-model 3
Sub-model 1
39
78(200%)
120(307.7%)
60(153.8%
VanillaeEnsemble
45
994
998
960
No Attack
14
9
12
12
Table MAT' diversity testing: successful lo adversarial attack. Each column shows the misclassification
samples out of 1000. The first (resp. second) row corresponds to lo adversarial attacks generated from sub-model (resp. MAT). The last row relates to clean accuracy with no attack.
Sub-model
Sub-model
Sub-model 3
MAT 64(41.0%)
Sub-model I
156
37(23.7%)
52(33.3%
MAT
114
548
106
128
No Attack
9
7
5
Comparing Table. and Table. 2 we observe the following phenomena. First, the transferability ratios of MAT
are F12 23.7%% r13 33.3%. which are much lower than r12 200%, 113 153.8% of vanilla ensemble.
This indicates the sub-models of MAT are more robust to -ataakkk, and attacks are easier to transfer between the sub-models of vanilla ensemble. Therefore, we con- clude that the sub-models of MAT are diverse enough to reduce the effective for adversarial transferability. Second, the performance of vanilla-ensemble is close to the worst sub-model performance. This is indicated by noticing the number of successful& lo -ataacks for vanilla-ensemble is 960, which is close to max (45, 994, 998} 998. This
Table 3: Testing: Robustness accuracy of MAT and its sub model: each row corresponds to dif- ferent attacks; and each column depicts the ro- bustness accuracy of MAT and its sub model.
MAT
sub
sub 2
sub
C. PGD
89.2%
85.0%
33. 1%
87.1%
6, PGD
66.6%
63.7%
69.0%
61.0%
means that aanila--nsembll does not provide extra ro- bustness compared to its sub-models However, the successful -ttack against MAT is only 128, which
E PGD
71.9%
72.0%
71.4%
67.8%
6

---

Under review conference paper at ICLR 2022
only takes up 23.4% of the worst sub-model performance max(114, 548, 1063 548, This suggests that the MAT clearly improves the robustness accuracy compared to that of sub-models.
Robustness of sub-models We empirically show that the sub-models of MAT are robust against S lz, & attacks. During testing, the same inputs are copied and fed to all the M sub-models, then MAT decides the
prediction by averaging Vi.m- The results are summarized Table 3 We observe that using MAT, each sub-model is relatively robust against all adversarial examples. The only weak sub-model is sub-model 2, which shows vulnerability against foo PGD attack (33.1)), however, MAT still performs well against &oo attacks (89.2%) because it averages the output of the three sub-models.
4.2 MAIN RESULT
In this section, we present two main results of MAT on MNIST and CIFAR-10 against various types of attacks and compare with different baselines. MAT generally boosts the previous methods without extra cost.
Main result on MNIST Our main results on MNIST are summarized in Figure 2 and Table 4 Figure 2 plots varying number of attack steps versus robust accuracy, and the varying radius using foo (resp. & and (1) versus robust accuracy. We can capture the following phenomena: (1) both of MAT and MAT+MSD exhibit generalization and robustness for a varying number of attack steps; (2) MAT outperforms MSD and MAT+MSD depending against & PGD attack; and when it comes P2DPGD attack and L PPD attack, the performance of MAT+MSD surpass the other two methods on MNIST for varying number of attacks; (3) MAT outperform MSD on fax PGD attack; 2) on l-- (resp. 41-) PGD attack, MAT+MSD surpass the other two method. Table. shows under the attack radius choice 0.3 (resp. 2 and 10) for &oo (resp. l and i
100
100
100
80
90
fourmmttmmsd
80
med
70
70
150
300
6 PSD
RGD
100
100
100
rourimat mmd
80
Inurtmee red
80
60
40
40
40
20
20
20
PGD
Figure 2: Robust accuracy curves of MAT +MSD, MAT, and MSD on MNIST under PGD foo (resp. lg, and (1) attack. First row: Robust accuracy against the number of steps of attack with radius 0.3 (resp. 2, and 10). Second row: Robust accuracy against varying attack radius using foo (resp. l2 and ET) PGD attack; the vertical dashed lines denotes the common radius choice in prior work evaluations as 0.3 (resp. 2 and 10) for Lo (resp. & and l1).
the robustness accuracy comparison under different attack metrics between MAT and AT, vanilla ensemble MSD, standard training is listed for clean accuracy comparisor All foo (resp. All la and All f1) attack is to defend against loo based attacks, i.e., the union of PGD Lo attack, HSJ attack, Auto Attack (resp. PGD l attack, C&W attack, Boundary attack for &g- based attacks; and PGD & attack, Salt & Pepper attack for 4
'All the hyperparameter including the step size of the PGD attack, attacks radius and termination max iteration are chosen the best tuning for MSD [88 in order to present results on same base line (which may not be the best for MAT), Please check Sec. C for detailed information about hyperparameters choice.
7

---

Under review conference paper at ICLR 2022
based attacks). This reveals that MAT could improve the overall model's robustness. The vanilla ensemble shows its vulnerability to all three kinds of adversarial attacks though it combines a three times larger model and requires three times longer training. MAT achieves 82.3% against the union of lo attacks on the MNIST dataset, which surpasses the previous state-of-the-an by 26.0% percent. For defending against the union of adversarial attacks, our method attains 51.7% robustness accuracy.
Table 4: Testing on MNIST: each row corresponds different attacks; each column compares the robustness accuracy of MAT with different baseline models.
standard training
8oc
Vanilla Ensemble
MSD
MAT (ours)
clean accuracy
99.5%
99.3%
98.6%
98.9%
99.0%
98.5%
99.5%
PGD Z, uttack
0.0%
96.0%
0.8%
0.0%
4.6%
62.2%
89.2%
HSJ attack
12.3%
98.0%
44.2%
12.1%
35.7%
87.4%
91.2%
Auto Attack
0.1%
94.9%
0.5%
0.2%
0.1%
57.0%
82.3%
All Eo attacks
0.0%
94.9%
0.4%
0%
0.0%
56.3%
82.3%
PGD attack
0.0%
78.1%
76.8%
11.4%
57.2%
72.0%
66.6%
C&W attack
0.0%
39.7%
76.8%
12.6%
60.1%
69.8%
59.4%
Boundary attack
0.7%
17.7%
78.2%
21.2%
46.9%
70.6%
77.8%
All f2 attacks
0.0%
8.7%
74.4%
9.8%
44.8%
67.3%
57.1%
PGD 2 attack
0.0%
53.3%
79.0%
19.9%
63.2%
63.7%
71.9%
Salt & Pepper attack
74.3%
64.9%
96. 26.2%
72.5%
68.4%
84.5%
96.0%
All & attacks
0.0%
53.3%
78. 78.2%
19.9%
60.1%
62.7%
71.6%
All attacks
0.0%
6.5%
0.8%
0.0%
1.5%
54.7%
51.7%
80
louttmaterso
60
50
5C
50
80
100
150
200
50
100
150
200
50
100
150
200
2. PGD ettack
4y PGD attack
PGD
80
80
80
80
69
40
20
20
20
0.000
3,023
0.050 0,0/5
0.100
0.0
3.5
10
1.5
2.0
10
20
30
40
7. PGD atuack
PGD ottack
6. PED attack
Figure 3: Robust accuracy curves of MAT +MSD, MAT, and MSD on CIFAR-10 under PGD E (resp. lg, and 41) attack. First row: Robust accuracy versus the number of attack steps 0.03 (resp. 0.5, and 12). Second row: Robust accuracy versus the varying radius using &o (resp. l and & PGD attack; the vertical dashed lines denote the common radius choice in testing as 0.03 (resp. 0.5 and 12) for foc (resp. Ea and (1).
Main result on CIFAR-10. Our main results on CIFAR-10 are summarized in Figure 3 and Table 5 We observe the following: (1) the performance of MAT comparable w.r.t MSD on foo and E PGD attack with varying steps and varying radius; (2) MSD exhibits better testing accuracy than the other two methods on CIFAR- for a varying number of 2 PPD attacks with radius r = 0.5, however, MAT outperforms MSD when the attack radius increases (greater than the common choice 0.5).
Table 5 shows that MAT achieve strong robustness against multiple kinds of attack. Crucially, the vanilla ensemble fails against the for attacks while MAT's is competitive against the state-of tff-thh-rrt MSD, while being computationally more efficient (K + forward passes for MAT versus (1 + M)K + 1 for MSD, with the same number of backward passes K 1). We include the time complexity analysis in Table and training time on each model Table 7 please see appendix B and appendix E for details.
8

---

Under review conference paper at ICLR 2022
Table 5: Testing on CIFAR- 10: each row corresponds to different attacks; each column compares the robustness accuracy of MAT with different baseline models.
standard training
800
Vanilla Ensemble
MSD
MAT (ours)
clean accuracy
95.3%
84.0%
89.3%
33.6%
84.0%
81.1%
78.9%
PGD P.u, attack
0.0%
48.8%
0.0%
0.0%
0.0%%
48.0%
47.9%
HSJ attack
7.5%
79.3%
69.0%
12.9%
67.3%
72.4%
74.7%
Auto Attack
0.1%
78.3%
0.0%
0.0%
0.0%
57.0%
56.4%
All foo attacks
0.0%
46.7%
0.0%
0.0%
0.0%
48.0%
47.9%
PGD attack
4.3%
61.1%
79.3%
0.0%
61.2%
68.2%
64.3%
C&W attack
3.4%
60.6%
79.3%
0.0%
57.5%
64.7%
63.3%
Boundary attack
25.2%
19.4%
82.1%
1.6%
23.2%
69.2%
64.2%
All la attacks
1.9%
18.2%
73.3%
0.0%
19.1%
64.3%
61.8%
PGD attack
13.1%
18.2%
34.0%
0.0%
14.2%
53.4%
47.6%
Salt & Pepper attack
53.5%
65.7%
75.4%
61.3%
65.6%
73.9%
75.0%
All E attacks
12.8%
18.0%
33.3%
0.0%
14.2%
33.45
47.6%
All attacks
0.0%
16.2%
0.0%
0.0%
0.0%
47.0%
46.4%
5 RELATED WORK
Adversarial Defense. Many defenses have been proposed, with most of them being broken by adaptive attacks I 3 28]. Adversarial Training [17]. though expensive in computation, has emerged as the best
practical way to create models resilient to white-box attacks. A slew of literature has developed variants of AT differing in the way samples are created (6). the way the models are iterated upon [24] 30), and the
way different attacks combined 27 [8] Our ensemble method orthogonal to these approaches and therefore can be combined with these, with varying levels of additional design and engineering effort, such as design the right combining function across the sub-models.
Adversarial Defense against multiple types of perturbation. multiple perturbations by introducing multiple steepest direction PGD attacks during training; their method ensembles three different attacks into one, called multi steepest direction. Our method instead seeks an ensemble over models. (66 proposed a provable defense against all lp norm with robust radius 5 times smaller than state-of offte--f-theaarr the-art heuristic defense methods [18). This technique has the benefit that it uses transfer learning and therefore does not need separate training for each kind of attack. This can be used seamlessly with MAT by starting each sub-network after transfer learning from the another sub-network. [14] trains DNNs against imperceptible adversarial examples, which are approximated using DNNs. While it shows robustness against wide range of attacks, its performance against specific attacks is limited. Our approach MAT outperforms prior solutions this space, either as we have proven empirically for the latest approach [8] through the author-publishee numbers on the same dataset.
18 tries to build a robust model against
6 CONCLUSION AND DISCUSSION
In this paper, present an adversarially diverse ensemble method called MAT in which each sub-model achieves both robustness and diversity by developing the idea of learning an ensemble via adversarial training (AT) and adapt the structure of recently discovered MIMO architecture. We require the same number of forward and backward passes as AT and yet achieve generalized robustness against multiple attack types. drawback of our scheme that if the network size constrained, it may not have the capacity to accommodate multiple sub-models. Another drawback that is common to most AT methods is that the AE perturbation hyperparameters must be manually selected (we simply use the parameters from MSD [88). Methods for automatically selecting the best hyperparameters using zero-th order optimization would broaden the applicability of our approach.
9

---

Under review conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We have publicly released all the code, the datasets and models used, along with the implementation of the attacks. To help with the reproducibility of the experiments, we make available all of our code, the trained models, the training hyperparameter values, and the raw results from testing. We point to the exact attacks that we have used (which are from existing codebases) in our evaluation. These are all to be found the anonymized Github link https: / /gggttb.bbb..../gggg/g/ggo...... com/ anonymous- lab-m1 /oalmmmmmo-ioe adv/ Code and pre-trained models for reproducing our experiments have been uploaded to the anonymized repository attpps:/giithh com/ anonymous lab ml/mimo- adv/
REFERENCES
[1] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Muchine Learning, volume of Proceedings of Machine Learning Research, pages 274-283. PMLR, 10-15 Jul 2018.
[2] W. Brendel, J. Rauber, and M. Bethge. Decision- adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representations, 2018.
[3] Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint 1902 2019.
[4] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 39-57. IEEE, 2017.
[5] Jianbo Chen, Michael I Jordan, and Martin J Wainwright. HopSkipllmpAAtackk query-efficient decision-based adversarial attack. 2020 IEEE Symposium on Security and Privacy (SP). IEEE, 2020.
[6] Jinhao Dong and Tong Lin. Margingan: Adversarial training in semi-supervised learning. Advances in neural information processing systems, 32:10440-10449, 2019.
[7] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.
Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015, 2015.
[8] Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Laksh- minarayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In International Conference Learning Representations, 2021.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[10] Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial example defense: Ensembles of weak defenses are not strong. In 11th USENIX Workshop on Offensive Technologies (WOOT 17), Vancouver, BC, August 2017. USENIX Association.
[11] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 0rXiv:1412.666 2014.
[12] Ludmila Kuncheva and Christopher Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine learning, 51(2):181-207, 2003.
10

---

Under review conference paper at ICLR 2022
[13] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world, 2017.
[14] Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In International Conference on Learning Representations, 2021.
[15] Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and Jun Zhu. Defense against ad- versarial attacks using high-level representation guided denoiser. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1778-1787, 2018.
[16] Xuanqing Liu, Minhao Cheng, Huan Zhang, and Cho-Jui Hsieh. Towards robust neural networks via random self-ensemble. In Proceedings of the European Conference on Computer Vision (ECCV), pages 369-385, 2018.
[17] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[18] Pratyush Maini, Eric Wong, and Zico Kolter. Adversarial robustness against the union of multiple perturbation models. In Proceedings of the 37th International Conference Machine Learning, pages 6640-6650, 2020.
[19] Tianyu Pang, Kun Xu, Chao Du, Ning Chen, and Jun Zhu. Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning, pages 4970-4979. PMLR, 2019.
[20] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07227 2016.
[21] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. 2017.
[22] Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. In International Conference on Learning Representations, 2019.
[23] Sanchari Sen, Balaraman Ravindran, and Anand Raghunathan. Empir: Ensembles of mixed precision deep networks for increased robustness against adversarial attacks. arXiv preprint arXiv:2004. 10162, 2020.
[24] Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances Neural Information Processing Systems, 2019.
[25] Leslic N Smith.
disciplined approach to neural network size, momentum, and weight decay. arXiv preprint arXiv:1803.09822 2018.
Part I-learning rate, batch
[26] David Stutz, Matthias Hein, and Bernt Schicle. Confidenco-caliiratee adversarial training: Generalizing to unseen attacks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pages 9155 9166. PMLR, 13-18 18 Jul 2020.
[27] Florian Tramer and Dan Boneh. Adversarial training and robustness for multiple perturbations. In Advances in Neural Information Processing Systems, volume 32, 2019.
11

---

Under review conference paper at ICLR 2022
[28] Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks adversarial example defenses. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1633-1645. Curran Associates, Inc., 2020.
[29] Gunjan Verma and Ananthram Swami. Error correcting output codes improve probability estimation and adversarial robustness of deep neural networks. Advances Neural Information Processing Systems, 2019.
[30] Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the conver- gence and robustness of adversarial training. In International Conference on Machine Learning, pages 6586-6595, PMLR, 2019.
[31] Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. International Conference on Learning Representations, 2020.
[32] Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. Advances in Neural Information Processing Systems, 2020.
[33] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. Proceedings 2018 Network and Distributed System Security Symposium, 2018.
[34] Zhuolin Yang, Linyi Li, Xiaojun Xu, Shiliang Zuo, Qian Chen, Benjamin Rubinstein, Ce Zhang, and Bo Li. Trs: Transferability reduced ensemble via encouraging gradient diversity and model smoothness. arXiv preprint arXiv:2104.000 00671, 2021.
[35] Zhi-Hua Zhou, Ensemble methods: foundations and algorithms. Chapman and Hal//CRC, 2019.
12

---

Under review conference paper at ICLR 2022
A
CODE AND PSEUDO CODE OF MAT CODE
A.1
Code for reproducing our experiments have been uploaded the anonymized repository: https:// github. com anonymous lab waa-mlll/ee mmlmmimo-adv/
A.2
PESUDO CODE
Algorithm MIMO adversarial training with approximate gradient (inner maximization)
input :network Sa. data_size N, batch_size B, data x, label y² pgd attack steps: K, num_of sfbseetet.... M
initialization;
iterate over all training examples
for = to ceil(N/B) do
Tade - (e v[));
0 - t-ninmmizel ze fo, Fados yii));
end
def maximimmae (fo. y));
adv x; for k = 1 to K do
11 adv (advi, ... adom)
V1..... yM = fa(advi,... adom);
loss = 0;
for m = 1 to do
loss loss CELoss(ym: vm)
end
grad V Vadoloss;
// approximate gradient step
for 1 M do adum t advm 1I (em) (am Vm (grad));
end end
return adv; def minimizze (fo, 2, y")"
loss
V1. YM fo(x1, fM)) for m = 1 to do
loss max(loss, CELOSS (yms Von ));
end grad- Volloss);
0 < 8 grad;
return
TIME COMPLEXITY OF DIFFERENT AT METHODS
Compared standard training (i.e., without ensemble), vanilla AT using K-step PGD attack requires K times more computation, due to the forward and backward passes individually for generating each AE plus the outer minimization. Furthermore, if we try use M-vanilla ensemble AT to promote robustness
13

---

Under review conference paper at ICLR 2022
over multiple attacks, it will require M times more computation compared vanilla AT. This becomes infeasible in practice, e.g., for CIFAR-10 with ResNet-50, 100 epochs. It takes 40 times of training time train one single model compared to standard training. To reduce the computation, we have the insight replace the vanilla ensemble with a MIMO architecture and to train for all the M attack types within the same network, each within its own subnetwork. This allows us to calculate all subnetworks' outputs Ue, i 1...., M in one forward pass instead of M passes, i.e., (V1..... MM) fe( TMM, instead (z1). foy (M )) in case of the M vanilla ensemble. This approximation works
of (yi...... yM) = (fe,(z1). because can fit M subnetworks within the original network, i.e., without needing to expand the size of the original network. This is because of the residual capacity in even reasonably sized networks (like ResNet-50 in our case) is sufficient.
Thus, M-MIMO AT reduces the number of forward passes to similar to that of vanilla AT training. However, for each subnetwork, one iteration of PGD attack still needs one backward pass, which leaves the computation of backward pass almost as high as that of na've ensemble with M networks. The computation cost per iteration of different methods summarized in Table 6
Table 6: Computation comparison between different methods (for each batch of data, ie., each iteration of the outer minimization)
Methods
Forward passes
Backwards passes
Standard Training
1
Standard MIMO Training
Vanilla AT (K step PGD)
K K+1
K+1
M-Vanilla Ensemble AT (K step PGD)
M(K +1)
M(K +1)
Multi-perturbation AT (K (K-step PGD) 27
MK+M
MK M
M-MSD AT (K-step PGD)
18
(1+ (+M +1
41
K+1
M-MIMO AT without Approx. AT(K-step PGD)
K+1
MK +1
(Ours) MAT: M-MIMO Approx. AT (K-step PGD)
K +1
K+1
(Ours) M M-MIMO MSD (K (K-step PGD)
(1 + M)K+1 +1
K+1
EXPERIMENTS SETTING
C.I HYPERPARAMETERS FOR TRAINING THE MODELS
Models. We use ResNet18 9 for all experiments on MNIST dataset. We also reproduce MSD results using the same neural network in their original paper. For CIFAR1O, use ResNet50.
Optimizers. For MNIST dataset, we use Adam optimizer with momentum 0.9, and weight decay 5 10-
II
For CIFARIO dataset, we use SGD optimizier
Step size. We use the step size scheduler follows 25 For MNIST, the step size linearly increase from 0 to 10 10-3 over the first 40% epochs, and down to over the last 60% epochs. For CIFARIO, the step size lincarly the first 40% epochs, and down 0.005 over the next 40% epochs, and finally back down to in the last 20% epochs. The overall number of epochs is 25 (resp.75) for MAT and MAT + MSD MNIST (resp. CIFAR10) datasets.
increase from 0 to 0.1
14

---

Under review conference paper at ICLR 2022
Hyperparameters for adversarial training. For all MNIST dataset, the inner maximization is optimized by (foo l2, (1) PGD attacks with max iteration 100, radius (0.3, 2.0, 10) and step size (0.01, 0.1, 0.8) respectively. For all CIFARIO dataset, the inner maximization is optimized by (foc : l2. & PGD attacks with max iteration 50, attack radius (0.03, 0.5, 12) and step size (0.003, 0.02, 1.0) respectively.
C.2 HYPERPARAMETER FOR ADVERSARIAL ATTACKS USED FOR ROBUST TEST
For all the results, we evaluate the first 1000 test examples. To verify the performance of MAT and compare it the performance of the baseline Adversarial Training, we evaluate the performance of model against foo l2, & PGD attacks [17. Carlini Wagner 4 Ez attack,and two decision-based black-box attack: Boundary attack and HopSkipJump attack 5
For PGD attacks, to guarantee the convergence, we increase the max iteration to 200. For & PGD attack, we fixed the number of pixels perturb per iteration as 20 during test. The other hyperparameterss stay the same in the software distribution.
For other attacks, we set the hyperparameters be consistent the original works.
D THE DIFFERENCE BETWEEN THE TRUE GRADIENT AND APPROXIMATE GRADIENT AFTER THE STEEPEST DIRECTION MAPPING OPERATOR )
We claim that the cross gradient does influence the optimization, since the the small value of cross gradient may affect the gradient value after the steepest descent direction. In Figure 4 we define E, [V,(V=,, + (a))))=
where |- denotes the cardinality the Ims denotes XOR operation, and = (1, Ep calculates the fraction of entries that are different in the approximate gradient after steepest-descent projection compared to the true gradient. Notice that the XOR operator with summation calculates the overall number of different pixels the two gradients have. denotes the overall pixels the gradient has. Figure 4 shows that the difference between the approximate gradient and the true gradient after the steep ascent direction mapping, where it decreases during training. Compared to Figure I which shows the difference between the true gradient and our approximation. The y axis denotes the fraction of pixels that changed by the cross gradient term. We can see from 4 especially, Ego does not converge to 0. They term might perform an important role to bring multiple robustness for the sub-model trained against a single attacks.
0.12
subnetwork1 subnetwork2 subnetwork3
0.10
0.08
0.06
0.04
0.2
25
50 epoch
75
100
25
50
75
100
epoch
Figure 4: Approximate gradient difference caused by steepest descent operator for 6,, loo norms: epoch versus the disagreement E.
15

---

Under review conference paper at ICLR 2022
E WALL CLOCK TIME DURING TRAINING
In this section we report the wall clock time to train the different models.
Table 7: The wall clock time for training the model. All the models are training using ResNet-18 18 (resp. ResNet-50) on MNIST (resp.CIFARIO),
Standard Training
AT
MSD
MAT
Training time on MNIST Training time on CIFAR10
10min23s 40min Is
8h 8h12min
12h56min
10h02min
89h42min
141h23min
123h07min
F SUB-MODEL TRAINED WITH SAME ADVERSARIAL ATTACK
As mentioned, MAT can also be integrated with other AT techniques to strengthen the model. In this section, we empirically show the three experiments where we combine MAT' with three previous methods, too AT, lz AT, and MSD respectively. The result in Table 8 shows that MAT' achieves higher robustness accuracy in most scenarios even against the attack type that AT approach has been designed for. Notice that these models are trained with the same models and hyperparameters. As we observe, for MSD+MAT, we also bring 20.4% extra robustness against lo attacks, and 29.2% more robustness accuracy against L attacks with 0.8% extra clean accuracy. The drawback is trade off 27% robust accuracy &g attacks. This is due to the imbalance choices of the hyperparameters giving relative weight to the three attack types &, l, and loo. In this table, we show our MAT could use with sub-models trained with same metrics. The results show that if we combine the MIMO structures and loo AT, we could achieves overall higher accuracy comparing to the standard coo AT.
G TRANSFERABILITY AND SUB-MODEL ROBUSTNESS
This section shows the sub-models performance of the vanilla ensemble (resp. MA7) against fg PGD attacks in Table 9 (resp. Table 10p and against & PGD attacks in Table I1 (resp. Table 12.. Comparing Table 9 and Table IO, when the attacker generates the adversarial examples on sub-model 2, we find that the adversarial examples is casier to transfer to other sub-models comparing to our MAT. This especially is due the vulnerability of the sub-model and sub-model 3. Since the sub-models not robust, we check the second row of the vanilla-ensemble model, the number of successful fz-attacks is 437 which is much higher than 383 of our MAT. Defending against & attacks, though the sub-model of the vanilla ensemble achieves low adversarial transferability, is due toe the vulnerable of the sub-models, which empirically proves our claim that to achieve robustness against adversarial examples, both sub-model 's robustness and the adversarial diversity between sub-models are required.
model still outperforms the vanilla ensemble by 12.1%. This
16

---

Under review conference paper at ICLR 2022
Table 8: Three sub-models trained with same Adversarial Attacks MNIST: the first row shows the clean accuracy; the remaining rows corresponds to different attacks; different column compares the robustness accuracy of fo AT (resp. & AT MSD) with M()(( (resp. MAT((() MAT+MSD)), where (f) (resp. MAT((22) denotes MAT with each sub model trained using &o -attack (resp. lz-attack), and MAT+MSD denotes MAT with each sub model trained using MSD.
to AT
MAT((
f2 AT MAT((2)
clean accuracy
99.3%
99.2%
clean accuracy
98.6%
98.5%
PGD-f Attack
96.0%
96.1%
PGD-(z Attack
76.8%
76.4%
HSJ Attack
98.0%
97.8%
C W Attack
76.8%
77.2%
Auto.Attack
94.9%
95.6%
Boundary Attack
78.2%
79.0%
All lx Attacks
94.9%
95.6%
All & Attacks
74.4%
75.7%
MAT((
MAT(E2)
MSD
MAT+MSD
clean accuracy
98.5%
99.3%
PGD-& Attack
62.2%
82.4%
HSJ Attack
87.4%
97.3%
All lo Attacks
62.2%
82.4%
PGD-L2 Attack
72.0%
77.7%
C & W Attack
69.8%
62.9%
Boundary Attack
70.6%
55.0%
All & Attacks
67.3%
39.7%
PGD- Attack
63.7%
93.7%
Salt Pepper Attack
84.5%
94.9%
All & Attacks
62.7%
91.9%
All Attacks
54.7%
39.5%
MAT + MSD
17

---

Under review conference paper at ICLR 2022
Table 9: Vanilla-Ensemile diversity testing: successful fz-adversarial attack. Each column shows the misclassification samples out of 1000. The first (resp. second) row corresponds to lz adversarial attacks generated from model 2 (resp. vanilla-ensemble). The last relates to clean accuracy with attack.
Sub-model
Sub-model
Sub-model
Sub-model 2
336(145.5%)
231
298( 129.0%)
234(101.3%)
557
206
704
437
No Attack
12
12
Table 10: MAT diversity testing: successful l adversarial attack. Each column shows the misclassification samples out 1000. The first (resp. second) row corresponds to E adversarial attacks generated from sub-model (resp. MAT). The last row relates clean accuracy with attack.
Sub-model
Sub-model
Sub-model
MAT
Sub-model 2
251(76.8%)
327
260(79.5%)
272(83.2%)
MAT
324
284
363
320
No Attack
Table 11: Vanilla-Ensemble diversity testing: successful f-aadversarill attack. Each column shows the misclassification samples out of 1000. The first (resp. second) row corresponds to &o adversarial attacks generated from model 2 (resp. vanilla-ensemble). The last row relates to clean accuracy with no attack.
Sub-model
Sub-model 2
Sub-model
Vanilla-Ensemble
Sub-mu 100-mmdel 3
100(16.4%)
64(1005%%
608
121
Vanilla- Vanilla-Ensemble
Ensemble
413
188
556
383
No Attack
14
12
12
Table 12: MAT diversity testing: successful adversarial attack. Each column shows the misclassification samples out of 1000. The first (resp. second) row corresponds to & adversarial attacks generated from sub-model (resp. MAT). The last row relates clean accuracy with no attack.
Sub-model
Sub-model 2
Sub-model
MAT
Sub-model
79(45.7%
54(31.2%
173
81(46.8%
MAT
273
249
287
262
No Attack
7
7
5
18


Reviews:
[{"review_id": "ib_ti3ujmD7", "reviewer": "Reviewer_YRoM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength]\n\n1 To defend against multiple attacks, the proposal MAT requires that a) each sub-model is robust against a specific attack, and b) submodels have reduced transferability. \n\n2 Compared with vanilla AT+ensemble, the proposal MAT is computationally efficient. \n\n[Weakness]\n\n1 In my opinion, this paper's novelty is not strong, e.g., sub-models should be diverse, sub-models are adversarially robust, and MIMO training strategy. \n\n2 Sections 3.1 and 3.2 is not clearly written. E.g., what does V in equation 5 mean? \n\n3 Compared with MSD (Maini et at.), the proposal MAT has marginal improvements against multiple attacks. \n\n[Questions]\n\n1 Could authors illustrate more about Eq. 6 and Eq. 7. From the current writing, I do not understand cross gradient in detail. \n\n2 A.2 PSEUDO CODE, where does penalty of cross gradient apply?", "summary_of_the_paper": "[Summary]\nThis paper proposes an ensemble method to defend against adversarial attacks. \nTo be specific, this paper combines MIMO strategies (Havasi et al.) and adversarial training (Madry et al.). ", "summary_of_the_review": "My main concern is the novelty of this paper;\nBesides, this paper is not well written. \n\n\n\n##### Post rebuttal #### \n\nMany thanks for authors' feedback. \nI have read other reviewers' comments and corresponding feedback. \nI agree with other reviewers' evaluations such as “limited novelty“, \"weak experimental evidence\", etc. \nThus, I keep my score unchanged. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "HnCQOqh1Ow", "reviewer": "Reviewer_U6VC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n(1) The motivation behind is simple and interesting.\n\n(2) The problem of defending multiple perturbations is significant.\n\n(3) The insights given by the author are interesting for me.\n\nCons:\n\n(1) My biggest concern for this paper is the experiment. First of all, the experimental settings are ambiguous. For example, what is the perturbation budget for your experiment? What is the step size of PGD attack? Did the author clip the perturbation for the C&W attack? Why didn't the author use AutoAttack for the main experiment on CIFAR-10 and MNIST? It seems that the author missed these important experimental settings, which make the results unconvincing.\n\n(2) The author missed some studies that also aim to improve model robustness against multiple adversarial perturbations [R1]. The author should compare and discuss with them.\n\n(3) According to the experimental results in Table 6, MAT+MSD achieves comparable performance with MSD, and even shows weaker robustness on the \"All Attacks\". How did the author explain this phenomenon?\n\n(4) Again, as shown in Section 4.4, the author claimed that ''The drawback is we trade-off 27% robust accuracy on $\\ell_2$ attacks. This is due to the imbalance of choices of the hyperparameters''. This is not convincing to me, and the discussion is somewhat tricky. In other words, if the author fine-tune the hyperparameter to increase the robustness against $\\ell_2$ attacks, would the model show weak robustness against other types of attacks?\n\n(5) Moreover, I also found the ''sub 2'' model only achieves 33.1\\% on $\\ell_{\\infty}$ PGD attacks, while other sub-networks achieve 80+\\% robustness on $\\ell_{\\infty}$ PGD. Could the author provide more explanations?\n\n(6) What about the defense performance on unseen perturbations? For example, training on $\\ell_1$ and $\\ell_2$ adversarial perturbations and evaluating the robustness on $\\ell_{\\infty}$ adversarial attacks.\n\nReferences\n\n[R1] Towards Defending Multiple Adversarial Perturbations via Gated Batch Normalization, arxiv, 2020.", "summary_of_the_paper": "This paper proposes an ensemble based adversarial training strategy, which could improve the worst-case model robustness against multiple $\\ell_p$-norm adversarial perturbations. To improve the training efficiency, the author introduces the Multiple-Input Multiple-Output (MIMO) strategy. The motivations and insights provided by this paper are interesting for me.", "summary_of_the_review": "The proposed method is very interesting and the motivation behind is intuitive. However, I think the experiments are not enough to prove its effectiveness, and some settings are even problematic.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DI808Z7wgMX", "reviewer": "Reviewer_zrcs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The algorithm is intuitive and the claims are well supported by experiments and analysis.\n2. The idea of cross gradients as a metric to analyse effect of transferable perturbations due to the unique architecture of MIMO models is interesting and worth further study on its own.\n3. The paper is organized well and the ideas are presented in a coherent manner.\n\nWeaknesses:\n1. MAT generally seems to perform worse than MSD for all three threat models when compared head to head. The robust accuracy for $\\ell_2$ attacks is significantly lower when compared to both MSD and AT. The paper claims in another section that this is due to choice of hyperparameters. Could the authors clarify this?\n2. It is not clear from the text what the threat model or the attack budget is.  Are the various copies of the input image allowed individual perturbations or are the perturbations forced to be equal? \n3. The authors do not test their defense against an adaptive attack. For example, an attacker could try to find salient sub-models using loss values for each input and perturb just their inputs. This also ties into my second question regarding the clarity of the threat model.\n4. As the authors claim robustness against multiple attacks, ideally robust accuracy should be reported for the worst case attack in the combined threat model for a more clear picture. \n5. The authors don't seem to have used the cross gradient terms as a penalty during training. Is there any specific reason for that?", "summary_of_the_paper": "In this work, the authors propose a new training algorithm MAT that adversarially trains Multi Input Multi Output (MIMO) models. They show that ensemble models based on MIMO when trained adversarially , show ``adversarial diversity'' and therefore are less vulnerable to transfer attacks. They empirically demonstrate that such models are robust to a variety of $\\ell_1, \\ell_\\infty$ and $\\ell_2$ attacks and achieve better performance than other methods claiming robustness across different threat models. The authors also show computational benefits of their algorithm over vanilla ensemble training. ", "summary_of_the_review": "The authors propose an algorithm for adversarial training of MIMO models. They claim better robustness against a wide variety of threat models, and demonstrate this with experiments on CIFAR-10 and MNIST. The claims are mostly well supported, however the improvements seem marginal.The technical contribution is therefore somewhat limited. The authors also need to clearly specify the threat model that they assume for their experiments. However, their results for the cross-gradient  across the sub-models is an interesting addition and does show that their method holds promise. Reviewing all of this, I currently this paper needs some work, and therefore recommend a marginal rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8-_FDvR9PdC", "reviewer": "Reviewer_7nra", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n- This paper focuses on the important problem of robustness to adversarial attacks using a straightforward idea that adds computationally negligible cost at inference time, which could be of interest to the community. Using MIMO architectures for AT is interesting.\n- It presents extensive experiments on MNIST, including varying ball sizes (fig. 2), and a comparison to the adaptive AutoAttack.\n\n----\n\n# Weaknesses\nGiven that the paper combines existing work (MIMO) and AT, the novelty in my opinion is limited. Moreover, the contributions are empirical. This raises the need for a more thorough empirical analysis to demonstrate the efficacy of the proposed approach.\n\n- (Major) Weak empirical evaluation. \n   - While the authors compare with existing methods on CIFAR-10, comparison on other real-world datasets is missing, given that the paper is empirical.\n   - While the authors perform a thorough analysis on MNIST, to argue in favor of MAT it would be helpful to have such analysis on CIFAR-10 as MNIST can be in some cases misleading. For example, the paper lacks results on CIFAR-10 with: (1) varying radius, \n   - The paper lacks experiments with a varying number of attack steps, in particular, showing its generalization and robustness for different a varying number of steps.\n  - Solely two architectures are used, thus it is unclear if MAT performs well using different models.\n\n- (Major) Unclear if the MAT method is outperforming existing ones.\n   - Focusing on CIFAR-10, the MSD method [20] seems to provide better performances across the different attacks, see Tab.5.  The authors acknowledge this and argue that MAT may need further hyperparameter tuning, which raises a concern about how the hyperparameters are selected, as ideally the selection of the presented results should be fair/fixed toward all methods.\n   - If I am correct, from Tab. 9 in the App.  MAT without $\\ell_\\infty$ is less robust to AutoAttack relative to MSD.\n- (Minor) While the flow is easy to follow, the writing is in some cases informal and thus ambiguous, e.g., in the abstract 'attacks do not transfer easily', and careful proofreading and polishing are needed.\n- (Minor) In the introduction the authors argue with certainty that ensemble methods fail due to two main reasons, there should be a citation, or alternatively, elaborate that this is argued in this paper and point to the relevant section.  See also the relevant question 4 below. \n\n----\n# Minor comments and Recommendations\n- It would be helpful to include the total training times per epoch for the different methods for fixed setup.\n- Include detailed captions, e.g. dataset for each table, etc. Similarly for the tables, rather 'main result on...'  be more precise, e.g. rows/columns are methods used for training or testing, etc. \n- Incomplete sentence in App. C.1\n- Sec 4.3: Our methods also increasing -> increase\n\n----\n# Questions\n1. Resnet-50 for CIFAR-10 is less-common, did the authors try with ResNet-18? Moreover, what are the models used by MSD [20]? Ideally, the authors should include a comparison between the two methods using the same model for a fair comparison.\n2. For how many epochs are MAT and the baselines trained?\n3. I would recommend moving the AutoAttack results on CIFAR-10 in the main paper, as such comparison is important. Moreover, how do these AutoAtttack results compare to state-of-the-art AT methods (other than MSD)?\n4. Looking solely at Tab.1 could also indicate that the two AT models are robust. What are the clean accuracies and PGD robustness of the models in Tab.1 both separately and the two as an ensemble? \n5. Did the authors compare against PGD-50-10?\n\n\n", "summary_of_the_paper": "This paper uses ensembles--and more precisely multi-input-multi-output (MIMO) neural networks for computational efficiency---for adversarial training (AT), resulting in a method called MAT, short for MIMO AT.\nMoreover, the authors investigate how the sub-networks should be trained to increase the overall robustness of the total MIMO model.\nIn particular, the proposed MAT generates the adversarial samples using the gradient with respect to the objective of the ensemble.\n\nThe authors demonstrate on MNIST and CIFAR-10 that the proposed method achieves generalization and robustness comparable to existing state-of-the-art methods.\n", "summary_of_the_review": "In summary, the proposed method is intuitive and simple, as well as computationally attractive. \nHowever, the novelty is limited and the paper is primarily empirical as it does not provide novel theoretical insights.\nAs such, the presented empirical evaluation is insufficient to convey the benefit of the proposed method, in particular:\n(i) apart from MNIST, from real-world datasets results are present solely on CIFAR-10, moreover (ii) these results are incomplete in terms of analysis with varying hyper-parameters (attack radius and the number of attack steps), and (iii)  does not show a clear advantage over existing methods, see above for further comments. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new ensemble training method for improving adversarial robustness to multiple attacks (e.g., $\ell_2$, $\ell_1$ and $\ell_\infty$). Specifically, authors adopt the recent Multi-Input Multi-Output (MIMO) ensemble architecture for computational efficiency. Then, the authors construct the adversarial examples using the outputs of multiple attacks simultaneously. With these examples, standard adversarial training is conducted on MIMO ensemble.

All reviewers are on the negative side. AC agrees with reviewers’ concerns on limited novelty and insufficient empirical evaluation. AC also thinks that the improvement is not that significant compared to the existing method, especially concerning the real-world dataset. Overall, AC recommends rejection.
Decision: Reject