Year: 2021

Abstract:
While ensemble methods have been widely used for robustness against random perturbations (i.e., the average case), ensemble approaches for robustness against adversarial perturbations (i.e., the worst case) have remained elusive despite multiple prior attempts. We show that ensemble methods can improve adversarial robustness to multiple attacks if the ensemble is adversarially diverse, which is defined by two properties: 1) the sub-models are adversarially robust themselves against multiple attacks, and yet 2) sub-models are diverse measuring by adversarial transferability. While at first glance, creating such an ensemble would seem computationally expensive, we demonstrate that an adversarially diverse ensemble can be trained with minimal computational overhead via a Multiple-Input Multiple-Output (MIMO) model. Specifically, we propose to train a MIMO model with adversarial training (MAT), where each sub-model can be trained on a different attack type. When computing gradients for generating adversarial examples during training, we use the gradient with respect to the ensemble objective. The benefits of this scheme are twofold: 1) it only requires 1 backward pass and 2) the cross-gradient information between the models promotes robustness against transferable attacks. We empirically demonstrate that MAT produces an ensemble of models that is adversarially diverse and significantly improves performance over single models or vanilla ensembles while being comparable to previous state-of-the-art methods. On MNIST, we obtain 99.5% clean accuracy and (82.3%, 57.1%, 71.6%) against (`∞, `2, `1) attacks, and on CIFAR10, we achieve 79.7% clean accuracy and (47.9%, 61.8%, 47.6%) against (`∞, `2, `1) attacks, which are comparable to previous state-of-the-art methods.

heading: 1 INTRODUCTION
text: The arms race between attacks and defenses against DNNs has created a diversity of attack types. These include black-box vs white-box attacks and gradient-based vs gradient-free attacks. Ideally, adversarially robust models would be robust against all types of attack. Many defenses have been developed to protect DNNs from adversarial attacks. For instance, gradient masking [21], input denoising [33, 15], and hidden layer randomization [16], are three techniques that in their time were heralded as strong defense techniques. However, [1, 3, 28] show that all of the above methods fail against adaptive adversaries, i.e., perturbations crafted with full knowledge of the defense measure. Among all attempts to make the deep model resilient, Adversarial Training (AT) [17] has stood the test of time and is regarded as the gold standard for its resilience to white-box adversaries [18]. AT trains the deep models with adversarial examples instead of with benign training examples. Due to its empirical robustness, this method remains state-of-the-art against a wide range of attacks. However, AT suffers from two fundamental drawbacks. First, it is computationally expensive, specifically the method to generate effective AEs. Second, the choice of attack type used during training significantly affects the type of robustness that is observed. For example, previous work [27, 22] shows that the model adversarial trained using the `∞ attack is robust against `1 and `2 PGD attacks but vulnerable
against gradient-free attacks. This phenomenon reveals that the robustness of adversarial training implicitly masks out the gradient information for `1 and `2 attacks. Previous works [27, 18] try to revise adversarial training by training the model using the worst case of PGD adversarial examples among different attack metrics and achieve better results. MSD [18] revises the Tramèr and Boneh’s method [27] by calculating `1, `2, `∞ attacks for each step of PGD iteration and updates with the one that achieves the largest loss. This method achieved better performance against `1, `2, `∞ attacks. However, though MSD is faster than [27], it still requires multiple forward passes for each step of the PGD attack, which makes it more expensive to train comparing to the standard AT.
As ensemble methods have proven to be effective in robustness to random perturbations (i.e.,, the average case), one natural idea to improve adversarial robustness is to exploit ensemble methods (or model averaging). Ensemble methods combine the predictions of sub-models based on some vote functions (like averaging or majority vote). Previous attempts of using ensemble methods for adversarial robustness have been shown to be vulnerable under strong adversaries [35, 19, 23, 29, 10]. They mainly failed for two reasons [34]. First, if the ensemble is diverse but the models are weak (i.e., not adversarially robust), an adaptive attack only needs to find a single AE that lies in the intersection of the AE space of the sub-models (i.e., a single worst case), which is easy [10]. Intuitively, this finds a single perturbation that is a “common mistake” across sub-models. Second, adversarial examples tends to transfer between similar models, i.e., the adversarial perturbation generated from one sub-model may easily mislead another sub-model especially when they share the similar training set or architecture [20]; and the transferability of attacks remains true even if the sub-models are adversarially robust themselves (see vanilla ensemble models in 4.2 of the experiments). Thus, neither diversity of the models nor adversarial robustness of each of the sub-models by themselves can improve adversarial robustness.
To overcome these two prior failures of ensemble methods, we propose to encourage adversarial diversity which is defined to have two key features simultaneously: 1) adversarial robustness of each sub-model and 2) robustness against transfer attacks between sub-models. Additionally, we would like to do this with minimal computational overhead because AT is already very computationally expensive. Our approach is inspired by a recent work called multi-input-multi-output (MIMO) training [8]. MIMO can train M sub-models simultaneously to form a diverse ensemble, and crucially the training time of the ensemble is almost the same as that of a single model. In this work, we show that by combining MIMO configuration with AT, it is possible to attain robustness against multiple attack types with an adversarially diverse ensemble, i.e., we can achieve adversarial diversity. The results in § 4.1 shows that by sharing model parameters, the sub-models act as an implicit way of bringing adversarial diversity while also remaining robust. Then in § 4.2, we present that our MAT adversarially trained with each sub-model bestowing resilience to a particular attack metric leads to robustness against multiple types of perturbations. We achieve robustness accuracy (82.3%, 57.1%, 71.6%) against (`∞, `2, `1) attack with epsilon (10, 2, 0.3) on MNIST. On CIFAR10 dataset, our model achieves robustness accuracy (47.9%, 61.8%, 47.6%) against (`1, `2, `∞) attack with radius (12, 0.5, 0.03). Finally, in § F, we provide an alternative application that one can utilize our MIMO+AT (MAT) by training the MAT with all sub-models trained with the same attack to strengthen previous AT methods to achieve better adversarial robustness without loss of clean accuracy. We provide three examples in this section as `∞ AT, `2 AT, and MSD. We summarize our contributions as follows:
1. We shed light on why prior ensemble methods may fail and hypothesize that ensemble methods can improve adversarial robustness if they are adversarially diverse, which is defined by two properties: 1) the sub-models are adversarially robust themselves and yet 2) adversarial attacks do not transfer easily between sub-models. 2.We propose to use MIMO AT (MAT) to train such an ensemble in a computationally efficient way and discuss how the cross-gradient information between the sub-models during training can increase the robustness to transfer attacks between models. 3. We empirically demonstrate that MAT is adversarially diverse and improves significantly over AT models and vanilla ensembles while being comparable or better than prior state-of-the-art methods for multi-attack robustness.

heading: 2 PRELIMINARY
text: In this section, we review some key concepts related to adversarial attacks, adversarial training (AT), and MIMO structure, all the notations defined here will be used throughout the paper.
PGD Adversarial Attacks. PGD attacks seek a point x′ in a neighborhood of the benign example x with perturbation δ, which maximizes the loss L. x′ is generated as: x′ = x+arg maxδ∈B‖·‖p ( ) L(fθ(x+δ), y ∗), where denotes the attack radius, fθ is the DNN with parameter θ, y∗ is the true label. Denote the perturbation δ = x′ − x. A general PGD attack adopts gradient-type algorithm to update δ as
δ(t+1) = ΠB‖·‖p ( )
( δ(t) + α · Vp (∇xL(fθ(x), y∗)) ) , with Vp(δ) = arg max
v∈B‖·‖p (1) v>δ. (1)
where p is the attack metric (usually `p norm), α is the step size, Π is the operator projecting the gradient onto the `p ball with radius (denoted by B‖·‖p( )), Vp is the operator mapping the gradient to the steepest descent direction on the `p ball with radius 1. Note that Vp is a fundamentally different operator than Π, and in the case of the p = ∞, V∞ reduces to the sign of the gradient (as in the FGSM attack). Intuitively, PGD adversarial attack uses iterate with a fixed step size α along the steepest direction Vp (∇xL(fθ(x), y∗)). FGSM [7], BIM [13] and [17] can all be treated as variants of the PGD Attack.
Adversarial Training. AT is regarded as one of the most practical yet effective methods for defending against adversarial attacks. This has stood the “test of time" through multiple iterations of proposed defenses and attacks that discover vulnerabilities in them. AT seeks to build a robust model against adversarial attacks by producing adversarial examples and injecting them into training data by solving a saddle point optimization problem
θ∗ = arg min θ ∑ i max δi∈B‖·‖p ( ) L (fθ(xi + δi), y∗i ) . (2)
Here i denote the i-th batch number of training set (xi, y∗i ). The inner maximization seeks to find the adversarial directions δi given the current classifier fθ, and the outer minimization tries to find the best θ to minimize the summation of all batches’ loss. PGD AT is widely accepted as a standard approximation of inner maximization methods since it is a universal first-order attack algorithm [17]. Though it roughly needs K + 1 times more computation compared to standard training, PGD-AT remains the de facto standard for AT since it walks the balance between strength of attack and computational cost of generating adversarial examples. The drawback of AT is that solving the inner maximization requires a pre-defined attack metric p, which only contributes to robustness under `p attacks. [27]. In this paper, we show how the adversarial robustness can be achieved through AT but crucially, one can generalize the defense to multiple attack types and achieve this at about the exact cost as AT.
MIMO Ensemble. Ensemble methods have been widely used to improve a model’s calibration and robustness over out-of-distribution samples by averaging over multiple neural network predictions [8, 31, 32]. However, ensemble over M deep models (sub-models) is usually M times more expensive from a computational and a memory standpoint. Furthermore, to improve uncertainty estimation, sub-models often sacrifice individual performance to achieve diversified predictions.
To address the computational drawbacks of vanilla ensembles, [8] proposed an ensemble structure called MIMO Ensemble. It showed a surprising result: the benefits of using multiple predictions can be achieved ‘for free’ under a single model’s forward pass. In particular, they showed that, using a multi-input multioutput (MIMO) configuration, one can utilize a single model’s capacity to train multiple sub-models that independently learn the task at hand. Since all the layers are shared except for the input layer and the output layer, this architecture uses only about 1% more parameters. Training a MIMO model has almost
the same time complexity as training a standard model while keeping the sub-models diverse without any special regularization and parameter tuning. During training, MIMO uses gradient method to solve following optimization problem for M sub-models.
minimize θ ∑ i M∑ m=1 L(fθ(xi,1, ..., xi,M )m︸ ︷︷ ︸ yi,m , y∗i,m), (3)
where (xi,1, ...xi,M ) are i.i.d sampled from the training set, with xi,m denoting the input of the i-th batch for the m-th subnetwork, yi,m, y∗i,m represent the outputs from the model and the ground truth respectively, and L is the cross entropy loss. Notice that the overall objective is the summation of each sub-model’s loss. The separation structure promotes the sub-models’ prediction to be less correlated because the output yj is independent to xi when i 6= j. During testing, the same inputs are copied and fed to all the M sub-models, then MIMO decides the prediction by averaging yi,m.
3 OUR APPROACH: MAT
Given the weaknesses of prior ensemble methods, we seek an algorithm that produces an adversarially diverse ensemble, where the sub-models are robust to the attacks transferred from other sub-models so that the ensemble can have better robustness comparing to single sub-model. To achieve this, we propose to combine MIMO and AT, an approach we call MAT.
There are two difficulties to overcome in adversarially training an ensemble model: 1) For the outer minimization, how should one choose the loss function to update the sub-model parameters θ? To be specific, choosing a single sub-model loss or the ensemble loss? 2) For the inner maximization, how should the adversarial examples x′1, . . . , x ′ M be created during training? To solve (1), [8] demonstrates that taking the gradient with respect to the ensemble loss contributes to higher clean accuracy and is more computationally efficient. Therefore, we choose to update parameter θ using SGD, where the gradient is taken with respect to the ensemble loss instead of the loss of each sub-model. To tackle (2), one natural choice is taking gradient with respect to each sub-model’s loss as in standard AT on each sub-model. However, we show in Sec. 3.2 that taking gradients with respect to the ensemble loss produces sub-models that re more robust to transfer attacks and, furthermore, it is more computationally efficient. Therefore, in the inner maximization, we generate adversarial examples by algorithms that compute gradients with respect to the ensemble loss instead of each sub-model’s loss. We explore these two choices in more detail in the next two sections.

heading: 3.1 NAÏVE MIMO ADVERSARIAL TRAINING
text: The foundation of MAT is MIMO adversarial training method. The goal is to train an ensemble that is adversarially diverse. To encourage diversity, we use different attack types for each sub-model. We adopt the common choice in the literature as three PGD attacks `1, `2, and `∞ to instantiate our network. We aim to solve the following problem
min θ ∑ i M∑ m=1 max δi,m∈B‖·‖pm ( m)
L(fθ(xi,1 + δi,1, ..., xi,M + δi,M )m︸ ︷︷ ︸ y′i,m , y∗i,m), (4)
where for each batch i, and each sub-model m, the inner maximization seeks the worst-case adversarial perturbation δi,m ∈ B‖·‖pm ( m) under the current classifier fθ. Notice that δi,m might be different for a different sub-model m. As a naïve approach, we use K-step PGD to update δki,m as the following
δ (k+1) i,m = ΠB‖·‖pm ( m) ( δ (k) i,m + αm · Vpm ( ∇xi,mL(y′i,m, y∗i,m) )) . (5)
Notice the difference between equation 5 and equation 1: for different m, equation 5 allows for different projections B‖·‖pm , stepsize αm as well as steep descent direction Vpm , which results in potential robustness over diverse attacks; however, equation 1 can only be seen as special case of equation 5 with m = 1, therefore it lacks multiple robustness after the training.

heading: 3.2 MIMO ADVERSARIAL TRAINING (MAT)
text: While the naïve MIMO adversarial training may produce models that are adversarially robust by themselves, it does not prevent adversarial examples that would transfer across multiple sub-models and, additionally, it is computationally expensive since it requires M backward passes for each δ computation. Therefore, to further improve the adversarial diversity of MAT, and also reduce the computational burden, we propose to replace equation 5 by the following
δ (k+1) i,m = ΠB‖·‖pm ( m) δ(k)i,m + αm · Vpm ∇xi,m M∑
j=1
L(y′i,j , y∗i,j)  , (6) where for each tuple (i,m), we used the approximate gradient∇xi,m ∑M j=1 L(y′i,j , y∗i,j) instead of the true gradient ∇xi,mL(y′i,m, y∗i,m), where the difference between the two is defined as the cross gradient
cross gradient (i,m) , ∇xi,m ∑ j 6=m L(y′i,j , y∗i,j) = ∇xi,m M∑ j=1 L(y′i,j , y∗i,j)−∇xi,mL(y′i,m, y∗i,m). (7)
We include the pseudo-code of MAT in appendix Sec. A.1, where the cross loss related to equation 7 allows the exchange of information between the networks during the attack stage which may help in reducing the transferability of Adversarial examples. It also plays an essential role in reducing the computation burden as well as improving the adversarial diversity.
To obtain all the perturbations δi,m,m = 1, . . . ,M using the true gradient update equation 5, we need M times backward propagation (time complexity analysis is in the appendix B). However, we only need 1 backward propagation when we adopt the approximate gradient step equation 6. Therefore equation 6 reduces the computational burden dramatically.
Next, we explore two properties of the cross gradient which contribute to adversarial diversity: 1) Reduced adversarial examples transferability between sub-models: In Section 2, we showed that minimization weakens the correlation between the sub-models due to MIMO structure. However, for problem equation 5, there’s still a possibility that the adversarial perturbation δi,m could flip the output yj of the j-th sub-model. Due to minimax structure of the problem equation 5, the flipping phenomena can be effectively reduced by introducing the cross loss during maximization training process, where the cross gradient (i,m) describes the change of cross loss ∑ j 6=m L w.r.t the change of xi,m locally. Therefore, introducing cross loss can
improve the robustness of MAT against different attacks. 2) Cross-gradients become small but non-zero during training: This can be validated by Fig. 1, where the cross gradients become very small during the training but the final cross gradient does not become exactly 0, showing that there is still weak correlation between sub-models.

heading: 4 EVALUATION
text: In § 4.1, we demonstrate that the sub-models trained by our MAT are more robust to adversarial examples generated from other sub-models (reduced adversarial transferability). Furthermore, we show that each sub-model is robust against multiple adversarial examples. In § 4.2, we present experimental results on MAT versus state-of-the-art multi-robustness defenses for a variety of attacks.

heading: 4.1 DIVERSITY AND ROBUSTNESS
text: Diversity of adversarial transferability among sub-models A group of classifiers can benefit from the ensemble if and only if each member is diverse enough. There are several different measures trying to quantify the property [12, 35]. In this section, we observe how the ensemble of sub-models perform against diverse adversarial attacks. To explore the diversity among sub-models of vanilla-ensemble (resp. MAT), the sub-models 1, 2, and 3 have been trained against respectively `1, `2, and `∞ PGD attacks; then, we evaluate vanilla-ensemble (resp. MAT) against `∞-adversarial attacks and report their clean accuracy (the results corresponds to `1 and `2 are similar and summarized in appendix G We study two kinds of `∞-adversarial examples. The first kind is `∞-adversarial attack generated from sub-model 1, from which we can study the transferability of `∞-adversarial attack generated from sub-model 1 by counting the successful attacks to sub-model 2, sub-model 3, and vanilla-ensemble (resp. MAT). In particular, we denote
Nii , #{successful attacks to sub-model i out of 1000 `∞-attacks generated from sub-model i}, Nij , #{successful attacks to sub-model j out of 1000 `∞-attacks generated from sub-model i}.
Then, we define the transferability of adversarial examples from i to j as rij , Nij/Nii. Furthermore, we use `∞-adversarial examples generated from vanilla-ensemble (resp. MAT) to attack each sub-model and their vanilla-ensemble (resp. MAT) to evaluate the overall performance of vanilla-ensemble (resp. MAT). The results are summarized in second row of Table. 1 (resp. Table. 2) for vanilla-ensemble (resp. MAT).
Comparing Table. 1 and Table. 2, we observe the following phenomena. First, the transferability ratios of MAT are r12 = 23.7%, r13 = 33.3%, which are much lower than r12 = 200%, r13 = 153.8% of vanilla ensemble.
This indicates the sub-models of MAT are more robust to `∞-attacks, and `∞-attacks are easier to transfer between the sub-models of vanilla ensemble. Therefore, we conclude that the sub-models of MAT are diverse enough to reduce the effective `∞ adversarial transferability. Second, the performance of vanilla-ensemble is close to the worst sub-model performance. This is indicated by noticing the number of successful `∞-attacks for vanilla-ensemble is 960, which is close to max{45, 994, 998} = 998. This means that vanilla-ensemble does not provide extra ro-
bustness compared to its sub-models. However, the successful `∞-attack against MAT is only 128, which
only takes up 23.4% of the worst sub-model performance max{114, 548, 106} = 548. This suggests that the MAT clearly improves the robustness accuracy compared to that of sub-models.
Robustness of sub-models We empirically show that the sub-models of MAT are robust against `∞, `2, `1 attacks. During testing, the same inputs are copied and fed to all the M sub-models, then MAT decides the prediction by averaging yi,m. The results are summarized in Table 3. We observe that using MAT, each sub-model is relatively robust against all adversarial examples. The only weak sub-model is sub-model 2, which shows vulnerability against `∞ PGD attack (33.1%), however, MAT still performs well against `∞ attacks (89.2%) because it averages the output of the three sub-models.

heading: 4.2 MAIN RESULT
text: In this section, we present two main results of MAT on MNIST and CIFAR-10 against various types of attacks and compare it with different baselines. MAT generally boosts the previous methods without extra cost.
Main result on MNIST Our main results on MNIST are summarized in Figure 2 and Table 4. Figure 2 plots varying number of attack steps versus robust accuracy, and the varying radius using `∞ (resp. `2 and `1) versus robust accuracy. We can capture the following phenomena: (1) both of MAT and MAT+MSD exhibit generalization and robustness for a varying number of attack steps; (2) MAT outperforms MSD and MAT+MSD depending against `∞- PGD attack; and when it comes to `2-PGD attack and `1-PGD attack, the performance of MAT+MSD surpass the other two methods on MNIST for a varying number of attacks; (3) MAT outperform MSD on `∞- PGD attack; 2) on `2- (resp. `1-) PGD attack, MAT+MSD surpass the other two method. Table. 4 shows under the attack radius choice 0.3 (resp. 2 and 10) for `∞ (resp. `2 and `1),
the robustness accuracy comparison under different attack metrics between MAT and AT, vanilla ensemble MSD, standard training is listed for clean accuracy comparison1. All `∞ (resp. All `2 and All `1) attack is to defend against `∞-based attacks, i.e., the union of PGD `∞ attack, HSJ attack, Auto Attack (resp. PGD `2 attack, C&W attack, Boundary attack for `2- based attacks; and PGD `1 attack, Salt & Pepper attack for `1-
1All the hyperparameter including the step size of the PGD attack, attacks radius , and termination max iteration are chosen as the best tuning for MSD [18] in order to present results on a same base line (which may not be the best for MAT). Please check Sec. C. for detailed information about hyperparameters choice.
based attacks). This reveals that MAT could improve the overall model’s robustness. The vanilla ensemble shows its vulnerability to all three kinds of adversarial attacks though it combines a three times larger model and requires three times longer training. MAT achieves 82.3% against the union of `∞ attacks on the MNIST dataset, which surpasses the previous state-of-the-art by 26.0% percent. For defending against the union of adversarial attacks, our method attains 51.7% robustness accuracy.
Main result on CIFAR-10. Our main results on CIFAR-10 are summarized in Figure 3 and Table 5. We observe the following: (1) the performance of MAT is comparable w.r.t MSD on `∞ and `1- PGD attack with varying steps and varying radius; (2) MSD exhibits better testing accuracy than the other two methods on CIFAR-10 for a varying number of `2-PGD attacks with radius r = 0.5, however, MAT outperforms MSD when the attack radius increases (greater than the common choice 0.5).
Table 5 shows that MAT achieve strong robustness against multiple kinds of attack. Crucially, the vanilla ensemble fails against the `∞ attacks while MAT is competitive against the state-of-the-art MSD, while being computationally more efficient (K + 1 forward passes for MAT versus (1 +M)K + 1 for MSD, with the same number of backward passes K + 1). We include the time complexity analysis in Table 6 and training time on each model in Table 7 please see appendix B and appendix E for details.

heading: 5 RELATED WORK
text: Adversarial Defense. Many defenses have been proposed, with most of them being broken by adaptive attacks [1, 3, 28]. Adversarial Training [17], though expensive in computation, has emerged as the best practical way to create models resilient to white-box attacks. A slew of literature has developed variants of AT differing in the way samples are created [6], the way the models are iterated upon [24, 30], and the way different attacks are combined [27, 18]. Our ensemble method is orthogonal to these approaches and therefore can be combined with these, with varying levels of additional design and engineering effort, such as to design the right combining function across the sub-models.
Adversarial Defense against multiple types of perturbation. [18] tries to build a robust model against multiple perturbations by introducing multiple steepest direction PGD attacks during training; their method ensembles three different attacks into one, called multi steepest direction. Our method instead seeks an ensemble over models. [26] proposed a provable defense against all `p norm with robust radius 3− 5 times smaller than state-of-the-art heuristic defense methods [18]. This technique has the benefit that it uses transfer learning and therefore does not need separate training for each kind of attack. This can be used seamlessly with MAT by starting each sub-network after transfer learning from the another sub-network. [14] trains DNNs against imperceptible adversarial examples, which are approximated using DNNs. While it shows robustness against a wide range of attacks, its performance against specific attacks is limited. Our approach MAT outperforms prior solutions in this space, either as we have proven empirically for the latest approach [18] or through the author-published numbers on the same dataset.

heading: 6 CONCLUSION AND DISCUSSION
text: In this paper, we present an adversarially diverse ensemble method called MAT in which each sub-model achieves both robustness and diversity by developing the idea of learning an ensemble via adversarial training (AT) and adapt the structure of a recently discovered MIMO architecture. We require the same number of forward and backward passes as AT and yet achieve generalized robustness against multiple attack types. A drawback of our scheme is that if the network size is constrained, it may not have the capacity to accommodate multiple sub-models. Another drawback that is common to most AT methods is that the AE perturbation hyperparameters must be manually selected (we simply use the parameters from MSD [18]). Methods for automatically selecting the best hyperparameters using zero-th order optimization would broaden the applicability of our approach.
REPRODUCIBILITY STATEMENT
We have publicly released all the code, the datasets and models used, along with the implementation of the attacks. To help with the reproducibility of the experiments, we make available all of our code, the trained models, the training hyperparameter values, and the raw results from our testing. We point to the exact attacks that we have used (which are from existing codebases) in our evaluation. These are all to be found at the anonymized Github link https://github.com/anonymous-lab-ml/mimo-adv/. Code and pre-trained models for reproducing our experiments have been uploaded to the anonymized repository https://github.com/anonymous-lab-ml/mimo-adv/.

heading: B TIME COMPLEXITY OF DIFFERENT AT METHODS
text: Compared to standard training (i.e., without ensemble), vanilla AT using K-step PGD attack requires K + 1 times more computation, due to the K forward and backward passes individually for generating each AE plus the outer minimization. Furthermore, if we try to use M -vanilla ensemble AT to promote robustness
over multiple attacks, it will require M times more computation compared to vanilla AT. This becomes infeasible in practice, e.g., for CIFAR-10 with ResNet-50, 100 epochs. It takes 40 times of training time to train one single model compared to standard training. To reduce the computation, we have the insight to replace the vanilla ensemble with a MIMO architecture and to train for all the M attack types within the same network, each within its own subnetwork. This allows us to calculate all subnetworks’ outputs yi, i = 1, . . . ,M in one forward pass instead of M passes, i.e., (y1, . . . , yM ) = fθ(x1, . . . , xM ), instead of (y1, . . . , yM ) = (fθ1(x1), · · · , fθM (xM )) in case of the M vanilla ensemble. This approximation works because we can fit M subnetworks within the original network, i.e., without needing to expand the size of the original network. This is because of the residual capacity in even reasonably sized networks (like ResNet-50 in our case) is sufficient.
Thus, M -MIMO AT reduces the number of forward passes to be similar to that of vanilla AT training. However, for each subnetwork, one iteration of PGD attack still needs one backward pass, which leaves the computation of backward pass almost as high as that of naïve ensemble with M networks. The computation cost per iteration of different methods is summarized in Table 6.

heading: C EXPERIMENTS SETTING
text: C.1 HYPERPARAMETERS FOR TRAINING THE MODELS
Models. We use ResNet18 [9] for all experiments on MNIST dataset. We also reproduce MSD results using the same neural network in their original paper. For CIFAR10, we use ResNet50.
Optimizers. For MNIST dataset, we use Adam optimizer [11]. For CIFAR10 dataset, we use SGD optimizier with momentum 0.9, and weight decay 5× 10−4.
Step size. We use the step size scheduler follows [25]. For MNIST, the step size linearly increase from 0 to 10−3 over the first 40% epochs, and down to 0 over the last 60% epochs. For CIFAR10, the step size linearly increase from 0 to 0.1 over the first 40% epochs, and down to 0.005 over the next 40% epochs, and finally back down to 0 in the last 20% epochs. The overall number of epochs is 25 (resp.75) for MAT and MAT + MSD on MNIST (resp. CIFAR10) datasets.
Hyperparameters for adversarial training. For all MNIST dataset, the inner maximization is optimized by (`∞, `2, `1) PGD attacks with max iteration 100, radius = (0.3, 2.0, 10) and step size α = (0.01, 0.1, 0.8) respectively. For all CIFAR10 dataset, the inner maximization is optimized by (`∞, `2, `1) PGD attacks with max iteration 50, attack radius = (0.03, 0.5, 12) and step size α = (0.003, 0.02, 1.0) respectively.
C.2 HYPERPARAMETERS FOR ADVERSARIAL ATTACKS USED FOR ROBUST TEST
For all the results, we evaluate the first 1000 test examples. To verify the performance of MAT and compare it to the performance of the baseline Adversarial Training, we evaluate the performance of our model against `∞, `2, `1 PGD attacks [17], Carlini Wagner [4] `2 attack,and two decision-based black-box attack: Boundary attack [2] and HopSkipJump attack[5].
For PGD attacks, to guarantee the convergence, we increase the max iteration to 200. For `1 PGD attack, we fixed the number of pixels to perturb per iteration as 20 during test. The other hyperparameters stay the same as in the software distribution.
For other attacks, we set the hyperparameters to be consistent to the original works.

heading: D THE DIFFERENCE BETWEEN THE TRUE GRADIENT AND APPROXIMATE GRADIENT AFTER THE STEEPEST DIRECTION MAPPING OPERATOR V
text: We claim that the cross gradient does influence the optimization, since the the small value of cross gradient may affect the gradient value after the steepest descent direction. In Figure 4, we define Ep = ∑ (Vp(∇xm (L1+L2+L3))⊕Vp(∇xmLm))
|xm| , where | · | denotes the cardinality of the xm, ⊕ denotes XOR operation, and p = {1,∞}. Ep calculates the fraction of entries that are different in the approximate gradient after steepest-descent projection compared to the true gradient. Notice that the XOR operator with a summation calculates the overall number of different pixels the two gradients have.|xm| denotes the overall pixels the gradient has. Figure 4 shows that the difference between the approximate gradient and the true gradient after the steep ascent direction mapping, where it decreases during training. Compared to Figure 1, which shows the difference between the true gradient and our approximation. The y axis denotes the fraction of pixels that changed by the cross gradient term. We can see from 4, especially, E∞ does not converge to 0. They term might perform an important role to bring multiple robustness for the sub-model trained against a single attacks.

heading: E WALL CLOCK TIME DURING TRAINING
text: In this section we report the wall clock time to train the different models.

heading: F SUB-MODEL TRAINED WITH SAME ADVERSARIAL ATTACK
text: As mentioned, MAT can also be integrated with other AT techniques to strengthen the model. In this section, we empirically show the three experiments where we combine MAT with three previous methods, `∞ AT, `2 AT, and MSD respectively. The result in Table 8 shows that MAT achieves higher robustness accuracy in most scenarios even against the attack type that AT approach has been designed for. . Notice that all these models are trained with the same models and hyperparameters. As we observe, for MSD+MAT, we also bring 20.4% extra robustness against `∞ attacks, and 29.2% more robustness accuracy against `1 attacks with 0.8% extra clean accuracy. The drawback is we trade off 27% robust accuracy on `2 attacks. This is due to the imbalance of choices of the hyperparameters giving relative weight to the three attack types `1, `2, and `∞. In this table, we show our MAT could use with sub-models trained with same metrics. The results show that if we combine the MIMO structures and `∞ AT, we could achieves overall higher accuracy comparing to the standard `∞ AT.

heading: G TRANSFERABILITY AND SUB-MODEL ROBUSTNESS
text: This section shows the sub-models performance of the vanilla ensemble (resp. MAT) against `2 PGD attacks in Table 9 (resp. Table 10) and against `1 PGD attacks in Table 11 (resp. Table 12). Comparing Table 9 and Table 10, when the attacker generates the adversarial examples on sub-model 2, we find that the adversarial examples is easier to transfer to other sub-models comparing to our MAT. This especially is due to the vulnerability of the sub-model 1 and sub-model 3. Since the sub-models is not robust, if we check the second row of the vanilla-ensemble model, the number of successful `2-attacks is 437 which is much higher than 383 of our MAT. Defending against `1 attacks, though the sub-model of the vanilla ensemble achieves low adversarial transferability, our model still outperforms the vanilla ensemble by 12.1%. This is due toe the vulnerable of the sub-models, which empirically proves our claim that to achieve robustness against adversarial examples, both sub-model’s robustness and the adversarial diversity between sub-models are required.
MAT + MSD

Review summaries:
[Summary] This paper proposes an ensemble method to defend against adversarial attacks.  To be specific, this paper combines MIMO strategies (Havasi et al.) and adversarial training (Madry et al.). 

This paper proposes an ensemble based adversarial training strategy, which could improve the worst-case model robustness against multiple $\ell_p$-norm adversarial perturbations. To improve the training efficiency, the author introduces the Multiple-Input Multiple-Output (MIMO) strategy. The motivations and insights provided by this paper are interesting for me.

This paper uses ensembles--and more precisely multi-input-multi-output (MIMO) neural networks for computational efficiency---for adversarial training (AT), resulting in a method called MAT, short for MIMO AT. Moreover, the authors investigate how the sub-networks should be trained to increase the overall robustness of the total MIMO model. In particular, the proposed MAT generates the adversarial samples using the gradient with respect to the objective of the ensemble.  The authors demonstrate on MNIST and CIFAR-10 that the proposed method achieves generalization and robustness comparable to existing state-of-the-art methods. 

In this work, the authors propose a new training algorithm MAT that adversarially trains Multi Input Multi Output (MIMO) models. They show that ensemble models based on MIMO when trained adversarially , show ``adversarial diversity'' and therefore are less vulnerable to transfer attacks. They empirically demonstrate that such models are robust to a variety of $\ell_1, \ell_\infty$ and $\ell_2$ attacks and achieve better performance than other methods claiming robustness across different threat models. The authors also show computational benefits of their algorithm over vanilla ensemble training.