Adversarial Robustness Against the Union of Multiple Perturbation Models
Authors: Pratyush Maini
Eric Wong
Zico Kolter
Venue:  (2020)
Keywords: adversarial
robustness
multiple perturbation
MNIST
CIFAR10

Abstract: Owing to the susceptibility of deep learning systems to adversarial attacks, there has been a great deal of work in developing (both empirically and certifiably) robust classifiers, but the vast majority has defended against single types of attacks. Recent work has looked at defending against multiple attacks, specifically on the MNIST dataset, yet this approach used a relatively complex architecture, claiming that standard adversarial training can not apply because it "overfits" to a particular norm. In this work, we show that it is indeed possible to adversarially train a robust model against a union of norm-bounded attacks, by using a natural generalization of the standard PGD-based procedure for adversarial training to multiple threat models. With this approach, we are able to train standard architectures which are robust against l_inf, l_2, and l_1 attacks, outperforming past approaches on the MNIST dataset and providing the first CIFAR10 network trained to be simultaneously robust against (l_inf, l_2, l_1) threat models, which achieves adversarial accuracy rates of (47.6%, 64.3%, 53.4%) for (l_inf, l_2, l_1) perturbations with epsilon radius = (0.03,0.5,12).

Under review as a conference paper at ICLR 2020
ADVERSARIAL ROBUSTNESS AGAINST THE UNION OF MULTIPLE PERTURBATION MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Owing to the susceptibility of deep learning systems to adversarial attacks, there has been great deal of work in developing (both empirically and certifiably) robust classifiers, but the vast majority has defended against single types of attacks. Recent work has looked at defending against multiple attacks, specifically on the MNIST dataset, yet this approach used a relatively complex architecture, claiming that standard adversarial training can not apply because it "overfits" to a particular norm. In this work, we show that it is indeed possible to adversarially train robust model against a union of norm- comm-bunded attacks, by using a natural generalization of the standard PGD-based procedure for adversarial training to multiple threat models. With this approach, we are able to train standard architectures which are robust against foo. f2, and & attacks, outperforming past approaches on the MNIST dataset and providing the first CIFARIO network trained to be simultaneously robust against (Leo 22, (1) threat models. which achieves adversarial accuracy of 46. 1% against the union of (foo, f2, (1) perturbations with radius (0.03, 0.5, 12).
INTRODUCTION
Machine learning algorithms have been shown to be susceptible to adversarial examples (Szegedy al. 2014) through the existence of data points which can be adversarially perturbed to be mis- classinied, but are "close enough" to the original example to be imperceptible to the human eye. Methods to generate adversarial examples, or "attacks", typically rely on gradient information, and most commonly use variations of projected gradient descent (PGD) to maximize the loss within small perturbation region, usually referred to as the adversary's 's threat model. Since then, a number of heuristic defenses have been proposed to defend against this phenomenon, e.g. distillation (Papernot ct al. 2016) or more recently bogit-pairing (Kannan et al., 2018). However, as time goes by, the original robustness claims of these defenses typically don't it hold up more advanced adversaries or more thorough attacks (Carlini & Wagner 2017 Engstrom et al. 2018 Mosbach et al., 2018). One heuristic defense that seems have survived (to this day) is to adversarial training against PGD adversary (Madry et al., 2018), which remains quite popular due to its simplicity and apparent empirical robustness. The method continues to perform well in empirical benchmarks even when compared to recent work in provable defenses, although it comes with no formal guarantees.
Some recent work, however, pointed out that adversarial training against foo perturbations "overfits" to the foo threat model, and used this as motivation to propose more complicated architecture in order to achieve robustness to multiple perturbation types on the MNIST dataset (Schott et al. 2019).
In this work, we offer a alternative viewpoint: while adversarial training can overfit to the individual threat models, we show that is indeed possible use adversarial training to learn a model which is simultancously robust against multiple types of Lp norm bounded attacks (we consider Loc. lz. and l attacks, but the approach can apply to more general attacks). First, we show while simple generalizations of adversarial training to multiple threat models can achieve some degree of robustness against the union of these threat models, the performance is inconsistent and converges to suboptimal tradeoff's which may not actually minimize the robust objective. Second, we propose slightly modified PGD-based algorithm called multi steepest descent (MSD) for adversarial training which more naturally incorporates the different perturbations within the PGD iterates, further improving the adversarial training approach by directly minimizing the robust optimization objective. Third, show empirically that our approach improves upon past work by being applicable to standard

---

Under review as a conference paper at ICLR 2020
network architectures, easily scaling beyond the MNIST dataset, and outperforming past results on robustness against multiple perturbation types.
2 RELATED WORK
After their original introduction, one of the first widely-considered attacks against deep networks had been the Fast Gradient Sign Method (Goodfellow et al. (015). which showed that a single, small step in the direction of the sign of the gradient could sometimes fool machine learning classifiers. While this worked to some degree, the Basic Iterative Method (Kurakin et al. 2017) (now typically referred to the PGD attack) and now lies at the core of many papers. Since then, a number of improvements and adaptations have been made to the base PGD algorithm overcome heuristic defenses and create stronger adversaries. Adversarial attacks were thought to be safe under realistic transformations (Lu et al. 2017) until the attack was augmented to be robust to them (Athalye al., 2018b). Adversarial examples generated using PGD on surrogate models can transfer to black box models (Papernot et al. 2017). Utilizing core optimization techniques such as momentum can greatly improve the attack success rate and transferability, and was the winner of the NIPS 2017 competition on adversarial examples (Dong jet al. 2018). [eeaat et al. (2188) showed that a number of ImageNet defenses were not as robust as originally thought, and and Athalye et al. 20) defeated many of the heuristic defenses submitted to ICLR 2018 shortly after the reviewing cycle ended, all with stronger PGD variations.
significantly more successful at creating adversarial examples,
Throughout this cycle of attack and defense, some defenses were uncovered that remain robust to this day. The aforementioned PGD attack, and the related defense known as adversarial training with PGD adversary (which incorporates PGD-attacked examples into the training process) has so far remained empirically robust (Madry et al. (2018). Verification methods to certify robustness properties of networks were developed, utilizing techniques such SMT solvers (Katz et al. (017]] SDP relaxations (Raghunathan et al., (b)18b), and mixed-integer linear programming (Tieng et al. 2019), the last of which has recently been successfully scaled to reasonably sized networks. Other work has folded verification into the training process to create provably robust networks (Wong & Kolter 2018: Raghunathan et al. (018a). some of which have also been scaled even larger networks (Wong et al.) [018 Mirman et al. 2018: Gowal et al.) 2018). Although some of these could potentially be extended to apply to multiple perturbations simultaneously, most of these works have focused primarily on defending against and verifying only single type of adversarial perturbation at a time.
Last but most relevant to this work are adversarial defenses that attempt to be robust against multiple types of attacks simultaneously. [Schott et al. (2019) used multiple variational autoencoders to construct complex architecture for the MNIST dataset that is not easily attacked by lxc. l2, and to adversaries. Importantly, |Schott et al. (2019) compare to adversarial training with an & -bonnddd PGD adversary described by Madry et al. (2018), claiming that the adversarial training defense overfits to the for metric, and they do not consider other forms of adversarial training. Following this, number of concurrent papers have since been released. While not studied as a defense, |Kang et al. (2019) study the transferability of adversarial robustness between models trained against different threat models. Croce & Hein (2019) propose a provable adversarial defense against all lp norms for p 2 1 using a regularizatiee term. Finally, Tramer & Boneh (2019) study the theoretical and empirical trade-offs of adversarial robustness in various settings when defending against multiple adversaries, however, they use rotation and translation adversary instead of an lz adversary for CIFAR1O.
Contributions In this work we demonstrate the effectiveness of adversarial training for learning models that are robust against union of multiple perturbation models. First, we show that while sim- ple aggregations of different adversarial attacks can achieve robustness against multiple perturbations models without resorting to complex architectures, the results are inconsistent across datasets and make suboptimal tradeoff's between the threat models. Second, we propose a modified PGD iteration that more naturally considers multiple perturbation models within the inner optimization loop of adversarial training. Third, we evaluate all approaches on the MNIST and CIFAR 10 datasets, showing that our proposed generalizations of adversarial training can significantly outperform past approaches for the union of fo.. lz. and E attacks. Specifically, on MNIST, our model achieves 58.7% (individ- ually 63. 7%, 82.6%, 62.3% adversarial accuracy against the union of all three attacks (foo, l2, f1)
2

---

Under review as a conference paper at ICLR 2020
Pa
Figure 1: (left) A depiction of the steepest descent directions for foo. l2, and & norms. The gradient is the black arrow, and the a radius step sizes and their corresponding steepest descent directions loo. f2, and & are shown in blue, red, and green respectively. (right) An example of the projection back to an & ball of radius c after a steepest descent step from the starting perturbation 6. The steepest descent step is the black arrow, and the corresponding projection back onto the &g ball is red arrow.
for e (0.3, 1.5, 12) respectively, substantially improving upon the multiple peniiipppppetuuraaiioommmooood robustness described in |Schott et al.] (2019) and also improving upon the simpler aggregations of multplle adversarial attacks. Unlike past work, we also train CIFARIO model, which achieves 46.1% (individually 47.6%, 64.3%, 53.4%) adversarial accuracy against the union of all three attacks (Loo: lz, (1) for c (0.03, 0.5, 12). Finally, for completeness, we also draw relevant comparisons to concurrent work, and show that the relative advantage of our approach still holds.
3 OVERVIEW OF ADVERSARIAL TRAINING
Adversarial training is an approach to learn a classifier which minimizes the worst case loss within some perturbation region (the threat model). Specifically, for some network fe parameterized by loss function É, and training data (I,, Je) the robust optimization problem of minimizing the worst case loss within Ep norm-hounded perturbations with radius is
min Σ max e(fo(0; y;), dEAp..
where Ap.c- {6 ||8l|p <) is the lp ball with radius centered around the origin. To simplify the notation, we will abbreviate {(fe(r + +),,)) = f(x + 8;0).
3.1 SOLVING THE INNER OPTIMIZATION PROBLEM
We first look at solving the inner maximization problem, namely
max f(x + 8; 0). AEAp..
(2)
This is the problem addressed by the "attackers" in the space of adversarial examples, hoping that the classifier can be tricked by the optimal perturbed image, x + 8* Typical solutions solve this problem by running form of projected gradient descent, which iteratively takes steps in the gradient direction increase the loss followed by a projection step back onto the feasible region, the ép ball. Since the gradients at the example points themselves (i.e., 8 0) are typically too small to make efficient progress, more commonly used variation called projected steepest descent.
Steepest descent For some norm || Ilp and step size the direction of steepest descent the loss function for a perturbation is
(6) = arg max Vi(x 66
(3)
Then, instead of taking gradient steps, steepest descent uses the following iteration
g((+1) 5(1) Up (8(1)..
(4)
3

---

Under review as a conference paper at ICLR 2020
In practice, the norm used in steepest descent is typically taken to be the same Ep norm used to define the perturbation region Ap..- However, depending on the norm used, the direction of steepest descent can be quite different from the actual gradient (Figure Tp. Note that single steepest descent step with respect to the &o norm reduces to Use (x:) a sign( Vf(r + 6: 0)), better known the adversarial examples literature as the Fast Gradient Sign Method (Goodfellow et al. 2015).
The second component of projected steepest descent for adversarial examples is to project iterates back onto the ep ball around x. Specifically, projected steepest descent performs the following iteration
Projections
g(++1) = Pay.. (8 + up((())
(5)
where PAp.. (6) is the standard projection operator that finds the perturbation 8 Ap,e that is "closest" in Euclidean space to the input 6, defined as
Pap. (6) = arg min |6 - 8'12. d'E.p..
(6)
Visually, depiction of this procedure (steepest descent followed by projection onto the perturbation region) for an & adversary can be found in Figure [ If we instead project the steepest descent directions with respect the loo norm onto the foo ball of allowable perturbations, the projected steepest descent iteration reduces to
g(++1) PA (0(t) + voa (8(1))
(8 +a wiinnsign((Viieii 88); [-c,c]
clip
where clip,- -- "clips" the input to lie within the range [-c, c). This is exactly the Basic Iterative Method used in Kurakin et al. (2017), typically referred to in the literature & PGD adversary.
3.2 SOLVING THE OUTER OPTIMIZATION PROBLEM
We next look at how to solve the outer optimization problem, or the problem learning the weights that minimize the loss of our classifier. While many approaches have been proposed in the literature, we will focus on a heuristic called adversarial training, which has generally worked well in practice.
Ad Adversarial training Although solving the min-max optimization problem may seem daunting, classical result known as Danskin's 's theorem (Danskin 1967) says that the gradient of maximization problem is equal to the gradient of the objective evaluated at the optimum. For learning models that minimize the robust optimization problem from Equation ( this means that
SEAp.. Σ max f(I; + +0:00
-
V
(8)
where 8' (I;; arg max (a;;+0; 0). In other words, this means that in order to backpropagate through the robust optimization problem, we can solve the inner maximization and backpropagate through the solution. Adversarial training does this by empirically maximizing the inner problem with PGD adversary. Note that since the inner problem not solved exactly, Danskin's 's theorem does not strictly apply. However, in practice, adversarial training does seem to provide good empirical robustness, at least when evaluated against the Ep threat model it was trained against.
4 ADVERSARIAL TRAINING FOR MULTIPLE PERTURBATION MODELS
We can now consider the core of this work, adversarial training procedures against multiple threat models. More formally, let S represent set of threat models, such that p E S corresponds to the ép perturbation model Ap.. and let As = UpeS Ap.c be the union of all perturbation models in S. Note that the chosen for each ball is not typically the same, but we still use the same notation for simplicity, since the context will always make clear which bp-ball we are talking about. Then, the generalization of the robust optimization problem in Equation (T) to multiple perturbation models is
min Σ 8
max E(x; + 8: 8).
(9)
SEAS
4

---

Under review as a conference paper at ICLR 2020
The key difference is in the inner maximization, where the worst case adversarial loss is now taken over multiple Ey perturbation models. In order to perform adversarial training, using the same motivational idea from Danskin's 's theorem, can backpropagate through the inner maximization by first finding (empirically) the optimal perturbation,
arg max E(x + 6;0).
(10)
dEAs
To find the optimal perturbation over the union of threat models, we begin by considering straightfor- ward generalizations of standard adversarial training, which will use PGD to approximately solve the inner maximization over multiple adversaries.
4.1 SIMPLE COMBINATIONS OF MULTIPLE PERTURBATIONS
First, we study two simple approaches generalizing adversarial training to multiple threat models. These methods can perform reasonably well in practice and are competitive with existing approaches without relying on complicated architectures. While these methods work to some degree, we later find empirically that these methods do not necessarily minimize the worst-case performance, and can converge to unexpected tradeoffs between multiple threat models.
Worst-case perturbation One way to generalize adversarial training to multiple threat models is to use each threat model independently, and train on the adversarial perturbation that achieved the maximum loss. Specifically, for each adversary p E S, we solve the innermost maximization with an Ep PGD adversary to get an approximate worst-case perturbation 6p.
8p arg max f(x + 8;
(11)
dEA...
and then approximate the maximum over all adversaries as
8 = arg max {(x + 6p:
(12)
6p
When |S|
then this reduces standard adversarial training. Note that if each PGD adversary solved their subproblem from Equation [TT exactly, then this is exactly the optimal perturbation 8*.
PGD augmentation with all perturbations Another way to generalize adversarial training is to train on all the ad adversarial perturbations for all E S to form a larger adversarial dataset. Specifically, instead of solving the robust problem for multiple adversaries in Equation (9) we instead solve
min
max l(x; + 0; óEAp.e
(13)
by using individual é, PGD adversaries to approximate the inner maximization for each threat model. Again, this reduces to standard adversarial training when |S| 1.
While these methods work reasonably well in practice (which is shown later in Section 5). both approaches solve the inner maximization problem independently for each adversary, each individual PGD adversary is not taking advantage of the fact that the perturbation region is enlarged by other threat models. To take advantage of the full perturbation region, propose modification to standard adversarial training, which combines information from all considered threat models into a single PGD adversary that is potentially stronger than the combination of independent adversaries.
4.2 MULTI STEEPEST DESCENT
To create a PGD adversary with full knowledge the perturbation region, we propose an algorithm that incorporates the different threat models within each step of projected steepest descent. Rather than generating adversarial examples for each threat model with separate PGD adversaries, the core idea to create a single adversarial perturbation by simultaneously maximizing the worst case loss over all threat models at each projected steepest descent step. We call our method multi steepest descent (MSD), which can be summarized as the following iteration:
g(++1) PA,,. (g(t) + (5(()) for prrpe S
p
(1+1) arg max f(x + 6 8(t+1))
(14)
5

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Multi steepest descent for learning classifiers that are simultaneously robust to Ep attacks
for S
Input: classifier Sa, data æ, labels y
Parameters: fp; Op for S, maximum iterations T, loss function &
(10)
for t = O do
for p S do
of(+2)
Pam. (5(t) + +1p(8(1))
end for g(++1) = arg m {ff( + 0g oz+")),,)
end for return 8(7)
The key difference here is that at each iteration of MSD, we choose projected steepest descent direction that maximizes the loss over all attack models p E S, whereas standard adversarial training and the simpler approaches use comparatively myopic PGD subroutines that only use one threat model at a time. The full algorithm is in Algorithm [ and can be used as drop in replacement for standard PGD adversaries learn robust classifiers with adversarial training. We direct the reader to Appendix for a complete description of steepest descent directions and projection operators for loo. f2, and & norms|]
RESULTS
In this section, we present experimental results on using generalizations of adversarial training to achieve simultaneous robustness to loc fg. and & perturbations on the MNIST and CIFAR 10 datasets. Our primary goal is show that adversarial training can in fact be adapted to a union of perturbation models using standard architectures to achieve competitive results, without the pitfalls described by Schott (209)) Our results improve upon the state-of- thette- in three key ways. First, we can use simpler, standard architectures for image classifiers, without relying on complex architectures or input binarization. Second, our method able to learn a single MNIST model which is simultancously robust to all three threat models, whereas previous work was only robust against two at time. Finally, our method is easily scalable to datasets beyond MNIST, providing the first CIFAR1O model trained to be simultaneously robust against exc. l2, and é adversaries.
We trained models using both the simple generalizations of adversarial training to multiple adversaries and also using MSD. Since the analysis by synthesis model is not scalable to CIFAR1O, 10, we additionally trained CIFAR10 10 models against individual PGD adversaries to measure the changes and tradeoffs in universal robustness. We evaluated these models with broad suite of both gradient and non-gradient based attacks using Foolbox (the same attacks used by Schott et al. (2019), and also incorporated all the (GD-based adversaries discussed in this paper. All aggregate statisties that combine multiple attacks compute the worst case error rate over all attacks for each example.
Summaries of these results specific thresholds can be found in Tables [ and 2 where B-ABS and ABS refer to binarized and non-binarized versions of the analysis by synthesis models from Schott et al. 2019). Pe refers to model trained against PGD adversary with respect to the p- norm, Worst- PGD and PGD-Aug refer to models trained using the worst-case and data augmentation generalizations of adversarial training, and MSD refers to models trained using multi steepest descent. Full tables containing the complete breakdown of these numbers over all individual attacks used in the evaluation are in Appendix C We report the results against individual attacks and threat models for completeness. however note that the goal of all these algorithms is to minimize the robust optimization objective from Equation 9.. While there may be different implicit tradeoff' between individual threat
'The pure steepest descent step is inefficient since only updates one coordinate at time. It can be improved by taking steps on multiple coordinates, similar to that used in Tramer Bonch (onch) and is also explained in Appendix(A) A https///gggiuuuooo/////gggalaa/t
Rauber
2017)

---

Under review as a conference paper at ICLR 2020
Table I: Summary of adversarial accuracy results for MNIST (higher is better)
Worst PGD
PGD
Poc
P3
Pi
B-ABS
ABS
Aug
MSD
Clean Accuracy
99.1%
99.4%
98.9%
99%
99%
98.9%
99.1%
98.2%
Lo attacks (e 0.3)
90.3%
0.4%
0.0%
77%
8%
68.4%
83.7%
63.7%
l attacks (e 1.5)
45.3%
87.0%
70.3%
39%
80%
82.1%
75.0%
82.6%
4 attacks (c 12)
1.4%
43.4%
71.8%
82%
78%
54.6%
15.6%
62.3%
All Attacks
1.4%
0.4%
0.0%
39%
8%
53.7%
15.6%
58.7%
models, in the end. the most meaningful metric for measuring the effective performance is the robust optimization objective, or the performance against the union of all attacks.
5.1 EXPERIMENTAL SETUP
Architectures and hyperparameters For MNIST, we use four layer convolutional network with
two convolutional layers consisting of 32 and 64 5 x filters and units padding, followed by fully connected layer with 1024 hidden units, where both convolutional layers followed by 2 x 2 Max Pooling layers and ReLU activations (this is the same architecture used by Madry et al. (2018)). This is in contrast past work on MNIST, which relied per-class variational autoencoders to achieve robustness against multiple threat models (Schott et al.) 2019), which was also not easily scalable larger datasets. Since our methods have the same complexity training, they also easily apply to standard CIFARIO 10 architectures, and in this paper we use the well known pew-actieaaaeee aetivvaiviioon version of the ResNet1 8 architecture consisting of nine residual units with two convolutional layers each (He et al. 2016).
standard adversarial
complete description of the hyperparameters used in Appendix B with hyperparameters for PGD adversaries in Appendix [B.1] and hyperparameters for adversarial training in Appendix [B.2 All reported c are for images scaled be between the range [0, 1]. All experiments can be run on modern GPU hardware (e.g. single 1080ti).
Attacks used for evaluation To evaluate the model, we incorporate the attacks from Schott et al. (2019) as well our PGD based adversaries using projected steepest descent, however we provide short description here. Note that we exclude attacks based on gradient estimation, since the gradient for the standard architectures used here are readily available.
For Eo attacks, although we find the fxx PGD adversary to be quite effective, for completeness, additionally use the Foolbox implementations of Fast Gradient Sign Method (Goodfellow et al.) 2015), PGD adversary (Madry et al. 2018), and the Momentum Iterative Method (Dong et al. 2018)) 8).
For f2 attacks, in addition to the lg PGD adversary, we use the Foolbox implementations of the same PGD adversary, the Gaussian noise attack (Rauber et al.] 2017), the boundary attack (Brendel et al. 2017). DeepFool -Moosavi-Dezzoolli al.
| (016), the pointwise attack (Schott et al. 2019), DDN
based attack (Rony et al. 2018), and C&W attack (Carlini & Wagner 2017)
For E1 attacks, we use both the 4 PGD adversary as well as additional Foolbox implementations of to attacks at the same radius, namely the salt & pepper attack (Rauber et al.) (017) and the pointwise attack (Schott et al., 2019). Note that an e adversary with radius strictly stronger than an to adversary with the same radius, and so we choose to explicitly defend against & perturbations instead of the lo perturbations considered by |choot et al. (2019).
We make 10 random restarts for each of the evaluation results mentioned hereon for both MNIST and CIFARIO We encourage future work in this area to incorporate the same, since the success of all attacks, specially decision based or gradient free ones, is observed to increase significantly over restarts.
3 All attacks of Boundary Attack, which by default makes 25 trials per iteration and DDN hased Attack which does not benefit from the same owing to a deterministic initialization of 8.
subset of the first 1000 test examples with 10 random restarts, with the exception
RRssults are from Schott et al. (2019), which used an Eo threat model of the same radius and evaluated against t attacks. So the reported number here upper bound the f adversarial accuracy. Further, they evaluate
7

---

Under review as a conference paper at ICLR 2020
MSD
Worst-PGD
PGD-Augment
1.0
1.0
1.0
0.8
0.8
0.8
Accuracy
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.1
0.2
0.3
0.4
1
2
4
5
10
20
30
t. radius
12 radius
radius
Figure 2: Robustness curves showing the adversarial accuracy for the MNIST model trained with MSD, PGD-Aug, Worst-PGD against &x (left), & (middle), and & (right) threat models over a range of epsilon.
Table 2: Summary of adversarial accuracy results for CIFAR10 (higher is better)
Poo
P2
Pi
Worst-PGD
PGD-Aug
MSD
Clean accuracy
83.3%
90.2%
73.3%
81.0%
84.6%
81.7%
too attacks (c 0.03)
50.7%
28.3%
0.2%
44.9%
42.5%
47.6%
E2 attacks (e = 0.5)
57.3%
61.6%
0.0%
61.7%
65.0%
64.3%
e attacks 12)
16.0%
46.6%
7.9%
39.4%
54.0%
53.4%
All attacks
15.6%
27.5%
0.0%
34.9%
40.6%
46.1%
5.2 MNIST
We first present results on the MNIST dataset, which are summarized in Table (a more detailed breakdown over each individual attack is in Appendix C.I). While considered "casy" dataset, note that the previous state-of-the-ar result for multiple threat models on MNIST (and our primary comparison) is only able defend against two out of three threat models time (Schott al.) 2019) using comparatively complex variational autoencoder architectures. The model trained with MSD achieves the best performance against all attacks, achieving an error of 58.7% (individually 63.7%, 82.6%, and 62.3) % against the union of (foe lz, and fg) perturbations with radius (0.3, 1.5, 12). Complete robustness curves over range of epsilons over each threat model can be found in Figure 2 A comparison of our results with concurrent work (Tramèr & Boneh 2019) can be found in Appendix D
5.3 CIFARIO
Next, we present results on the CIFARIO dataset, which are summarized in Table (a more detailed breakdown over each individual attack is in Appendix C.2h. Our MSD approach reaches the best
performance against the union of attacks, and achieves 46.1% 1% (individually 47.6% 64.3%, 53.4%) (0.03, 0.5. 12). Interestingly, note that the P; model trained against an é PGD adversary is not very robust when evaluated against other attacks, even though it can defend reasonably well against the é, PGD attack in isolation (Table in Appendix CC22). Complete robustness curves over a range of epsilons over each threat model can be found in Figure
adversarial accuracy against the union of (fxc, l2, & perturbations of size
comparison our results with concurrent work (Tramèr Bonch 2019) can be found in Appendix D While adversarial defenses are generally not intended to defend against attacks outside of the threat model, we show some experiments exploring this aspect in Appendix
their model without restarts and the adversarial accuracy against all attacks is an upper bound based on the reported accuracies for the individual threat models. Finally, all ABS results were computed using numerical gradient estimation, since gradients readily available.
8

---

Under review as a conference paper at ICLR 2020
MSD
Worst-PGD
PGD-Augment
0.8
0.8
0.8
Accuracy
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.00
0.03
0.06 0.09
0
1
2
3
4
0
25
50
75 100
to radius
f2 radius
f1 radius
Figure 3: Robustness curves showing the adversarial accuracy for the CIFARIO model trained with MSD, PGD-Aug, Worst-PGD against &o (left), & (middle), and & (right) threat models over a range of epsilon.
On tradeoffss and variability of the simpler defenses One major drawback to the simpler methods for generalizing adversarial training to multiple threat models is their variability and unclear tradeoff's different settings. For example, MNIST reduce the robust optimization objective: the &oo threat model dominates the training process and we get suboptimal tradeoff between threat models which isn' robust to the union. Similarly, on CIFAR1O 10 we see that the worst-case approach for adversarial training also converges to model which has suboptimal robust performance against the union threat models. This highlights the inconsistency of the simpler generalizations of adversarial training: depending the dataset and the threat models, they may not ultimately minimize the robust optimization objective from Equation ()) and the tradeoffss may vary significantly with the problem setting. On the other hand, in both problem settings, we find MSD is consistent at finding more optimal tradeoff which minimizes the worst-cas cast- loss in the union of the threat models. As result, rather than using one of the simpler methods and convergence to a potentially unclear tradeoff between threat models, we recommend using MSD which directly minimizes the worst performance among the specified threat models.
that the data augmentation approach fails to
CONCLUSION
In this paper, we showed that adversarial training can be quite effective when training against union of multiple perturbation models. We compare two simple generalizations of adversarial training and an improved adversarial training procedure, multi steepest descent, which incorporates the different perturbation models directly into the direction steepest descent. MSD based adversarial training procedure is able outperform past approaches, demonstrating that adversarial training in fact learn networks that are robust to multiple perturbation models simultaneously (as long as they are included in the threat model) while being scalable beyond MNIST and using standard architectures.
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference Machine Learning, ICML 2018, July 2018a. URL wttp///wwwvv.rr//o//o abs/
1802.00420
Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing robust adversarial examples. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 284 293, Stockholmsmssan Stockholm Sweden, 10-15 Jul 2018b. PMLR. URL http: htthttp //proceeddngg.mlpprreesss/
athalye18b.html 8b.html
Wieland Brendel. Jonas Rauber, and Matthias Bethge. Decision-basee adversarial attacks: Reliable attacks against black-box machine learning models. arXiv preprint arXiv: arXiv:/712.000 17/2,04122 2017.

---

Under review as a conference paper at ICLR 2020
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 2017 IEEE Symposium on, pp. 39-57. IEFE, 2017.
Pin-Yr Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Ead: Elastic-net net attacks to deep neural networks via adversarial examples, 2017.
Francesco Croce and Matthias Hein. Provable robustness against all adversarial Ipprrturtoonssssssrraaattooooo for p21. CoRR, abs/1905.112133 2019. UtL.bttp:://axxi..orrr/// httpo//g//aa/19.1111111
John M Danskin. The theory of max-min and its application to weapons allocation problems, volume 5. Springer Science & Business Media, 1967.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
John Duchi, Shai Shalev-Shwarte Yoram Singer, and Tushar Chandra. Efficient projections onto the 11-ball for learning in high dimensions. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pp. 272-279, New York, NY. USA, 2008 ACM. ISBN 978- URL http:///........orr//011111
2-60558-205-4. doi: 10.1145/139015..11 1390191.
13901 56. 390191
Logan Engstrom, Andrew Ilyas, and Anish Athalye. Evaluating and understanding the robustness of adversarial logit pairing. arXiv preprint 1rXiv:1807.10277 2018.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http://
seciv.orgabbs/111222
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy A. Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. CoRR, abs/1810.122115 2018. URL
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Daniel Kang, Yi Sun, Tom Brown, Dan Hendryeks, and Jacob Steinhardt. Transfer of adversarial robustness between perturbation types. arXiv preprint vrXivv1905.01000 2019.
Harini Kannan, Alexey Kurakin, and Ian J. Goodfellow. Adversarial logit pairing. CoRR, abs/1803 #bbs/1003.0633,, 2018. URL
Guy Katz, Clark Barrett, David Dill, Kyle Julian, and Mykel Kochenderfer. Reluplex: An efficient smt solver for verifying deep neural networks. arXiv preprint arXivv 2017.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples the physical world. ICLR Workshop, 2017. http:/tttss///rxiioo/r////////
Jiajun Lu, Hussein Sibai, Evan Fabry, and David Forsyth. No need to worry about adversarial examples in object detection in autonomous vehicles. arXiv preprint 10Xhv:1707.33511, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learn- ing Representations, 2018. URL httttt/t/////// httpa//////pppeeeeeeeeeeeeeeerre httpa////////////////////aaa/ana wttp:eeewweeeeooooooeooooooooooe
Matthew Mirman, Timon Gehr, and Martin Vechev. Differentiable abstract interpretation for provably robust neural networks. In International Conference Machine Learming (ICML), 2018. URL
www.iicm.lc. httpeee/www...cmcccc/oooofffeeee /2t/////e///////// Bwt.01///ccheuuee 2os.uuutttessooeeeee222e22
Seyed-Mohsen Gboosvi--eezfooll: Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
10

---

Under review as a conference paper at ICLR 2020
Marius Mosbach, Maksym Andriushchenko, Thomas Trost, Matthias Hein, and Dietrich Klakow. Logit pairing methods can fool gradient- based attacks. arXiv preprint 1010:v:810.120000 2018.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks, In Security and Privacy (SP), 2016 IEEE Symposium on, pp. 582-597. IEEE, 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, ASIA CCS 17, pp. 506-519, New York, NY, USA, 2017. ACM. ISBN 978-1-4503-494444, doi: 10. 10.114573052973.3533000 URL
Aditi Raghunathan, Jacob Steinhardt, and Perey Liang. Certified defenses against adversarial
examples. In International Conference Learning Representations, 2018a. URL [ttpp:
Rb
Aditi Raghunathan, Jacob Steinhardt, and Percy S Liang. Semidefinite relaxations for cer- tifying robustness to adversarial examples. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems 31, pp. 10900-10910. Curran Associates, Inc.. 2018b. oeeaaeeeeoeniiiiiieeeeeeaaaaooos for relaxations
URL
cert
Jonas Rauber, Wieland Brendel, and Matthias Bethge. Foolbox: A python toolbox to benchmark the robustness of machine learning models. arXiv preprint arXiv:1707.04131, 2017. URL
http::/arzdv.org//ab3/
Jérôme Rony, Luiz G. Hafemann, Luiz S. Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and norm for efficient gradient-based 1.2 adversarial attacks and defenses. CoRR, abs/181 1.. 109600, 2018.
Lukas Schott, Jonas Rauber, Matthias Bethge, and Wieland Brendel. Towards the first adversarially robust neural network model on MNIST. In International Conference on Learning Representations, 2019. http:/tppp///pppeeeeee..eeeeeees foorumuu/////uu/ 2at.::unmnni--11115CCCCC3C
Leslie N Smith. A disciplined approach to neural network hyper-paramettes: Part l-lcarning rate, batch size, momentum, and weight decay. arXiv preprint arXiv: htt.:/803309933 2018
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations, 2019. URL
httpss:/cpeereeiew..eee wwwp://eee//ouuufffeeeooe for.mmid-diiii Forum?idd-HyGldiRRtt
Florian Tramèr and Dan Boneh. Adversarial training and robustness for multiple perturbations. arXiv preprint arxiv: /904. /3000, 2019.
Jonathan Uesato, Brendan O' Donoghuee Pushmeet Kohli, and Aaron van den Oord. Adversarial risk and the dangers of evaluating against weak attacks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings Stockholmsmssaan Stockholm Sweden, 10-15 Jul
of Machine Learning Research, pp. 2018. PMLR. /tte:t//////oop//oo//oeeeo pttt:::/pproceedngg..lt. wttpppeee//////e/aattaaeeee
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pp. 5283-52852000 5292,, 2018.
Eric Wong, Frank Schmidt, Jan Hendrik Metzen, and J. Zico Kolter. Scaling provable adversarial defenses. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 8410 8419. Curran Asso- ciates, Inc., 2018 URL adversarial fenses.pdf
8050 scalinceeee wownnnc--robblleee

---

Under review as a conference paper at ICLR 2020
Algorithm 2 Projection of some perturbation 8 E IR" onto the & ball with radius denote element-wise absolute value.
We use
to
Input: perturbation 8, radius
Sort 66 into y " 2 12 2 ... 2 7n
[n] : 7 =0}
P = max
-r
= (E 7i -) c)
2, == sign (,) max {7- 0} for =1...
return
A STEEPEST DESCENT AND PROJECTIONS FOR Los lz, AND & ADVERSARIES
In this section, we show what the steepest descent and projection steps are for lp adversaries for p E {oo, 2, 1]; these are standard results, but included for complete description of the algorithms. Note that this differs slightly from the adversaries considered in |Schott et al. (2019): while they used an ly adversary, we opted to use an & adversary with the same radius. The &u ball with radius is contained within & ball with the same radius, achieving robustness against an & adversary is strictly more difficult.
Lo space The direction steepest descent with respect the foo norm is (6) = sign(Vl(a + 0;)
(15)
and the projection operator onto Ax,
PA PAx., -...((-..(
(16)
l2 space The direction of steepest descent with respect the & norm is
Vl(x + 8:0)
02 [v[ +6:0)112
(17)
and the projection operator onto the fg ball around a is
8
P PA. (0) = max ca
(18)
4 space
The direction of steepest descent with respect to the & norm
U1 (6) a sign [Ol(x 5;0)
(19)
05
where
i = arg max Vi(r + 0),1
(20)
and ex- unit vector with one in position i*, Finally, the projection operator onto the ball,
PA. (0) arg min ||ō - 81
(21)
1 Se
can be solved with Algorithm 2 and we refer the reader Duchi al. (2008) for its derivation.
A.1 ENHANCED STEEPEST DESCENT STEP
Note that the steepest descent step for & only updates single coordinate per step. This can be quite inefficient, as pointed out by Tramèr & Boneh (201)). To tackle this issue, and also empirically improve the attack success rate, Tramer & Boneh 2019) instead select the top k coordinates according to Equation to update. In this work, we adopt a similar but slightly modified scheme: we randomly sample to be some integer within some range [k1, kz]. and update each coordinate with step size a' a/ We find that the randomness induced by varying the number of coordinates aids in avoiding the gradient masking problem observed by Tramer & Boneh (2019).
12

---

Under review as a conference paper at ICLR 2020
A.2 RESTRICTING THE STEEPEST DESCENT COORDINATE
The steepest descent direction for both the Eo and & norm end up selecting single coordinate direction to move the perturbation. However, if the perturbation is already at the boundary of pixel space (for MNIST, this is the range [0,1] for each pixel), then it's possible for the PGD adversary get stuck in a loop trying to use the same descent direction to escape pixel space. To avoid this, we only allow the steepest descent directions for these attacks to choose coordinates that keep the image in the range of real pixels.
B EXPERIMENTAL DETAILS
B.I HYPERPARAMETERS FOR PGD ADVERSARIES
In this section, we describe the parameters used for all PGD adversaries in this paper.
MNIST
The too adversary used step size a = 0.01 within radius of e = 0.3 for 50 iterations.
The E2 adversary used step size 0.1 within radius of e 1.5 for 100 iterations.
The f adversary used a step size of a 0.05 within radius of e for 50 iterations. By default the attack is run with two restarts, once starting with d = and once by randomly initializing d in the allowable perturbation ball. b 22 = 20 as described in A.I
The MSD adversary used step sizes of radius = (0.3, 1.5, 12) for 100 iterations.
0.2,1, for the (Lou, E2 (1) directions within
At test time, we increase the number of iterations to (100, 200, 100) for (Loss E E1).
CIFAR10 The Loo adversary used a step size a 0.003 within a radius of c = 0.03 for 40 iterations.
The f2 adversary used step size a = 0.05 within radius of 0.5 for 50 iterations.
The & adversary used step size 0.1 within radius of 12 for 50 iterations. kn 5, 1/ 20 as described in A.
The MSD adversary used step sizes of (0.003, 0.05, 0.05) for the (Lo,, L1) directions within a radius of = (0.03, 0.3, 12) for 50 iterations. Note that the MSD model trained for & radius of 0.3 in fact robust higher radius of 0.5.
B.2 TRAINING HYPERPARAMETERS
In this section, we describe the parameters used for adversarial training. For all the models, we used the SGD optimizer with momentum 0.9 and weight decay 5 10-4.
MNIST We train the models to maximum of 20 epochs. We used variation of the learning rate schedule from Smith (2018), which piecewise linear from 0.1 over the first 7 epochs, down to 0.001 over the next 8 epochs, and finally back down to 0.0001 in the last 5 epochs.
We used variation of the learning rate schedule from Smith 2018) to achieve super- convergence 50 epochs, which is piecewise linear from 0.1 over the first 20 epochs, down to 0,005 over the next 20 epochs, and finally back down to 0 in the last 10 epochs.
CIFARIO
C EXTENDED RESULTS
Here, we show the full tables which break down the overall adversarial error rates over individual attacks for both MNIST and CIFAR10 10, along with robustness curves for all models in the paper.
13

---

Under review as a conference paper at ICLR 2020
Table 3: Summary of adversarial accuracy results for MNIST
Worst PGD
PGD
Poo
P2
Pi
B-ABS
ABS
Aug
MSD
Clean Accuracy
99.1%
99.4%
98.9%
99%
99%
98.9%
99.1%
98.2%
PGD-L
90.3%
0.4%
0.0%
68.4%
83.7%
63.7%
FGSM
94.9%
68.6%
6.4%
85%
34%
82.4%
90.9%
81.8%
PGDD-Follbox
92.1%
8.5%
0.1%
86%
13%
72.1%
85.7%
67.9%
MIM
92.3%
14.5%
0.1%
85%
17%
73.9%
87.3%
71.0%
6o attacks (c= 0.3)
90.3%
0.4%
0.0%
77%
8%
68.4%
83.7%
63.7%
PGD-62
83.8%
87.0%
70.8%
85.3%
87.9%
84.2%
PGD-Foolbox
93.4%
89.7%
74.4%
63%
87%
86.9%
91.5%
86.9%
Gaussian Noise
98.9%
99.6%
98.0%
89%
98%
97.4%
99.0%
97.8%
Boundary Attack
52.6%
92.1%
83.0%
91%
83%
86.9%
79.1%
88.6%
DeepFool
95.1%
92.2%
76.5%
41%
83%
87.9%
93.5%
87.9%
Pointwise Attack
74.3%
97.4%
96.6%
87%
94%
92.7%
89.0%
95.1%
DDN
82.7%
87.0%
70.8%
85.1%
85.2%
84.3%
CWL2
88.2%
88.1%
75.5%
85.2%
87.5%
85.1%
82 attacks (E = 1.5)
45.3%
87.0%
70.3%
39%
80%
82.1% %
75.0%
82.6%
PGD-&
51.8%
49.9%
71.8%
66.5%
57.4%
64.8%
Salt & Pepper
55.5%
96.3%
95.6%
96%
95%
86.4%
71.9%
92.2%
Pointwise Attack
2.4%
66.4%
85.2%
82%
78%
60.1%
17.1%
72.8%
& attacks (c t 12)
1.4%
43.4%
71.8%
82%
78%
54.6%
15.6%
62.3%
All attacks
1.4%
0.4%
0.0%
39%
8%
53.7%
15.6%
58.7%
C.I MNIST RESULTS
Expanded table of results contains the full table of results for attacks on all models on the MNIST dataset. All attacks restarts, with the exception Boundary Attack, which by default makes trials per iteration, and DDN attack, which does not benefit from restarts owing the results for B-ABS and ABS models are from [ccoott et al. (2019). which uses gradient estimation techniques whenever gradient is needed, and the robustness against all attacks for B-ABS and ABS is an upper bound based on the reported results. Further, these models are not evaluated with restarts, pushing the reported results even higher than actual.
subset of the first 1000 test examples with 10 random
deterministic starting point. Note that
C.2 CIFARIO RESULTS
Expanded table of results Table contains the full table of results for all attacks on all models on the CIFARIO 10 dataset. All attacks were run on subset of the first 1000 test examples with random restarts, with the exception of Boundary Attack, which by default makes 25 trials per iteration, and DDN attack, which does not benefit from restarts owing to deterministic starting point. Further note that salt & pepper and pointwise attacks in the l section technically lu attacks, but produce perturbations in the & ball. Finally, is clear here that while the training against an & PGD adversary defends against said PGD adversary, it does not
transfer robustness against other attacks.
D COMPARISON WITH CONCURRENT WORK
In this section we compare the results of our trained MSD model with that Tramèr & Boneh (2019), who study the theoretical and empirical trade-offs of adversarial robustness various settings when defending against multiple adversaries. Training methods presented by them in their comparisons, namely Advavg and Adumaz closely resemble the naive approaches discussed this paper: PGD- Aug and Worst-PGD respectively. We the results is from their work, and additionally compare the position of our MSD models at the revised thresholds used by Tramèr & Boneh (2019) without specially retraining them.
The results of Tables 5 and show that the relative advantage of MSD over naive techniques does hold up. While do make comparison the most relevant concurrnt work for completeness, the
14

---

Under review as a conference paper at ICLR 2020
Table 4: Summary of adversarial accuracy results for CIFARIO
Poc
P2
P1
Worst-PGD
PGD-Aug
MSD
Clean accuracy
83.3%
90.2%
73.3%
81.0%
84.6%
81.7%
PGD-Lc
50.3%
48.4%
29.8%
44.9%
42.8%
49.8%
FGSM
57.4%
43.4%
12.7%
54.9%
51.9%
55.0%
PGD-Foolbox
52.3%
28.5%
0.6%
48.9%
44.6%
49.8%
MIM
52.7%
30.4%
0.7%
49.9%
46.1%
50.6%
So attacks (E 0.03)
50.7%
28.3%
0.2%
44.9%
42.5%
47.6%
PGD-22
59.0%
62.1%
28.9%
64.1%
66.9%
66.0%
PGD-Foolbox
61.6%
64.1%
4.9%
65.0%
68.0%
66.4%
Gaussian Noise
82.2%
89.8%
62.3%
81.3%
84.3%
81.8%
Boundary Attack
65.5%
67.9%
2.3%
64.4%
69.2%
67.9%
DeepFool
62.2%
67.3%
0.9%
64.4%
67.4%
65.7%
Pointwise Attack
80.4%
88.6%
46.2%
78.9%
83.8%
81.4%
DDN
60.0%
63.5%
0.1%
64.5%
67.7%
66.2%
CWL2
62.0%
71.6%
0.1%
66.9%
71.5%
68.7%
E2 attacks (E - 0.05)
57.3%
61.6%
0.0%
61.7%
65.0%
64.3%
PGD-4
16.5%
49.2%
69.1%
39.5%
54.0%
53.4%
Salt & Pepper
63.4%
74.2%
35.5%
75.2%
80.7%
75.6%
Pointwise Attack
49.6%
62.4%
8.4%
63.3%
77.0%
72.8%
f attacks (<= 12)
16.0%
46.6%
7.9%
39.4%
54.0%
53.4%
All attacks
15.6%
27.5%
0.0%
34.9%
40.6%
46.1%
Table 5: Comparison with contemporary work on MNIST (higher is better). Results for all models except MSD are taken as is from Tramer & Boneh (0199
Vanilla
Adu
Adv
Adua
Adwang
Adumaa
MSD
Clean accuracy
99.4%
99.1% 1%
98.9%
98.5%
97.3%
97.2%
98.2%
foo attacks (e = 0.3)
0.0%
91.1%
0.0%
0.4%
76.7%
71.7%
63.7%
8 attacks 2.0)
12.4%
50.6%
71.8%
58.3%
56.0%
67.4%
E attacks (e 10)
8.5%
11.3%
78.5%
68.0%
53.9%
62.6%
70.0%
All attacks
0.0%
6.8%
0.0%
0.4%
49.9%
52.4%
60.9%
following differences can bias the robust accuracies reported for the MSD models relatively lower than expected (and correspondingly, the robust accuracies reported for the other models are relatively higher than expected):
Use of random restarts: We observe our experiments that using up to 10 restarts for all attacks leads to decrease in model accuracy from 5 to 10% across all models. Tramer & Bonch do not mention restarting their attacks for these models and so the results for models apart from MSD in Tables could potentially be lowered with random restarts.
2. Different training and testing thresholds:
The MSD model for the MNIST dataset was trained at (0.3, 1.5, 12) for the loc. E2, E perturbation balls respectively, while Tramer & Boneh (2019) tested at c = (0.3, 2.0, 10). This may lower the robust accuracy at these thresholds for the MSD model, since
not trained for that particular threshold. Likewise, the MSD model for CIFAR 10 was also trained at (0.03, 0.05, 12) for the Eoon f2. L perturbation balls respectively, while Tramer & Boneh 2019 tested
O, 2000 255
255
3. Different perturbation models: For the CIFARIO results in Table 6 Advavg & Adumaa models trained and tested only for & and foc adversarial perturbations, whereas the MSD model is robust to the union of f,, l and fao. achieving a much harder task.
4. Larger Suite of Attacks Used: The attacks used by Tramèr & Boneh are PGD, EAD Chen et al. (017) and Pointwise Attack (Schott et al., 2019) for ε1; PGD, C&W (Carlini & Wagner 2017) and Boundary Attack (Brendel et al. 2017) for lz; and PGD for foo adversaries. We use a more expansive suite of attacks as shown in Appendix C Some the attacks like DDN, which proved to be strong adversaries in most cases, not considered
15

---

Under review as a conference paper at ICLR 2020
Table 6: Comparison with contemporary work on CIFARIO (higher better). Results for all models
except MSD are taken as is from Tramer& & Boneh 2019)
Vanilla
Adeso
Advı
Aduang
Aduman
MSD
Clean accuracy
95.7%
92.0%
90.8%
91.1%
91.2%
82.1%
Loo attacks (e
0.0%
71.0%
53.4%
64.1%
65.7%
65.6%
( attacks (c
2000
0.0%
16.4%
66.2%
60.8%
62.5%
62.0%
233
All attacks
0.0%
16.4%
53.1%
59.4%
61.1%
61.7%
Table Performance on CIFAR-I0-C
Accuracy
Standard model
66.0%
Poo
75.0%
P2
82.7%
PI
57.8%
Worst-PGD
70.8%
PGD-Aug 76.8%
MSD
74.2%
by Tramer & Boneh (219)) and thus were only used to attack the MSD models in Tables and
6
E ATTACKS OUTSIDE THE THREAT MODEL
In this section, we present some additional experiments exploring the performance of our model on attacks which lie beyond the threat model. Note that there is no principled reason why we would believe this to the case (as most adversarial defenses tend not generalize beyond the threat model defended against), and this is presented for exploratory reasons.
Common corruptions We measure the performance of all the models on CIFAR-10-C, which is CIFAR1O benchmark which has had common corruptions applied it (e.g. noise, blur, and compression). We report the results in Table 7 We find that that, apart from the P1 model, the rest achieve some improved robustness against these common corruptions above the standard CIFAR10 model.
Defending against & and &oo and evaluating on & We also briefly study what happens when one trains against E and for threat models, while evaluating against the f2 adversary. Specifically, we take the MSD approach on MNIST and simply remove the &2 adversary from the threat model. This results in model which has its L and foo robust performance against a PGD adversary drop by 1 % and its &2 robust performance against PGD adversary (which was not trained for) drops by 2% in comparison to the original MSD approach all three threat models.
As a result, we empirically observe that including the & threat model in this setting actually improved overall robustness against all three threat models. Unsurprisingly the & performance drops some degree, but the model does not lose all of its robustness.
16


Reviews:
[{"review_id": "SklAX8hm5H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to do adversarial training on multiple L_p norm perturbation models simultaneously, to make the model robust against various types of attacks. \n\n[Novelty] I feel this is just a natural extension of adversarial training. If we define the perturbation set in PGD to be S, then in general S can be union of perturbation set of several L_p norm, and the resulting algorithm will be MSD (everytime you do a gradient update and then find the worst case projection in S). It would be interesting to study the convergence of this kind of algorithms, since S is no longer convex, the projection is trickier to define. Unfortunately this is not discussed in the paper. \n\nIn terms of experiments, this is an interesting data point to show that we can have a model that is (weakly) robust to L1, L2 and Linf norms simultaneously. However, the results are not surprising since there's more than 10% performance decreases compared to the original adversarial training under each particular attack. So it's still not clear whether we can get a model that simultaneously achieves L1, L2, Linf robust error comparable to original PGD training. \n\n[Performance] \n- It seems MSD is not always better than others (worst PGD and PGD Aug). For MNIST, MSD performs poorly on Linf norm and it's not clear why.\n- There's significant performance drop in clean accuracy, especially MSD on MNIST data. \n\n[Suggestions]\n- As mentioned before, studying the convergence properties of the proposed methods will be interesting. \n- It will be interesting if you can train on a set of perturbation models and make it also robust to another perturbation not in the training phase. For instance, can we apply the proposed method to L{1,inf} in training and generalize to L2 perturbation? \n\n=====\nThanks for the response. I still have concerns about novelty so would like to keep my rating unchanged. \n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SkexvzFOKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper: The paper describes adversarial training aiming to build models that are robust to multiple adversarial attacks - with L_1, L_2 and L_inf norms.  The method is a based on adversarial training against a union of adversaries. That union is created by taking (projected) gradient steps like PGD (Kurakin 2017), but choosing the maximal loss over GD steps for L1, L2, L_inf at each step. \n\nStrengths: The topic is trendy and interesting. The proposed algorithm is simple and easy to implement. The experimental results demonstrate improvement over several baselines. \n\nWeaknesses: \n-- I am missing a more systematic comparisons to baseline defenses in the experiments. Figures 2 and 3 should have shown the accuracy as a function of radius also for PGD-aug, PGD-worst, Schott et al. Also what about comparisons to the latest SoTA defenses, e.g. recent baselines from from  \nwww.robust-ml.org/defenses/. \n\n-- An implicit expectation from this paper is that it addresses the key issue of  \"Defend against one attack but face a different attack\". The paper could have done more to advance our understanding of this issue. Specifically: \n\nThe approach improves over baselines for the \"all attacks\" mode, but under-performs compared with PGDaug and PGDworst when attacked with a single norm (Tab 1). \n\nWhile this is expected and probably cannot be avoided, it leaves the reader with an unclear conclusion about risk tradeoffs. It would have been useful to clarify the regime of mixtures of attacks where the various approaches are best. For instance, if one uses a of mix attack samples from the three norms, what mixtures would it be best to defend using MSD, wand what mixtures would it be best to use PGD-aug? or ABS? \n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "HylxMAzTtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper adversarially trains models against l_p norms where p is of there different values. They then propose a method which does somewhat better than the obvious way of adversarially training against more than one l_p perturbation.\nThe motivation for the paper is limited, in that they suggest previous works have suggested adversarial training itself \"overfits\" to the given l_p norm. This isn't surprising that it works, since the straightforward baseline works. They make it seem surprising by suggesting that ABS suggested adversarial training is doomed and cannot provide robustness to l_1, l_2, l_\\infty norms simultaneously. The other motivation is that this is a step toward studying an expanded threat model, but the authors have not demonstrated that the learned representations are any bit more robust to common corruptions (could the authors show the generalization performance on CIFAR-10-C or generalization to unforeseen corruptions?). Without further evidence, we are left to believe this only helps for this narrow threat model. Overall the paper is deficient in creativity and generality, so I vote for rejection.\n\nSmall comments:\n\n> take more time than a single norm,  it is a step closer towards the end goal of truly robust models, with adversarial robustness against all perturbations.\nPlease show model performance on CIFAR-10-C since if the model is more robust, it should hopefully be more robust to stochastic adversaries.\n\n> has claimed that adversarial training “overfits” to the particular type of perturbation used to generate the adversarial examples\nWouldn't this be that l_\\infty training fits specifically to l_\\infty examples, not that robust optimization cannot handle more than one norm at a time? Who is claiming that?\n\n> First, we show that even simple aggregations of different adversarial attacks can achieve competitive universal robustness against multiple perturbations models without resorting to complex architectures.\nI am not sure this was in doubt. The phrase \"universal robustness\" is misleading.\n\nHow were the budgets chosen for l_2 and l_1? Those values seem small.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks to the authors for submitting the paper and providing further explanations and experiments. This paper aims to ensure robustness against several perturbation models simultaneously. While the authors' response has addressed several issues raised by the reviewers, the concern on the lack of novelty remains. Overall, there is not enough support among the reviewers for the paper to be accepted.
Decision: Reject