MMA Training: Direct Input Space Margin Maximization through Adversarial Training
Authors: Gavin Weiguang Ding
Yash Sharma
Kry Yik Chau Lui
Ruitong Huang
Venue:  (2020)
Keywords: adversarial robustness
perturbation
margin maximization
deep learning

Abstract: We study adversarial robustness of neural networks from a margin maximization perspective, where margins are defined as the distances from inputs to a classifier's decision boundary.
Our study shows that maximizing margins can be achieved by minimizing the adversarial loss on the decision boundary at the "shortest successful perturbation", demonstrating a close connection between adversarial losses and the margins. We propose Max-Margin Adversarial (MMA) training to directly maximize the margins to achieve adversarial robustness. 
Instead of adversarial training with a fixed $\epsilon$, MMA offers an improvement by enabling adaptive selection of the "correct" $\epsilon$ as the margin individually for each datapoint. In addition, we rigorously analyze adversarial training with the perspective of margin maximization, and provide an alternative interpretation for adversarial training, maximizing either a lower or an upper bound of the margins. Our experiments empirically confirm our theory and demonstrate MMA training's efficacy on the MNIST and CIFAR10 datasets w.r.t. $\ell_\infty$ and $\ell_2$ robustness.

Published as a conference paper at ICLR 2020
MMA TRAINING: DIRECT INPUT SPACE MARGIN MAXIMIZATION THROUGH ADVERSARIAL TRAINING
Gavin Weiguang Ding', Yash Sharma Kry Yik Chau Lui', Ruitong Huang Borealii Al PUniversity of Tuebingen MMax Planck Institute for Intelligent Systems
ABSTRACT
We study adversarial robustness of neural networks from a margin maximization perspective, where margins defined the distances from inputs to a classifier's decision boundary. Our study shows that maximizing margins can be achieved by minimizing the adversarial loss the decision boundary at the "shortest success- ful perturbation", demonstrating a close connection between adversarial losses and the margins. We propose Max-Margin Adversarial (MMA) training to di- rectly maximize the margins to achieve adversarial robustness. Instead of adver- sarial training with a fixed 6, MMA offers an improvement by enabling adaptive selection of the "correct" as the margin individually for each data point. In ad- margin
dition, we rigorously analyze adversarial training with the perspective maximization, and provide an alternative interpretation for adversarial training, maximizing either a lower or an upper bound of the margins. Our experiments empirically confirm our theory and demonstrate MMA training's s efficacy the MÑIST and CIFARIO datasets w.r.t. for and & robustness. Code and models are available https: //gittttuuuut com/ Borea 1 SAT /mma tr irrinnngg inng..
INTRODUCTION
Despite their impressive performance on various learning tasks, neural networks have been shown to be vulnerable to adversarial perturbations (Szegedy et al., 2013; Big- gio et al., 2013). An artificially constructed imperceptible perturbation can cause a significant drop in the prediction accuracy of an otherwise accurate network. The level of distortion is measured by the magnitude of the perturba- tions (e.g. in loo E2 norms), i.c. the distance from the original input to the perturbed input. Figure 1 shows an example, where the classifier changes its prediction from panda to bucket when the input is perturbed from the blue sample point to the red one. Figure also shows the nat- ural connection between adversarial robustness and the margins of the data points, where the margin is defined as the distance from a data point to the clas- sifier's decision boundary. Intuitively, the margin of data point is the minimum distance that has to be perturbed to change the classifier's prediction. Thus, the larger the margin is, the farther the distance from the input to the decision boundary is, the more robust the classifier is w.r.t. this input.
shortest successful perturbation
perturbed image
margin
bucket
decis panda boundary
original image
perturbation magnified times
Illustration of decision bound- ary, margin, and shortest successful per- turbation on application of an adversar- ial perturbation.
Figure 1:
Although naturally connected to adversarial robustness, "directly" maximizing margins has not yet been thoroughly studied in the adversarial robustness literature. Instead, the method of minimax adversarial training (Huang et al., 2015; Madry et al., 2017) is arguably the most common defense to adversarial perturbations due to its effectiveness and simplicity. Adversarial training attempts to minimize the maximum loss within fixed sized neighborhood about the training data using projected gradient descent (PGD). Despite advancements made in recent years (Hendrycks et al., 2019; Zhang et al., 2019a; Shafahi et al., 2019: Zhang et al., 2019b; Stanforth et al., 2019; Carmon et al., 2019), adversarial training still suffers from fundamental problem, the perturbation length e has to be set and is fixed throughout the training process. In general, the setting of e is arbitrary, based assumptions whether perturbations within the defined ball are "imperceptible" or not. Recent work (Guo et al., 2018: Sharma et al., 2019) has demonstrated that these assumptions do not
Work done Borealis AI.

---

Published as a conference paper at ICLR 2020
consistently hold true, commonly used settings assumed to only allow imperceptible perturbations in fact do not. c is set too small, the resulting models lack robustness, if too large, the resulting models lack in accuracy. Moreover, individual data points may have different intrinsic robustness, the variation in ambiguity in collected data is highly diverse, and fixing one e for all data points across the whole training procedure is likely suboptimal.
Instead of improving adversarial training with a fixed perturbation magnitude, we revisit adversarial robustness from the margin perspective, and propose Max-Margin Adversarial (MMA) training for "direct" imput margin maximization. By directly maximizing margins calculated for each data point, MMA training allows for optimizing the "current robustness" of the data, the "correct" at this point in training for each sample individually, instead of robustness w.r.t. a predefined magnitude.
While it is intuitive that one can achieve the greatest possible robustness by maximizing the margin of classifier, this maximization has technical difficulties. In Section 2, we overcome these difficul- ties and show that margin maximization can be achieved by minimizing classification loss w.r.t model parameters, at the "shortest successful perturbation". This makes gradient descent viable for margin maximization, despite the fact that model parameters are entangled in the constraints.
We further analyze adversarial training (Huang al., 2015; Madry al., 2017) from the perspective of margin maximization in Section We show that, for each training example, adversarial training with fixed perturbation length e is maximizing a lower (or upper) bound of the margin, if is smaller (or larger) than the margin of that training point. As such, MMA training improves adversarial training. in the sense that it selects the "correct" c, the margin value for each example.
Finally in Section 4, we test and compare MMA training with adversarial training on MNIST and CIFARIO w.r.t. foc and lg robustness. Our method achieves higher robustness accuracies on average under a variety of perturbation magnitudes, which echoes its goal of maximizing the average margin. Moreover, MMA training automatically balances accuracy vs robustness while being insensitive to its hyperparameter setting, which contrasts sharply with the sensitivity of standard adversarial training to its fixed perturbation magnitude. MMA trained models not only match the performance of the best adversarially trained models with carefully chosen training e under different scenarios, also matches the performance of ensembles of adversarially trained models.
In this paper, focus our theoretical efforts the formulation for directly maximizing the input space margin, and understanding the standard adversarial training method from margin maxi- mization perspective. We focus our empirical efforts on thoroughly examining our MMA training algorithm, comparing with adversarial training with fixed perturbation magnitude.
1.1 RELATED WORKS
Although not often explicitly stated, many defense methods are related to increasing the margin. One class uses regularization constrain the model's Lipschitz constant (Cisse al., 2017; Ross & Doshi Velez, 2017; Hein & Andriushchenko, 2017; Sokolic et al., 2017; Tsuzuku et al., 2018), thus samples with small loss would have large margin since the loss cannot increase too fast. If the Lipschitz constant is merely regularized at the data points, it is often too local and not accurate in neighborhood. When globally enforced, the Lipschitz constraint on the model is often too strong that it harms accuracy. So far, such methods have not achieved strong robustness. There are also efforts using first-order approximation to estimate and maximize input space margin (Matyasko Chau, 2017; Elsayed et al., 2018; Yan et al., 2019). Similar to local Lipschitz regularization, the reliance on local information often does not provide accurate margin estimation and efficient maximization. Such approaches have also not achieved strong robustness far. Croce et al. (2018) aim to enlarge the linear region around input example, such that the nearest point, to the input and on the decision boundary, is inside the linear region. Here, the margin can be calculated analytically and hence maximized. However, the analysis only works on ReL. networks, and the implementation so far only works on improving robustness under small perturbations. We defer some detailed discussions on related works to Appendix B, including a comparison between MMA training and SVM.
1.2 NOTATIONS AND DEFINITIONS
We focus on K-class classification problems. Denote S = {a., y1} as the training set of input- label data pairs sampled from data distribution D. We consider the classifier a score function
2

---

Published as a conference paper at ICLR 2020
fo()) ................................ (x)), parametrized by 6, which assigns score fz( to the i-th class. The predicted label of is then decided by arg max; fā(x).
Let Lg (z, 3) = II(@ + y) be the 0-1 loss indicating classification error, where I[() is the indicator function. For an input (x, we define its margin w.r.t. the classifier fa(-) as:
(z, y) = 1 = min || s.t. à: Lg (I + 6, y) =1
(I)
where 8 arg Ln || the "shortest successful perturbation"". We give an equivalent
definition of margin with the "logit margin loss" (I, y) maxjyy f/ - 1 (zx). The level set (a: 71: y) 0} corresponds the decision boundary of class y. Also, when (,, v) < O, the classification is correct, and when L.M (zz y) 2 O, the classification is wrong. Therefore, we can define the margin in Eq. (1) in an equivalent way by (-) as:
(2 v) || min |ö|| s.t. 8 L (x: + 0,
where 8 = arg min (2+8,))) 22 ||ō]| is again the "shortest successful perturbation". For the rest of the paper, we use the term "margin" denote dg(x,,) y) in Eq. (2). For other notions of margin, we will use specific phrases, e.g. "SLM--arrgi" or "logit margin."
2 MAX-MARGIN ADVERSARIAL TRAINING
We propose to improve adversarial robustness by maximizing the average margin of the data distri- bution D, called Max-Margin Adversarial (MMA) training, by optimizing the following objective:
m 8
Σ
max (0, dmax - do( + 3 Σ U1)b
(3)
VES
jES,
where St = {i (F., v.) < 0} is the set of correctly classified examples, S = {i D (2x, 21) 2 0} is the set of wrongly classified examples, J-(-) regular classification loss function, e.g. cross entropy loss, dI v1) is the margin for correctly classified samples, and / is the coefficient for balancing correct classification and margin maximization. Note that the margin do( y1) is inside the hinge loss with threshold dmax (a hyperparameter), which forces the learn- ing to focus on the margins that are smaller than dmax- Intuitively, MMA training simultaneously minimizes classification loss wrongly classified points in S and maximizes the margins of cor- rectly classified points in (æ(
(i, Wi) until it reaches dmax- Note that we do not maximize margins on wrongly classified examples. Minimizing the objective in Eq. (3) turns
be technical chal- lenge. While VoJo Uj) can be easily computed by standard back propagation, computing the gradient of de (z y:) needs some technical developments.
In the next section, we show that margin maximization can still be achieved by minimizing a classifi- cation loss w.r.t. model parameters, at the "shortest successful perturbation". For smooth functions, a stronger result exists: the gradient of the margin w.r.t. model parameters can be analytically calculated, scaled gradient of the loss. Such results make gradient descent viable for margin maximization, despite the fact that model parameters are entangled in the constraints.
2.1
MARGIN MAXIMIZATION
Recall that
dz v) = ||8* mmin |8|| s.t. 8: LLM + 8. 2 0.
Note that the constraint of the above optimization problem depends on model parameters, thus mar- gin maximization is max-min nested optimization problem with a parameter-dependent constraint in its inner minimization.? Computing such gradients for a linear model is easy due to the existence of its closed- form solution, e.g. SVM, but it is not so for general functions such as neural networks.
The next theorem provides a viable way to increase (x(x, y). Theorem 2.1. Gradient descent on (x u) w.r.t. 0 with proper step size increases (x, v). where 5t = arg min 1 (x++6 ||0| is the shortest successful perturbation given the current
Since the scores (fd (z)..... S (x)) output by So are also called logits in neural network literature.
In adversarial training (Madry al., 2017), the constraint the inner max does NOT have such problem.
3

---

Published as a conference paper at ICLR 2020
the loss function before an update, L(,, 0b)
the sheetest successful perturbation
dy (xr y)
after updating the loss function, 81
the function after update, L(-,0))
the original data point, Tt.
update the shortest successfuul pertutbation 6 an update at the fixed C
the shortest suocassful perturbation before updating the loss function, og
Before making update
66
61
x
87
20
2
%
(b) After an update at the "shortest successful perturbation", the margin increases.
(c) Afler an update al the fixed Case the margin increnses.
(d) After an update the fixed Case the margin decreases.
Figure 2: A 1-D example on how margin is affected by decreasing the loss at different locations.
Theorem 2.1 summarizes the theoretical results, where show separately later
1) how to calculate the gradient of the margin under some smoothness assumptions;
2) without smoothness, margin maximization can still be achieved by minimizing the loss at the shortest successful perturbation.
Calculating gradients of margins for smooth loss and norm: Denote L + 8, by L(ō, for brevity. It is easy to see that for a wrongly classified example (x. y), 8 is achieved at 0 and thus Vadg (I. 3) O. Therefore we focus on correctly classified examples. Denote the Lagrangian as Lo A) = || + AT.(,, 0). For a fixed 0, denote the optimizers of Lo X) by 5" and X The following theorem shows how to compute Vodo(x.1) y).
Proposition 2.1. Let e(ō) [[ Given fixed 0, assume that 8* is unique, E(0) and L(8, 0) are
8²e((") x
116"
8L(6" .9)
C² functions in neighborhood of (8*00) and the matrix
06
26
06
aL18" .)
0
88
full rank, then
(e((8))
OL .0)
OL(8",07
05
05
Vedo(I, u) = C(0, y)
where C(O,, 20 v,
is a scalar.
00
0L(6*
88
Remark 2.1. By Proposition 2.1, the margin's 's gradient w.r.. to the model parameter 8 is propor- tional to the loss' gradient w.rt. @ at 8*, the shortest successful perturbation. Therefore to perform gradient ascent on the margin, we just need to find 8* and perform gradient descent on the loss.
Margin maximization for non-smooth loss and norm: Proposition 2.1 requires the loss function and the norm to be C2 at 8*. This might not be the case for many functions used in practice, e.g. ReLU networks and the for norm. Our next result shows that under weaker condition of directional differentiabilityy (instead of C²), learning 9 to maximize the margin can still be done by decreasing L(8*, 0) w.r.t. 0, 0 Og. Due space limitations, we only present informal statement here. Rigorous statements can be found in Appendix A.2.
Proposition 2.2. Let 8* be unique and L(ö, 0) be the loss of a deep ReLU network. There exists some direction T in the parameter space, such that the loss L(ō, 0) can be reduced in the direction of Furthermore, by reducing L(8, 6=á., the margin also guaranteed to be increased.
Figure illustrates the relationship between the margin and the adversarial loss with an imaginary example. Consider a I-D example in Figure (a), where the input example is a scalar. We perturb x in the positive direction with perturbation 8. As we fix (x, y). we overload L(5, 8) = L.M + 8, y), which is monotonically increasing on 5, namely larger perturbation results in higher loss. Let L(. ()) (the dashed curve) denote the original function before an update step, and og arg argmin mi ((5.9))20 [|8]] denote the corresponding margin (same as shortest successful perturbation in
4

---

Published as a conference paper at ICLR 2020
ID). As shown in Figure 2 (b), the parameter is updated to 01 such that L( 01) is reduced, the new margin of = arg 148,0 ||ö]| is enlarged. Intuitively, reduced value of the loss at the shortest successful perturbation leads an increase in margin.
2.2 STABILIZING THE LEARNING WITH CROSS ENTROPY SURROGATE. LOsS
In practice we find the gradients of the "logit margin loss" L to be unstable. The piecewise nature of the L loss can lead to discontinuity of its gradient, causing large fluctuations on the boundary between the pieces. It also does not fully utilize information provided by all the logits.
In our MMA algorithm, we instead the "soft logit margin loss" (SLM)
L (x, v)
log Σ exp(it (z)) - (z),
jFy
which serves as surrogate loss to the "logit margin loss" L (x. v) by replacing the the max function by the LogSumExp (sometimes also called softmax) function. One immediate property is that the SLM loss is smooth and convex (w.r.t. logits). The next proposition shows that SLM loss is a good approximation to the LM loss. Proposition 2.3.
Li
SLM -o log(Ko 1) S Lg (1, (I. S Lig
SLL (x, y),
(4)
where K denote the number of classes. Remark 2.2. By using the soft logit margin loss, MMA maximizes a lower bound of the margin, the SLM-margin, dg d5LM (x., y):
dg (z, v) = 110^1| || min ||o|| s.t. 8: L.g. (x + 5, 2 0.
To see that, note by Proposition 2.3, L (xx y) upper bounds L (x, 4). So we have 15 L (xc +
5. v) S 0} C (5 I. (x + 6, y) S 0}. Therefore, dg (x, S do (xr 3). i.e. the SI.M-margin is lower bound of the margin.
We next show the gradient of the SLM loss is proportional to the gradient of the cross entropy loss, [, 3) log E, exp(ff f (z), thus minimizing Lg (x + yy) w.r.t. @ "is" minimizing LSLM(x 8*,y).
Proposition 2.4. For a fixed (x, y) and 0, VOLo CE (x. v) = r(0, x, y) V Lo SIM (x, v), and L. = r(0, x, 1) L (1x
V
SLM
(5)
where the scalar r(0,2, Σnsn exp(f
(6)
E, exp((((()))
Therefore, simplify the learning algorithm, we perform gradient descent on model parameters using _ (z + 5", y). As such, we use I. on both clean and adversarial examples, which in practice stabilizes training:
min Lag N.MMA (S), where T. I.MMA(S)
Σ nMH
1.g (r, + 0 + Σ 1. (0 jES,
where 8* = arg min p L (a+t6)2)) || is found with the SLM loss, and Ho = {i de(x <
d } is the set of examples that have margins smaller than the hinge threshold.
2.3 FINDING THE OPTIMAL PERTURBATION 8*
To implement MMA, we still need to find the 8*. which intractable in general settings. We propose an adaptation of the projected gradient descent (PGD) (Madry et al.. 2017) attack to give an approximate solution of 8*, the Adaptive Norm Projective Gradient Descent Attack (AN-PGD). In AN-PGD, we apply PGD on an initial perturbation magnitude Ginit to find a norm-constrained perturbation d, then we search along the direction of 8, to find a scaled perturbation that gives - 0, we then use this scaled perturbation approximate e". Note that AN-PGD here only serves as an algorithm to give an approximate solution of 8*, and it can be decoupled from the remaining parts of MMA training. Other attacks that serve similar purpose can also fit into MMA training framework, e.g. the Decoupled Direction and Norm (DDN) attack (Rony et al., 2018). Algorithm describes the Adaptive Norm PGD Attack (AN-PGD) algorithm.
5

---

Published as a conference paper at ICLR 2020
Algorithm 1 Adaptive Norm PGD Attack for approximately solving 8"
Inputs: (x, is the data example, cinit is the initial norm constraint used the first PGD attack.
Outputs: 8", approximate shortest successful perturbation. Parameters: Emaw is the maximum perturbation length. PGD(x, y, represents PGD perturbation with magnitude E.
1: Adversarial example 61 = PGD(2, (z. y, Cinst)
2: Unit perturbation 6, _
Toill
3: if prediction
52 is correct then
Bisection search to find the zero-crossing of L(x + no.. y) w.r.. 77,77 E [[]6 lls Emaz]
5: else
6:
Bisection search to find e', the zero-crossing of L( + npuu y) w.r.t 77 E [0, [|6
7: end if
8: 8" 6'8
training
training sel
Iest
Training
20 903
poft mar 1555
Fogh maram
10
00
(a) OMMA-32
(b) PGD-8
(c) MMA-32
Figure 3: Visualization of loss landscape in the input space for MMA and PGD trained models.
Remark 2.3. Finding the 8" in Propositions 2.1 and 2.2 requires solving non-convex optimization problem, where the optimality cannot be guaranteed in practice. Previous adversarial training methods, e.g- Madry et al. (2017), suffer the same problem. Nevertheless, we show later in Figure 4, our proposed MMA training algorithm does achieve the desired behavior of maximizing the margin of each individual example in practice.
2.4 ADDITIONAL. CLEAN LOSS DURING TRAINING
In practice, we observe that when the model is only trained with the objective function in Eq. (7), the input space loss landscape is very flat, which makes PGD less efficient in finding 6" for training, as shown in Figure 3. Here we choose 50 examples from both the training and test sets respectively, then perform the PGD attack with e 8/25c and keep those failed perturbations. For each example, we linearly interpolate 9 more points between the original example and the perturbed example, and plot their logit margin losses. In each sub-figure, the horizontal axis is the relative position of the interpolated example: e.g. 0.0 represents the original example, 1.0 represents the perturbed example with 8/255, 0.5 represents the average of them. The vertical axis is the logit margin loss. Recall that when L.
x + 8, y) < O, the perturbation 8 fails.
OMMA-32 in Figure 3a represents model trained with only MMMA in Eq. (7) with dmax 8. PGD-8 Figure 3b represents model trained with PGD training (Madry et al., 2017) with e = 8. As one can see, 32 has "flatter" loss curves compared to PGD-8. This could potentially weaken the adversary during training, which leads to poor approximation of 8* and hampers training.
To alleviate this issue, we add an additional clean loss term to the MMA objective in Eq. (7) to lower the loss on clean examples, such that the input space loss landscape is steeper. Specifically, we use the following combined loss
LC
Σ CE LC (I,,8,) ) + MMA
3
(8)
3
jES
The model trained with this combined loss and dmax = 32 is the MMA-32 shown in Figure 3c. Adding the clean loss is indeed effective. Most of the loss curves are more tilted, and the losses of perturbed examples are lower. We use _ for MMA training in the rest of the paper due to its higher performance. A more detailed comparison between L and L MMA is delayed to Appendix E.
2.5 THE MMA TRAINING ALGORITHM

---

Published as a conference paper at ICLR 2020
Algorithm 2 Max-Margin Adversarial Training.
Inputs: The training set {(r,, W:]}. Outputs: the trained model fa(.). Parameters: e contains perturbation lengths of training data. Emin is the minimum perturbation length. Cmaz is the maxi- mum perturbation length. A(z, y, Etnit) represents the approximate shortest successful perturbation returned by algorithm A (c.g. AN-PGD) on the data example (zz y) and at the initial norm Einit.
1: Randomly initialize the parameter I of model f. and initialize every element of e emnn
2: repeat
Read minibatch
{(11, (i..yy),, y....... (Im, }mm)
4: Make predictions on B and into two: wrongly predicted Bg and correctly predicted B
5: Initialize an empty batch "
6: for (z,, Mt) in B1 do
7: Retrieve perturbation length e, from
8: 5; = A(z,, Vi, 6s)
9: Update the in as |162 If || < dmax then put (z,, + (-5,,1) into B
10:
end for
11:
Calculate gradients of EjeDo (x,,, + E Lg (2 35) + L (2,, Us), the com-
bined loss, on Bo, B1, and BT w.r.t 0, according to Eqs. (7) and (8)
12:
Perform one step gradient step update
13:
until meet training stopping criterion
Algorithm 2 summarizes our practical MMA training algorithm. During training for each minibatch,
we 1) separate it into 2 batches based on if the current prediction matches the label; 2) find 8* for each example in the "correct batch"; 3) calculate the gradient of & based on Eqs. (7) and (8).
3 UNDERSTANDING ADVERSARIAL TRAINING W.R.T. MARGIN
Through our development of MMA training in the last section, we have shown that margin maxi- mization is closely related adversarial training with the optimal perturbation length || In this section, we further investigate the behavior of adversarial training in the perspective of margin max- imization. Adversarial training (Huang al., 2015; Madry et al., 2017) minimizes the "worst-case" loss under a fixed perturbation magnitude e, as follows.
min BR max Lo(x + 5,9). S
Looking again Figure 2, we can see that adversarial training update step does not necessarily increase the margin. In particular, as we perform an update to reduce the value of loss at the fixed perturbation e, the parameter is updated from O to 0₁. After this update, we imagine two different scenarios for the updated loss functions Lo, (.) (the solid curve) in Figure (c) and (d). In both (c) and (d), Lo, is decreased by the same amount. However, the margin is increased in (c) with oj > S6. but decreased in (d) with &f < og. Formalizing the intuitive analysis, we present two theorems connecting adversarial training and margin maximization. For brevity, fixing (I. y), let L 0) = L.M (z 6, y), de = do( y). and (p( = mins. [l
Theorem 3.1. Assuming an update from adversarial training changes On to G,, such that p*
max T.55.0)) 0p) > m 8l|se T(5,0, 0,), then
1) ife dey then p* = 0, cg (p*)
de. deg EG, (p"):
2) ife < dey then p² < 0, 60 (p*) < da 60, (p²) < day, and eo (p*) 2 E0g (o"):
3) i > dag, then p² 2 0, 6, (p*) 2 dao Co, (p*) 2 do,, and éo (p*) 2 Cop (p*).
Remark 3.1. In other words, adversarial training, with the logit margin loss and a fixed e
1) exactly maximizes the margin, if is equal the margin;
2) maximizes a lower bound of the margin, if is less than the margin;
3) maximizes an upper bound of the margin, if is greater than the margin.
Next we look at adversarial training with the cross entropy loss through the connection between cross entropy and the soft logit margin loss from Proposition 2.4. We first look at adversarial training on the SLM loss. Fixing (x. y). let (SLM = d5 dfLM (I. y). and <ŠLM,((p) = min (uM 1|6].
Corollary 3.1. Assuming an update from adversarial training changes Og to 61. such that
max Lig (x (x+ 5, maxxol|se LSIM (x+ 8,y), ife S dSTM then p² mmax L% (x +
5. 3) S SLmMo, (p") ) day, SSMMO, (0²) S do and SSLM,, (0*) SSLM. (p*).
7

---

Published as a conference paper at ICLR 2020
Remark 3.2. In other words, if o is less than or equal to the SLM-margin adversarial training, with the SLM loss and a fixed perturbation length e, maximizes a lower bound of the SLM 5LM-margin, thus a lower bound of the margin.
Recall Proposition 2.4 shows that L and Li have the same gradient direction w.r.t both the model parameter and the input. In adversarial training (Madry et al., 2017), the PGD attack only uses the gradient direction w.r.t. the input, but not the gradient magnitude. Therefore, in the inner maximization loop, using the SLM and CE loss will result in the same approximate 8 Furthermore, Vo (z + 5",y) and V. + 8",9) y) have the same direction. If the step size is chosen appropriately, then a gradient update that reduces Lg x2 + y) will also reduce Lg LŞIM(z + ++,8)) .y) Combined with Remark 3.2, these suggest:
Adversarial training with cross entropy loss (Madry et al., 2017) approximately maximizes lower bound of the margin, if is smaller than or equal to the mt.ggnnr
On the other hand, when e larger then the margin, such a relation no longer exists. When e is too large, adversarial training is maximizing increase the margin. This suggests that for adversarial training with a large e, starting with smaller then gradually increasing it could help, since the lower bound of the margin is maximized at the start of training. Results in Scctions 4.1 and 4.2 corroborate exactly with this theoretical prediction.
upper bound of the margin, which might not necessarily
4 EXPERIMENTS
We empirically examine several hypotheses and compare MMA training with different adversarial training algorithms on the MNIST and CIFAR 10 datasets under /eglo--norm constrained perturba- tions. Due to space limitations, we mainly present results on CIFARIO- for representative models in Table 1. Full results are in Table 2 to 15 in Appendix F. Implementation details are also left to the appendix, including neural network architecture, training and attack hyperparameters.
Our results confirm our theory and show that MMA training is stable to its hyperparameter dmaxs and balances better among various attack lengths compared to adversarial training with fixed per- turbation magnitude. This suggests that MMA training is a better choice for defense when the perturbation length is unknown, which is often the case in practice.
Measuring Adversarial Robustness: We use the robust accuracy under multiple projected gra- dient descent (PGD) attacks (Madry et al., 2017) as the robustness measure. Specifically, given an example, each model is attacked by both repeated randomly initialized whitebox PGD attacks and numerous transfer attacks, generated from whitebox PGD attacking other models. If any one of these attacks succeed, then the model is considered "not robust under attack" on this example. For each dataset-norm setting and for each example, under a particular magnitude c, we first perform N randomly initialized whitebox PGD attacks on each individual model, then use N (m - 1) PGD) attacks from all the other models perform transfer attacks, where m is the total number of models considered under each setting (e.g. 10 In our experiments, we use N = 10 for models trained on CIFARIO, thus the total number of the "combined" (whitebox and transfer) is 320 for CIFARIO-4 10-Loo (m = 32). 3 We use ClnAcc for clean accuracy, AvgAcc for the average over both clean accuracy and robust accuracies at different c's. AvgRobAce for the average over only robust accuracies under attack.
of attacks
Correspondence between dmax and In MMA training. maximize each example's margin until the margin reaches dmax.. In standard adversarial training, each example is trained to be robust at e. Therefore in MMA training, we usually set d to be larger than é in standard adversarial training. is difficult to know the "correct" value of dinxx, therefore we test different dmax values, and compare group of MMA trained models to group of PGD trained models.
4.1 EFFECTIVENESS OF MARGIN MAXIMIZATION DURING TRAINING
As discussed in Section 3, MMA training enlarges margins of all training points, while PGD training, by minimizing the adversarial loss with a fixed e, might fail to enlarge margins for points with initial margins smaller than e. This is because when dæ(x, y) < e, PGD training is maximizing bound of de (I. y), which may not necessarily increase the margin. To verify this, we track how the
upper
N = 50 for models trained on MNIST. Total number of attacks is 900 for MNIST- (m = 18), 1200 for MNIST-& (m- 24) and 260 for CIFAR10-62 (m 26). We explain relevant details in Appendix C.
8

---

Published as a conference paper at ICLR 2020
norm
ep:1 15
20
43
57
71
85
113
127
141
Figure 4: Margin distributions during training, under the CIFARIO-E, case. Each blue histogram represents the margin value distribution of MMA-3.0, and the orange represents PGD-2.5.
Table 1:
Accuracies of representative models trained on CIFARIO with foo -nom constrained at- tacks. Robust accuracies are calculated under combined whhteeboxttrnssfee PGD attacks. AvgAcc averages over clean and all robust accuracies; AvgRobAcc averages over all robust accuracies.
CIFARIO Model
RobAcc under different combined (whiiebox+irrnsseef attacks
Cln Acc
AvgAce
AvgRobAce
8
12
16
20
24
28
32
PGD-8
85.14
27.27
20.03
67.73
46.47
26.63
12.33
4.69
1.56
0.62
0.22
PGD-16
68.86
28.28
23.21
$7.99
46.09
33.64
22.73
13.37
7.01
3.32
1.54
PGD-24
10.90
9.95
9.83
10.60
10.34
10.11
10.00
9.89
9.69
9.34
8.68
67.96
46.19
12:22
PGDI S-
70.68
28.44
23.16
59.43
47.00
33.64
21.72
12.66
6.54
2.98
1.31
PGDLS-24
58.36
26.53
22.55
49.05
41.13
32.10
23.76
15.70
5.86
3.11
MMA-12
88.59
26.87
19.15
67.
43.42
24.07
11.45
4.27
1.43
0.45
0.16
21.65
29.83
MMA-32
84.36
29.39
22,51
64.82
47.18
31.49
18.91
10.16
4.77
1.97
0.81
PGD
87.38
28.10
20.69
64.59
46.95
28.88
15.10
6.35
2.35
0.91
0.39
PGDLSSens
76.73
29.52
23.62
60.52
48.21
35.06
22.
12.28
6.17
3.14
1.43
TRADES
84.92
30.46
23.65
70.96
52.92
33.04
18.23
8.34
3.57
1.4
0.69
margin distribution changes during training processes in models under the CIFARIO-62 case, MMA-3.0 vs PGD-2.5. We use MMA-duua to denote the MMA trained model with the combined loss in Eq. (8) and hinge threshold dmaxx and PGD-c to represent the PGD trained (Madry et al., 2017) model with fixed perturbation magnitude
Specifically, we randomly select 500 training points, and measure their margins after each training epoch. We use the norm of the 1000-step DDN attack (Rony et al., 2018) to approximate the margin. The results are shown in Figure 4, where each subplot is histogram (rotated by 90%) of margin values. For the convenience of comparing across epochs, we use the vertical axis to indicate margin value, and the horizontal axis for counts the histogram. The number below each subplot is the corresponding training epoch. Margins mostly concentrate near 0 for both models at the beginning. As training progresses, both enlarge margins on average. However, in PGD training, portion of the margins stay close to 0 throughout the training process, at the same time, also pushing some margins be even higher than 2.5, likely because PGD training continues maximize lower bounds of this subset of the total margins, as discussed in Section 3, the value that the PGD-2.5 model is trained for. MMA training, on the other hand, does not "give up" on those data points with small margins. At the end of training, 37.8% of the data points for PGD-2.5 have margins smaller than 0.05, while only 20.4% for MMA. As such, PGD training enlarges the margins of "casy data" which already robust enough, but "gives up" "hard data" with small margins. Instead, MMA training pushes the margin every data point, by finding the proper c. In general, when the attack magnitude unknown, MMA training is capable in achieving better balance between small and large margins, and thus a better balance among adversarial attacks with various e's as whole.
4.2 GRADUALLY INCREASING HELPS PGD TRAINING WHEN E IS LARGE
Our previous analysis in Section suggests that when the fixed perturbation magnitude c small, PGD training increases the lower bound of the margin. On the other hand, when is larger than the margin, PGD training does not necessarily increase the margin. This is indeed confirmed by our experiments. PGD training fails larger e, in particular
24/255 for the CIFAR10-E, as shown in Table I. We can see that PGD-24's accuracies at all test e 's are around 10%.
'Here choose CIFAR10-E, because the DDN-62 attack both fast and effective for estimating margins.

---

Published as a conference paper at ICLR 2020
Aiming improve PGD training, propose a variant of PGD training, named PGD with Linear pepposeaaaaa Scaling (PGDLS), in which we grow the perturbation magnitude from 0 to the fixed magnitude linearly in 50 epochs. According to our theory, gradually increasing the perturbation magnitude could avoid picking e that is larger than the margin, thus managing to maximizing the lower bound of the margin rather than its upper bound, which is more sensible. It can also be seen as "global magnitude scheduling" shared by all data points, which is to be contrasted to MMA training that gives magnitude scheduling for each individual example. We use PGDLS-e represent these models and show their performances also in Table 1. We can see that PGDLS-24 is trained successfully, whereas PGD-24 fails. At c = 8 or 16, PGDLS also performs similar or better than PGD training, confirming the benefit of training with small perturbation at the beginning.
4.3 COMPARING MMA TRAINING WITH PGD TRAINING
From the first three columns in Table 1, we can see that MMA training very stable with respect to its hyperparameter, the hinge threshold dmax- When dmax is set to smaller values (e.g. 12 and 20), MMA models attain good robustness across different attacking magnitudes, with the best clean accuracies in the comparison set. When dmaa is large, MMA training can still learn a reasonable model that is both accurate and robust. For MMA-32, although dmax is set to a "impossible-to- be-robust" level at 32/255, it still achieves 84.36% clean accuracy and 47.18% robust accuracy at 8/ 8/255, thus automatically "ignoring" the demand to be robust at larger c's, including 20, 24, 28 and 32, recognizing its infeasibility due to the intrinsic difficulty of the problem. In contrast, PGD trained models are more sensitive to their specified fixed perturbation magnitude. In terms of the overall performance, notice that MMA training with large dimax. e.g. 20 or 32, achieves high Avg values, e.g. 28.86% or 29. 39.9.. However, for PGD training to achieve similar performance, e needs to be carefully picked (PGD-16 and P6DLS-16), and their clean accuracies suffer a significant drop.
We also compare MMA models with ensembles of PGD trained models. (GDDens/PGDLLS-ee rep- resents the ensemble of PGD/PGDLS trained models with different e's. The ensemble produces prediction by performing a majority vote on label predictions, and using the softmax scores as the tie breaker. MMA training achieves similar performance compared to the ensembled PGD models. PGD-ens maintains a good clean accuracy, but it is still marginally outperformed by MMA-32 w.r.t robustness at various c's. Further note that 1) the ensembling requires significantly higher computa- tion costs both at training and test times; 2) Unlike attacking individual models, attacking ensembles still relatively unexplored in the literature, thus our whitebox PGD attacks on the ensembles may not be sufficiently effective; 5 and 3) as shown in Appendix F, for MNIST-fae/f,, MMA trained models significantly outperform the PGD ensemble models.
TRADES (Zhang et al., 2019b) improves PGD training by decoupling the minimax training objective into an accuracy term and an robustness regularization term. We compare MMA-32 with TRADES 6 in Table as their clean accuracies are similar. We see that TRADES outperforms MMA the attacks of lengths that are less than = 12/255, but it sacrifices the robustness under larger attacks. We note that MMA training and TRADES' idea of optimizing a calibration loss are progresses in orthogonal directions and could potentially be combined.
Comparison with TRADES:
Testing on gradient free attacks: As sanity check for gradient obfuscation (Athalye al., 2018), we also performed the gradient-fre SPSA attack (Uesato et al., 201 8), to all our -MMA trained models on the first 100 test examples. We find that, in all cases, SPSA does not compute adversarial examples successfully when gradient-based PGD did not.
5 CONCLUSIONS
In this paper, we proposed to directly maximize the margins to improve adversarial robustness. We developed the MMA training algorithm that optimizes the margins via adversarial training with perturbation magnitude adapted both throughout training and individually for the distinct datapoints in the training dataset. Furthermore, we rigorously analyzed the relation between adversarial training and margin maximization. Our experiments on CIFARIO and MNIST empirically confirmed our theory and demonstrate that MMA training outperforms adversarial training in terms of sensitivity to hyperparameter setting and robustness to variable attack lengths, suggesting MMA is a better choice for defense when the adversary is unknown, which is often the case in practice.
5 Attacks on ensembles are explained in Appendix D.
DDowloadded models from https: /ttppp:///itii /ttph/bttttttt/////// com/ www.o//wcddggyy.///ooooo DRS
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENT
We thank Marcus Brubaker, Bo Xin, Zhanxing Zhu, Joey Bose and Hamidreza Saghir for proofread- ing earlier drafts and useful feedbacks, also Jin Sung Kang for helping with computation resources.
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense se- curity; Circumventing defenses to adversarial examples. In International Conference on Machine Learning. pp. 274-283, 2018. 4.3
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smdicć, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Security and Privacy (SP), 2017 IEEE Symposium on, pp. 39-57. IEEE, 2017. D, F.1
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, Percy Liang, and John C Duchi. Unlabeled data improves adversarial robustness. arXiv preprint arXiv: /905. /ttp//955.33373 13736, 2019.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. International Conference on Machine Learning, pp. 854-863, 2017. 1.1
Jeremy M Cohen, Elan Rosenfeld, and J Zico Kolter. Certified adversarial robustness via randomized smoothing. arXiv preprint Vr9iv:/902.. 02918, 2019. B
Francesco Croce, Maksym Andriushchenko, and Matthias Hein. Provable robustness of relu net- works via maximization of linear regions. arXiv preprint arXiv: 2018. 1.1
Gavin Weiguang Ding, Kry Yik-Chau Lui, Xiaomeng Jin, Luyu Wang, and Ruitong Huang, On the sensitivity of adversarial robustness input data distributions. In International Conference on Learning Representations, 2019a. F
Gavin Weiguang Ding, Luyu Wang, and Xiaomeng Jin. AdverTorch v0.1: An adversarial robustness toolbox based on pytorch. arXiv preprint arXiv:1902.07623, 2019b.
Gamaleldin FF Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. arXiv preprint arXiv:1803.0550 08509,5598, 2018. 1.1, B
Chuan Guo, Jared S Frank, and Kilian Q Weinberger. Low frequency adversarial perturbation. arXiv preprint arXiv: wtt.:/.0990.778 2018.
Matthias Hein and Maksym Andriushchenko. Formal guarantees the robustness of classifier
against adversarial manipulation. arXiv preprint arXiv: / hato::/705.87555 2017. 1.1
Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre-training can improve model robustness and uncertainty. arXiv preprint arXiv: /0t1::99000099000 2019. 1
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Learning with strong adver- sary. arXiv preprint arXiv:1511.0000
2015. 3,3
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 1706v 2017. 1, 2, 2, 2.3, 2,4, 3, 4, C.1.C
Alexander Matyasko and Lap-Pui Chau. Margin maximization for robust classification using deep learning. Neural Networks (IJCNN), 2017 International Joint Conference on, pp. 300-307.
IEEE, 2017. 1.1, B
Jérôme Rony, Luiz G Hafemann, Luis S Oliveira, Ismail Ben Ayed, Robert Sabourin, and Eric Granger. Decoupling direction and norm for efficient gradient- based adversarial attacks and defenses. arXiv preprint arXiv: /8/ 1.06000, 2018. 2.3, 4.1, B.

---

Published as a conference paper at ICLR 2020
Andrew Slavin Ross and Finale Doshi Velez. Improving the adversarial robustness and in- terpretability of deep neural networks by regularizing their imput gradients. arXiv preprint arX1v:1711.09404. 2017. 1.1
Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! arXiv preprint arXiv:1904. 12843, 2019. 1
Yash Sharma, Gavin Weiguang Ding, and Mareus Brubaker. the effectiveness of low frequency perturbations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3389-3396. AAAI Press, 2019.
Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues, Robust large margin deep neural networks. IEEE Transactions on Signal Processing, 2017. 1.1,
Robert Stanforth, Alhussein Fawzi, Pushmeet Kohli, et al. Are labels required for improving adver- sarial robustness? arXiv preprint arXiv:1905.
2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable certifi- cation of perturbation invariance for deep neural networks. In Advances in Neural Information Processing Systems, pp. 6542-6551, 2018. 1.1.
Jonathan Uesato, Brendan O' D'Dooggue, Aaron van den Oord, and Pushmeet Kohli. Adversarial risk and the dangers of evaluating against weak attacks. arXiv preprint arXiv:1802.05660 2018. 4.3,
D
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013. B
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In International Conference on Machine Learning, pp. 6586-6595, 2019. B
Ziang Yan, Yiwen Guo. and Changshui Zhang. Adversarial margin maximization networks. IEEE transactions on pattern analysis and machine intelligence, 2019. 1.1,B
Nanyang Ye and Zhanxing Zhu, Bayesian adversarial learning. In Proceedings of the 32nd Interna- fional Conference on Neural Information Processing Systems, pp. 6892-6901. Curran Associates Inc., 2018. B
Yao-Liang Yu. The differentiability of the upper envelop. Technical note, 2012. A.2
Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
arXiv:1605.07146, 2016.
Runtian Zhai, Chen Dan, Di Hc, Huan Zhang, Boqing Gong, Pradeep Ravikumar, Cho-Jui Hsich, and Liwei Wang. Macer: Attack-free and scalable robust training via maximizing certified radius. In International Conference on Learning Representations, 2020. B
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. arXiv preprint arXiv: /905.008770..7 2019a.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P Xing, Laurent El Ghaoui, and Michael I Jordan. Theoretically principled trade-off between robustness and accuracy. arXiv preprint arXiv: arXiv:/901.0m /00//9991.3333 2019b. 1, 4.3
Huan Zhang, Hongge Chen, Zhao Song, Duane Boning, Inderjit S Dhillon, and Cho-Jui Hsieh. The limitations of adversarial training and the blind-spot attack. arXiv preprint MXXI:1990.004644 2019c. F
12

---

Published as a conference paper at ICLR 2020
Appendix
A PROOFS
A.1 PROOF OF PROPOSITION 2.1
Proof. Recall c(6) ||8|. Here we compute the gradient for do (z. y) in its general form. Consider the following optimization problem:
do(x,y)
min <(d), 6EA(B)
where A()) = {8: Lo(x+0, = 0}, c and L(6, 8) are both C² functions' Denotes its Lagrangian by L(6, A), where
[ A) = E + ALo(E + d,y) For fixed 0, the optimizer 8 and X must satisfy the first-order conditions (FOC)
Ōc(6) OLo(x 6.9)
0,
(10)
06
b=
Læ( + 8, 11 =0, Put the FOC equations in vector form,
C((5,)),0) (
Øc(6) + @.o(e+tby)) 36
)
05
0.
Lo + 8,9)
Note that G is C continuously differentiable since and L(ö, 0) are C2 functions. Furthermore, the Jacobian matrix of w.r.t (6, A) is
((6.)) G(("), )1),,)
(P((8')
+ x 0² L.8" .(8.0))
8L(8" .0) 062
082
06
aL(6"
0
88
which by assumption is full rank. Therefore, by the implicit function theorem, 5* and X* can be expressed as a function of 6, denoted by 8* (0) and X (0).
To further compute Vædg ar, v). note that do y) = <(6" (0)). Thus, Vøda y) = de(5*) (8) X (0(5"00)007)
(11)
06 00
06
00
where the second equality is by Eq. (10). The implicit function theorem also provides way of computing 08" (9) which complicated involving taking inverse of the matrix ((O"))(("",,>))
86
Here we present a relatively simple way to compute this gradient. Note that by the definition of 8* (0),
(O)) 8) = 0.
and 8 (0) is differentiable implicit function of 0 restricted to this level set. Differentiate with w.r.t. on both sides:
AL(8*,0) 8)
aL 0) (8)
+
(12)
08
08
00
Combining Eq. (11) and Eq. (12),
ØL(5*, 0)
Vodo(x, x* (0)
(13)
00
Lastly, note that
de(o)
OLo(I d, v)
08
カ= A=
Note that simple application of Danskin's 's theorem would not he valid as the constraint set A(0) depends the parameter 0.
13

---

Published as a conference paper at ICLR 2020
Therefore, one way calculate (0) is by
0c(8)
aLe (1
(0)
06
06
( 6.y)
(1. A.y)
06
86
6=
A.2 PROOF OF PROPOSITION 2.2
We provide detailed and formal statements of Proposition 2.2.
For brevity, consider K Kllayers fully-connected ReLU network, f(0; x) fa(x) as function of 0.
f(0; x) = Dx
(14)
where the Dy are diagonal matrices dependent on ReLU's activation pattern over the layers, and Wx 's and V are the weights (i.e. 0). Note that x(); is piecewise polynomial functions of # with finitely many pieces. We further define the directional derivative of a function g, along the direction of U. to be:
9 tō) 9(0)
g'(0: 8) = lim 1.0
t
Note that for every direction U, there exists a > 0 such that f(0; ) is a polynomial restricted to a line segment [0, 0 + atjj. Thus the above limit exists and the directional derivative is well defined.
We first show the existence of 7 and for 1(0 + (7) given any Let 8, (() SuP L(ö, 0g + tū).
Proposition A.1. For 0, t E [0, 1}. and Op E O, there exists a direction T7 E O such that the derivative of (,) exists and is negative. Moreover, it is given by
No T,e (6 oo: 17).
Proof. [Proof sketch] Since is local minimum, there exists a direction d, such that L' (((86,00' 0o: σ) 01(6" (tif) is negative.
ōt
The Danskin theorem provides a way to compute the directional gradient along this direction ह. We basically apply version of Danskin theorem for directional absolutely continuous maps and semi- continuous maps (Yu, 2012). 1. the constraint set {ō || S é} is compact; 2. L(0g + tū; æ + the integral
6, y) is piecewise Lipschitz and hence absolutely continuous (an induction argument representation over the finite pieces). 3. L(Ab + tē; + 8, y) is continuous on both ó and along the direction d and hence upper semi continuous. Hence we can apply Theorem I in Yu (2012). 0
Therefore, for any > 0, 0o not a local minimum, then there exits direction d, such that for 0, 0g tū for proper t,
sup L(d, 0 + tū) |
sup L(5, 0o) S
(15)
Our next proposition provides an alternative way increase the margin of fa. Proposition A.2. Assume fe, has margin Eo. and 0, such that t,, (t) S Lo, F.eo (0), then fo. has larger margin than Eg-
Proof. Since Se, has a margin co. thus
max L(On; 0, |
Further by te (t) S t. (0),
sup L(5, 0, + tū) ||
sup L(ō, 0o).
Se
To see the equality (constraint not binding), we use the following argument. The envolope function's continuity is passed from the continuity of L(0p; 5, 8,y). The inverse image of a closed set under continuous function closed. If 8* lies in the interior of max Al|Seo (0u: + 6, y) 2 0, we would have contradiction. Therefore the constraint is not binding, due to the continuity of the envolope function. By Eq. (15), max L(0; + 6, y) < 0. So for the parameter 0,, fo, has margin 60-
14

---

Published as a conference paper at ICLR 2020
Therefore, the update 0u 01 0u + tē increases the margin of fo-
A.3 PROOF OF PROPOSITION 2.3 Proof.
1 (z. y) (max (1)) - (x)
(16)
log(exp(max (mmxx((xx)))) - -1)())
(17)
57y
S log(exp(
Σ fx())) (x)
(18)
i7y
(19)
log((K - 1) exxp(max/o l (x))) (x)
(20)
jfy
log( K - 1) + (max l/ (x)) - (x)
(21)
izy
log(( - 1) L (I, y)
(22)
Therefore,
L I.SLM (x, v) llog (K 1) s L.L (x, y) S (2. (2. v)-
A.4 A LEMMA FOR LATER PROOFS
The following lemma helps relate the objective of adversarial training with that of our MMA train-
ing. Here, we denote Lo( + 6, y) as I. B) for brevity.
Lemma A.1. Given (I. y) and 0 assume that L(ō, 6) is continuous in 5, then for 2 0, and
p 2 L(0,0) E Range (6, 0)), it holds that
min |08|| = - max L(5, 0) = PP; 1
(23)
L(
max L(8,0) = S
min ||0|| S E.
(24)
L(6,0)2P
Proof. Eq. (23). We prove this by contradiction. Suppose m L(8, 0) > p. When = 0, this violates our asssumption 2 L(0, 8) in the theorem. So assume > 0. Since L(6, 0) is a continuous function defined on a compact set, the maximum attained by 8 such that || S and I.65,0) 0) > p- Note that (66 0)) is continuous and p Z L(0, 0), then there exists 8 E (0, 8) i.e. the line segment connecting O and 5, such that ||ō||| < 6 and T.(ō. 0) This follows from the intermediate value theorem by restricting T.(5, 0) onto (0, 8). This contradicts min || =
If max L 8) < p. then {o ||o]| < C {ō: L(5, < p}. Every point p E (5: : || S c) is in the open set {6 L(8, 8) < p). there exists an open ball with some radius Tp centered at p such
that B,, C {8 L(8.0)
p}. This forms open cover for {8 ||8|| < c). Since {8 : ||8| < c)
is compact, there is an open finite subcover U. such that: (5 [6: S { C C (5 L(5 0) <)
Since Ue is finite, there exists h > 0 such that (5 |8|1 S 6 hb C {8 1(6,0) p). Thus
{ō: L(5, 0) 2 p} C {6: ||8|| > E + h}, contradicting min [6]
again.
Eq. (24). Assume that min || > e, then {6 L(6,0) 2 p} C {8
||8|| e). Taking
complementary set of both sides, {6: ||8|| < {8 L(6,0) < p). Therefore, by the compactness
of (ō || S c). max 1olls. T(5 0) < p. contradiction.
A.5 PROOF OF THEOREM 3.1
Proof. Recall that L(8, 0) LLM + 5.y). do do v), Ep(p) min ||]|| and p* max I.65, 0n) max 81|5e T(6,01)-
15

---

Published as a conference paper at ICLR 2020
We first prove that Ve, cū, (p*) 2 (0)) by contradiction. We assume Cū, (p*) < Ci (p").
(g) Let
53 (p) arg ming, L(6,8))p []ó]] which |,ō; (p')) < 1,0% (p*) Il. By Eq. (24), we have | (p } 1
6. Therefore, 116b, (0')] < c. Then there exist a 6# E {ō: ||0|| S E) such that L(d#, 01) 2 p² This
contradicts 6 d L(6, 01) < p'. Therefore o (p*) 2 c (p').
For e= da. By definition of margin in Eq. (1), we have p= da L(ó, 0o) = 0. Also
by definition of eg(p), c, (0) da, and c, (0) dø.
For 2), 6 < dog- We have p = max L(6, Bo) 1 max 1(6,0o) = 0. Therefore
100 (p") - c (0) de, and c, (p') S c, (0) = dø,
For 3), 6 > do.- We have p² = 6| [,6,, 0o) z max 1(6,0) = 0. Therefore
60 (p*) 2 og (0) = do, and eú, (0*) 2 c,. (0) = de.
B MORE RELATED WORKS
We next discuss few related works in details.
First-order Large Margin: Previous works (Matyasko & Chau, 2017; Elsayed et al., 2018; Yan et al., 2019) have attempted to use first-order approximation to estimate the input space margin. For first-order methods, the margin will be accurately estimated when the classification function is linear. MMA's margin estimation is exact when the shortest successful perturbation 8' can be solved, which is not only satisfied by linear models, but also by broader range of models, e.g. models that are convex w.r.t. input æ. This relaxed condition could potentially enable more accurate margin estimation which improves MMA training's ' performance.
Lipschit Regularization: Sokolic al. (2017): Tsuzuku et al. (2018) enlarges their mar- gin by controlling the global Lipschitz constant, which in return places a strong constraint on the model and harms its learning capabilities. Instead, our method, alike adversarial training, uses ad- versarial attacks to estimate the margin to the decision boundary. With strong method, our estimate much more precise in the neighborhood around the data point, while being much more flexible due to not relying on a global Lipschitz constraint.
Hard Hard-Margin SVM (Vapnik, 2013) in the separable case: Assuming that all the training examples are correctly classified and using our notations on general classifiers, the hard-margin SVM objective can be written as:
max min (z))
s.t. Lo(z)) Vi.
(25)
On the other hand, under the same "separable and correct" assumptions, MMA formulation in Eq. (3) can be written as
maxx Σ ddlts))
s.t. Lo(z;) < Vi,
(26)
which maximizing the average margin rather than the minimum margin in SVM. Note that the theorem on gradient calculation of the margin in Section 2.1 also applies to the SVM formulation of differentiable functions. Because of this, we can also use SGD to solve the following "SVM-style" formulation:
max
min do(z;) - Σ Jo(2)) iESt
(27)
JES,
As our focus is using MMA to improve adversarial robustness which involves maximizing the aver- age margin, we delay the maximization of minimum margin to future work.
Maximizing Robust Radius for Randomized Smoothing: Randomized smoothing (Cohen et al., 2019) is an effective technique for certifying the & robust radius of a "smoothed classifier" with high probability bound. Here the certified robust radius is lower bound of the input space margin of the "smoothed classifier". Zhai et al. (2020) introduce a method to directly maximizing this certified robust radius.
Bayesian Adversarial Learning: Ye & Zhu (2018) introduce full Bayesian treatment of adver- sarial training. In practice, the model is trained be robust under adversarial perturbations with
16

---

Published as a conference paper at ICLR 2020
different magnitudes. Although both their and our methods adversarially train the model with dif- ferent perturbation magnitudes, the motivations are completely different.
Wang et al. (2019) improve adversarial training's 's convergence by dynamically adjusting the strength (the number of iterations) of PGD attacks, based on its proposed First-Order Stationary Condition. In sense, MMA training is also dynamically adjusting the attack strength. However, our aim is margin maximization and the strength here represents the magnitude of the perturbation.
Dynamic Adversarial Training:
B.1 DETAILED COMPARISON WITH ADVERSARIAL TRAINING WITH DDN
For 22 robustness, we also compare models adversarially trained on the "Decoupled Direction and Norm" (DDN) attack (Rony et al., 2018), which concurrent to our work. The DDN attack aims to achieve successful perturbation with minimal lg norm, thus, DDN could be used as a drop-in placement for the AN-PGD attack for MMA training. We performed evaluations on the downloaded DDN trained models.
The DDN MNIST model a larger ConvNet with similar structure to our LeNet5, and the CIFARIO model is wideresnet-28-10 10, which is similar but larger than the wideresnet- -2--4 that we use.
DDN training, "training adversarial examples generated by the DDN attack", differs from MMA in the following ways. When the DDN attack does not find a successful adversarial example, it re- turns the clean image, and the model will use it for training. In MMA, when a successful adversarial example cannot be found, it is treated as perturbation with very large magnitude, which will be ignored by the hinge loss when we calculate the gradient for this example. Also, in DDN training, there exists a maximum norm of the perturbation. This maximum norm constraint does not exist for MMA training. When a perturbation is larger than the hinge threshold, it will be ignored by the hinge loss. There also are differences in training hyperparameters which we refer the reader to Rony al. (2018) for details.
Despite these differences, in our experiments MMA training achieves similar performances under the lo cases. While DDN attack and training only focus &g cases, we also show that the MMA training framework provides significant improvements over PGD training in the foo case.
C DETAILED SETTINGS FOR TRAINING
With respect to the weights on CE and MMA in Eq. (8), we tested 3 pairs of weights. (1/3, 2/3), (1/2, 1/2) and (2/3, 1/3), in our initial CIFARIO lo experiments. We observed that (1/3, 2/3), namely (1/3 for C and 2/3 for L MMA) gives better performance. We then fixed and use the same values for all the other experiments in the paper, including the MNIST experiments and & experiments.
We train LeNet5 models for the MNIST experiments and use wide residual networks (Zagoruyko & Komodakis, 2016) with depth 28 and widen factor 4 for all the CIFAR10 experiments. For all the experiments, we monitor the average margin from AN-PGD on the validation set and choose the model with largest average margin from the sequence of checkpoints during training. The validation set contains first 5000 images of training set. It is only used to monitor training progress and not used in training. Here all the models are trained and tested under the same type of norm constraints, namely if trained on fgo, then tested on foo: trained on lz, then tested on Ez.
The LeNet5 is composed of 32-channel conv filter + ReI.U + size 2 max pooling 64-channel conv filter ReLU + size 2 max pooling + fc layer with 1024 units ReLU + layer with 10 output classes. We do not preprocess MNIST images before feeding into the model.
For training LeNet5 on all MNIST experiments, for both PGD and MMA training, we use the Adam optimizer with an initial learning rate of 0.0001 and train for 100000 steps with batch size 50. In our initial experiments, we tested different initial learning rate at 0.0001, 0.001, 0.01, and 0.1 and do not find noticeable differences.
17

---

Trained models: Various PGD/PGDLS models are trained with different perturbation magnitude e, denoted by PGD-e or PGDLS-e. PGD-ens/PGDLS-en represents the ensemble of PGD/PGDLS trained models with different e's. The ensemble makes prediction by majority voting on label pre- dictions, and uses softmax scores as the tie breaker. We perform MMA training with different hinge thresholds dmax, also with/without the additional clean loss (see next section for details). We use OMMA to represent training with only L in Eq. (7), and MMA to represent training with the combined loss in Eq. (8). When train For each dmaxx value, we train two models with different ran- dom seeds, which serves two purposes: 1) confirming the performance of MMA trained models are not significantly affected by random initialization; 2) to provide transfer attacks from an "identical" Models shown in Published as a conference paper at ICLR 2020
We use the WideResNet-28-- as described in Zagoruyko & Komodakis (2016) for our experiments, where 28 is the depth and 4 is the widen factor. We use "per image standardization" " to preprocess CIFARIO images, following Madry et al. (2017).
For training WideResNeet on CIFARIO variants, we use stochastic gradient descent with momentum 0.9 and weight decay 0.0002. We train 50000 steps in total with batch size 128. The learning rate set to 0.3 step 0, 0.09 at step 20000, 0.03 at step 30000, and 0.009 at step 40000. This setting is the same for PGD and MMA training. In our initial experiments, we tested different learning rate at 0.03, 0.1, 0.3, and 0.6, and kept using 0.3 for all our later experiments. We also tested longer training schedule, following Madry et al. (2017), where we train 80000 steps with different learning rate schedules. We did not observe improvement with this longer training, therefore kept using the 50000 steps training.
For models trained on MNIST, we use 40-step PGD attack with the soft logit margin (SLM) loss defined in Section 3, for CIFARI0 we use 10 step-PGD, also with the SLM loss. For both MNIST and CIFARIO, the step size of PGD attack at training time number of steps In AN-PGD, we always perform 10 step bisection search after PGD, with the SLM loss. For AN-PGD, the maximum pertur- bation length is always 1.05 times the hinge threshold: émaz 1.05dmax. The initial perturbation length at the first epoch, fjnát, have different values under different settings, Einit = 0.5 for MNIST lz. Cinit 0.1 for MNIST foo. Eiuit 0.5 for CIFARIO E2. Esrit 0.05 for CIFARIO Eg. In epochs
2.5e
after the first, Emni will be set to the margin of the same example from last epoch.
model. As such, MMA trained models are named as OMMA/MMA-daax the main body correspond to those with seed "sd0".
For MNIST- we train PGD/PGDLS models with 0.1, 0.2, 0.3, 0.4, 0.45. We also train OMMA/MMA models with dmax = 0.45, each with two random seeds. We also consider a standard model trained on clean images, PGD-ens, PGDLS-ens, and the PGD trained model from Madry et al. (2017). Therefore, we have in total m = 5 2 4 = 18 models under MNIST-
For MNIST- we train PGD/PGDLS models with OMMA/MMA models with dmax = 2.0, 4.0,6.0, 6.0, each with two random seeds. We also consider a standard model trained on clean images, PGD-ens, PGDLS-ens, and the DDN trained model from Rony et al. (2018). Therefore, we have in total m = 2 x 4 + x x 2 + 4 = 24 models under MNIST-
1.0, 2.0, 4.0. We also train
For CIFAR10- we train PGD/PGDLS models with c = 4, 8, 12, 16, 20, 24, 28, 32. We also train OMMA/MMA models with dmax = 12, 20, 32, each with two random seeds. We also consider a standard model trained on clean images, PGD-ens, PGDLS-ens. and the PGD trained model from Madry et al. (2017). Therefore, we have in total 2 x 8 + 2 x 3 x 2 + 4 = 32 models under CIFARIO-
For CIFAR10-E2, we train PGD/PGDLS models with e = 0.5, 1.0, 1.5, 2.0, 2.5. We also train OMMA/MMA models with dmax 1.0, 2.0, 2.0,3.0, 3.0, each with two random seeds. We also consider standard model trained on clean images, PGD-ens, PGDLS-ens, and the DDN trained model from Rony al. (2018). Therefore, we have in total 2 x 5 + 3 x 2 4 26 models under MNIST-62.
With regard to ensemble models, for MNIST-&, PGD/PGDLS-ens. CIFARIO-6 PGD/PGDLS-ens. MNIST-6<. PGDLS-ens, and CIFARIO- PGDLS-ens, they all use the PGD (or PGDLS) models
Description can he found at https: /wwww tensor ow, org/api_ /tt.d/c/ccttt/////////o per_im watiiiaaeataddrreeeee zation... We implemented our own version in PyTorch.
18

---

Published as a conference paper at ICLR 2020
trained at all testing (attacking) c's. For CIFARIO- PGD-ens, PGD- 2GD-24,28,32 are excluded for the same reason.
D DETAILED SETTINGS OF ATTACKS
For both for and &g PGD attacks, we use the implementation from the AdverTorch toolbox (Ding et al., 2019b). Regarding the loss function of PGD, we use both the cross entropy (CE) loss and the Carlini & Wagner (CW) loss. 10
As previously stated, each model will have N whitebox PGD attacks on them, N/2 of them are CE-PGD attacks, and the other N/2 are CW-PGD attacks. Recall that N 50 for MNIST and N = 10 for CIFARIO. At test time, all the PGD attack run 100 iterations. We manually tune the step size parameter on a few MMA and PGD models and then fix them thereafter. The step size for MNIST- when 0.3 is 0.0075, the step size for CIFARIO-E when 8/255 is 2/255, the step size for MNIST-C2 when e = 1.0 is 0.25, the step size for CIFARIO-E2 when = 1.0 is 0.25. For other values, the step size is linearly scaled accordingly.
The ensemble model we considered uses the majority vote for prediction, and uses softmax score as the tie breaker. So it is not obvious how to perform CW-PGD and CE-PGD directly on them. Here we take strategies. The first one is a naive strategy, where we minimize the sum of losses of all the models used in the ensemble. Here, similar to attacking single models, we CW and CE loss here and perform the same number attacks.
The second strategy is still a PGD attack with a customized loss towards attacking ensemble models. For the group of classifiers in the ensemble, at each PGD step, if less than half of the classifiers give wrong classification, we sum up the CW losses from correct classifiers as the loss for the PGD attack. If more than half of the classifiers give wrong classification, then we find the wrong prediction that appeared most frequently among classifiers, and denote it as labelo, with its corresponding logit, logit0. For each classifier, we then find the largest logit that is not logitO, denoted as logitl. The loss we maximize, in the PGD attack, is the sum of "logitl logito" from each classifier. Using this strategy, we perform additional (compared to attacking single models) whitebox PGD attacks on ensemble models. For MNIST, we perform 50 repeated attacks, for CIFAR10 we perform 10. These are also 100-step PGD attacks.
We expect more carefully designed attacks could work better on ensembles, but we delay it to future work.
For the SPSA attack (Uesato et al., 2018), we run the attack for 100 iterations with perturbation size 0.01 (for gradient estimation), Adam learning rate 0.01, stopping threshold -5.0 and 2048 samples for each gradient estimate. For CIFARIO- we use = 8/255. For MNIST- we use 0.3.
E EFFECTS OF ADDING CLEAN LOSS IN ADDITION TO THE MMA LOss
We further examine the effectiveness of adding a clean loss term to the MMA loss. We represent MMA trained models with the MMA loss in Eq. (7) as MMA-dmux- In Section 2.4, we introduced MMAC-dmax models resolve MMA-djax model's s problem of having flat input space loss land- scape and showed its effectiveness qualitatively. Here we demonstrate the quantitative benefit of adding the clean loss.
We observe that models trained with the MMA loss in Eq. (7) have certain degrees of TransferGaps. The term TransferGaps represents the difference between robust accuracy under "combined (white- box+transfer) attacks" and under "only whitebox PGD attacks". In other words, it is the additional attack success rate that transfer attacks bring. For example, OMMA-32 achieves 53.70% under whitebox PGD attacks, but achieves a lower robust accuracy at 46.31% under combined (white- box+transfer) attacks, therefore has TransferGap of 7.39% (See Appendix F for full results.). After adding the clean loss, MMA-32 reduces its TransferGap at c = 8/255 to 3.02%. This corre- sponds to our observation in Section 2.4 that adding clean loss makes the loss landscape more tilted, such that whitebox PGD attacks can succeed more easily.
The CW loss is almost equivalent to the logit margin (LM) loss. We use CW loss for the ease of comparing with literature. Here CW(x) min (max fz) fy(x), 0). When the classification is correct, CW and LM loss have the same gradient.
19

---

Published as a conference paper at ICLR 2020
Recall that MMA trained models robust to gradient free attacks, as described in Section 4.3. Therefore, robustness of MMA trained models and the TransferGaps are likely not due to gradient masking.
We also note that TransferGaps for both MNIST- and lg cases are almost zero for the MMA trained models, indicating that TransferGaps, observed on CIFARIO cases, are not solely due to the MMA algorithm, data distributions (MNIST vs CIFARI0) also play an important role.
Another interesting observation is that, for MMA trained models trained on CIFARIO, adding ad- ditional clean Ioss results in decrease in clean accuracy and an increase in the average robust accuracy, e.g. OMMA-32 has CInAce 86. 11%, and AvgRobAcc 28.36%, whereas MMA-32 has ClnAcc 84.36%, and AvgRobAce 29.39%. The fact that "adding additional clean loss results in a model with lower accuracy and more robustness" seems counter-intuitive. However, it actually confirms our motivation and reasoning of the additional clean loss: it makes the input space loss landscape steeper, which leads to stronger adversaries at training time, which in turn poses more emphasis on "robustness training", instead of clean accuracy training.
F FULL RESULTS AND TABLES
We present all the empirical results in Table 4 to 15. Specifically, we show model performances under combined (whitebox+transfer) attacks in Tables 4 to 7. This is our proxy for true robustness measure. We show model performances under only whitebox PGD attacks in Tables 8 to 11. We show TransferGaps in Tables 12 to 15.
In these tables, PGD-Madry et al. models are the "secret" models downloaded from https: // github. www.c//////ca/c Maaaa//aadLLaab/aa /aiaaa/mmmiii /mnisttniitttttttmm/iii/.ittm/tt mmmist_chaalleee challenge and https: //gtttb..../////// wtt.cccom//aa/. com/MadzyLab/ 1ifarl0__chaleenge/. DDN-Rony et al. models are downloaded from https: //tttsu. com/ watarrrrrrrrrettttavrreeeeeeeeee
hfagt////ssttaa //github. com/ comm/yaodongyu/TRADE..
TRADES models are downloaded from https:
For MNIST PGD-Madry et al. models, whitebox attacks brings the robust accuracy at c 0.3 down to 99.79%, which is at the same level with the reported 89.62% on the website, also with 50 repeated random initialized PGD attacks. For CIFARIO PGD-Madry et al. models, our whitebox attacks brings the robust accuracy at E = 8/255 down to 44.70% which is stronger than the reported 45.21 % on the website, with 10 repeated random initialized 20-step PGD attacks. As our PGD attacks are 100-step, this is not surprising.
We also compared TRADES with MMA trained models under for. attacks. On MNIST, overall TRADES outperforms MMA except that TRADES fails completely at large perturbation length of 0.4. This may be because TRADES is trained with attacking length 0.3, and therefore cannot defend larger attacks. On CIFAR 10, we compare MMA-32 with TRADES as their clean accuracies similar. Similar to the results on MNIST, we that TRADES outperforms MMA the attacks of lengths that are less than c = 12/255. but it sacrifices the robustness under larger attacks. We also note that MMA training and TRADES' idea of optimizing calibration loss progresses in orthogonal directions and could potentially be combined.
As we mentioned previously. DDN training can be seen as specific instantiation of the general MMA training idea, and the DDN-Rony al. models indeed performs very similar to MMA trained models when dmax is set relatively low. Therefore, we do not discuss the performance of DDN-Rony et al. separately.
In Section 4, we have mainly discussed different phenomena under the case of CIFAR10- For CIFAR10-4; we see very similar patterns in Tables 7, 11 and 15. These include
MMA training is fairly stable to duaxx and achieves good robustness-accuran trade-offs. On the other hand, to achieve good AvgRobAce, PGD/PGDLS trained models need to have large sacrifices on clean accuracies.
Adding additional clean loss increases the robustness of the model, reduce TransferGap, at a cost of slightly reducing clean accuracy.
As a simpler datasets, different adversarial training algorithms, including MMA training, have very different behaviors on MNIST as compared CIFARIO.
20

---

Published as a conference paper at ICLR 2020
We first look MNIST- Similar to CIFARIO cases, PGD training is incompetent on large e's, e.g. PGD-0.4 has significant drop on clean accuracy (to 96.64%) and PGD-0.45 fails to train. PGDLS training, the other hand, is able to handle large e's training very well on MNIST- and MMA training does not bring extra benefit on top of PGDLS. We suspect that this is due to the "easiness" of this specific task on MNIST, where finding proper for each individual example is not necessary, and a global scheduling of is enough. We note that this phenomenon confirms our understanding of adversarial training from the margin maximization perspective in Section 3.
Under the case of MNIST- we notice that MMA training almost does not need to sacrifice clean accuracy in order to get higher robustness. All the models with dmax Z 4.0 behaves similarly w.r.t. both clean and robust accuracies. Achieving 40% robust accuracy at e = 3.0 seems to be the robustness limit of MMA trained models. On the other hand, PGD/PGDLS models are able to get higher robustness at e = 3.0 with robust accuracy of 44.5%, although with some sacrifices to clean accuracy. This is similar to what we have observed in the of CIFARIO.
We notice that on both MNIST- and MNIST- unlike CIFARIO cases, PGD(LS)-ens model performs poorly in terms of robustness. This is likely due to that PGD trained models on MNIST usually have a very sharp robustness drop when the used for attacking larger than the used for training.
Another significant differences between MNIST cases and CIFARIO cases is that TransferGaps very small for OMMA/MMA trained models on MNIST cases. This again is likely due to that MNIST is an "easier" dataset. It also indicates that the TransferGap is not purely due the MMA training algorithm, it also largely affected by the property of datasets. Although previous litera- ture (Ding et al., 2019 9a; Zhang et al., 2019c) also discusses related topies the difference between MNIST and CIFAR 10 w.r.t. adversarial robustness, they do not directly explain the observed phe- nomena here. We delay thorough understanding of this topic future work.
F.1 ADDITIONAL EVALUATION WITH THE CW-62 ATTACK
We run the CW-&2 attack (Carlini Wagner, 2017) on the first 1000 test examples for models trained with &e attacks. This gives the minimum & norm a perturbation needs to change the prediction. For each model, we show the clean accuracy, also mean, median, 25th percentile and 75th percentile of these minimum distances in Tables 2 and 3 below.
Looking CW-f2 results, have very similar observations compared to the observations made based on robust accuracies (under PGD attacks) at different c's and the average robust accu- racy.
On MNIST, for MMA trained models as d increases from 2.0 to 6.0, the mean minimum distance goes from 2.15 to 2.76, with the clean accuracy only drops to 97.7% from 99.3%. On the contrary, when we perform PGD/PGDLS with a larger 6, e.g. 6 = 4.0, the clean accuracy drops significantly to 91.8%, with the mean minimum distance increased to 2.53. Therefore MMA training achieves better performance on both clean accuracy and robustness under the MNIST- case.
On CIFAR10, we observed that MMA training is fairly stable to dmaxx and achieves good robustness- accuracy trade-offs. With carefully chosen value, (GDD-1.//PGDI--11 models also achieve similar robustness-accus accurracy tradcoff's as compared to MMA training. However, when we perform PGD training with larger e, the clean accuracy drops significantly.
21

---

75% Norm Published as a conference paper at ICLR 2020
Table 2: CW-C2 attack results models trained on MNIST with f2-norm constrained attacks.
Avg Norm
Norm
STD
99.0
1.57
1.19
1.57
1.95
PGD-1.0
99.2
1.95
1.58
1.95
2.33
PGD-3.0
96.8
2.54
1.79
2.76
3.39
PGD-4.0
91.8
2.53
1.38
2.75
3.75
PGDLS-1.0
99.3
1.88
1.53
1.88
2.24
PGDLS-200
PGDIS
96.9
2,51
1.81
2.74
3.29
PGDLS-4.0
91.8
MMA 2.0
99.3
2.15
1.81
2.22
2.58
MMA-4.0
98.5
2.67
1.86
2.72
3.52
MMA-6.0
97.7
2.76
1.84
2.72
3.61
Table 3: CW-l2 attack results on models trained on CIFARIO with l2- form constrained attacks.
CIFAR10 Model
Cin Acc
Avg Norm
25% Norm
Median Norm
STD
95.1
0.09
0.05
0.08
0.13
PGD-0.5
89.2
0.80
0.32
1.21
PGD-1.0
83.8
1.01
0.30
0.91
1.62
PGD-1.5
PGD-2.0
71.6
1.16
0.00
0.96
2,03
PGD-2.5
65.2
1.19
0.00
2.10
PGDLS-005
90.5
0.79
0.34
0.76
1.17
PGDLS-10
PGDLS- 1.5
77.6
1.10
0.11
0.97
1,83
PGDLS-2.5
66.0
L.19
0.00
0.88
2.13
MMA-1.0
88.4
0.85
0.32
0.80
1.27
MMA-2.0
84.2
1.03
0.26
0.92
1.62
22

---

Published as a conference paper at ICLR 2020
Table 4: Accuracies of models trained MNIST with too norm constrained attacks. These bust accuracies are calculated under both combined (whitebox+ttransfer) PGD attacks. sd0 and sd1 indicate 2 different random seeds.
MNIST Model
RobAcz under different c, comhined (whitchox-ttrassee attacks
Cln Ace
AvgAce
AvgRohAce
0.2
0.4
99.21
73.58
PGD-0.1
99.40
48.85
36.22
96.35
48.51
0.01
0.00
PGD-0.2
99.22
57.92
47.60
0.84
0.00
PGD-0.3
98.96
76.97
71.47
97.90
96,00
91.76
0.22
96.64
PGD-0.45
11.35
11.35
11.35
11.35
11:35
11.35
11.35
PGDLS-0L
99.43
46.85
33.71
95.41
39.42
0.00
0.00
PGD
99.38
58.36
48. 10
97.38
89.49
5.53
0.00
PGDLS-003
99.10
76.56
70.93
97.97
95.66
0.00
PGDLS-0.4
98.98
93.07
91.59
98.12
96.29
93.01
78.96
PGDLS-0045
98.89
87.28
MMA-0.45.xhr -.4-0055-d0
98.95
94.13
92.93
97.87
96.01
92.59
85.24
MMA-0.45-sdI
98.90
94.04
92.82
97.82
96.00
92.63
84.83
OMMA-0-45-dDD
98.98
0MMA-0445-sd
99.02
94.03
92:78
97.93
96.02
92.44
84.73
PGD-ens PGDLS-ens
99.28
57.98
47.65
97.25
89,99
3.37
0.00
99.34
59.04
48.96
97.48
90.40
7.96
0.00
PGD Madry et al.
98.53
76.04
70.41
97.08
94.83
89.64
0.11
TRADES
99.47
77.94
72.56
98.81
97.27
94.11
0.05
Table 5: Accuracies of models trained on CIFAR10 with -orm constrained attacks. These ro- bust accuracies are calculated under both combined (whitebox+transserr) PGD attacks. sd0 and sd1 indicate 2 different random seeds.
CO Model
RobAcc under different e, combined (whitebox4transsee) attacks
Cln Acc
AvgAce
AvgRobAce
STD
94.92
10.55
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
PGD-4
90.44
22.95
14.51
66.31
33.49
12.22
3.01
0.75
0.24
0.06
0.01
PGD-8
67.73
46.47
PGD-12 12
77.86
28.51
22.34
63.88
48.22
32.13
18.67
9.48
4.05
1.56
0.70
PGD-16
PGD-20
61 1.06
27.34
23.12
51.72
43. 13
33.73
24.55
15.66
9.05
4.74
2.42
PGD-24
10.90
9.95
9.83
10.60
10.34
10.11
10.00
9.89
9.69
9.34
8.68
PGD-28
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10,00
10.00
10.00
PGD-32
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10.00
PGDLS-44
89.87
22.39
13.96
63.98
31.92
11.47
3.32
0.68
0.16
0.08
0.05
PGDLS-8
85.63
27.20
19.90
67.96
46.19
26.19
12.22
4.51
1.48
0.44
FGDLS-12
79.39
28.45
22.08
64.62
48.U8
31.34
17.86
8.69
3.95
1.48
0.65
PGDLS.
70.68
28.44 44
26.
59.43
47.00
33.64
21.72
12.66
6.54
2.98
1.31
PGDI 5-20
65.81
27.60
22.83
54.96
44.39
33.13
22.53
13.80
7.79
4.08
1.95
PGDLS 24
58.36
26.53
22.55
49.05
41.13
32.10
23.76
15.77
9.66
5.86
3.11
RGD
50.07
20.97
PGDLS-32
38.80
19.88
17.52
26. 16
24.96
23.22
19.96
16.22
12.92
9.82
6.88
MMA-12-sa
88.59
26.87
19.15
67.96
43.42
24.07
11.45
4.27
1.43
0.45
0.16
MMA-12-sdl
88.91
26.23
18.39
.0.08
42.97
22.57
9.76
3.37
0.92
0.35
0.12
MMA-20-s00
86.56
21.65
66.92
3.25
MMA-20-xdl
85.87
28.72
21.57
65.44
46.11
29.96
17.30
8.27
3.60
1.33
0.56
MMA-32-sdD
MMA-32-sd1
84.76
29.08
22.11
64.41
45.95
30.36
18.24
9.85
4.99
2.20
0.92
OMMA-12-sd0
88.52
26.31
18.54
66.96
42.58
23.22 23.22
10.29
3.43
1.24
0.46
12-s-12-sd1
87.82
26.24
18.54
66.23
43.10
23.57
10.32
3.56
1.04
0.388
0.14
19.95
OMMA-20-sd!
87.44
27.77
20.31
66.28
45.60
27.33
14.00
6.04
2.23
0.74
0.25
OMMA-32-sd0
86.11
28.36
21.14
66.02
46.31
88
15.98
7.44
2.94
1.12
OMMA-32-sd1
86.36
28.75
21.55
66.86
477.12
29.63
16.09
7.56
3,38
1.31
0.47
PGD-ens
87.38
28.10
20.69
64.59
46.95
28.88
15.10
6.35
2.35
0.39
PGDLS-cus
76.73
29.52
23.62
60.52
48.21
35.06
22.14
12.28
6.17
3.14
1.43
PGD MuD-MMary et
87.14
27.22
19.73
68.01
44.68
25.03
12.15
5.18
1.95
0.64
0.23
TRADES
84.92
30.46
23.65
70.96
$2.92
33.04
18.23
8.34
3.57
23

---

Published as a conference paper at ICLR 2020
Table 6: Accuracies of models trained on MNIST with -orm constrained attacks. These robust accuracies are calculated under both combined (whitebox+transfer) PGD attacks. sd0 and sd indi- cate 2 different random seeds.
MNIST Model
Cin Acc
AvgAcc
AvgRobAce
RobAcc under different cumbined wwhhitebbusttrnnstee attacks
STD
99.21
41.84
27.49
86.61
22.78
0.59
0.00
PGD-1.0
99.30
48.78
36.15
95.06
46.84
2,71
0,00
PGD-2.0
56.14
PGD-3.0
97.14
60.36
51.17
90.01
71.03
38.93
4.71
PGD-4.0
93.41
59.52
51.05
82.34
66.25
43.44
12.18
PGDLS-1.0
99.39
47.61
34.66
94.33
42.44
1.89
0.00
PGDLS-2.0
PGDLS-3.0
97.52
60.13
50.78
90.86
71.91
36.80
3.56
59.49
67.211
43.68
MMA-22 2.0 sdo
99.27
53.85
42.50
95.59
68.37
6.03
0.01
MMA-200sd
99.28
54.34
43.10
95.78
68.18
8.45
MMA-4.0-ad
98.71
62.25
53.13
93.93
74.01
39.34
5.24
73.70
MMA-660-s00
98.32
62.32
53,31
93.16
72.63
38.78
8.69
MMA-660-sd1
98.50
62.49
53.48
93.48
73.50
38.63
8.32
OMMA-2.0-66
99.26
54.01
42.69
95.94
67:78
7.03
0.03
0MMAA-200-a1
54.04
42.74
68.83
0MMA-4.0-s00
98.61
62.17
$3.06
94.06
73.51
39.66
5,02
0MMA-6.0-sd0
96.16
62.45
53,52
92.90
72.59
39 09.68
8.93
0MMA-6.0-sd1
98.45
62.24
53.19
93.37
72.93
37.63
PGD-ens
98.87
56.13
45
94.37
70.16
16.79
0.46
54.71
DDN-RRony& al.
99.02
59.93
50.15
95.65
77.65
25.44
Table 7: Accuracies of models trained on CIFARIO with f2-norm constrained attacks. These ro- bust accuracies calculated under both combined (whitebox+transfer) PGD attacks. sd0 and sd1 indicate 2 different random seeds.
CIFARIO Model
RobAcc under different combined (whitebox+trasseec attacks
Acc
AvgAce
AvgRobAce
0.5
1.0
1.5
2.0
2.5
STD
94.92
15.82
0.00
0.01
0.00
0.00
0.00
0.00
PGD-0.5
89.10
33.63
22.53
65.61
33.21
11.25
2.31
0.28
PGD-1.0
83.25
39.70
30.99
66.69
46.08
26.05
11.92
4.21
PGD-1.5
34.94
48.32
33.72
PGD-20
71.05
41.78
35.92
59.76
47.85
35.29
23.15
13.56
PGD-2.5
65.17
40.93
36.08
55.60
45.76
35.76
26.00
17.27
PGDLS-005
89.43
33.41
22.21
65.49
32.40
10.73
2.09
0.33
PGDLS-LO
39.46
30.63
67.29
45.30
25.43
4.03
PGDLS-1.5
77.03
41.74
34,68
63.76
48.43
33.04
19.00
9.17
PGDLS-225
66.21
41.21
36.21
56.45
46.66
35.93
25.51
16.51
MMA-1.0.st s.0-s00
88.02
35,55
25. 15.066
66.18
37.75
15.588
4.74
1.03
MMA-1 1-0-sd
88.92
35.69
25.05
66.81
37.16
15.71
4.49
1.07
MMA-200-d0
84.22
40.48
31.73
65.91
45..66
27.40
14.18
3,50
2.0-2.0.sd
85. 16
39.81
30.75
65.36
44.44
26.42
12.63
4,88
MMA-3.0-s00
82.11
41.59
33 33.49
64,22
46.41
30.23
17.85
8.75
MMA-3.0-sd1
81.79
41.16
33.03
63.58
45.59
29.77
17.52
8.69
OMMA-1.0-she
89.02
35.18
24.41
65.43
36.89
14.77
4.18
0.79
OMMA- 1-0---10-sd1
89.97
35.20
24.25
66.16
36.10
14.04
4.17
0.79
0MMA-2.0-x00
OMMA-2.0-44
85.04
39.68
30.61
64.69
44.36
25.89
12.92
5.19
0MMA-3.0--d0
40.62
31.99 31.97
64.14
45.61
28. 28
15.00
OMMA-3.0.so
84.00
40.66
32.00
63.81
45.22
28.47
15.41
7.08
PGD-ens
27.91
PGDLS ens
86.11
40.38
31.23
63.74
46.21
27.58
13.32
5.31
DDN al.
89.05
36.23
25.67
66.51
39.02
16.60
5.02
1.20
24

---

Published as a conference paper at ICLR 2020
Table 8: Accuracies of models trained MNIST with foc -orm constrained attacks. These robust accuracies are calculated under only whitebox PGD attacks. sd0 and sd1 indicate different random seeds.
MNIST Model
Cln Acc
AvgAce
AvgRohAce
RohAcc under different c, whitehox only
0.1
0.2
0.3
0.4
35.02
18.97
PGD-001
99.40
48.91
36.29
96.35
48.71
0.09
0.00
PGD-0.2
99.22
57.93
47.60
97.44
92.12
0.86
0.00
PGD-0.3
98.96
77.35
71.95
97.90
96.00
91.86
2.03
PGD.0.4
91.51
92.27
PGD-0.45
11.35
11.35
11.35
11.35
11.35
11.35
11.35
PGDLS-0.1
99.43
46,94
33.82
95.41
39.85
0.02
0.00
PGDI S-0.2
99.38
58.44
48.20
97.38
89.49
5.95
0.00
PGDLS-0.3
76.85
71.29
97.98
95.66
90.63
0.90
PGDLS-0.4
98.98
95.49
94,61
98.13
96.42
94.02
89.89
PGDLS-0.45
95.72
94.92
97.91
94.54
MMA 0.45 -d0
98. 95
94,97
93.97
97.89
96.26
93.57
88.16
MMA-0.45-sd
98.90
94.83
93.81
97.83
96.18
93.34
87.91
OMMA 1.45--0.45sdhl
95.06
97.91
93.63
0MMA-0.45-dd
99.02
95.45
94.55
97.96
96,30
94.16
89,80
PGD
99.28
58.02
47.70
97.31
90.11
3.38
0.00
PGDsSsens
99.34
$9.09
49.02
97.50
90.56
8.03
0.00
PGD. Madry al.
98.53
76.08
70.47
97.08
94.87
89.79
0.13
TRADES
99.47
77.95
72.57
98.81
97.27
94.13
0.07
Table 9: Accuracies of models trained on CIFAR10 with -oom constrained attacks. These robust accuracies are calculated under only whitebox PGD attacks. sd0 and sd1 indicate different random seeds.
CI Model
RobAce under different whitebox only
Cln Acc
AvgAce
AvgRobAce
STD
94.92
10.55
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
PGD-4
90.44
22.97
14,53
66.33
33.51
12.27
3.03
0.77
0.25
0.07
0.02
PGD-8
20.05
67.73
46.49
12.37
PGD-12 12
77.86
28.55
22.39
63.90
48.25
32.19
18.78
9.58
4.12
1.59
0.72
PGD-16
23.36
PGD-20
61 1606
27.73
23.57
51.75
43.32
34.22
25.19
16.36
9.65
5.33
2.73
PGD-24
10.90
9.98
9.86
10.60
10.34
10.11
10.01
9.91
9.74
9.39
8.81
PGD-28
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10,00
10.00
10.00
PGD-32
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10.00
10.00
PGDLS--4
89.87
22.43
14,00
63.98
31.93
11:57
3.43
0.77
0.18
0.09
0.05
PGDLS-8
85.63
27.22
19.92
67.96
46.19
26.24
12.28
4.54
1.52
FGDLS-12
79.39
28.50
22.14
64.63
48.10
31.40
17.99
8.80
4.01
1.51
0.67
PGDLS.
70.68
28.53
23,26
59.44
47.04
33.78
21.94
12.79
6.66
3.07
1.34
PGDI 5-20
65.81
27.82
23.07
54.96
44.46
33.41
22.94
14.27
8.07
4.37
2.08
PGDLS 24
58.36
27.25
23.36
49.09
41.47
32.90
24.84
16.93
10.88
7.04
3.76
RGD
50.07
22,63
35.m7
19.40
PGDLS-32
38.80
22.79
20.79
26. 19
25.34
24.72
23.21
20.98
18.13
15.12
12.66
MMA-12-sa
88.59
27.54
19.91
67.99
43.62
24.79
12.74
5.85
2.68
1.09
0.51
MMA-12-sdl
88.91
26.68
18.90
67.17
43.63
23.62
10.80
4.07
1.20
0.50
0.18
MMA-20-s00
86.56
24.87
67.07
MMA-20-sdl
85.87
33.07
26.47
65.63
48.11
34.70
24.73
16.45
10.97
7.00
4.14
MMA-32-sdD
30.60
50,20
MMA-32-sd1
84.76
33.49
27.08
64.66
48.23
35.65
25.74
17.86
11.86
7.79
4,88
OMMA-12-sd0
88.52
29.34
21.99
67.49
46.11
29.22
16.65
8.62
4.36
2.05
12-s-1
87.82
30.30
23.11
66.77
46.37
31.19
19:40
10.93
5.72
2.84
1.29
OMMA-20-sd!
87.44
34.49
27.87
67.40
51.55
37.94
26.48
17.76
11.31
6.74
3.76
OMMA-32-sd0
86.11
38.87
32.97
67.57
53,70
42.56
32.88
24.91
18.57
13.79
9.76
OMMA-32-sd1
86.36
39.13
33.23
68.80
56.02
44.62
33.97
24.71
17.37
11.94
8.39
PGD-ens
87.38
28.83
21.51
64.85
47.67
30.37
16.63
7.79
3.01
1.25
0.52
PGDLS-cus
76.73
30.60
24.83
61.16
49.46
36.63
23.90
13.92
7.62
3.91
2.05
PGD MuD-MMary et
87.14
27.36
19.89
68.01
44.70
25.15
12.52
5,50
2.25
0.73
0.27
TRADES
84.92
30.46
23.66
70.96
52.92
18.26
8.37
3.59
25

---

Published as a conference paper at ICLR 2020
Table 10: Accuracies of models trained on MNIST with Eg-norm constrained attacks. These robust accuracies are calculated under only whitebox PGD attacks. sd0 and sd indicate 2 different random seeds.
MNIST Model
Cin Ace
AvgAce
AvgRohAce
RobAcc under diflerent whitebbu only
2.0
3.0
STD
99.21
41.90
27.57
86.61
23.02
0.64
0.00
PGD-1.0
99,30
49.55
37.11
95.07
48.99
4.36
0.01
PGD-2.0
45.79
PGD-3.0
97.14
60.94
51.89
90.02
71.53
40.72
5.28
PGD-4.0
93.41
59.93
51.56
82.41
66.49
44.36
12.99
PGDLS-1.0
99,39
48.17
35.36
94.35
43.96
2.97
0.16
PGDLS-2.0
44.19
95.22
69.73
PGDLS 3.0
97.52
60.60
$1.37
90.87
72.24
38.39
3.99
PGDE55-400
51.44
82.73
67.37
MMA-2.0 sdD
99.27
53.97
42.64
95.59
68.66
6.32
0.01
MMA-2.0-sd1
99.28
54.46
43.26
95.79
68.45
8.79
0,01
MMA-4.0-sd0
98.71
62.51
53.45
93.93
74.06
40.02
5.81
MMA-6.0-sd0
98.32
62.60
53.67
93,16
72.72
39.47
9.35
MMA--600sd11
98.50
62.73
53.79
93:48
73.57
39.25
OMMA-2.0-sa 0-sd0
99.26
54,12
42.83
95.94
68.08
7.27
0.03
0MMA-200-d1
99.21
42.85
95.72
6.72
OMMA-4.0 4.0-
98.61
62.44
$3,40
94.06
73.60
40.29
5.66
93.72
0MMA-6.0-sd0
98.16
62.67
53.79
92.90
72.71
40.28
9.29
0MMA-600-d1
98.45
62.52
53.54
93.37
73.02
38.49
9.28
PGD-ens
98.87
56.57
45.99
94.73
70.98
17.76
0.51
PGDLS ens
94.86
DDN-Roay et
99.02
60.34
50.67
95.65
77.79
26.59
Table 11: Accuracies of models trained on CIFAR10 with fo-n constrained attacks. These robust accuracies calculated under only whitebox PGD attacks. sdO and sd1 indicate different random seeds.
CIFAR10 Model
RobAcc under different whitebox
Ace
AvgAcc
AvgRobAce
0.5
1.0
1.5
2.0
2.5
STD
94.92
15.82
0.00
0.01
0.00
0.00
0.00
0.00
PGD-0.5
89.10
33.64
22.55
65.61
33.23
11.29
2.34
0.29
PGD-1.0
83.25
39.74
31.04
66.69
46.11
26.16
12.00
4.26
PGD-1.5
35.02
62.74
48.35
33.80
20.17
PGD-2.0
71.05
41,88
36.05
59.80
47.92
35.39
23.34
13,81
PGD-2.5
65.17
41.03
36.20
55.66
45.82
35,90
26.14
17.49
PGDLS-0.5
89.43
35.44
22.25
65.50
32.42
10.78
2.17
0.36
PGDLS-1.0
39.50
30.68
67.30
45.35
25.49
4.08
PGDLS-1L
77.03
41,80
34.75
63.76
48.46
33.11
19.12
9,32
21.38
PGD1.S-2.5
66.21
41.34
36.36
56.49
46.72
36.13
25.73
16.75
MMA- AMA-1.00-iB0
88.02
35.58
25.09
66.19
37.80
15.61
4.79
1.06
XMA-1.0-sd
88.92
35,74
25.10
66.81
37.22
15:78
4.57
1.14
MMA-2.0-d00
41.22
32.62
65.98
46.1
28.56
15.60
6.86
MMA-2.0-sd1
85.16
40.60
31.69
65.45
45.27
28.07
13.99
5.67
MMA-3.0-sd0
82.11
43.67
35,98
64.25
47.61
33.48
22.07
12.50
MMAA3300sd1
81.79
43.75
36.14
63.82
47.33
33.79
22.36
13,40
OMMA-110-sd0
89.02
35.49
24.79
65.46
37.38
15.34
4,76
1.00
OMMA-1.0--0
89.97
35.41
24.49
66.24
36.47
14.44
4.43
0.89
0MMA
18.23
OMMA-2.0 2.0
85.04
42.96
34.55
65.23
46,32
31.07
19.36
10.75
0MMA-3.0-d0
46.46
99
64.67
49.34
36.40
26.50
18.02
0MMA-3.0.40
84.00
45.59
37.91
64.31
48.50
35.92
24.81
16.03
32.46
63.27
46.66
PGDLS ens
86.11
41.39
32.45
64.04
46.99
29.11
15.51
6.59
DDN-Rony Rony
89.05
36.25
25.69
66.51
39.02
16.63
5.05
1.24
26

---

Published as a conference paper at ICLR 2020
Table 12: The TransferGap of models trained MNIST with -nomm constrained attacks. Trans- ferGap indicates the gap between robust accuracy under only whitebox PGD attacks and under combined (whitebox+transfer)) PGD attacks, sd0 and sdl indicate 2 different random seeds.
MNIST Model
TransferGap:
RobAcc drop after adding transfer attacks
Cin Ace
AvgÂce
AvgRobAce
0.1
0.2
0.00
0.01
0.00
0.00
0.00
PGD-0.1
0.06
0.07
0.00
0.20
0.08
0.00
PGD-0.2
0.00
0.00
0.00
0.02
0.00
PGD-013
0.38
0.48
0.00
0.00
0.10
1.81
PGD-04
7.55
PGD-0.45
0.00
0.00
0.00
0.00
0.00
0.00
PGDLS-0.1
0.09
0.11
0.00
0.43
0.02
0.00
PGDLS-0.2
0.08
0.11
0.00
0.00
0.42
0.00
PGDI.S-0.4
2.42
3.02
0.01
0.13
1.01
10.93
PGDLS-0045
1.22
0.00
0.30
1.25
3.32
(MA-0.45-.d0
0.83
1.04
0.02
0.25
0.98
2,92
MMA-0.45-sdI
0.80
0.99
0.01
0.18
0.71
3.08
0MMA-0.45.shh
1.40
1.28
4.13
OMMA-0045-sdI
1.42
1.78
0.03
0.28
1.72
5,07
PGD-cus PGDLS-ens
0.04
0.05
0.06
0.12
0.01
0.00
0.05
0.06
0.02
0.16
0.07
0.00
PGD-Madry et al.
0.04
0.05
0.00
0.04
0.15
0.02
TRADES
0.0
0.01
0.0
0.0
0.01
0.02
Table 13: The TransferGap of models trained CIFARIO with Lo -nrm constrained attacks. TransferGap indicates the gap between robust accuracy under only whitebox PGD attacks and under combined (whitebox+transfen PGD attacks, sd0 and sd1 indicate 2 different random seeds.
CIFARIO Model
TransferGap: RobAcc drop after adding transfer attacks
Cin Ace
AvgÅce
AvgRobAce
20
24
28
PGD44
0.02
0.02
0.02
0.02
0.05
0.02
0.02
0.01
0.01
0.01
PGD.8
0.02
0.02
0.00
0.02
0.04
0.02
0.02
0,00
0.01
PGD-12
0.05
0.05
0.02
0.03
0.06
0.11
0.10
0.07
0.03
0.02
PGD-20
0.39
0.44
0.03
0.19
0.49
0.64
0.70
0.60
0.59
0.31
PGD-24
0.03
0.00
0.00
0.01
0.02
0.05
0.05
PGD-18
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
PGD.32
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
PGDLS-4
0.04
0.04
0.00
0.01
0.10
0.11
0.09
0.02
0.01
0.00
PGDLS
PGDL.S-12
0.05
0.05
0.01
0.02
0.06
0.13
0.11
0.06
0.03
0.02
PGDLS.
0.09
0.10
0.01
0.04
0.14
0,22
0.13
0.12
0.09
0.03
PGDI 5-20
0.21
0.24
0.00
0.07
0.28
0.41
0.47
0.28
0.29
0.13
PGDLS-24
0.73
0.82
0.04
0.34
0.80
1.08
1.23
1.22
1.18
0.65
PGDLS-28
1.66
PGDLS.3 32
2.91
3.28
0.03
0.38
1.50
3.25
4.76
5.21
5.30
5.78
MMA-12-sd0
0.67
0.76
0.03
0.20
0.72
1.29
1.58
1.25
0.64
0.35
MMA-13 4MM-11-sd1
0.45
0.50
0.09
0.66
1.05
1.04
0.70
0.28
0.15
0.06
MMA-20-sd0
2,86
3.22
0.15
1.85
4.23
5.23
4.31
2.89
1.68
MMA-20-d11
4.35
4,90
0.19
2.00
4.74
7.43
8.18
7.37
5.67
3.58
300022
MMA-32-sd1
4.42
4.97
0.25
2.28
5.29
7,50
8.01
6.87
5.59
3.96
DMMA--1-.44
3.02
3.40
0.53
3.53
6.00
6.36
5.19
3.12
1.59
0.90
0MMA-12-sd
4.06
4.57
0.54
3.67
7.62
9.08
7.37
4.68
2.46
1.15
OMMA-20-s00
9.66
7.59
7.68
0MMA-2U-sd!
6.72
7.56
1.12
5.95
10.61
12.48
11.72
9.08
6.00
3.51
7.39
OMMA-32-d1
10.38
11.67
1.94
8.90
14.99
17.88
17.15
13.99
10.63
7.92
PGD-ens
0.73
0.82
0.26
0.72
0.66
0.34
0.13
PGDLS-cns
1.08
1.21
0.64
1.25
1.57
1.76
1.64
1.45
0.77
0.62
PGD-Madry et
0.14
0.16
0.00
0.02
0.12
0.37
0.32
0.30
0.09
0.04
TRADES
0.0
0.01
0.0
0.0
0.03
0.02
0.02
0.02
27

---

Published as a conference paper at ICLR 2020
Table 14: The TransferGap of models trained on MNIST with f2-norm constrained attacks. Trans- ferGap indicates the gap between robust accuracy under only whitebox PGD attacks and under combined (watefeoxttrnnsfer) PGD attacks. sd0 and sdl indicate different random seeds.
MNIST Model
TransferGop: RobAre drop after adding transfer allacks
Cin Ace
AvgAce
AvgRobAce
STD
0.06
0.07
0.00
0.24
0.00
PGD-1.0
0.76
0.96
0.01
2.15
1.65
0.01
PGD-2.0
0.24
PGD-3.
0.57
0.72
0.01
0.50
1.79
0.57
PGD-4.0
0.41
0.51
0.07
0.24
0.92
PGDLS-1.0
0.56
0.70
0.02
1.52
1.08
0.16
0.44
PGDLS-34
0.47
0.59
0.01
0.33
1.59
0.43
0.06
MMA 2.0-2.0xxd00
0.12
0.15
0.00
0.29
0.29
0.00
MMA-200dd1
0.13
0.16
0.01
0.27
0.34
0.01
MMA-4.0-s00
0.26
0.33
0.00
0.05
0.68
0.57
MMA-6.0-s00
0.29
0.36
0.00
0.09
0.69
0.66
MMA-6.0-sd1
0.25
0.31
0.00
0.07
0.54
OMMA-2.0-00
0.11
0.13
0.00
0.30
0.24
0.00
0MMA-220-s1
0.09
0MMA-4.0--B0
0.27
0.34
0.00
0.09
0.63
0.64
0MMA-6.0-c00
0.22
0.27
0.00
0.12
0.60
0.36
0MMA-6.0-sd1
0.28
0.00
0.09
PGD-ens PGDLS
0.44
0.55
0.36
0.82
0.97
0.05
0.27
DDN-Rony al.
0.41
0.51
0.00
0.14
1.15
0.77
Table 15: The TransferGap of models trained on CIFARIO with f2-norm constrained attacks. Trans- ferGap indicates the gap between robust accuracy under only whitebox PGD attacks and under combined (whitebox+transfer) PGD attacks. sd0 and sd indicate 2 different random seeds.
CIFARIO Model
TransferGap:
RobAce drop after adding transfer attacks
Ace
AvgAcc
AvgRobAce
U.5
1.0
1.5
2.0
2.5
STD
0.00
0.00
0.00
0.00
0.00
0.00
0.00
PGD-0.5
0.02
0.02
0.00
0.02
0.04
0.03
0.01
PGD-1.0
0.04
0.05
0.00
0.03
0.11
0.08
0.05
PGD-1.5
0.06
PGD-2.0
0.11
0.13
0.04
0.07
0.10
0.19
0.25
PGD-2.5
0.10
0.12
0.06
0.06
0.14
0.22
PGDLS-005
0.03
0.04
0.01
0.02
0.05
0.08
0.03
PGDLS-LO
0.05
0.05
PGDLS-1.5
0.06
0.07
0.00
0.03
0.07
0.12
0.15
PGD1.S-2.5
0.13
0.15
0.04
0.06
0.20
0.22
0.24
MMA-1.0s::0
0.03
0.03
0.01
0.05
0.05
0.03
MMA-1.0-sd
0.05
0.06
0.00
0.06
0.07
0.08
0.07
MMA-200-00
0.74
0.07
1.42
1.36
2MA-200.dl
0.79
0.94
0.09
0.83
1.65
1.36
0.79
MMA-3.0-s00
2.08
2.49
0.03
1.20
3.77
MMA-3.0-sd1
2.59
3.11
0.24
1.74
4.02
4.84
4.71
OMMA-1.0-508
0.31
0.38
0.03
0.49
0.57
0.58
0.21
OMMA-1.0-sd
0.20
0.24
0.08
0.37
0.40
0.26
0.10
0.0--200-00
OMMA-2.0.44
3.28
3.94
0.54
1.96
5.18
6.44
5.56
0MMA-3.0-s0
5.85
0.53
3.75
11.50
OMMA-3.0.
4.93
5.92
0.50
3.28
7.45
9.40
8.95
PGDLS ens
1.01
1.22
0.30
0.78
1.53
2.19
1.28
Mony-Rnyyce al.
0.02
0.02
0.00
0.00
0.03
0.03
0.04
28


Reviews:
[{"review_id": "r1xsfMaxFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nThis paper proposes an adaptive margin-based adversarial training (eg. MMA) approach to train robust DNNs by maximizing the shortest margin of inputs to the decision boundary. Theoretical analyses have been provided to understand the connection between robust optimization and margin maximization. The main difference between the proposed approach to standard adversarial training is the adaptive selection of the perturbation bound \\epsilon. This makes adversarial training with large perturbation possible, which was previously unachievable by standard adversarial training (Madry et al.) Empirical results match the theoretical analysis.\n\nPros:\n1. The margin maximization idea has been well-explained, both intuitively and theoretically.\n2. Interesting theoretical analyses and understandings of robust optimization from the margin perspective.\n3. Clear advantage of MMA over standard adversarial training under large perturbations.\n\n8. The proposed PGDLS is very interesting, actually quite good and much simpler, without extra computational cost. A similar idea was discussed in paper [3], where they gradually increase the convergence quality of training adversarial examples, and show the convergence guarantee  of \"dynamic training\".\n9. The gradient-free SPSA helps confirm the improvements of MMA under large perturbations are not a side effect of gradient masking.\n\n\nCons:\n1. The idea of \"shortest successful perturbation\" appears like a type of weak training attack, looking for minimum perturbations to just cross the classification boundary, like deepfool [1] or confidence 0 CW-L2 attack [2]. \n2. The margin d_\\theta in Equation (1)/(2)/... defined on which norm? L_\\infty or L2 norm? I assume it's the infinity norm. In Theorem 2.1, the \\delta^{*} = argmin ||\\delta||, is a norm? Looks like a mistake. \n3. The minimum margin \\delta^{*} is a bit confusing, is it used in maximization or just in the outer minimization? The last paragraph of page 3, L(\\theta, \\delta) or L(\\delta, \\theta), consistency check?\n4. Why do we need the \"gradients of margins to model parameters\" analysis from Proposition 2.1 to remark 2.2? Given the \\delta^{*} found in the inner maximization (eg. attacking) process (step 1), minimizing the loss over this \\delta^{*}  seems quite a straightforward step 2. Why don't go directly from Theorem 2.1 to Proposition 2.4, since the extensions from LM loss to SLM and CE loss via Proposition 2.3 -> Proposition 2.4., just proves that the standard classification loss CE can already maximize the margin given \\delta^{*}? \n5. Section 4 Experiments. The experimental settings are not clear, and are not standard. What CIFAR10-\\ell_{\\infty} means: is it the CW-L2 attack, used for training, or for testing? How the test attacks were generated, the m and N, are confusing: for each test image, you have 260 samples for CIFAR10 (which means 260*10K in total), or just 260 in total (this is far less than a typical setting causing inaccurate results)? How are the d_max determined, and what are their relationship to standard \\epsilon? How the m models were trained?\n6. Fairness of the comparison. Since MMA changes \\epsilon, how to fairly compare the robustness to standard epsilon bounded adversarial training is not discussed. Is it fair to compare MMA-3.0 vs PGD-2.5, since they have different epsilon? Why robustness was not tested against strong, unrestricted attacks like CW-L2 [2], and report the average L2 perturbations required to completely break the robustly trained model (and show MMA-trained models enforce large perturbations to succeed)?\n7. Significance of the results. Normally, \\epsilon_{infty} > 16/255 will cause perceptual difference. Under 16/255, PGD-8/16, PGDLS-8/16 are still the best. At this level, it is quite a surprise that MMA does not improve robustness, although it does increase clean accuracy. This means the theoretical analysis only stand under certain circumstances. I don't think the optimal \\epsilon < margin can explain this, as it does not make sense to me the margin can be larger than 16/255. On the other hand, I thought the theoretical parts were discussing the  ROBUSTNESS, not the CLEAN ACCURACY? But it turns out the MMA benefits a lot the clean accuracy?  Why do we need robustness against large \\infty perturbations, this definitely deserves more discussion, as when perturbation goes large, the L2 attack (eg. CW-L2) makes more sense than PGD-\\infty.\n\n\n[1] Moosavi-Dezfooli, Seyed-Mohsen, Alhussein Fawzi, and Pascal Frossard. \"Deepfool: a simple and accurate method to fool deep neural networks.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.\n[2] Carlini, Nicholas, and David Wagner. \"Towards evaluating the robustness of neural networks.\" 2017 IEEE Symposium on Security and Privacy (SP). IEEE, 2017.\n[3] Wang, Yisen, et al. \"On the Convergence and Robustness of Adversarial Training.\" International Conference on Machine Learning. 2019.\n\n============\nMy rating stays the same after reading through all the responses. I appreciate the authors' clarification on the notations and experimental settings. My 8/9 are positive points. My major concern is still the effectiveness of the proposed approach, and fairness of the comparison.  It seems that MMA only works when the perturbation is large, which often larger than the \\epsilon used to train baseline adversarial training methods such as Trades. The authors seem have misunderstood my request for CWL2 results, I was just suggesting that the average L2 perturbation of CWL2 attack can be used as a fair test measure for robustness, instead of the AvgRobAcc used in the paper, and the susceptible comparison between  MMA-12 vs PGD-8, or MMA-32 vs Trades. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SyenNZBnFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method, Max-Margin Adversarial (MMA) training, for robust learning against adversarial attacks. In the MMA, the margin in the input space is directly maximized. In order to alleviate an instability of the learning, a softmax variant of the max-margin is introduced. Moreover, the margin-maximization and the minimization of the worst-case loss are studied. Some numerical experiments show that the proposed MMA training is efficient against several adversarial attacks. \n\n* review:\nOverall, this paper is clearly written, and the readability is high. Though the idea in this paper is rather simple and straightforward, some theoretical supports are presented. A minor drawback is the length of the paper. The authors could shorten the paper within eight pages that is the standard length of ICLR paper. \n\n- In proposition 2.4: the loss L^{CE} should be clearly defined. \n- In equation (8), how is the weight of L^CE and L^MMA determined?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "SJxahVPkcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper propose to use maximal margin optimization for correctly classified examples while keeping the optimization on misclassified examples unchanged. Specifically, for correctly classified examples, MMA adopts cross-entropy loss on adversarial examples, which are generated with example-dependent perturbation limit. For misclassified examples, MMA directly applies cross-entropy loss on natural examples.\n\nProblems:\n1. For the performance measurement, why use the AvgRobAcc? does it make any sense to combine black-box results and white-box results?\n2. For the epsilon, since it is different from the standard adversarial settings, how to guarantee the fair comparison? For example, how to evaluate the performance of  MMA-12 to PGD-8 under the same test attack PGD-8?\n3. For the baseline, the authors lack some necessary baselines, like the following [1] and [2]\n[1] Theoretically Principled Trade-off between Robustness and Accuracy. ICML 2019\n[2] On the Convergence and Robustness of Adversarial Training. ICML2019", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a new loss function that combines the usual cross-entropy term with a margin maximization term applied to the correctly classified examples. There have been a lot of recent ideas on how to incorporate margin into the training process for deep learning. The paper differs from those in the way that it computes margin. The paper shows that training with the proposed max margin loss results in robustness against some adversarial attacks.
There were initially some concerns about baseline comparisons; one of the reviewers requesting comparison against TRADES, and the other making comments on CW-L2. In response, authors ran additional experiments and listed those in their rebuttal and in the revised draft. This led some reviewers to raise their initial scores. At the end, majority of reviewers recommended accept. Alongside with them, I find extensions of classic large margin ideas to deep learning settings (when margin is not necessarily defined at the output layer) an important research direction for constructing deep models that are robust and can generalize. 
Decision: Accept (Poster)