SOAR: Second-Order Adversarial Regularization
Authors: Avery Ma
Fartash Faghri
Nicolas Papernot
Amir-massoud Farahmand
Venue:  (2021)
Keywords: Adversarial Robustness

Abstract: Adversarial training is a common approach to improving the robustness of deep neural networks against adversarial examples. In this work, we propose a novel regularization approach as an alternative. To derive the regularizer, we formulate the adversarial robustness problem under the robust optimization framework and approximate the loss function using a second-order Taylor series expansion. Our proposed second-order adversarial regularizer (SOAR) is an upper bound based on the Taylor approximation of the inner-max in the robust optimization objective. We empirically show that the proposed method improves the robustness of networks against the $\ell_\infty$ and $\ell_2$ bounded perturbations on CIFAR-10 and SVHN.

Under review as a conference paper at ICLR 2021
SOAR: SECOND-ORDER ADVERSARIAL REGULARIZATION
Anonymous authors Paper under double-bling review
ABSTRACT
Adversarial training is a common approach to improving the robustness of deep neural networks against adversarial examples. In this work, we propose a novel regularization approach as an alternative. To derive the regularizer, we formulate the adversarial robustness problem under the robust optimization framework and approximate the loss function using a second-order Taylor series expansion. Our proposed second-order adversarial regularizer (SOAR) is an upper bound based on the Taylor approximation of the inner-max in the robust optimization objective. We empirically show that the proposed method improves the robustness of networks against the too and & bounded perturbations on CIFAR-10 and SVHN.
INTRODUCTION
Adversarial training (Szegedy et al., 2013) the standard approach for improving the robustness of deep neural networks (DNN), any other model, against adversarial examples. It is a data augmentation method that adds adversarial examples to the training set and updates the network with newly added data points. Intuitively, this procedure encourages the DNN not make the same mistakes against an adversary. By adding sufficiently enough adversarial examples, the network gradually becomes robust the attack was trained on. One of the challenges with such data augmentation approach is the tremendous amount of additional data required for learning a robust model. Schmidt et al. (2018) show that under Gaussian data model, the sample complexity of robust generalization is vd times larger than that of standard generalization. They further suggest that current datasets (c.g., CIFAR-10) may not be large enough to attain higher adversarial accuracy. data augmentation procedure, however, is an indirect way improve the robustness Our proposed alternative is to define regularizer that penalizes DNN parameters prone to attacks. Minimizing the regularized loss function leads estimators robust adversarial examples.
DNN.
Adversarial training and our proposal can both be formulated in terms of robust optimization frame- work for adversarial robustness (Ben-Tal et al., 2009; Madry et al., 2018; Wong & Kolter, 2018; Shaham et al., 2018; Sinha et al., 201 8). In this formulation, one is seeking to improve the worst- case performance of model, where the performance is measured by a particular loss function f. Adversarial training be understood approximating such worst-case loss by finding the corresponding worst-case data point, i.c., x + with some specific attack techniques. Our proposed method is more direct. It is based approximating the loss function {(x + 8) using its second-order Taylor series expansion, i.e.,
1 2 V
((1 5) ~ f(x) + [
and then upper bounding the worst-case loss using the expansion terms. By considering both gradient and Hessian of the loss function with respect (w.r.t)) the input, we can provide more accurate consider both l and foo attacks. In
approximation to the worst-case loss. In our derivations, our derivations, the second-order expansion incorporates both the gradient and Hessian of the loss function with respect (w.r.t.) the input. We call the method Recond-Order Adversarial Regularizer (SOAR) (not to be confused with the Soar cognitive architecture Laird 2012). In the course of development of SOAR, we make the following contributions:
We show that an over- wer-parameterieee linear regression model can be severely affected by an adversary, even though its population loss is zero. We robustify it with regularizer that

---

Under review as a conference paper at ICLR 2021
exactly mimics the adversarial training. This suggests that regularization can be used instead of adversarial training (Section 2).
Inspired by such a possibility, we develop a regularizer which upper bounds the worst-case effect of an adversary under an approximation of the loss. In particular, we derive SOAR, which approximates the inner maximization of the robust optimization formulation based on the second-order Taylor series expansion of the loss function (Section 4).
We study SOAR in the logistic regression setting and reveal challenges with regularization using Hessian w.r.t. the input. We develop simple initialization method to circumvent the issue (Section 4.1).
We empirically show that SOAR significantly improves the adversarial robustness of the network against &o attacks and & attacks 1IFAR-10 and SVHN. Specifically, we evalu- ate using a PGD1000 white-box attack (Madry et al., 1), transferred PGD1000 attacks, AutoAttack (Croce & Hein, 2020), and SimBA (Guo al., 2019).
2 LINEAR REGRESSION WITH AN OVER- PTEE-PARAMETRIIED MODEL
This section shows that for over weer--arameterized linear models, gradient descent (GD) finds solution that has zero population loss, but is prone to attacks. It also shows that one can avoid this problem with defining an appropriate regularizer. Hence, we do not need adversarial training to robustify such a model. This simple illustration motivates the development of our method in next sections. We only briefly report the main results here, and defer the derivations to Appendix A.
Consider a linear model (x) = ( [w, x) with x, w E Rdd Suppose that w = (1,0,0,.... O)T and the distribution of (x1, 0, ...... 0) : I1 E J. This setup can be thought of as using an peeapaarmetere model that has many irrelevant dimensions with data that is only covering the relevant dimension of the input space. This is a simplified model of the situation when the data manifold has a dimension lower than the input space. We consider the squared error pointwise loss ((1; w) = 2 K x w) (x, w )²² Denote the residual by r(æ; w) (x, w w²) ), and the population loss by [(w) = E (Xx:))
p is such that it is confined on 1-dimensional subspace
Suppose that we initialize the weights as w() = W ~ (0, L'xx,, and use GD on the population loss, i.c., w(t + 1) w(t) - ((w). It is easy to see that the partial derivatives w.r.t. w2.....d are all zero, i.e., no weight adaptation happens. With a proper choice of learning rate 3, we get that the asymptotic solution is w 2 lim w(t) = (wi, (0), random weights on dimensions 2, ...... d do not change.
That is, the initial
We make two observations. The first is that L(w)- = 0, i.c., the population loss is zero So from the perspective of training under the original loss, we are finding the optimal solution. The second observation is that this model is vulnerable to adversarial examples. An FGSM-like attack that perturbs x by Ax (0, Azg. Arg, ............ ... Aza)" with Ar, e sign(w: (0)) (for i 2.....d. d) has the population loss of Wx.w [(X + Ar); w)) x d(="d"o")) under the adversary at the asymptotic solution W. When the dimension is large, this loss is quite significant. The culprit is obviously that GD is not forcing the initial weights go to when there is no data from irrelevant and unused dimensions. This simple problem illustrates how the optimizer and an over- parammmtttte model might interact and lead to solution that is prone attacks.
An effective solution is to regularize the loss such that the weights of irrelevant dimensions to go to zero. Generic regularizers such ridge and Lasso regression lead to biased estimate of wf, and thus, one is motivated to define a regularizer that is Bishop (1995) showed the close connection between training with random perturbation and Tikhonov Regularization. Inspired by this idea, we develop regularizer that mimics the adversary itself. For this FGSM-like adversary, the population loss the perturbed point is
for improving adversarial robustmess.
(1) Minimizing Crobustined (w) is equivalent to minimizing the model at the point x' = x + Az. The regularizer =E [x;;; w)] || , + & || incorporates the effect of adversary in exact form. Nonetheless, there are two limitations of this approach. The first is that it is designed for a particular choice of attack, an FGSM-like one. We would like a regularizer that is robust larger class
Crobustiffcd (w:) A TE [1(X + Ar; w)) = L(w) + eIE [r(X; (X] [ 1 +
2 ||
2

---

Under review as a conference paper at ICLR 2021
of attacks. The second is that this regularizer is designed for linear model and the squared error loss. How can we design a regularizer for more complicated models, such as DNNs? We address these questions by formulating the problem of adversarial robustness within the robust optimization framework (Section 3), and propose an approach to approximately solve it (Section 4).
ROBUST OPTIMIZATION FORMULATION
Designing an adversarial robust estimator can be formulated as robust optimization problem (Huang et al., 2015; Madry et al., 2018; Wong & Kolter, 2018: Shaham et al., 2018). To describe it, let us introduce our notations first. Consider an input space X' C Rd an output space D. and a parameter (or hypothesis) space W, parameterizing a model f : X' x W + y In the supervised learning scenario, we are given a data distribution D over pairs of examples {(X;, Y.)) Given the prediction of fa;; w) and target value y, the pointwise loss function of the model is denoted by E(zz y; w) & ((f(x; w), y). Given the distribution of data, one can define the population loss as [(w) = E [8(X, w)) The goal of the standard supervised learning problem is to find a w e W that minimizes the population loss. A generic approach to do this is through empirical risk minimization (ERM). Explicit or implicit regularization is often used to control the complexity of the hypothesis to avoid over- or under-fitting (Hastie et al., 2009).
As shown in the previous section, it is possible to find a parameter w that minimizes the loss through ERM, but leads notion in the model, it requires defenders to reconsider the training objective. It is also important to formalize and constrain the power of the adversary, so we understand the strength of the attack to which the model is resistant. This can be specified by limiting that the adversary can only modify any input x to x 8 with 8 E A C X. Commonly used constraints are g-balls w.r.t. the l no-oorss, though other constraint sets have been used too (Wong et al., 2019b). This goal can be formulated as a robust optimization problem where the objective is to minimize the adversarial population loss given some perturbation constraint A:
model that is vulnerable to adversarial examples. To incorporate the robustness
min E (xXY)~ max ((X + 6, w] SEA
We have an interplay between two goals: 1) the inner-max term looks for the worst-case loss around the input, while 2) the outer-min term optimizes the hypothesis by minimizing such loss.
Note that solving the inner- max problem is often computationally difficult, so one may approximate it with surrogate loss obtained from a particular attack. Adversarial training and its variants (Szegedy et al., 2013; Goodfellow et al., 2014; Kurakin et al., 2016; Madry et al., 2018; Wong et al., 2019a) can be intuitively understood as an approximation of this min-max problem via different 8(z).
As shown in Section 2, one can design regularizer that provides the exact value of the loss function at the attacked point for a particular choice of model, loss function, and adversary, cf. (1). Under the robust optimization framework, the regularizer and adversarial training are two realizations of the inner-max objective in (2), but using such a regularizer relieved us from using a separate inner optimization procedure, is done in adversarial training. Motivated by that example and the robust optimization framework discussed here, we develop a regularizer that can be understood as an upper-bound on the worst-case value of the loss an attacked point under a second-order approximation of the loss function.
4 SECOND-ORDER ADVERSARIAL REGULARIZER (SOAR)
The main idea of SOAR to approximate the loss function using the second-order Taylor series expansion around an input æ and then solve the inner maximization term of the robust optimization formulation (2) using the approximated form. We show this for both Eg and fox attacks; the same idea can be applied to other Ep norms. We describe crucial steps of the derivation in this section, and defer details to Appendix B
Assuming that the loss is twice- wiice-differentiable, we can approximate the loss function around input by the second-order Taylor expansion
[xx + 8, yı w) Re la (x 8, v: w) A ((, + V,ll,, w) T + 1 V w)o. (3)
T
2
3

---

Under review as a conference paper at ICLR 2021
For brevity, drop w. y and use V to denote Vg. Let us focus the lp attacks, where the constraint set in (2) is A = {ō |ôl1, I } for some > and 2 1. We focus on the loo attack because of its popularity, but we also derive the formulation for the &g attacks.
As warm-up, let us solve the inner optimization problem by considering the first-order Taylor series expansion. We have
FFAR (I) - max ((x) + Vi(x) ((x) & V[(()) 10 Se
(4)
The term V[(x)]l defines the First-Order Adversarial Regularizer (FOAR). This is similar to the regularizer introduced by Simon-Gabriel et al. (2019) with the choice of eo perturbation set. For a general lp- fp-attach with 1 S I oc, we have ||Ve(=))l, with g satisfying p- + q-- 1. We shall empirically evaluate FOAR-base approach (for the lor attack), but our focus is going to be on solving the inner maximization problem based on the second-order Taylor expansion:
1 + 2
max (xx) + V((x) |
((x(()
(5)
for p 2, co. The second-order expansion in (3) can be rewritten as
8 ((x = ((x) + 2 [ v((x)
(((())
v((x)
6
1 8 H6' 2
1 2
2 = ((x)
(6)
[6: 1]. This allows us to derive an upper bound on the expansion terms using the characteristics of single Hessian term H. Note that 8' is d 1-dimensional vector and H is (d + 1) x (d + 1) matrix. We need to find an upper bound on 8' H& under the attack constraint.
where 8' = 16:
For the &o attack, solving this maximizing problem is not easy in (4) since the Boolean quadratic programming problem in formulation (5) is NP-hard. But we can relax the constraint set and find an upper bound for the maximizer. Note that with 5 E Rdd, -bal of size is enclosed by 22-ball of size /de with the same centre. Therefore, we can upper bound the inner maximization by
max l(x + 6 < max {(x + 88,
||
|1
dds
which after substituting the second-order Taylor series expansion leads an f2-constrained quadratic optimization problem
((x) +
max 8' HJ |161,5vde
(8)
2'
with 8' = jō; 1] as before. The l version of SOAR does not require this extra step, and we have instead of /e in (8). A more detailed discussion on the above relaxation procedure is included in Appendix B.2.
Proposition 1. Let & Idd R be twice-differentiable function. For any ε >> O, we have
max Exmals + 8) S {(x) + dea +
E Ha|l 2'
| Se
where H defined (6) and N(0, (()+))))++))
This result upper bounds the maximum of the second-order approximation End over an too ball with radius e. and relates it to an expectation of a Rlessian-vector product. Note that there is a simple correspondence between (1) and regularized loss in (9). The latter can be understood as upper bound on the worst-case damage of an adversary under a second-order approximation of the loss. For the lg attack, the same line of argument leads to e2 + 1 instead of de
Let us take a closer look Hz. By decomposing [zd, 21]
we get
(²((x)za++ V((x)
H
Vi(x) 2d + z1
4

---

Under review as a conference paper at ICLR 2021
The term (2) be computed using Finite Difference (FD) approximation. Note that E [||ا|||]]= Vd for our Normally distributed z. To ensure that the approximation direction has the
same magnitude, we use the normalized Ed
Zd instead, and use the approximation below
T=all2
Vi(a + hza) v((x)
(2((2 Re |eall2
(10)
h
To summarize, SOAR regularizer evaluated at with a direction 2, and FD step size h > is R(1;2, dea +1
|zall2 ( valm)
+ V((a))
(11)
Ve(x) +
The expectation in (9) can then be approximated by taking multiple samples of drawn from z (0, ((+1))(++1)) These samples would be concentrated around its expectation. One can show that P H|| E []] > t} S 2xp(( ( ) where e is constant and ||]| is the é induced norm (see Theorem 6.3.2 of Vershynin 2018). In practice, we observed that taking more than one sample of do not provide significant improvement for increasing adversarial robustness, and include an empirical study on the the effect of sample sizes in Appendix E.4.
Before we discuss the remaining details, recall that we fully robustify the model with an appropriate regularizer in Section 2. Note the maximizer of the loss based on formulation (2) is exactly the FGSM direction, and (1) shows the population loss with our FGSM-IP like choice of Ar. To further motivate a second-order approach, note that we can obtain the first two terms in (1) with a first-order regularizer such FOAR; and we recover the exact form with a second-order formulation in (5).
Next, we study SOAR in the simple logistic regression setting. which shows potential failure of the regularizer and reveals why we might observe gradient masking. Based on that insight, we provide the remaining details of the method afterwards in Section 4.1.
4.1 AVOIDING GRADIENT MASKING
Consider a linear classifier ʃ Rd Rd + R with [(x; = =(((,, x)), where x, w E Rd the input and the weights, and o(-) is the sigmoid function. Note that the output of f has the interpretation being a Bernoulli distribution. For the cross-entropy loss function É(2z y; w) = -ly log (a;; w) + y) log(1 [<; w) the gradient w.r.t. the input x: is V((x)= (f(x; w) y)w and the Hessian w.r.t. the input x is (²((z)= = [x; w) - f(z; w))
The second-order Taylor series expansion (3) with the gradient and Hessian evaluated at is
1
{(x + ~ {(x) + + yi w) 8 u((;; w)8 (aoj)Twee ww To,
(12)
2
where rxx, w) f(x; w)) is the residual term describing the difference between the predicted probability and the correct label, and u = u(x; w) = f(; w) (1- f(r; w)). Note that u can be interpreted as how confident the model is about its predication (correct incorrect), and is close to 0 whenever the classifier is predicting a value close to 0 or 1. With this linear model, the maximization (8) becomes
1 [rw]] 8 _ud" www 8] =(xx +evdlr(a y (x)) ||||| + de² u(a; w) ||w||22
((x)
max svie
2
The regularization term is encouraging the norm of w to be small, weighted according to the residual r(x, yı w) and the uncertainty u(x; w.)
Consider linear interpolation of the cross-entroply loss from æ to a perturbed input x'. Specifically, we consider f(ax + (1 a)z²' for a E [0, 1). Previous work has empirically shown that the value of increases from 0 to (Madry al., 2018). In such case, since there is very little curvature at z, if we use Hessian exactly at xr, it leads to an inaccurate approximation of the value {xx'. Consequently, we have poor approximation of the inner-max, and the derived regularization will not be effective.
the loss behaves logistically
For the approximation in (12), this issue corresponds to the scenario in which the classifier is very confident about the clean input at z. Standard training techniques such as minimizing the cross- entropy loss optimize the model such that it returns the correct label with high confidence. Whenever
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Computing the SOAR objective for a single training data
Input A pair of training data (z, y). foo constraint of E, Finite difference step-size h.
x'' t x + 7/ where 7 + (7, T2...., na)" and 1); ~ L1( -1(-2.))
x' - ((..5) {z' + sign (Va((²))) where ll is the projection operator.
Sample 2 ~ (0, I(+(()))))
Compute SOAR regularizer R(z': 2, h) as (11).
Compute the pointwise objective: (SOAR( v) ((1',)) 2,,
the classifier is correct with a high confidence, both r and u will be close to zero, As a result, the effect of the regularizer diminishes, i.e., the weights are no longer regularized. In such case, the Taylor series expansion, computed using the gradient and Hessian evaluated at z; becomes an inaccurate approximation to the loss, and hence its maximizer is not good solution the inner maximization problem.
Note that this does not mean that one cannot use Taylor series expansion to approximate the loss. In fact, by the mean value theorem, there exists an h* E (0. 1) such that the second-order Taylor expansion is exact: f(x + 5) {(x) Vi(x) 8 + 30 V2t(I + h 8)8. The issue is that if compute the Hessian at x (instead of at x + h 6), our approximation might not be very good whenever the curvature profile of the loss function at a: is drastically different from the one at x: + h
More importantly, a method relying on the gradient masking can be easily circumvented (Athalye al., 2018). Our early experimental results had also indicated that gradient masking occurred with SOAR when the gradient and Hessian were evaluated at x. In particular, we observe that SOAR with zero-initialization leads to models with nearly 100% confidence on their predictions, leading to an ineffective regularizer. The result is reported in Table 5 in Appendix D.
This suggests a heuristic to improve the quality of SOAR. That is to evaluate the gradient and Hessian, through FD approximation (10) at a less confident point in the Loo ball of I. We found that evaluating the gradient and Hessian at 1-step PGD adversary successfully circumvent the issue (Line 1-2 in Algorithm 1). We compare other initializations in Table 5 in Appendix D. To ensure the regularization of the original loo ball of e, we use & for PGDI initialization, and then & in SOAR.
Based on this heuristic, the regularized pointwise objective for data point (I,,)) v) is
ESOAR( (z. 9) ((x'.)) + ((,';2,h))
(13)
where ~ (o0, I [d+11)x and the point x' is initialized at PGD1 adversary. Algorithm 1 summarizes SOAR on single training data. We include the full training procedure Appendix C. Moreover, we include additional discussions and experiments on gradient masking in Appendix E. 1.
4.2 RELATED WORK
Several regularization-brn basse alternatives to adversarial training have been proposed. Simon-Gabriel al. (2019) studied regularization under the first-order Taylor approximation. The proposed reg- ularizer for the foo perturbation set is the same as FOAR. Qin et al. (2019) propose local linearity regularization (LLR), where the local linearity measure is defined by the maximum error of the first- order Taylor approximation of the loss. LLR minimizes the local linearity mesaure, and minimizes the magnitude of the projection of gradient along the corresponding direction of the local linearity mesaure. It is motivated by the observation of flat loss surfaces during adversarial training.
CURE MMosavi-Dezzooo et al., 2019) the closest to our method. They empirically observed that adversarial training leads to reduction in the magnitude of eigenvalues of the Hessian w.r.t. the input. Thus, they proposed directly minimizing the curvature of the loss function to mimic the effect of adversarial training. An important advantage of our proposed method is that SOAR is derived from a complete second-order Taylor approximation of the loss, while CURE exclusively focuses on the second-order term for the estimation of the curvature. Note the final optimization objective in SOAR, FOAR, LLR and CURE contains derivative w.r.t. the input of the DNN, and such a technique was first introduced to improve generalization by Drucker & Le Cun (1992) double backpropagation.

---

Under review as a conference paper at ICLR 2021
Table I: Performance on CIFAR-10 10 against Lo bounded white-box PGD attacks (c = 8/255).
Method
Standard
FGSM
PGD20
PGDI00
PGD200
PGD1000
PGD20-50
Standard
95.79%
44.77%
0.03%
0.01%
0.00%
0.00%
0.00%
WideResNell
ADV
87.14%
55.38%
45.64%
45.05%
45.01%
44.95%
45.21%
TRADES
84.92%
60.87%
55.40%
55.10%
55.11%
55.06%
55.05%
MART
83.62%
61.61%
56.29%
56.11% %
56.10%
56.07%
55.98%
MMA
84.36%
61.97%
52.01%
51.26%
51.28%
51.02%
50.93%
Standard
92.54%
21 21.59%
0.14%
0.09%
0.10%
0.08%
0.10%
ADV
80.64%
50.96%
42.86%
42.27%
42.21%
42.17%
42.55%
ResNet
TRADES
75.61%
50.06%
45.38%
45.19%
45.18%
45.16%
45.24%
MART
75.88%
52.55%
46.60%
46.29%
46.25%
46.21%
46.40%
MMA
82.37%
47.08%
37.26%
36.71%
36.66%
36.61%
36.85%
FOAR 65.84%
36.96%
32.28%
31.87%
31.89%
31.89%
32.08%
SOAR
87.95%
67.15%
56.06%
55.00%
54.94%
54.69%
54.20%
Another related line adversarial regularization methods do not involve approximation the loss function nor robust optimization. TRADES (Zhang al., 2019) introduces regularization term that penalizes the difference between the output of the model on a training data and its corresponding adversarial example. MART (Wang al., 2020) reformulated the training objective by explicitly differentiating between the mis-classified and correctly classified examples. Ding et al. (2019b) present another regularization approach that leverages adaptive margin maximization (MMA) correctly classified example to robustify the model.
EXPERIMENTS
In this section, we verify the effectiveness of the proposed regularization method against to PGD attacks on CIFARIO. Our experiments show that training with SOAR leads significant improve- ments in adversarial robustness, which is achieved without significantly sacrificing standard accuracy. We focus on foc this section and defer evaluations on & in Appendix E.5. Additionally, we provide a detailed discussion and evaluations on the SVHN dataset in Appendix E.6.
We train ResNet-10 10 (He al., 2016) on the CIFAR-10 dataset. The baseline methods consist of: (I) Standard: training with adversarially perturbed data; (2) ADV: training with 10-step PGD adversarial examples; (3) TRADES; (4) MART and (5) MMA. Empirical studies Madry al. (2018) and Wang al. (2020) reveal that their approaches benefit from increasing model capacity to achieve higher adversarial robustness, as such, we include WideResNet (Zagoruyko & Komodakis, 2016) for all baseline methods. We were not able reproduce the results of two closely related works, CURE and LLR, which we discuss further in Appendix E, 1. In Appendix E. 13, we compare SOAR and FOAR with different initializations. FOAR achieves the best adversarial robustness using PGDI initialization, so we only present this variation of FOAR in this section.
The optimization procedure is described in detail in Appendix E.2. Note that all methods in this section are trained to defend against foe norm attacks with 8/ /55 as this is popular choice of the literature. The PGD adversaries discussed in Sections 5.1 and 5.2 are generated with : 8/255 and a step size of 2/ 2/255 (pixel values are normalized to [0. 1]). PGD20-50 denotes 20-step PGD attacks with 50 restarts. Section 5.3, we compare SOAR with baseline methods on foc AutoAttack (Croce & Hein, 2020) adversaries with varying e. Additionally, results of all methods on ResNer10 are obtained by averaging over independently initialized and trained models, where the standard deviations are reported in Appendix 10. We use the provided pretrained WideResNet model provided in the public repository of each method. Lastly, discussions on challenges (i.e., difficult to train from scratch, catastrophic overfitting, BatchNorm, etc.) encountered while implementing SOAR and our solutions (i.e., using pretrained model, clipping regularizer gradient, early stopping, etc.) are included in Appendix E.7.
7

---

Under review as a conference paper at ICLR 2021
Table Performance on CIFAR-10 against foo bounded black-box attacks (e 8/255).
Method
SimBA
PGD20-R
PGD20-W
PGD1000-R
PGD1000-W
WideResNer
ADV
49.20%
84.69%
86.30%
84.69%
86.24%
TRADES
58.97%
82.18%
83.88%
82.24%
83.95%
MART
59.60%
80.79%
82.62%
80.96%
82.76%
MMA
58.30%
80.73%
82.74%
80.76%
82.74%
ADV
47.27%
77.19%
79.48%
77.22%
79.55%
TRADES
47.67%
72.28%
74.39%
72.24%
71.37%
ResNet
MART
48.57%
72.99%
74.91%
72.99%
75.04%
MMA
43.53%
78.70%
80.39%
78.72%
81.35%
FOAR 35.97%
63.56%
63.20%
63.60%
65.27%
SOAR
68.57%
79.25%
86.35%
79.49%
86.47%
5.1 ROBUSTNESS AGAINST PGD WIITE-BOXX ATTACKS
Before making the comparison between SOAR and the baselines in Table 1, note that FOAR achieves 32.28% against PGD20 attacks. Despite its uncompetitive performance, this shows that approximating the robust optimization formulation based on Taylor series expansion is a reasonable approach. Furthermore, this justifies our extension to a second-order approximation, as the first- order alone is not sufficient. Lastly, we observe that training with SOAR significantly improves the adversarial robustness against all PGD attacks, leading higher robustness in all k-step PGD attacks on the ResNet model. SOAR remains competitive compared to baseline methods trained on high-capacity WideResNet architecture.
5.2 ROBUSTNESS AGAINST BLACK-BOL ATTACKS
Many defences only reach an illusion robustness through methods collectively known as gradient masking (Athalye et al., 2018). These methods often fail against attacks generated from undefended independently trained model, known as transfer-bas based black-box attacks. Recent works (Tramèr et al., 2017; Ilyas et al., 2019) have proposed hypotheses for the success of transfer-based black-box attacks. our evaluation, the transferred attacks are PGD20 and PGD1000 perturbations generated from two source models: ResNet and WideResNet, which are denoted by the suffix -R and -W respectively. The source models are trained separately from the defence models on the unperturbed training set. Additionally, Tramer et al. (2020) recommends score- base- black-box attacks such as SimBA (Guo et al., 2019). They are more relevant real-world applications where gradient information is not accessible, and are empirically shown be more effective than transfer-based attacks. Because they are solely based on the confidence score of the model, score-based attacks are resistant gradient-masking. All black-box attacks in this section are foo constrained g = 8/255.
SOAR achieves the best robustness against all baseline methods trained on ResNet, as shown in Table 2. Compared with the baselines trained on WeteResNet, SOAR remains the most robust model against transferred PGD20-W and PGD1000-
approaching its standard accuracy on unperturbed data. Note that all defence methods are substantially more vulnerable to the score-based SimBA attack. SOAR regularized model is the most robust method against SimBA.
5.3 ROBUSTNESS AGAINST AUTOATTACK
During the ICLR rebuttal phase, we evaluated SOAR against Autoattack (Croce & Hein, 2020). In this section, we focus on the -oundedd Autoattack, and similar results with the bo-boudded attack included in Appendix E.5. We noticed that SOAR has shown greater vulnerabilities AutoAutack compared to the attacks discussed in Sections 5.1 and 5.2. AutoAttack consists an ensemble of four attacks: two parameter- free versions of the PGD attack (APGD-CE and APGD-DLRS DLR), white-bon box fast adaptive boundary (FAB) attack (Croce & Hein, 2019), a score-based black- box Square Attack (Andriushchenko al., 2020). Notice that the major difference between the two PGD attacks is the
8

---

Under review as a conference paper at ICLR 2021
Table Performance of the ResNet models on CIFAR-I0 against the four Ex bounded attacks used as an ensemble in AutoAttack (E = 8/255, 6/255, 4/255).
Method
Untargeted APGD-CE
Targeted APGD-DLR
Targeted FAB
Square Attack
ADV
41.57|51.88 61..1.144199955 61.9%
38.99|49.615 60.11 %
50.05
47.84/56.09 63.02%
TRADES
54.69 60.667%
40.27 49.08 (58.08888...... 25%
40.64 49.39 58.45%
46.16|551.04 61.22%
MART
45.01 15.01|53.52) 62.05%
39.22/ 18.18844888 18|88388%
39.90 1111191.11111 58.66%
46.90|5/1.51 62.03%
MMA
36.59/46.31 |6.3||881159
(5.77745.90/57.888
(5.50)66.54(582222
45.24|544.05 63.99%
FOAR
(.15/40.16|49.88
97.56)36.22/46.9122
27.92 47.04%
35.92|43 43. 18.18818 18/51805%
SOAR
53.40 68.37) 18.25/33.22//22.66%
20.22 (33 29%
35.94 63.90%
loss they are based on: APGD-CE is based on the cross-entrope loss similar to (Madry et al., 2018), and APGD-DLR is based on the logit difference similar to (Carlini & Wagner, 2017).
To better understand the source of SOAR's vulnerability, we tested it against the four attacks individually. First, we observed that the result against untargeted APGD-CE similar to the one shown in Section 5.1. This is expected because the attacks are both formulated based on cross-entropy- based PGD. However, there is a considerable degradation in the accuracy of SOAR against targeted APGD-DLR and targeted FAB. At 8/255, SOAR is most vulnerable to targeted APGD-DLE with a robust accuracy of only 18. 18. To further investigate SOAR's 's robustness against AutoAttack, tested with different to verify if SOAR can at least improve robustness against &o attacks with smaller e. We observed that at & = 4/255 the robustness improvement of SOAR becomes more consistent. Interestingly, we also noticed that model with better robustness at 8/255 does not guarantee better robustness at = 4/255, as is the case for Square Attack on ADV and SOAR.
Combing the results with the four attacks and with different E, we provide three hypotheses on the vulnerability of SOAR. First, SOAR might overfit to particular type of attack: adversarial examples generated based on the cross-entropy loss. APGD-DLR based on logit difference and FAB is based on finding minimal perturbation distances, which are both very different from the cross- coss-entropy loss. Second, SOAR might rely on gradient masking to certain extent, and thus PGD with cross-entropy loss is difficult to find adversaries while they still exist. This also suggests that the results with black-box attacks might insufficient to conclusively eliminate the possibility of gradient masking. Third, since SOAR provide more consistent robustness improvement at smaller e, this suggests that the techniques discussed in Section 4 did not completely address the problems raised from the second-order approximation. This makes the upper-bound of the inner-max problem loose, hence making SOAR improves robustness against attacks with E smaller than what it was formulated with.
Finally, emphasize that this should not rule SOAR as a failed defence. Previous work shows that mechanism based on gradient masking can be completely circumvented, resulting in 0% accuracy against non-gradient-basel attacks (Athalye et al., 2018). Our result SimBA and Square Attack shows that this is not the case with SOAR, even at e = 8/255, and thus the robustness improvement cannot be only due gradient masking. Overall, we think SOAR's vulnerability to AutoAttack an interesting observation and requires further investigation.
CONCLUSION
This work proposed SOAR, regularizer that improves the robustness of DNN to adversarial examples. SOAR was obtained using the second-order Taylor series approximation of the loss function w.r.t. the input, and approximately solving the inner maximization of the robust optimization formulation. We showed that training with SOAR leads to significant improvement in adversarial robustness under lo and E2 attacks. This is only one step in designing better regularizers to improve the adversarial robustness. Several directions deserve further study, with the prominent one being SOAR's 's vulnerabilities AutoAttack. Another future direction is to understand the loss surface of DNN better in order to select a good point around which an accurate Taylor approximation can be made. This is important for designing regularizers that not affected by gradient masking.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: query-efficieet black-box adversarial attack via random search. In European Conference on Computer Vision, pp. 484-501 Springer, 2020. 8
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. In Proceedings of the 35th International Conference Machine Learning, pp. 274-2842 2018. 6, 8, 9, 26
John Beasley. Heuristic algorithms for the unconstrained binary quadratie programming problem. London, England, 1998. 16
Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization, volume 28.
Princeton University Press, 2009.
Yoshua Bengio, Jéróme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In
Proceedings of the 26th annual international conference on muchine learning, pp. 41-48, 2009. 24
Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, (1):108-116, 1995. 2, 14
Qi-Zhi Cai, Chang Liu, and Dawn Song. Curriculum adversarial training. In Proceedings of the 27th International Joint Conference Artificial Intelligence, pp. 3740-3747, 2018. 23, 24
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. IEEE, 2017. 9
Francesco Croce and Matthias Hein. Minimally distorted adversarial examples with fast adaptive boundary attack. arXiv preprint
nen::: 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fres attacks, arXiv preprint arXiv:2003.016900 2020. 2, 7,
Gavin Weiguang Ding, Kry Yik Chau Lui, Xiaomeng Jin, Luyu Wang, and Ruitong Huang. sensitivity of adversarial robustness to input data distributions. In International Conference Learning Representations, 2019a. 13
Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. In International Conference on Learning Representations, 2019b. 7
Gavin Weiguang Ding, Luyu Wang, and Xiaomeng Jin. AdverTorch v0.1: An adversarial robustness toolbox based on pytorch. arXiv preprint arXiv:1902.07623, 2019c. 21
Harris Drucker and Yann Le Cun. Improving generalization performance using double backpropaga- tion. IEEE Transactions on Neural Networks, 3(6):991-997 1992.
Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W Taylor. Batch normalization is a cause of adversarial vulnerability. arXiv preprint 1rXiv:1905.02161, 2019. 24
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. 2014. 3, 13
Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger. Simple black-box adversarial attacks. In International Conference on Machine Learning, pp. 2484-2493 2019. 2, 8, 21
Trevor Hastie, Robert Tibshirani. and Jerome Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd edition). Springer, 2009. 3
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.7
10

---

Under review as a conference paper at ICLR 2021
Ruitong Huang, Bing Xu, Dale Schuurmans, and Csaba Szepesvári. Learning with strong adversary. arXiv preprint arXiv:1511.00000 2015. 3
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in Neural Information Processing Systems, pp. 125-136, 2019. 8
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456, 2015. 24
Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint arXiv:1611.0022 2016. 3
John Laird. The Soar cognitive architecture. MIT press, 2012.
Ricardo M Lima and Ignacio E Grossmann. On the solution of nonconvex cardinality boolean quadratic programming problems: a computational study. Computational Optimization and Applications, 66( 6611):133, 2017. 16
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. 1, 3, 7,
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 178899086, 2019. 6, 20
Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli. Adversarial robustness through local linearization. In Advances in Neural Information Processing Systems, pp. 13824-13833, 2019. 6, 20
Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J Fleet. Adversarial manipulation of deep representations. arXiv preprint / 2015.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. In Advances in Neural Information Processing Systems, pp. 5014-5026, 2018. 1, 13
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of supervised models through robust optimization. Neurocomputing, 307:195-204, 2018. 3,
Carl-Johann Simon-Gabriel, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, and David Lopez-Paz. First-order adversarial vulnerability of neural networks and input dimension. In International Conference on Machine Learning, pp. 5809-5817, 2019. 4, 6, 22
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014. 24
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying some distributional robustness with principled adversarial training. In International Conference Learning Representations, 2018. 1
Chawin Sitawarin, Supriyo Chakraborty, and David Wagner. Improving adversarial robustness through progressive hardening. arXiv preprint arXiv:2003.09347. 2020. 24
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. 1.3
Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint arXiv: 1704.0330 2017. 8

---

Under review as a conference paper at ICLR 2021
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. arXiv preprint WrXiv:2002.08347,, 2020.
Roman Vershynin. Hfigh-dimensional probability. Cambridge University Press: An Introduction with Applications in Data Science, 2018. 5
Jianyu Wang and Haichao Zhang. Bilateral adversarial training: Towards fast training of more robust models against adversarial attacks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6629 6629-6638, 2019. 23
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International Conference Learning Representations, 2020. 2,25
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pp. 5286-5295, 2018.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2019a. 3,
Eric Wong, Frank Schmidt, and Zico Kolter. Wasserstein adversarial examples via projected sinkhorn iterations. In International Conference on Machine Learning, pp. 6808 6817, 2019b.
Yao- Yua-Yan Yang, Cyrus Rashtchian, Hongyang Zhang, Ruslan Salakhutdinov, and Kamalika Chaud- huri. Adversarial robustness through local lipschitzness. arXiv preprint arXiv:2003.024600 2020. 20
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.0711 2016. 7
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. International Conference Machine Learning, pp. 7472-7482, 2019. 7,
12

---

Under review as a conference paper at ICLR 2021
A DERIVATIONS OF SECTION 2: LINEAR REGRESSION WITH AN OVER- PARAMETRIZED MODEL
We derive the results reported in Section 2 in more detail here. Recall that we consider a linear model
aw (z) = (w, x²) with z, E Rd. We suppose that w' = (1, (1,0, o,.....0 0)* and the distribution of J p is such that is confined on 1-dimensional subspace { (21, 0, 0............. .......01 0) 21 E R }. So the pi (z) (((2))((())) 5(za), where 8(.) is Dirac's delta function.
density of æ is ((I1,...,
We initialize the weights at the first time step as w(() ~ o( and use GD to find the minimizer of the population loss. The partial derivatives of the population loss are
al(w)
[[[i wi Ip (ar )ada = (Wi wi))u: 0mg ((w (x,) )xdr = (w, := 0, 0,
where 141 E [X Notice that the gradient in dimension
is non-zero, unless (wi (a:1-wj]i wi) 0. Assuming that f41 * 0, this implies that the gradient won't be zero unless wi = w|. On the other hand, the gradients for j = 2, d. Therefore, under the proper choice of learning rate 3, we get that the asymptotic solution of GD solution is w - lim w(() [wi], (o2(0), wy) wa(()) It is clear that [(w) 0, i.e., the population loss is zero, as noted already as our first observation in that section.
dimensions j = ..... d are all zero, so GD does not change the value of (ty()
Also note that we can easily attack this model by perturbing æ by Ar = (0, Az2, Ar3.... Aaa) T. The pointwise loss + A
1 2 (ww wi)= + ( w Az 11² |r w) + (w, Ar)
1
U(r + Ar; w)
With the choice of Az; sign((;(()) (for i 2, .,.... d) and Ar 0, an FGSM-like attack (Goodfellow et al., 2014) at the learned weight v leads to the pointwise loss of
1(x Az; 1 -2
wy (o,(0))
2 [w)l)
We comment that our choice of Az is not from the same distribution as the training data æ. This choice aligns with the hypotheses in Ding al. (2019a); Schmidt al. (2018) that adversarial examples come from a shifted data distribution; however, techniques such as feature adversaries (Sabour et al., 2015) focus on designing perturbations to be close to input distributions. We stress that the goal here is to illustrate the loss under this particular attack.
In order to get a better sense this loss, we compute its expected value w.r.t. the randomness of weight initialization. We have that (including the extra wm) (0) term too)
EW~N(0.oPLxa))
[
E
W
E
Σ WJ=1
([W,,,
where we used the independence of the r.v. W; and W, when * The expectation E [W, [2] is the variance o² of W., The r.v. [Wyl has folded normal distribution, and its expectation E [[, is
/ oo. Thus, we get that
[W]]]]
do d(d 1)
2 Loo
for d >>> 1. The expected population loss of the specified attack Ar at the asymptotic solution / is
Ex,w [[X + Am); m)) v d
The dependence of this loss dimension d is significant, showing that the learned model is quite vulnerable to attacks. We note that the conclusions would not change much with initial distributions other than the Normal distribution.
13

---

Under review as a conference paper at ICLR 2021
An effective solution is regularize the loss to encourage the weights of irrelevant dimensions going to zero. generic regularizer is to use the & f--norm of the weights, i.e., formulate the problem as ridge regression. In that case, the regularized population loss is
Cridge (w)
2 E [[X,, w) (XX, , w" }13
2 ||
One can that the solution of V Cridoo 0 20 (A) ff wf and wy (A) = 0 f j # 1.
Ri ML+A
w
j=1
wT(A)
1#1.
The use of this generic regularizer seems reasonable in this example, as it enforces the weights for dimensions 2 to to become zero. Its only drawback is that it leads to a biased estimate of wf. The bias, however, can be made small with a small choice for X. We can obtain a similar conclusion for the & regularizer (Lasso).
As such, one has to define a regularizer that is specially-designed for improving adversarial ro- bustness. Bishop (1995) showed the strong connection between training with random pertur- bation and Tikhonov Regularization, Inspired by this idea, we develop regularizer that mim- ics the adversary itself. Let assume that particular adversary attacks the model by adding Az = (0, sign (O)).,........... ..,, sign (O)) The population loss at the perturbed point is
1 Crohustified (w) 2 E1X + Aa; w)] = 2 B
r(u; w) + 8
|wyl
1=2
E² - |u2aalli
= [(w) + =EE [r(X; W22ddll 11 +
2
where ||122al1 = Σラ |myl'² This is the same objective as (1) reported in Section 2. Note that minimizing Lrobustiteed (v) is equivalent to minimizing the model at the point x' r + Ar. The
regularizer =IE (r(:; w)] [| 1 + = || incorporates the effect of adversary in exact form. This motivated the possibility of designing a regularized tailored to prevent attacks.
A.1 DERIVATION OF THE POPULATION LOSS UNDER ITS FIRST AND SECOND ORDER APPRXOIMATION
First, show that the FGSM direction is the maximizer of the loss when the perturbation is foo constrained. Based on the pointwise loss at x + Ar, we have
max 1(I ||
Az; w)
r(z; w)
max |AX
(w, Ar)
Se
We use the Ktuchy-Schward inequality to obtain
(w, Ax)
kw, Ar)
max |Ax|l
||l|| ||Ax|lac =1lw]11
|aX|I Se
which leads
argmax [(x + Ax;w) = sign(w).
|X
Next, we show that the first-order approximation of [[X + Ar; w)] obtains the first two terms in (1).
Note the gradient of the loss w.r.t. the input is
Val(r; = ((w, Ar (w", Ar))(w w²) r(2; w) w²)
A similar, but complicated result, would hold the adversary could also attack the first dimension.
14

---

Under review as a conference paper at ICLR 2021
and the Hessian w.r.t. the input is
(= (w w ww -w)
The first-order Taylor series approximation is
Lmhustifidd (w) Lia(w) - (1(;w)) + V Ar
[(w) + [x) - w²) Ar]
C(w) + E [r(X; Ar]
[(w) + =E [r(;; )
w2aa11
Note that w= Ar Tr because of our particular choice of Ar and w*. Here we obtain the first two terms in (1).
The second-order Taylor series approximation
Lnstustitied (w) (w) a ((X; w) + vx((;;)) Ar Ax (2)(z;w)Ar
=(ww) + cE r w)] |22e1|1 ||
+ Ar T (w - ww)(ww ==-)= Ax
E²
[(w) + = (r(:; |2
+ 2
which recovers the exact form in (1).
This completes the motivation of using second-order Taylor series approximation with our warm-up toy example.
15

---

Under review as a conference paper at ICLR 2021
B DERIVATIONS OF SECTION 4: SECOND-ORDER ADVERSARIAL REGULARIZER (SOAR)
B.I RELAXATION
Note the Boolean quadratic programming (BQP) problem in formulation (5) is NP-hard (Beasley, 1998; Lima & Grossmann, 2017). Even though there exist semi-definite programming (SDP) relax- ations, such approaches require the exact Hessian w.r.t. the input, which is computationally expensive to obtain for high-ding sigh-dimensional inputs. And even if we could compute the exact Hessian, SDP itself is computationally expensive approach, and not suitable be within the inner loop of DNN training. As such, we relax the lo constraint to an & constraint, which as we see, leads to a computationally efficient solution.
B.2 THE ISSUE RELATED TO THE LOOSENESS OF THE BOUND IN EQ (7)
In the ICLR rebuttal phase, the reviewer pointed out that, from the perspective of the volume ratio between the two Ep balls, replacing || S with |نll2 I dds can be problematic since the volume of {8 ||8|| S }} is 2d whereas the volume of {ō ||ó||2 S V(dE} is r(i+4/2) Their ratio goes to 0 as the dimension increases. The implication is that the search space for the foo maximizer is infinitesimal compared to the one for the l maximizer, leading to a loose upper-bound bound.
4/2 dd/3gd
As preliminary study on the tightness of the bound, we evaluated the two slides of (7) by approxi- mating the maximum using PGD attacks. In particular, we approximate max|| 8(x + 6) using ((x + d where doe is generated using 20-iteration &ox -PG with = 25 Similarly, we approxi- mate max [|]]|s | vde f(x + 5) using {(x + 62) where 8, is generated using 100-iteration PgDPGD with 1.74. The reason for this particular configuration of attack parameter is match the used
during our previous evaluations.
Table 4: Comparing ((r + )0)) and l(x + 62). We approximate de and & using the PGD attacks with their corresponding tp norm.
Checkpoints
f(x + 5,0)
E(x + 82)
Beginning of SOAR
76.03
112.10
End of SOAR
7.0
20.15
From this preliminary study, observe that there is indeed a gap between the approximated LHS and RHS of (7), and thus, we think is a valuable future research direction to explore other possibilities that allow us use second-order approximation study the worst-case loss subject to an constrained perturbation.
B.3 UNIFIED OBJECTIVE
We could maximize each term inside (8) separately and upper bound the max by max 14l, V max ((())) dd [V[(x)) d (V2(()) where Fmax (z²((x)) is the largest singular value of the Hessian matrix, V²((x). Even though the norm of the gradient and the singular value of the Hessian have an intuitive appeal, separately optimizing these terms might lead to looser upper bound than necessary. The reason is that the
maximizer of the first two terms are argmax /[(<>) T 5|
vi(z) and the direction corresponding (v)))
to the largest singular value of (()))) In general, these two directions
aligned.
16

---

Under review as a conference paper at ICLR 2021
B.4 PROOF OF PROPOSITION
Proof. By the inclusion of the boo-ball of radius e within the E2-ball of radius de and the definition
of H in (6), have
max (z)d(1)) -
max 11ó1g, vd
Ezau (z)
3 Se
6
[2((x)
v((x)
8
max (x)) 1
+
2
1
V((x) T
[
T
5
8
((x) + max
H
1
1
((x) +
max
8 H8'
, Svde244
2
It remains to upper bound |a Se 6 H8 with E' de I. We use the Cauchy-Schwar inequality to obtain
max 8 H8' I ||
max 16
H8
max
118112 HH 112 = max 22
H8 112 = E2 H
||
Se
where the last equality is obtained using properties of the fz-induced matrix norm (this is the spectral norm). Since computing [|H]| would again require the exact input Hessian, and we would like to avoid it, we further upper bound the spectral norm by the Frobenius norm
H
max HH) < |H] F.
The Frobenius norm itself satisfies
Tr(HT H) [2
(14)
where NO Therefore, estimate [ by sampling random vectors and compute the sample average of |H
17

---

Under review as a conference paper at ICLR 2021
C SOAR ALGROITHM: A COMPLETE ILLUSTRATION
In Algorithm 1, we present the inner-loop operation of SOAR using a single data point. Here we summarize the full training procedure with SOAR in Algorithm Note that is presented if the optimizer is SGD, but we may use other optimizers as well.
Algorithm 2: Improving adversarial robustness via SOAR
Input :Taiinnng dataset. Learning rate B, training batch size b, number of iterations N, foo constraint of Finite difference step-size h.
Initialize network with pre-trained weight w;
for i E 10, N do
Get mini-batch B {(T1, 91) .... (T6, (b)} from the training set.
for j == 1, ..... m (in parallel) do
2'j - 2j where f- (71 ....... and 7 ~ 21- -1
2// T7 (( { + sign (Vaz {(())) } where IT is the projection operator.
Sample (0,I (0,Ia+t)x(++)))
Compute the SOAR regularizer R(z'si z, h) as (11).
Compute the pointwise objective: fSOAR (I,, Uz) U5)) 2,
end
11
Wi++1 f u4
3 x 6 Σ
2==
Vw, CSOAR-
12 end
18

---

Under review as a conference paper at ICLR 2021
D POTENTIAL CAUSES OF GRADIENT MASKING
Table 5: Average value of the highest probability output for all test set data, that is, 1
N EEW m P( where P(rn); represent the probability of class i given data Zn.
Method
Standard
Random
PGD1
Standard
98.11% - 0.07
97.81% + 0.06
96.83% + 0.11
ADV
70.33% + 0.45
70.04% + 0.45
65.46% + 0.40
SOAR
zero init
99.99% + 0.01
99.97% + 0.01
99.99% + 0.01
-random init
99.98% + 0.01
99.98% + 0.00
100.0% 0.00
-PGD1 init
97.71% + 0.10
97.63% + 0.08
97.94% 0.09
We summarize the average value of the highest probability output for test set data initialized with zero, random and PGD1 perturbations Table 5. We notice that training with SOAR using zero or random initialization leads to models with nearly 100% confidence their predictions. This is aligned with the analysis of SOAR for linear classifier (Section 4.1), which shows that the regularizer becomes ineffective the model outputs high confidence predictions. Indeed, results in Table 7 show that those models are vulnerable under black- box attacks.
Results in Table 5 suggest that highly confident predictions could be indication for gradient masking. We demonstrate this using the gradient-based PGD attack. Recall that we generate PGD attacks by first initializing the clean data In with randomly chosen / within the &oo ball of size followed by gradient ascent at In + 77. Suppose that the model makes predictions with 100% confidence on any given input. This leads to piece-wise loss surface that is either zero (correct predictions) or infinity (incorrect predictions). The gradient of this loss function is either zero or undefined, and thus making gradient ascent ineffective. Therefore, white-box gradient- gradient-based attacks unable to find adversarial examples.
19

---

Under review as a conference paper at ICLR 2021
E SUPPLEMENTARY EXPERIMENTS
E. DISCUSSION ON THE REPRODUCIBILITY OF CURE AND LLR
We not able reproduce results of two closely related works, CURE (Moosavi-Dezfooli et al., 2019) and LLR (Qin et al., 2019). For CURE, we found the open-source implementation", but were not able reproduce their reported results using their implmentation. We were not able to reproduce the results of CURE with our own implementation either. For LLR, Yang et al. (2020) were not able to reproduce the results, they also provided open-source implementation" Regardless, compare SOAR to the reported result by CURE and LLR in Table 6:
Table 6: Comparison of SOAR, CURE and LLR on CIFAR-10 against &oo bounded adversarial perturbations 8/255).
Method
Standard Accuracy
PGD20
Architecture
SOAR
87.95%
56.06%
ResNet-10
CURE
81.20%
36.30%
ResNet-18
CURE
83.10%
41.40%
WideResNet
LLR
86.83%
54.24%
WideResNet
E.2 TRAINING AND EVALUATION SETUP
CIFAR-10: Training data augmented with random crops and horizontal flips.
We used an open-s ResNet- 10 implementation'. More specifically, we initialize the model with ResNet (Basica) paasicllock, 1,,,, 11). Note that we remove the BatchNorm layers in the ResNet- 10 architecture, and we discuss this further in Appendix E.7.
ResNet:
WideResNet: We used the implementation' of WideResNet-34-10 model found in public repository maintained by the authors of TRADES (Zhang et al., 2019).
Standard training on ResNet and WideResNet: Both are trained for a total of 200 epochs, with an initial learning rate of 0. The learning rate decays by an order of magnitude at epoch 100 and 150. We used minibatch size 128 for testing and training. We used SGD optimizer with momentum of 0.9 and weight decay of 2e-4.
The optimization setting is the same as the one used for standard training. Additionally, ensure that the final model has the highest adversarial robustness, we save the model at the end of every epoch, and the final evaluation is based on the with the highest PGD20 accuracy.
Adversarial training with PGD10 examples on ResNet:
SOAR on ResNet: SOAR refers to continuing the training of the Standard model on ResNet. trained for a total of 200 epochs with an initial learning rate of 0,004 and decay by order of magnitude at epoch 100. We used SGD optimizer with momentum of 0.9 and a weight decay of 2e-4. We FD step-size h 0.01 for the regularizer. Additionally, we apply clipping of 10 on the regularizer, and we discuss this clipping operation in Appendix E.7.
We used the same optimization setup as the ones in their respective public repository". We briefly summarize it here. The model is trained for a total of 120 epochs, with an initial learning rate of 0. The learning rate decays by an order of magnitude at epoch 75, 90, 100. We used SGD optimizer with momentum of 0.9 and weight decay of 2.-4. We performed a hyperparameter sweep on the strength of the regularization term I and selected one that resulted in the best performance against PGD20 attacks. A complete result is reported in Appendix 12.
MART and TRADES on ResNet:
htttta////gttaa..cc/a/aaaeeeee// rowpsssseecserruuuseesseeess 'https: httpt////sttttcc.oommmmmett/ee/s
http:////ggghbbb.ccoo//o/o////// far
20

---

Under review as a conference paper at ICLR 2021
MMA ResNet: We used the same optimization setup the one in its public repository'. We briefly summarize it here. The model is trained for a total of 50000 iterations, with an initial learning rate of 0.3. The learning rate changes to 0.09 at the 20000 iteration, 0.03 at the 30000 iteration and lastly 0.009 at the 40000 iteration. We used SGD optimizer with momentum of 0.9 and weight decay of 2e-4. We performed hyperparameter sweep on the margin term and selected the one that resulted in the best performance against PGD20 attacks. A complete result is reported in Appendix E.12.
ADV, TRADES, MART and MMA on NedeResNet: We the pretrained checkpoint provided in their respective repositories. Note that we use the pretrained checkpoint for PGDIO adversarially trained WideResNet Madry's CIFARIO Challenge".
Evaluations: For FGSM and PGD attacks, we use the implementation in AdverTorch (Ding et al., 2019c). For SimBA (Guo al., 2019), we use the authors' open-source implementation'.
E.3 ADVERSARIAL ROBUSTNESS OF THE MODEL TRAINED USING SOAR WITH DIFFERENT INITIALIZATIONS
Table 7: Performance of SOAR with different initializations on CIFAR-10 against white-box and
transfer-based blxck-box& foe bounded adversarial perturbations (E 8/255).
Method
Standard accuracy
White-box PGD20
Black-box PGD20
SOAR
zero init
91. 91.73%
89.24%
2.86%
rand init
91.70%
90.82%
9.16%
-PGD1 init
87.95%
56.06%
79.25%
We report the adversarial robustness of the model trained using SOAR with different initialization techniques in Table 7. The second column shows the accuracy against white-box PGD20 adversaries. The third column shows the accuracy against black-box PGD20 adversaries transferred from an independently initialized and standard-traine -rained ResNet- 10 model.
Note that despite the high adversarial accuracy against white-box PGD attacks, models trained using SOAR with zero and random initialization perform poorly against transferred attacks. This suggests the presence of gradient masking when using SOAR with zero and random initializations. Evidently, SOAR with PGD1 initialization alleviates the gradient masking problem.
E.4 COMPARING THE VALUES OF THE SOAR REGULARIZED LOSS COMPUTED USING DIFFERENT NUMBERS OF RANDOMLY SAMPLED
Table 8: Values of the regularized loss computed using different numbers of at the beginning and the end of SOAR regularization.
Checkpoints
72 1
π
10 100
Beginning of SOAR 10.58
10.58 1.56
10.58
End of SOAR
1.57
1.56
Suppose slightly modify Eq (13) by (soAR( v. = ((xx))) + - R(1: h) to incor-
porate the effect of using multiple randomly sampled vectors 2(2) in computing the SOAR regularized loss. Notice that the current implementation is equivalent to using 1. We observed the model two checkpoints, at the beginning and the end of SOAR regularization, the value of the regularized loss remains unchanged as we increase n from 1 to 100.
hhttps://github.comm//bb
leenge
attack
21

---

Under review as a conference paper at ICLR 2021
E.5 ROBUSTNESS UNNDER & ATTACKS ON CIFAR-10 10
We evaluate SOAR and two of the baseline methods, ADV and TRADES, against l2 white-box and black-box attacks on CIFAR-10 10 in Table 9. No & results were reported by MART and we not able to reproduce the &g results using the implementation by MMA, thus those two methods are not included our evaluation.
In Section 4, we show that the loc formulation of SOAR with | = e is equivalent to the & formulation of SOAR with ||ól|| =vd. In other words, models trained with SOAR to be robust & should also obtain improved robustness against & attacks with 8 255 322 32 #3 = 1.74. In our evaluation, all & adversaries used during ADV and TRADES are generated with 10-step PGD (s 1.74) and step size of 0.44. Note that the goal here is to show the improved robustness of SOAR against & attacks other than being SOTA, thus the optimization procedures are the same the ones used in the lo evaluation.
against Loo attacks with
253
We observe that training with SOAR improves the robustness of the model against &g attacks. Instead of fixed lg norm, we demonstrate the improved robustness using increasing range of s. For all attacks, we use 100 iterations of PGD and a step size of 2.3e In Table 9, we find that training with SOAR leads to significant increase in robustness against white-box and black-box lg adversaries. As & increases, SOAR model remain robust against white-box le attacks (e = 1), while other methods falls off. The last column of Table 9 shows the robustness against transferred f2 attacks (s = 1.74). The source model is a ResNet10 network trained separately from the defence models on the unperturbed training set. We observe that SOAR achieves the second highest robustness compared to baseline methods against transferred &g attacks. This result empirically verifies our previous claim that Eg and Lo formulation of SOAR only differs by a factor of Vd. Moreover, it aligns with findings by Simon- Gabriel et al. (2019), that empirically showed adversarial robustness through regularization gains robustness against more than one norm-ball attack at the same time.
100
Table 9: fg robustness of the adversarially trained model (under lS formulations) at different epsilon values. 100-step PGD used for all attacks. Accuracy (%) against P2G attacks.
Method
60
120
255 255
1.74
Transfer
255
255
ResNet
ADV
68. 13%
62.03%
47.53%
28.09%
70.86%
TRADES
68.59%
62.39%
45.42%
25.67%
69.02%
SOAR
75.39%
66.81%
60.90%
56.89%
69.52%
Table 10: Performance of the ResNet models on CIFAR- against the four &g- bo-boddd attacks used as an ensemble in AutoAttack (e 1.74, 1.0, 0.5).
Method
Untargeted APGD-CE
Targeted APGD-DIR
Targeted FAB
Square Attack
ADV
24.61 44.44/50..00000 60.62
22.00 42.99 (5/.788. 78
02.57/43.31 58.89
45. 45.69|57.84
TRADES
20.22 43.50 61.51
15.05 37.997
14.44/36.37 59. 19
(3.60)57.988668 21.34|49.64/71..)
SOAR
51.37 56.51 64.18
16.5
1.05)17.70|53.42
E.6
ADDITIONAL EVALUATION ON SVHN DATASET
We use the same ResNet- 10 architecture the one for CIFAR-10 evaluation. Training data is augmented with random crops and horizontal flips. For Standard training, we use the same opti- mization procedure the used for CIFAR-10. For SOAR and TRADES, we use the exact same hyperparameter for the regularizer. For SOAR, we use carly-stopping at epoch 130 to prevent catastrophic over-fitting Besides, the optimization schedule is identical for SOAR and TRADES as the ones used for -1.
We emphasize again that the goal of evaluting using SVHN is to demonstrate the improved robustness sweep. The
with SOAR on a different dataset, thus we did not perform an additional optimization procedures the same the ones used in the CIFAR-10 evaluation.
22

---

Under review as a conference paper at ICLR 2021
Table 11: Performance on SVHN against So bounded white-box attacks (e = 8/255)..
Method
Standard
FGSM
PGD20
PGDI00
PGD200
PGD1000
PGD20-50
Standard
94.93%
34.41%
2.71%
2.27%
2.20%
2.09%
2.15%
ADV 91.8%
61.0%
43.2%
42.1%
TRADES 84.48%
58.01%
48.90%
48.15%
48.11%
48.05%
48.01%
SOAR
91.23%
72.70%
58.99%
56.80%
56.56%
56.38%
56.70%
Table 12: Performance on SVHN against &0 bounded black-box attacks (s = 8/255).
Method
SimBA
PGD20
PGD1000
TRADES
19.13%
75.35%
75.27%
SOAR 64.20%
75.42%
75.81%
For PGDIO adversarial training, we observe that ResNet-10 is not able to learn anything meaningful. Specifically, when trained with PGDIO examples, ResNet- 10 does not perform better than randomly- initialized network in both standard and adversarial accuracy. Cai et al. (2018) made similar observation on ResNet-50, where training accuracy is not improving over long period adversarial training with PGDI0. 10. They further investigated models with different capacities and found that ResNet-50 might not be sufficiently deep for PGDIO adversarial training on SVHN. Wang & Zhang (2019) reported PGD10 adversarial training result SVHN with WideResNet, which we include in Table 11.
For MART, we not able to translate their CIFAR-10 results SVHN. We performed the same hyperparameter sweep as the one in Table 18, as well as different optimization settings, but none resulted in meaningful model. It is likely that the potential cause is the small capacity of ResNet- 10. For MMA, the implementation included in its public repository is very specific to the CIFAR-10 dataset, so we did not include it in the comparison.
Overall, we observe a similar performance on SVHN vs. on CIFAR-IO. Compared to the result in Table 1, we observe slight increase standard accuracy and robust accuracy for both SOAR and TRADES. In particular, the standard accuracy increases by 8.87% and 3.28 3.28%, and the PGD20 accuracy increases by 3.52% and 2.93% for TRADES and SOAR respectively. More notably, we observe SVHN that SOAR regularized model gains robustness without significantly sacrificing its standard accuracy.
Table 12 compares the performance of SOAR to TRADES on SimBa and on transferred fox attacks. The evaluation setting for transferred attacks identical to the one used for CIFAR-IO, where we use an undefended independently trained ResNet- 10 the source model. Despite a smaller gap on the accuracy against transferred attacks, we see that SOAR regularized model yields a significant higher accuracy against the stronger SimBA attacks.
Note that we did not perform any extensive hyperparameter sweep on SVHN, and we simply took what worked CIFAR-10. We stress that the goal is to demonstrate the effectiveness of SOAR, and its performance relative to other baseline methods.
Next, evaluate SOAR and TRADES under fg bounded white-box and black-box attacks. All &z PGD adversaries are generated using the same method as the one in the evaluation for CIFAR-10. Also, we do not include ADV due to the same result discussed above. Our results show that training with SOAR significantly improves the robustness against &s PGD white-box attacks compared to TRADES, For transferred attacks, TRADES and SOAR performs similarly.
E.7 CHALLENGES
Batch Normalization: We observe that networks with BatchNorm layers do not benefit from SOAR in adversarial robustness. Specifically, we performed an extensive hyper-parameter search for SOAR on networks with NatchNorm layers, and not able to achieve meaningful improvement
23

---

Under review as a conference paper at ICLR 2021
Table 13: Performance on SVHN against lg bounded white-box and black-box attacks. 100-step PGD is used for all attacks.
Method
Standard
E
60 255
120 255
2555 255
8 1.739
Transfer
Standard
94.93%
53.46%
30.04%
18.20%
14.60%
15.52%
TRADES 91.94%
55.17%
47.47%
32.34%
17.52%
50.22%
SOAR
91.23%
82.35%
69.65%
56.70%
48.38%
49.13%
in adversarial robustness. A related work by Galloway et al. (2019) focuses on the connection between BatchNorm and adversarial robustness. In particular, their results show that on VGG-based architecture (Simonyan & Zisserman, 2014), there is a significant gap in adversarial robustness between networks with and without BatchNorm layers under standard training. Needless to say, the interaction between SOAR and BatchNorm requires further investigations, and we consider this as an important future direction. As such, we use small-capacity ResNet (ResNet-10) and modified it by removing its BatchNorm layers. Specifically, we removed BatchNorm layers from all models used in the baseline experiments with ResNet. Note that BatchNorm layers makes the training process less sensitive to hyperparameters (loffe & Szegedy, 2015), and removing them makes difficult to train a very deep network such as WideResNet. As such, we did not perform SOAR on WideResNet.
our experiment,
Starting from pretrained model: We notice that it is difficult to train with SOAR on a newly- initialized model. Note that it is a common technique to perform fine-tuning on pretrained model for a specific task. In CURE, regularization is performed after a model is first trained with a cross-entropy loss to reach a high accuracy on clean data. They call the process adversarial fine-tuning Cai et al. (2018); Sitawarin et al. (2020) study the connection between curriculum learning (Bengio et al., 2009) and training using adversarial examples with increasing difficulties. Our idea is similar. The model is first optimized for an easier task (standard training), and then regularized for a related, but more diffieult task (improving adversarial robustness). Since the model has been trained to minimize its standard loss, the loss gradient can be very small compared to the regularizer gradient, and thus we apply a clipping of 10 on the regularizer.
Catastrophic Overfitting:
We observe that when the model achieves a high adversarial accuracy and continues training for a long period of time, both the standard and adversarial accuracy drop significantly. A similar phenomenon was observed in (Cai et al., 2018; Wong et al., 2019a), which they refer to as catastrophic forgetring and catastrophic over-fitting respectively. Wong et al. (2019a) use early-stopping as simple solution. We observe that with large learning rate, the model reaches a high adversarial accuracy faster and catastrophic over-fitting happens sooner. As such, our solution is to fix the number of epochs to 200 and then carefully sweep over various learning rates to make sure that catastrophic over- over-fitting do not happen.
We emphasize that our primary goal is to propose regu- larization as an alternative approach to improving adversarial robustness. We discussed techniques towards an efficient implementation, however, there is still potential for a faster implementation. In our current implementation, a single epoch with WideResNet takes: 19 mins on PGD10 adversarial training, 26.5 mins on SOAR, 29 mins on MART, and 39.6 mins on TRADES. We observe that despite being a faster method than MART and TRADES, SOAR is still quite slow compared to PGDI0 adversarial training. We characterize the computation complexity as a function of the number of forward and backward passes required for single mini- batch. Standard training: forward pass and backward pass; Adversarial training with k-step PGD: k+1 forward passes and k+1 backward passes; FOAR: 1 forward pass and 2 backward passes; SOAR: 3 forward passes and 4 backward passes
Discussion on Computation Complexity:
E.8 DIFFERENTIABILITY OF RELU AND ITS EFFECT ON SOAR
The SOAR regularizer is derived based on the second-order Taylor approximation of the loss which requires the loss to be wiice-differentiable. Although ReLU is not differentiable O, the probability of its input being at exactly 0 is very small. That is also why we can train ReLU networks through backpropagation. This is true for the Hessian too. In addition, notice that from computation
24

---

Under review as a conference paper at ICLR 2021
viewpoint, never need to compute the exact Hessian approximate it through first-order approximation.
E.9 POTENTIAL ROBUSTNESS GAIN WITH INCREASING CAPACITIES
Empirical studies in Madry et al. (2018) and Wang et al. (2020) reveal that their approaches ben- efit from increasing model capacity to achieve higher adversarial robustness. We have similar observation with SOAR.
Table 14: Performance on CIFAR-10 10 against foo bounded white-box attacks 8/255).
Model
Standard
PGD20
PGD100
PGD200
CNN6
81.73%
32.83%
31.20%
31.15%
CNNS
83.65%
47.30%
46.07%
45.83%
ResNet-10
87.95 87.95%
56.06%
55.00%
54.94%
Table 14 compares the performance SOAR against Lo bounded white-box attacks on networks with different capacities. CNN6(CNN8) refers to simple +-layer(8-layer) convolutional network, and ResNet- 10 is the network we use in Section Evidently, as network capacity increases, we observe improvements in both standard accuracy and adversarial accuracy. As such, expect similar gain in performance with larger capacity networks such as WideResNet.
E.10 EXPERIMENT RESULTS ON RESNET10 TABLE AND TABLE 2 WITH STANDARD DEVIAITIONS
All results ResNet10 are obtained by averaging over 3 independently initialized and trained models. Here, we report the standard deviation of the results in Table and Table 2. Notice we omit results on PGD100 and PGD200 due to space constraint.
Table 15: Performance on CIFAR-10 against Eo bounded white-box attacks (c = 8/255). (Table 1 with standard deviations)
Method
Standard
FGSM
PGD20
PGD1000
PGD20-50
Standard
92.54%(0.07)
(0.59%(
0.14%(0.03)
00.02(0.02)
0.10%(0.002
ADV
80.64%(0.40) (0.96%(
(2.86%(0.18))
42.17%((.15)
42.55%(0.15)
ResNet
TRADES
(0.11%().11) 50.06%((.09)
15.38%(0.15)
(5.16%(0.17)
45.21%(0.16)
MART
75.88%(1.12) 52.55%((.28)
46.60 46.60%()
46.21%((.477)
46.40%(0.45
MMA
(0.15) (7.08%(0037
37.2 (7.26%).338
36.64%(0.41)
(0.85%(
FOAR
(0.24%) 32.28%(0.11) 36.96%(..17)
31.89%(0.10)
32.08%(0.07)
SOAR
87.95%(0.08)
67.15%(..32)
(6.06%(0.155
51.69%(0.17)
(1.20%(0.16)
Table 16: Performance on CIFAR-10 against fx bounded black-box attacks (E = 8/255). (Table with standard deviations)
Method
SimBA
PGD20-R
PGD20-W
PGD1000-R
PGD1000-W
ADV
17.27%(1.01)
77.19%(0.30)
79.48%(0.38)
77.32%().31)
(9.35%(..37)
TRADES
(2.28%((.19) (0.52%()
74.39%(0.28)
72.24%((.14)
(0.27)(0.21)
ResNet
MART
(8.57%() 72.99% (0.90)
74.91%(1.
72.99%(
75.04%(1.12)
MMA
43.53%((.225 78.70%(0.09)
80.39%
78. 78.79%(0.09)
(1.35%(0.17)
FOAR
35.97%(0.22 63.56%(0.30)
65.20%(0.21)
63.60%().33)
63.27%(0.32)
SOAR
68.57%(..95)
(0.25%().45)
(6.35%(..04)
(0.49)(
86.47%(0..1))
25

---

sign of those non-zero gradient elements to generate Under review as a conference paper at ICLR 2021
E.11 ADDITIONAL EXPERIMENTS ON GRADIENT MASKING
To verify that SOAR improves robustness of the model without gradient masking, we include the following experiments empirically support claim.
First, from the result in Appendix E.3, we conclude that SOAR with zero initilaization results in gradient masking. This is shown by the high accuracy 289.24%, close standard accuracy) under white-box PGD attacks and low accuracy (2.86%) under black-box transferred attacks, Next, prior work has verified that adversarial training with PGD20 adversaries (ADV) results model without gradient masking et al. (2018). Therefore, let us use models trained using ADV and SOAR(zero-inis as examples of models with/without gradient masking respectively.
In the lo attack setting, PGD uses the sign of the gradient sign( V V=((=)) to generate perturbations. As such, one way to verify the strength of gradient elements in the gradient. A model with gradient masking is expected to have much less non-zero elements than one without.
measure the average number of none-zero
our experiment, the average non-zero element in gradient 3072 for ADV trained (no GM), 3069 for SOAR (PGDI-init) and 1043 for SOAR (zero-init, has GM). We observe that SOAR with PGD1-init has similar number of non-zero gradient elements compared to ADV, meaning PGD adversary meaningful perturbations.
In Section 5, the 20-iteration &o PGD adversaries generated with step-size of 256 and E 283 Suppose we E 1 instead of 255 and other parameters remain the same, that is, allow the maximum lo perturbation to reach the input range (0,, 1] and generate PGD20 attacks. We observe such attacks result in near black-and- boock-and-hhite white images on SOAR with PGD-1 init; it has 0% accuracy against such PGD20 attacks, similar to the 3.3% on ADV trained model. On the other hand, the robust accuracy for SOAR (zero-init) is 9.7%.
E. 12 HYPERPARAMETER SWEEP FOR TRADES, MART AND MMA ON RESNET
The following results show the hyperparameter sweep TRADES, MART and MMA respectively. We include the one with the highest PGD20 accuracy in Section 5.
Table 17: Hyperparameter sweep of TRADES on ResNet: evaluation based performance on CIFAR
10 against foo bounded adversarial perturbations (e 8/255).
B
Standard
FGSM
PGD20
PGD100
PGD200
PGD20-50
15
72.80%
48.85%
44.80%
44.63%
44.63%
44.67%
73.46%
49.18%
45.09%
44.97%
44.96%
44.98%
74.46%
49.39%
44.86%
44.74%
44.71%
44.75%
75.61%
50.06%
45.38%
45.19%
45.18%
45.24%
76.74%
50.05%
45.04%
44.80%
44.78%
44.87%
5
78.39%
50.22%
44.40%
44.11%
44.14%
44.26%
3
80.20%
49.81%
42.53%
42.12%
42.09%
42.23%
1
83.69%
45.61%
35.27%
34.62%
34.52%
34.87%
0.5
84.00%
46.16%
31.40%
33.48%
33.38%
33.75%
0.1
84.64%
41.12%
16.52%
14.80%
14.53%
14.83%
26

---

Under review as a conference paper at ICLR 2021
Table 18: Hyperparameter sweep of MART ResNet: evaluation based performance CIFAR-10
against &oc bounded adversarial perturbations (= 8/255).
Standard
FGSM
PGD20
PGD100
PGD200
PGD20-50
15
72. 78%
52.13%
46.41%
46.05%
45.98%
46.14%
73.03%
51.43%
46.29%
46.05%
45.99%
46.15%
45.
51.57%
46.17%
45.84%
45.83%
45.95%
75.88%
52.55%
46.60%
46.29%
46.25%
46.40%
75.77%
52.54%
16.31%
45.94%
45.92%
46.07%
5
76.84%
52.07%
46.00%
45.67%
45.59%
45.80%
78.25%
52.22%
45.32%
44.84%
44.82%
45.10%
80.30%
52.12%
44.10%
43.58%
43.52%
43.86%
0.5
80.79%
52.06%
43.95%
43.32%
43.24%
43.61%
0.1
80.82%
51.97%
43.39%
42.78%
42.76%
43.01%
Table 19: Hyperparameter sweep of MMA on ResNet: evaluation based performance CIFAR-10
against foo bounded adversarial perturbations (c 8/255).
Margin
Standard
FGSM
PGD20
PGD100
PGD200
PGD20-50
12
84.54%
46.11%
34.56%
33.89%
33.81%
34.12%
20
83.12%
47.48%
36.99%
36.41%
36.37%
36.59%
32
82.37%
17.08%
37.2 37.26%
36.71%
36.66%
36.85%
48
82.03%
46.77%
36.64%
36.02%
35.98%
36.22%
60
81.67%
46.79%
36.91%
36.35%
36.32%
36.51%
72
81.73%
46.40%
36.55%
36.01%
35.98%
36.21%
E. ADVERSARIAL ROBUSTNESS OF MODEL TRAINED USING FOAR DIFFERENT INITIALIZATIONS
FOAR achieves the best adversarial robustness using PGDI initialization, so we only present this variation of FOAR in Section 5.
Table 20: Performance of FOAR with different initializations on CIFAR-I0 against white-box and transfer-based black-box foo bounded adversarial perturbations (s 8 8/255).
Initialization
Standard
FGSM
PGD20
PGD100
PGD200
PGD1000
PGD20-50
zero
71.43%
33.39%
23.65%
22.83%
22.79%
23.23%
68.58%
rand 73.96%
33.62%
24.71%
23.96%
23.93%
24.37%
69.04%
PGD1
65.84%
36.96%
32.28%
31.87%
31.89%
32.08%
63.56%
27


Reviews:
[{"review_id": "l5aXHfGm8hn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose a second order approximation based training procedure to build robust networks against \\ell_inf and \\ell_2 attacks. \n\nThe paper is very well written and ideas are clearly stated. The core intuition rests on building an approach similar to adversarial training, but without the computational expense and empirically inexact inner maximization procedure. The inexactness of inner maximization in adversarial training is indeed a crucial issue, which the authors highlight.  \n\nThe authors first formulate a second order objective which uses the Hessian information of the loss function Eq. 6. Through Proposition 1 and the approximation in Eq. 10, they derive a bound on this regularizer which only depends on first order terms, hence making it computationally inexpensive (Algorithm 1 SOAR). To my knowledge, the derivations are clean.\n\nThe experiments are only done on ResNet and not on WideResNet, although authors have reported the performance of other robust algorithms on WideResNet. The show that robust accuracy of ResNet-10 with SOAR is better than all competing algorithms using ResNet-10 and comparable to that other algorithms when used on WideResNet. It will be interesting to see the gains of SOAR on WideResNet as well.\n\nAnother key observation from the experiments is that SOAR does not lead to as significant drops in standard accuracy (i.e. not under attack), which I do think is also another important contribution of this paper. \n\nI do have a small issue with authors portraying regularization and robust training to be two separate objectives. Regularization is not specific to generalization properties, so I don't see why they dedicate several paragraphs to show that the two are the same. \n\nTypos:\n\nl_FOAR Eq.4 - should be <= not =", "extra_scores": {}, "extra_text": {}}, {"review_id": "IoQ7arTXGQX", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposed a regularizer loss as an alternative to adversarial training to improve the robustness of neural networks against adversarial attacks. The new regularizer is derived from a second-order Tyler series expansion of the loss function in the model robustness optimization problem. Clear mathematical derivation and thoughtful empirical experimental results are provided. The proposed method outperformed baseline adversarial training methods with better or on part robustness and higher standard accuracy.\n\nPros:\n- The paper is really well-written and easy to understand. Both the intuition behind the method is conveyed clearly and the potential drawback of the method is discussed with solutions.\n- The performance of the proposed method is outstanding with a large margin compared with other baseline methods.The experiments are extensive and rigorous. \n- The idea is easy to implement.\n- There is a lot of valuable discussion and experiments presented in the supplemental materials. In fact, some might be better to be moved to the main text.\n\nCons:\n- As also discussed in the challenges in Appendix E6, an expected advance of this approach would be the training efficiency. However it is not well discussed in the main text. It would make the paper an even better one with some efficiency optimization. \n- I have some concerns about the sampling steps in the regularization loss. Only a single sample is drawn for both eta and z. How does this sample affect the stability of the training? Some empirical analysis on the intermediate values would be nice.\n- The author mentioned in the appendix that batch normalization layers are removed from the SOAR experiments. Do you also remove these layers from the baseline experiments? If not, how does the removal of BN layers impact the performance?\n\nOther detail comments:\n- I think the clipping of the regularization loss should also be mentioned and discussed in the main text.\n- Missing space in conclusion: l2 attacks\n\n---------------------------------------------\npost-rebuttal\nI would like to thank the reviewers for their efforts to improve the draft. Most of my concerns were resolved. However, I agree with other reviewers on the fact that the proposed method only present advantages in certain limited networks and scenarios.  To calibrate this weakness, I downgrade my score to 7. Overall, I think the paper explored an interesting and promising direction to improve network robustness using second-order regularization and solid progress was made. I will recommend accepting the paper for publication.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2qkrEUk9CA7", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduced a regularization scheme based on the second-order Taylor expansion of the loss objective to improve the robustness of the trained models.\n\nTheoretically, it used the simple linear model as an example to demonstrate 1) with a proper regularization term, the training algorithm can encode robustness without explicitly solving the inner maximization problem, i.e., generating adversarial examples. 2) using the second-order estimation at the origin may lead to gradient masking and using a perturbed input can tackle this problem.\n\nThe algorithm proposed is based on an upper bound of a quadratic.\n\nThe experiments show some results on CIFAR10 and SVHN in comparison with some baselines in both white-box and black-box settings.\n\nComments on the intuition and methodology:\n\n1. We know that most modern deep learning models are based on ReLU activations (or its variants like leaky ReLU), which is not second-order differentiable. In these cases, the Hessian matrix of the loss is ill defined. The author should discuss this.\n\n2. I cannot see the \"uniqueness\" or the advantage of regularization introduced in Equation (1). Actually, the quadratic form of (1) comes from the mean squared loss of $l$ instead of second-order Taylor expansion. In addition, the regularization term in (1) depends on $\\Delta x$. That's to say, using a different perturbation $\\Delta x$, the regularization term would be different. It is not necessarily in a quadratic form and can also drive the trainable parameter $w$ to the most robust one.\n\n3. The upper bound derived in Section 4 is not a strictly upper bound of the loss function, because it is based on a second-order approximation. The highlight of the word \"worst-case\" is a bit misleading. In addition, the author did not point out why we need a \"upper bound\" here? In adversarial training, we are actually optimizing a lower bound of the inner maximization problem. It is reasonable as long as this lower bound is tight, corresponding to a stronger attacker. I think the key point is not the upper bound of the approximation, but how close the objective function you are training on is to the inner maxima of Equation (2).\n\n4. In Equation (7), replacing $\\|\\delta\\|_\\infty \\leq \\epsilon$ with $\\|\\delta\\|_2 \\leq \\epsilon \\sqrt{d}$ is risky even if $\\delta$ is a dense vector. The volume of $\\\\{\\delta | \\|\\delta\\|_\\infty \\leq \\epsilon\\\\}$ is $2^d\\epsilon^d$, while the volume of $\\\\{\\delta | \\|\\delta\\|_2 \\leq \\epsilon \\sqrt{d}\\\\}$ is $\\frac{\\pi^{d / 2}}{\\Gamma(1 + d / 2)} d^{d / 2}\\epsilon^d$. The volume ratio of them goes quickly to zero as the dimension $d$ increases. As a result, the RHS of (7) is a much looser bound than LHS.\n\n5. The authors should discuss the complexity of Algorithm 1 and compare it with baselines.\n\n6. The authors should justify the claim \"In such a case, the Taylor series expansion, computed using the gradient and Hessian evaluated at x, becomes an inaccurate approximation to the loss, and hence its maximizer is not a good solution to the inner maximization problem.\" in the second last paragraph of page 5. A regularizer of small values does not mean it is not a good regularizer in this case.\n\n7. The sentence \"our approximation might not be very good whenever\nthe loss function is close to being flat at x\" in the end of page 5 is incorrect. The approximation error is approximately the third-order Taylor expansion term and does not connect to the flatness at $x$.\n\nComments on the experiments:\n\n1. The author should evaluate the robustness of the model by the state-of-the-art attack, such as AutoAttack introduced in https://github.com/fra31/auto-attack. \n\n2. The results on the baselines is not convincing. The accuracy is a bit worse than the one claimed in their papers. For example, regarding adversarial training, the authors should use early stopping introduced in https://arxiv.org/abs/2002.11569 as it typically overfits in the end.\n\nThe writing is easy to follow. As far as what I know, there is no key missing references.\n\nBased on the concerns listed above, my score is a reject unfortunately, but I look forward to the discussion with the authors, I will re-evaluate the paper after the rebuttal period.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a regularization approach based on the second-order Taylor expansion of the loss objective to improve robustness of the trained models against \ell_inf and \ell_2 attacks. It is interesting to explore the second order-based regularization approach for network robustness. However, as pointed out by the reviewer, a major drawback of this approach is that SOAR is broken under a stronger attack - AutoPGD-DLR.  In addition, the theoretical bound seems very loose in the \ell_inf case.  
Decision: Reject