Adversarial Training Generalizes Data-dependent Spectral Norm Regularization
Authors: Kevin Roth
Yannic Kilcher
Thomas Hofmann
Venue:  (2020)
Keywords: Adversarial Robustness
Adversarial Training
Spectral Norm Regularization

Abstract: We establish a theoretical link between adversarial training and operator norm regularization for deep neural networks. Specifically, we present a data-dependent variant of spectral norm regularization and prove that it is equivalent to adversarial training based on a specific $\ell_2$-norm constrained projected gradient ascent attack. This fundamental connection confirms the long-standing argument that a network's sensitivity to adversarial examples is tied to its spectral properties and hints at novel ways to robustify and defend against adversarial attacks. We provide extensive empirical evidence to support our theoretical results. 

Under review as a conference paper at ICLR 2020
ADVERSARIAL TRAINING GENERALIZES DATA- DEPENDENT SPECTRAL NORM REGULARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
We establish a theoretical link between adversarial training and operator norm regularization for deep neural networks. Specifically, we present a deeeedent variant of spectral norm regularization and prove that it is equivalent to adversarial training based on a specific f--norm constrained projected gradient ascent attack. This fundamental connection confirms the long-standing argument that a network's s sensitivity to adversarial examples is tied its spectral properties and hints at novel ways to robustify and defend against adversarial attacks. We provide extensive empirical evidence to support our theoretical results.
INTRODUCTION
Deep neural networks have been used with great success for perceptual tasks such image clas- sification (Simonyan & Zisserman 2014 LeCun et al.] 2015) or speech recognition (Hinton et al.) 2012). While they are known to be robust to random noise, it has been shown that the accuracy of deep nets dramatically deteriorates in the face of so-called adversarial examples (Biggio et al.) 2013 Szegedy et al.) 2013] Goodfellow al., 2014), i.c. small perturbations of the input signal, often imperceptible to humans, that are sufficient to induce large changes in the model output. This apparent vulnerability is worrisome as deep nets start to proliferate in the real-world, including in safety-critical deployments. Consequently, there has been surge in methods that find adversarial perturbations (Sabour et al. 22055] Papernot et al.) 2016 Kurakin et al., 2016 Moosavi Dezfooli et al., 2016 Moosavi-Dezfooli et al. 2017 Madry al. 2017 Athalye al. 2018))
The most direct strategy of robustification, called adversarial training, aims to harden machine learning model by immunizing it against an adversary that maliciously corrupts training examples before passing them to the model (Goodfellow et al., [2014] Kurakin et al. 2016;] Miyato et al. 2015 2017 Madry et al. 2017). A different strategy of defense is to detect whether the input has been perturbed by detecting characteristic regularities either in the adversarial perturbations themselves or in the network activations they induce (Grosse et al. 2017) Feinman et al. 2017 Xu al. 2017 Roth
Metzen et al. 2017 Carlini & Wagner 2017: Roth et al. 2019).
Despite practical advances in finding adversarial examples and defending against them, it is still an open question whether (i) adversarial examples are unavoidable, i.e. no robust model exists, cf. Fawzi et al. 2018: Gilmer et al., 2018), (ii) learning a robust model requires too much training data, cf. (Schmidt et al. 2018), (iii) learning a robust model from limited training data is possible but computationally intractable (Bubeck al. 2018), or (iv) we just have not found the right training algorithm yet. i.e. adversarial examples exist because of intrinsic flaws of the model or learning objective that can ultimately be overcome.
In this work, we investigate the origin of adversarial vulnerability in neural networks by focusing on the attack algorithms used to find adversarial examples. In particular, we make the following contributions:
We present data-dependent variant of spectral norm regularization that directly regularizes large singular values of neural network regions that are supported by the data, as opposed to existing methods that regularize a global, data-independent upper bound.

---

Under review as a conference paper at ICLR 2020
We establish a theoretical link between adversarial training and operator norm regulariza- tion for deep neural networks. Specifically, we prove that data-dependent spectral norm regularization is equivalent to adversarial training based specific &z f--nrrm constrained projected gradient ascent attack.
We conduct extensive empirical evaluations showing that (i) adversarial perturbations align with dominant singular vectors, (ii) adversarial training and data- data-dependent spectral norm regularization dampen the singular values, and (iii) both training methods give rise to models that are significantly more linear around data points than normally trained ones.
RELATED WORK
The idea that conservative measure of the sensitivity of network against adversarial examples can be obtained by computing the spectral norm of the individual weight layers appeared already in
the seminal work of SZegedy al. (2013). A number of works have since suggested to regularize
the spectral norm (Yoshida & Miyato| 2017|Hein Hein & Andriushchenko 2017; Isuzuku et al., [2018] 2018) and Lipschitz constant (Cisse al. 2017
Miyato et al. 2018 Bartlett et al. (017])Fammn et al.
Raghunathan et al. 2018) as means to improve model robustness against adversarial attacks. In
the same vein, training methods based on input gradient regularization have been proposed (Gu &
Rigazio 2014
Lyu et al.) 2015) Cisse et al. 2017))
The most direct and popular strategy of robustification, however, is to use adversarial examples as data augmentation during training (Goodfellow et al. 2014: Shaham et al. 2015 Kurakin et al.) 2016 Miyato et al., 2017: Madry et al., 2017). Adversarial training can be viewed as a variant
of (distributionally) robust optimization El Ghaoui & Lebret 1997; Xu et al., 2009 Bertsimas &
Copenhaver 2018; Namkoong & Duchi 2017;Sinha et al. 2017] Gao & Kleywegt 2016) where a machine learning model is trained to minimize the worst-case loss against adversary that shift the entire training data within an uncertainty set. Interestingly, for certain problems and uncertainty sets, such as for linear regression and induced matrix norm balls, robust optimization has been shown to be equivalent to regularization (EI Ghaoui & Lebret| 1997 Xu et al.) 20099 Bertsimas Copenhaver 2018 Bietti et al., 2018). Similar results on the equivalence of robustness and regularization have been obtained also for (kemeliized SVMs (Xu et al.. 2009).
More recently, related works have started to develop a learning theory for robust optimization, in- cluding Lipschitz-sensstivee generalization bounds (Neyshabur et al. (015) and spectrally spectrally-normalzzed normalized margin bounds for neural networks (Bartlett et al.) (017), particularly as bounds on the spectral norm or Lipschitz constant can easily be translated to bounds on the minimal perturbation required to fool a machine learning model.
We extend these lines of work by establishing a theoretical link between adversarial training and data-dependent spectral norm regularization. This fundamental connection confirms the long- standing argument that network's 's sensitivity to adversarial examples is tied to its spectral properties and opens the door for adversarially robust generalization bounds via spectral norm based ones.
3 BACKGROUND
3.1 GLOBAL SPECTRAL NORM REGULARIZATION
In this section we rederive spectral norm regularization à la Yoshida & Miyato (2017), while also setting up the notation for later. Let x and denote input-label pairs generated from a data distribution P. Let f X C Ra + Rdd denote the logits of a 8 p-parmeetrrized piecewise linear classifier, i.e. f()
bbb-))++b b-. where ef is the activation function, and W', be denote the layer-wise weight matrix and bias vector, collectively denoted by 0. Let us furthermore assume that each activation function is ReLU (the argument can easily be generalized to other piecewise lincar activations). In this case, the activations o² act as meputddependent diagonal matrices &e == diag(o$) ). where element in the diagonal of 1(x 0) is one if the corresponding pre-activation & == W ()) + be is positive and equal to zero otherwise.
NNoe that convolutional layers can he constructed as matrix multiplications by converting the convolution operator into Toeplitz matrix.
2

---

Under review as a conference paper at ICLR 2020
Following Raghu al. (2017), we call Øx ( ".x....,._.)) E {0,1) the "activation pattern", where m is the number of neurons in the network. For any activation pattern 6 E {0, 1 we can define the preimage (Ø) {x E I Øx Φ}. inducing a partitioning of the input space via RR" = Up (Φ). Note that some (σ) = 0. as not all combinations of activiations may be feasible. See Figure 1 in (Raghu et al. 2017) or Figure in (Novak et al. 2018 for an illustration of ReLU tesselations of the imput space.
We can linearize f within a neighborhood around x as follows
f(x Ax) ~ f(x) J (xx) (with equality if + Ax X(x)))
(1)
where J ((x) denotes the Jacobian of f x
J fxx) = W 1ph-1
W. TT. .
W
(2)
We have the following bound for ||Ax||2 +
[lf(x + Ax) f(x)lle
|| f(x) AxIl2
|| ((a)) Ax|12
S (J)()))
sup ||xx||=00
(3)
| /
|
||Ax|
where ((J(()) is the spectral norm (largest singular value) of the linear operator J f(x)- From a robustness perspective we want (J f(x) to be small in regions that are supported by the data.
Based on the decomposition in Equation 2 and the non-expansiveness of the activations, ( I 1 for every & E {1,..., L-1), Yoshida & Miyato (2077) suggest upper-bound the spectral norm of the Jacobian by the product of the spectral norms of the individual weight matrices
(J)) S 'WW' Vx
(4)
The layer-wise spectral norms o² == a (W*) can be computed iteratively using the power method Starting with random vector Vo, the power method iteratively computes
u /f/ af l2. uf < W'vk-1 v v/llVall. v - (W")TTf
(5)
The (final) singular value can be obtained via of = (uf) W
Yoshida & Miyato (2017) suggest to turn this upper-bound into global (data-independent) regularizer by learning the parameters 0 via the following penalized empirical risk minimization
min 0 + E (x.))~ [((,,
+ 2 E=1
aWW)22
(6)
where f(. denotes arbitrary classification loss. Note, since the parameter gradient of (W /2 u (ve] with of, u' and v being the dominant singular value and singular vectors of W (approximated via the power method), Yoshida & Miyato (2017) global spectral norm regularizer effectively adds a term Ao (v²)" for each layer E {I,..., L} to the parameter gradient of the loss function. In terms of computational complexity, because the global regularizer decouples from the empirical loss, the power-method iterations can be amortized across data-points and a single power method iteration per parameter update step usually suffices in practice (Yoshida & Miyato 2017).
3.2 GLOBAL VS. LOCAL REGULARIZATION
The advantage of global bounds is that they trivially generalize from the training to the test set. The problem however that they can be arbitrarily loose, e.g. penalizing the spectral norm over irrelevant regions of the ambient space. To illustrate this, consider the ideal robust classifier that is essentially piecewise constant on class-conditional regions, with sharp transitions between the classes. The global spectral norm will be heavily influenced by the sharp transition zones, whereas local data-dependent bound can adapt to regions where the classifier is approximately constant (Hein & Andriushchenko, 2017). We would therefore expect global regularizer to have the largest effect in the empty parts of the input space. A local regularizer, on the contrary, has its main effect around the data manifold.
3

---

Under review as a conference paper at ICLR 2020
4 ADVERSARIAL TRAINING GENERALIZES SPECTRAL NORM REGULARIZATION
4.1 DATA-DEPENDENT SPECTRAL NORM REGULARIZATION
We now show how to directly regularize the data-dependent spectral norm of the Jacobian J (x)) Under the assumption that the dominant singular value is non-degenerate" the problem of computing the largest singular value and the corresponding left and right singular vectors can efficiently be solved via the power method. Let vo be random vector approximation the dominant right singular vector of J f(x)- The power method iteratively computes
Ug üz/l|ūull, Un <- Vk-) W4. T T
W Vk-1 (forward pass)
Vis < vall, V f J 1x (x) up))
(backward pass)
The (final) singular value can be computed via a = _ f(x) Va- Note that the right singular vector Vi gives the direction in input space that corresponds to the steepest ascent of f(x) along ug-
We turn this into regularizer by learning the parameters 0 via the following sacobian-based spectral norm penalized empirical risk minimization
min 0 - E (x. f(v, + (u s(m)))
(8)
where u and v are the data-dependent singular vectors of J f(x), computed via Equation 7
By optimality / stationarity [ (u J v) v f(x; f(x) || f() and lincarization
cJ (x)) f(x + cv) f(x) (which holds with equality if x + E X(x))) we can regular- ize learning also via the following sum-of-squares based spectral norm regularizer
min
(x ((v. f(x)) [1)(x+ f(x)
where the data-dependnnt singular vector of J f(x) is computed via Equation 7 and Á = Ae Both variants can readily be implemented in modern deep learning frameworks. We found the numerically stable than the Jacobian
sun-of-squares based spectral norm regularizer to be based one, which why we used this variant in our experiments.
In terms of computational complexity, the deta-dependent regularizer is equally expensive PGA- based adversarial training, and both are a constant (number of power method iterations) times more expensive than the data-independent variant, plus an overhead that depends on the batch size, which is usually mitigated in modern frameworks by parallelizing computations across a batch of data.
4.2 POWER METHON FORMULATION OF ADVERSARIAL TRAINING
Adversarial training (Goodfellow et al. 2014 Kurakin et al. 2016 Madry et al. 2017) aims to improve the robustness of machine learning model by training it against an adversary that independently perturbs each training example subject to a proximity constraint, e.g. in fn-norm,
min + E
(xx.)) E(y.
f(x)) + À max Latv (U, f(x)) CBC
(10)
where (:dv(:)) denotes the loss function used to find adversarial perturbations (does not need to be the same as the classification loss -))
The adversarial example x* is typically computed iteratively, e.g. via norm constrained projected
gradient ascent (Madry et al. 2017 Kurakin et al. 2016) (the general f,-norm constrained case is similar)
V (y. f f(xx-1)) X (x)) X eadv (v, (x)=1))
Xo ~ (x)(
(11)
Due to numerical errors, we can safely assume that the dominant singular value is non-degenerate.
J V/l
4

---

Under review as a conference paper at ICLR 2020
where (x) is the projection operator into the norm ball (x) == fx* ||x* -x|2 S c).
a step-size size or weighting factor, trading off the previous iterate X;-1 with the current gradient direction V v (y, fxxx=1))//) (ooe-))lllVV (v, fxx-1))ll =: Vk. and y is the true predicted label. For targeted attacks the sign in front of a is flipped, so as to descend the loss function into the direction of the target label.
By the chain-rule, the computation of the gradient-step Vą can be decomposed into a logit-gradient and Jacobian vector product, while the projection into the f--nrrm ball (x) can be expressed as normalization (see Section A.2 in the Appendix). The ε2-norm constrained projected gradient ascent attack can thus equivalently be written in the following power method like form (the normalization of üg is optional and can be absorbed into the normalization of V%.)
u1 <- u
ig +- Vzladv (y, a=f
(forward pass)
< va/l/iall,,
Vx 4 J 1/ = (x([(x=-1) uus)) us))
(backward pass)
(12)
X5 < (8 + avvs)
(projection)
Note that the logit-gradient Veladv (y: ((xe=>>)- can be computed in single forward pass, by directly expressing terms of the arguments of the adversarial loss.
Comparing the update equations for projected gradient ascent based adversarial training with those of data-dependent spectral norm regularization, we can see that adversarial training generalizes spectral norm regularization in Iwo ways: (i) via the choice of the adversarial loss function and (ii) by iterating X₁ within the norm ball (which also introduces an additional parameter o)).
The adversarial loss function determines the direction UI of the directional derivative (x(f(xx-1) ug), cf. A.33 in the Appendix for an example using the softmax cross-entropy loss. The following theorem shows that adversarial training based on a specific f2-norm constrained projected gradient ascent attack is indeed equivalent to data-dependent spectral norm regularization.
Theorem 1. For e small enough such that (x) C ((xx) and in the limit a - 0o, E2-norm constrained projected gradient ascent based adversarial training with -an-or-squares loss on the logits of the clean and perturbed inpul fadw (f(x), f(xx)) = [1 - [(x")] 122 is equivalent to data-dependent spectral norm regularization.
The proof can be found Section A.2 in the Appendix.
The conditions on and a can be considered specifics of the iteration method. The condition that be small enough such that B3 (x) is contained in the ReI.U cell around x ensures that the Jacobian J f(x*) J (x) for all x B (x), while the condition that - means that in the update equation for X2 all the weight is placed on the current gradient direction Vy whereas no weight is put on the previous iterate X Note that the limit - oo is well-defined since it is inside the projection operation (the projection of X divides by a again).
Note that, in practice, the Theorem is applicable long the Jacobian of the network remains approximately constant in the uncertainty ball under consideration, in which case the correspondence between adversarial training and data-dependent spectral norm regularization holds approximately in a region much larger than ((xx). In Section 5 we verify this in the experimental setting.
EXPERIMENTAL RESULTS
5.1 DATASET, ARCHITECTURE & TRAINING METHODS
We trained Convolutional Neural Networks (CNNs) with ReLU activations and batch normalization on the CIFARIO data set (Krizhevsky & Hinton, 2009). We use 7-layer CNN as our default platform, since it has good test set accuracy at acceptable computational requirements (we used an estimated 2.5k GPU hours (Titan X) in total for all our experiments). We train each classifier with a number of different training methods: (i) 'Standard': standard empirical risk minimization with softmax cross- cooss-entropy loss, (ii) 'Adversarial': &g fg-norm constrained projected gradient ascent (PGA) based adversarial training with a softmax cross-entropy loss, (iii) 'global SNR global spectral norm regularization à la Yoshida & Miyato (2017). and (iv) 'd.-d. SNR': data detenddenndent spectral norm regularization.
5

---

Under review as a conference paper at ICLR 2020
Table 1: CIFARIO test accuracies and hyper-parameters for the CNN7 and training methods considered. The regularization constants were chosen such that the models achieve roughly the same test set accuracy on clean examples as the adversarially trained model does.
TRAINING METHOD
ACCURACY
HYPUR- PARAMEERS
STANDARD TRAINING
93.5%
ADVERSARIAL TRAINING
83.6%
1.75, 22/ITERS, ITERS
GLOBAL SPECTRAL NORM REG.
80.4%
A 3 1-100" ITE55=
DATA-DER SPECTRAL NORM REG.
84.6%
== 3-10--2---0 100 1.75, ITERS = 10
As default attack strategy adversarial loss ladv and 10 attack iterations. We verified that all our conclusions also hold for larger numbers of attack iterations, however, due computational constraints we limit the attack iterations to 10. The attack strength used for training was chosen to be the smallest value such that almost all adversarially perturbed imputs to the standard model are successfully misclassified, which is e 1.75 (indicated by a vertical dashed line in the Figures below). The regularization constants of the other training methods were then chosen in such a way that they roughly achieve the same test set accuracy on clean examples as the adversarially trained model does. Further details regarding the experimental setup can be found in Section [A.4 in the Appendix. Table I summarizes the test set accuracies and hy per-paaraeeters for the training methods we considered. Shaded areas in the plots below denote standard errors with respect to the number test set samples over which the experiment was repeated.
f2-norm constrained PGA white-box attack with cross- cross-entropy
5.2 SPECTRAL PROPERTIES
Effect of training method on singular value spectrum. We compute the singular value spectrum of the Jacobian J f(x) for networks f trained with different training methods and evaluated a number of different test set examples (200 except if stated otherwise). Since we are interested in computing the full singular value spectrum, and not just the dominant singular value and singular vectors as during training, the power method would be too impractical to use, as it gives us access to only one (the dominant) singular value-vector pair at a time. Instead, we first extract the Jacobian (which is per se defined as computational graph in modern deep learning frameworks) as an input-dim output-dim dimensional matrix and then use available matrix factorization routines to compute the full SVD of the extracted matrix. For each training method, the procedure is repeated for 200 randomly chosen clean and corresponding adversarially perturbed test set examples. Further details regarding the Jacobian extraction can be found in Section A.5 in the Appendix.
The results are shown in Figure I (left). We can see that, compared to the spectrum of the normally trained and global spectral norm regularized model, the spectrum of adversarially trained and data- dependent spectral norm regularized models is significantly damped after training. In fact, the data-dependent spectral norm regularizer seems dampen the singular values even slightly more effectively than adversarial training, while global spectral norm regularization has almost no effect compared to standard training.
Alignment of adversarial perturbations with singular vectors. We compute the cosine-similarity of adversarial perturbations with singular vectors v, of the Jacobian f(x)) extracted at number of test set examples, as function of the rank of the singular vectors returned by the SVD decomposition. For comparison we also show the cosine-similarity with the singular vectors of random network as well as the cosine-similarity with random perturbations.
The results are shown in Figure (right). (right). We can see that for all training methods (except the random network) adversarial perturbations are strongly aligned with the dominant singular vectors while the alignment decreases towards the bottom-ranked singular vectors. For the random network, the alignment is roughly constant with respect to rank. Interestingly, this strong alignment with dominant singular vectors also explains why input gradient regularization and fast gradient method (FGM) based adversarial training do not sufficiently protect against adversarial attacks, namely because the input gradient, resp. a single power method iteration, do not yield a sufficiently good approximation for the dominant singular vector in general.

---

Under review as a conference paper at ICLR 2020
Tanel Staneood
640-
Tra red: Standars Tertsttoggra PSA
Trannl Advanarta
red Meeono,, Pertoratisee HiA
Trandl cete
red: Pattstuleers
8.00
ned Rovdon Int, Petrrtationn PGA
Tra Surbled. telabulzzo farsum
4.00
Singalar
A.5
k10-
6.6
251
200
Rank
Bank
Figure 1: (Left) Singular value spectrum of the Jacobian f(x) for networks trained with different training methods. (Right) Cosine-similarity of adversarial perturbations with singular vectors v, of the Jacobian J f(x)- as a function of the rank r of the singular vector. For comparison we also show the posine-similarig with the singular vectors of random network well the alignment with random perturbations. Curves were aggregated over 200 samples from the test set.
5.3 LOCAL LINEARITY
Validity of linear approximation. In order to determine the size of the area where a locally linear approximation is valid, we measure the deviation from linearity of df-1(( + 2) as the distance ||2||2 to x is increased in random and adversarial directions, i.e. we measure 1|o]--1(x + 2) (x))(=
(x)+ (x-1(x)) 2)122 l| a function of the distance ||z||2, for random and adversarial perturbations z, aggregated over 200 data points x in the test set, with adversarial perturbations serving as proxy for the direction in which the linear approximation holds the least. The purpose of this experiment is to investigate how good the linear approximation for different training methods is, as an increasing number of activation boundaries are crossed with increasing perturbation radius. See Figure 1 in (Raghu et al. 2017) or Figure 3 in (Novak et al., 2018) for an illustration of activation boundary tesselations in the input space.
The results are shown in Figure 2 (left). We can see that adversarial training and data-dependent spectral norm regularization give rise to models that are considerably more linear than the clean trained one, both in random as well as adversarial directions. Compared to the normally trained model, the adversarially trained and spectral norm regularized ones remain flat in random directions for pertubations of considerable magnitude and even remain flat in the adversarial direction for perturbation magnitudes up the order of the used during adversarial training, while the deviation from linearity seems to increase roughly linearly with | thereafter. The global spectral norm regularized model behaves similar to the normally trained one.
Largest singular value over distance. Figure 2 (right) shows the largest singular value of the linear operator f(x+x) as the distance ||z||2 from x is increased, both along random and adversarial directions, for different training methods. We can see that the naturally trained network develops large dominant singular values around the data point during training, while the adversarially trained and data deta-deeendent spectral norm regularized models manage to keep the dominant singular value low in the vicinity of x.
5.4 ADVERSARIAL ROBUSTNESS
Adversarial classification accuracy. A plot of the classification accuracy on adversarially perturbed test examples, as a function of the perturbation strength is shown in Figure (left). We can see that the adversarial accuracy of the data-dependent spectral norm regularized model comparable to that of the adversarially trained model, while global spectral norm regularization does not seem to robustify the model against adversarial attacks. This is in line with our earlier observation that adversarial perturbations tend to align with dominant singular vectors and that adversarial training and detendeenndent spectral norm regularization dampen the singular values. Additional results against loc -PA attack provided in Section in the Appendix. The conclusions for this and the other experiments remain the same. This indicates that the main effect of adversarial training is captured by data-dependent spectral norm regularization.
7

---

Under review as a conference paper at ICLR 2020
Tra rel Sander,, Pertmmdisne Hit,
Trasee Startir,, Petortatin: Pii
Tra red: Sardad. Puttrfulirrs Random red (daeoas, pertration:: AGA
Traas Stwiirs, Petrrst Random
390
Tranze Advema "etattelan
Devizzion from Linearity
red. dits Porurtation: handen goou 3KK, perurtu ISA Tha redi goos 2H3, Parsccatios: Random
Trahne 5&8, Perturationn Sandem
Top Singular
Transe glital SMA tarturaation RSA
Trancc glatel 5AR teturbations Nandom
0.55
12.
19.0
37.
283
15.0
20.0
Distance from
Discance from
+ - (x) a function of the distance ||z||2 from for random and adversarial perturbations (Right) Largest singular value the linear operator J (x++z) as function of the magnitude ||z||2 of random and adversarial perturbations z. The dashed vertical line indicates the used during adversarial training. Curves were aggregated over 200 samples from the test set.
Figure 2: (Left) Deviation from linearity 1x
18
Stancord Traned Adveraaii
0.8
Tra red Adeetari,, Petornaiion FMA.
Traned Standard Tertatbatios: FSA
Transll cele-zup, 5
Singala Vieter Aligamment
Traned date dep. Shn Purtsbblion FSA
44
Trainol: glub 5M
glod naturbuccte PGA
0.20
Accuncy
66
6.5
0.00
19 e'
0..
12.9
Perurbation Magninude
Pertuchation Magnitade
Figure 3: (Left) Classification accuracy as a function of perturbation strength (Right) Alignment of adversarial perturbations with dominant singular vector of J f(x a function of perturbation magnitude e. The dashed vertical line indicates the used during adversarial training. Curves were aggregated over 2000 samples from the test set.
Alignment of adversarial perturbations with dominant singular vector. Figure the essine-similarity of adversarial perturbations of mangitude e with the dominant singular vector of J f(x). as a function of perturbation magnitude e. For comparison, we also include the alignment with random perturbations. For all training methods, the larger the perturbation magnitude c, the lesser the adversarial perturbation aligns with the dominant singular vector of J f(x), which is to be expected for simultaneously increasing deviation from linearity. The alignment similar for adversarially trained and data-dependent spectral norm regularized models and for both larger than that of global spectral norm regularized and naturally trained models.
3 (right) shows
CONCLUSION
We established a theoretical link between adversarial training and operator norm regularization for deep neural networks. Specifically, we presented data-dependent variant of spectral norm regularization that directly regularizes large singular values of a neural network in regions that are supported by the data and proved that it is equivalent to adversarial training based on specific f2-norm constrained projected gradient ascent attack. This fundamental connection confirms the long-standing argument that a network's sensitivity to adversarial examples is tied to its spectral properties and opens the door for adversarially robust generalization bounds via data-dependent spectral norm based ones. We also conducted extensive empirical evaluations showing that (i) adversarial perturbations align with dominant singular vectors, (ii) adversarial training and data- dependent spectral norm regularization dampen the singular values, and (iii) both training methods give rise models that are significantly more linear around data points than normally trained ones.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint 100.vv/802.0000 2018.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky, Spectrally-normalizeed margin bounds for neural networks. In Advances in Neural Information Processing Systems. pp. 6241-62222 6250, 2017.
Dimitris Bertsimas and Martin Copenhaver. Characterization of the equivalence of robustification and regularization in linear and matrix regression. European Journal of Operational Research, 270 (3):931-942, 2018.
Alberto Bietti, Grégoire Mialon, and Julien Mairal. On regularization and robustness of deep neural networks. arXiv preprint arXiv: 180000001 2018.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smdic,, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Sébastien Bubeck, Eric Price, and Ilya Razenshteyn. Adversarial examples from computational constraints. arXiv preprint arXiv: 2018
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop Artificial Intelligence and Security, pp. 3-14. ACM, 2017.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness adversarial examples. In International Conference on Machine Learning, pp. 854-863, 2017.
Laurent El Ghaoui and Hervé Lebret. Robust solutions to least-squares problems with uncertain data. SIAM Journal on matrix analysis and applications, 18(4): 1997.
Farzan Farnia, Jesse M Zhang, and David Tse. Generalizable adversarial training via spectral normalization. arXiv preprint arXiv: arXiv: 1,00455 2018.
Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. Adversarial vulnerability for any classifier. arXiv preprint arXiv:1802.0866 2018
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint arXiv: arXivv1703.000 1703.0001, 2017.
Rui Gao and Anton J Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. arXiv preprint 1rXiv:1604.02199 2016
Justin Gilmer, Luke Metz, Fartash Faghri, Samuel S Schoenholz, Maithra Raghu, Martin Wattenberg, and Ian Goodfellow. Adversarial spheres. arXiv preprint arXiv:1801.02774 2018.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6533 2014.
Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, and Patrick McDaniel. On the (statistical) detection of adversarial examples. arXiv preprint arXiv: 17..107222...... 2017.
Shixiang Gu and Luca Rigazio. Towards deep neural network architectures robust to adversarial examples. arXiv preprint 41X5v:1412.5068, 2014.
Matthias Hein and Maksym Andriushchenko. Formal guarantees on the robustness of a classifier against adversarial manipulation. In Advances in Neural Information Processing Systems, pp. 2266-2276, 2017.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6) 2012.

---

Under review as a conference paper at ICLR 2020
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv: 02533, 2016.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553))436, 2015.
Chunchuan Lyu, Kaizhu Huang, and Hai-Ning Liang. A unified gradient regularization family for adversarial examples. In 2015 IEEE International Conference on Data Mining, pp. 301-309. IEEE,
2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706.06083 2017.
Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. arXiv preprini 172iv:/702.04267, 2017.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii. Distributional smoothing with virtual adversarial training. arXiv preprint arXiv: 2015.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: reg- ularization method for supervised and semi-supervised learning, arXiv preprint arXiv:1704.03977 2017.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05953 2018.
Seyed Mohsen Moosavi Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), number EPFL-CONF-218057 2016.
Seyed-Mohsen Moosavi-Dezzoolli Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. Universal
adversarial perturbations. arXiv preprint, 2017.
Hongseok Namkoong and John C Duchi. Variance-based regularization with convex objectives. In Advances in Neural Information Processing Systems, pp. 2975-2984, 2017.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401, 2015.
Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization in neural networks: empirical study, arXiv preprint arXiv: 2018.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv: arXiv:1605.0m 1605.07277 2016.
Maithra Raghu, Ben Poole, Jon Kleinberg, Surya Ganguli, and Jascha Sohl Dickstein. On the expressive power of deep neural networks. In Proceedings of the 34th International Conference on Machine Learning-Vai Volume 70, pp. 2847-2854. JMLR. org. 2017.
Aditi Raghunathan, Jacob Steinhardt, and Percy Liang. Certified defenses against adversarial examples. arXiv preprint arXiv:1801.0934. 2018.
Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting 2019.
adversarial examples. arXiv preprint
Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J Fleet. Adversarial manipulation of deep representations. arXiv preprint arXiv: 15.1:5122..... 2015.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. arXiv preprint arXiv: 1804. /285, 2018
10

---

Under review as a conference paper at ICLR 2020
Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of neural nets through robust optimization. arXiv preprint arXiv:1511.05432, 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations (ICLR), 2014.
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifiable distributional robustness with principled adversarial training. arXiv preprint arXiv: 10571, 2017.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv:1312.6199, 2013.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks. In Advances in Neural Information Processing Systems, pp. 6541-6550, 2018.
Huan Xu, Constantine Caramanis, and Shie Mannor. Robustness and regularization of support vector machines. Journal of Machine Learning Research, 10(Jul): 158--1500, 2009.
Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv:1704.01/55, 2017.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning, arXiv preprint 1014,, 2017.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 ROBUST OPTIMIZATION AND REGULARIZATION FOR LINEAR REGRESSION
In this section, we recapitulate the basic ideas on the relation between robust optimization and regularization presented in Bertsimaa & Copenhaver (2018). Note that the notation deviates slightly from the main text: most importantly, the perturbations refer to perturbations of the entire training data X, is common in robust optimization. Consider linear regression with additive perturbations A the data matrix X
min max 9 (y (X + A)w), with loss g R: - R w DEU
(13)
where U denotes the uncertainty set. A general way to construct U is ball of bounded matrix norm perturbations 24 {A ||||| & X}. Of particular interest are induced matrix norms
[(Aw) h(w) f
| || := max
9: J:R" R. h R - - R
(14)
where is a semi-norm and h is norm. It is obvious that if fulfills the triangle inequality then one can upper bound
g (y (X++XXA)V A))) S g Xw) + +(())) sly Xw) Ah(w), AA EU,
(15)
by using (a) the triangle inequality and (b) the definition of the matrix nomm.
The question then is, under which circumstances both inequalities become equalities at the maximizing A It is straightforward to check (Bertsimas & Copenhaver 2018 Theorem that specifically may choose the rank matrix
A
rv g(r)
where r= Xw,
arg max {v TyTw) w). (h* is the dual norm).
vv)=
(16)
If g(r) then one can pick any u for which g form A Auv (such u has exist if g is not identically zero). This shows that, for robust linear regression with induced matrix norm uncertainty sets, robust optimization is equivalent regularization.
A.2 PROOF OF MAIN THEOREM
Theorem 1 For small enough such that (x) C. x(xx) and in the limit a - 30, by-norm constrained projected gradient ascent based adversarial training with a sum-of-squares loss on the logits of the clean and perturbed input (xx,
= = [x) [[][fxxxxxx"")) f(x")) is equivalent to
date-dependent spectral norm regularization.
To proof the theorem we need show that the update equations in 12 for f2-norm constrained projected gradient ascent based adversarial training with the sum-of-squares loss function and the above conditions on and reduce to the corresponding update equations for data-dependent spectral norm regularization in equation
We start with the following lemma.
Lemma 1. In the limit a -> 30, the projection X [Xg + avR) reduces to X X + CVK.
Proof. The projection x 11g3(x) (X + oVR) can be expressed as follows,
X₁ c(xx x)/ max(e, | x(|2) with X X + avk.
(17)
where the max(e, 1|8 x/12) ensures that if || x||2 then X Et.
12

---

Under review as a conference paper at ICLR 2020
Thus,
lim (x)g(x)(Xx) (Xh- + avv)
(18)
8x -x x
-lim x +
(19)
a-+00
max(c; |34 x(12)
XB-1 + QVA
lim
(20)
a-700 max(c, [x + aV2 - x1
a(Vi + 1 (xx=11 x))
= x t lim
(21)
a-000 max Vv (xx-11 - x)112)
a(vx + (xx-1 -x))
x 6 lim
(22)
0 allvv (xt-1 - xx)12
x x + v
(23)
cVR
(24)
where in the third-to-last line we used that the max will be attained at its second argument in the
limit - op since ||v + _ (xx-1 x)ll2 > 0 for vs * 0, and in the last line we used that Vą is a
normalized by construction, i.e. by how it is defined in the update equations in (2)
Proof of Theorem 1. With the lemma in place, it is now easy to show that for small enough, such that B (x) C X ((x), and in the limit a - 20, adversarial training reduces to data-dependent spectral norm regularization. Let show this step by step.
In the limit a + co, we have for the forward pass
ú Vr5il((()) 2113 11
(25)
f(xx-1) - f(x)
(26)
-((() x)
(27)
cJ Vk-
(28)
where the last equality holds by the lemma. For the backward pass, we have
Vk J J(xe-1) Us J J
(29)
since J(xx) f(x) for all xx (x).
Together, this gives
ug + u
us + f(x)
Va - v Vvlll,
Ve + J un
(30)
X - x
Observing that the first two update equations don't depend Xt: and absorbing the into the normalization of ug. we precisely end up with the data-dependent spectral norm regularization in Equation 7
Up < uz/llūull>
üg < J VA-
VR-1
(31)
V4 + V. /
Vx J Jix
We have thus proved that the update equations for the adversarial attack in Theorem 1 correspond exactly to those of data deta-depend spectral norm regularization.
It is also easy to see that the adversarial training objective in Equation (10) corresponds to that of data-dependent spectral norm regularization for the specific choice of adversarial loss function
1 [11((x) f(x*) >
min 0 >EE,E (xx))) C(v. f(x)) + À
max x (xE
(32)
13

---

Under review as a conference paper at ICLR 2020
which by the condition (x) C X(ox) and x" = x + is equivalent
d
min E (x,))~ ((y, f(x))
(J v)
(33)
2
where the data-dependent singular vector v is computed via Equation
L
A.3 EFFECT OF THE ADVERSARIAL. LOSS FUNCTION THE LOGIT-SPACE DIRECTION
The adversarial loss function determines the logit-space direction uı of the directional derivative in the power method like formulation of adversarial training Equation T2
Let us consider this for the softmax cross-entropy loss, defined as
exp(zy)
ladv log(),(z)) --y log Σ exp(zx))
8y(z)
(34)
Ed=1 exp(zk)
Untargeted 22-PGA on softmax cross-entropy loss: (forward pass)
ü < Valadv 2)x= 82 (x([(xx)) -ôy
(35)
Targeted f2-PGA on softmax cross-entropy loss: (forward pass)
U + V (Wadv, z) 2)lc=)(x))] Sinnar sx(f(x)))
(36)
Notice that the logit gradient can be computed in forward pass by analytically expressing it in terms of the arguments of the objective function (this why we call the ug update a forward pass).
Interestingly, for temperature-deeeeddet softmax cross-entropy loss, the logit-space direction becomes a "label-flip" vector in the low-temperature limit (high inverse temperature 3 + 00)
where the softmax (z) == exp(3zy)/(Ei exp(Bzx)) converges to the argmax: 8 s² 8
arg max(z), z). E.g. for targeted attacks [ug [ deysar diy(xx)) This implies that in the high B limit, itcrative PGA finds an input space perturbation Vą that corresponds to the steepest ascent of f along the "label flip" direction u 8-900
note canonical link functions. Interestingly, the gradient of the loss w.r.t. the logits of the classifier takes the form "prediction target" for both the sum-of-squares error as well as the softmax cross-entropy loss. This is in fact general result modelling the target variable with conditional distribution from the exponential family along with canonical link (activation) function. This means that in both cases, adversarial attacks try to find perturbations input space that induce logit perturbation that aligns with the difference between the current prediction and the attack target.
A.4 DATASET, ARCHITECTURE & TRAINING METHODS
We trained Convolutional Neural Networks (CNNs) with seven hidden layers and batch normalization on the CIFARIO data set Krizhevsky & Hinton (200)) The CIFAR10 dataset consists of 60k 32 x 32 colour images in 10 classes, with 6k images per class. It comes in pre-packaged train- test split, with 50k training images and 10k test images, and can readily be downloaded from https: http://wwwwcc....ooooooo http://oooottto.eeeee//////////e
htm.
We conduct our experiments on pre-trained standard convolutional neural network, employing 7 convolutional layers, augmented with BatchNorm, ReLU nonlincarities and MaxPooling. The network achieves 93.5% accuracy on a clean test set. Relevant links to download the pre-trained model can be found in our codebase.
We adopt the following standard preprocessing and data augmentation scheme: Each training image is zero-padded with four pixels on each side, randomly cropped produce new image with the original dimensions and horizontally flipped with probability one half. We also standardize each image have zero mean and unit variance when passing to the classifier.
14

---

Under review as a conference paper at ICLR 2020
The attack strength used for PGA was chosen to be the smallest value such that almost all adver- sarially perturbed inputs to the standard model are successfully misclassified, which is c = 1.75. The regularization constants of the other training methods were then chosen in such way that they roughly achieve the same test set accuracy on clean examples as the adversarially trained model does, Lee. we allow comparable drop in clean accuracy for regularized and adversarially trained models. When training the derived regularized models, we started from a pre-trained checkpoint and ran hyper-parameter search over number of epochs, learning rate and regularization constants. Table [ summarizes the test set accuracies and hyper- parameters for all the training methods we considered.
A.5 EXTRACTING JACOBIAN AS A MATRIX
Since we know that any neural network with its nonlinear activation function set to fixed values represents linear operator, which, locally, is good approximation to the neural network itself, we develop a method to fully extract and specify this linear operator in the neighborhood of any input datapoint x. We have found the naive way of determining each entry of the linear operator by consecutively computing changes to individual basis vectors to be numerically unstable and therefore have settled for more robust alternative:
In a first step, we run a set of randomly perturbed versions of x through the network (with fixed activation functions) and record their outputs at the particular layer that is of interest to (usually the logit layer). In a second step, we compute a linear regression on these input-output pairs to obtain a weight matrix W as well a bias vector b, thereby fully specifying the linear operator. The singular vectors and values of W can be obtained by performing an SVD.
A.6 FURTHER EXPERIMENTAL RESULTS FOR loo -PG
Tra Staddrt Pertuttations fanden
Traine: Adveraaal fanubalors FSA
Tra net Asvesaroo Pettraasite HGS
Devintton from Eieesity
Trahhe: Adveraral Nanurabors Randor
Tra rest Asvanane Puturssicre: Rardom
Trites des, 983, Annutatione
Tops Singraler Valuee
Tin rest dra-trp SME Penumatise HGs.
Trated: dess des. 381, becuraatona fundom
Tru rest desedap, hutartarre fandon
20-
S1R Pettrationn kantom
09
Distance from
Distanoe From
(x) + J (x) 2) as a function of the distance ||z||2 from x for random and -PGA adversarial perturbations z. (Right) Largest singular value of the linear operator J f(x+2) PPA adversarial perturbations z. The dashed vertical line indicates the used during adversarial training. Curves were aggregated over 200 samples from the test set.
Figure 4: (Left) Deviation from linearity [|04-1(x 2)
function of the magnitude | of random and
Trahse:
Singular Vactor Algimmen
Talred: dep 285, Parsunations FGA
Tranne: glital 5N1
Traned: gicsa Partutationn PSA
Trainel: Standori, Poturiaions: Ravdon
4:5
Actuncy
02
03
6.0
25
7.5
350
35.9
20.3
0.0
22.00
40.0
56.
500
704
Pertarbation Magaitude
Pesturbation Magnitude
Figure 5: (Left) Classification accuracy function of perturbation strength (measured in 8-bit). (Right) Alignment of &o -PGA adversarial perturbations with dominant singular vector of fix) as a function of perturbation magnitude e. The dashed vertical line indicates the used during adversarial training. Curves were aggregated over 2000 samples from the test set.
15

---

Under review as a conference paper at ICLR 2020
A.7 FURTHER EXPERIMENTAL RESULTS FOR GLOBAL SNR WITH 10 ITERATIONS
In the main section, we have implemented the baseline version of global SNR as close as possible to the descriptions in Yoshida & Miyato (0117). However, this included recommendation from the authors to perform only a single update iteration to the spectral decompositions of the weight matrices per training step. As this is computationally less demanding than the 10 iterations per training step spent on both adversarial training, as well as data-dependent spectral norm regularization, we verify that performing 10 iterations makes no difference the method of Yoshida Miyato (017)). Figures
[6] and and 7 reproduce the curves for global SNR from the main part (having used I iteration) and overlap with the same experiments, but done with global SNR using 10 iterations. As can be seen, there is no significant difference between the two.
red goos SNG, Parusattinn: PGA guou cerarooeee
Trahnd gaba Farburtariae PGA
pMbb Feturburens
030
Deviation from Linearity
Trained pibbi Pertuttatinn Sandem
20-
Top Singuhar
Distonen from
Distance from
Figure 6: (Left) Deviation from linearity 1x 2) (x) + Jos 16x)2)ll2 as a function of the distance ||z]|a from x for random and adversarial perturbations z. (Right) Largest singular value of the linear operator J (x+=z) as function of the magnitude |/z||2 of random and adversarial perturbations z. The dashed vertical line indicates the used during ad adversarial training. Curves were aggregated over 200 samples from the test set.
Traive ghiss 5NR
Trainw guenal SKA, AeT wwa.araeenc PSA
Truincd: global 5M lan!
Thamnez glocal SWR 120 busil Potlabbllsno POA
Singolar Vectur Aliymment
8.0
6.5
02-
0.08
0.00
-2.0
Perturhation Magninde
Pertorbstion Magnitude
Figure 7: (Left) Classification accuracy as a function of perturbation strength e. (Right) Alignment of adversarial perturbations with dominant singular vector of J f(x) a function of perturbation magnitude e. The dashed vertical line indicates the e used during adversarial training. Curves were aggregated over 2000 samples from the test set.
A.8 HYPERPARAMETER SWEEP
We tested the following hyperparameter configurations during training and selected the best perform- ing for each training method. Table 2 details the searched parameters.
A.9 ADVERSARIAL TRAINING WITH LARGE a
Figure X shows the result of empirically raising the adversarial learning rate during adversarial training with PGA to very large values. As can be seen, the adversarial robustness initially rises, but after some threshold levels and does change significantly even at very large values.
16

---

Under review as a conference paper at ICLR 2020
Table Hyperparameter sweep during training.
TRAINING METHOD
HYPERPARAMETER
VALUES TESTED
ADVERSARIAL TRAINING
0.5,0.75, 1.0, 1.25, 1.5, 1.75, 2.0,
2.7, 3.0, 3.25,
6/ITERS, 2:/1TERS, 3 4e/TEES,
ITERS
1,2,3,5, 10,8 15, 30, 40,
GLORAIL SPECTRAL REG.
1.10-2, 33.33..000
10-100- 10-10-
1.10 8,
8
8,1.10 2,3
1.1110
3. 1
1.10
10-0,,.. 33-10
ITERS
1,10
NORM
1-10-9.0
1.00-93-1007
100.1111111-00000 13-10-44
1.10 1-10-3,3-
10
1. 2.3
11 1001
10"01
1. 10
10
10 3. 10*
ITERS
1,2,3,5,8,
15, 40.
1.0
Test Set Accuracy
0.6
0.2
Clean Samples
Adversarial Samples
1.0
100.0 10000010000.000 BO0000.0
used for Adversarial Training
Figure 8: Test set accuracy on clean and adversarial examples after using adversarial (PGA) training with given step size a. The dashed line indicates the a used when generating adversarial examples at test time.
17


Reviews:
[{"review_id": "SyeD7eqHcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This largely theretical paper establishes a theoretical link between adversarial training and operator norm regularization\nfor DNNs. It is well written and structured, and it falls squarely within the the remit of the conference. The experimental apparatus is thorough and the derivations, proofs and the maths at large seem sound to me, even if I have not checked them in full detail. The study delivers a data-dependent variant of spectral norm regularization affecting large singular values of the DNN. It is proved to be equivalent to adversarial training based on a type of norm-constrained projected gradient ascent attack.\nResults are novel and relevant and, in my opinion, they merit acceptance.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xIuisysH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Adversarial training generalizes data-dependent spectral norm regularization\n\nThis paper shows that, projected gradient descent based adversarial training is similar to the data-dependent spectral norm regularization, and under very restrictive condition, the authors show that this two methods are the same. Some experiments are conducted to support the theory.\n\nOverall, I think this paper is marginal, while the experiments are not convincing. First, the relation between spectral normalization and adversarial training have been investigated by [1], while the fast computational of maximum singular value with power methods have also been proposed in [1]. The authors only give a data-dependent version of the spectral normalization based on the Jacobian of the neural networks, which I think is somewhat weak. The experiments are limited with specific settings that are not generally used in practice, which alleviate my confidence on this paper’s results. Also, the experiment section contain several not so important information. I think the authors should do far more experiments to support the main claim, while move these additional justification to the appendix.\n\nDetailed comments:\n1. I think the claim of theorem 1 is somewhat ambiguous. How to guarantee there exists such epsilon satisfies this condition? Is this the case we face in the real world? What will happen if \\alpha is not sufficiently large? If we don’t use logits pairing and \\ell_2 norm constraint, will the claim hold? I think the correlation behinds the spectral norm and adversarial training is well investigated and use this correlation as the intuition behinds work is enough. This theorem cannot convince me that the proposed methods have a strong theoretical basis.\n2. Generally, the neural networks have a large number of parameters (~ millions) for image classification task. The global spectral norm regularization only needs to calculate the spectral norm of each layer’s weight matrix, whose computational cost is acceptable. However, to calculate the Jacobian and use the power methods, we will additionally do several forward pass and backward pass just as adversarial training. As a regularization technique, is this calculation tolerable? If this is some variant of the adversarial training, I don’t find the experiment results support the claim that it will outperform the adversarial training consistantly.\n3. Why don’t use some standard neural network architecture like ResNet? As this results is not comparable to other existing work, I’m not sure if this result is meaningful. Also, are the comparisons fair? For example, the regularization coefficient of global spectral norm regularization and data-dependent spectral norm regularization are far more different. And the authors use only 1 iteration to calculate the singular value in global spectral norm regularization, why to do that? Also, what’s the result compared with \\ell_p norm constraint adversarial training?\n4. The evaluation of some assumption on the network is better moved to appendix, as this is only some sanity check, not the core contribution. More experiments with ResNet, WideResNet, MobileNet etc. on CIFAR100 and ImageNet are more convincing.\n5. What’s the attack method in the main context?\n6. I think the discussion in Appendix A.5 is somewhat confusing. If the authors want to argue that the network is locally linear so that we can approximate with linear regression, why should we use the power methods?\n\nStill, I feel the contribution of this paper is somewhat weak. I don’t see any improvements of the proposed algorithms compared with the standard adversarial training, as well as the theoretical contribution like adversarial generalization. The experiments are not convincing, as the setting is different from the general setting the community used in adversarial training. I’m not familiar with the results in global spectral normalization and it’s possible that the global spectral normalization may have little gain in adversarial robustness, but in my opinion, the main contribution [1] is the generalization analysis of spectral normalized adversarial trained neural networks, which this paper lacks. On the empirical side, the computation efficiency and performance of the proposed algorithms don’t outperform adversarial training much. So I tend to reject this paper.\n\n\n[1] Farnia, Farzan, Jesse Zhang, and David Tse. \"Generalizable Adversarial Training via Spectral Normalization.\" International Conference on Learning Representations, 2019.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1laVbdjYH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the link between adversarial training and the proposed data-dependent operator norm regularization for ReLU network. Under specific conditions, in theory, the authors show the equivalence between the l2 PGD training and the regularization method. Empirical experiments are conducted to support the theory.\n\nWhile this paper gives interesting observation on both theory and empirical study, I think this paper is not qualified for publishing in ICLR due to the following reasons: (1) limited theoretical results; (2) No significant improvement for practical algorithm;\n\nMain argument:\n\nThe main theorem 1 seems to be weak as it is only valid for small perturbation region \\epsilon and it is unclear how this assumption is consistent to the practice. It is also unclear how the assumption that \\alpha \\to \\infty influences the practical algorithm.\n\nIt would be better to generalize the theorem to other \\ell_p attack, instead of just \\ell_2. \n\nDiscussion of computational complexity of the proposed regularization method compared with PGD is missed.\n\nThe adversarial robustness is related to the (local) Lipschitz continuity and many other types of regularization decreases the (local) Lipschitz constant. Could you further give result that distinguishes the proposed norm?\n\nIt would be better to give improved algorithm for adversarial training based on the current result. The current contribution for further theoretical is too weak as the main theorem requires strong assumption. And I don’t see significant contribution to empirical algorithm.\n\n\nMinor\nEqu (10) seems not the typical one used and seems not the one studied later.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper shows an theoretical equivalence between the L2 PGD adversarial training and operator norm regularization. It gives an interesting observation and support it from both theoretical arguments and practical experiments. There has been a significant discussion between the reviewers and authors. Although the authors made efforts in rebuttal, it still leaves many places to improve and clarify, especially in improving the mathematical rigor of the  proof and experiments using state-of-the-art networks. 


Decision: Reject