Towards Understanding the Regularization of Adversarial Robustness on Neural Networks
Authors: Yuxin Wen
Shuai Li
Kui Jia
Venue:  (2020)
Keywords: Adversarial robustness
Statistical Learning
Regularization

Abstract:   The problem of adversarial examples has shown that modern Neural Network (NN) models could be rather fragile. Among the most promising techniques to solve the problem, one is to require the model to be {\it $\epsilon$-adversarially robust} (AR); that is, to require the model not to change predicted labels when any given input examples are perturbed within a certain range. However, it is widely observed that such methods would lead to standard performance degradation, i.e., the degradation on natural examples.  In this work, we study the degradation through the regularization perspective.  We identify quantities from generalization analysis of NNs; with the identified quantities we empirically find that AR is achieved by regularizing/biasing NNs towards less confident solutions by making the changes in the feature space (induced by changes in the instance space) of most layers smoother uniformly in all directions; so to a certain extent, it prevents sudden change in prediction w.r.t.  perturbations. However, the end result of such smoothing concentrates samples around decision boundaries, resulting in less confident solutions, and leads to worse standard performance.  Our studies suggest that one might consider ways that build AR into NNs in a gentler way to avoid the problematic regularization.


Under review as a conference paper at ICLR 2020
TOWARDS UNDERSTANDING THE REGULARIZATION OF ADVERSARIAL ROBUSTNESS ON NEURAL NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The problem of adversarial examples has shown that modern Neural Network (NN) models could be rather fragile. Among the most promising techniques to solve the problem, one is to require the model to be e-adversarillly robust (AR); that is, to require the model not to change predicted labels when any given input examples are perturbed within certain range. However, it is observed that such methods would lead to standard performance degradation, i.c., the degradation on natural examples. In this work, we study the degradation through the regularization perspective. We identify quantities from generalization analysis of NNs; with the identified quantities we empirically find that AR is achieved by regularizing/biasing NNs towards less confident solutions by making the changes in the feature space (induced by changes in the instance space) of most layers smoother uniformly in all directions; so to certain extent, it prevents sudden change in prediction w.r.t. perturbations. However, the end result of such smoothing concentrates samples around decision boundaries, resulting in less confident solutions, and leads to worse standard performance. Our studies suggest that one might consider ways that build AR into NNs in gentler way avoid the problematic regularization.
INTRODUCTION
Despite the remarkable performance (Krizhevsky et al., 2012) of Deep Neural Networks (NNs), they are found to be rather fragile and easily fooled by adversarial examples (Szegedy et al., 2014). More intriguingly, these adversarial examples are generated by adding imperceptible noise to normal examples, and thus are indistinguishable for humans. NNs that are more robust to adversarial examples tend to have lower standard accuracy (Su et al., 2018), i.e., the accuracy measured on natural examples. The trade-off between robustness and accuracy has been observed (Kurakin et al., 2017; Madry et al., 2018; Tsipras et al., 2019). To understand such phenomenon, Tsipras et al. (2019) show that for linear models, if examples are closed to decision boundaries, robustness provably conflicts with accuracy, though the proof seems unlikely to generalize to NNs. Zhang et al. (2019) show that gap exists between surrogate risk gap and 0-1 risk gap if many examples decision boundaries, and better robustness can be achieved by pushing examples away from decision boundaries. But pushing examples away again degrades NN performance in their experiments. A more established remedy is developed to require NNs to be e-adversarially robust (AR), e.g., via Adversarial Training (Madry al., 2018), Lipschitz-Margin Training (Tsuzuku al., 2018); that is, they require the model not to change predicted labels when any given input examples are perturbed within a certain range. Note that such hard requirement is different from penalties on the risk function employed by Lyu et al. (2015) and Miyato et al. (2018), which is not our subject of investigation (more discussion in appendix A). practice, hard-requirement methods are found to lead to worse performance measured in standard classification accuracy. We aim to study this branch of methods.
close
We investigate how adversarial robustness influence the behaviors of NNs to make them more robust but have lower performance. In earlier time (Szegedy et al., 2014), adversarial training has been suggested as a form of regularization: it augments the training of NNs with adversarial examples, thus might improve the generalization of the end models. How does possible improvement in generalization end up degrading performance? It prompts us to analyze the regularization effects of AR NNs. A successful regularization technique is expected improve test performance,

---

Under review as a conference paper at ICLR 2020
Skerdeerd Degalien di Singltee Velues
Adv. Adv. 16
Adv. 4
2.0
0.04
Adv.
Adv. 16
00
p-obabitte
2:0.0
1.5
0.02
1.0
0.11
0.5
0.00
30
40
50
-50
-25
Magnibudo Margin
25
1.00
Adverserial Robusthess
(a) STD of Singular Values
(b) Margin Distribution
(c) Accuracy
Figure 1: Experiment results ResNet56 (He et al., 2016) trained the CIFARIO 10 dataset. For the details of the experiments, refer to section 5. (a) The standard deviation of singular values of each layer of NNs with adversarial robustness (AR) strength 4, 16 (AR strength 8 is dropped for clarity of the plot). To emphasize, the x-axis is the layer index overall 56 layers are involved. (b) The probability distribution of margins of NNs with AR strength 4, 8, 16. (c) The standard and adversarial accuracy of NNs with AR 4, 8, 16.
but an improved performance is only one of the possible outcomes of improved generalization. Technically, improved generalization implies the reduction in gap between training errors and test errors. Regularization achieves the gap reduction by reducing the size of the hypothesis space, which reduces the variance, but meanwhile increases the bias of prediction made- constant classifier can have zero generalization errors, but also have low test performance. Thus, when a hypothesis space is improperly reduced, another possible outcome is biased poorly performing models with reduced generalization gaps.
Through series of theoretically motivated experiments, we find that AR is achieved by regularizinggbiasing NNs towards less confident solutions by making the changes in the feature space of most layers (which are induced by changes in the instance space) smoother uniformly in all directions; so to a certain extent, it prevents sudden change in prediction w.r.t. perturbations. However, the end result of such smoothing concentrates examples around decision boundaries and leads to worse standard performance. We elaborate the above statement in details shortly in section 1.1.
Key results.
Overall, the investigation of generalization behaviors of NNs points out possible directions where we might go if we are to resolve the issue of the test performance degradation done by AR. The main result shows that the hypothesis space of NNs improperly reduced, thus we might investigate how to avoid it when enforcing AR. Though beyond the scope of this work, we conjecture that the improper reduction comes from the indistinguishabiliy of the change induced in the intermediate layers of NNs by adversarial noise and that by inter-class difference. To guarantee AR, NNs are asked to smoothe out difference uniformly in all directions in high dimensional space, and thus are biased towards diffident solutions that make similar/concentrated predictions. We leave the investigation of the conjecture as future works.
1.1 AR LEADS TO DIFFIDENT NNS WITH MORE INDECISIVE MISCLASSIFICATIONS
This section elaborates the key results we briefly present previously.
1. AR reduces the variance of (norms of) the activation/outputs (compared with NNs with different AR strength) at most layers that are emitted/induced by feeding perurbations (of any directions) to that layer from the previous layer. Through a series of theoretically motivated experiments, the results prompt us to look at the singular value distributions of the weight matrix of each layer of the NNs. Shown in fig. 1a, we find that overall the standard deviation (STD) of singular values associated with a layer of the NN trained with lower AR strength is larger than that of the NN with higher AR strength 16' the green dots are mostly below the red dots. Note that given matrix W and an example æ, singular values of W determine how the norm || W x is changed
'The AR strength is characterized by the maximally allowed Lx norm of adversarial examples that are used to train the NNs we use adversarial training (Madry et al., 2018) to build adversarial robustness into NNs. Details be found in appendix B.1
2

---

Under review as a conference paper at ICLR 2020
when compared with | More specifically, let minn max be the maximal and minimal singular
values, if x is not in the null space of W. then we have ||a]| E [dmin||<]] Tmax [ []. where
||. denotes 2-norm. This applies to norm ||8x||| of perturbation well: that is, given possible
changes áx of æ of the same norm || = c, where e: is constant, the variance of a (WW roughly determines the variance of || [ 8a where (W) denotes all singular values {a,} of W. In more details, note that by SVD decomposition, Wox = >, ou dæ, thus o; determines how the component v dæ in the direction of v; is amplified. To see an example, suppose that Owin = max = oo. then the variance of o (W) variance of [ (given an ensemble of perturbations óæ of the same norm c) is zero as well. The conclusion holds as well for ReLU W òz), where W here is a weight matrix of layer of a NN, and ReLU denotes Rectifier Linear Unit activation function (proved by applying Cauchy interlacing law by row deletion (Chafai) to lemma 4.1). Consequently, by reducing the variance of singular values of weight matrix of a layer of the NN, AR reduces the norm variance of layer activations induced by input perturbations.
W is zero, and || W dx|| = ou ou||6az||. | In this case, the
2. The reduced norm variance induced by example perturbations concentrates examples, and it empirically concentrates them around decision boundaries; that is. predictions are more diffident. The reduced variance implies that the outputs of each layer of the NN more concentrated, but it does not tell where they are concentrated. Note that in the previous paragraph, the variance relationship discussed between || 6.z| and ||ōz||| equally applies to [Wal| and ||>||, where is an actual example instead of perturbations. Thus, to find out the concentration of perturbations, we can look at the concentration of samples. Technically, we look margins of examples. In a multi-class setting, suppose NN computes a score function f : R - R,, where L is the number of classes; a way to convert this to a classifier is to select the output coordinate with the largest magnitude, meaning x arg max, (z)). The confidence of such classifier could be quantified by margins. It measures the gap between the output for the correct label and other labels, meaning fz(z) m fz). Margin piece-wise linearly depends margins is also in a picce-wise linear relationship with the variance of the scores, which are computed linearly from the activation of NN layer. Thus, the consequence of concentration of activation discussed in the previous paragraph can be observed in the distribution of margins. More details of the connection between singular values and margins are discussed in section 5.2.2, after we present lemma 4.1. A zero margin implies that a classifier has equal propensity to classify an example to two classes, and the example is on the decision boundary. We plot the margin distribution of the test set of CIFAR10 10 in fig. 1b, and find that margins are increasingly concentrated around zero that is, the decision boundaries - as AR strength grows.
the scores, thus the variance of
3. The sample concentration around decision boundaries smoothes sudden changes induced per- turbations, but also increases indecisive misclassification. The concentration of test set margins implies that the induced change in margins by the perturbation in the instance space is reduced by AR. The statement may not be immediately obvious, so we explain in details as follows. Given two examples x, x' from the test set, dæ changes the example x to x'. The concentration of overall margins implies the change induced by óz is smaller statistically in NNs with higher AR strength. Thus, for adversarial perturba- tion applied on æ, statistically the change of margins is smaller as well - experimentally it is reflected in the increased adversarial robustness of the network, as shown in the increasing curve in fig. Ic. That is, the sudden changes of margins originally induced by adversarial perturbations are smoothed (to change slowly). However, the cost of such smoothness lower confidence in prediction, and more test examples are aigghtyyineeciiieely moved to the wrong sides of the decision boundaries- incurring lower accuracy, as shown in the decreasing curve in fig. Ic.
x' can be taken significant perturbation that
Lastly, we note that experiments in this section are used to illustrate our main arguments in this section. Further consistent quality results are reported in section 5 by conducting experiments on CIFAR10/100 and Iiny-ImageNer with networks of varied capacity.
1.2 OUTLINE AND CONTRIBUTIONS
As briefly discussed at the beginning, this work carries out generalization analysis on NNs with AR. The quantities we investigate in the previous section are identified by the generalization errors (GE) upper bound we establish at theorem 1. which characterizes the regularization of AR NNs. The key result is actually obtained at the end of a series of analysis, thus we present the outline of the analysis here.
3

---

Under review as a conference paper at ICLR 2020
Outline. After presenting some preliminaries in section 3, we proceed to analyze the regularization of AR on NNs, and establish a GE upper bound in section 4. The bound prompts us to look at the GE gaps in experiments. In section 5.1, we find that for NNs trained with higher AR strength, the surrogate risk gaps (GE gaps) decrease for a range of datasets, i.e., CIFARIO/ and Tiny-ImageNet (ImageNet, 2018). It implies AR effectively regularizes NNs. We go further study the finer behavior change of NNs that might lead to such a gap reduction. Again, we follow the guidance of theorem 4.1. We look at the margins in section 5.2.1, then the singular value distribution in section 5.2.2, and discover the main results described in section 1.1. More corroborative experiments are run in appendix B.4 to show that such phenomenon exists in broad range of NNs with varied capacity, and more complementary results are present in appendix B.3 to explain some seemingly abnormal observations. More related works are present section
Contributions. Overall, the core contribution in this work is to show that adversarial robustness (AR) regularizes NNs way that hurts its capacity to learn to perform test. More specifically:
We establish a generalization error (GE) bound that characterizes the regularization of AR on NNs. The bound connects margin with adversarial robustness radius e via singular values of weight matrices of NNs, thus suggesting the two quantities that guide us to investigate the regularization effects of AR empirically.
Our empirical analysis tells that AR effectively regularizes NNs to reduce the GE gaps. To understand how reduced GE gaps turns out to degrade test performance, we study variance of singular values of layer-wise weight matrices of NNs and distributions of margins of samples, when different strength of AR are applied on NNs.
The study shows that AR achieved by regularizing/biasing NNs towards less confident solutions by making the changes in the feature space of most layers (which are induced by changes in the instance space) smoother uniformly in all directions; so to a certain extent, it prevents sudden change in prediction w.r.t. perturbations. However, the end result of such smoothing concentrates samples around decision boundaries and leads to worse standard performance.
RELATED WORKS
Robustmess in machine learning models is a large field. We review some more works that analyze robustness from the statistical perspective. The majority of works that study adversarial robustness from the generalization perspective study the generalization behaviors of machine learning models under adversarial risk. The works that study adversarial risk include Attias et al. (2018); Schmidt et al. (2018); Cullina et al. (2018); Yin and Bartlett (2018); Khim and Loh (2018); Sinha et al. (2018). The bounds obtained under the setting of adversarial risk characterize the risk gap introduced by adversarial examples, thus, is intuitive that a larger risk gap would be obtained for a larger allowed perturbation limit which is roughly among the conclusions obtained in those bounds. That is to say, the conclusion normally leads to a larger generalization error as an algorithm asked to handle more adversarial examples, for that it focuses on characterizing the error of adversarial examples, not that of natural examples. However, adversarial risk is not our focus. In this paper, we study when a classifier needs to accommodate adversarial examples, what is the influence that the accommodation has on generalization behaviors of empirical risk of natural data.
PRELIMINARIES
Assume an instance space Z X' x y, where X' is the space of input data, and 3/ is the label space. Z : (X,Y) are the random variables with an unknown distribution μ,, from which we draw samples. We use Sm = {2+ (x;, ,:m to denote the training set of size m whose examples are drawn independently and identically distributed (i.i.d.) by sampling Z. Given loss function I, the goal of learning to identify a function T X + y in a hypothesis space (a class T of functions) that minimizes the expected risk
R(I T) Ez (T(X), Y)), Since i is unknown, the observable quantity serving as the proxy to the expected risk R is the empirical risk
Rm (1 T) 1 Σ 1T)
m 1=1
4

---

Under review as a conference paper at ICLR 2020
Our goal is to study the discrepancy between R and Rm, which is termed as generalization enror is also sometimes termed as generalization gap the literature
GB(l o T) = -R(( o T) - Rm TT)-
(1)
Definition (Covering number). Given metric space (S: p), and subset $ C S, we say that subset S of a c-cover S. i8 S, 38 S such that p(3.8) 8) S The e-covering number of S
is
S, p) = min is an e-covering of S)
Various notions of adversarial robustness have been studied in existing works (Madry et al., 2018; Tsipras al., 2019; Zhang al., 2019). They are conceptually similar; this work, we formalize its definition to make clear the object for study.
Definition ((p, adaversaaiaal robustness). Given multi-class classifier f X R and metric on X, where L is the number of classes, f is said to be adversarially robust w.r.t adversarial perturbation of strength e, if there exists an > 0 such that Yz (x. E and 8æ {(((xx) S c). we have
fa + dæ:) f:( + 5a)) 2 0,
where ÿ = arg max, (z) and + y. e is called adversarial robustness radius. When the metric used is clear, we also refer (p, -)-adversaiial robustness as e-adversarial robustness.
Note that the definition is example-wise one; that is, it requires each example to have guarding area, in which all examples are of the same class. Also note that the robustness is w.r.t. the predicted class, since ground-truv truth label is unknown for test.
We characterize the GE with ramp risk, which is typical risk to undertake theoretical analysis (Bartlett et al., 2017; Neyshabur et al., 2018b).
Definition 3 (Margin Operator). A margin operator M : R x .1..... L} + R is defined as
M(s, 3) == Sy max
Definition 4 (Ramp Loss). The ramp loss by R IR is defined as
r r< -7 (r) = 1+r/7 rE [- r> 0
0
Definition (Ramp Risk). Given a classifier f. ramp risk is the risk defined as R2(f)) == E(L((-(((X))
where X, are random variables in the instance space Z previously.
We will use a different notion of margin in theorem 4.1, I, and formalize its definition follows. We reserve the unqualified word "margin" specifically for the margin discussed previously - the output of margin operator for classification. We call this margin to-be-inttrr introduced instance-space margin (IM).
Definition 6 (Smallest Instance-space Margin). Given an element (ar E Z, let v(x)) be the distance from x to its closest point on the decision boundary, i.e., the instance-space margin (IM) of example Given e-covering of Z, let
Umin
min [
v(x).
=E(xEX
Umin the smallest instance- space margin of elements the covering balls that contain training examples.
4 THEORETICAL INSTRUMENTS FOR EMPIRICAL STUDIES ON AR
In this section, we rigorously establish the bound mentioned in the introduction. We study the map T defined in section 3 as NN (though technically, T now is map from X' to R., instead of to ) such an abuse of notation should be clear the context). To begin with, we introduce an assumption, before state the generalization error bound guaranteed by adversarial robustness.
5

---

Under review as a conference paper at ICLR 2020
Ducidia booedday the topethara ques
niioalnno cedita
leperylane wee out Does the topollasic space
(b)
Figure 2: (a) Illustration of the regularization effect of adversarial robustness. If a NN T is adversarially robust, for a given example x (drawn as filled squares or circles) and points x' in the yellow ball {æ' p(xx, x²) S <} around æ, the predicted labels of æ, x' should be the same, and the loss variation is potentially bigger as x'' moves from the center to the edge, as shown as intenser yellow color the edge of ball. Collectively, the adversarial robustness of each example requires an instance-span space margin (IM) to exist for the decision boundary, shown as the shaded cyan margin. As normally known, margin related to generalization ability that shrinks the hypothesis space. In this case, the IM required by adversarial robustness would weed out hypotheses that do not have an adequate IM, such as the red dashed line shown in the illustration.
(b) Illustration of lemma 4. 1. Given NN with ReLU activation function, the feature map I at layer 1 is divided into regions where I(x)) is piecewise linear w.r.t. T. The induced linear map W is given by diag (TI (4)) Wi, where diag(m) (T) (4)) is a diagonal matrix whose diagonal entries given by vector (ạ) that has 0-1 values. For example, in region p. I1 = W x and distance between instances a are vertical elongated, while in region q. 1 W and distance are horizontally elongated. Thus given æ, x', the difference [x)( (xx)) ]l between (x) and (x') is the length of the transformed line segment x: x drawn, of which each segment linearly transformed in different way.
Assumption 4.1 (Monotony). Given a point a E X. let x' be the point on the decision boundary of NN that is closest to Then, for all x" the line segment + t(xxx =), E [0, 1), the margin M(Tx", "CTT",") .u) decreases monotonously.
The assumption is a regularity condition on the classifier that rules out undesired oscillation between and æ'. To see how, notice that the margin defined in definition reflects how confident the decision made. Since x' on the decision boundary, means the classifier unsure how should be classified. Thus, when the difference x' x is gradually added to ideally we want the confidence that we have on classifying x to decrease a consistent way to reflect the uncertainty.
Theorem 4.1. Let T denote NN with ReLU and MaxPooling nonlinear activation functions (a definition is put at eq. (6) for readers' convenience). ly the ramp loss defined at definition 4, and the instance space assumed section 4. Assume that Z is a k-dimensional regular manifold that accepts an e-covering with covering number (Sx)a and assumption 4.1 holds. T is eg en-adversarially robust (defined definition 2), S co, and denote Vaiin the smallest IM margin the covering balls that contain training examples (defined at definition 6), amin the smallest singular values of weight matrices w,.i=,,.... ........-..................... of a NN, (w;)i=t.....1) the set of vectors made up with ith rows of WL L (the last layer's weight matrix), then given an i.i.d. training sample S, {2 (x yss) ) drawn from Z. its generalization error GE(L T') (defined at eq. (1)) satisfies that, for any 1 > 0,
with probability at least
Umin
2 Olog()) emm
2 2lo((1/))
GE(L, T) S max (0,
(3)
where
T--1
Umin
min w.
I,,, uogl 112
II imin Umin
lower bound of margins of examples in covering bally that contain training samples.

---

Under review as a conference paper at ICLR 2020
The proof of theorem 4.1 is in appendix C. The bound identifies quantities that would be studied experimentally in section to understand the regularization of AR on NNs. The first term in eq. (3) in theorem 4.1 suggests that quantities related to the lower bound of margin umin might be useful to study how c-adversarial robustness (c-AR) regularizes NNs. However, c-AR is guaranteed in the instance space that determines the smallest instance-space margin Umin- To relate GE bound with e-AR, we characterize in eq. (4) the relationship between margin with IM, via smallest singular values of NNs' weight matrices, suggesting that quantities related singular values of NNs' weight matrices might be useful to study how AR regularizes NNs as well. An illustration on how AR could influence generalization of NNs through IM is also given in fig. 2a. The rightmost term in eq. (3) is standard term in robust framework (Xu and Mannor, 2012) in learning theory, and is not very relevant to the discussion. The remaining of this paper are empirical studies that are based on the quantities, e.g., margin distributions and singular values of NNs' weight matrices, that are related to the identified quantities, i.c., Umin minn. These studies aim to illuminate with empirical evidence on the phenomena that AR regularizes NNs, reduces GE gaps, but degrades test performance.
Before turiing into empirical study, we further present a lemma to illustrate the relation characterized in eq. (4) without the need to jump into proof of theorem 4. It would motivate our experiments later in section 5.2.2. We state the following lemma that relates distances between elements in the instance space with those in the feature space of any intermediate network layers.
Lemma 4.1. Given two instances x, x' X, let (xxx ) be the activation g( (Wig(W(-..... (W (Wz)) ))))
at layer of x (c.f. definition of NNs at appendix C.2), then there exist n E N sets of matrices (W? )==1..,;] = 1 .,, that each of the matrix W is obtained by setting some rows of W, to zero, and {gr (q.)j=....n arbitrary distinctive symbols indexed by that index W such that
[|](x)) - -1(2')1 = Σ j=1
II W dt(x x')
where 81 = 8,+1 = Cj. = 1, ej s, E [0, 1] each (s3, eg) is a segment in the line segment parameterized by t that connects a and x'.
Its proof is in appendix C, and an illustration is given in fig. 2b. Essentially, it states that difference in the feature space of NN, induced by the difference between elements in the instance space, is a summation of the norms of the linear transformation (T W,)) applied on segments of the line segment that connects æ, x' in the instance space. Since :. is obtained by setting rows of W; to zero, the singular values of these induced matrices are intimately related to weight matrices W; of NN by Cauchy interlacing law by row deletion (Chafai). Since the margin of an example æ is linear transform of the difference between (z) and the Iz=)(x') of an element x' on the decision boundary, singular values of [W.)=s....- determine the amplliicatiinn/shiinnee of the IM a x'.
5 EMPIRICAL STUDIES ON REGULARIZATION OF ADVERSARIAL ROBUSTNESS
In this section, guided by theorem 4. we undertake empirical studies to explore AR's is's regularization effects on NNs. We first investigate the behaviors of off-the-shelf architectures of fixed capacity on various datasets in section 5.1 and 5.2. More corroborative controlled studies that explore the regularization effects of AR on NNs with varied capacity are present in appendix B.4.
5.1 ADVERSARIAL ROBUSTNESS EFFECTIVELY REGULARIZES NNs ON VARIOUS DATASETS
This section aims explore whether AR can effectively reduce generalization errors more specifically, the surrogate risk gaps. We use adversarial training (Madry et al., 2018) to build
2 Note that in the previous paragraph, though we identifies quantities umin and oni related to the upper bound of GE, the quantities we actually would study empirically are margin distribution and all singular values that characterize the GE of all samples, not just the extreme case (upper bound). The analytic characterization of the GE of all samples not possible since we do not have enough information (at least we do not know the true distribution of samples). That's why to arrive at close-form analytic characterization of GE, we resort to the extreme oon-asympootic large-sample behaviors. The analytic form is a neat way to present how relevant quantities influence GE. In the rest of the paper, we would carry on empirical study on the distributions of margins and singular values mostly to investigate AR's influence GE of all samples.
7

---

Under review as a conference paper at ICLR 2020
7.8
CRARO
1.-
0.5
125
12-
3
10-
045
15
15
15
10 15
20 35
26 25
15
15
a
(b)
Figure 3: Experiment results on CIFARIO/100, and Tiny- ImageNe..... Net A, B are ResNet-56 and ResNet- 1 10 (He et al., 2016) respectively. The unit of x-axis is the adversarial robustness (AR) strength of NNs, c.f. the beginning of section 5. (a) Plots of loss gap (and error rate gap) between training and test datasets v.s. AR strength. (b) Plots of losses (and error rates) on training and test datasets v.s. AR strength.
adversarial robustness into NNs. The AR strength is characterized by the maximally allowed (o0 norm of adversarial examples that are used to train the NNs. Details on the technique to build adversarial robustness into NNs is given in appendix B.I.
Our experiments conducted CIFARIO, CIFAR100, and Tiny-ImageNet (ImageNet, 2018) that represent learning tasks of increased difficulties. We use ResNet-56 and ResNet- 10 (He et al., 2016) for CIFARIO/100, and Wide ResNet WWRR--50-2bootlenne
(Zagoruyko and Komodakis, 2016) for Tiny-ImageNet (ImageNet, 2018). These networks are trained with increasing AR strength. Results plotted in fig. 3, where Net A stands for ResNet56, and Net B for ResNet-110.
Regularization of AR on NNs. We observe fig. 3a (shown as blue lines marked by circles) that GE gaps (the gaps between training and test losses) decrease as strength of AR increase; we also observe in fig. 3a that training losses increase as AR strength increase; these results (and more results in subsequent fig. 6) imply that AR does regularize training of NNs by reducing their capacities to fit training samples. Interestingly, in the CIFAR 10/100 results in fig. 3b, the test losses show a decreasing trend even when test eтror rates increase. It suggests that the network actually performs better measured in test loss as contrast to the performance measured in test error rates. This phenomenon results from that more diffident wrong predictions adversarial training, which will be explained in details in section 5.2, when we carry on finer analysis. We note that on Tiny-ImaeeNet, the test loss does not decrease as those on CIFAR10/100 is likely because the task is considerably harder, and regularization hurts NNs even measured in test loss.
made by NNs thanks to
Trade-off' between regularization of AR and test error rates. The error rate curves in fig. 3b also tell that the end result of AR regularization leads to biased-performing NNs that achieve degraded test performance. These results consistent across datasets and networks.
Seemingly abnormal phenomenon. An seemingly abnormal phenomenon CIFARIO observed in fig. 3a is that the error rate gap actually increases. It results from the same underlying behaviors of NNs. which we would introduce in section 5.2, and an overfitting phenomenon that AR cannot control. Since it would be digress to explain, it is put in appendix B.3.
We finally note that the adversarial robustness training reproduced is relevant, of which the defense effect is comparable with existing works. One may refer to fig. 11 in appendix D.2 for the details. We can see from it that similar adversarial robustness to Madry et al. (2018) and Li ct al. (2018) is achieved for CIFAR 10/100, Imag-em in the NNs we reproduce.
8

---

Under review as a conference paper at ICLR 2020
5.2 REFINED ANALYSIS THROUGH MARGINS AND SINGULAR VALUES
The experiments in the previous sections confirm that AR reduces GE, but decreases accuracy. We study the underlying behaviors of NNs to analyze what have led to here. More specifically, we show that adversarial training implements e-adversarial robustness by making NNs biased towards less confident solutions; that is, the key finding we present in section 1.1 that explains both the prevented sudden change in prediction w.r.t. sample perturbation (i.e., the achieved AR), and the reduced test accuracy.
5.2.1 MARGINS THAT CONCENTRATE MORE AROUND ZERO LEAD TO REDUCED GE GAP
To study how GE gaps are reduced, theorem 4.1 suggests we first look at the margins of examples a lower bound of margins is Umin in eq. (4). The analysis on margins has been widely used tool in learning theory (Bartlett et al., 2017). reflects the confidence that a classifier has on an example, which after being transformed by loss function, is the surrogate loss. Thus, the loss difference between examples are intuitively reflected in the difference in confidence characterized by margins. To study how AR influences generalization of NNs. plot in fig. 4 the margin distributions of samples which are obtained by training ResNet-56 on CIFARIO and CIFARI00 with increased AR strength (the same setting as for fig. 3). Applying the same network of ResNet-56 respectively on CIFAR-IO and CIFAR-100 100 of different learning difficulties creates learning settings of larger- and smaller-capacity NNs.
104
500
E04
504
$000
sro
Fece
10
660
(a) CIFARIO Test
(b) CIFARIO0 Test
(c) CIFARIO Training
(d) CIFAR100 Training
Figure 4: Margin distributions of NNs with AR strength 4, 8, 16 on Training and Test sets of CIFAR10/100.
Concentration and reduced accuracy. In fig. 4, we can see that in both CIFARIO/100, the distributions of margins become more concentrated around zero as AR grows. The concentration moves the mode margin distribution towards zero and more examples slightly across the decision boundaries, where the margins are zero, which explains the reduced accuracy.
Concentration and reduced loss/GE gap. The concentration has different consequences on train- ing and test losses. Before describing the consequences, to directly relate the concentration to loss gap, we further introduce estimated probabilities of examples. This is because though we use ramp loss in theoretical analysis, in the experiments, we explore the behaviors of more practically used cross entropy loss. The loss maps one-to-one to estimated probability, but not to margin, though they both serve as a measure of confidence. Suppose p(x) is the output of the softmax function of dimension L (L is the number of target classes), and is the target label. The estimated probability of æ would be the y-th dimension of (p(æ)), i.e., (p(x)),, On the training sets, since the NNs optimized to perform well on the sets, only tiny fraction of them are classified wrongly. To concentrate the margin distribution more around zero, is to make almost all of predictions that are correct more diffident. Thus, a higher expected training loss ensues. On the test sets, the estimated probabilities of the target class concentrate more around middle values, resulting from lower confidence/margins in predictions made by NNs, as shown in fig. 5a (but the majority of values are still at the ends). Note that wrong predictions away from decision boundaries (with large negative margins) map to large loss values in the surrogate loss function. Thus, though NNs with larger AR strength have lower accuracy, they give more predictions whose estimated probabilities are at the middle (compared with NNs with smaller AR strength). These predictions, even if relatively more of them are wrong, maps to smaller loss values, as shown in fig. 5b, where we plot the histogram of loss values of test samples. In the end, it results in expected test losses that lower, increase in lower rate than the training

---

Under review as a conference paper at ICLR 2020
losses CI Imng- shown in fig. 3b. The reduced GE gap results from the increased training losses, and decreased or less increased test losses.
5.2.2 AR MAKES NNs SMOOTHE. PREDICTIONS W.R.T. INPUT PERTURABTIONS IN ALL. DIRECTIONS
60
Of
(a) Prob. Histogram
(b) Loss Histogram
(c) CIFARIO
(d) CIFARIO0
Figure 5: (b)(b) are histograms of estimated probabilities and losses respectively of the test set sample of NNs trained AR strength 4, 8, 16. We plot a subplot of a narrower range inside the plot of the full range show the histograms of examples that are around the middle values to show the change induced by AR that induces more middle valued confidence predictions. (c) are standard deviations of singular values of weight matrices of NNs at each layer trained on 11FAR10/100 with AR strength 4, 16. The AR strength 8 is dropped for clarity.
The observation in section 5.2.1 shows that AR make NNs just less confident by reducing the variance of predictions made and concentrate margins more around zero. In this section, we study the underlying factors of AR that make NNs become less confident.
To begin with, we show that the singular values of the weight matrix of each layer determine the perturbation in margins of samples induced by perturbations the instance space. Such a connection between singular values and the perturbation of outputs of single layer, i.e., BeLU(Wázz) has been discussed section 1.1. In the following, with lemma 4.1, we describe how the relatively more complex connection between margins and singular values of each weight matrix of layers of NNs holds. Observe that margins are obtained by applying a piece-wiss wiie linear mapping (c.f. the margin operator in definition 3) to the activation of the last layer of NN. It implies the perturbations in activation of the last layer induce changes in margins in a piece-wise linear way. Meanwhile, the perturbation in the activation of the last layer (induced by perturbation in the instance space) is determined by the weight matrix's 's singular values of each layer of NNs. More specifically, this explained as follows. Lemma 4.1 shows that the perturbation 81 induced by dæ, is given by Ej [I'm :hth 8xndt] Note that for each i, W.:: is matrix. By Cauchy interlacing law by row deletion (Chafai), the singular values W., the weight matrix of layer i, determine the singular values of W Thus, suppose 7 1, we have the change (measured in norm) induced by perturbation Σj=1 S.;
The singular values of W1 would determine the variance (of norms) of activation change induced by perturbations dæ, similarly as explained in section 1.1 except that the norm change now is obtained by summation of terms W dwddt|| (each of which is the exact form discussed in section 1.1) weighted by 1/(e, s,). Similarly, for the case where 2... L 1, the singular values of Wi determine the variance of changes induced by the perturbation of the previous layer (induced by perturbation from further previous layer recursively) of layer 1. Consequently, we choose to study these singular values.
We plot the standard deviation of singular values of each layer of ResNet56 trained on CIFAR10/100 earlier, shown in fig. 5c 5d. Overall, we can see that the standard deviation of singular values associated with layer of the NN trained with AR strength 4 is mostly larger than that of the NN with AR strength 16. The STD reduction in CIFAR 100 is relatively smaller than CI, since as observed in fig. 4b, the AR induced concentration effect of margin distributions is also relatively less obvious than that in fig. 4a. More quantitative analysis is given in appendix B.2, This leads us to our key results described in section 1.1.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Idan Attias, Arych Kontorovich, and Yishay Mansour. Improved generalization bounds for robust learning. Technical report, 2018. URL https::/aaxiiie wwwp:ii.orgg////111111 11100.1110000 02180 pdf.
Peter Bartlett, Dylan J. Foster, and Matus Telgarsky. Spertrally-normmaiized margin bounds for neural networks. In NIPS, pages 1-24, 2017.
Gary Bécigneul. On the effect of pooling on the geometry of representations. Technical report, mar 2017. URL http:////// rxi.. orgoorggaaaoo hatg//b///// /100711110000 06726.
Djalil Chafai. Singular Values Of Random Matrices. Technical report.
Daniel Cullina, Arjun Nitin Bhagoji, and Prateek Mittal. PAC-learning in the presence of evasion adversaries. In NIPS, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep Sparse Rectifier Neural Networks. In AISTATS, 2011.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and Harnessing Adversarial Examples. In ICLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity Mappings in Deep Residual Networks. ECCV, 2016.
Tiny ImageNet. Tiny imagenet, 2018. URL
weweeet.hereeee herokuapp.com com/.
Kui Jia, Shuai Li, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal Deep Neural Networks. Technical report, 2019. URL ptttp://arxiae hwtp:////g/aaa//a////// 05929.
Justin Khim and Po-Ling Loh. Adversarial Risk Bounds for Binary Classification via Function Transformation. Technical report, 2018. URL http://///// //ap:ivaaaiiv.////aa/aaaeeeeraaa htpgg/abs188000 .09519.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In NIPS, 2012.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. Adversarial Machine Learning at Scale. In ICLR, 2017.
Chen-yu Lee, Saining Xie, and Patrick Gallagher. Deeply-Supervised Nets. In AISTATS, 2015.
Yao Li, Martin Rengiang Min, Wenchao Yu, Cho-Jui Hsich, Thomas C. M. Lee, and Erik Kruus. Opti- mal Transport Classifier: Defending Against Adversarial Attacks by Regularized Deep Embedding. Technical report, 2018. URL http: / arxiv http://awwwww.w.a.//ssoo
811 07950.
Chunchuan Lyu, Kaizhu Huang, and Hai Ning Liang. A unified gradient regularization family for adversarial examples. In JCDM, 2015.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant Adversarial Attacks. ICLR, 2018.
Takeru Miyato, Shin Ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. PAMI, pages 1-16, 2018. ISSN 19393539. doi: 10. 10.11999AMM..211..855555 2.85888....
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PAC-Bayesian Approach to Spectrally-Noommiizee Margin Bounds for Neural Networks. ICLR, 2018a.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The Role Of parmmtttaniiinntioonnn In Generalization Of Neural Networks. In ICLR, 2018b.
F W Pfeiffer. Automatic differentiation in prose. In ICLR Workshop, 2017.
11

---

Under review as a conference paper at ICLR 2020
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander M Madry. Adversarially Robust Generalization Requires More Data. In NIPS, 2018.
Hanie Sedghi, Vineet Gupta, and Philip M. Long. The Singular Values of Convolutional Layers. In ICLR, may 2018.
Shalev-Shwartz and S. Ben-David. Understanding Machine Learning: From Theory to Algorithms. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014. ISBN 9781 1771007557135
Aman Sinha, Hongseok Namkoong, and John Duchi. Certifying Some Distributional Robustness with Principled Adversarial Training. In ICLR, 2018.
Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Robust Large Margin Deep Neural Networks. IEEE Transactions on Signal Processing, 65(16):4265-4288, aug 2017. ISSN 1053-587X.
doi: (0.1009/SS2201727000033
Dong Su, Huan Zhang, Hongge Chen, Jinfeng Yi, and C V Aug. Robustness the Cost of Accuracy
- A Comprehensive Study the Robustness of. In ECCV, 2018.
Christian Szegedy, W Zaremba, and Sutskever. Intriguing properties of neural networks. In ICLR, 2014.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Mądry, Robustness May Be Odds with Accuracy. In ICLR, 2019.
Yusuke Tsuzuu,, Issei Sato, and Masashi Sugiyama. Lipschitz-Margin Training: Scalable Certifica- tion of Perturbation Invariance for Deep Neural Networks. In NIPS, 2018.
Nakul Verma. Distance Preserving Embeddings for General n-Dimensional Manifolds. Journal of Machine Learning Research, 14:2415-2444 2013.
Huan Xu and Shie Mannor. Robustness and generalization. Machine Learning, 86(3):391-423, 2012. ISSN 08856125. doi: 10. 10.11077/100994--11-5556
Dong Yin and Peter Bartlett. Rademacher Complexity for Adversarially Robust. Technical report, 2018.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In BMVC, 2016.
Chiyuan Zhang, Samy Bengio. Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICLR, 2016.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric P. Xing, Laurent El Ghaoui, and Michael I. Jordan. Theoretically Principled Trade-off between Robustness and Accuracy. ICML, 2019.
Hongyi Zhang, Yann Dauphin, and Tengyu Ma. Fixup Initialization: Residual Learning Without Normalization. In ICLR, 2018.
12

---

Under review as a conference paper at ICLR 2020
APPENDICES
A FURTHER RELATED WORKS
Hard and soft adversarial robust regularization. We study the behaviors of NNs that are trained in the way that adversarial examples are required to be classified correctly. We note that the adversarial robustness required can also be built in NNs in a soft way by adding a penalty term in the risk function. Relevant works includes Lyu et al. (2015) and Miyato et al. (2018). This line of works is not our subject of investigation. They focus on increasing test performance instead of defense performance. The focus of our works is to study the behaviors that lead to standard performance degradation when a network is trained to has a reasonable defense ability to adversarial examples. For example, a adversarial examples generated by PGD methods (Madry et al., 2018) in fig. 11 is a defense ability that can serve as a baseline for a reasonable defense performance. is natural that in the setting where the requirement to defend against adversarial examples is dropped, the regularization can be weakened (added as penalty term) to only aim to improve the test performance of the network. In this case, no performance degradation would occur, but the defense performance is also poor.
50% accuracy
B FURTHER EMPIRICAL STUDIES ON ADVERSARIAL ROBUSTNESS
B. TECHNIQUE TO BUILD ADVERSARIAL ROBUSTNESS
To begin with, we describe the technique that we use to build AR into NNs. As mentioned in the caption of fig. we choose arguably the most well received technique, i.e., the adversarial training method (Madry et al., 2018). Specifically, we use -PG (Madry et al., 2018) untargeted attack adversary, which creates an adversarial example by performing projected gradient descent starting from a random perturbation around a natural example. Then, NNs are trained with adversarial examples. NNs with different AR strength are obtained by training them with increasingly stronger adversarial examples. The adversarial strength of adversarial examples is measured in the lx0 norm of the perturbation applied to examples. 1 -norm is rescaled to the range of 255 to present perturbations applied to different datasets in a comparable way; that means in fig. 1 3 4 6 11, AR is measured in this scale. We use 10 steps of size 2/255 and maximum of = [4/255, 8/255, 16/255] respectively for different defensive strength in experiments. For example, NN with AR strength 8 is a NN trained with adversarial examples generated by perturbations whose 7 norm are at most 8. Lastly, we note that although adversarial training could not precisely guarantee adversarial robustness radius of e, a larger Ino norm used in training would make NNs adversarially robust in a larger ball around examples. Thus, though the precise adversarial robustness radius is not known, we know that we are making NNs adversarially robust w.r.t. a larger e. Consequently, it enables us to study the influence of A-AR on NNs by studying NNs trained with increasing (o norm.
B.2 QUANTITATIVE ANALYSIS OF VARIANCE REDUCTION IN SINGULAR VALUES
Here, we provide more quantitative analysis on fig. 5c and fig. 5d, as noted previously in section 5.2.2.
Quantitatively, we can look at the accumulated standard deviation (STD) difference in all layers. We separate the layers into two group: the group that the STD (denoted o²)) of singular values of layer (of the NN trained) with AR strength 4 that is larger than that (denoted a.!5) of AR strength 16; and the group that is smaller. In CIFARIO, for the first group, the summation of the differenne/incrementt of STD of the two networks < of 5.0) is 4.7465, and the average is 0.1158. For the second groups, the summation (E, 0,66 of) is 0.4372, and the average is 0.0312. In CIFAR100, the summation of the first group is 3.7511, and the average is 0.09618; the summation of the second group is 0.4372, and the average is 0. 100. The quantitative comparison shows that the accumulated STD decrease in layers that have their singular value STDs decreased (comparing STD of the NN with AR strength 16 with STD of the NN with AR strength 4) is magnitude larger the accumulated STD increase in the layers that have their singular value STDs increased. The magnitude difference is significant since the STDs of singular values of most layers around 1.
13

---

Under review as a conference paper at ICLR 2020
B.3 DISCREPANCY BETWEEN TRENDS OF LOSS AND ERROR RATE GAPS IN LARGE CAPACITY NNs
In section 5.1, we have noted an inconsistent behaviors of CIFARIO, compared with that of CIFARI00 and Imay--mageNet: the error gap reduces for CIFAR 100 and Imag-em, but increases for CIFARIO. It might suggest that AR does not effectively regularize NNs in the case of CIFAR10 10. However, we show in this section that the abnormal behaviors of CIFARIO 10 are derived from the same margin concentration phenomenon observed in section 5.2.1 due to capacity difference, and compared with the error gaps, the GE/loss gaps are more faithfully representatives of the generalization ability of the NNs. Thus, the seemingly abnormal phenomenon corroborate, not contradict, the key results present in section 1.
Using CIFARIO and CIFARION as examples and evidence in the previous sections, we explain how the discrepancy emerges from AR's influence on margin distributions of the same network trained on tasks with different difficulties. Further evidence that the discrepancy arises from capacity difference would be shown at appendix B.4, where we run experiments to investigate GE gap of NNs with varied capacities on the same task/dataset.
1. On CIFARIO, the margin distribution of training sets not only concentrate more around zero, but also skews towards zero. As shown in the margin distribution on training sets of CIFARIO in fig. 4c, we find that the large eπтоr gap is caused by the high training accuracy that is achieved with a high concentration of training samples just slightly beyond the decision boundary. This phenomenon does not happen in CIFAR100. Comparing margin distribution on the test set in fig. 4a, the margin distribution on the training set in fig. 4c is highly skewed, i.e., asymmetrically distributed w.r.t. mean. While the margin distributions of CIFAR 100 training set in fig. 4d is clearly less skewed, and looks much more like a normal distribution, as that of the margin distribution the test set.
2. The high skewness results from the fact that the NN trained on CIFARIO is of large enough capacity to overfit the training set. As known, CIFARIO0 is a more difficult task w.r.t. CIFARIO with more classes and less training examples in each class. Thus, relatively, even the same ResNet56 network is used, the capacity of the network trained CIFARIO is larger than the one trained on CIFARI00. Recall that NNs have a remarkable ability to overfit training samples (Zhang et al., 2016). And note that though AR requires should be of the same class, since the ball is supposed only to include imperceptible perturbation to the example, few of the training samples are likely in the same ball. Thus, the ability to overfit the training set is not regularized by AR: if NNs can overfit all training samples, it can still overfit some more examples that are almost imperceptibly different. For CIFARIO. 10, since NNs have enough capacity, the NN simply overfits the training set.
ball around an example, the examples in the ball
3. However, as shown in the observed overfitting phenomenon in fig. 4c, the high training accuracy is made up of correct predictions with relatively lower confidence (compared with NNs with lower AR), which is bad and not characterized by the error rate; and the low test accuracy are made up of wrong predictions with relatively lower confidence as well (as explain in section 5.2.1), which is good, and not characterized by error rate as well. Thus, the error gap in this case does not characterize the generalization ability (measured in term of prediction confidence) of NNs well, while the GE gap more faithfully characterizes the generalization ability, and show that AR effectively regularizes NNs. In the end, AR still leads biased poorly performing solutions since the overfitting in training set does not prevent the test margin distribution concentrating more around zero, which leads to higher test errors of CIFARIO as shown in 3b. It further suggests that the damage AR done to the hypothesis space is not recovered by increasing capacity, however the ability of NNs to fit arbitrary labels is not hampered by AR.
B.4 FURTHER EVIDENCE OF REGULARIZATION EFFECTS ON NNS WITH VARIED CAPACITY
In previous sections, we observe AR consistently effectively regularizes NNs; meanwhile, we also observe that in the case where a NN has a large capacity, it can spuriously overfit training samples and lead to an increased error gap. In this section, we present additional results by applying AR to networks of controlled capacities. This is to ensure that our observations and analysis in previous sections exist not just at some singular points, but also in continuous area in the hypothesis space.
14

---

Under review as a conference paper at ICLR 2020
0668
645-
Spcciu Rore
Specter Boer Urcontc
karm
Brnil Dap
-125
-115
22.5
125
460
6.00
0.5
Drow
Traitnn
300
E2-
C2-
A1
051
14
360
12.5
15.6
650-
13
12.
215
11.3
co.
10
300
305 25.0
15
15.
12.5 15.
062
11.5
6.20
Seeched
22-
Stethe Hore
22.5
4.-
"pecrnll fores
125
6.3
C5-
164
[3-
E2-
E2.
34 255 360 12.5
25.0
93
C32
75
12.0
225
15.3
C3
50
75
300
225
25.0
80
15
U.S.
12.5
11.4
04
(a) Gap Curves
(b) Error Rate & Loss Curves
Figure 6: The four plots from upper left to lower bottom (in each subfigure) are NNs with increasingly smaller spectral complexity, where "Spectral Norm I" means for each weight matrix of the NN, its spectral norm is at most 1. (a) Plots of training/test loss gap (and error gap) against adversarial robustness strength. (b) Training/test losses and error rates against increased strength of adversarial robustness.
To control capacities of NNs quantitatively, we choose the measure based on spectral norm (Bartlett et al,, 2017; Neyshabur et al., 2018a). In spectral norm based capacity measure bound (Bartlett et al., 2017; Neyshabur et al., 2018a), the NN capacity is normally proportional to quantity called spectral complexity (SC), which is defined as follows.
Definition (Spectral Complexity). Spectral complexity SC(T) of a NN T is the multiplication of spectral norms of weight matrices of layers in a NN.
L
SC(T)
II Will
==1
where (W.)i=1... denotes weight matrices of layers of the NN.
To control SC, we apply the spectral normalization (SN) (Sedghi et al., 2018) on NNs. The technique renormalizes the spectral norms of the weight matrices of a NN to designated value after certain iterations. We carry out the normalization at the end of each epoch. We train ResNet56 with increasingly strong AR and with increasingly strong spectral normalization. The results are shown in fig. 6.
As can be seen, as the capacity of NNs decreases (from upper left to bottom right in each sub- figure), the error gap between training and test gradually changes from an increasing trend to decreasing trend, while the loss gap keeps a consistent decreasing trend. It suggests that the overfitting phenomenon is gradually prevented by another regularization techniques, i.e., the spectral normalization. As result, the regularization effect of AR starts to emerge even in the error gap, which previously manifests only in the loss gap. The other curves corroborate our previous observations and analysis as well.
B.5 FURTHER EVIDENCE ON THE SMOOTHING EFFECT OF ADVERSARIAL ROBUSTNESS
We quantitatively measure the smoothing effect around examples here by measuring the average maximal loss change/variation induced by the perturbation (of fixed infinity norm) applied on examples. We found that the loss variation decreases as networks become increasingly adversarially robust. Note that the loss of an example a proxy to the confidence of the example - is the logarithm of the estimated probability (a characterization of confidence) of the NN classifier.
For given maximal perturbation range characterized by the infinity norm, we generate adversarial examples within that norm for all test samples. For each example, the maximal loss variation/change of the adversarial example w.r.t. the natural example is computed for networks with different adversarial strength. To obtain statistical behaviors, compute the average and standard deviation of such maxima of all test samples. The results are shown in fig. 7. The exact data can be found in table 1.
15

---

Under review as a conference paper at ICLR 2020
CIFAR1O
CIFAR100
Averoge Loss Varistion
0.3
Average Loss Variation
0.4
0.3
0.2
0.2
0.1
0.0
-0.1
-0.
5
10
15
10 15
Adversarial Robustness
Adversarial Robustness
Figure 7: Average maximal loss variation induced by adversarial examples in networks with increasing adversarial robustness. The experiments are carried on CIFARI0/100 e represents the maximal perturbation can be applied on natural test examples to generate adversarial examples. It is measured in the infinity norm. The larger the c, the stronger the perturbation is. The error bars represent standard deviation.
We can see that the average loss variation decreases with adversarial robustness. The standard deviation decreases with network adversarial robustness well. The phenomenon that the standard deviation is comparably large with the mean might need some explanation. This is because different examples have different losses, thus the loss varies in relatively different regimens the more wrongly classified examples vary in a larger magnitude, and vice versa for more correctly classified examples. This phenomenon leads to the large standard deviation of the loss variation.
Table
Data of the smoothing effect of PGD ad advesariial training in fig. 7.
Defensive Strength
Dataset
Attack Strength
4
8
16
c 1
0.0273 + 0.0989
0.0236 + 0.0778
0.0215 = 0.0588
CIFARIO
== 2
0.0590 = 0.1637
0.0477 = 0.1337
0.0443 = 0.1102
c
0.1337 0.2494
0.1006 0.2137
0.0888 = 0.1816
1
0.0550 = 0.1276
0.0430 = 0.1043
0.0379 = 0.0886
CIFARI00
c
0.1072 0.2138
0.0839 = 0.1802
0.0732 = 0.1568
6 4
0.1868 + 0.2946
0.1563 = 0.2712
0.1355 = 0.2494
B.6 FURTHER EMPIRICAL STUDY ON USING FGSM IN ADVERSARIAL TRAINING TO BUILD ADVERSARIAL ROBUSTNESS
We also use FGSM (Goodfellow et al., 2015) in the adversarial training build adversarial robustness into NNs. The results are consistent with the results obtained using PGD. The experiments are carried on CIFAR10/100. We present key plots that support the results obtained in the main content here. All the setting are same with that described in appendix 1 of PGD, except that we replace PGD with FGSM.
Adversarial robustness reduces generalization gap and standard test performance. In sec- tion 5.1, we find that NNs with stronger adversarial robustness tend to have smaller toss/generalization gap between training and sets. Consistent phenomenon has been observed in networks adversari- ally trained with FGSM on CIFARI0/100 as shown in fig. 8a. Consistent standard test performance degradation has been observed adversarially trained with FGSM on CIFAR10/100 as well, as shown in fig. 8b. The exact data can be found in table 2.
Adversarial robustness concentrates examples around decision boundaries. In section 5.2.1, find that the distributions of margins become more concentrated around AR grows.
16

---

Under review as a conference paper at ICLR 2020
Loss Gap of CIFAR10
Loss Gap of CIFAR100
Test Error of CIFARIO
Test Error CIFAR100
0.43
FGSM
1.6
FGSM
17
FGSM
FGSM
43
0.42
1.5
16
1.4
15
42
14
1.3
0.40
1.2
13
40
12
0.39
1.1
1.0
38
10
10
15
5 Adverserial rial Robustness
10
15
15 Adversarial Robustness
5
10
Adversarial Robustness
Adversaria Robustness
(b)
Figure 8: Experiment results on CIFARI0/100. The network is ResNet-56 (He et al., 2016). The unit of x-axis the adversarial robustness (AR) strength of NNs, c.f. the beginning of section 5. (a) Plots of loss gap between training and test datasets v.s. AR strength. (b) Plots of error rates on training and test datasets v.s. AR strength.
The phenomenon has been observed consistently in networks adversarially trained with FGSM CIFAR 10/100, as shown in fig. 9. Phenomenon in fig. 5a 5b is also reproduced consistently in fig. 10a 10b. Please refer to section 5.2.1 for the analysis of the results. Here we mainly present counterparts of the results analyzed there.
513
Eass
2004
408
400
(a) CIFARIO Test
(b) CIFARI00 Test
(c) CIFARIO Training
(d) CIFARI00 Training
Figure 9: Margin distributions of NNs with AR strength 4, 8, 16 on Training and Test sets of CIFARIO/100.
Adversarial robustness reduces the standard deviation of singular values of weight matrices in the network. In section 5.2.2, we find that for NNs with stronger adversarial robustness, the standard deviation of singular values of weight matrices is smaller most layers. The phenomenon has been consistently observed in NNs trained with FGSM CIFAR 10/100, shown in fig. 10c and 10d. Please refer to section 1.1 and section 5.2.2 for the analysis of the results. Here we mainly present counterparts of the results analyzed there..
In conclusion, all key empirical results have been consistently observed in NNs trained with FGSM.
17

---

Under review as a conference paper at ICLR 2020
1000
A8-4
AJ.
200
46..14
(a) Prob. Histogram
(b) Loss Histogram
(c) CIFARIO
(d) CIFARIO0
Figure 10: (a)(b) are histograms of estimated probabilities and losses respectively of the test set sample of NNs trained AR strength 4, 8, 16. We plot subplot of narrower range inside the plot of the full range to show the histograms of examples that are around the middle values to show the change induced by AR that induces more middle valued confidence predictions. (c) standard deviations of singular values of weight matrices of NNs at each layer trained on CIFAR 10/100 with AR strength 4, 16. The AR strength 8 is dropped for clarity.
Table 2: Data of fig.
Defensive Strength
Dataset
8
16
Test Acc. 89.32
86.67
82.83
Trm Loss
0.038
0.086
0.252
CIFARIO
Test Loss
0.467
0.495
0.637
Loss
0.429
0.409
0.385
Test Acc.
62.01
59.78
56.30
Tm Loss
0.469
0.656
0.822
CIFARI00
Test Loss
1.776
1.723
1.797
A Loss
1.307
1.067
0.975
18

---

Under review as a conference paper at ICLR 2020
C PROOF OF THEOREM 4.1
C.I ALGORITHMIC ROBUSTNESS FRAMEWORK
In order characterize the bound the GE, we build on the algorithmic robustness framework (Xu and Mannor, 2012). We introduce the framework below.
Definition ((1 -robust, An algorithm is (K, x(()) robust, for K and c(-) : 2 + R. if Z can be partitioned into K disjoint sets, denoted by C = {Ck) K for all s; = (Zi yb) E Sm, (x, y) 2, Ck
k such that the following holds
vs; (z,, Ck. V2 = (z,y) C [((f v:) 1(f(x)) y)1 S c(Sm).
The gist of the definition is to constrain the variation of loss values on test examples w.r.t. those of training ones through local property of the algorithmically learned function f. Intuitively, if E S.. and 2 E 2 are "close" (e.g., in the same partition C²). their loss should also be close, due to the intrinsic constraint imposed by f-
For any algorithm that is robust, Xu & Mannor (Xu and Mannor, 2012) proves
Theorem C.1 (Xu Mannor (Xu and Mannor, 2012)). Ifa learning algorithm is (K, -roo-uusssttst and C is bounded, a.k.a. ((f(x), y) I M 4z Z. for any O, with probability at least 77 have
2K log (2) 2 log((/n)
GE(Js..) S <( + M
m
To control the first term, an approach is constrain the variation of the loss function. Covering number Shalev-Shwwrtz & Ben-David, (Shaley-Shwartz and Ben-Davia 2014), Chapter 27) provides a way to characterize the variation of the loss function, and conceptually realizes the actual number K of disjoint partitions.
For any regular k-dimensional manifold embedded in space equipped with a metric p. e.g., the image data embedded in I.2 (R2), the square integrable function space defined on R.2 it has covering number (X;; p. <) of (Cx/c)* (Verma, 2013), where Cx is constant that captures its "intrinsic" properties, and is the radius of the covering ball. When calculate the GE bound of NNs, would assume the data space a k-dimensional regular manifold that accepts a covering.
Adversarial robustness makes NNs (K, - algorithm, and is able to control the variation of loss values on test examples. Building on covering number and theorem C.I, we are able to prove theorem 4.1.
C.2 NEURAL NETWORKS
NN is a map that takes an input æ from the space X' and builds its output by recursively applying a linear map W; followed by a pointwise non-linearity g:
I; = g 1,;;
where indexes the times of recursion, which is denoted layer in the community. i i 1...., L.
To = x, and g denotes the activation function. which is restricted to Rectifier Linear Unit (ReLU) (Glorot al., 2011) !) pooling operator (Bécigneul, 2017) in this paper. To compactly summarize the operation T, we denote
Tr g(Woo(WLL- ... ((Wx))))
(6)
C.3 PROOF
Proof of lemma 4.1. By Theorem Sokolic et al. (2017), we have
[14()) - (xx')) = f J(x - t(xx x))dt(x - x')
where (w) denotes the Jacobian of Tı(1r) a.
19

---

Under review as a conference paper at ICLR 2020
By lemma 3.2 in Jia et al. (2019), when only have pooling layers and ReLU nonlinear layer in NNs, (x) is linear operator at a local region around æ. For terminology concerning regions, we follow the definitions in Jia et al. (2019). More specifically, we have
II W =
where W the linear mapping (matrix) induced by (az) at æ. It a matrix obtained by selectively setting certain rows of W; to zero. For the more concrete form of W refer to lemma 3.2 in Jia et al. (2019). In Jia et al. (2019),
noted as W.,. where q is a region where æ is in.
Suppose that from to x', the line segment æ - x'' passes through regions (g;))=1.....r The line segment illustrated fig. 2b as the boldest black line segment at the upper half of the figure. In the illustration, x x' passes through three regions, colored coded as gray, dark yellow, light blue respectively. The line segment is divided into three sub-segments. Suppose ((t) = x + f(æ' x). Then the three sub-segments can be represented by l(t) as ((1)) to (ee), ((s2) to ((22). and (ss) to l(ez) respectively, as noted on the line segment in the illustration. Originally, the range of the integration in eq. (7) is from 0 to 1, representing the integration the instance space. Now, since for each of these regions trespassed by the line segment, the Jacobian J (a) is linear operator, denoted as W the integration in eq. (7) from to I can be decomposed as a summation of integration on segments ((s1) to ((e1) etc. In each of these integration, the Jacobian
the line segment ((0) to [(1) in
J (x) is the multiplication of linear matrices W i.e., IT W Thus, eq. (7) can be written
Σ
W dt(x -aa)
where 8;; ey denotes the start and end of the segment [5,, ez] C [0, 1] of the segment [o, 1] that passes through the region qz-
In the cases that linear operator is applied on the feature map ((()) without any activation function, we can also obtain a similar conclusion. Actually, such cases are just degenerated cases of feature maps that have activation functions.
Corollary C.1. Given two elements x, x', and Ir(ar) = Wig W.-.... -g (()))) we have
n
W W dt(x 20) i=1
Σ
where symbols are defined similar in lemma 4.1.
Now, we are ready prove theorem C.I.
Proof of theorem C.I. Similar with the proof of theorem C.I,, partition space 2 into the e-cover of Z, which by assumption is a k-dimension manifold. Its covering number is upper bounded by C denoting K / and O; the ith covering ball. For how the covering ball is obtained from the e-cover, refer to theorem in Xu and Mannor (2012). We study the constraintrregularizatiion that adversarial robustness imposes on the variation of the loss function. Since we only have adversarial robustness, the radius of the covering balls at most this why we use the same symbol. Beyond e, adversarial robustness does not give information the possible variation anymore. Let T' denotes the NN without the last layer.
First, we analyze the risk change in covering ball C.. The analysis is divided into two cases: all training samples in C₁ are classified correctly; 2) all training samples in C, are classified wrong. Note that other cases exist, for that the radius of C2 is restricted to be and work e-adversarial robust classifiers. guarantees that all samples in a ball are classified as the same class. Thus, either all training samples all classified correctly, or wrongly.
20

---

Under review as a conference paper at ICLR 2020
We first study case 1). Given any example (x. v) C₁ let arg m w.'xx, Its ramp loss is
L (2z, v) max - (w,, wgy w T'x).
Note that within C2. (wy wg) T 0, thus (ax y) is mostly 1, and would not reach the region where in definition 4. Let uxx) = (w, wg " x, and u minveEC, u(x). We have
by (2e S max " Mmin S ma
Minin
where Umin denotes the smallest margin among all partitions.
The inequality above shows adversarial robustness requires that T' x should vary slowly enough, so that in the worst case, the loss variation within the adversarial radius should satisfy the above inequality. The observation leads to the constraint on the loss difference e(-) defined earlier in definition in the following.
Given any training example 2 == (x. Ct and any element x' == (x', v/ E C₁. where C2 is the covering ball that covers æ, have
1 y) L (a' 3/))
u(x)
max max u(xx) H
= =/max(0,
Umin }.
S <max(0,
7
Now we relate the margin to the margin the instance space.
Given = (æ, 3) 2, and æ', of which æ' is the closest points (measured in Euclidean norm) on the decision boundary, we can derive the inequality below.
u(x) u(x) u(x')
(9)
J x t(x x²) d')( x¹)
(10)
L-1 1 (wy - wg))
1(x 2o' dd(x x')
1 (wy w
L
1(2 x²) dt((c x¹)
(11)
1=1
B2
(wy, wi )
W dt(x x²)
(12)
j=1
L
σ mmnn | - x 0'112 f(' dt
min v.VEA.vv
||w, Wg ll2
(13)
==1
min V.jEV.v7
||,, wg l12
o || x æ'112
where J (x) denotes the Jacobian of 1((x) at æ. Equation (10) can be reached by Thcorem 3 in
Sokolic et (2017). Equation (11) can be reached because (wy - wyg) w (yx==((=='))) (a)::: - xx) the actually classification score u(x), u(=') difference between æ, x', and by assumptions 4. they positive throughout. Equation (12) is reached due to corollary C.I - in this case, the matrix W; in corollary C.1 is of rank one.
To arrive from eq. (12) to eq. (13), we observe that x' is the closest point to æ on the decision w.)T") If the difference x' satisfies x x' Y (T"). we can always remove the part in the A AT")",, which would identify a point that is closer æ, but still the decision boundary, which would be contradiction. Then if x: xf is
boundary. Being the closest means x' 1 N((w,,
21

---

Under review as a conference paper at ICLR 2020
orthogonal to the null space, informal reasoning above formally in the following.
bound the norm using the least singular values. We develop the
Similarly in Lemma 3.4 in Jia et al. (2019), by Cauchy interlacing law by row deletion, assuming 1 WI L-1 w (A' denotes the null space; the math statement means æ is orthogonal the null space of J [x)),, we have
II W x:
II
o | |211|
(14)
where iiin is the smallest singular value of W. Then conclusion holds as well multiplication of matrices I1 L-1 W since the multiplication of matrices are also matrix.
Notice that in each integral in eq. (12), we are integrating over constant. Thus, have it equates to
L-1 / (x - x')
Σ e - 87) (wy w
T
Now we show that in each operand, æ - x' 1 (V((w, - wgy II= W Denote g) as
N(w, wy) I w Suppose that it does not hold. Then we can decompose xx into two components A 2 where A 1. TTy A / Taj. We can find new point x" A that
on the boundary. However, in this case
x x" 1 A1ll2 I ||Ai|12 + 11A21le | 1
Recall that x' is the closest point to the decision boundary. This leads to contradiction. Repeat this argument for all j I,..., n, then we have æ - x be orthogonal to all (VT,,) Thus, by the inequality eq. (14) earlier, we can arrive at eq. (13) notice that wy wg is matrix with one column, thus also satisfies the above reasoning.
Through the above inequality, can transfer the margin to margin in the instance space. Let v(x)) be the shortest distance in ||| |l2 norm from an element E X' to the decision boundary. For a covering ball C, let o be min 2(2x). Let Umin the smallest umin among covering balls C, that contain at least training example. We have that
Umin
min
1w,, wgll
ll
II ==1
σ min Umin
Consequently, can obtain upper bound of eq. (8) parameterized Umin as follows
max (0, - Umin < mmx (0, min [w,,
gf min Mmin
Notice that only because eo-adversarial robustness, we can guarantee that Umin is non-zero, thus the bound is influenced by AR.
Then, we study case 2), in which all training samples z e C2 are classified wrong. In this case, for all E Ct,, the given by ÿ = arg max; w T" x in the margin operator is the same, for that I is the wrongly classified class. Its ramp loss is
(x, v) = max (w,, TT''
Note that in the case 1), it is the y that stays fixed, while ý may differ from example to example; while in the case 2), it is the ÿ stays fixed, while y may differ.
Similarly, within C₁ as required by adversarial robustness, (w, -w:) TT'x S 0, thus we always
have 1 - 1 7, (w. w,,) T 2 1, implying
( v)
22

---

Under review as a conference paper at ICLR 2020
Thus, V2 (xx,), (e'E))C
1 (x, (2',8'))
(15)
Since only these two cases are possible, by eq. (8) and eq. (15), we have V2, E C₁
(2, v) (.', ./0 max(0,
Umin }.
(16)
The rest follows the standard proof algorithmic robust framework.
Let N, be the set of index of points of examples that fall into Cj. Note that (IN is an IDD multimonial random variable with parameters m and (Iu(C)))iis.... Then
RR( T) Rm (1 T))
K
E Ez ((Tx
((Ta;;
K
[i
1
m
E EZ~z ((X,
ITT az y:)l
m
m
K
K ((TX, Y) [N
( (),)
max ((Txx)) 2) ((Txx U3)) zEC,
(17)
JEN;
K
N
+ max =E
M(TT, v)
(C.))l
(18)
Remember that = (ar, y)-
By eq. (16) we have eq. (17) is equal or less than max (0, Mmin )}. By Breteganolle- Huber-Carol inequality, eq. (18) is less equal to v 1og(2)2t-1 The proof is finished.
2 log(1/n)
7 m
23

---

Under review as a conference paper at ICLR 2020
Rostel CICO
19
Adverar
Figure 11: The plot of accuracy on adversarial examples v.s. adversarial defense strength built in NNs. The dotted line of which the intersections are marked by stars are adversarial accuracy in Madry al. (2018) (CIFARIO), in Li et al. (2018) (Tiny ImageNet) under similar adversarial attack strength.
D IMPLEMENTATION DETAILS
We summarize the details of the experiments in this section. The experiments are run with PyTorch (Pfeiffer, 2017).
D.1 DATASETS
CIFAR10/100. Each CIFAR dataset consists of 50, 000 training data and 10, 000 test data. CIFAR-10 and CIFAR-100 have 10 and 100 classes respectively. Our data augmentation follows the standard manner in Lee et al. (2015): during training, we zero-pad pixels along each image side, and sample 32 x 32 region cropped from the padded image its horizontal flip; during testing, we use the original non- non-padded image.
Tiny-ImageNet. Tiny-ImaeNNet subset of ImageNet dataset, which contains 200 classes rather than 1, 000 classes. Each class has 500 training images and 50 validation images. Images in the Imny--mageNet dataset are 64 x 64 pixels, as opposed to 256 x 256 the full ImageNet set. The data augmentation is straightforward: an input image is 56 x 56 randomly cropped from resized image using the scale, aspect ratio augmentation as well as scale jittering. single 56 x 56 cropped image is used for testing.
D.2 EXPERIMENTS IN SECTION 5.1
CIFARI0/100 Models and Training. The models for CIFAR10/100 are the same as the ones in appendix B.4, except that we do use spectral normalization anymore. CIFAR100 has 100 output neurons instead of 10.
Tiny-ImageNet Model. For Tiny ImageNet dataset, we use 50-layered wide residual networks with 4 groups of residual layers and [3. 4, 6, bottleneck residual units for each group respectively. The 3 x 3 filter of the bottleneck residual units have 164 x k. 128 x k. 256 x k. 512 x k] feature maps with the widen factor & 2 as mentioned in Zagoruyko and Komodakis (2016). We replace the first x convolution layer with x filters with stride and padding 1. The max pooling layer after the first convolutional layer is also removed to fit the 56 x 56 input size. Batch normalization layers are retained for this dataset. The weights of convolution layers for Tiny ImageNet are initialized with Xavier uniform (Glorot and Bengio, 2010). Again, all dropout layers omitted.
-mageNe Training. The experiments on the ImageNe dataset are based on a mini-batch batch size of 256 for 90 epochs. The initial learning rate is set be 0.1 and decayed at 10 30 and 60 epochs respectively. All experiments are trained on the training set with stochastic gradient descent with the momentum of 0.9.
24

---

Under review as a conference paper at ICLR 2020
Table 3: Raw data of CIFARIO dataset for plots in fig. 3 and fig. 11.
Defensive Strength
Method
8
16
Trn Acc.
99.51
98.45
95.97
Test Acc.
88.86
87.51
84.89
AAc.
10.65
10.94
11.08
ResNet-56
Trm Loss
0.014
0.043
0.105
+ Adv Tm
Test Loss
0.683
0.649
0.650
Loss
0.669
0.606
0.545
PGD
65.92
65.21
72.16
Trn Acc.
99.95
99.62
98.42
Test Acc.
89.20
87.09
85.02
Acc.
10.75
12.53
13.40
ResNet-1 10
Trn Loss
0.002
0.010
0.044
+ Adv Tm
Test Loss
0.825
0.813
0.729
Loss 0.823
0.803
0,685
PGD
58.02
66.94
72.40
Table 4: Raw data of CIFARI00 dataset for the plot in fig. and fig. 11.
Defensive Strength
Method
4
16
Trn Acc.
88.73
86.97
82.17
Test Acc.
61.31
60.87
59.43
AAcc.
27.42
26.10
22.74
ResNet-56
I'm Loss
0.357
0.413
0.570
Adv Tr
Test Loss
2.063
2.106
1.978
ALoss
1.706
1.693
1.408
PGD
30.52
40.99
48.81
Trn Acc.
96.91
94.55
90.90
Test Ace.
61.48
61.26
59.56
AAcc
35.43
33.29
31.34
ResNet-110
Irn Loss
0.098
0.171
(1.278
Adv Tr
Test Loss
2.645
2.413
2.323
ALoss
2.547
2.241
2.045
PGD
33.33
42.08
50.99
Results. The data for fig. are given in table 3, and 5. More specifically, the data on CIFARIO given in table 3. The result on CIFAR100 are given in table 4. The result on Tiny-ImageNet are given table 5.
Adversarial Robustness Attack Method. The adversarial accuracy is evaluated against 4 PGGD (Madry al., 2018) untargeted attack adversary, which one of the strongest white-box attack methods. When considering adversarial attack, they usually train and evaluate against the same perturbation. And for our tasks, we only use the moderate adversaries that generated by 10 iterations with steps of size and maximum of 8. When evaluating adversarial robustness, we only consider clean examples classified correctly originally, and calculate the accuracy of the adversarial examples generated from them that are still correctly classified. The adversarial accuracy is given in table 4 5, the row named "PGD", and plotted in fig. 11.
D.3 EXPERIMENTS IN APPENDIX B.4
Models. We use ResNet-typp networks (Zhang et al., 2018). Given that we need to isolate factors that influence spectral complexity, use ResNet without additional batch normalization (BN) layers. To train ResNet without BN, we rely on the fixup initialization proposed in Zhang et al. (2018). The scalar layers in Zhang et al. (2018) also omitted, since it changes spectral norms of layers.
25

---

Under review as a conference paper at ICLR 2020
Table Raw data of Tiny-ImageNet dataset for the plot fig. and fig.
Defensive Strength
Method
0
4
16
Trn Acc.
79.12
73.71
66.17
60.73
Test Acc.
63.43
62.09
61.09
57.36
A Acc.
15.69
11.62
5.08
3.37
Wide ResNet Adv Trn
Loss
0.874
1.080
1.384
1.641
Test Loss
1.561
1.637
1.689
1.806
Aoss
0.687
0.557
0.305
0.165
PGD
0.00
32.26
41.20
53.12
Table 6: Raw data for fig. 6. SP denotes spectral norm.
Strength
Defensive Strength
Spectral Normalization
4
8
16
Trm Acc.
96.91
94.38
90.58
Test Acc.
90.47
88.87
85.82
AAcc. 6.44
5.51
4.76
SP
Imm Loss
0.092
0.159
0.265
Test Loss
0.316
0.353
0.432
ALsss
0.224
0.194
0. 0.168
PGD
57.93
69.98
75.98
T'm Acc.
99.65
98.51
95.94
Test Acc.
90.02
88.07
85.43
AAcc.
9.63
10.44
10.51
Loss
0.010
0.039
0.107
Test Loss
0.606
0.580
0.577
ALoss
0.596
0.541
0.470
PGD
56.83
67.73
73.41
Tm Acc.
99.57
98.33
95.96
Test Acc.
89.53
88.09
85.32
AAcc.
10.04
10.24
10.64
SP
Tra Loss
0.012
0.045
0.105
Test Loss
0.649
0.602
0.611
ALoss
0.638
0.557
0.506
PGD
54.91
65.96
72.37
T' Acc.
99.51
98.45
95.97
Test Acc.
88.86
87.51
84.89
Acc. 10.65
10.94
11.08
SP Uncontrolled
I'n Loss
0.014
0.043
0.105
Test Loss
0.683
0.649
0.650
ALsss
0.669
0.606
0.545
PGD
65.92
65.24
72.16
Dropout layers omitted well. Following Sedghi et al. (2018), we clip the spectral norm every epoch rather than every iteration.
Training. The experiments CIFAR1O datasets based mini-batch size of 256 for 200 epochs. The learning rate starts at 0.05, and is divided by 10 at 100 and 150 epochs respectively. All experiments are trained training with stochastic gradient descent based on the momentum of 0.9.
Results. The data for fig. 6 are given in table 6.
26


Reviews:
[{"review_id": "r1lOwA8TFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "===== Summary ===== \nThe paper presents new theory to develop understanding about why adversarially robust neural networks show lower test performance compared to their standard counterparts despite being more robust to perturbations in the data. The main hypothesis is that the degradation in performance in adversarially robust networks is due to many samples being concentrated around the decision boundary, which makes the network less confident about its decisions. The paper studies this hypothesis by deriving a bound on the generalization error based on the margin between the samples in the training set and the decision boundary. The paper then presents empirical demonstrations that aim to illustrate the theoretical findings. \n\nContributions:\n1. Derive a generalization bound on the performance of adversarially robust networks  that depends on the margin between training examples and the decision boundary.\n2. Provide empirical evaluations that aim to illustrate the theoretical results.\n\n===== Review =====\nThe problem that the paper addresses is very significant to the robust optimization field and the study of adversarial robustness in neural networks. Thus, I believe that the results could represent a significant contribution. However, due to the way that the information is presented, it is difficult to validate the correctness of the theory and the insights from the paper. Consequently, I consider that the paper should be rejected.\n\n===== Detailed Comments =====\n- First, and foremost, the paper should be proof-read for English grammar and writing style. In its current form, it is difficult to follow the main argument of many of the paragraphs. This is exceedingly important because the main subject of the paper is already difficult to digest as is.\n\n- In the related work section, a lot of previous work is referenced without any context about what the contribution of each of those papers is. Each of these papers should be mentioned along with their corresponding contributions. Otherwise, it is difficult to frame the paper within the context of the current literature. Moreover, not providing context makes it difficult to determine which parts of the paper are original and which are the result from previous work. \n\n- The first item in Section 1.1 is difficult to follow because there are many gaps in logic that are left to the reader to fill in. It is reasonable to expect the reader to fill in some of the details, but since the sole purpose of this section is to build intuition about what is about to be presented in the paper, then each step in the explanation should follow as seamlessly as possible.\n\n- The motivation for studying the margins between the training set and the decision boundary is not clear until Section 5.2.1 where it is mentioned that this is a widely used tool in learning theory. This should be presented earlier since not every reader will be completely familiar with learning theory. Moreover, it would also be useful to provide some intuition about how margins relate to the confidence of a classifier.\n\n- After presenting the main result of the paper — Theorem 4.1 — very little intuition is provided about each of the terms in the bounds. It would greatly increase the clarity of the result if each term was explained intuitively, so that the readers can gain the main insight of the paper before reading the proofs. This would also help motivate better the empirical evaluations in the following section. \n\n- The plots in FIgure 3 and Figure 4 are very difficult to understand and unintuitive. For Figure 3, the main reason the plots are difficult to read is because different colors are used for different networks. For Figure 4, it is difficult to understand what is happening because the tallest curves are plotted on top of the shortest ones. Hence, the information from the other curves is mostly lost. This seems like a very minor comment, but this graphs are not very complicated, so they should be easy to understand; yet it takes several minutes to take in what is happening in the graph.\n\n- For the proof of Lemma 4.1it is not clear how to get from Equation (7) to the main result of the lemma even after referencing Theorem 3 of Sokolic et. al. (2017) and Jia et. al. (2019). This could also be my lack of expertise on the topic; however, since the proof is already in the appendix, the proof should not be sparse in the amount of detail that it provides. \n\n===== References =====\nJure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Robust Large Margin Deep Neural Networks. IEEE Transactions on Signal Processing, 65(16):4265–4280, aug 2017. ISSN 1053-587X. doi: 10.1109/TSP.2017.2708039.\n\nKui Jia, Shuai Li, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal Deep Neural Networks. Technical report, 2019. URL http://arxiv.org/abs/1905.05929.\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeOtaIFKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper focuses on analyzing the regularization of adversarial robustness (AR) on neural networks (NNs). They establish a generalization error (GE) bound characterizing the regularization of AR, and identify two quantities: margin distributions and singular values of NNs' weight matrices. With empirical studies, they show that AR is achieved by regularizing NNs towards less confident solutions and making feature space changes smoother uniformly in all directions, which prevents sudden change wrt perturbations but leads to performance degradation.\n\nThe paper is well written with theoretically motivated experiments and detailed analysis. I'd suggest accepting the paper.\n\nWith proposed GE bound connecting 'margin' with AR radius via 'singular values of weight matrices of NNs', they present 3 key results with empirical experiments on CIFAR10/100 and Tiny-ImageNet.\n1) AR reduces the variance of outputs at most layers given perturbations.\n2) Empirically examples are concentrates around decision boundaries.\n3) The samples concentration around decision boundaries smooths sudden perturbation change, but also degrades model performance.\nThe paper only shed light on their conjecture that the performance degradation comes from the indistinguishable changes induced by adversarial noise and by inter-class difference. It'd nicer to further analyze on how to obtain AR without sacrificing performance on natural examples.\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1e_-c4kcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to explain a phenomenon that the increasing robustness for adversarial examples might lead to performance degradation on natural examples. The authors analyzed it from the following aspects: \n\n1)\tAdversarial robustness reduces the variance of output at most layers in terms of reducing the standard deviation (STD) of singular values associated with a layer of NN. The authors provide the experiment to show that the stronger robustness for adversarial examples leads to smaller STD of singular values of parameter of layers. \n\n2)\tThe reduced norm variance can cause the margins concentrated around zeros. Specifically， the authors provide the relevant lemma to show the relationship between margin and singular vectors. Moreover，the authors also conducted the experiment to show that stronger robustness over adversarial examples can lead to zero concentration of margin. The authors think a small margin might cause shrinking the hypothesis space which might cause low generalization.\n\n\n3)\tThe authors have derived a bound of generalization which is related to Instance-space Margin，and minimum singular values. This proved that strong robustness on adversarial examples might reduce the generalization. \n\n\nIn general, this paper seems technically sound. It is good that to some theoretic analysis can be derived in particular a bound of generalization can be given. Moreover, some experiments were made trying to verify the theory. Despite interesting, there are still some major concerns regarding the paper：\n\n1.\tThe authors mentioned that “it is widely observed that such methods would lead to standard performance degradation, i.e., the degradation on natural examples.” I am afraid that this may not be always the case. In some other related work (see C1), adversarial training can perhaps achieve the performance lift on both the adversarial examples and natural examples (if a trade-off parameter can be well specified). Some clarification or further discussion may be necessary regarding this point.\n\n2.\tOnly PGD was used as the adversarial perturbations in the experimental part of this paper. It would be more convincing if the authors could perform analysis on different adversarial training methods, e.g. FGSM and even the unified gradient perturbations developed in C2. There are also more adversarial attacks in the literature.\n\n3.\tThe authors stated that the sample concentration around decision boundaries smoothness sudden changes, which was verified by the accuracy degradation. This is ok but it would be better to visualize or quantifying directly whether this can indeed make the boundary smoother. One possible way is to plot the confidence when moving the points near the decision boundary to check whether the confidence changes smoothly. \n\n4.\tFinally, this paper seems to be written in a hurry. The paper may need substantial improvement on the English writing. There are still quite a few typos and grammar errors in the paper; this makes the paper less attractive though it contains some theoretic merits.\n\nIn summary, it is good that a theoretical bound can be derived from the paper, but this paper's quality may need more enhancement particularly on its writing and experimental parts. \n\nC1:  Virtual Adversarial Training: a Regularization Method for Supervised and Semi-Supervised, Learning\" http://arxiv.org/abs/1704.03976\n\nC2: A Unified Gradient Regularization Family for Adversarial Examples C. Lyu, K. Huang, and H. Liang, ICDM 2015.\n\n\n============\nI have carefully read the response as  well as the revised paper. To me, the response has addressed those of my major concerns. I an inclined to increase my rating and would suggest to weak accept this paper.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates why adversarial training can sometimes degrade model performance on clean input examples. 

The reviewers agreed that the paper provides valuable insights into how adversarial training affects the distribution of activations. On the other hand, the reviewers raised concerns about the experimental setup as well as the clarity of the writing and felt that the presentation could be improved. 

Overall, I think this paper explores a very interesting direction and such papers are valuable to the community. It's a borderline paper currently but I think it could turn into a great paper with another round of revision. I encourage the authors to revise the draft and resubmit to a different venue. 

 
Decision: Reject