Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspaces
Authors: Odelia Melamed
Gilad Yehudai
Gal Vardi
Venue: NeurIPS 2023 poster (2023)
Keywords: Adversarial Examples
Robustness
Neural Networks
Classification
TL;DR: We prove that adversarial examples exist on trained 2-layer ReLU networks, when the data lies on a low dimensional linear subspace, and suggest two ways to mitigate it.

Abstract: Despite a great deal of research, it is still not well-understood why trained neural networks are highly vulnerable to adversarial examples.
In this work we focus on two-layer neural networks trained using data which lie on a low dimensional linear subspace.
We show that standard gradient methods lead to non-robust neural networks, namely, networks which have large gradients in directions orthogonal to the data subspace, and are susceptible to small adversarial $L_2$-perturbations in these directions.
Moreover, we show that decreasing the initialization scale of the training algorithm, or adding $L_2$ regularization, can make the trained network more robust to adversarial perturbations orthogonal to the data.

Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspaces
Odelia Melamed
Gilad Yehudait
Gal Vardi
Abstract
Despite great deal of research, it is still not well-understood why trained neural networks are highly vulnerable adversarial examples. In this work we focus on two-layer neural networks trained using data which lie on low dimensional linear subspace. We show that standard gradient methods lead non-robust neural networks, namely, networks which have large gradients in directions orthogonal the data subspace, and are susceptible small adversarial Lz-perturbations these directions. Moreover, we show that decreasing the initialization scale of the training algorithm, or adding T.2 regularization, can make the trained network more robust to adversarial perturbations orthogonal to the data.
Introduction
Neural networks are observed to be susceptible to adversarial perturbations [Szegedy et al., 2013]. often imperceptible by humans. Many works have shown attacks, where adding a very small perturbation to the input may change the prediction of the network [Carlini and Wagner, 2017, Papernot et al., 2017, Athalye et al., 2018]. Other works have shown defense mechanisms called adversarial training [Papernot et al., 2016, Madry et al., 2017, Wong and Kolter, 2018). Despite great deal of research, it is still not well-understood why neural-network training methods tend towards such non-robust solutions.
Several recent works have given theoretical explanation for the existence of adversarial perturbations under different settings. One line of work [Daniely and Shacham, 2020, Bubeck et al., 2021a, Bartlett al., 2021, Montanari and Wu, 2022] have shown that random networks are susceptible adversarial perturbations. These results might explain why neural networks are non-robust at initialization, but they do not explain why trained neural networks are non-robust. Recently, Vardi et al. [2022] showed that for data which is nearly orthogonal, after training for infinitely many iterations, the implicit bias of neural networks towards margin maximization leads non-robust solutions. Despite these works, it is still unclear why trained neural networks tend to be non-robust to adversarial perturbations, and specifically what are the assumptions on the input data which leads to non-robustness.
One common belief about "real-life" datasets, that they approximately lie on a low dimensional "data-manil" manifold" in high dimensional space. In this setting, the existence of perturbations orthogonal to the data-mane manifold that change the network's 's predictions is especially undesired, since such pertur- bations do not make the input closer to data points from other classes. Indeed, such a perturbation only increases the distance between the input and all "real-life" examples. Shamir et al. [2021] have demonstrated empirically that under such data-manifold assumption, the decision boundary of trained classifier clings to the data-manifole in way that even very small perturbations orthogonal to the manifold can change the prediction.
Weizmann Institute of Science, Israel, odelia. me weeedweei ac il Weizmann Institute of Science, Israel, gi lad. yehuda eee zmann ac. i1 I TTTIChicago and the Hehrew University Jerusalem, ga va rdi @t c.edu.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this paper we focus on data which lies on a low dimensional "data-mmnifoldd.. Specifically, we assume that the data lies on a linear subspace P C Rd of dimension d l for some > O. We study adversarial perturbations in the direction of p1, i.e. orthogonal to the data subspace. We show that the gradient projected on P- is large, and in addition there exist a universal adversarial perturbation in a direction orthogonal to P. Namely, the same small adversarial perturbation applies to many inputs. The norm of the gradient depends on the term ? d' while the perturbation size depends on the term $ i.c. low dimensional subspace implies reduced adversarial robustness. Finally, we also study how changing the initialization scale or adding L2 regularization affects robustness. We show that in our setting, decreasing the initialization scale, adding sufficiently large regularization term, can make the network significantly more robust. We also demonstrate empirically the effects of the initialization scale and regularization the decision boundary. Our experiments suggest that these effects might extend to deeper networks.
2 Related Works
Despite extensive research, the reasons for the abundance of adversarial examples in trained neural networks are still not well understood [Goodfellow et al., 2014b, Fawzi et al., 2018, Shafahi et al., 2018, Schmidt et al., 2018, Khoury and Hadfield-Menell, 2018, Bubeck et al., 2019, Allen-Zhu and
Li, 2020, Wang et al., 2020, Shah et al., 2020, Shamir et al., 2021, Ge et al., 2021, Wang et al., 2022, Dohmatob and Bietti, 2022]. Below discuss several prior works on this question.
In string of works, it was shown that small adversarial perturbations can be found for any fixed input in certain ReLU networks with random weights (drawn from the Gaussian distribution). Building on Shamir et al. [2019], it was shown in Daniely and Shacham [2020] that small adversarial L2- perturbations can be found in random ReLU networks where each layer has vanishing width relative to the previous layer. Bubeck et al. [2021a] extended this result to two-layer neural networks without the vanishing width assumption, and Bartlett et al. [2021] extended it to a large family of ReLU networks of constant depth. Finally, Montanari and Wu [2022] provided a similar result, but with weaker assumptions on the network width and activation functions. These works aim to explain the abundance of ad advesaarial examples in neural networks, since they imply that adversarial examples are common in random networks, and in particular in random initializations of gradient-basee methods. However, trained networks are clearly not random, and properties that hold in random networks may not hold in trained networks. Our results also involve analysis of the random initialization, but consider the projection of the weights onto the linear subspace orthogonal to the data, and study its implications on the perturbation size required for flipping the output's sign in trained networks.
In Bubeck et al. [2021b] and Bubeck and Sellke [2021], the authors proved under certain assumptions, that overparameterizatioo is necessary if one wants to interpolate training data using a neural network with small Lipschitz constant. Namely, neural networks with a small number of parameters are not expressive enough to interpolate the training data while having a small Lipschitz constant. These results suggest that overparameterization might be necessary for robustness.
Vardi et al. [2022] considered a setting where the training dataset S consists of points, and proved that every network to which gradient flow might converge is non-robust w.r.t. S. Namely, building on known properties of the implicit bias of gradient flow when training two-layer ReLU networks w.r.t. the logistic loss, they proved that for every two-layer ReLU network to which gradient flow might converge as the time / tends to infinity, and every point x; from S, it is possible to flip the output's 'ss sign with a small perturbation. We note that in Vardi et al. [2022] there is a strict limit on the number of training samples and their correlations, as well as the training duration. Here, we have no assumptions on the number of data points and their structure, besides lying on a low-dimensional subspace. Also, in Vardi et al. [2022] the adversarial perturbations are shown to exist only for samples in the training set, while here we show existence of adversarial perturbation for any sample which lies on the low-dimensional manifold.
is widely common to assume that "real-life data" (such as images, videos, text, etc.) lie roughly within some underlying www-iimensional data manifold. This common belief started many successful research fields such as GAN [Goodfellow et al., 2014a], VAE [Kingma and Welling, 2013], and diffusion (Sohl-Dickstein et al., 2015]. In Fawzi et al. [2018] the authors consider setting where the high dimensional input data is generated from a low-dimensional latent space. They theoretically analyze the existence of adversarial perturbations on the manifold generated from the latent space,
2

---

although they do not bound the norm of these perturbations. Previous works analyzed adversarial perturbations orthogonal to the data manifold. For example, Khoury and Hadfield-Menell [2018] considering several geometrical properties of adversarial perturbation and adversarial training for low dimensional data manifolds. Tanay and Griffin [2016] analyzed theoretically such perturbations for linear networks, and Stutz et al. [2019] gave empirical analysis for non-linear models. Moreover, several experimental defence methods against adversarial examples were obtained, using projection of it onto the data manifold to eliminate the component orthogonal to the data (see, e.g., Jalal et al. [2017], Meng and Chen [2017]. Samangouei et [2018]).
Shamir et al. [2021] showed empirically on both synthetic and realistic datasets that the decision boundary of classifiers clings onto the data manifold, causing very close off off-manifold adversarial examples. Our paper continues this direction, and provides theoretical guarantees for off- manifold perturbations on trained two-layer ReL.U networks, in the special case where the manifold is linear subspace.
3 Setting
Notations. We denote by U(A) the uniform distribution over a set A. The multivariate normal distribution with mean μ and covariance E is denoted by [M, 2), and the univariate normal distribution with mean a and variance o² is denoted by ((a, )²). The set of integers {1..., m) is denoted by [m]. For vector i and ending with + j. For vector and linear subspace P we denote by p! the subspace orthogonal to P, and by p (rr) the projection of x on p.. We denote by 0 the zero vector. We use Ta for the identity matrix of size d.
IR", we define Uei+j to be the j + coordinates of starting from
3.1 Architecture and Training
In this paper we consider a two-layer fully-connected neural network Rd - with ReLU activation, imput dimension d and hidden dimension m:
(I, W Wam)
Σ (.oww.))
Wm). Here, (xz) max(z, 0) is the ReLU function and Wim (w), Wm When Wim is clear from the context, we will write for short (xr).
We initialize the first layer using standard Kaiming initialization [He et al., 2015], ie. wA ~ N (0, A4a, and the output layer as u, u ( for every E [m]. Note that in standard initialization, each u, would be initialized normally with a standard deviation of ymm for ease of analysis we fix the initial value to be equal to the standard deviation where only the sign is random.
We consider dataset with binary labels. Given a training dataset (e1,91),... ... (ær, yr) E Rd x {- we train w.r.t. the logistic loss (a.k.. binary cross entropy): L(q) log(1 e e-9), and minimize the empirical error:
Σ Lys N(x,, W1:m))
min
We assume throughout the paper that the network is trained using either gradient descent (GD) stochastic gradient descent (SGD). Our results hold for both training methods. We assume that only the weights of the first layer (i.e. the w, 's) are trained, while the weights of the second layer (i.e. the u, (s) are fixed.
3.2 Assumptions on the Data
Our main assumption in this paper is that the input data lie a low dimensional manifold, which embedded in high dimensional space. Specifically, we assume that this "data manifold" is linear subspace, denoted by P, which has dimension d f. We denote by & the dimension of the
3

---

data "off- "off-manifold", i.e. the linear subspace orthogonal to the data subspace, which is denoted by pl. In this work we study adversarial perturbations in P! Note that adding perturbation from p- of any size to input data point which changes its label is an unwanted phenomenon, because this perturbation is orthogonal any possible data point from both possible labels. We will later show that under certain assumptions there exists adversarial perturbation in the direction of p1 which also has small norm. This reason for this assumption so that the projection of the first layer weights on p- remain fixed during training. An interesting question is to consider general "data manifolds", which we elaborate on Section 7.
To demonstrate that the oow-diensioona data assumption arises in practical settings, in Figure 1 we plot the cumulative variance of the MNIST and CIFARIO datasets, projected on linear manifold. These are calculated by performing PCA on the entire datasets, and summing over the square of the singular values from largest to smallest. For CIFARIN 10, the accumulated variance reaches 90% at 98 components, and 95% at 216 components. For MNIST, the accumulated variance reaches 90% at 86 components, and 95% at 153 components. This indicates that both datasets can projected to much smaller linear subspace, without losing much of the information.
1.0
0.35
0.-
3.
2.6
0.6-
04-
Cumulative variance
Cumulative variance
0.4
90% percentile
95 % percentile
95% percentile
350
1020
-340
2000
2510
1003
100
200
500
430
5:0
600
700
802
Components
Camsewars
(a)
Figure 1: The cumulative variance for the (a) CIFARIO; and (b) MNIST datasets, calculated by performing PCA on the entire datasets, and summing over the square of the singular values from largest to smallest.
Remark 3.1 (On the Margin of the Network). Given neural network N Fd - E. and dataset (Z1,30 y1),..., (Iy, Ye) with binary labels which the network label correctly, we define the margin of the network as y minig(r) (xx))
In our setting, possible to roughly estimate the margin without assuming much about the data, besides its boundedness. Note that the gradient of the loss decays exponentially with the output of the
=2 Hence, if we train for at most polynomially many iterations network, because 0L(g) 140-0
ôq
and label all the data points correctly (i.e. the margin is larger than O), then training effectively stops after the margin reaches Ollog?(d) This is because if the margin is log" (d), then the gradient is of size:
= logg -og"(dde log)(d) e- log) I (d)) d' log(d)
('(lo2(d))
which is smaller than any polynomials in d. This means that all the data points on the margin (which consists of at least one point) will have an output of O (polyloggd)).
The number of points which lie exactly on the margin is difficult to assess, since it may depend on both the dataset and the model. Some empirical results this direction are given in Vardi et [2022], J. where it is observed (empirically) that for data sampled uniformly from sphere and trained with two-layer network, over 90% of the input samples on the margin. Also, in Haim et [2022) shown that for CIFARIO, large portion of the dataset lies the margin.

---

Large Gradient Orthogonal to the Data Subspace
One proxy for showing non non-robustness of models, is to show that their gradient w.r.t the imput data is large (ef. Bubeck et al. [2021b], Bubeck and Sellke [2021]). Although a large gradient does not guarantee that there is also adversarial perturbation, it is an indication that small change in the input might significantly change the output. Moreover, by assuming smoothness of the model, is possible to show that having large gradient may suffice for having adversarial perturbation.
In this section we show that training a network on a dataset which lies entirely on a linear subspace yields large gradient in direction which is orthogonal to this subspace. Moreover, the size of the gradient depends on the dimension of this subspace. Specifically, the smaller the dimension of the data subspace, the larger the gradient is in the orthogonal direction. Our main result in this section is the following:
Theorem 4.1. Suppose that a network ((x) = u ((w, is trained on a dataset which lies i=1
on linear subspace P C Rd of dimension d - 2- for & 2 1, and let P. Let S = {i E [m]
zo) 2 0}, and let ls == (S). Then, w.p 2 1 f/16 (over the initialization) we have:
(2N(20) dx
kl
2md
The full proof be found in Appendix B. Here we provide a short proof intuition: First, we use a symmetry argument to show that suffices to consider w.l.o.g. the subspace M == spanfe)..... Ed-e), where e; are the standard unit vectors. Next, we note that since the dataset lies on M, only the first d - coordinates of each weight vector w; are trained, while the other & coordinates are fixed their initial value. Finally, using standard concentration result on Gaussian random variables we can lower bound the norm of the gradient. Note that our result shows that there might be a large gradient orthogonal the data subspace. This correspond "off-manifoodd adversarial examples, while the full gradient (i.e. without projecting on P might be even larger.
The lower bound on the gradient depends on two terms: f and 5. The first term is the fraction of active neurons for the input zo. i.c. the neurons whose inputs non-negative. Note that inactive neurons do not increase the gradient, since they do not affect the output. The second term corresponds to the fraction of directions orthogonal to the data. The larger the dimension of the orthogonal subspace, the more directions in which it is possible to perturb the input while still being orthogonal to the data. If both of these terms constant, i.e. there is constant fraction of active neurons, and "off manifold" directions, we can give a more concrete bound on the gradient:
m
Corollary 4.1. For e = M(d), k a(m), in the setting of Theorem 4. I, with probability 2 1- 1-d- we have:
I p.
(~r
(oN(ze)
|| -21).
2(1)
Fx
Consider the case where the norm of each data point @(vā) = ((VE), i.e. every coordinate is of size ((1. By Remark 3.1, for a point To on the margin, its output is of size polylog(d). Therefore, for the point zo, gradient of size (1(1) corresponds to an adversarial perturbation of size polylog(d). which is much smaller than ||اø|| || = (vdd). We note that this is rough and informal estimation, since, as we already discussed, a large gradient at zo does not necessarily imply that an adversarial perturbation exists. the next section, we will prove the existence of adversarial perturbations.
5 Existence of an Adversarial Perturbation
In the previous section we have shown that at any point Zo which lies on the linear subspace of the data P. there is large gradient in the direction of p.. In this section show that not only the gradient is large, there also exists an adversarial perturbation in the direction of P. which changes the label of data point from P (under certain assumptions). The main theorem of this section is the following:
x}) )) is trained on dataset which lies Theorem 5.1. Suppose that a network N N(x) = Σ (,0((1;
1=1
on linear subspace P. C Rd of dimension d - l, where f 2 32(m 1) logomm²²d) Let Do P.
5

---

and denote yo == sign( ((oo)). Ler I- == fie [m] < 0 and 1+ == fi [m] > 0). and
denote k- := ii I- To? 2 04/, and k+ == (i E 1 (w, zo) 2 0}} Let kyn kー if
Yo and luyo h+ if you
-. For w E R denote = p (w), and denote the perturbation
= yoo a
Σ w Σ iET_
wi =ET+
where
8umdN(zo) tkyo
Then, w.p. 2 5(me f/16 + d 1/2)
we have that ||=|| S 8/2N(mu)
m Roo
Vz and:
sign N + 2)) + sign(N((x)))
The full proof can be found in Appendix C. Here we give short proof intuition: As in the previous sec- tion, we show using symmetry argument that w.l.o.g. we can assume that P = span ea-e)-
Now, given the perturbation from Theorem 5.1 want understand how adding it to the input changes the output. Suppose that yo = 1. We can write
Na 2)
Σ VET
u (ws, + (w, +
Σ
uo((wu; + (w,, 2))
iET
We can see that for all i:
(163, 2) = a.. (Wi,
Σ
pp (w)) - Σ
Ip (w)))
jEL-
JEL+
-a. (wi
sign (ly (ppp)(w))..
For T_ we can write:
(w,, 2) = a |[I (wx))l a (w;),
Σ
sign (2 I (w²))
jyts
and using similar calculation, for E 1 we can write:
(w, 2) -a (w.)ll plIpp (wx), Σ
sign(u, Ip (wy))
jtt
Using concentration inequalities of Gaussian random variables, and the fact that p (w;) did not change from their initial values, we can show that:
Vem
m (w)),
Σ (ug (w,)) J
while p (w:)ll 11 ~ 4. Thus, for a large enough value of we have that (we, 2) S 0 for i E 4 and (w,, 2) R a.
Vi d (Vt mm) for i E I_.
From the above calculations we can see that adding the perturbation 2 does not increase the output of the neurons with a positive second layer. On the other hand, adding can only increase the input of the neurons with negative second layer, and for those neurons which are also active it increases their output as well if we assume that > m. This means, that if there are enough active neurons with negative second layer (denoted by k= in the theorem), then the perturbation can significantly change the output. In the proof we rely only on the active negative neurons to change the label of the output (for the case of yo 1. if yo = -1 we rely on the active positive neurons). Note that the active positive neurons may become inactive, and the inactive negative neurons may become active. Without further assumptions it is not clear what is the size of the perturbation make this change for every neuron. Thus, the only neurons that are guaranteed to help change the label are the active negative ones, which by assumptions é are guaranteed to increase their output.
Note that our perturbation is not the direction of the gradient w.r.t. the input. The direction of the gradient would be the sum of all the active neurons, i.e. the sum (with appropriate signs) over all E [m] such that (w;, vo) 2 O. Our analysis would not have worked with such a perturbation, because could not guarantee that inactive neurons would stay inactive.

---

The assumption that é = 92M) (up to log factors) is a technical limitation of our proof technique. We note that such an assumption also used in other theoretical papers about adversarial perturbations (e.g. Daniely and Shacham [2020]).
Note that the direction of the perturbation does not depend on the input data Ho, only its size depends on Zo. In fact, Theorem 5,1 shows that there is a single universal direction for an adversarial perturbation that can flip the label of any data point in P. The size of the perturbation depends on the dimension of the linear subspace of the data, the number of active neurons for Io. the total number of neurons in the network and the size of the output. In the following corollary we give specific bound on the size of the perturbations under assumptions on the different parameters of the problem:
Corollary 5.1. In the setting of Theorem 5. 1, assume in addition that l ((d) and kyo = ((m). Then, there exists a perturbation 2 such that w.p. 2 1 5 (me t/116 (x)())) and:
(t/16/16 + d= 1/) we have ||
(N + 2)) # sioo))
The above corollary follows directly by noticing from Theorem 5.1 that:
| S O (
m
O(N(2)))
Kyo
where we plugged in the additional assumptions. The assumptions in the corollary above are similar to the assumptions in Corollary 4.1. Namely, that the dimension of the data subspace is constant fraction from the dimension of the entire space, and the number of active neurons is a constant fraction of the total number of neurons. Note that here we only consider active neurons with specific sign in the second layer.
Note that the size of the perturbation in Corollary 5.1 is bounded by N(ro). By Remark 3.1, the output of the network for data points the margin can be at most Ologg(d)), since otherwise the network would have essentially stopped training. Therefore, if we consider an input zo on the margin, and ||z = @(vd) = @(VE), then the size of the adversarial perturbation is much smaller than || For any other point, without assuming it is the margin, and since we do not assume anything about the training data (except for being in P), we must assume that the size of the perturbation required to change the label will depend the size of the output.
6 The Effects of the Initialization Scale and Regularization on Robustness
In Section 5, we presented the inherent vulnerability of trained models to small perturbations in a direction orthogonal to the data subspace. In this section, we return to common proxy for robustness that we considered in Scction 4 - the gradient at an input point Tg. We suggest two ways that might improve the robustness of the model in the direction orthogonal to the data, by decreasing bound of the gradient in this direction. We first upper bound the gradient of the model in the general case where we initialize w; (0, 12),), and later discuss strategies to use this upper bound for improving robustness.
upper
Theorem 6.1. Suppose that network N(x) = Σ uior ((w;,( xx)) is trained on dataset which lies i=1
on linear subspace P C Rd of dimension d - for E > 1, and assume that the weights w, are initialized from (0, I2).. Let xo E P, let S {i E [m] (W;, 2n) 2 0}. and let = (S]. Then,
w.p. 2 1 e- to we have:
(aN(z)) Iu
2kl
31
m
The full proof uses the same concentration bounds ideas as the lower bound proof and can be found in Appendix D. This bound is result of the untrained weights: since the projection of the data points on P1 zero, the projection of the weights vectors on P... are not trained and are fixed at their initialization. We note that Theorem 4. readily extends to the case of initialization from (0, Ia). which case the lower bound it provides matches the upper bound from Theorem 6. up to constant factor. In what follows, we suggest two ways to affect the post-training weights in the P. direction: (1) To initialize the weights vector using smaller-variance initialization, and (2) Add an Lg-norm regularization on the weights. We next analyze their effect on the upper bound.
7

---

6.1 Small Initialization Variance
From Theorem 6.1, one can conclude strong result about the model's gradient without the depen- dency of its norm on and k.
Corollary 6.1. For 2v2' in the settings of Theorem 6.1, with probability 2 1 - e to we have
(2N(20)
1
dx
Vd
The proof follows directly from Theorem 6.1, by noticing that k S m and { S d. Consider for example an input To P with || e @(vd), and suppose that (zo) 92(1). The above corollary implies that if the initialization has a variance of 1/d2 (rather than the standard choice of
1/d) then the gradient is of size O (1/va). Thus, it corresponds to perturbations of size (vvd). which is the same order as ||z0|).
6.2 L Regularization
We consider another way influence the projection onto p of the trained weights vectors: adding L2 regularization while training. We will update the logistic loss function by adding an additive factor 1 ||W1sm|l For dataset (I1, 01), (Zr: Vr), we now train over the following objective:
1 Σ L(w - Nay, Wis))) + > |imm/
This regularization will cause the previously untrained weights to decrease in each training step which will decrease the upper bound on the projection of the gradient:
Theorem 6.2. Suppose that network N(x) = Σ uo (ww((,,, is trained for T training steps,
=
1=1
using L2 regularization with parameter X 2 0 and step size rj > O, on a dataset which lies on linear subspace P C Rdd of dimension d - for l z starting from standard initialization (i.e., wi NO0,, (a)). Let To E P, let = {i [m] (we, Fo) 2 0}, and let == =S]. Then, w.p.
we have
(ON(TT) dr
2kE md
TpI
< < - n1=
The full proof can be found in Appendix D.1. The main idea of the proof is to observe the projection of the weights on p changing during training. As before, we assume w.l.o.g. that P = sep..ee....... Ed-e} and denote by úq := Ipp (w;). During training, the weight vector's last É ccoordinates are only affected by the regularization term of the loss. These weights decrease in constant multiplicand of the previous weights. Thus, we can conclude that for every t 2 O we have: wi!") (1 m where vn is the i-th weight vector at time l. It implies that setting is equivalent initializing the weights with standard deviation (1-nA)T and training the model without vd
regularization for T steps. As a result, we get the following corollary:
Corollary 6.2. For (1 - nn)T < 1
.2d in the settings of 6.2, w.p. 2 1 - fe we get that:
(ao(2))
1
dx
Vd
6.3 Experiments
In this section, we present our robustness-improoinng experiments. two datasets: (1) A 7-point dataset on a one-dimensional linear subspace in a two-dimensional input
We explore our methods on
'Fo the off
code of the experiments
https: /tip::/ com/ ode iamel/

---

space, and (2) A 25-point dataset on two- ivo-dimensional linear subspace in three-dimensional input space. In Figures and we present the boundary of two-layer ReLU network trained over these two datasets. We train the networks until reaching a constant positive margin. We note that unlike our theoretical analysis, the experiments in Figure we trained all layers and initialize the weights using the default Py PyTorch initialization, to verify that the observed phenomena occur also in this setting. In the experiment in Figure we use a different initialization scale for the improving effect to be smaller and visualized easily. In Figures 2a and 3a trained with default settings. In Figures 2b and 3b we initialized the weights using an initialization with a smaller variance (i.e., initialization divided by constant factor). Finally, in Figures 2c and 3c train with L2 regularization.
075
98
099
44.00 -0.0
8.0
235
40.00
(a)
(b)
(c)
Figure 2: Experiments on a one-dimensional dataset. We plot the dataset points and the decision boundary in 3 settings: (a) Vanilla trained network, (b) The network's 's weight are initialized from smaller variance distribution, and (c) Training with regularization.
Consider the adversarial perturbation in the direction P., orthogonal to the data subspace, in Figures 2 and 3. In figure (a) of each experiment, needed to cross the boundary in the subspace orthogonal to the data. In the middle figure (b), see that the boundary in the orthogonal subspace is much further. This is direct result of the projection of the weights onto this subspace being much smaller. In the right experiment (c), we a similar effect created by regularization. In Appendix E we add the full default-scaled experiment in the two-dimensional setting to demonstrate the robustness effect. There, in both the mmall-initialization and regularization experiments, the boundary lines are almost orthogonal the data subspace. In Appendix E we also conduct further experiments with deeper networks and standard Py PyTorch initialization, showing that our theoretical results also observed empirically in settings going beyond our theory.
that rather small adversarial perturbation
22
21
02
68 Q
12
10
16
(a)
(b)
Figure 3: Experiments two-dimensional dataset demonstrating smaller robustness effect. We plot the dataset points and the decision boundary settings: (a) Vanilla trained network, (b) The network's 's weights are initialized from smaller variance distribution, and (c) Training with regularization. Colors are used to emphasise the values in the a axis.
In Figure 4 we plot the distance from the decision boundary for different initialization scales of first layer. We trained a 3-layer network, initialized using standard initialization except for the first layer which divided by the factor represented the X -axiss After training, we randomly picked 200 points and used a standard projected gradient descent adversarial attack change the label of

---

each point, which is described in the Y -axxi (perturbation norm, with error bars), The datasets are: (a) Random points from a sphere with 28 dimensions, which lies space with 784 dimensions; and (b) MNIST, where the data is projected on 32 dimensions using PCA. The different lines are adversarial attacks projected either on data subspace, on its orthogonal subspace, or without projection. can be seen that small initialization increases robustness off the data subspace, and also on the non-projected attack, while having almost no effect for the attacks projected on the data subspace.
data iudipace
On cata rubipece
Patter
(b)
Figure 4: The distance to the decision boundary for different initializations of the first layer. The X-axis represents the factor which the initialization of the first layer divided by. The Y-axis shows the size of a standard perturbed gradient descent adversarial attack to change the label of each point for 200 randomly picked points. The datasets are: (a) Random points from sphere with 28 dimensions, which lies in space with 784 dimensions; and (b) MNIST, where the data is projected on 32 dimensions using PCA. The different lines are adversarial attacks projected either on data subspace, on its orthogonal subspace, or without projection.
7 Conclusions and Future Work
In this paper we considered training a two-layer network on dataset lying on C Rd where P is d É dimensional subspace. We have shown that the gradient of any point zo E P projected on p- large, depending on the dimension P and the fraction active neurons on Zi- We additionally showed that there exists an adversarial perturbation in the direction of p+. The size of the perturbation depends in addition on the output of the network on To. which by Remark 3.1 should be voly-logarithmic in d, least for points which lie on the margin of the network. Finally, we showed that by either decreasing the initialization scale or adding L2 regularization we can make the network robust "off-manifold" perturbations, by decreasing the gradient in this direction.
One interesting question is whether our results can be generalized to other manifolds, beyond linear subspaces. We state this as an informal open problem:
Open Problem 7.1. Let M be manifold, and D a distribution over M x {+1). Suppose we train a network N Rd + R on a dataset sampled from D. Let xo E M, then under what conditions on M, D and N, there exists small adversarial perturbation in the direction of (Tz, i.e. orthogonal to the tangent space Tr, M, of M at Zg-
Our result can be seen as special case of this conjecture, where at all points x' E M, the tangent spaces are equal T.M M. Another future direction would be to analyze deeper networks, or different architectures such convolutions. Finally, it would also be interesting to analyze robustness of trained networks w.r.t. different norms such as L1 or Loo.
Acknowledgments and Disclosure of Funding
We thank Ohad Shamir for the many helpful discussions about this work. would also like to thank Michal Irani for contributing computational resources. GY was supported in part by the European Research Council (ERC) grant 754705.. GV acknowledges the support of the NSF and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning.
10

---

References
Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. Preprint, arXiv:2005. 10/90, 2020.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Cireumventing defenses to adversarial examples. In International conference on machine learning, pages 274-283. PMLR, 2018.
Peter Bartlett, Sébastien Bubeck, and Yeshwanth Cherapanamjeri. Adversarial examples in multi- layer random relu networks. Advances in Neural Information Processing Systems, 34, 2021.
Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. Advances in Neural Information Processing Systems, 34:2881 11122811-200 1-2822 2021.
Sébastien Bubeck, Yin Tat Lee, Eric Price, and Ilya Razenshteyn. Adversarial examples from computational constraints. In International Conference on Machine Learning, pages 831-840. PMLR, 2019.
Sébastien Bubeck, Yeshwanth Cherapanamjeri, Gauthier Gidel, and Rémi Tachet des Combes. A single gradient step finds adversarial examples on random two-layers neural networks. Advances in Neural Information Processing Systems, 34, 2021a.
Sébastien Bubeck, Yuanzhi Li, and Dheeraj M Nagaraj. law of robustness for two-layers neural networks. In Conference on Learning Theory, pages 804-820. PMLR, 2021b.
Nicholas Carlini and David Wagner. Adversarial examples detection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 3-14, 2017.
easily detected: Bypassing
Amit Daniely and Hadas Shacham. Most relu networks suffer from & adversarial perturbations. Advances in Neural Information Processing Systems, 33, 2020.
Elvis Dohmatob and Alberto Bietti. On the (non-) robustness of two-layer neural networks in different learning regimes. arXiv preprint arXiv:2203. 1864, 2022.
Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. Adversarial vulnerability for any classifier. Preprint, 1rXiv:1802.08800 2018
songwei Ge, Vasu Singla, Ronen Basri, and David Jacobs. Shift invariance can reduce adversarial robustmess. Preprint, arXiv:2103.0269) 2021.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Zoubin Ghahramani, Max Welling. Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, pages 2680, 2014a. URL attps::/psse /ttppe/di//ccceiiggg...... www.iiinnnns..uissepeee.. cc/ paper/201 /20 4 hhwhhaaahhhhaa 5ca 3e 9b1 122922222222 61 F2w06494099977700 a foof Aactt html.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. Preprint, 14Xi::1412.6572,, 2014b.
Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks. arXiv preprint arXiv:2206.0 07758, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Ajil Jalal, Andrew Ilyas, Constantinos Daskalakis, and Alexandros G Dimakis. The robust manifold defense: Adversarial training using generative models. arXiv preprint 1rXiv:1712.09114 2017.
Mare Khoury and Dylan HadfieldMenell. On the geometry of adversarial examples. Preprint, arXiv:1811.00525, 2018

---

Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding
variational
bayes.
arXiv preprint
B. Laurent and P. Massart. Adaptive estimation of a quadratic functional by model selection. The Annals of Staristics, 28(5):1302 1338, 2000. doi:
URL https:
//doi.org/10 1214222//aa aaos/ 1015957395.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. Preprint, arXhv:1706006883, 2017.
Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. 2017.
Andrea Montanari and Yuchen Wu. Adversarial examples in random neural networks with general activations. arXiv preprini arXiv:2203. 17209, 2022.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pages 582-597. IEEE, 2016.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security, pages 506-519, 2017.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint 1805006055 2018.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adver- sarially robust generalization requires more data. Preprint, arXiv: 1004. 1285,, 2018.
Ali Shafahi, Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial 2018.
examples inevitable? Preprint,
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Preprint, arXiv:2006.07710, 2020.
Adi Shamir, Itay Safran, Eyal Ronen, and Orr Dunkelman. A simple explanation for the existence of adversarial examples with small hamming distance. Preprini, arXiv: 1901. 2019.
Adi Shamir, Odelia Melamed, and Oriel BenShmuel. The dimpled manifold model of adversarial examples in machine learning. Preprint, arXiv:2106.10151, 2021.
Jascha -ocl-Diccsttin, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning. pages 2256-2265. PMLR, 2015.
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generaliza- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9977-,977, 2019.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. Preprint, arXiv:1312.6199, 2013.
Thomas Tanay and Lewis Griffin. A boundary tilting persepective on the phenomenon of adversarial examples. arXiv preprint arXiv:1608. 07690, 2016.
Gal Vardi, Gilad Yehudai, and Ohad Shamir. Gradient methods provably converge to non-robust networks. Preprint, arXiv:2202.0433: 04347, 2022.
Haohan Wang, Xindi Wu, Zeyi Huang, and Eric Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8684-8694, 2020.
Yunjuan Wang, Enayat Ullah, Poorya Mianjy, and Raman Arora. Adversarial robustness is at odds with lazy training. arXiv preprint arXiv:2207. 00411, 2022.
Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning, pages 5286-5295. PMI.R, 2018.
12

---

A Rotation Invariance w.r.t. the Initialized Weights
In this paper, we analyze neural networks trained on high-dimensional data that lies on low dimen- sional linear subspace denoted by P. We assume that the dimension of /' is d - L. Throughout the pa- per it will be more convenient to analyze data which lies the subspace M span([ .... ed-e)) because then the "off manifold" directions correspond exactly to certain coordinates of the input. In this section we show that we can essentially analyze the data as if it is rotated lie on M, and would imply the same consequences as the original data from P.
Theorem A.1. Let P S IR be a subspace of dimension Let R be an orthogonal matrix such that R - M, let X C P be training dataset and let XR= = {R- x I X}. Assume we train a neural network N(x) = =is (wo x) as explained in Section 3, and denote by Nx and N the network trained on and Xn respectively for the same number of iterations. Let Zo E P. then have:
L, and let M = spanfer..... Cd-e).
(8N (8N~ (20)
)|
1. W.p. p (over the initialization) we have
e (resp. S for some E I.
Ut
iff w.p. also [MM- ( (oN*n(RZa)
){
2 (resp. S c).
2. For any c, 2 0. w.p. p (over the initialization) there exists z E p- with |||| = such that sign(N (IO 2)) * sign(N (ognNN((x))) (zo)), w.p. p there exists E M with || = e such that (N (Reo + z²)) sign(N (N
Proof. Denote by Wim (Wi,.... Wm) and by RWim (Rwi, Rwm). Let wif)) the weights of the network trained on the dataset X where wi.m is some initialization, and w (t) lam = 1im. In the proof. (inf) ..... ) the weights of the network trained on X and initialized Rw( when taking derivatives w.r.t. the wyys 's we will explicitly write N(z, W1:m).
We first show by induction the number of training steps that w Rw(?) Fort 0 it is clear by the assumption on the initialization. Assume is true for f, then we have for some x E X:
0N (Rx, W w Ow,
uo ( RR)))
u,o' u,o' ((Rw! Re)) Rx
(ww((( xx) )x)
R 0N w (i
Ow;
This is true for every i [m] and for every x X. Also note that by our induction assumption we have:
m
m
Nxx, 1:m
u.o (t)
x)) = Σ u: ((iw) RR)) = N (Rr, wim)
(I)
Finally, the derivative of the loss single data point E X with label can be written as:
aL (N w (1 v)
L' (N w ( -w) ON (z, w (t) 1mm
Ow;
Ow;
where the first term depends only on the value of N N(x, (t) Hence, taking a single gradient step of N with weights w (t) and dataset X will change the weights by the same term up to multiplication
by R as if taking gradient step with with weights w wim. and dataset Xn. This finishes the induction.
La be an initialization for the training of N where there exists E P- with ||=||| = c such (0) that sign(N X (xo + =)) A sign(NX (zo)). Then, by Eq. (1) the initialization low 1:m for the training of Nx is such that for 2' R2 have ||a/|| and sign(NX X (Rxo + 22)) # sign(NXX Xx (Ræg)) This argument holds also in the opposite direction. Let A C {Wim E gddm be the set of all
Let w (0)
13

---

initializations to N* where there exists E Pl with |ж|| = such that Sign(N* (ro (x0 + 2)) * sign( (zo)), then by the above the set R A [RWi.m Witm E A] are exactly all the initializations N where there exists 2' E . with || such that sign (N (Rxo + of sign( (Rro)). Since we initialize the wy 's using Gaussian initialization which is spherically symmetric, we have that Pr(A) Pr(RAA) This proves item (2). Item (1) follows from similar arguments (which we do not repeat for conciseness).
Under the assumption that the data lies on M spanfe1. fi......... Ed-e)- and no regularization is used, can show that the weights of the first layer projected on M do not change during training. This an essential part of the proofs, as allows us to analyze those weights as random Gaussian vectors, and apply concentration bounds on them.
Theorem A.2. Let M = span/et,.... Ed-th.
Assume we train neural network N (I, Wi:m) == E (w z) as explained in Section (where Wim = (wi,..., Wm)). Denote by w == Tm for w E RR,, then after training. for each E [m], úú did not change from their initial value.
Proof. Note that for each E [mi] and MM we have:
(N (az, W Wism)
M
= M (u.o'(w) (wTx))) u,oo x) 0.
Ow:
Taking the derivative the loss we have:
AL (N (Zo Wim)) 1)
= Mare (N,,,,, wi:m) ..) (N,x, Wim))
1
Ow;
Ow;
ON Wism)
= (N wim) a1s
0.
Ow:
The above calculation did not depend on the specific value of the wy's. Hence, the value of the úū for every E [m] did not change during training from their initial value.
B Proofs from Section
Before proving the main theorem, we will first need the next two lemmas about the concentration of Gaussian random variables:
Lemma B.1. Let w E IR" such that w ~ 02 Then:
1 I 2²n
P |wol
se it
Proof. Note that w has the Chi-squared distribution. A concentration bound by Laurent and Massart [Laurent and Massart, 2000, Lemma 1] implies that for all we have
Pr
n
2 2vnt 1e-
Plugging-in & == 16. we get
2
1 n
W 2
1
Pr
n
Pr
-n/16
nn/16
2
Thus, we have
Pr
||o|||
I ov 2
n/16
n/16
Lemma B.2. Let W1...., Wm R such that for all i [m], wi ~ NN0,0 I²I)), then we have:
2
w
<
m n
14

---

Proof. We denote the j-th coordinate of the vector w; R" by Wig. Note, for any i {1, fl....,m) m)
m
and j (1,..., n we have was (V o.). We denote by the sum vector == 2 Wi, and by
sy the j- th coordinate of By this definition, 8y Σ way sum of m independent Gaussian
variables and therefore also Gaussian variable. Particularly, No0 Io) We use Lemma B. with variance mo² and get that:
IP
S m n
S
2
We are now ready prove the main theorem of this section:
Proof of Theorem 4.1. Let M span(ei, ... ed-e). By Theorem A.1(I), given training dataset X C P. it is enough to consider training set X = {Rx x E X). where f is an orthogonal matrix such that R- P M, and training is done over X g. From now on, we assume that the training data, as well as zo lic on M, and the consequences of this proof would also imply for a dataset X and zg E
The projection of the gradient on M + is equal to:
(ON Ox
M
Σ w (w) 20
Σ aara (u,ws) lies = Elar (u:w;)
iES
Denote by 0; (wi)d d-8++1d Ltd, the last coordinates wy By Theorem A.2 we get that for every i E [m], ii did not change from their initial value during training.
Recall that initialized ú; N0, Je
Note that the set S is independent of the value of the
w; s. This is because w; does not effect the training, hence will not effect w, M (w;). Also, after choosing do we have (ú,, .o) O, since Ω. O, which means that the choice of $ is independent of the i; ss We can conclude that the random variables W; for E S are sampled independently.
m). |us| vm and they are not trained, get that u,ú;; are also Gaussian random variables with the same mean, and variance multiplied by 1. Therefore, from Lemma B.2 get that w.p. 2 1 e e9/16
Note, since for all i E {I,...,m)
kl dm
Σ upu 2 iES
Combining the above, get:
(ON(
)
1 kl 2
T
dr
2
dm
C Proofs from Section 5
Before proving the main theorem, we prove a few lemmas about concentration of Gaussian random variables:
Lemma C.1. Let w 1a with ~ N(O, o² I2))). Then:
Pr || 2 20 n S e
15

---

Proof. Note that has the Chi-squared distribution. A concentration bound by Laurent and
Massart [Laurent and Massart, 2000, Lemma 1] implies that for all t we have
w
Pr
2
2v
nt 28 se
Plugging-in / == 16. we get
Pr
w
2 2n
- Pr
1 w
2 n/2 + n/8 S 66-11
Thus, we have
Pr
|1o|| 2 ovv2m
S se-1116.
Lemma C.2. Let u E R", and v ~~~ o Then, for every t >> O we have
12
Pr[l(u, o)| 2 [|u]] [] S 2exp
2n
Proof. We first consider < " 'Tu]] v). As the distribution NO, 1,)) (,) is rotation invariant, one can rotate
u and v to get and 8 such that Tal = E1. the first standard basis vector and [Tull v) TTll Ø). Note, v and have the same distribution. We can see that ( T ū) ~ ~O, o²) since is the first
coordinate of U. By a standard tail bound, we get that for / 0:
t2
u
Pr
K |u||
2 = Pr 1(
|ull
0)
2
t
=
Pr 1il
2 $J S
2 exp
2
Therefore
Pr[lau, o)1 2 ||u||| S 2exp
2n
Lemma C.3. Let u ~ NNO, and NO0, oIIn). Then, for every t we have
(u,, w)لl 2 o 2ntt s e nn/16 + 2c -2 1200 Pr
Proof. Using Lemma C.1 we get that w.p. S en/16 we have ||u|| 2 σ1 /n. Moreover, by
(- 1² 20g
Lemma C.2, w.p. S 2 exp
have (u, w>1 ||u||| t. By the union bound, we get
Pr (u, w/|l 2 σ1 2nt
2nt
< Pr
||u||
2 21
+ Pr (u, .> 2 ||u|| s e nn/16 + 2exp
20
We are now ready to prove the main theorem of this section:
Theorem 5.1. By Theorem A.1(2), We also assume w.l.o.g. that Vo = 1, the case Yo = -1 is proved a similar manner. Denote by (w) (w))-+++1d 1ad the last coordinates of w. By Theorem A.2 we have that w; have not changed after training from their initial value.
assume w.l.o.g. that P M spanfer...... +
We write N (xo + 2) as:
16

---

m
(æ +2
Σ u:o((u,, + (ws, 2))
Σ u: (wss va) + (wi, 2)) +
Σ (oo(wu + (w;, 2))
iET
iEl
Σ
u. ((ws, + (is, 2)) + Σ
u zo) + (w), 2))
iEl+
where the last equality since (2) = 0, hence (w, 2) = (w, 2) for every w E Rd. We will bound each term of the above separately.
For the first term in Eq. (2), where i I_ we can write:
(wi, 2) = a ||00 1|" + a(iis
Σ jØL
sign(u,)
By our assumptions, tū ~ /V (0, 4 and شه sign(u, m 1 i.idd Gaussian random variables, which are also symmetric hence multiplying them by -1 does not change their distribution. From Lemma B.I we get w.p. 2 1/18 that
N (0, le) since it is a sum of
a 2
a
2d
From Lemma C.3, and using /
(m 1) log()
we get w.p. 2 - 6 + 2e 2e- (dd22mm
1 - -f/16 + 2m -im--d-- 1/2 that
(w Σ sign(u;)(ūg)
< 1122
Vd
1
d 22l(m 1) log(md))
(3)
Applying union bound over the above two events, and for every E I_, we get w.p. 2 1 2 (me- 4/18 + 1/1/ that:
al a 20 1) log(m²d)
(w,, 2) 2d d
For the second term in Eq. (2), where i 1+ write in a similar way:
(w,, Σ) a ||, + a Σ sign(u,)) (wy)
Using the same argument as above, we get w.p 2 2 (me 1/16 + 1/2)) that:
al + 20(m 1) log(m2d)
(wi, < 2d
By assuming that é 2 8(m - 1) log(m2d) we get that (Ø,, S 0. Denote C ==
of 2d
á 222mm 1) log(mmd)) then going back to Eq. (2), using the above bounds and applying union bound, we get w.p. 2 1 (me 2/16 6
+ d 1/2) -E/16
that:
N N + 2) < Σ u. ( σo) +C) + Σ u,o(((,
iEr_
SET+
Σ
u xo) +C) + Σ (w0 + Σ (;o(( - Σ
unor u,o((ur 20))
sef
iE4
SEl.
SEI
Σ
un u (Wi, + +C)
Ui (ww; + N(xo)
IET_
Σ
uy (a((w;, + CC) a(w,, zo))) N(xo)
17

---

Define F- == I- (w) 2 0}, and k_ = [F-|. We have that:
Σ ui (o((ws, + o o(ww, mol)) -
Σ
us (o((w,, Fu) + + (w0;,
h-C
u.
mm
REF
where the first inequality is since we only sum over negative terms, and the second inequality is since both (w,, xo) 2 (because i F-) and C' (because f 32(m 1) log(m² ()). Combining all
of the above, we get that:
K-C
Nxo + 2) S
+ N(o))
(4)
Vm
By our assumption that f 32(m log(m2d) have that
18
Vl /log(dm?)
C a
V2vm
2d
d
avi
2 1) log(m=d)
d
2
al
4d
Plugging in C' and a 8/mid((aa)
to Eq. (4) we get that:
k
k_
N(ro + +2)5
N(xo)
Vm
k-
f SymdN(ro) + NZo0 = -Nooz) < 0,
Vm 4d
k_
and in particular (an(N((o)) * sign(N((x) + 2)).
We are left with calculating the norm of z:
|||||
I (w:) - Σ
Ilaes (wi)
iET-
REL+
I (ws)
= a Σ sign u4 W
Since for each i [m]] W; ~ -N (0, de), then -ign((;;) ū. also have the same distribution, because this is a symmetric distribution. Hence, Em -sign(u,)w; ~ A (o, m Ie) as sum of Gaussian
random variables. Using Lemma C.I we get w.p 2 1 - 8/16 that Dm sign ) s 2nd Plugging in we get that:
m 2ml 8vmdNN(aa) 8v2N(20)
d
|| S
V
k_
K
18

---

D Proofs for Section
For proving the main theorem, we will use the following lemma that upper bounds the norm of sum of Gaussian random variables:
Lemma D.I. Let Wis Wm E R" such that for all E [m], wa ~o (O))) then we have:
P
Σ
Wi
2mo
Se
Proof. We denote the j-th coordinate of the vector 1n by way-- Note, for any i E [m]] and j E [r]] we have Wi, NN(0, o²). We denote by s the sum vector == Σ Wis and by 8j the j-th coordinate of By this definition, 8j w is of m independent Gaussian variables and We use Lemma C.1 with variance
Σ
therefore also Gaussian variable. Therefore, NO, mo²In)) m and get that:
P
Σ
W
2 2mo n
S Se
im
We now prove the main theorem of this section:
Proof of Theorem 6.1. Similar to the lower bound the norm, let M = span By Theorem A.1(1). given training dataset X C P. it is enough to consider training X = Rr x E X), where R is an orthogonal matrix such that R P M, and training is done over Xt. From now on, assume that the training data, well as 20 lie on M, and the consequences of this proof would also imply for a dataset and Zo E P.
The projection of the gradient on M is equal to:
(ON dr
Ta
= M Σ 1am0l (wr. (e..
= The (uww)) lies = Σ Tar (u:w;)
iES
Denote by w; (w;) (-)))-e+iad the last coordinates of w. By Theorem A.2 get that for every
i [m], Wi did not change from their initial value during training.
Recall that we initialized ii ~ (0, T2). Note that the set S is independent of the value of the
w;:'s. 's. This is because dh does not effect the training, hence will not effect w; M (w;). Also, after choosing we have (u, 20) O, since tt O, which means that the choice of S is independent of the w. We can conclude that the random variables w; for E S are sampled independently.
Therefore, from Lemma B.2 get that w.p. 2 1 - -
Σ th < 3
iES
Note, since for all i E [m], |ual
7m and they are not trained, we get w.p. 2 1 ef/16 that:
1
(aN(zT)
2kl
T
< B/
Or
19

---

D.1 Explicit Lg regularization
Proof of Theorem 6.2. As before, for this proof we rotate the data subspace to lie on M spanfe1.... Cd-E) l and rotate the model's 's weights accordingly. For a dataset (æ1, Vi), (Ir, yr.). we train over the following objective:
Σ Luy N Mag Wim))) +
1 Σ |/immll
Theorem A.2, we showed for all (æz, Uz) that if we train the model using the loss L we get:
M [OL (N Wiam) Uj)
0
Ow,
Now, we analyze the training process using the new loss which includes the regularization term. We
denote by w the weight vector w, after & training steps, and by wi :- 11 (w!))) its projection
on the subspace orthogonal to M. We look at the projected gradient of w!"" ( w.r.t. the loss:
0 &E=z1 L (N(zg. wim) UU
2 W
IT M+
Ow;
Ow.
0L (N
w lim Us)
014 wc:
Σ a /=1
+ a
Sw.
Ow;
0 w 13/
=Il
Ow;
(Xce?") (1)
=II
For training step of size 7) using gradient descent we get that:
i
((+1) vn(") yy
Thus, after total of T iteration of training we get that:
= (1 -1- (1- w!
Therefore, the projection of gradients after training onto P will be the same as if they were initialized N (o, (1-mA)=7 Ia and trained using logistic loss without regularization, The rest of the proof
the same as Theorem 6.1 for B (1-nA)² Vd
E Further Experiments and Experimental Details
E.1
Further Experiments
In Figure 5 we present the boundary of a two-layer ReLU network trained over a 25-point dataset on two-dimensional linear subspace, similar to Figure 3. We train the networks until reaching constant positive margin. The difference between the figures that Figure we initialize the weights using the default Porch initialization, while in Figure 3 we initialized using smaller scale
20

---

for the robustness effect to be smaller, and visualized more easily. The experiment in Figure is demonstrating an extreme robustness effect, occurring when using the standard settings.
10
0.
66
26
04
04
52
-1 00
-1
-0.2
20 2g Ray 25 R
650
0.2
82
Leg az Pay
(a)
(b)
(c)
Figure 5: Experiments on two-dimensional dataset. We plot the dataset points and the decision boundary in settings: (a) Vanilla trained network, (b) The network's 's weights are initialized from smaller variance distribution, and (c) Training with regularization. Colors are used emphasise the values in the axis.
In Figure we go beyond the theory discussed this paper, and present similar phenomena in all three settings for five-layer ReLU network. In Figure 6a we see the boundary of the regularly trained network within small distance p- from the data points. Figure 6b we use small initialization for all five layers, and present boundary almost orthogonal to the data manifold. In Figure 6c, the boundary of regularized trained network is a similar form. This experiment suggests that our theoretical results might be extended also to deeper networks, where all layers trained.
-238
-22
(a)
(b)
(c)
Figure 6: Experiments one-dimensional dataset with deep network. We plot the dataset points and the decision boundary in settings: (a) Vanilla trained network, (b) The network's weights are initialized from smaller variance distribution, and (c) Training with regularization.
E.2 One-dimensional dataset experiment layer network (Figure 2)
Dataset For all the three experiments we used a 7-point data set, spread equally on the two dimensional line y from (-1.-1) (1, 1).
Network For all the three experiments we used two-layer ReLU network of width 100 with biases in both layers. The weights of both layers were initialized using (1+3) default PyTorch initialization for linear layers, (2) default initialization divided by 3.
Training We used train step of size 0.02 for (1+3) and 0.04 for (2). We trained both layers until the margin reached 0.3. The losses we used were (1+2) Logistic loss, (3) Logistic loss with 0.005 L2 regularization.
21

---

E.3
Two-dimensional dataset experiment smaller effect (Figure 3)
Dataset For all the three experiments we used a 25-point data set, spread equally on a grid which lies on the 2 = 0.5 axis.
Network For all the three experiments we used two-layer ReLU network of width 4000 with biases in both layers. The weights in the first layer were initialized in (1+3) from (O 1/31s), and in (2) from (0. 1/36/z). The weight of the output layer were initialized the uniform distribution over the set 11,
Training For all the experiments we trained both layers until the margin reached 0.3 and we used train step of size 0.002. The losses we used were (1+2) Logistic loss, (3) Logistic loss with 0.8 La regularization on the weights of the first layer.
E.4 Two-dimensional dataset experiment (Figure 5)
Dataset For all the three experiments we used a 25-point data set, spread equally on a grid which lies on the y axis.
Network For all the three experiments we used two-layer RelU network of width 400 with biases in both layers. The weights in any layer were initialized using (1+3) default PyTorch initialization for linear layers, (2) default initialization divided by 3.
Training For (1) experiments we used train step of size 0.005, and for (2+3) used step of size 0.05. We trained both layers until the margin reached 0.1. The losses we used were (1+2) Logistic loss, (3) Logistic loss with 0.005 L2 regularization.
E.5 One-dimensional dataset experiment 5 layer network (Figure 6)
Dataset For all the three experiments we used a 7-point data set, spread equally on the two dimensional line y from (-1, (1, 1).
For all the three experiments we used 5-layer RelU network of width 100 with biases in all layers. The weights in any layer were initialized using (1+3) default Py Torch initialization for linear layers, (2) default initialization divided by 3.
Network
Training For (1+3) experiments used train step of size 0.02, and for (2) used step of size 0.06. we trained all layers until the margin reached 0.3. The losses we used were (1+2) Logistic loss, (3) Logistic loss with 0.01 L2 regularization.
22


Reviews:
[{"review_id": "ZOYOwCnS7H", "reviewer": "Reviewer_ZcHg", "summary": "The paper shows that on two-layer neural networks trained using data which lie on a low dimensional linear subspace that standard gradient methods lead to non-robust neural networks, that networks which have large gradients in directions orthogonal to the data subspace, and are susceptible to small adversarial L2-perturbations in these directions. The paper shows that decreasing the initialization scale of the training algorithm and adding L2 regularization, can make the trained network more robust to adversarial perturbations orthogonal to the data.", "questions": "1.Does the proposed method which decreasing the gradient in directions orthogonal to the data subspace hurt the accuracy on clean data?\n2.The paper claim that large gradients exists in directions orthogonal to the data subspace, and are susceptible to small adversarial L2-perturbations in these directions, do other adversarial perturbations such as L1 or L_{\\infty} also exist in directions orthogonal to the data subspace?\n3.Is the proposed method effective in deeper networks?\n4.small typo errors,e.g. line 287,extra )", "limitations": "The number of data points is few, it's better to evaluate on more data points.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and organized, and is innovative and original. The structure of the paper is clear and rigorous.", "weaknesses": "The experiments are insufficient, the number of data points used to evaluate the proposed method is few.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and constructive comments.\n\n\"The experiments are insufficient, the number of data points used to evaluate the proposed method is few.\": \nThe experiments in the paper were done mostly for visualization purposes, with low dimensions and dataset size. We added several new experiments on larger datasets, both synthetic and MNIST/CIFAR10, please see the comment above to all the reviewers and the attached PDF. These experiments both show empirically on real datasets that they approximately lie on a low dimensional subspace, and that a small initialization scale indeed increases the robustness outside of the data subspace.\n\nRegarding the questions:\n\n1) Our methods, that is changing the initialization scale and $L_2$ regularization, may affect the accuracy on clean data.  We note that it does not necessarily hurt it; namely, it may as well improve the accuracy. Indeed, $L_2$ regularization is often used in practice for improving accuracy, regardless of its effect on adversarial robustness. Analyzing the effect of our methods on the accuracy requires additional assumptions on the data, since our results are general in the sense that we have almost no assumptions on the data, besides residing on a low dimensional subspace.\n\n2) In this work, we focused on adversarial perturbations w.r.t the $L_2$ norm. We believe that there are also adversarial perturbations w.r.t. other norms in directions orthogonal to the data, although proving such perturbations exist requires a different analysis, which is beyond the scope of the paper.\n\n3) Yes, our methods are effective for deeper networks, at least empirically, as demonstrated in Figure 4 in Appendix E. In the attached PDF, we describe further experiments on a low-dimensional sphere and on the MNIST dataset using deeper networks, demonstrating the same robustness effect described in the paper.\n"}, {"review_id": "mckwYhJqHc", "reviewer": "Reviewer_5bBJ", "summary": "This paper focus on the data lies on a low dimensional data manifold (linear subspace P). There’re no additional assumptions on the number of data and their structure (orthogonality). The paper considers the perturbation on P^\\orth space. The paper claims that standard gradient descent leads to non-robust neural network; while decreasing the initialization scale, adding L2 regularizer can make the trained network more robust.\n", "questions": "1. In Theorem 5.1 or Corollary 5.1, the perturbation size depends on the output size. However, in classification problems, you can always rescale the predictor ($w,u$) while maintaining the sign of prediction to be unchanged. To that end, $N(x_0)$ can be arbitrarily small, implying the size of perturbation can be arbitrarily small, regardless of the size of $k_{y_0}$ and $\\ell$. Then the result is a bit counterintuitive.\n\n2. I'm wondering whether theorem 4.1 lower bound can also depend on the network initialization so that it matches theorem 6.1.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Studying the adversarial perturbation on the low-dimensional subspace is an interesting problem. \n2. The paper is well-written and easy to follow. The mathematical derivation seems sound.\n", "weaknesses": "1. This paper specifically assumes the dataset lies on a linear subspace, and the perturbation lies in the orthogonal direction of the subspace. The analysis of thm4.1 still relies on part of the weight being unchanged after training, and the gradient lower bound depends on the unchanged weights. To my understanding, such analysis still relies on random initialization property and therefore not much different compared to previous work, which might previous work analysis also holds under this paper’s low dimensionality assumption. To that end, the perturbation lies in the orthogonal direction of the subspace this constraint is trying to bypass the gradient update algorithm. In other words, does it mean that theorem 4.1 still holds even if you consider adversarial training?\n\n2. Although the author motivates the idea by saying real-world dataset mostly lie in low-dimensional subspace, the experiments are based on extremely simple synthetic data. I’m wondering whether similar results hold for a simple dataset like MNIST.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and helpful comments.\n\nRegarding the weaknesses:\n\n1) We acknowledge in the paper that some of our proof techniques were also used in previous works about robustness. The main difference is that in this work we consider trained networks, and our results are general, with only very minor assumptions on the data, while previous works considered random and untrained networks. While the weights corresponding with the off-data dimensions are indeed unchanged, the other weights are updated and affect those gradients via activations. Therefore, even if we use stronger data constraints, results based on random networks do not apply here.  Analyzing adversarial training will require additional assumptions on the input data, this goes beyond the scope of the paper and is an interesting future research direction.  If we didn't understand the question correctly, we would be happy if the reviewer could clarify it.\n\n2) To show our results indeed extend to the MNIST dataset, we use simple PCA decomposition that demonstrates the implicit low linear data subspace. One can see a small fraction of the dimensions capture most of the variance. We also added an additional experiment on the MNIST dataset demonstrating the phenomena presented in the paper. Details about the experiments can be found in our global comment above, and the results are in the attached PDF.\n\nRegarding the questions:\n\n1) Theorem 5.1 and Corollary 5.1 indeed depend on the size of the output, but also on the scale of the initialization. Please note that scaling the output (by scaling the weights) in these results is equivalent to scaling the size of the initialization, which, as we show, affects the size of the adversarial perturbation.\n\n2) This is a good question. Yes, Theorem 4.1 could have been written where the scale of the initialization is a parameter of the problem and then the upper and lower bounds in Theorem 4.1 and Theorem 6.1 would have been tight (up to a constant factor). We wrote the theorem this way to emphasize the phenomenon in standard initializations, and will consider changing it for the camera-ready version."}, {"review_id": "JhXMeDFCqc", "reviewer": "Reviewer_X8ox", "summary": "This paper studies the vulnerability of two-layer ReLU networks under adversarial attack when the data is in a low-dimensional manifold. The paper also observes that adding L2 regularization in clean training also improves the adversarial robustness.", "questions": "Please address my concerns mentioned in the weakness section.", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The intuition of this paper is interesting, with a high quality and a good clarity. Although existing literature considers that the vulnerability of clean training is caused by the off-manifold attack, this is the first one to study the theoretical perspective in detail.", "weaknesses": "(1) My major concern is that this paper mainly studies the side off the data manifold, but not the side on the data manifold. Some claims need more justifications. For example, in the abstract, the authors mention that \"the standard gradient methods lead to non-robust neural networks, .. and are susceptible to small adversarial L2 perturbations in these directions\". To claim that the perturbation is \"small\", the authors need to provide a result on what is the smallest on-manifold attack strength which results in an incorrect prediction. It is essential to make comparison in order to claim a quantity is \"small\", i.e., while Theorem 4.1 presents the L2 norm of the off-manifold gradient, what is the on-manifold gradient? \n\n(2) For the numerical experiments, please also share the code or report the observations when the training iteration T is large enough so that the loss converges to zero. In the paper\n\nBa, J., Erdogdu, M., Suzuki, T., Wu, D., & Zhang, T. (2020, April). Generalization of two-layer neural networks: An asymptotic viewpoint. In International conference on learning representations.\n\nit is observed that training a neural network in regression in their setting is the same as a penalized linear regression, i.e., the training already introduces some sort of regularization. It would be great if the authors of this submission can provide the code or some more experimental results to support the claims in this submission.\n\nDue to concern (1) and (2), my current score is 5. I like the intuition of this paper so I will consider raising my score if these major concerns can be addressed properly. For (1), a full theoretical proof may takes too much time. A brief justification would be helpful enough.\n\nSome other comments:\n\n(3) While Theorem 6.2 provides justifications on how L2 regularization improves adversarial robustness, it would be helpful if the authors can also provide the convergence and the generalization performance of the regularized neural network, which may potentially implies a trade-off between clean performance and adversarial robustness.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and helpful comments.\n\nWeaknesses:\n\n1) We emphasize that an analysis of the perturbations on the data manifold would require having some additional assumptions on the data, e.g. about its distribution. We believe that one of the major benefits of our work is that the results are general in the sense that the only assumption we have on the data is that it resides on a low-dimensional subspace. Hence, we can only analyze the gradient and the adversarial perturbations off the manifold. We think it is of great interest to study on-manifold perturbations and compare them to those off-manifold. However, this is beyond the scope of our paper. \n\n2) We will include all the code for the experiments in the camera-ready version. \n\n    The visualizations in the paper demonstrate the evolution of the decision boundary, rather than the loss of the predictor on the train/test data.\n    Therefore we measured convergence using minimal margin criteria, as noted in Appendix E. \n    We added several new experiments which demonstrate the theoretical phenomenon described in the paper, on both synthetic and real datasets. In these experiments, we trained the network until reaching a cross-entropy loss of $10^{-3}$.\n    Full details about the experiments can be found in our global comment above, and the results are in the attached PDF.\n\n3) We agree it would be helpful to provide the convergence and generalization performance of the trained predictors and potentially observe a trade-off between robustness and performance. However, to analyze these, we would need further assumptions on the data beyond residing on a low-dimensional manifold. This is because, different data distributions (on the manifold) would mean different generalization or convergence guarantees. We think that a very interesting follow-up work would be to study specific on-manifold distributions, and use the tools we developed in this paper to compare on- and off-manifold robustness with generalization and convergence."}, {"review_id": "w0n0KFE2CP", "reviewer": "Reviewer_2uZr", "summary": "The paper investigates adversarial robustness of trained ReLU neural networks when data lies within a linear subspace.  For the theoretical part, which is the bulk of the paper, the networks have two layers and only the first layer is trained.  Then the key observation is that the assumption on the data causes the training to change only the projections of the first-layer weight vectors to the linear subspace, i.e. their components in the orthogonal subspace remain as at initialisation.  Based on this, the authors prove several results, including a lower bound on the projection to the orthogonal subspace of the gradient of the network output at any point in the linear subspace, existence of a direction in the orthogonal subspace for universal adversarial perturbations, and that either reducing the initialisation scale or using L2 regularisation can reduce the norm of the projections to the orthogonal subspace of the gradients (in the latter case, the projections of the first-layer weight vectors to the orthogonal subspace are changed during training only by the L2 regulariser).  The results contains some assumptions, most notably for the universal adversarial perturbation, the width of the network has to be roughly at most a constant fraction of the input dimension.  The theory is supplemented by experiments on small examples in dimensions 2 and 3; in them all layers are trained and a network with five layers is also considered, which suggests that it may be possible to extend the theoretical results beyond the assumptions in the paper.\n", "questions": "Are you able to say how the theoretical results would change if the second-layer weights were trained as well?\n", "limitations": "The assumption of small network width in relation to the input dimension is restrictive, although a similar assumption featured in the related paper by Daniely and Shacham (NeurIPS 2020).\n", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is a nice mostly theoretical follow-up to the mostly empirical \"The Dimpled Manifold Model of Adversarial Examples in Machine Learning\" by Shamir, Melamed, and BenShmuel (CoRR abs/2106.10151), which in particular brings in the considerations of the effects of initialisation scale and L2 regularisation.  The theorems are proved in detail in the appendix, with helpful sketches provided in the main.  The experiments and the resulting figures aid the intuition, and suggest directions for future theoretical work.\n", "weaknesses": "The proofs of the theoretical results are perhaps not surprising or difficult.\n\nI believe the code for the experiments is relatively simple and essentially given by the text in Appendix E, however it was not submitted with the supplementary materials.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and constructive comments.\n\nCode for the experiments: We indeed report all the experimental details in Appendix E. We will publish the full code with the camera-ready version.\n\n\"Are you able to say how the theoretical results would change if the second-layer weights were trained as well?\": This is a good question and an interesting future research direction. We believe it is possible to show a similar result when the second layer weights are also trained. However, such a result would require stronger assumptions on the input data, beyond residing on a low dimensional manifold. Therefore we tried to avoid it in our paper and keep the conclusions as general as possible. Empirically, we trained all layers and showed similar results in the experiments attached here and in the paper.\n"}]

Author rebuttal:
We thank the reviewers for the thorough reviews and constructive comments.
In the attached PDF we add two experiments, aiming at showing empirically the effects of small initialization and that real datasets approximately lie in a low dimensional subspace.

1) To show that real datasets approximately lie in a low dimensional subspace we performed PCA on MNIST and CIFAR10, and calculated the cumulative variance. For MNIST, it reached $90\%$ of the total variance by accumulating $86$ components, and $95\%$ variance by accumulating $153$ components. Similarly, CIFAR10 reaches $90\%$ of the total variance by accumulating $98$ components, and $95\%$ variance by accumulating $216$ components. Note that MNIST is a $784$-dimensional dataset, and CIFAR10 is a $3072$-dimensional dataset. This indicates that most of the information for both of these datasets indeed lies in a low-dimensional subspace.

2) We trained a $3$-layer fully-connected neural network on two datasets for different initialization scales of the first layer of the network, and used a standard projected gradient descent adversarial attack to calculate the effect of the initialization scale on the distance from the decision boundary. The datasets are:

   - MNIST projected on a $32$ dimensional subspace using PCA.
   - $500$ random samples from a $20$-dimensional sphere which lies in a $784$-dimensional space.

    The adversarial attack is either: (a) Projected on the data subspace; (b) Projected off the data subspace (i.e. the space orthogonal to it); or (c) Unconstrained attack. Each experiment was repeated $5$ times (with a newly trained networks each time), and error bars are given in the plot. 

    This experiment shows the dramatic effect of changing the initialization scale on the attacks projected off the data subspace. Also, changing the initialization scale improves robustness for the unconstrained attacks, while having almost no effect for the attacks projected on the data manifold.
    


Meta-review: The paper investigates adversarial robustness of trained 2-layer neural networks when data lies within a low-dimensional linear subspace.  It shows that standard gradient methods lead to non-robust neural networks.  Additionally, it demonstrates that reducing the initial scale of the training algorithm can enhance the trained network's resilience to adversarial perturbations. Exploring adversarial perturbations within the context of low-dimensional subspaces is quite interesting. The paper is well-written, with clear and innovative mathematical derivations.
Decision: Accept (poster)