Title: Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspaces
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adversarial Examples

Robustness

Neural Networks

Classification
Abstract: Despite a great deal of research, it is still not well-understood why trained neural networks are highly vulnerable to adversarial examples.
In this work we focus on two-layer neural networks trained using data which lie on a low dimensional linear subspace.
We show that standard gradient methods lead to non-robust neural networks, namely, networks which have large gradients in directions orthogonal to the data subspace, and are susceptible to small adversarial $L_2$-perturbations in these directions.
Moreover, we show that decreasing the initialization scale of the training algorithm, or adding $L_2$ regularization, can make the trained network more robust to adversarial perturbations orthogonal to the data.

Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspaces

Abstract

Despite a great deal of research, it is still not well-understood why trained neural networks are highly vulnerable to adversarial examples. In this work we focus on two-layer neural networks trained using data which lie on a low dimensional linear subspace. We show that standard gradient methods lead to non-robust neural networks, namely, networks which have large gradients in directions orthogonal to the data subspace, and are susceptible to small adversarial L 2 -perturbations in these directions. Moreover, we show that decreasing the initialization scale of the training algorithm, or adding L 2 regularization, can make the trained network more robust to adversarial perturbations orthogonal to the data.

1 Introduction

Neural networks are observed to be susceptible to adversarial perturbations [Szegedy et al., 2013], often imperceptible by humans. Many works have shown attacks, where adding a very small perturbation to the input may change the prediction of the network [Carlini and Wagner, 2017, Papernot et al., 2017, Athalye et al., 2018]. Other works have shown defense mechanisms called adversarial training [Papernot et al., 2016, Madry et al., 2017, Wong and Kolter, 2018]. Despite a great deal of research, it is still not well-understood why neural-network training methods tend towards such non-robust solutions.

Several recent works have given theoretical explanation for the existence of adversarial perturbations under different settings. One line of work [Daniely and Shacham, 2020, Bubeck et al., 2021a, Bartlett et al., 2021, Montanari and Wu, 2022] have shown that random networks are susceptible to adversarial perturbations. These results might explain why neural networks are non-robust at initialization, but they do not explain why trained neural networks are non-robust. Recently, Vardi et al. [2022] showed that for data which is nearly orthogonal , after training for infinitely many iterations, the implicit bias of neural networks towards margin maximization leads to non-robust solutions. Despite these works, it is still unclear why trained neural networks tend to be non-robust to adversarial perturbations, and specifically what are the assumptions on the input data which leads to non-robustness.

One common belief about 'real-life' datasets, is that they approximately lie on a low dimensional 'data-manifold' in a high dimensional space. In this setting, the existence of perturbations orthogonal to the data-manifold that change the network's predictions is especially undesired, since such perturbations do not make the input closer to data points from other classes. Indeed, such a perturbation only increases the distance between the input and all 'real-life' examples. Shamir et al. [2021] have demonstrated empirically that under such a data-manifold assumption, the decision boundary of a trained classifier clings to the data-manifold in a way that even very small perturbations orthogonal to the manifold can change the prediction.

In this paper we focus on data which lies on a low dimensional 'data-manifold'. Specifically, we assume that the data lies on a linear subspace P ⊆ R d of dimension d -ℓ for some ℓ > 0 . We study adversarial perturbations in the direction of P ⊥ , i.e. orthogonal to the data subspace. We show that the gradient projected on P ⊥ is large, and in addition there exist a universal adversarial perturbation in a direction orthogonal to P . Namely, the same small adversarial perturbation applies to many inputs. The norm of the gradient depends on the term ℓ d , while the perturbation size depends on the term d ℓ , i.e. a low dimensional subspace implies reduced adversarial robustness. Finally, we also study how changing the initialization scale or adding L 2 regularization affects robustness. We show that in our setting, decreasing the initialization scale, or adding a sufficiently large regularization term, can make the network significantly more robust. We also demonstrate empirically the effects of the initialization scale and regularization on the decision boundary. Our experiments suggest that these effects might extend to deeper networks.

2 Related Works

Despite extensive research, the reasons for the abundance of adversarial examples in trained neural networks are still not well understood [Goodfellow et al., 2014b, Fawzi et al., 2018, Shafahi et al., 2018, Schmidt et al., 2018, Khoury and Hadfield-Menell, 2018, Bubeck et al., 2019, Allen-Zhu and Li, 2020, Wang et al., 2020, Shah et al., 2020, Shamir et al., 2021, Ge et al., 2021, Wang et al., 2022, Dohmatob and Bietti, 2022]. Below we discuss several prior works on this question.

In a string of works, it was shown that small adversarial perturbations can be found for any fixed input in certain ReLU networks with random weights (drawn from the Gaussian distribution). Building on Shamir et al. [2019], it was shown in Daniely and Shacham [2020] that small adversarial L 2 -perturbations can be found in random ReLU networks where each layer has vanishing width relative to the previous layer. Bubeck et al. [2021a] extended this result to two-layer neural networks without the vanishing width assumption, and Bartlett et al. [2021] extended it to a large family of ReLU networks of constant depth. Finally, Montanari and Wu [2022] provided a similar result, but with weaker assumptions on the network width and activation functions. These works aim to explain the abundance of adversarial examples in neural networks, since they imply that adversarial examples are common in random networks, and in particular in random initializations of gradient-based methods. However, trained networks are clearly not random, and properties that hold in random networks may not hold in trained networks. Our results also involve an analysis of the random initialization, but we consider the projection of the weights onto the linear subspace orthogonal to the data, and study its implications on the perturbation size required for flipping the output's sign in trained networks.

In Bubeck et al. [2021b] and Bubeck and Sellke [2021], the authors proved under certain assumptions, that overparameterization is necessary if one wants to interpolate training data using a neural network with a small Lipschitz constant. Namely, neural networks with a small number of parameters are not expressive enough to interpolate the training data while having a small Lipschitz constant. These results suggest that overparameterization might be necessary for robustness.

Vardi et al. [2022] considered a setting where the training dataset S consists of nearly-orthogonal points, and proved that every network to which gradient flow might converge is non-robust w.r.t. S . Namely, building on known properties of the implicit bias of gradient flow when training two-layer ReLU networks w.r.t. the logistic loss, they proved that for every two-layer ReLU network to which gradient flow might converge as the time t tends to infinity, and every point x i from S , it is possible to flip the output's sign with a small perturbation. We note that in Vardi et al. [2022] there is a strict limit on the number of training samples and their correlations, as well as the training duration. Here, we have no assumptions on the number of data points and their structure, besides lying on a low-dimensional subspace. Also, in Vardi et al. [2022] the adversarial perturbations are shown to exist only for samples in the training set, while here we show existence of adversarial perturbation for any sample which lies on the low-dimensional manifold.

It is widely common to assume that 'real-life data' (such as images, videos, text, etc.) lie roughly within some underlying low-dimensional data manifold. This common belief started many successful research fields such as GAN [Goodfellow et al., 2014a], VAE [Kingma and Welling, 2013], and diffusion [Sohl-Dickstein et al., 2015]. In Fawzi et al. [2018] the authors consider a setting where the high dimensional input data is generated from a low-dimensional latent space. They theoretically analyze the existence of adversarial perturbations on the manifold generated from the latent space,

although they do not bound the norm of these perturbations. Previous works analyzed adversarial perturbations orthogonal to the data manifold. For example, Khoury and Hadfield-Menell [2018] considering several geometrical properties of adversarial perturbation and adversarial training for low dimensional data manifolds. Tanay and Griffin [2016] analyzed theoretically such perturbations for linear networks, and Stutz et al. [2019] gave an empirical analysis for non-linear models. Moreover, several experimental defence methods against adversarial examples were obtained, using projection of it onto the data manifold to eliminate the component orthogonal to the data (see, e.g., Jalal et al. [2017], Meng and Chen [2017], Samangouei et al. [2018]).

Shamir et al. [2021] showed empirically on both synthetic and realistic datasets that the decision boundary of classifiers clings onto the data manifold, causing very close off-manifold adversarial examples. Our paper continues this direction, and provides theoretical guarantees for off-manifold perturbations on trained two-layer ReLU networks, in the special case where the manifold is a linear subspace.

3 Setting

Notations. We denote by U ( A ) the uniform distribution over a set A . The multivariate normal distribution with mean µ and covariance Σ is denoted by N ( µ, Σ) , and the univariate normal distribution with mean a and variance σ 2 is denoted by N ( a, σ 2 ) . The set of integers { 1 , .., m } is denoted by [ m ] . For a vector v ∈ R n , we define v i : i + j to be the j +1 coordinates of v starting from i and ending with i + j . For a vector x and a linear subspace P we denote by P ⊥ the subspace orthogonal to P , and by Π P ⊥ ( x ) the projection of x on P ⊥ . We denote by 0 the zero vector. We use I d for the identity matrix of size d .

3.1 Architecture and Training

In this paper we consider a two-layer fully-connected neural network N : R d → R with ReLU activation, input dimension d and hidden dimension m :

Here, σ ( z ) = max( z, 0) is the ReLU function and w 1: m = ( w 1 , . . . , w m ) . When w 1: m is clear from the context, we will write for short N ( x ) .

We initialize the first layer using standard Kaiming initialization [He et al., 2015], i.e. w i ∼ N ( 0 , 1 d I d ) , and the output layer as u i ∼ U ({ -1 √ m , 1 √ m }) for every i ∈ [ m ] . Note that in standard initialization, each u i would be initialized normally with a standard deviation of 1 √ m , for ease of analysis we fix the initial value to be equal to the standard deviation where only the sign is random.

We consider a dataset with binary labels. Given a training dataset ( x 1 , y 1 ) , . . . , ( x r , y r ) ∈ R d × {-1 , 1 } we train w.r.t. the logistic loss (a.k.a. binary cross entropy): L ( q ) = log(1 + e -q ) , and minimize the empirical error:

We assume throughout the paper that the network is trained using either gradient descent (GD) or stochastic gradient descent (SGD). Our results hold for both training methods. We assume that only the weights of the first layer (i.e. the w i 's) are trained, while the weights of the second layer (i.e. the u i 's) are fixed.

3.2 Assumptions on the Data

Our main assumption in this paper is that the input data lie on a low dimensional manifold, which is embedded in a high dimensional space. Specifically, we assume that this 'data manifold' is a linear subspace, denoted by P , which has dimension d -ℓ . We denote by ℓ the dimension of the

data 'off-manifold', i.e. the linear subspace orthogonal to the data subspace, which is denoted by P ⊥ . In this work we study adversarial perturbations in P ⊥ . Note that adding a perturbation from P ⊥ of any size to an input data point which changes its label is an unwanted phenomenon, because this perturbation is orthogonal to any possible data point from both possible labels. We will later show that under certain assumptions there exists an adversarial perturbation in the direction of P ⊥ which also has a small norm. This reason for this assumption is so that the projection of the first layer weights on P ⊥ remain fixed during training. An interesting question is to consider general 'data manifolds', which we elaborate on in Section 7.

To demonstrate that the low-dimensional data assumption arises in practical settings, in Figure 1 we plot the cumulative variance of the MNIST and CIFAR10 datasets, projected on a linear manifold. These are calculated by performing PCA on the entire datasets, and summing over the square of the singular values from largest to smallest. For CIFAR10, the accumulated variance reaches 90% at 98 components, and 95% at 216 components. For MNIST, the accumulated variance reaches 90% at 86 components, and 95% at 153 components. This indicates that both datasets can be projected to a much smaller linear subspace, without losing much of the information.

0

500

1000

1500

2000

2500

3000

# Components

0.4

0.6

0.8

0.9

0.95

1.0

Cumulative variance

Cumulative variance

90% percentile

95% percentile

(a)

0

100

200

300

400

500

600

700

800

# Components

0.2

0.4

0.6

0.8

0.9

0.95

1.0

Cumulative variance

Cumulative variance

90% percentile

95% percentile

(b)

Remark 3.1 (On the Margin of the Network) . Given a neural network N : R d → R and a dataset ( x 1 , y 1 ) , . . . , ( x r , y r ) with binary labels which the network label correctly, we define the margin of the network as γ := min i ∈ [ r ] y i N ( x i ) .

In our setting, it is possible to roughly estimate the margin without assuming much about the data, besides its boundedness. Note that the gradient of the loss decays exponentially with the output of the network, because ∣ ∣ ∣ ∂L ( q ) ∂q ∣ ∣ ∣ = ∣ ∣ ∣ -qe -q 1+ e -q ∣ ∣ ∣ . Hence, if we train for at most polynomially many iterations and label all the data points correctly (i.e. the margin is larger than 0 ), then training effectively stops after the margin reaches O (log 2 ( d )) . This is because if the margin is log 2 ( d ) , then the gradient is of size:

which is smaller than any polynomials in d . This means that all the data points on the margin (which consists of at least one point) will have an output of O (polylog( d )) .

The number of points which lie exactly on the margin is difficult to assess, since it may depend on both the dataset and the model. Some empirical results in this direction are given in Vardi et al. [2022], where it is observed (empirically) that for data sampled uniformly from a sphere and trained with a two-layer network, over 90% of the input samples lie on the margin. Also, in Haim et al. [2022] it is shown that for CIFAR10, a large portion of the dataset lies on the margin.

4 Large Gradient Orthogonal to the Data Subspace

One proxy for showing non-robustness of models, is to show that their gradient w.r.t. the input data is large (cf. Bubeck et al. [2021b], Bubeck and Sellke [2021]). Although a large gradient does not guarantee that there is also an adversarial perturbation, it is an indication that a small change in the input might significantly change the output. Moreover, by assuming smoothness of the model, it is possible to show that having a large gradient may suffice for having an adversarial perturbation.

In this section we show that training a network on a dataset which lies entirely on a linear subspace yields a large gradient in a direction which is orthogonal to this subspace. Moreover, the size of the gradient depends on the dimension of this subspace. Specifically, the smaller the dimension of the data subspace, the larger the gradient is in the orthogonal direction. Our main result in this section is the following:

Theorem 4.1. Suppose that a network N ( x ) = m ∑ i =1 u i σ ( ⟨ w i , x ⟩ ) is trained on a dataset which lies on a linear subspace P ⊂ R d of dimension d -ℓ for ℓ ≥ 1 , and let x 0 ∈ P . Let S = { i ∈ [ m ] : ⟨ w i , x 0 ⟩ ≥ 0 } , and let k := | S | . Then, w.p ≥ 1 -e -ℓ/ 16 (over the initialization) we have:

The full proof can be found in Appendix B. Here we provide a short proof intuition: First, we use a symmetry argument to show that it suffices to consider w.l.o.g. the subspace M := span { e 1 , . . . , e d -ℓ } , where e i are the standard unit vectors. Next, we note that since the dataset lies on M , only the first d -ℓ coordinates of each weight vector w i are trained, while the other ℓ coordinates are fixed at their initial value. Finally, using standard concentration result on Gaussian random variables we can lower bound the norm of the gradient. Note that our result shows that there might be a large gradient orthogonal to the data subspace. This correspond to 'off-manifold' adversarial examples, while the full gradient (i.e. without projecting on P ⊥ ) might be even larger.

The lower bound on the gradient depends on two terms: k m and ℓ d . The first term is the fraction of active neurons for the input x 0 , i.e. the neurons whose inputs are non-negative. Note that inactive neurons do not increase the gradient, since they do not affect the output. The second term corresponds to the fraction of directions orthogonal to the data. The larger the dimension of the orthogonal subspace, the more directions in which it is possible to perturb the input while still being orthogonal to the data. If both of these terms are constant, i.e. there is a constant fraction of active neurons, and 'off-manifold' directions, we can give a more concrete bound on the gradient:

Corollary 4.1. For ℓ = Ω( d ) , k = Ω( m ) , in the setting of Theorem 4.1, with probability ≥ 1 -e -Ω( d ) we have:

Consider the case where the norm of each data point is Θ( √ d ) = Θ( √ ℓ ) , i.e. every coordinate is of size Θ(1) . By Remark 3.1, for a point x 0 on the margin, its output is of size polylog( d ) . Therefore, for the point x 0 , gradient of size Ω(1) corresponds to an adversarial perturbation of size polylog( d ) , which is much smaller than ∥ x 0 ∥ = Θ( √ d ) . We note that this is a rough and informal estimation, since, as we already discussed, a large gradient at x 0 does not necessarily imply that an adversarial perturbation exists. In the next section, we will prove the existence of adversarial perturbations.

5 Existence of an Adversarial Perturbation

In the previous section we have shown that at any point x 0 which lies on the linear subspace of the data P , there is a large gradient in the direction of P ⊥ . In this section we show that not only the gradient is large, there also exists an adversarial perturbation in the direction of P ⊥ which changes the label of a data point from P (under certain assumptions). The main theorem of this section is the following:

Theorem 5.1. Suppose that a network N ( x ) = ∑ u i σ ( ⟨ w i , x ⟩ ) is trained on a dataset which lies on a linear subspace P ⊆ R d of dimension d -ℓ , where ℓ ≥ 32( m -1) log( m 2 d ) . Let x ∈ P ,

m i =1 0

and denote y 0 := sign ( N ( x 0 )) . Let I -:= { i ∈ [ m ] : u i < 0 } and I + := { i ∈ [ m ] : u i > 0 } , and denote k -:= |{ i ∈ I -: ⟨ w i , x 0 ⟩ ≥ 0 }| , and k + := |{ i ∈ I + : ⟨ w i , x 0 ⟩ ≥ 0 }| . Let k y 0 = k -if y 0 = 1 and k y 0 = k + if y 0 = -1 . For w ∈ R d denote ˆ w := Π P ⊥ ( w ) , and denote the perturbation

we have that ∥ z ∥ ≤ 8 √ 2 N ( x 0 ) · m k y 0 · √ d ℓ and:

̸

The full proof can be found in Appendix C. Here we give a short proof intuition: As in the previous section, we show using a symmetry argument that w.l.o.g. we can assume that P = span { e 1 , . . . , e d -ℓ } .

Now, given the perturbation z from Theorem 5.1 we want to understand how adding it to the input changes the output. Suppose that y 0 = 1 . We can write

We can see that for all i :

For i ∈ I -we can write:

and using a similar calculation, for i ∈ I + we can write:

̸

Using concentration inequalities of Gaussian random variables, and the fact that Π P ⊥ ( w i ) did not change from their initial values, we can show that:

̸

while ∥ Π P ⊥ ( w i ) ∥ 2 ≈ ℓ d . Thus, for a large enough value of ℓ we have that ⟨ w i , z ⟩ ≤ 0 for i ∈ I + and ⟨ w i , z ⟩ ≈ α · √ ℓ d · ( √ ℓ - √ m ) for i ∈ I -.

From the above calculations we can see that adding the perturbation z does not increase the output of the neurons with a positive second layer. On the other hand, adding z can only increase the input of the neurons with negative second layer, and for those neurons which are also active it increases their output as well if we assume that ℓ > m . This means, that if there are enough active neurons with a negative second layer (denoted by k -in the theorem), then the perturbation can significantly change the output. In the proof we rely only on the active negative neurons to change the label of the output (for the case of y 0 = 1 , if y 0 = -1 we rely on the active positive neurons). Note that the active positive neurons may become inactive, and the inactive negative neurons may become active. Without further assumptions it is not clear what is the size of the perturbation to make this change for every neuron. Thus, the only neurons that are guaranteed to help change the label are the active negative ones, which by our assumptions on ℓ are guaranteed to increase their output.

Note that our perturbation is not in the direction of the gradient w.r.t. the input. The direction of the gradient would be the sum of all the active neurons, i.e. the sum (with appropriate signs) over all i ∈ [ m ] such that ⟨ w i , x 0 ⟩ ≥ 0 . Our analysis would not have worked with such a perturbation, because we could not guarantee that inactive neurons would stay inactive.

̸

The assumption that ℓ = Ω( M ) (up to log factors) is a technical limitation of our proof technique. We note that such an assumption is also used in other theoretical papers about adversarial perturbations (e.g. Daniely and Shacham [2020]).

Note that the direction of the perturbation z does not depend on the input data x 0 , only its size depends on x 0 . In fact, Theorem 5.1 shows that there is a single universal direction for an adversarial perturbation that can flip the label of any data point in P . The size of the perturbation depends on the dimension of the linear subspace of the data, the number of active neurons for x 0 , the total number of neurons in the network and the size of the output. In the following corollary we give a specific bound on the size of the perturbations under assumptions on the different parameters of the problem:

Corollary 5.1. In the setting of Theorem 5.1, assume in addition that ℓ = Θ( d ) and k y 0 = Θ( m ) . Then, there exists a perturbation z such that w.p. ≥ 1 -5 ( me -ℓ/ 16 + d -1 / 2 ) we have ∥ z ∥ = O ( N ( x 0 )) and:

̸

The above corollary follows directly by noticing from Theorem 5.1 that:

where we plugged in the additional assumptions. The assumptions in the corollary above are similar to the assumptions in Corollary 4.1. Namely, that the dimension of the data subspace is a constant fraction from the dimension of the entire space, and the number of active neurons is a constant fraction of the total number of neurons. Note that here we only consider active neurons with a specific sign in the second layer.

Note that the size of the perturbation in Corollary 5.1 is bounded by N ( x 0 ) . By Remark 3.1, the output of the network for data points on the margin can be at most O (log 2 ( d )) , since otherwise the network would have essentially stopped training. Therefore, if we consider an input x 0 on the margin, and ∥ x 0 ∥ = Θ( √ d ) = Θ( √ ℓ ) , then the size of the adversarial perturbation is much smaller than ∥ x 0 ∥ . For any other point, without assuming it is on the margin, and since we do not assume anything about the training data (except for being in P ), we must assume that the size of the perturbation required to change the label will depend on the size of the output.

6 The Effects of the Initialization Scale and Regularization on Robustness

In Section 5, we presented the inherent vulnerability of trained models to small perturbations in a direction orthogonal to the data subspace. In this section, we return to a common proxy for robustness that we considered in Section 4 - the gradient at an input point x 0 . We suggest two ways that might improve the robustness of the model in the direction orthogonal to the data, by decreasing an upper bound of the gradient in this direction. We first upper bound the gradient of the model in the general case where we initialize w i ∼ N ( 0 , β 2 I d ) , and later discuss strategies to use this upper bound for improving robustness.

Theorem 6.1. Suppose that a network N ( x ) = m ∑ i =1 u i σ ( ⟨ w i , x ⟩ ) is trained on a dataset which lies on a linear subspace P ⊆ R d of dimension d -ℓ for ℓ ≥ 1 , and assume that the weights w i are initialized from N ( 0 , β 2 I d ) . Let x 0 ∈ P , let S = { i ∈ [ m ] : ⟨ w i , x 0 ⟩ ≥ 0 } , and let k = | S | . Then, w.p. ≥ 1 -e -ℓ 16 we have:

The full proof uses the same concentration bounds ideas as the lower bound proof and can be found in Appendix D. This bound is a result of the untrained weights: since the projection of the data points on P ⊥ is zero, the projection of the weights vectors on P ⊥ are not trained and are fixed at their initialization. We note that Theorem 4.1 readily extends to the case of initialization from N ( 0 , β 2 I d ) , in which case the lower bound it provides matches the upper bound from Theorem 6.1 up to a constant factor. In what follows, we suggest two ways to affect the post-training weights in the P ⊥ direction: (1) To initialize the weights vector using a smaller-variance initialization, and (2) Add an L 2 -norm regularization on the weights. We next analyze their effect on the upper bound.

6.1 Small Initialization Variance

From Theorem 6.1, one can conclude a strong result about the model's gradient without the dependency of its norm on ℓ and k .

Corollary 6.1. For β = 1 d √ 2 , in the settings of Theorem 6.1, with probability ≥ 1 -e -ℓ 16 we have

The proof follows directly from Theorem 6.1, by noticing that k ≤ m and ℓ ≤ d . Consider for example an input x 0 ∈ P with ∥ x 0 ∥ = Θ( √ d ) , and suppose that N ( x 0 ) = Ω(1) . The above corollary implies that if the initialization has a variance of 1 /d 2 (rather than the standard choice of 1 /d ) then the gradient is of size O ( 1 / √ d ) . Thus, it corresponds to perturbations of size Ω( √ d ) , which is the same order as ∥ x 0 ∥ .

6.2 L 2 Regularization

We consider another way to influence the projection onto P ⊥ of the trained weights vectors: adding L 2 regularization while training. We will update the logistic loss function by adding an additive factor 1 2 λ ∥ w 1: m ∥ 2 . For a dataset ( x 1 , y 1 ) , .., ( x r , y r ) , we now train over the following objective:

This regularization will cause the previously untrained weights to decrease in each training step which will decrease the upper bound on the projection of the gradient:

m training steps, using L 2 regularization with parameter λ ≥ 0 and step size η > 0 , on a dataset which lies on a linear subspace P ⊆ R d of dimension d -ℓ for ℓ ≥ 1 w i ∼ N ( 0 , 1 d I d ) ). Let x 0 ∈ P , let S = { i ∈ [ m ] : ⟨ w i , x 0 ⟩ ≥ 0 } , and let k := | S | . ≥ 1 -e -ℓ 16 we have

Theorem 6.2. Suppose that a network N ( x ) = ∑ i =1 u i σ ( ⟨ w i , x ⟩ ) is trained for T , starting from standard initialization (i.e., Then, w.p.

The full proof can be found in Appendix D.1. The main idea of the proof is to observe the projection of the weights on P ⊥ changing during training. As before, we assume w.l.o.g. that P = span { e 1 , . . . , e d -ℓ } and denote by ˆ w i := Π P ⊥ ( w i ) . During training, the weight vector's last ℓ coordinates are only affected by the regularization term of the loss. These weights decrease in a constant multiplicand of the previous weights. Thus, we can conclude that for every t ≥ 0 we have: ˆ w ( t ) i = (1 -ηλ ) t ˆ w (0) i , where ˆ w ( t ) i is the i -th weight vector at time t . It implies that our setting is equivalent to initializing the weights with standard deviation (1 -ηλ ) T √ d and training the model without regularization for T steps. As a result, we get the following corollary:

Corollary 6.2. For (1 -ηλ ) T ≤ 1 √ 2 d , in the settings of 6.2, w.p. ≥ 1 -e -ℓ 16 we get that:

6.3 Experiments

In this section, we present our robustness-improving experiments. 4 We explore our methods on two datasets: (1) A 7-point dataset on a one-dimensional linear subspace in a two-dimensional input

space, and (2) A 25-point dataset on a two-dimensional linear subspace in a three-dimensional input space. In Figures 2 and 3 we present the boundary of a two-layer ReLU network trained over these two datasets. We train the networks until reaching a constant positive margin. We note that unlike our theoretical analysis, in the experiments in Figure 2 we trained all layers and initialize the weights using the default PyTorch initialization, to verify that the observed phenomena occur also in this setting. In the experiment in Figure 3 we use a different initialization scale for the improving effect to be smaller and visualized easily. In Figures 2a and 3a we trained with default settings. In Figures 2b and 3b we initialized the weights using an initialization with a smaller variance (i.e., initialization divided by a constant factor). Finally, in Figures 2c and 3c we train with L 2 regularization.

(a)

(b)

(c)

1.00

1.00

1.00

0.75

0.75

0.75

0.50

0.50

0.50

0.25

0.25

0.25

0.00

00°0

0.00

0.25

0.25

0.25

0.50

0.50

0.50

0.75

0.75

0.75

1.00 +

1.00

0.750.50

0.25

0.00

0.25

0.50

0.75

1.00

1.00

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

1.00

1.00

0.75

-0.500.25

0.00

0.25

0.50

0.75

1.00

Consider the adversarial perturbation in the direction P ⊥ , orthogonal to the data subspace, in Figures 2 and 3. In figure (a) of each experiment, we can see that a rather small adversarial perturbation is needed to cross the boundary in the subspace orthogonal to the data. In the middle figure (b), we see that the boundary in the orthogonal subspace is much further. This is a direct result of the projection of the weights onto this subspace being much smaller. In the right experiment (c), we can see a similar effect created by regularization. In Appendix E we add the full default-scaled experiment in the two-dimensional setting to demonstrate the robustness effect. There, in both the small-initialization and regularization experiments, the boundary lines are almost orthogonal to the data subspace. In Appendix E we also conduct further experiments with deeper networks and standard PyTorch initialization, showing that our theoretical results are also observed empirically in settings going beyond our theory.

(a)

(b)

(c)

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.4*

0.4

0.4

0.2°

0.2

0.2

0.0

0.0

0.0

0.0

0.0

0.0

0.2

0.2

0.2

0.4

0.6

8'0

1.0 1.0

0.8

0.6

0.4

0.2

0.0

0.4

0.6

0.8

1.01.0

0.8

0.6

0.4

0.2

0.0

0.4

0.6

0.8

1.0 1.0

0.8

0.6

0.4

0.2

0.0

In Figure 4 we plot the distance from the decision boundary for different initialization scales of the first layer. We trained a 3 -layer network, initialized using standard initialization except for the first layer which is divided by the factor represented in the X -axis. After training, we randomly picked 200 points and used a standard projected gradient descent adversarial attack to change the label of

each point, which is described in the Y -axis (perturbation norm, with error bars). The datasets are: (a) Random points from a sphere with 28 dimensions, which lies in a space with 784 dimensions; and (b) MNIST, where the data is projected on 32 dimensions using PCA. The different lines are adversarial attacks projected either on data subspace, on its orthogonal subspace, or without projection. It can be seen that small initialization increases robustness off the data subspace, and also on the non-projected attack, while having almost no effect for the attacks projected on the data subspace.

(a)

(b)

16

18

17

+

Off data subspace

In all input space

On data subspace

十

In all input space

Off data subspace

On data subspace

15

14

13

Distance to Boundary

2110

Distance to Boundary

4

2

0

0.5

1.0

2.0

3.0

Factor

4.0

5.0

6.0

7.0

0

0.2

0.5

1.0

2.0

3.0

Factor

4.0

5.0

6.0

7.0

7 Conclusions and Future Work

In this paper we considered training a two-layer network on a dataset lying on P ⊆ R d where P is a d -ℓ dimensional subspace. We have shown that the gradient of any point x 0 ∈ P projected on P ⊥ is large, depending on the dimension of P and the fraction of active neurons on x 0 . We additionally showed that there exists an adversarial perturbation in the direction of P ⊥ . The size of the perturbation depends in addition on the output of the network on x 0 , which by Remark 3.1 should be poly-logarithmic in d , at least for points which lie on the margin of the network. Finally, we showed that by either decreasing the initialization scale or adding L 2 regularization we can make the network robust to 'off-manifold' perturbations, by decreasing the gradient in this direction.

One interesting question is whether our results can be generalized to other manifolds, beyond linear subspaces. We state this as an informal open problem:

Open Problem 7.1. Let M be a manifold, and D a distribution over M ×{± 1 } . Suppose we train a network N : R d → R on a dataset sampled from D . Let x 0 ∈ M , then under what conditions on M, D and N , there exists a small adversarial perturbation in the direction of ( T x 0 M ) ⊥ , i.e. orthogonal to the tangent space T x 0 M , of M at x 0 .

Our result can be seen as a special case of this conjecture, where at all points x, x ′ ∈ M , the tangent spaces are equal T x M = T x ′ M . Another future direction would be to analyze deeper networks, or different architectures such as convolutions. Finally, it would also be interesting to analyze robustness of trained networks w.r.t. different norms such as L 1 or L ∞ .

Acknowledgments and Disclosure of Funding

We thank Ohad Shamir for the many helpful discussions about this work. We would also like to thank Michal Irani for contributing computational resources. GY was supported in part by the European Research Council (ERC) grant 754705 . GV acknowledges the support of the NSF and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning.

References

Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. Preprint, arXiv:2005.10190 , 2020.

Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International conference on machine learning , pages 274-283. PMLR, 2018.

Peter Bartlett, Sébastien Bubeck, and Yeshwanth Cherapanamjeri. Adversarial examples in multilayer random relu networks. Advances in Neural Information Processing Systems , 34, 2021.

Sébastien Bubeck and Mark Sellke. A universal law of robustness via isoperimetry. Advances in Neural Information Processing Systems , 34:28811-28822, 2021.

Sébastien Bubeck, Yin Tat Lee, Eric Price, and Ilya Razenshteyn. Adversarial examples from computational constraints. In International Conference on Machine Learning , pages 831-840. PMLR, 2019.

Sébastien Bubeck, Yeshwanth Cherapanamjeri, Gauthier Gidel, and Rémi Tachet des Combes. A single gradient step finds adversarial examples on random two-layers neural networks. Advances in Neural Information Processing Systems , 34, 2021a.

Sébastien Bubeck, Yuanzhi Li, and Dheeraj M Nagaraj. A law of robustness for two-layers neural networks. In Conference on Learning Theory , pages 804-820. PMLR, 2021b.

Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security , pages 3-14, 2017.

Amit Daniely and Hadas Shacham. Most relu networks suffer from ℓ 2 adversarial perturbations. Advances in Neural Information Processing Systems , 33, 2020.

Elvis Dohmatob and Alberto Bietti. On the (non-) robustness of two-layer neural networks in different learning regimes. arXiv preprint arXiv:2203.11864 , 2022.

Alhussein Fawzi, Hamza Fawzi, and Omar Fawzi. Adversarial vulnerability for any classifier. Preprint, arXiv:1802.08686 , 2018.

Songwei Ge, Vasu Singla, Ronen Basri, and David Jacobs. Shift invariance can reduce adversarial robustness. Preprint, arXiv:2103.02695 , 2021.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014 , pages 2672-2680, 2014a. URL https://proceedings.neurips.cc/ paper/2014/hash/5ca3e9b122f61f8f06494c97b1afccf3-Abstract.html .

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. Preprint, arXiv:1412.6572 , 2014b.

Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani. Reconstructing training data from trained neural networks. arXiv preprint arXiv:2206.07758 , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Ajil Jalal, Andrew Ilyas, Constantinos Daskalakis, and Alexandros G Dimakis. The robust manifold defense: Adversarial training using generative models. arXiv preprint arXiv:1712.09196 , 2017.

Marc Khoury and Dylan Hadfield-Menell. On the geometry of adversarial examples. Preprint, arXiv:1811.00525 , 2018.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Laurent and P. Massart. Adaptive estimation of a quadratic functional by model selection. The Annals of Statistics , 28(5):1302 - 1338, 2000. doi: 10.1214/aos/1015957395. URL https: //doi.org/10.1214/aos/1015957395 .

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. Preprint, arXiv:1706.06083 , 2017.

Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. 2017.

Andrea Montanari and Yuchen Wu. Adversarial examples in random neural networks with general activations. arXiv preprint arXiv:2203.17209 , 2022.

Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP) , pages 582-597. IEEE, 2016.

Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM on Asia conference on computer and communications security , pages 506-519, 2017.

Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense-gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv:1805.06605 , 2018.

Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Adversarially robust generalization requires more data. Preprint, arXiv:1804.11285 , 2018.

Ali Shafahi, W Ronny Huang, Christoph Studer, Soheil Feizi, and Tom Goldstein. Are adversarial examples inevitable? Preprint, arXiv:1809.02104 , 2018.

Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Preprint, arXiv:2006.07710 , 2020.

Adi Shamir, Itay Safran, Eyal Ronen, and Orr Dunkelman. A simple explanation for the existence of adversarial examples with small hamming distance. Preprint, arXiv:1901.10861 , 2019.

Adi Shamir, Odelia Melamed, and Oriel BenShmuel. The dimpled manifold model of adversarial examples in machine learning. Preprint, arXiv:2106.10151 , 2021.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6976-6987, 2019.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. Preprint, arXiv:1312.6199 , 2013.

Thomas Tanay and Lewis Griffin. A boundary tilting persepective on the phenomenon of adversarial examples. arXiv preprint arXiv:1608.07690 , 2016.

Gal Vardi, Gilad Yehudai, and Ohad Shamir. Gradient methods provably converge to non-robust networks. Preprint, arXiv:2202.04347 , 2022.

Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8684-8694, 2020.

Yunjuan Wang, Enayat Ullah, Poorya Mianjy, and Raman Arora. Adversarial robustness is at odds with lazy training. arXiv preprint arXiv:2207.00411 , 2022.

Eric Wong and Zico Kolter. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International Conference on Machine Learning , pages 5286-5295. PMLR, 2018.

A Rotation Invariance w.r.t. the Initialized Weights

In this paper, we analyze neural networks trained on high-dimensional data that lies on a low dimensional linear subspace denoted by P . We assume that the dimension of P is d -ℓ . Throughout the paper it will be more convenient to analyze data which lies on the subspace M = span ( { e 1 , . . . , e d -ℓ } ) , because then the 'off manifold' directions correspond exactly to certain coordinates of the input. In this section we show that we can essentially analyze the data as if it is rotated to lie on M , and it would imply the same consequences as the original data from P .

Theorem A.1. Let P ⊆ R d be a subspace of dimension d -ℓ , and let M = span { e 1 , . . . , e d -ℓ } . Let R be an orthogonal matrix such that R · P = M , let X ⊆ P be a training dataset and let X R = { R · x : x ∈ X } . Assume we train a neural network N ( x ) = ∑ m i =1 u i σ ( w ⊤ i x ) as explained in Section 3, and denote by N X and N X R the network trained on X and X R respectively for the same number of iterations. Let x 0 ∈ P , then we have:

̸

For any c, p ≥ 0 , w.p. p (over the initialization) there exists z ∈ P ⊥ with ∥ z ∥ = c such that sign( N X ( x 0 + z )) = sign( N X ( x 0 )) , iff w.p. p there exists z ′ ∈ M ⊥ with ∥ z ′ ∥ = c such that sign( N X R ( Rx 0 + z ′ )) = sign( N X R ( Rx 0 )) .

̸

Proof. Denote by w 1: m := ( w 1 , . . . , w m ) and by R w 1: m = ( Rw 1 , . . . , Rw m ) . Let w ( t ) 1: m the weights of the network trained on the dataset X where w (0) 1: m is some initialization, and ˜ w ( t ) 1: m = ( ˜ w ( t ) 1 , . . . , ˜ w ( t ) m ) the weights of the network trained on X R and initialized at R w (0) 1: m . In the proof, when taking derivatives w.r.t. the w i 's we will explicitly write N ( x, w 1: m ) .

We first show by induction on the number of training steps that ˜ w ( t ) 1: m = R w ( t ) 1: m . For t = 0 it is clear by the assumption on the initialization. Assume it is true for t , then we have for some x ∈ X :

This is true for every i ∈ [ m ] and for every x ∈ X . Also note that by our induction assumption we have:

Finally, the derivative of the loss on a single data point x ∈ X with label y can be written as:

where the first term depends only on the value of N ( x, w ( t ) 1: m ) . Hence, taking a single gradient step of N with weights w ( t ) 1: m and dataset X will change the weights by the same term up to multiplication by R as if taking a gradient step with with weights ˜ w ( t ) 1: m and dataset X R . This finishes the induction.

̸

Let w (0) 1: m be an initialization for the training of N X , where there exists z ∈ P ⊥ with ∥ z ∥ = c such that sign( N X ( x 0 + z )) = sign( N X ( x 0 )) . Then, by Eq. (1) the initialization R w (0) 1: m for the training of N X R is such that for z ′ = Rz we have ∥ z ′ ∥ = c and sign( N X R ( Rx 0 + z ′ )) = sign( N X R ( Rx 0 )) . This argument holds also in the opposite direction. Let A ⊆ { w 1: m ∈ R d · m } be the set of all

̸

Plugging-in t = n 16 , we get

Thus, we have

Lemma B.2. Let w 1 , . . . , w m ∈ R n such that for all i ∈ [ m ] , w i ∼ N ( 0 , σ 2 I n ) , then we have:

̸

̸

initializations to N X where there exists z ∈ P ⊥ with ∥ z ∥ = c such that sign( N X ( x 0 + z )) = sign( N X ( x 0 )) , then by the above the set R · A = { R w 1: m : w 1: m ∈ A } are exactly all the initializations to N X R where there exists z ′ ∈ M ⊥ with ∥ z ′ ∥ = c such that sign( N X R ( Rx 0 + z ′ )) = sign( N X R ( Rx 0 )) . Since we initialize the w i 's using a Gaussian initialization which is spherically symmetric, we have that Pr( A ) = Pr( RA ) . This proves item (2). Item (1) follows from similar arguments (which we do not repeat for conciseness).

Under the assumption that the data lies on M = span { e 1 , . . . , e d -ℓ } , and no regularization is used, we can show that the weights of the first layer projected on M ⊥ do not change during training. This is an essential part of the proofs, as it allows us to analyze those weights as random Gaussian vectors, and apply concentration bounds on them.

Theorem A.2. Let M = span { e 1 , . . . , e d -ℓ } . Assume we train a neural network N ( x, w 1: m ) := ∑ m i =1 u i σ ( w ⊤ i x ) as explained in Section 3 (where w 1: m = ( w 1 , . . . , w m ) ). Denote by ˆ w := Π M ⊥ ( w ) for w ∈ R d , then after training, for each i ∈ [ m ] , ˆ w i did not change from their initial value.

Proof. Note that for each i ∈ [ m ] and x ∈ M we have:

Taking the derivative of the loss we have:

The above calculation did not depend on the specific value of the w i 's. Hence, the value of the ˆ w i 's for every i ∈ [ m ] did not change during training from their initial value.

B Proofs from Section 4

Before proving the main theorem, we will first need the next two lemmas about the concentration of Gaussian random variables:

Lemma B.1. Let w ∈ R n such that w ∼ N ( 0 , σ 2 I n ) . Then:

Proof. Note that ∥ ∥ w σ ∥ ∥ 2 has the Chi-squared distribution. A concentration bound by Laurent and Massart [Laurent and Massart, 2000, Lemma 1] implies that for all t > 0 we have

Proof. We denote the j -th coordinate of the vector w i ∈ R n by w i,j . Note, for any i ∈ { 1 , . . . , m } and j ∈ { 1 , . . . , n } we have w i,j ∼ N (0 , σ 2 ) . We denote by s the sum vector s := m ∑ i =1 w i , and by s j the j - th coordinate of s . By this definition, s j = m ∑ i =1 w i,j is a sum of m independent Gaussian variables and therefore also a Gaussian variable. Particularly, s ∼ N ( 0 , mσ 2 I n ) . We use Lemma B.1 with variance mσ 2 and get that:

We are now ready to prove the main theorem of this section:

Proof of Theorem 4.1. Let M = span { e 1 , . . . , e d -ℓ } . By Theorem A.1(1), given a training dataset X ⊆ P , it is enough to consider a training set X R = { Rx : x ∈ X } , where R is an orthogonal matrix such that R · P = M , and training is done over X R . From now on, we assume that the training data, as well as x 0 lie on M , and the consequences of this proof would also imply for a dataset X and x 0 ∈ P .

The projection of the gradient on M ⊥ is equal to:

Denote by ˆ w i = ( w i ) d -ℓ +1: d , the last ℓ coordinates of w i . By Theorem A.2 we get that for every i ∈ [ m ] , ˆ w i did not change from their initial value during training.

Recall that we initialized ˆ w i ∼ N ( 0 , 1 √ d I ℓ ) . Note that the set S is independent of the value of the ˆ w i 's. This is because ˆ w i does not effect the training, hence will not effect w i -Π M ⊥ ( w i ) . Also, after choosing x 0 we have ⟨ ˆ w i , ˆ x 0 ⟩ = 0 , since ˆ x 0 = 0 , which means that the choice of S is independent of the ˆ w i 's. We can conclude that the random variables ˆ w i for i ∈ S are sampled independently.

Note, since for all i ∈ { 1 , . . . , m } , | u i | = 1 √ m and they are not trained, we get that u i ˆ w i are also Gaussian random variables with the same mean, and variance multiplied by 1 m . Therefore, from Lemma B.2 we get that w.p. ≥ 1 -e -ℓ/ 16 :

C Proofs from Section 5

Before proving the main theorem, we prove a few lemmas about concentration of Gaussian random variables:

Lemma C.1. Let w ∈ R n with w ∼ N ( 0 , σ 2 I n ) . Then:

Combining the above, we get:

Proof. Note that ∥ ∥ w σ ∥ ∥ 2 has the Chi-squared distribution. A concentration bound by Laurent and Massart [Laurent and Massart, 2000, Lemma 1] implies that for all t > 0 we have

Plugging-in t = n 16 , we get

Thus, we have

Lemma C.2. Let u ∈ R n , and v ∼ N ( 0 , σ 2 I n ) . Then, for every t > 0 we have

Proof. We first consider ⟨ u ∥ u ∥ , v ⟩ . As the distribution N ( 0 , σ 2 I n ) is rotation invariant, one can rotate u and v to get ˜ u and ˜ v such that ˜ u ∥ u ∥ = e 1 , the first standard basis vector and ⟨ u ∥ u ∥ , v ⟩ = ⟨ ˜ u ∥ u ∥ , ˜ v ⟩ . Note, v and ˜ v have the same distribution. We can see that ⟨ ˜ u ∥ u ∥ , ˜ v ⟩ ∼ N (0 , σ 2 ) since it is the first coordinate of ˜ v . By a standard tail bound, we get that for t > 0 :

Therefore

Lemma C.3. Let u ∼ N ( 0 , σ 2 1 I n ) , and v ∼ N ( 0 , σ 2 2 I n ) . Then, for every t > 0 we have

Proof. Using Lemma C.1 we get that w.p. ≤ e -n/ 16 we have ∥ u ∥ ≥ σ 1 √ 2 n . Moreover, by Lemma C.2, w.p. ≤ 2 exp ( -t 2 2 σ 2 2 ) we have |⟨ u, v ⟩| ≥ ∥ u ∥ t . By the union bound, we get

We are now ready to prove the main theorem of this section:

Theorem 5.1. By Theorem A.1(2), we can assume w.l.o.g. that P = M = span { e 1 , . . . , e d -ℓ } . We also assume w.l.o.g. that y 0 = 1 , the case y 0 = -1 is proved in a similar manner. Denote by ¯ w := ( w ) d -ℓ +1: d , the last ℓ coordinates of w . By Theorem A.2 we have that ¯ w i have not changed after training from their initial value.

We can write N ( x 0 + z ) as:

where the last equality is since ( z ) 1: d -ℓ = 0 , hence ⟨ w,z ⟩ = ⟨ ¯ w, ¯ z ⟩ for every w ∈ R d . We will bound each term of the above separately.

For the first term in Eq. (2), where i ∈ I -we can write:

̸

̸

By our assumptions, ¯ w i ∼ N ( 0 , 1 d I ℓ ) and ∑ j = i sign ( u j ) ¯ w j ∼ N ( 0 , m -1 d I ℓ ) , since it is a sum of m -1 i.i.d. Gaussian random variables, which are also symmetric hence multiplying them by -1 does not change their distribution. From Lemma B.1 we get w.p. ≥ 1 -e -ℓ/ 16 that

α · ∥ ¯ w i ∥ 2 ≥ α · ℓ 2 d . From Lemma C.3, and using t = √ ( m -1) log( dm 2 ) d we get w.p. ≥ 1 -e -ℓ/ 16 +2 e -t 2 d/ 2( m -1) = 1 -e -ℓ/ 16 +2 m -1 d -1 / 2 that

̸

Applying union bound over the above two events, and for every i ∈ I -, we get w.p. ≥ 1 -2 ( me -ℓ/ 16 + d -1 / 2 ) that:

For the second term in Eq. (2), where i ∈ I + we can write in a similar way:

̸

Using the same argument as above, we get w.p ≥ 1 -2 ( me -ℓ/ 16 + d -1 / 2 ) that:

By assuming that ℓ ≥ 8( m -1) log( m 2 d ) we get that ⟨ ¯ w i , z ⟩ ≤ 0 . Denote C := αℓ 2 d -α d √ 2 ℓ ( m -1) log( m 2 d ) , then going back to Eq. (2), using the above bounds and applying union bound, we get w.p. ≥ 1 -4 ( me -ℓ/ 16 + d -1 / 2 ) that:

Define F -:= { i ∈ I -: ⟨ w i , x 0 ⟩ ≥ 0 } , and k -= | F -| . We have that:

where the first inequality is since we only sum over negative terms, and the second inequality is since both ⟨ w i , x 0 ⟩ ≥ 0 (because i ∈ F -) and C ≥ 0 (because ℓ ≥ 32( m -1) log( m 2 d ) ). Combining all of the above, we get that:

By our assumption that ℓ ≥ 32( m -1) log( m 2 d ) we have that

Plugging in C and α = 8 √ mdN ( x 0 ) k -ℓ to Eq. (4) we get that:

̸

and in particular sign ( N ( x 0 )) = sign ( N ( x 0 + z )) .

We are left with calculating the norm of z :

Since for each i ∈ [ m ] , ¯ w i ∼ N ( 0 , 1 d I ℓ ) , then -sign ( u i ) ¯ w i also have the same distribution, because this is a symmetric distribution. Hence, ∑ m i =1 -sign ( u i ) ¯ w i ∼ N ( 0 , m d I ℓ ) as a sum of Gaussian random variables. Using Lemma C.1 we get w.p ≥ 1 -e -ℓ/ 16 that ∥ ∑ m i =1 -sign ( u i ) ¯ w i ∥ 2 ≤ 2 mℓ d . Plugging in α we get that:

D Proofs for Section 6

For proving the main theorem, we will use the following lemma that upper bounds the norm of a sum of Gaussian random variables:

Lemma D.1. Let w 1 , .., w m ∈ R n such that for all i ∈ [ m ] , w i ∼ N ( 0 , σ 2 I n ) , then we have:

Proof. We denote the j -th coordinate of the vector w i ∈ R n by w i,j . Note, for any i ∈ [ m ] and j ∈ [ n ] we have w i,j ∼ N (0 , σ 2 ) . We denote by s the sum vector s := m ∑ i =1 w i , and by s j the j -th coordinate of s . By this definition, s j = m ∑ i =1 w i,j is a sum of m independent Gaussian variables and therefore also a Gaussian variable. Therefore, s ∼ N ( 0 , mσ 2 I n ) . We use Lemma C.1 with variance mσ 2 and get that:

We now prove the main theorem of this section:

Proof of Theorem 6.1. Similar to the lower bound of the norm, let M = span { e 1 , . . . , e d -ℓ } . By Theorem A.1(1), given a training dataset X ⊆ P , it is enough to consider a training set X R = { Rx : x ∈ X } , where R is an orthogonal matrix such that R · P = M , and training is done over X R . From now on, we assume that the training data, as well as x 0 lie on M , and the consequences of this proof would also imply for a dataset X and x 0 ∈ P .

The projection of the gradient on M ⊥ is equal to:

Denote by ˆ w i = ( w i ) d -ℓ +1: d , the last ℓ coordinates of w i . By Theorem A.2 we get that for every i ∈ [ m ] , ˆ w i did not change from their initial value during training.

Recall that we initialized ˆ w i ∼ N ( 0 , β 2 I ℓ ) . Note that the set S is independent of the value of the ˆ w i 's. This is because ˆ w i does not effect the training, hence will not effect w i -Π M ⊥ ( w i ) . Also, after choosing x 0 we have ⟨ ˆ w i , ˆ x 0 ⟩ = 0 , since ˆ x 0 = 0 , which means that the choice of S is independent of the ˆ w i 's. We can conclude that the random variables ˆ w i for i ∈ S are sampled independently.

Therefore, from Lemma B.2 we get that w.p. ≥ 1 -e -ℓ/ 16 :

Note, since for all i ∈ [ m ] , | u i | = 1 √ m and they are not trained, we get w.p. ≥ 1 -e -ℓ/ 16 that:

D.1 Explicit L 2 regularization

Proof of Theorem 6.2. As before, for this proof we rotate the data subspace P to lie on M = span { e 1 , . . . , e d -ℓ } and rotate the model's weights accordingly. For a dataset ( x 1 , y 1 ) , .., ( x r , y r ) , we train over the following objective:

In Theorem A.2, we showed for all ( x j , y j ) that if we train the model using the loss L we get:

Now, we analyze the training process using the new loss which includes the regularization term. We denote by w ( t ) i the weight vector w i after t training steps, and by ˆ w ( t ) i := Π M ⊥ ( w ( t ) i ) its projection on the subspace orthogonal to M . We look at the projected gradient of w ( t ) i w.r.t. the loss:

For a training step of size η , using gradient descent we get that:

Thus, after a total of T iteration of training we get that:

Therefore, the projection of gradients after training onto P ⊥ will be the same as if they were initialized to ∼ N ( 0 , (1 -ηλ ) 2 T d I d ) and trained using logistic loss without regularization. The rest of the proof is the same as Theorem 6.1 for β = (1 -ηλ ) T √ d .

E Further Experiments and Experimental Details

E.1 Further Experiments

In Figure 5 we present the boundary of a two-layer ReLU network trained over a 25 -point dataset on a two-dimensional linear subspace, similar to Figure 3. We train the networks until reaching a constant positive margin. The difference between the figures is that in Figure 5 we initialize the weights using the default PyTorch initialization, while in Figure 3 we initialized using a smaller scale

for the robustness effect to be smaller, and visualized more easily. The experiment in Figure 5 is demonstrating an extreme robustness effect, occurring when using the standard settings.

(a)

(b)

(c)

1.0

1.0

1.0

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0七

-1.00

-0.75

0.50

0.25

0.00

0.25

0.50

0.75

-1.00

0.75

-0.50

0.25

0.00

0.25

0.50

0.75

1.001.00

0.75

0.50

o0.250.000.20.500.752.00

-1.00

0.75

-0.50

0.25

0.00

0.25

0.50

0.75

In Figure 6 we go beyond the theory discussed in this paper, and present similar phenomena in all three settings for a five-layer ReLU network. In Figure 6a we can see the boundary of the regularly trained network within a small distance in P ⊥ from the data points. In Figure 6b we use small initialization for all five layers, and present a boundary almost orthogonal to the data manifold. In Figure 6c, the boundary of a regularized trained network is in a similar form. This experiment suggests that our theoretical results might be extended also to deeper networks, where all layers are trained.

(a)

(b)

(c)

1.00

1.00

1.00

0.75

0.75

0.75

0.50

0.50

0.50

0.25

0.25

0.25

0.00

0.00

0.00

0.25

0.25

0.25

0.50

0.50

0.50

0.75

0.75

0.75

1.00

00'-

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

1.00

1.00

0.75

0.500.25

0.00

0.25

0.50

0.75

1.00

1.00

1.00

0.75

0.50

0.25

00°0

0.25

0.50

0.75

1.00

E.2 One-dimensional dataset experiment - 2 layer network (Figure 2)

Dataset For all the three experiments we used a 7-point data set, spread equally on the two dimensional line y = x from ( -1 , -1) to (1 , 1) .

Network For all the three experiments we used two-layer ReLU network of width 100 with biases in both layers. The weights of both layers were initialized using (1+3) default PyTorch initialization for linear layers, (2) default initialization divided by 3 .

Training We used train step of size 0 . 02 for (1+3) and 0 . 04 for (2). We trained both layers until the margin reached 0 . 3 . The losses we used were (1+2) Logistic loss, (3) Logistic loss with 0 . 005 L 2 regularization.

E.3 Two-dimensional dataset experiment - smaller effect (Figure 3)

Dataset For all the three experiments we used a 25-point data set, spread equally on a grid which lies on the z = 0 . 5 axis.

Network For all the three experiments we used two-layer ReLU network of width 4000 with biases in both layers. The weights in the first layer were initialized in (1+3) from N ( 0 , 1 / 3 I 3 ) , and in (2) from N ( 0 , 1 / 36 I 3 ) . The weight of the output layer were initialized to the uniform distribution over the set {-1 , 1 } .

Training For all the experiments we trained both layers until the margin reached 0 . 3 and we used train step of size 0 . 002 . The losses we used were (1+2) Logistic loss, (3) Logistic loss with 0 . 8 L 2 regularization on the weights of the first layer.

E.4 Two-dimensional dataset experiment (Figure 5)

Dataset For all the three experiments we used a 25-point data set, spread equally on a grid which lies on the x -y axis.

Network For all the three experiments we used two-layer RelU network of width 400 with biases in both layers. The weights in any layer were initialized using (1+3) default PyTorch initialization for linear layers, (2) default initialization divided by 3 .

Training For (1) experiments we used train step of size 0 . 005 , and for (2+3) we used step of size 0 . 05 . We trained both layers until the margin reached 0 . 1 . The losses we used were (1+2) Logistic loss, (3) Logistic loss with 0 . 005 L 2 regularization.

E.5 One-dimensional dataset experiment - 5 layer network (Figure 6)

Dataset For all the three experiments we used a 7-point data set, spread equally on the two dimensional line y = x from ( -1 , -1) to (1 , 1) .

Network For all the three experiments we used 5-layer RelU network of width 100 with biases in all layers. The weights in any layer were initialized using (1+3) default PyTorch initialization for linear layers, (2) default initialization divided by 3 .

Training For (1+3) experiments we used train step of size 0 . 02 , and for (2) we used step of size 0 . 06 . we trained all layers until the margin reached 0 . 3 . The losses we used were (1+2) Logistic loss, (3) Logistic loss with 0 . 01 L 2 regularization.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates adversarial robustness of trained ReLU neural networks when data lies within a linear subspace.  For the theoretical part, which is the bulk of the paper, the networks have two layers and only the first layer is trained.  Then the key observation is that the assumption on the data causes the training to change only the projections of the first-layer weight vectors to the linear subspace, i.e. their components in the orthogonal subspace remain as at initialisation.  Based on this, the authors prove several results, including a lower bound on the projection to the orthogonal subspace of the gradient of the network output at any point in the linear subspace, existence of a direction in the orthogonal subspace for universal adversarial perturbations, and that either reducing the initialisation scale or using L2 regularisation can reduce the norm of the projections to the orthogonal subspace of the gradients (in the latter case, the projections of the first-layer weight vectors to the orthogonal subspace are changed during training only by the L2 regulariser).  The results contains some assumptions, most notably for the universal adversarial perturbation, the width of the network has to be roughly at most a constant fraction of the input dimension.  The theory is supplemented by experiments on small examples in dimensions 2 and 3; in them all layers are trained and a network with five layers is also considered, which suggests that it may be possible to extend the theoretical results beyond the assumptions in the paper.\n\nSTRENGTHS:\nThe paper is a nice mostly theoretical follow-up to the mostly empirical \"The Dimpled Manifold Model of Adversarial Examples in Machine Learning\" by Shamir, Melamed, and BenShmuel (CoRR abs/2106.10151), which in particular brings in the considerations of the effects of initialisation scale and L2 regularisation.  The theorems are proved in detail in the appendix, with helpful sketches provided in the main.  The experiments and the resulting figures aid the intuition, and suggest directions for future theoretical work.\n\nWEAKNESSES:\nThe proofs of the theoretical results are perhaps not surprising or difficult.\n\nI believe the code for the experiments is relatively simple and essentially given by the text in Appendix E, however it was not submitted with the supplementary materials.\n\nQUESTIONS:\nAre you able to say how the theoretical results would change if the second-layer weights were trained as well?\n\nLIMITATIONS:\nThe assumption of small network width in relation to the input dimension is restrictive, although a similar assumption featured in the related paper by Daniely and Shacham (NeurIPS 2020).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the vulnerability of two-layer ReLU networks under adversarial attack when the data is in a low-dimensional manifold. The paper also observes that adding L2 regularization in clean training also improves the adversarial robustness.\n\nSTRENGTHS:\nThe intuition of this paper is interesting, with a high quality and a good clarity. Although existing literature considers that the vulnerability of clean training is caused by the off-manifold attack, this is the first one to study the theoretical perspective in detail.\n\nWEAKNESSES:\n(1) My major concern is that this paper mainly studies the side off the data manifold, but not the side on the data manifold. Some claims need more justifications. For example, in the abstract, the authors mention that \"the standard gradient methods lead to non-robust neural networks, .. and are susceptible to small adversarial L2 perturbations in these directions\". To claim that the perturbation is \"small\", the authors need to provide a result on what is the smallest on-manifold attack strength which results in an incorrect prediction. It is essential to make comparison in order to claim a quantity is \"small\", i.e., while Theorem 4.1 presents the L2 norm of the off-manifold gradient, what is the on-manifold gradient? \n\n(2) For the numerical experiments, please also share the code or report the observations when the training iteration T is large enough so that the loss converges to zero. In the paper\n\nBa, J., Erdogdu, M., Suzuki, T., Wu, D., & Zhang, T. (2020, April). Generalization of two-layer neural networks: An asymptotic viewpoint. In International conference on learning representations.\n\nit is observed that training a neural network in regression in their setting is the same as a penalized linear regression, i.e., the training already introduces some sort of regularization. It would be great if the authors of this submission can provide the code or some more experimental results to support the claims in this submission.\n\nDue to concern (1) and (2), my current score is 5. I like the intuition of this paper so I will consider raising my score if these major concerns can be addressed properly. For (1), a full theoretical proof may takes too much time. A brief justification would be helpful enough.\n\nSome other comments:\n\n(3) While Theorem 6.2 provides justifications on how L2 regularization improves adversarial robustness, it would be helpful if the authors can also provide the convergence and the generalization performance of the regularized neural network, which may potentially implies a trade-off between clean performance and adversarial robustness.\n\nQUESTIONS:\nPlease address my concerns mentioned in the weakness section.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focus on the data lies on a low dimensional data manifold (linear subspace P). There’re no additional assumptions on the number of data and their structure (orthogonality). The paper considers the perturbation on P^\\orth space. The paper claims that standard gradient descent leads to non-robust neural network; while decreasing the initialization scale, adding L2 regularizer can make the trained network more robust.\n\nSTRENGTHS:\n1. Studying the adversarial perturbation on the low-dimensional subspace is an interesting problem. \n2. The paper is well-written and easy to follow. The mathematical derivation seems sound.\n\nWEAKNESSES:\n1. This paper specifically assumes the dataset lies on a linear subspace, and the perturbation lies in the orthogonal direction of the subspace. The analysis of thm4.1 still relies on part of the weight being unchanged after training, and the gradient lower bound depends on the unchanged weights. To my understanding, such analysis still relies on random initialization property and therefore not much different compared to previous work, which might previous work analysis also holds under this paper’s low dimensionality assumption. To that end, the perturbation lies in the orthogonal direction of the subspace this constraint is trying to bypass the gradient update algorithm. In other words, does it mean that theorem 4.1 still holds even if you consider adversarial training?\n\n2. Although the author motivates the idea by saying real-world dataset mostly lie in low-dimensional subspace, the experiments are based on extremely simple synthetic data. I’m wondering whether similar results hold for a simple dataset like MNIST.\n\nQUESTIONS:\n1. In Theorem 5.1 or Corollary 5.1, the perturbation size depends on the output size. However, in classification problems, you can always rescale the predictor ($w,u$) while maintaining the sign of prediction to be unchanged. To that end, $N(x_0)$ can be arbitrarily small, implying the size of perturbation can be arbitrarily small, regardless of the size of $k_{y_0}$ and $\\ell$. Then the result is a bit counterintuitive.\n\n2. I'm wondering whether theorem 4.1 lower bound can also depend on the network initialization so that it matches theorem 6.1.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
The paper shows that on two-layer neural networks trained using data which lie on a low dimensional linear subspace that standard gradient methods lead to non-robust neural networks, that networks which have large gradients in directions orthogonal to the data subspace, and are susceptible to small adversarial L2-perturbations in these directions. The paper shows that decreasing the initialization scale of the training algorithm and adding L2 regularization, can make the trained network more robust to adversarial perturbations orthogonal to the data.

STRENGTHS:
The paper is well written and organized, and is innovative and original. The structure of the paper is clear and rigorous.

WEAKNESSES:
The experiments are insufficient, the number of data points used to evaluate the proposed method is few.

QUESTIONS:
1.Does the proposed method which decreasing the gradient in directions orthogonal to the data subspace hurt the accuracy on clean data?
2.The paper claim that large gradients exists in directions orthogonal to the data subspace, and are susceptible to small adversarial L2-perturbations in these directions, do other adversarial perturbations such as L1 or L_{\infty} also exist in directions orthogonal to the data subspace?
3.Is the proposed method effective in deeper networks?
4.small typo errors,e.g. line 287,extra )

LIMITATIONS:
The number of data points is few, it's better to evaluate on more data points.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07