SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Odelia Melamed *
Gilad Yehudai†
Gal Vardi†
Despite a great deal of research, it is still not well-understood why trained neural networks are highly vulnerable to adversarial examples. In this work we focus on two-layer neural networks trained using data which lie on a low dimensional linear subspace. We show that standard gradient methods lead to non-robust neural networks, namely, networks which have large gradients in directions orthogonal to the data subspace, and are susceptible to small adversarial  $L_{2}$ -perturbations in these directions. Moreover, we show that decreasing the initialization scale of the training algorithm, or adding  $L_{2}$  regularization, can make the trained network more robust to adversarial perturbations orthogonal to the data.
Neural networks are observed to be susceptible to adversarial perturbations [Szegedy et al., 2013], often imperceptible by humans. Many works have shown attacks, where adding a very small perturbation to the input may change the prediction of the network [Carlini and Wagner, 2017, Papernot et al., 2017, Athalye et al., 2018]. Other works have shown defense mechanisms called adversarial training [Papernot et al., 2016, Madry et al., 2017, Wong and Kolter, 2018]. Despite a great deal of research, it is still not well-understood why neural-network training methods tend towards such non-robust solutions.
Several recent works have given theoretical explanation for the existence of adversarial perturbations under different settings. One line of work [Daniely and Shacham, 2020, Bubeck et al., 2021a, Bartlett et al., 2021, Montanari and Wu, 2022] have shown that random networks are susceptible to adversarial perturbations. These results might explain why neural networks are non-robust at initialization, but they do not explain why trained neural networks are non-robust. Recently, Vardi et al. [2022] showed that for data which is nearly orthogonal, after training for infinitely many iterations, the implicit bias of neural networks towards margin maximization leads to non-robust solutions. Despite these works, it is still unclear why trained neural networks tend to be non-robust to adversarial perturbations, and specifically what are the assumptions on the input data which leads to non-robustness.
One common belief about "real-life" datasets, is that they approximately lie on a low dimensional "data-manifold" in a high dimensional space. In this setting, the existence of perturbations orthogonal to the data-manifold that change the network's predictions is especially undesired, since such perturbations do not make the input closer to data points from other classes. Indeed, such a perturbation only increases the distance between the input and all "real-life" examples. Shamir et al. [2021] have demonstrated empirically that under such a data-manifold assumption, the decision boundary of a trained classifier clings to the data-manifold in a way that even very small perturbations orthogonal to the manifold can change the prediction.
*Weizmann Institute of Science, Israel, odelia.melamed@weizmann.ac.il
†Weizmann Institute of Science, Israel, gilad.yehudai@weizmann.ac.il
$^{\ddagger}$ TTI-Chicago and the Hebrew University of Jerusalem, galvardi@ttic.edu.
In this paper we focus on data which lies on a low dimensional "data-manifold". Specifically, we assume that the data lies on a linear subspace  $P \subseteq \mathbb{R}^d$  of dimension  $d - \ell$  for some  $\ell > 0$ . We study adversarial perturbations in the direction of  $P^\perp$ , i.e. orthogonal to the data subspace. We show that the gradient projected on  $P^\perp$  is large, and in addition there exist a universal adversarial perturbation in a direction orthogonal to  $P$ . Namely, the same small adversarial perturbation applies to many inputs. The norm of the gradient depends on the term  $\frac{\ell}{d}$ , while the perturbation size depends on the term  $\frac{d}{\ell}$ , i.e. a low dimensional subspace implies reduced adversarial robustness. Finally, we also study how changing the initialization scale or adding  $L_2$  regularization affects robustness. We show that in our setting, decreasing the initialization scale, or adding a sufficiently large regularization term, can make the network significantly more robust. We also demonstrate empirically the effects of the initialization scale and regularization on the decision boundary. Our experiments suggest that these effects might extend to deeper networks.
Despite extensive research, the reasons for the abundance of adversarial examples in trained neural networks are still not well understood [Goodfellow et al., 2014b, Fawzi et al., 2018, Shafahi et al., 2018, Schmidt et al., 2018, Khoury and Hadfield-Menell, 2018, Bubeck et al., 2019, Allen-Zhu and Li, 2020, Wang et al., 2020, Shah et al., 2020, Shamir et al., 2021, Ge et al., 2021, Wang et al., 2022, Dohmatob and Bietti, 2022]. Below we discuss several prior works on this question.
In a string of works, it was shown that small adversarial perturbations can be found for any fixed input in certain ReLU networks with random weights (drawn from the Gaussian distribution). Building on Shamir et al. [2019], it was shown in Daniely and Shacham [2020] that small adversarial  $L_{2}$ -perturbations can be found in random ReLU networks where each layer has vanishing width relative to the previous layer. Bubeck et al. [2021a] extended this result to two-layer neural networks without the vanishing width assumption, and Bartlett et al. [2021] extended it to a large family of ReLU networks of constant depth. Finally, Montanari and Wu [2022] provided a similar result, but with weaker assumptions on the network width and activation functions. These works aim to explain the abundance of adversarial examples in neural networks, since they imply that adversarial examples are common in random networks, and in particular in random initializations of gradient-based methods. However, trained networks are clearly not random, and properties that hold in random networks may not hold in trained networks. Our results also involve an analysis of the random initialization, but we consider the projection of the weights onto the linear subspace orthogonal to the data, and study its implications on the perturbation size required for flipping the output's sign in trained networks.
In Bubeck et al. [2021b] and Bubeck and Sellke [2021], the authors proved under certain assumptions, that overparameterization is necessary if one wants to interpolate training data using a neural network with a small Lipschitz constant. Namely, neural networks with a small number of parameters are not expressive enough to interpolate the training data while having a small Lipschitz constant. These results suggest that overparameterization might be necessary for robustness.
Vardi et al. [2022] considered a setting where the training dataset  $S$  consists of nearly-orthogonal points, and proved that every network to which gradient flow might converge is non-robust w.r.t.  $S$ . Namely, building on known properties of the implicit bias of gradient flow when training two-layer ReLU networks w.r.t. the logistic loss, they proved that for every two-layer ReLU network to which gradient flow might converge as the time  $t$  tends to infinity, and every point  $\mathbf{x}_i$  from  $S$ , it is possible to flip the output's sign with a small perturbation. We note that in Vardi et al. [2022] there is a strict limit on the number of training samples and their correlations, as well as the training duration. Here, we have no assumptions on the number of data points and their structure, besides lying on a low-dimensional subspace. Also, in Vardi et al. [2022] the adversarial perturbations are shown to exist only for samples in the training set, while here we show existence of adversarial perturbation for any sample which lies on the low-dimensional manifold.
It is widely common to assume that "real-life data" (such as images, videos, text, etc.) lie roughly within some underlying low-dimensional data manifold. This common belief started many successful research fields such as GAN [Goodfellow et al., 2014a], VAE [Kingma and Welling, 2013], and diffusion [Sohl-Dickstein et al., 2015]. In Fawzi et al. [2018] the authors consider a setting where the high dimensional input data is generated from a low-dimensional latent space. They theoretically analyze the existence of adversarial perturbations on the manifold generated from the latent space,
although they do not bound the norm of these perturbations. Previous works analyzed adversarial perturbations orthogonal to the data manifold. For example, Khoury and Hadfield-Menell [2018] considering several geometrical properties of adversarial perturbation and adversarial training for low dimensional data manifolds. Tanay and Griffin [2016] analyzed theoretically such perturbations for linear networks, and Stutz et al. [2019] gave an empirical analysis for non-linear models. Moreover, several experimental defence methods against adversarial examples were obtained, using projection of it onto the data manifold to eliminate the component orthogonal to the data (see, e.g., Jalal et al. [2017], Meng and Chen [2017], Samangouei et al. [2018]).
Shamir et al. [2021] showed empirically on both synthetic and realistic datasets that the decision boundary of classifiers clings onto the data manifold, causing very close off-manifold adversarial examples. Our paper continues this direction, and provides theoretical guarantees for off-manifold perturbations on trained two-layer ReLU networks, in the special case where the manifold is a linear subspace.
3 Setting
Notations. We denote by  $\mathcal{U}(A)$  the uniform distribution over a set  $A$ . The multivariate normal distribution with mean  $\mu$  and covariance  $\Sigma$  is denoted by  $\mathcal{N}(\mu, \Sigma)$ , and the univariate normal distribution with mean  $a$  and variance  $\sigma^2$  is denoted by  $\mathcal{N}(a, \sigma^2)$ . The set of integers  $\{1,..,m\}$  is denoted by  $[m]$ . For a vector  $v \in \mathbb{R}^n$ , we define  $v_{i:i+j}$  to be the  $j+1$  coordinates of  $v$  starting from  $i$  and ending with  $i+j$ . For a vector  $x$  and a linear subspace  $P$  we denote by  $P^\perp$  the subspace orthogonal to  $P$ , and by  $\Pi_{P^\perp}(x)$  the projection of  $x$  on  $P^\perp$ . We denote by  $\mathbf{0}$  the zero vector. We use  $I_d$  for the identity matrix of size  $d$ .
3.1 Architecture and Training
In this paper we consider a two-layer fully-connected neural network  $N:\mathbb{R}^d\to \mathbb{R}$  with ReLU activation, input dimension  $d$  and hidden dimension  $m$ :
$$
N (x, \mathbf {w} _ {1: m}) = \sum_ {i = 1} ^ {m} u _ {i} \sigma \left(w _ {i} ^ {\top} x\right).
$$
Here,  $\sigma(z) = \max(z, 0)$  is the ReLU function and  $\mathbf{w}_{1:m} = (w_1, \ldots, w_m)$ . When  $\mathbf{w}_{1:m}$  is clear from the context, we will write for short  $N(x)$ .
We initialize the first layer using standard Kaiming initialization [He et al., 2015], i.e.  $w_{i} \sim \mathcal{N}\left(0, \frac{1}{d} I_{d}\right)$ , and the output layer as  $u_{i} \sim \mathcal{U}\left(\left\{-\frac{1}{\sqrt{m}}, \frac{1}{\sqrt{m}}\right\}\right)$  for every  $i \in [m]$ . Note that in standard initialization, each  $u_{i}$  would be initialized normally with a standard deviation of  $\frac{1}{\sqrt{m}}$ , for ease of analysis we fix the initial value to be equal to the standard deviation where only the sign is random.
We consider a dataset with binary labels. Given a training dataset  $(x_{1},y_{1}),\ldots ,(x_{r},y_{r})\in \mathbb{R}^{d}\times$ $\{-1,1\}$  we train w.r.t. the logistic loss (a.k.a. binary cross entropy):  $L(q) = \log (1 + e^{-q})$  , and minimize the empirical error:
$$
\min  _ {w _ {1}, \dots , w _ {m}} \sum_ {i = 1} ^ {r} L \left(y _ {i} \cdot N \left(x _ {i}, \mathbf {w} _ {1: m}\right)\right).
$$
We assume throughout the paper that the network is trained using either gradient descent (GD) or stochastic gradient descent (SGD). Our results hold for both training methods. We assume that only the weights of the first layer (i.e. the  $w_{i}$ 's) are trained, while the weights of the second layer (i.e. the  $u_{i}$ 's) are fixed.
3.2 Assumptions on the Data
Our main assumption in this paper is that the input data lie on a low dimensional manifold, which is embedded in a high dimensional space. Specifically, we assume that this "data manifold" is a linear subspace, denoted by  $P$ , which has dimension  $d - \ell$ . We denote by  $\ell$  the dimension of the
data "off-manifold", i.e. the linear subspace orthogonal to the data subspace, which is denoted by  $P^{\perp}$ . In this work we study adversarial perturbations in  $P^{\perp}$ . Note that adding a perturbation from  $P^{\perp}$  of any size to an input data point which changes its label is an unwanted phenomenon, because this perturbation is orthogonal to any possible data point from both possible labels. We will later show that under certain assumptions there exists an adversarial perturbation in the direction of  $P^{\perp}$  which also has a small norm. This reason for this assumption is so that the projection of the first layer weights on  $P^{\perp}$  remain fixed during training. An interesting question is to consider general "data manifolds", which we elaborate on in Section 7.
To demonstrate that the low-dimensional data assumption arises in practical settings, in Figure 1 we plot the cumulative variance of the MNIST and CIFAR10 datasets, projected on a linear manifold. These are calculated by performing PCA on the entire datasets, and summing over the square of the singular values from largest to smallest. For CIFAR10, the accumulated variance reaches  $90\%$  at 98 components, and  $95\%$  at 216 components. For MNIST, the accumulated variance reaches  $90\%$  at 86 components, and  $95\%$  at 153 components. This indicates that both datasets can be projected to a much smaller linear subspace, without losing much of the information.
(b)
Figure 1: The cumulative variance for the (a) CIFAR10; and (b) MNIST datasets, calculated by performing PCA on the entire datasets, and summing over the square of the singular values from largest to smallest.
Remark 3.1 (On the Margin of the Network). Given a neural network  $N: \mathbb{R}^d \to \mathbb{R}$  and a dataset  $(x_1, y_1), \ldots, (x_r, y_r)$  with binary labels which the network label correctly, we define the margin of the network as  $\gamma := \min_{i \in [r]} y_i N(x_i)$ .
In our setting, it is possible to roughly estimate the margin without assuming much about the data, besides its boundedness. Note that the gradient of the loss decays exponentially with the output of the network, because  $\left|\frac{\partial L(q)}{\partial q}\right| = \left|\frac{-qe^{-q}}{1 + e^{-q}}\right|$ . Hence, if we train for at most polynomially many iterations and label all the data points correctly (i.e. the margin is larger than 0), then training effectively stops after the margin reaches  $O(\log^2(d))$ . This is because if the margin is  $\log^2(d)$ , then the gradient is of size:
$$
\left| L ^ {\prime} (\log^ {2} (d)) \right| = \left| \frac {- \log^ {2} (d) e ^ {- \log^ {2} (d)}}{1 + e ^ {- \log^ {2} (d)}} \right| \leq \log^ {2} (d) \cdot d ^ {- \log (d)},
$$
which is smaller than any polynomials in  $d$ . This means that all the data points on the margin (which consists of at least one point) will have an output of  $O(\mathrm{polylog}(d))$ .
The number of points which lie exactly on the margin is difficult to assess, since it may depend on both the dataset and the model. Some empirical results in this direction are given in Vardi et al. [2022], where it is observed (empirically) that for data sampled uniformly from a sphere and trained with a two-layer network, over  $90\%$  of the input samples lie on the margin. Also, in Haim et al. [2022] it is shown that for CIFAR10, a large portion of the dataset lies on the margin.
4 Large Gradient Orthogonal to the Data Subspace
One proxy for showing non-robustness of models, is to show that their gradient w.r.t. the input data is large (cf. Bubeck et al. [2021b], Bubeck and Sellke [2021]). Although a large gradient does not guarantee that there is also an adversarial perturbation, it is an indication that a small change in the input might significantly change the output. Moreover, by assuming smoothness of the model, it is possible to show that having a large gradient may suffice for having an adversarial perturbation.
In this section we show that training a network on a dataset which lies entirely on a linear subspace yields a large gradient in a direction which is orthogonal to this subspace. Moreover, the size of the gradient depends on the dimension of this subspace. Specifically, the smaller the dimension of the data subspace, the larger the gradient is in the orthogonal direction. Our main result in this section is the following:
Theorem 4.1. Suppose that a network  $N(x) = \sum_{i=1}^{m} u_i \sigma(\langle w_i, x \rangle)$  is trained on a dataset which lies on a linear subspace  $P \subset \mathbb{R}^d$  of dimension  $d - \ell$  for  $\ell \geq 1$ , and let  $x_0 \in P$ . Let  $S = \{i \in [m] : \langle w_i, x_0 \rangle \geq 0\}$ , and let  $k := |S|$ . Then,  $w.p \geq 1 - e^{-\ell / 16}$  (over the initialization) we have:
$$
\left\| \Pi_ {P ^ {\perp}} \left(\frac {\partial N (x _ {0})}{\partial x}\right) \right\| \geq \sqrt {\frac {k \ell}{2 m d}}.
$$
The full proof can be found in Appendix B. Here we provide a short proof intuition: First, we use a symmetry argument to show that it suffices to consider w.l.o.g. the subspace  $M \coloneqq \operatorname{span}\{e_1, \ldots, e_{d - \ell}\}$ , where  $e_i$  are the standard unit vectors. Next, we note that since the dataset lies on  $M$ , only the first  $d - \ell$  coordinates of each weight vector  $w_i$  are trained, while the other  $\ell$  coordinates are fixed at their initial value. Finally, using standard concentration result on Gaussian random variables we can lower bound the norm of the gradient. Note that our result shows that there might be a large gradient orthogonal to the data subspace. This corresponds to "off-manifold" adversarial examples, while the full gradient (i.e. without projecting on  $P^\perp$ ) might be even larger.
The lower bound on the gradient depends on two terms:  $\frac{k}{m}$  and  $\frac{\ell}{d}$ . The first term is the fraction of active neurons for the input  $x_0$ , i.e. the neurons whose inputs are non-negative. Note that inactive neurons do not increase the gradient, since they do not affect the output. The second term corresponds to the fraction of directions orthogonal to the data. The larger the dimension of the orthogonal subspace, the more directions in which it is possible to perturb the input while still being orthogonal to the data. If both of these terms are constant, i.e. there is a constant fraction of active neurons, and "off-manifold" directions, we can give a more concrete bound on the gradient:
Corollary 4.1. For  $\ell = \Omega(d)$ ,  $k = \Omega(m)$ , in the setting of Theorem 4.1, with probability  $\geq 1 - e^{-\Omega(d)}$  we have:
$$
\left\| \Pi_ {P ^ {\perp}} \left(\frac {\partial N (x _ {0})}{\partial x}\right) \right\| = \Omega (1).
$$
Consider the case where the norm of each data point is  $\Theta (\sqrt{d}) = \Theta (\sqrt{\ell})$ , i.e. every coordinate is of size  $\Theta (1)$ . By Remark 3.1, for a point  $x_0$  on the margin, its output is of size polylog(d). Therefore, for the point  $x_0$ , gradient of size  $\Omega (1)$  corresponds to an adversarial perturbation of size polylog(d), which is much smaller than  $\| x_0\| = \Theta (\sqrt{d})$ . We note that this is a rough and informal estimation, since, as we already discussed, a large gradient at  $x_0$  does not necessarily imply that an adversarial perturbation exists. In the next section, we will prove the existence of adversarial perturbations.
5 Existence of an Adversarial Perturbation
In the previous section we have shown that at any point  $x_0$  which lies on the linear subspace of the data  $P$ , there is a large gradient in the direction of  $P^{\perp}$ . In this section we show that not only the gradient is large, there also exists an adversarial perturbation in the direction of  $P^{\perp}$  which changes the label of a data point from  $P$  (under certain assumptions). The main theorem of this section is the following:
Theorem 5.1. Suppose that a network  $N(x) = \sum_{i=1}^{m} u_i \sigma(\langle w_i, x \rangle)$  is trained on a dataset which lies on a linear subspace  $P \subseteq \mathbb{R}^d$  of dimension  $d - \ell$ , where  $\ell \geq 32(m - 1) \log(m^2 d)$ . Let  $x_0 \in P$ ,
and denote  $y_0 \coloneqq \text{sign}(N(x_0))$ . Let  $I_- \coloneqq \{i \in [m] : u_i < 0\}$  and  $I_+ \coloneqq \{i \in [m] : u_i > 0\}$ , and denote  $k_- \coloneqq |\{i \in I_- : \langle w_i, x_0 \rangle \geq 0\}|$ , and  $k_+ \coloneqq |\{i \in I_+ : \langle w_i, x_0 \rangle \geq 0\}|$ . Let  $k_{y_0} = k_-$  if  $y_0 = 1$  and  $k_{y_0} = k_+$  if  $y_0 = -1$ . For  $w \in \mathbb{R}^d$  denote  $\hat{w} \coloneqq \Pi_{P^\perp}(w)$ , and denote the perturbation
$$
z := y _ {0} \cdot \alpha \left(\sum_ {i \in I _ {-}} \hat {w} _ {i} - \sum_ {i \in I _ {+}} \hat {w} _ {i}\right) w h e r e \alpha = \frac {8 \sqrt {m} d N \left(x _ {0}\right)}{\ell k _ {y _ {0}}}. T h e n, w. p. \geq 1 - 5 (m e ^ {- \ell / 1 6} + d ^ {- 1 / 2})
$$
we have that  $\| z\| \leq 8\sqrt{2} N(x_0)\cdot \frac{m}{k_{y_0}}\cdot \sqrt{\frac{d}{\ell}}$  and:
$$
\operatorname {s i g n} \left(N \left(x _ {0} + z\right)\right) \neq \operatorname {s i g n} \left(N \left(x _ {0}\right)\right).
$$
The full proof can be found in Appendix C. Here we give a short proof intuition: As in the previous section, we show using a symmetry argument that w.l.o.g. we can assume that  $P = \operatorname{span}\{e_1, \ldots, e_{d - \ell}\}$ .
Now, given the perturbation  $z$  from Theorem 5.1 we want to understand how adding it to the input changes the output. Suppose that  $y_0 = 1$ . We can write
$$
N \left(x _ {0} + z\right) = \sum_ {i \in I _ {-}} u _ {i} \sigma \left(\langle w _ {i}, x _ {0} \rangle + \langle w _ {i}, z \rangle\right) + \sum_ {i \in I _ {+}} u _ {i} \sigma \left(\langle w _ {i}, x _ {0} \rangle + \langle w _ {i}, z \rangle\right)
$$
We can see that for all  $i$ :
$$
\begin{array}{l} \langle w _ {i}, z \rangle = \alpha \cdot \langle w _ {i}, \sum_ {j \in I _ {-}} \Pi_ {P ^ {\perp}} (w _ {j}) - \sum_ {j \in I _ {+}} \Pi_ {P ^ {\perp}} (w _ {j}) \rangle \\ = - \alpha \cdot \left\langle w _ {i}, \sum_ {j = 1} ^ {m} \operatorname {s i g n} \left(u _ {j}\right) \Pi_ {P ^ {\perp}} \left(w _ {j}\right) \right\rangle . \\ \end{array}
$$
For  $i \in I_{-}$  we can write:
$$
\langle w _ {i}, z \rangle = \alpha \left\| \Pi_ {P ^ {\perp}} \left(w _ {i}\right) \right\| ^ {2} - \alpha \langle \Pi_ {P ^ {\perp}} \left(w _ {i}\right), \sum_ {j \neq i} \operatorname {s i g n} \left(u _ {j}\right) \Pi_ {P ^ {\perp}} \left(w _ {j}\right) \rangle ,
$$
and using a similar calculation, for  $i \in I_{+}$  we can write:
$$
\langle w _ {i}, z \rangle = - \alpha \left\| \Pi_ {P ^ {\perp}} (w _ {i}) \right\| ^ {2} - \alpha \langle \Pi_ {P ^ {\perp}} (w _ {i}), \sum_ {j \neq i} \operatorname {s i g n} (u _ {j}) \Pi_ {P ^ {\perp}} (w _ {j}) \rangle .
$$
Using concentration inequalities of Gaussian random variables, and the fact that  $\Pi_{P^{\perp}}(w_i)$  did not change from their initial values, we can show that:
$$
\left| \langle \Pi_ {P ^ {\perp}} (w _ {i}), \sum_ {j \neq i} \operatorname {s i g n} (u _ {j}) \Pi_ {P ^ {\perp}} (w _ {j}) \rangle \right| \approx \frac {\sqrt {\ell m}}{d},
$$
while  $\| \Pi_{P^{\perp}}(w_i)\| ^2\approx \frac{\ell}{d}$ . Thus, for a large enough value of  $\ell$  we have that  $\langle w_i,z\rangle \leq 0$  for  $i\in I_{+}$  and  $\langle w_i,z\rangle \approx \alpha \cdot \frac{\sqrt{\ell}}{d}\cdot (\sqrt{\ell} -\sqrt{m})$  for  $i\in I_{-}$ .