SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {P} \left[ \| w \| ^ {2} \leq \frac {1}{2} \sigma^ {2} n \right] \leq e ^ {- \frac {n}{1 6}}.
$$
Proof. Note that  $\left\| \frac{w}{\sigma} \right\|^2$  has the Chi-squared distribution. A concentration bound by Laurent and Massart [Laurent and Massart, 2000, Lemma 1] implies that for all  $t > 0$  we have
$$
\Pr \left[ n - \left\| \frac {w}{\sigma} \right\| ^ {2} \geq 2 \sqrt {n t} \right] \leq e ^ {- t}.
$$
Plugging-in  $t = \frac{n}{16}$ , we get
$$
\Pr \left[ n - \left\| \frac {w}{\sigma} \right\| ^ {2} \geq \frac {1}{2} n \right] = \Pr \left[ \left\| \frac {w}{\sigma} \right\| ^ {2} \leq \frac {1}{2} n \right] \leq e ^ {- n / 1 6}.
$$
Thus, we have
$$
\Pr \left[ \| w \| \leq \sigma \sqrt {\frac {n}{2}} \right] \leq e ^ {- n / 1 6}.
$$
Lemma B.2. Let  $w_{1},\ldots ,w_{m}\in \mathbb{R}^{n}$  such that for all  $i\in [m]$ ,  $w_{i}\sim \mathcal{N}(\mathbf{0},\sigma^{2}I_{n})$ , then we have:
$$
\mathbb {P} \left[ \left\| \sum_ {i = 1} ^ {m} w _ {i} \right\| ^ {2} \leq \frac {1}{2} m \sigma^ {2} n \right] \leq e ^ {- \frac {n}{1 6}}.
$$
Proof. We denote the  $j$ -th coordinate of the vector  $w_{i} \in \mathbb{R}^{n}$  by  $w_{i,j}$ . Note, for any  $i \in \{1, \ldots, m\}$  and  $j \in \{1, \ldots, n\}$  we have  $w_{i,j} \sim \mathcal{N}(0, \sigma^{2})$ . We denote by  $s$  the sum vector  $s := \sum_{i=1}^{m} w_{i}$ , and by  $s_{j}$  the  $j$ -th coordinate of  $s$ . By this definition,  $s_{j} = \sum_{i=1}^{m} w_{i,j}$  is a sum of  $m$  independent Gaussian variables and therefore also a Gaussian variable. Particularly,  $s \sim \mathcal{N}(\mathbf{0}, m\sigma^{2}I_{n})$ . We use Lemma B.1 with variance  $m\sigma^{2}$  and get that:
We are now ready to prove the main theorem of this section:
Proof of Theorem 4.1. Let  $M = \operatorname{span}\{e_1, \ldots, e_{d - \ell}\}$ . By Theorem A.1(1), given a training dataset  $X \subseteq P$ , it is enough to consider a training set  $X_R = \{Rx : x \in X\}$ , where  $R$  is an orthogonal matrix such that  $R \cdot P = M$ , and training is done over  $X_R$ . From now on, we assume that the training data, as well as  $x_0$  lie on  $M$ , and the consequences of this proof would also imply for a dataset  $X$  and  $x_0 \in P$ .
The projection of the gradient on  $M^{\perp}$  is equal to:
$$
\Pi_ {M ^ {\perp}} \left(\frac {\partial N (x _ {0})}{\partial x}\right) = \Pi_ {M ^ {\perp}} \left(\sum_ {i = 1} ^ {m} u _ {i} w _ {i} \mathbb {1} _ {\langle w _ {i}, x _ {0} \rangle \geq 0}\right) = \sum_ {i = 1} ^ {m} \Pi_ {M ^ {\perp}} \left(u _ {i} w _ {i}\right) \mathbb {1} _ {i \in S} = \sum_ {i \in S} \Pi_ {M ^ {\perp}} \left(u _ {i} w _ {i}\right).
$$
Denote by  $\hat{w}_i = (w_i)_{d - \ell +1:d}$ , the last  $\ell$  coordinates of  $w_{i}$ . By Theorem A.2 we get that for every  $i\in [m]$ ,  $\hat{w}_i$  did not change from their initial value during training.
Recall that we initialized  $\hat{w}_i\sim \mathcal{N}(\mathbf{0},\frac{1}{\sqrt{d}} I_\ell)$ . Note that the set  $S$  is independent of the value of the  $\hat{w}_i$ 's. This is because  $\hat{w}_i$  does not effect the training, hence will not effect  $w_{i} - \Pi_{M^{\perp}}(w_{i})$ . Also, after choosing  $x_0$  we have  $\langle \hat{w}_i,\hat{x}_0\rangle = 0$ , since  $\hat{x}_0 = \mathbf{0}$ , which means that the choice of  $S$  is independent of the  $\hat{w}_i$ 's. We can conclude that the random variables  $\hat{w}_i$  for  $i\in S$  are sampled independently.
Note, since for all  $i \in \{1, \ldots, m\}$ ,  $|u_i| = \frac{1}{\sqrt{m}}$  and they are not trained, we get that  $u_i \hat{w}_i$  are also Gaussian random variables with the same mean, and variance multiplied by  $\frac{1}{m}$ . Therefore, from Lemma B.2 we get that w.p.  $\geq 1 - e^{-\ell / 16}$ :
$$
\left\| \sum_ {i \in S} u _ {i} \hat {w} _ {i} \right\| \geq \sqrt {\frac {1}{2}} \sqrt {\frac {k l}{d m}}.
$$
Combining the above, we get:
$$
\left\| \Pi_ {M ^ {\perp}} \left(\frac {\partial N (x _ {0})}{\partial x}\right) \right\| \geq \sqrt {\frac {1}{2}} \sqrt {\frac {k l}{d m}}.
$$
C Proofs from Section 5
Before proving the main theorem, we prove a few lemmas about concentration of Gaussian random variables:
Lemma C.1. Let  $w \in \mathbb{R}^n$  with  $w \sim \mathcal{N}(0, \sigma^2 I_n)$ . Then:
$$
\Pr \left[ \| w \| ^ {2} \geq 2 \sigma^ {2} n \right] \leq e ^ {- \frac {n}{1 6}}.
$$
$$
\operatorname * {P r} \left[ \left\| \frac {w}{\sigma} \right\| ^ {2} - n \geq 2 \sqrt {n t} + 2 t \right] \leq e ^ {- t}.
$$
$$
\Pr \left[ \left\| \frac {w}{\sigma} \right\| ^ {2} \geq 2 n \right] \leq \Pr \left[ \left\| \frac {w}{\sigma} \right\| ^ {2} - n \geq n / 2 + n / 8 \right] \leq e ^ {- n / 1 6}.
$$
$$
\Pr \left[ \| w \| \geq \sigma \sqrt {2 n} \right] \leq e ^ {- n / 1 6}.
$$
Lemma C.2. Let  $u \in \mathbb{R}^n$ , and  $v \sim \mathcal{N}(\mathbf{0}, \sigma^2 I_n)$ . Then, for every  $t > 0$  we have
$$
\operatorname * {P r} \left[ | \langle u, v \rangle | \geq \| u \| t \right] \leq 2 \exp \left(- \frac {t ^ {2}}{2 \sigma^ {2}}\right).
$$
Proof. We first consider  $\langle \frac{u}{\|u\|}, v \rangle$ . As the distribution  $\mathcal{N}(\mathbf{0}, \sigma^2 I_n)$  is rotation invariant, one can rotate  $u$  and  $v$  to get  $\tilde{u}$  and  $\tilde{v}$  such that  $\frac{\tilde{u}}{\|u\|} = e_1$ , the first standard basis vector and  $\langle \frac{u}{\|u\|}, v \rangle = \langle \frac{\tilde{u}}{\|u\|}, \tilde{v} \rangle$ . Note,  $v$  and  $\tilde{v}$  have the same distribution. We can see that  $\langle \frac{\tilde{u}}{\|u\|}, \tilde{v} \rangle \sim \mathcal{N}(0, \sigma^2)$  since it is the first coordinate of  $\tilde{v}$ . By a standard tail bound, we get that for  $t > 0$ :
$$
\operatorname * {P r} \left[ | \langle \frac {u}{\| u \|}, v \rangle | \geq t \right] = \operatorname * {P r} \left[ | \langle \frac {\tilde {u}}{\| u \|}, \tilde {v} \rangle | \geq t \right] = \operatorname * {P r} \left[ | \tilde {v} _ {1} | \geq t \right] \leq 2 \exp \left(- \frac {t ^ {2}}{2 \sigma^ {2}}\right).
$$
$$
\Pr \left[ | \langle u, v \rangle | \geq \| u \| t \right] \leq 2 \exp \left(- \frac {t ^ {2}}{2 \sigma^ {2}}\right).
$$
Lemma C.3. Let  $u\sim \mathcal{N}(\mathbf{0},\sigma_1^2 I_n)$ , and  $v\sim \mathcal{N}(\mathbf{0},\sigma_2^2 I_n)$ . Then, for every  $t > 0$  we have
$$
\operatorname * {P r} \left[ | \langle u, v \rangle | \geq \sigma_ {1} \sqrt {2 n} t \right] \leq e ^ {- n / 1 6} + 2 e ^ {- t ^ {2} / 2 \sigma_ {2} ^ {2}}.
$$
Proof. Using Lemma C.1 we get that w.p.  $\leq e^{-n / 16}$  we have  $\| u\| \geq \sigma_1\sqrt{2n}$ . Moreover, by Lemma C.2, w.p.  $\leq 2\exp \left(-\frac{t^2}{2\sigma_2^2}\right)$  we have  $|\langle u,v\rangle |\geq \| u\| t$ . By the union bound, we get
$$
\operatorname * {P r} \left[ | \langle u, v \rangle | \geq \sigma_ {1} \sqrt {2 n} t \right] \leq \operatorname * {P r} \left[ \| u \| \geq \sigma_ {1} \sqrt {2 n} \right] + \operatorname * {P r} \left[ | \langle u, v \rangle | \geq \| u \| t \right] \leq e ^ {- n / 1 6} + 2 \exp \left(- \frac {t ^ {2}}{2 \sigma_ {2} ^ {2}}\right).
$$
Theorem 5.1. By Theorem A.1(2), we can assume w.l.o.g. that  $P = M = \operatorname{span}\{e_1, \ldots, e_{d - \ell}\}$ . We also assume w.l.o.g. that  $y_0 = 1$ , the case  $y_0 = -1$  is proved in a similar manner. Denote by  $\bar{w} := (w)_{d - \ell + 1:d}$ , the last  $\ell$  coordinates of  $w$ . By Theorem A.2 we have that  $\bar{w}_i$  have not changed after training from their initial value.
We can write  $N(x_0 + z)$  as:
$$
\begin{array}{l} N (x _ {0} + z) = \sum_ {i = 1} ^ {m} u _ {i} \sigma (\langle w _ {i}, x _ {0} \rangle + \langle w _ {i}, z \rangle) \\ = \sum_ {i \in I _ {-}} u _ {i} \sigma (\langle w _ {i}, x _ {0} \rangle + \langle w _ {i}, z \rangle) + \sum_ {i \in I _ {+}} u _ {i} \sigma (\langle w _ {i}, x _ {0} \rangle + \langle w _ {i}, z \rangle) \\ = \sum_ {i \in I _ {-}} u _ {i} \sigma \left(\langle w _ {i}, x _ {0} \rangle + \langle \bar {w} _ {i}, \bar {z} \rangle\right) + \sum_ {i \in I _ {+}} u _ {i} \sigma \left(\langle w _ {i}, x _ {0} \rangle + \langle \bar {w} _ {i}, \bar {z} \rangle\right) \tag {2} \\ \end{array}
$$
where the last equality is since  $(z)_{1:d - \ell} = 0$ , hence  $\langle w,z\rangle = \langle \bar{w},\bar{z}\rangle$  for every  $w\in \mathbb{R}^d$ . We will bound each term of the above separately.
For the first term in Eq. (2), where  $i \in I_{-}$  we can write:
$$
\langle \bar {w} _ {i}, \bar {z} \rangle = \alpha \| \bar {w} _ {i} \| ^ {2} + \alpha \langle \bar {w} _ {i}, \sum_ {j \neq i} \operatorname {s i g n} (u _ {j}) \bar {w} _ {j} \rangle .
$$
By our assumptions,  $\bar{w}_i\sim \mathcal{N}\left(\mathbf{0},\frac{1}{d} I_\ell\right)$  and  $\sum_{j\neq i}\mathrm{sign}(u_j)\bar{w}_j\sim \mathcal{N}\left(\mathbf{0},\frac{m - 1}{d} I_\ell\right)$ , since it is a sum of  $m - 1$  i.i.d. Gaussian random variables, which are also symmetric hence multiplying them by  $-1$  does not change their distribution. From Lemma B.1 we get w.p.  $\geq 1 - e^{-\ell /16}$  that
$$
\alpha \cdot \| \bar {w} _ {i} \| ^ {2} \geq \alpha \cdot \frac {\ell}{2 d}.
$$
From Lemma C.3, and using  $t = \sqrt{\frac{(m - 1)\log(dm^2)}{d}}$  we get w.p.  $\geq 1 - e^{-\ell /16} + 2e^{-t^2 d / 2(m - 1)} = 1 - e^{-\ell /16} + 2m^{-1}d^{-1 / 2}$  that
$$
\begin{array}{l} \langle \bar {w} _ {i}, \sum_ {j \neq i} \operatorname {s i g n} (u _ {j}) \bar {w} _ {j} \rangle \leq \frac {1}{\sqrt {d}} t \sqrt {2 \ell} \\ = \frac {1}{d} \cdot \sqrt {2 \ell (m - 1) \log \left(m ^ {2} d\right)}. \tag {3} \\ \end{array}
$$
Applying union bound over the above two events, and for every  $i \in I_{-}$ , we get w.p.  $\geq 1 - 2\left(me^{-\ell /16} + d^{-1 / 2}\right)$  that:
$$
\langle \bar {w} _ {i}, \bar {z} \rangle \geq \frac {\alpha \ell}{2 d} - \frac {\alpha}{d} \sqrt {2 \ell (m - 1) \log (m ^ {2} d)}.
$$
For the second term in Eq. (2), where  $i \in I_{+}$  we can write in a similar way:
$$
\langle \bar {w} _ {i}, \bar {z} \rangle = - \alpha \left\| \bar {w} _ {i} \right\| ^ {2} + \alpha \langle \bar {w} _ {i}, \sum_ {j \neq i} \operatorname {s i g n} (u _ {j}) \bar {w} _ {j} \rangle .
$$
Using the same argument as above, we get  $\mathrm{w.p} \geq 1 - 2\left(me^{-\ell /16} + d^{-1 / 2}\right)$  that:
$$
\langle \bar {w} _ {i}, \bar {z} \rangle \leq - \frac {\alpha \ell}{2 d} + \frac {\alpha}{d} \sqrt {2 \ell (m - 1) \log (m ^ {2} d)}.
$$
By assuming that  $\ell \geq 8(m - 1)\log (m^2 d)$  we get that  $\langle \bar{w}_i,z\rangle \leq 0$ . Denote  $C\coloneqq \frac{\alpha\ell}{2d} -\frac{\alpha}{d}\sqrt{2\ell(m - 1)\log(m^2d)}$ , then going back to Eq. (2), using the above bounds and applying union bound, we get w.p.  $\geq 1 - 4$  ( $me^{-\ell /16} + d^{-1 / 2}$ ) that:
$$
\begin{array}{l} N (x _ {0} + z) \leq \sum_ {i \in I _ {-}} u _ {i} \sigma (\langle w _ {i}, x _ {0} \rangle + C) + \sum_ {i \in I _ {+}} u _ {i} \sigma (\langle w _ {i}, x _ {0} \rangle) \\ = \sum_ {i \in I _ {-}} u _ {i} \sigma (\langle w _ {i}, x _ {0} \rangle + C) + \sum_ {i \in I _ {+}} u _ {i} \sigma (\langle w _ {i}, x _ {0} \rangle) + \sum_ {i \in I _ {-}} u _ {i} \sigma (\langle w _ {i}, x _ {0} \rangle) - \sum_ {i \in I _ {-}} u _ {i} \sigma (\langle w _ {i}, x _ {0} \rangle) \\ = \sum_ {i \in I _ {-}} u _ {i} \sigma (\langle w _ {i}, x _ {0} \rangle + C) - \sum_ {i \in I _ {-}} u _ {i} \sigma (\langle w _ {i}, x _ {0} \rangle) + N (x _ {0}) \\ = \sum_ {i \in I _ {-}} u _ {i} \left(\sigma \left(\langle w _ {i}, x _ {0} \rangle + C\right) - \sigma \left(\langle w _ {i}, x _ {0} \rangle\right)\right) + N \left(x _ {0}\right). \\ \end{array}
$$
Define  $F_{-} \coloneqq \{i \in I_{-} : \langle w_{i}, x_{0} \rangle \geq 0\}$ , and  $k_{-} = |F_{-}|$ . We have that:
$$
\begin{array}{l} \sum_ {i \in I _ {-}} u _ {i} \left(\sigma \left(\langle w _ {i}, x _ {0} \rangle + C\right) - \sigma \left(\langle w _ {i}, x _ {0} \rangle\right)\right) \leq \sum_ {i \in F _ {-}} u _ {i} \left(\sigma \left(\langle w _ {i}, x _ {0} \rangle + C\right) - \sigma \left(\langle w _ {i}, x _ {0} \rangle\right)\right) \\ = \sum_ {i \in F _ {-}} u _ {i} C = - \frac {k _ {-} C}{\sqrt {m}}, \\ \end{array}
$$
where the first inequality is since we only sum over negative terms, and the second inequality is since both  $\langle w_i,x_0\rangle \geq 0$  (because  $i\in F_{-}$ ) and  $C\geq 0$  (because  $\ell \geq 32(m - 1)\log (m^2 d)$ ). Combining all of the above, we get that:
$$
N \left(x _ {0} + z\right) \leq - \frac {k _ {-} C}{\sqrt {m}} + N \left(x _ {0}\right). \tag {4}
$$
By our assumption that  $\ell \geq 32(m - 1)\log (m^2 d)$  we have that
$$
\begin{array}{l} C = \alpha \left(\frac {1}{2} \frac {\ell}{d} - \sqrt {2} \sqrt {m - 1} \frac {\sqrt {\ell}}{d} \sqrt {\log (d m ^ {2})}\right) \\ = \frac {\alpha \sqrt {\ell}}{d} \left(\frac {\sqrt {\ell}}{2} - \sqrt {2 (m - 1) \log (m ^ {2} d)}\right) \\ \geq \frac {\alpha \ell}{4 d}. \\ \end{array}
$$
Plugging in  $C$  and  $\alpha = \frac{8\sqrt{m}dN(x_0)}{k_\ell}$  to Eq. (4) we get that:
$$
\begin{array}{l} N (x _ {0} + z) \leq - \frac {k _ {-} C}{\sqrt {m}} + N (x _ {0}) \\ \leq - \frac {k _ {-}}{\sqrt {m}} \cdot \frac {\ell}{4 d} \cdot \frac {8 \sqrt {m} d N (x _ {0})}{k _ {-} \ell} + N (x _ {0}) = - N (x _ {0}) <   0, \\ \end{array}
$$
and in particular  $\mathrm{sign}(N(x_0))\neq \mathrm{sign}(N(x_0 + z))$
We are left with calculating the norm of  $z$ :
$$
\begin{array}{l} \| z \| = \alpha \cdot \left\| \sum_ {i \in I _ {-}} \Pi_ {M ^ {\perp}} \left(w _ {i}\right) - \sum_ {i \in I _ {+}} \Pi_ {M ^ {\perp}} \left(w _ {i}\right) \right\| \\ = \alpha \cdot \left\| \sum_ {i = 1} ^ {m} - \operatorname {s i g n} \left(u _ {i}\right) \Pi_ {M ^ {\perp}} \left(w _ {i}\right) \right\| \\ = \alpha \cdot \left\| \sum_ {i = 1} ^ {m} - \operatorname {s i g n} \left(u _ {i}\right) \bar {w} _ {i} \right\|. \\ \end{array}
$$
Since for each  $i \in [m]$ ,  $\bar{w}_i \sim \mathcal{N}\left(\mathbf{0}, \frac{1}{d} I_\ell\right)$ , then  $-\mathrm{sign}(u_i) \bar{w}_i$  also have the same distribution, because this is a symmetric distribution. Hence,  $\sum_{i=1}^{m} -\mathrm{sign}(u_i) \bar{w}_i \sim \mathcal{N}\left(\mathbf{0}, \frac{m}{d} I_\ell\right)$  as a sum of Gaussian random variables. Using Lemma C.1 we get w.p  $\geq 1 - e^{-\ell/16}$  that  $\| \sum_{i=1}^{m} -\mathrm{sign}(u_i) \bar{w}_i \|^2 \leq \frac{2m\ell}{d}$ . Plugging in  $\alpha$  we get that:
$$
\| z \| \leq \sqrt {\frac {2 m \ell}{d}} \cdot \frac {8 \sqrt {m} d N (x _ {0})}{k _ {-} \ell} = 8 \sqrt {2} N (x _ {0}) \cdot \frac {m}{k _ {-}} \cdot \sqrt {\frac {d}{\ell}}.
$$
D Proofs for Section 6
For proving the main theorem, we will use the following lemma that upper bounds the norm of a sum of Gaussian random variables:
Lemma D.1. Let  $w_{1},\ldots ,w_{m}\in \mathbb{R}^{n}$  such that for all  $i\in [m]$ ,  $w_{i}\sim \mathcal{N}(\mathbf{0},\sigma^{2}I_{n})$ , then we have:
$$
\mathbb {P} \left[ \left\| \sum_ {i = 1} ^ {m} w _ {i} \right\| ^ {2} \geq 2 m \sigma^ {2} n \right] \leq e ^ {- \frac {n}{1 6}}
$$
Proof. We denote the  $j$ -th coordinate of the vector  $w_{i} \in \mathbb{R}^{n}$  by  $w_{i,j}$ . Note, for any  $i \in [m]$  and  $j \in [n]$  we have  $w_{i,j} \sim \mathcal{N}(0, \sigma^{2})$ . We denote by  $s$  the sum vector  $s := \sum_{i=1}^{m} w_{i}$ , and by  $s_{j}$  the  $j$ -th coordinate of  $s$ . By this definition,  $s_{j} = \sum_{i=1}^{m} w_{i,j}$  is a sum of  $m$  independent Gaussian variables and therefore also a Gaussian variable. Therefore,  $s \sim \mathcal{N}(\mathbf{0}, m\sigma^{2}I_{n})$ . We use Lemma C.1 with variance  $m\sigma^{2}$  and get that:
$$
\mathbb {P} \left[ \left\| \sum_ {i = 1} ^ {m} w _ {i} \right\| ^ {2} \geq 2 m \sigma^ {2} n \right] \leq e ^ {- \frac {n}{1 6}}.
$$
We now prove the main theorem of this section:
Proof of Theorem 6.1. Similar to the lower bound of the norm, let  $M = \operatorname{span}\{e_1, \ldots, e_{d - \ell}\}$ . By Theorem A.1(1), given a training dataset  $X \subseteq P$ , it is enough to consider a training set  $X_R = \{Rx : x \in X\}$ , where  $R$  is an orthogonal matrix such that  $R \cdot P = M$ , and training is done over  $X_R$ . From now on, we assume that the training data, as well as  $x_0$  lie on  $M$ , and the consequences of this proof would also imply for a dataset  $X$  and  $x_0 \in P$ .
Recall that we initialized  $\hat{w}_i\sim \mathcal{N}(\mathbf{0},\beta^2 I_\ell)$ . Note that the set  $S$  is independent of the value of the  $\hat{w}_i$ 's. This is because  $\hat{w}_i$  does not effect the training, hence will not effect  $w_{i} - \Pi_{M^{\perp}}(w_{i})$ . Also, after choosing  $x_0$  we have  $\langle \hat{w}_i,\hat{x}_0\rangle = 0$ , since  $\hat{x}_0 = \mathbf{0}$ , which means that the choice of  $S$  is independent of the  $\hat{w}_i$ 's. We can conclude that the random variables  $\hat{w}_i$  for  $i\in S$  are sampled independently.
Therefore, from Lemma B.2 we get that w.p.  $\geq 1 - e^{-\ell /16}$
$$
\left\| \sum_ {i \in S} \hat {w} _ {i} \right\| \leq \beta \sqrt {2 k \ell}.
$$
Note, since for all  $i \in [m]$ ,  $|u_i| = \frac{1}{\sqrt{m}}$  and they are not trained, we get w.p.  $\geq 1 - e^{-\ell / 16}$  that:
$$
\left\| \Pi_ {M ^ {\perp}} \left(\frac {\partial N (x _ {0})}{\partial x}\right) \right\| \leq \beta \sqrt {\frac {2 k \ell}{m}}.
$$
D.1 Explicit  $L_{2}$  regularization
Proof of Theorem 6.2. As before, for this proof we rotate the data subspace  $P$  to lie on  $M = \operatorname{span}\{e_1, \ldots, e_{d - \ell}\}$  and rotate the model's weights accordingly. For a dataset  $(x_1, y_1), \ldots, (x_r, y_r)$ , we train over the following objective:
$$
\sum_ {j = 1} ^ {r} L \left(y _ {j} \cdot N \left(x _ {j}, \mathbf {w} _ {1: m}\right)\right)) + \frac {1}{2} \lambda \left\| \mathbf {w} _ {1: m} \right\| ^ {2}
$$
In Theorem A.2, we showed for all  $(x_{j},y_{j})$  that if we train the model using the loss  $L$  we get:
$$
\Pi_ {M ^ {\perp}} \left(\frac {\partial L \left(N (x _ {j} , \mathbf {w} _ {1 : m}) \cdot y _ {j}\right)}{\partial w _ {i}}\right) = 0
$$
Now, we analyze the training process using the new loss which includes the regularization term. We denote by  $w_{i}^{(t)}$  the weight vector  $w_{i}$  after  $t$  training steps, and by  $\hat{w}_{i}^{(t)} \coloneqq \Pi_{M^{\perp}}\left(w_{i}^{(t)}\right)$  its projection on the subspace orthogonal to  $M$ . We look at the projected gradient of  $w_{i}^{(t)}$  w.r.t. the loss:
$$
\begin{array}{l} \Pi_ {M ^ {\perp}} \left(\frac {\partial \sum_ {j = 1} ^ {r} L \left(N (x _ {j} , \mathbf {w} _ {1 : m} ^ {(t)}) \cdot y _ {j}\right)}{\partial w _ {i}} + \frac {\partial \frac {1}{2} \lambda \left\| w _ {i} ^ {(t)} \right\| ^ {2}}{\partial w _ {i}}\right) = \\ = \sum_ {j = 1} ^ {r} \Pi_ {M ^ {\perp}} \left(\frac {\partial L \left(N \left(x _ {j} , \mathbf {w} _ {1 : m} ^ {(t)}\right) \cdot y _ {j}\right)}{\partial w _ {i}}\right) + \Pi_ {M ^ {\perp}} \left(\frac {\partial \frac {1}{2} \lambda \left\| w _ {i} ^ {(t)} \right\| ^ {2}}{\partial w _ {i}}\right) \\ = \Pi_ {M ^ {\perp}} \left(\frac {\partial^ {\frac {1}{2}} \lambda \left\| w _ {i} ^ {(t)} \right\| ^ {2}}{\partial w _ {i}}\right) \\ = \Pi_ {M ^ {\perp}} \left(\lambda w _ {i} ^ {(t)}\right) \\ = \lambda \hat {w _ {i}} ^ {(t)}. \\ \end{array}
$$
For a training step of size  $\eta$ , using gradient descent we get that:
$$
\hat {w} _ {i} ^ {(t + 1)} = \hat {w} _ {i} ^ {(t)} - \eta \lambda \hat {w} _ {i} ^ {(t)}.
$$
Thus, after a total of  $T$  iteration of training we get that:
$$
\hat {w} _ {\dot {\boldsymbol {i}}} ^ {(T)} = (1 - \eta \lambda) ^ {T} \hat {w} _ {\dot {\boldsymbol {i}}} ^ {(0)}.
$$
Therefore, the projection of gradients after training onto  $P^{\perp}$  will be the same as if they were initialized to  $\sim \mathcal{N}\left(0,\frac{(1 - \eta\lambda)^{2T}}{d} I_d\right)$  and trained using logistic loss without regularization. The rest of the proof is the same as Theorem 6.1 for  $\beta = \frac{(1 - \eta\lambda)^T}{\sqrt{d}}$ .
E Further Experiments and Experimental Details
E.1 Further Experiments
In Figure 5 we present the boundary of a two-layer ReLU network trained over a 25-point dataset on a two-dimensional linear subspace, similar to Figure 3. We train the networks until reaching a constant positive margin. The difference between the figures is that in Figure 5 we initialize the weights using the default PyTorch initialization, while in Figure 3 we initialized using a smaller scale
for the robustness effect to be smaller, and visualized more easily. The experiment in Figure 5 is demonstrating an extreme robustness effect, occurring when using the standard settings.
(b)
Figure 5: Experiments on two-dimensional dataset. We plot the dataset points and the decision boundary in 3 settings: (a) Vanilla trained network, (b) The network's weights are initialized from a smaller variance distribution, and (c) Training with regularization. Colors are used to emphasise the values in the  $z$  axis.
In Figure 6 we go beyond the theory discussed in this paper, and present similar phenomena in all three settings for a five-layer ReLU network. In Figure 6a we can see the boundary of the regularly trained network within a small distance in  $P^{\perp}$  from the data points. In Figure 6b we use small initialization for all five layers, and present a boundary almost orthogonal to the data manifold. In Figure 6c, the boundary of a regularized trained network is in a similar form. This experiment suggests that our theoretical results might be extended also to deeper networks, where all layers are trained.
(a)
Figure 6: Experiments on one-dimensional dataset with deep network. We plot the dataset points and the decision boundary in 3 settings: (a) Vanilla trained network, (b) The network's weights are initialized from a smaller variance distribution, and (c) Training with regularization.
E.2 One-dimensional dataset experiment - 2 layer network (Figure 2)
Dataset For all the three experiments we used a 7-point data set, spread equally on the two dimensional line  $y = x$  from  $(-1, -1)$  to  $(1, 1)$ .
Network For all the three experiments we used two-layer ReLU network of width 100 with biases in both layers. The weights of both layers were initialized using  $(1 + 3)$  default PyTorch initialization for linear layers, (2) default initialization divided by 3.
Training We used train step of size 0.02 for  $(1 + 3)$  and 0.04 for (2). We trained both layers until the margin reached 0.3. The losses we used were  $(1 + 2)$  Logistic loss, (3) Logistic loss with  $0.005L_{2}$  regularization.
E.3 Two-dimensional dataset experiment - smaller effect (Figure 3)
Dataset For all the three experiments we used a 25-point data set, spread equally on a grid which lies on the  $z = 0.5$  axis.
Network For all the three experiments we used two-layer ReLU network of width 4000 with biases in both layers. The weights in the first layer were initialized in  $(1 + 3)$  from  $\mathcal{N}(0,1 / 3I_3)$ , and in (2) from  $\mathcal{N}(0,1 / 36I_3)$ . The weight of the output layer were initialized to the uniform distribution over the set  $\{-1,1\}$ .
Training For all the experiments we trained both layers until the margin reached 0.3 and we used train step of size 0.002. The losses we used were (1+2) Logistic loss, (3) Logistic loss with  $0.8L_{2}$  regularization on the weights of the first layer.
E.4 Two-dimensional dataset experiment (Figure 5)
Dataset For all the three experiments we used a 25-point data set, spread equally on a grid which lies on the  $x - y$  axis.
Network For all the three experiments we used two-layer RelU network of width 400 with biases in both layers. The weights in any layer were initialized using  $(1 + 3)$  default PyTorch initialization for linear layers, (2) default initialization divided by 3.
Training For (1) experiments we used train step of size 0.005, and for  $(2 + 3)$  we used step of size 0.05. We trained both layers until the margin reached 0.1. The losses we used were  $(1 + 2)$  Logistic loss, (3) Logistic loss with  $0.005L_{2}$  regularization.
E.5 One-dimensional dataset experiment - 5 layer network (Figure 6)
Network For all the three experiments we used 5-layer RelU network of width 100 with biases in all layers. The weights in any layer were initialized using  $(1 + 3)$  default PyTorch initialization for linear layers, (2) default initialization divided by 3.