SlowGuess/NeurIPS_2023_OCR huggingface dataset:
From the above calculations we can see that adding the perturbation  $z$  does not increase the output of the neurons with a positive second layer. On the other hand, adding  $z$  can only increase the input of the neurons with negative second layer, and for those neurons which are also active it increases their output as well if we assume that  $\ell > m$ . This means, that if there are enough active neurons with a negative second layer (denoted by  $k_{-}$  in the theorem), then the perturbation can significantly change the output. In the proof we rely only on the active negative neurons to change the label of the output (for the case of  $y_0 = 1$ , if  $y_0 = -1$  we rely on the active positive neurons). Note that the active positive neurons may become inactive, and the inactive negative neurons may become active. Without further assumptions it is not clear what is the size of the perturbation to make this change for every neuron. Thus, the only neurons that are guaranteed to help change the label are the active negative ones, which by our assumptions on  $\ell$  are guaranteed to increase their output.
Note that our perturbation is not in the direction of the gradient w.r.t. the input. The direction of the gradient would be the sum of all the active neurons, i.e. the sum (with appropriate signs) over all  $i \in [m]$  such that  $\langle w_i, x_0 \rangle \geq 0$ . Our analysis would not have worked with such a perturbation, because we could not guarantee that inactive neurons would stay inactive.
The assumption that  $\ell = \Omega(M)$  (up to log factors) is a technical limitation of our proof technique. We note that such an assumption is also used in other theoretical papers about adversarial perturbations (e.g. Daniely and Shacham [2020]).
Note that the direction of the perturbation  $z$  does not depend on the input data  $x_0$ , only its size depends on  $x_0$ . In fact, Theorem 5.1 shows that there is a single universal direction for an adversarial perturbation that can flip the label of any data point in  $P$ . The size of the perturbation depends on the dimension of the linear subspace of the data, the number of active neurons for  $x_0$ , the total number of neurons in the network and the size of the output. In the following corollary we give a specific bound on the size of the perturbations under assumptions on the different parameters of the problem:
Corollary 5.1. In the setting of Theorem 5.1, assume in addition that  $\ell = \Theta(d)$  and  $k_{y_0} = \Theta(m)$ . Then, there exists a perturbation  $z$  such that w.p.  $\geq 1 - 5\left(me^{-\ell/16} + d^{-1/2}\right)$  we have  $\|z\| = O(N(x_0))$  and:
The above corollary follows directly by noticing from Theorem 5.1 that:
$$
\left\| z \right\| \leq O \left(N \left(x _ {0}\right) \cdot \frac {m}{k _ {y _ {0}}} \cdot \sqrt {\frac {d}{\ell}}\right) = O \left(N \left(x _ {0}\right)\right),
$$
where we plugged in the additional assumptions. The assumptions in the corollary above are similar to the assumptions in Corollary 4.1. Namely, that the dimension of the data subspace is a constant fraction from the dimension of the entire space, and the number of active neurons is a constant fraction of the total number of neurons. Note that here we only consider active neurons with a specific sign in the second layer.
Note that the size of the perturbation in Corollary 5.1 is bounded by  $N(x_0)$ . By Remark 3.1, the output of the network for data points on the margin can be at most  $O(\log^2 (d))$ , since otherwise the network would have essentially stopped training. Therefore, if we consider an input  $x_0$  on the margin, and  $\| x_0\| = \Theta (\sqrt{d}) = \Theta (\sqrt{\ell})$ , then the size of the adversarial perturbation is much smaller than  $\| x_0\|$ . For any other point, without assuming it is on the margin, and since we do not assume anything about the training data (except for being in  $P$ ), we must assume that the size of the perturbation required to change the label will depend on the size of the output.
6 The Effects of the Initialization Scale and Regularization on Robustness
In Section 5, we presented the inherent vulnerability of trained models to small perturbations in a direction orthogonal to the data subspace. In this section, we return to a common proxy for robustness that we considered in Section 4 - the gradient at an input point  $x_0$ . We suggest two ways that might improve the robustness of the model in the direction orthogonal to the data, by decreasing an upper bound of the gradient in this direction. We first upper bound the gradient of the model in the general case where we initialize  $w_i \sim \mathcal{N}(\mathbf{0}, \beta^2 I_d)$ , and later discuss strategies to use this upper bound for improving robustness.
Theorem 6.1. Suppose that a network  $N(x) = \sum_{i=1}^{m} u_i \sigma(\langle w_i, x \rangle)$  is trained on a dataset which lies on a linear subspace  $P \subseteq \mathbb{R}^d$  of dimension  $d - \ell$  for  $\ell \geq 1$ , and assume that the weights  $w_i$  are initialized from  $\mathcal{N}(\mathbf{0}, \beta^2 I_d)$ . Let  $x_0 \in P$ , let  $S = \{i \in [m] : \langle w_i, x_0 \rangle \geq 0\}$ , and let  $k = |S|$ . Then,  $w.p. \geq 1 - e^{-\frac{\ell}{16}}$  we have:
$$
\left\| \Pi_ {P ^ {\perp}} \left(\frac {\partial N (x _ {0})}{\partial x}\right) \right\| \leq \beta \sqrt {\frac {2 k \ell}{m}}.
$$
The full proof uses the same concentration bounds ideas as the lower bound proof and can be found in Appendix D. This bound is a result of the untrained weights: since the projection of the data points on  $P^{\perp}$  is zero, the projection of the weights vectors on  $P^{\perp}$  are not trained and are fixed at their initialization. We note that Theorem 4.1 readily extends to the case of initialization from  $\mathcal{N}(\mathbf{0},\beta^{2}I_{d})$ , in which case the lower bound it provides matches the upper bound from Theorem 6.1 up to a constant factor. In what follows, we suggest two ways to affect the post-training weights in the  $P^{\perp}$  direction: (1) To initialize the weights vector using a smaller-variance initialization, and (2) Add an  $L_{2}$ -norm regularization on the weights. We next analyze their effect on the upper bound.
6.1 Small Initialization Variance
From Theorem 6.1, one can conclude a strong result about the model's gradient without the dependency of its norm on  $\ell$  and  $k$ .
Corollary 6.1. For  $\beta = \frac{1}{d\sqrt{2}}$ , in the settings of Theorem 6.1, with probability  $\geq 1 - e^{-\frac{\ell}{16}}$  we have
$$
\left\| \Pi_ {P ^ {\perp}} \left(\frac {\partial N (x _ {0})}{\partial x}\right) \right\| \leq \frac {1}{\sqrt {d}}.
$$
The proof follows directly from Theorem 6.1, by noticing that  $k \leq m$  and  $\ell \leq d$ . Consider for example an input  $x_0 \in P$  with  $\| x_0 \| = \Theta(\sqrt{d})$ , and suppose that  $N(x_0) = \Omega(1)$ . The above corollary implies that if the initialization has a variance of  $1/d^2$  (rather than the standard choice of  $1/d$ ) then the gradient is of size  $O\left(1/\sqrt{d}\right)$ . Thus, it corresponds to perturbations of size  $\Omega(\sqrt{d})$ , which is the same order as  $\| x_0 \|$ .
6.2  $L_{2}$  Regularization
We consider another way to influence the projection onto  $P^{\perp}$  of the trained weights vectors: adding  $L_{2}$  regularization while training. We will update the logistic loss function by adding an additive factor  $\frac{1}{2}\lambda\|\mathbf{w}_{1:m}\|^2$ . For a dataset  $(x_{1},y_{1}),..,(x_{r},y_{r})$ , we now train over the following objective:
$$
\sum_ {j = 1} ^ {r} L \left(y _ {j} \cdot N \left(x _ {j}, \mathbf {w} _ {1: m}\right)\right)) + \frac {1}{2} \lambda \left\| \mathbf {w} _ {1: m} \right\| ^ {2}.
$$
This regularization will cause the previously untrained weights to decrease in each training step which will decrease the upper bound on the projection of the gradient:
Theorem 6.2. Suppose that a network  $N(x) = \sum_{i=1}^{m} u_i \sigma(\langle w_i, x \rangle)$  is trained for  $T$  training steps, using  $L_2$  regularization with parameter  $\lambda \geq 0$  and step size  $\eta > 0$ , on a dataset which lies on a linear subspace  $P \subseteq \mathbb{R}^d$  of dimension  $d - \ell$  for  $\ell \geq 1$ , starting from standard initialization (i.e.,  $w_i \sim \mathcal{N}(\mathbf{0}, \frac{1}{d} I_d)$ ). Let  $x_0 \in P$ , let  $S = \{i \in [m] : \langle w_i, x_0 \rangle \geq 0\}$ , and let  $k := |S|$ . Then,  $w.p. \geq 1 - e^{-\frac{\ell}{16}}$  we have
$$
\left\| \Pi_ {P ^ {\perp}} \left(\frac {\partial N (x _ {0})}{\partial x}\right) \right\| \leq (1 - \eta \lambda) ^ {T} \sqrt {\frac {2 k \ell}{m d}}.
$$
The full proof can be found in Appendix D.1. The main idea of the proof is to observe the projection of the weights on  $P^{\perp}$  changing during training. As before, we assume w.l.o.g. that  $P = \operatorname{span}\{e_1, \ldots, e_{d - \ell}\}$  and denote by  $\hat{w}_i \coloneqq \Pi_{P^{\perp}}(w_i)$ . During training, the weight vector's last  $\ell$  coordinates are only affected by the regularization term of the loss. These weights decrease in a constant multiplicand of the previous weights. Thus, we can conclude that for every  $t \geq 0$  we have:  $\hat{w}_i^{(t)} = (1 - \eta \lambda)^t\hat{w}_i^{(0)}$ , where  $\hat{w}_i^{(t)}$  is the  $i$ -th weight vector at time  $t$ . It implies that our setting is equivalent to initializing the weights with standard deviation  $\frac{(1 - \eta\lambda)^T}{\sqrt{d}}$  and training the model without regularization for  $T$  steps. As a result, we get the following corollary:
Corollary 6.2. For  $(1 - \eta \lambda)^T \leq \frac{1}{\sqrt{2d}}$ , in the settings of 6.2, w.p.  $\geq 1 - e^{-\frac{\ell}{16}}$  we get that:
6.3 Experiments
In this section, we present our robustness-improving experiments. We explore our methods on two datasets: (1) A 7-point dataset on a one-dimensional linear subspace in a two-dimensional input
4For the code of the experiments see https://github.com/odeliamel/off-manifold-robustness
space, and (2) A 25-point dataset on a two-dimensional linear subspace in a three-dimensional input space. In Figures 2 and 3 we present the boundary of a two-layer ReLU network trained over these two datasets. We train the networks until reaching a constant positive margin. We note that unlike our theoretical analysis, in the experiments in Figure 2 we trained all layers and initialize the weights using the default PyTorch initialization, to verify that the observed phenomena occur also in this setting. In the experiment in Figure 3 we use a different initialization scale for the improving effect to be smaller and visualized easily. In Figures 2a and 3a we trained with default settings. In Figures 2b and 3b we initialized the weights using an initialization with a smaller variance (i.e., initialization divided by a constant factor). Finally, in Figures 2c and 3c we train with  $L_{2}$  regularization.
(b)
Figure 2: Experiments on a one-dimensional dataset. We plot the dataset points and the decision boundary in 3 settings: (a) Vanilla trained network, (b) The network's weight are initialized from a smaller variance distribution, and (c) Training with regularization.
Consider the adversarial perturbation in the direction  $P^{\perp}$ , orthogonal to the data subspace, in Figures 2 and 3. In figure (a) of each experiment, we can see that a rather small adversarial perturbation is needed to cross the boundary in the subspace orthogonal to the data. In the middle figure (b), we see that the boundary in the orthogonal subspace is much further. This is a direct result of the projection of the weights onto this subspace being much smaller. In the right experiment (c), we can see a similar effect created by regularization. In Appendix E we add the full default-scaled experiment in the two-dimensional setting to demonstrate the robustness effect. There, in both the small-initialization and regularization experiments, the boundary lines are almost orthogonal to the data subspace. In Appendix E we also conduct further experiments with deeper networks and standard PyTorch initialization, showing that our theoretical results are also observed empirically in settings going beyond our theory.
(b)
Figure 3: Experiments on two-dimensional dataset demonstrating a smaller robustness effect. We plot the dataset points and the decision boundary in 3 settings: (a) Vanilla trained network, (b) The network's weights are initialized from a smaller variance distribution, and (c) Training with regularization. Colors are used to emphasise the values in the  $z$  axis.
In Figure 4 we plot the distance from the decision boundary for different initialization scales of the first layer. We trained a 3-layer network, initialized using standard initialization except for the first layer which is divided by the factor represented in the  $X$ -axis. After training, we randomly picked 200 points and used a standard projected gradient descent adversarial attack to change the label of
each point, which is described in the  $Y$ -axis (perturbation norm, with error bars). The datasets are: (a) Random points from a sphere with 28 dimensions, which lies in a space with 784 dimensions; and (b) MNIST, where the data is projected on 32 dimensions using PCA. The different lines are adversarial attacks projected either on data subspace, on its orthogonal subspace, or without projection. It can be seen that small initialization increases robustness off the data subspace, and also on the non-projected attack, while having almost no effect for the attacks projected on the data subspace.
(b)
Figure 4: The distance to the decision boundary for different initializations of the first layer. The  $X$ -axis represents the factor which the initialization of the first layer is divided by. The  $Y$ -axis shows the size of a standard perturbed gradient descent adversarial attack to change the label of each point for 200 randomly picked points. The datasets are: (a) Random points from a sphere with 28 dimensions, which lies in a space with 784 dimensions; and (b) MNIST, where the data is projected on 32 dimensions using PCA. The different lines are adversarial attacks projected either on data subspace, on its orthogonal subspace, or without projection.
7 Conclusions and Future Work
In this paper we considered training a two-layer network on a dataset lying on  $P \subseteq \mathbb{R}^d$  where  $P$  is a  $d - \ell$  dimensional subspace. We have shown that the gradient of any point  $x_0 \in P$  projected on  $P^\perp$  is large, depending on the dimension of  $P$  and the fraction of active neurons on  $x_0$ . We additionally showed that there exists an adversarial perturbation in the direction of  $P^\perp$ . The size of the perturbation depends in addition on the output of the network on  $x_0$ , which by Remark 3.1 should be poly-logarithmic in  $d$ , at least for points which lie on the margin of the network. Finally, we showed that by either decreasing the initialization scale or adding  $L_2$  regularization we can make the network robust to "off-manifold" perturbations, by decreasing the gradient in this direction.
One interesting question is whether our results can be generalized to other manifolds, beyond linear subspaces. We state this as an informal open problem:
Open Problem 7.1. Let  $M$  be a manifold, and  $\mathcal{D}$  a distribution over  $M \times \{\pm 1\}$ . Suppose we train a network  $N: \mathbb{R}^d \to \mathbb{R}$  on a dataset sampled from  $\mathcal{D}$ . Let  $x_0 \in M$ , then under what conditions on  $M$ ,  $\mathcal{D}$  and  $N$ , there exists a small adversarial perturbation in the direction of  $(T_{x_0}M)^\perp$ , i.e. orthogonal to the tangent space  $T_{x_0}M$ , of  $M$  at  $x_0$ .
Our result can be seen as a special case of this conjecture, where at all points  $x, x' \in M$ , the tangent spaces are equal  $T_xM = T_{x'}M$ . Another future direction would be to analyze deeper networks, or different architectures such as convolutions. Finally, it would also be interesting to analyze robustness of trained networks w.r.t. different norms such as  $L_1$  or  $L_\infty$ .
We thank Ohad Shamir for the many helpful discussions about this work. We would also like to thank Michal Irani for contributing computational resources. GY was supported in part by the European Research Council (ERC) grant 754705 . GV acknowledges the support of the NSF and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning.
A Rotation Invariance w.r.t. the Initialized Weights
In this paper, we analyze neural networks trained on high-dimensional data that lies on a low dimensional linear subspace denoted by  $P$ . We assume that the dimension of  $P$  is  $d - \ell$ . Throughout the paper it will be more convenient to analyze data which lies on the subspace  $M = \operatorname{span}(\{e_1, \ldots, e_{d - \ell}\})$ , because then the "off manifold" directions correspond exactly to certain coordinates of the input. In this section we show that we can essentially analyze the data as if it is rotated to lie on  $M$ , and it would imply the same consequences as the original data from  $P$ .
Theorem A.1. Let  $P \subseteq \mathbb{R}^d$  be a subspace of dimension  $d - \ell$ , and let  $M = \text{span}\{e_1, \ldots, e_{d - \ell}\}$ . Let  $R$  be an orthogonal matrix such that  $R \cdot P = M$ , let  $X \subseteq P$  be a training dataset and let  $X_R = \{R \cdot x : x \in X\}$ . Assume we train a neural network  $N(x) = \sum_{i=1}^{m} u_i \sigma(w_i^\top x)$  as explained in Section 3, and denote by  $N^X$  and  $N^{X_R}$  the network trained on  $X$  and  $X_R$  respectively for the same number of iterations. Let  $x_0 \in P$ , then we have:
1. W.p.  $p$  (over the initialization) we have  $\left\| \Pi_{P^{\perp}}\left(\frac{\partial N^{X}(x_{0})}{\partial x}\right)\right\| \geq c$  (resp.  $\leq c$ ) for some  $c \in \mathbb{R}$ , iff w.p.  $p$  also  $\left\| \Pi_{M^{\perp}}\left(\frac{\partial N^{X_{R}}(Rx_{0})}{\partial x}\right)\right\| \geq c$  (resp.  $\leq c$ ).
2. For any  $c, p \geq 0$ , w.p.  $p$  (over the initialization) there exists  $z \in P^{\perp}$  with  $\|z\| = c$  such that  $\operatorname{sign}(N^{X}(x_{0} + z)) \neq \operatorname{sign}(N^{X}(x_{0}))$ , iff w.p.  $p$  there exists  $z' \in M^{\perp}$  with  $\|z'\| = c$  such that  $\operatorname{sign}(N^{X_{R}}(Rx_{0} + z')) \neq \operatorname{sign}(N^{X_{R}}(Rx_{0}))$ .
Proof. Denote by  $\mathbf{w}_{1:m} := (w_1, \ldots, w_m)$  and by  $R\mathbf{w}_{1:m} = (Rw_1, \ldots, Rw_m)$ . Let  $\mathbf{w}_{1:m}^{(t)}$  the weights of the network trained on the dataset  $X$  where  $\mathbf{w}_{1:m}^{(0)}$  is some initialization, and  $\tilde{\mathbf{w}}_{1:m}^{(t)} = (\tilde{w}_1^{(t)}, \ldots, \tilde{w}_m^{(t)})$  the weights of the network trained on  $X_R$  and initialized at  $R\mathbf{w}_{1:m}^{(0)}$ . In the proof, when taking derivatives w.r.t. the  $w_i$ 's we will explicitly write  $N(x, \mathbf{w}_{1:m})$ .
We first show by induction on the number of training steps that  $\tilde{\mathbf{w}}_{1:m}^{(t)} = R\mathbf{w}_{1:m}^{(t)}$ . For  $t = 0$  it is clear by the assumption on the initialization. Assume it is true for  $t$ , then we have for some  $x \in X$ :
$$
\begin{array}{l} \frac {\partial N (R x , \tilde {\mathbf {w}} _ {1 : m} ^ {(t)})}{\partial w _ {i}} = u _ {i} \sigma^ {\prime} (\langle \tilde {w} _ {i} ^ {(t)}, R x \rangle) R x \\ = u _ {i} \sigma^ {\prime} \left(\left\langle R w _ {i} ^ {(t)}, R x \right\rangle\right) R x \\ = u _ {i} \sigma^ {\prime} (\langle w _ {i} ^ {(t)}, x \rangle) R x \\ = R \cdot \frac {\partial N (x , \mathbf {w} _ {1 : m} ^ {(t)})}{\partial w _ {i}}. \\ \end{array}
$$
This is true for every  $i \in [m]$  and for every  $x \in X$ . Also note that by our induction assumption we have:
$$
N \left(x, \mathbf {w} _ {1: m} ^ {(t)}\right) = \sum_ {i = 1} ^ {m} u _ {i} \sigma \left(\left\langle w _ {i} ^ {(t)}, x \right\rangle\right) = \sum_ {i = 1} ^ {m} u _ {i} \sigma \left(\left\langle R w _ {i} ^ {(t)}, R x \right\rangle\right) = N \left(R x, \tilde {\mathbf {w}} _ {1: m} ^ {(t)}\right). \tag {1}
$$
Finally, the derivative of the loss on a single data point  $x \in X$  with label  $y$  can be written as:
$$
\frac {\partial L \left(N (x , \mathbf {w} _ {1 : m} ^ {(t)}) \cdot y\right)}{\partial w _ {i}} = L ^ {\prime} \left(N (x, \mathbf {w} _ {1: m} ^ {(t)}) \cdot y\right) \cdot \frac {\partial N (x , \mathbf {w} _ {1 : m} ^ {(t)})}{\partial w _ {i}},
$$
where the first term depends only on the value of  $N(x, \mathbf{w}_{1:m}^{(t)})$ . Hence, taking a single gradient step of  $N$  with weights  $\mathbf{w}_{1:m}^{(t)}$  and dataset  $X$  will change the weights by the same term up to multiplication by  $R$  as if taking a gradient step with weights  $\tilde{\mathbf{w}}_{1:m}^{(t)}$  and dataset  $X_R$ . This finishes the induction.
Let  $\mathbf{w}_{1:m}^{(0)}$  be an initialization for the training of  $N^X$ , where there exists  $z \in P^\perp$  with  $\|z\| = c$  such that  $\mathrm{sign}(N^X(x_0 + z)) \neq \mathrm{sign}(N^X(x_0))$ . Then, by Eq. (1) the initialization  $R\mathbf{w}_{1:m}^{(0)}$  for the training of  $N^{X_R}$  is such that for  $z' = Rz$  we have  $\|z'\| = c$  and  $\mathrm{sign}(N^{X_R}(Rx_0 + z')) \neq \mathrm{sign}(N^{X_R}(Rx_0))$ . This argument holds also in the opposite direction. Let  $A \subseteq \{\mathbf{w}_{1:m} \in \mathbb{R}^{d \cdot m}\}$  be the set of all
initializations to  $N^X$  where there exists  $z \in P^\perp$  with  $\|z\| = c$  such that  $\mathrm{sign}(N^X(x_0 + z)) \neq \mathrm{sign}(N^X(x_0))$ , then by the above the set  $R \cdot A = \{R\mathbf{w}_{1:m} : \mathbf{w}_{1:m} \in A\}$  are exactly all the initializations to  $N^{X_R}$  where there exists  $z' \in M^\perp$  with  $\|z'\| = c$  such that  $\mathrm{sign}(N^{X_R}(Rx_0 + z')) \neq \mathrm{sign}(N^{X_R}(Rx_0))$ . Since we initialize the  $w_i$ 's using a Gaussian initialization which is spherically symmetric, we have that  $\operatorname{Pr}(A) = \operatorname{Pr}(RA)$ . This proves item (2). Item (1) follows from similar arguments (which we do not repeat for conciseness).
Under the assumption that the data lies on  $M = \operatorname{span}\{e_1, \ldots, e_{d - \ell}\}$ , and no regularization is used, we can show that the weights of the first layer projected on  $M^{\perp}$  do not change during training. This is an essential part of the proofs, as it allows us to analyze those weights as random Gaussian vectors, and apply concentration bounds on them.
Theorem A.2. Let  $M = \text{span}\{e_1, \ldots, e_{d - \ell}\}$ . Assume we train a neural network  $N(x, \mathbf{w}_{1:m}) := \sum_{i=1}^{m} u_i \sigma(w_i^\top x)$  as explained in Section 3 (where  $\mathbf{w}_{1:m} = (w_1, \ldots, w_m)$ ). Denote by  $\hat{w} := \Pi_{M^\perp}(w)$  for  $w \in \mathbb{R}^d$ , then after training, for each  $i \in [m]$ ,  $\hat{w}_i$  did not change from their initial value.
Proof. Note that for each  $i \in [m]$  and  $x \in M$  we have:
$$
\Pi_ {M ^ {\perp}} \left(\frac {\partial N (x , \mathbf {w} _ {1 : m})}{\partial w _ {i}}\right) = \Pi_ {M ^ {\perp}} \left(u _ {i} \sigma^ {\prime} (w _ {i} ^ {\top} x) x\right) = u _ {i} \sigma^ {\prime} (w _ {i} ^ {\top} x) \hat {x} = \mathbf {0}.
$$
Taking the derivative of the loss we have:
$$
\begin{array}{l} \Pi_ {M ^ {\perp}} \left(\frac {\partial L (N (x , \mathbf {w} _ {1 : m}) \cdot y)}{\partial w _ {i}}\right) = \Pi_ {M ^ {\perp}} \left(L ^ {\prime} (N (x, \mathbf {w} _ {1: m}) \cdot y) \cdot \frac {\partial N (x , \mathbf {w} _ {1 : m})}{\partial w _ {i}}\right) \\ = L ^ {\prime} \left(N (x, \mathbf {w} _ {1: m}) \cdot y\right) \cdot \Pi_ {M ^ {\perp}} \left(\frac {\partial N (x , \mathbf {w} _ {1 : m})}{\partial w _ {i}}\right) = \mathbf {0}. \\ \end{array}
$$
The above calculation did not depend on the specific value of the  $w_{i}$ 's. Hence, the value of the  $\hat{w}_{i}$ 's for every  $i\in [m]$  did not change during training from their initial value.
B Proofs from Section 4
Before proving the main theorem, we will first need the next two lemmas about the concentration of Gaussian random variables:
Lemma B.1. Let  $w \in \mathbb{R}^n$  such that  $w \sim \mathcal{N}(\mathbf{0}, \sigma^2 I_n)$ . Then: