SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we compare the symmetric solutions found in erf [2] and ReLU networks [5] to our one-neuron solution  $(n = 1)$ . The main difference is that both earlier studies constrain the search space to the symmetric subspace whereas we first prove that the non-trivial critical points are contained in this subspace in Theorem 5.1 for a broad class of activation functions, including erf and ReLU. Solving the low-dimensional loss, we recover the same solution for ReLU and erf as in [2, 5] for unit-orthonormal teachers.
Symmetric Solution of Saad and Solla [2] for erf activation. The authors focus on the 'symmetric subspace' parameterized as
$$
Q _ {i i} = r _ {i} ^ {2} = Q, \quad Q _ {i j} = p _ {i j} r _ {i} r _ {j} = C, \quad R _ {i n} = u _ {i n} r _ {i} = R. \tag {21}
$$
In this case, the loss is parameterized by three values, that is  $Q, C, R$ , hence can be expressed analytically in terms of these values. Solving the fixed point equations, they find the following critical/fixed point (their Eq.22)
$$
Q = C = \frac {1}{2 k - 1}, \quad R = \frac {1}{\sqrt {k (2 k - 1)}} \tag {22}
$$
which implies  $r_i = 1 / \sqrt{2k - 1}$  and  $\rho_{ij} = 1$  in our parameterization. This selection of parameters forces all student vectors to be equal therefore reducing the system to a one-neuron network. There are two main improvements in our analysis
1. We prove that student-teacher correlations  $u_{ij}$  are equal to each other at a non-trivial critical point, and give necessary conditions on the activation function (Assumption 3.1) to satisfy this property. We show in Lemma F.2 that not only erf but a large class of common activation functions satisfy Assumption 3.1.
2. Our student network has a flexible outgoing weight (shallow neural network) as opposed to a fixed outgoing weight  $+1$  (soft-committee machine) in Saad and Solla [2]. It is instructive to compare the generalization errors of the one-neuron network
(soft-committee machine)  $L_{\mathrm{erf,soft}}^{*}(k) = \frac{k}{3} - k^{2}\frac{2}{\pi}\arcsin \left(\frac{1}{2k}\right)$
(shallow network)  $L_{\mathrm{erf}}^{*}(k) = k\frac{2}{\pi}\arcsin \left(\frac{1}{2}\right) - k^{2}\frac{2}{\pi}\arcsin \left(\frac{1}{2k}\right)\approx k\left(\frac{1}{3} -\frac{1}{\pi}\right),$
which are identical since  $\arcsin(0.5) = \pi/6$  (Saad and Solla [2] uses  $\epsilon_g(k) = \frac{1}{2} L_{\mathrm{erf;soft}}^*(k)$  that's why there is a factor 0.5 difference with respect to their Eq. (23)). However, if we set teacher outgoing weights to say  $a_t$ , the shallow network adapts and reaches the generalization error  $a_t^2 L_{\mathrm{erf}}^*(k)$  but the error of the soft-committee machine is
$$
\begin{array}{l} L _ {\text {e r f ; s o f t}} ^ {*} (k) = k ^ {2} g \left(\frac {1}{\sqrt {2 k - 1}}, \frac {1}{\sqrt {2 k - 1}}, 1\right) - 2 k ^ {2} a _ {t} g \left(\frac {1}{\sqrt {2 k - 1}}, 1, \frac {1}{\sqrt {k - 1}}\right) + a _ {t} ^ {2} k g (1, 1, 1) \\ = O \left(a _ {t}\right) + a _ {t} ^ {2} k \frac {1}{3}. \\ \end{array}
$$
which has a worse coefficient  $\frac{1}{3}$  compared to  $\frac{1}{3} -\frac{1}{\pi}$  as expected.
Symmetric Solution of Tian [5] for ReLU activation. The authors focus on a particular two-dimensional subspace  $(x,y)$  that allows the specialization of student neurons, namely
$$
w _ {i} = x v _ {i} + y \sum_ {j \neq i} v _ {j}. \tag {23}
$$
In particular, they consider the 'symmetric subspace'  $x = y$  which is the case when all student neurons collapse to one neuron, and show that the dynamics converge to the following fixed point
$$
x = y = \frac {1}{\pi k} (\sqrt {k - 1} - \arccos  \left(\frac {1}{\sqrt {k}}\right) + \pi). \tag {24}
$$
Summing over  $k$  neurons then produces the following one-neuron due to the positive homogeneity
$$
w ^ {*} = \frac {1}{\pi} (\sqrt {k - 1} - \operatorname {a r c c o s} (\frac {1}{\sqrt {k}}) + \pi) \sum_ {j = 1} ^ {k} v _ {j}.
$$
Our formula (Corollary G.5) gives the identical result due to
$$
w ^ {*} a ^ {*} = \frac {k}{h (1)} h (\frac {1}{\sqrt {k}}) \sum_ {j = 1} ^ {k} \frac {1}{\sqrt {k}} v _ {j} = \frac {1}{\pi} (\sqrt {k - 1} - \arccos (\frac {1}{\sqrt {k}}) + \pi) \sum_ {j = 1} ^ {k} v _ {j}.
$$
In this case, there is no difference between the optimal solution of the soft-committee machine and the shallow network since ReLU is positive-homogeneous as expected.
B
Figure 5: Structure of the optimal solution of the one-neuron network for various activation functions. We trained 20 seeds of one-neuron students learning from the unit-orthonormal teacher networks with  $k = 2, \dots, 10$  neurons. All students converge to the same optimal solution up to symmetries (that is, positive-scaling symmetry for ReLU and sign symmetry for odd activation functions such as tanh and erf). A For ReLU, the magnitude  $\| w^{*} \| a^{*}$  exactly matches with the result of Corollary G.5. For softplus, the magnitude is very close to  $\sqrt{k}$ ; for sigmoid, tanh, and erf, it is below  $\sqrt{k}$ . B The norm of the incoming vector is smaller than  $1 / \sqrt{k}$  for softplus, sigmoid, tanh, and erf. C The outgoing weight is larger than  $k$  for softplus and tanh, and it is virtually  $k$  for sigmoid and erf.
C Further Experiments
All experiments in this paper are implemented using the gradient flow package implemented by Brea et al. [16] which is particularly suited to studying gradient flow on the population loss. For activation functions for which there is an analytic formula, it is already implemented in the package; for the others, we used the approximator option for a speed-up compared to the numerical integration option. This method uses a neural network in the background fitted to approximate Gaussian integrals. We trained for  $10^{5}$  ode iterations for erf and relu experiments;  $10^{3}$  ode iterations for softplus, tanh, and sigmoid. For erf experiments, all seeds converged to configurations with gradient norm below  $5 \cdot 10^{-8}$ . For ReLU experiments, a fraction of seeds failed to converge (large gradient norm at the end of training). In Appendix C.3, we report among the seeds that succeeded in converging. Weights initialized as Gaussians with zero mean and standard deviation 0.1 or with Glorot initialization [43]. This is in contrast with Saad and Solla [2], Tian [5] where (order) parameters are initialized with positive values (as opposed to the rotationally symmetric initializations done in practice). For each  $(n,k)$  pair, we implemented 10 or 20 seeds of random initializations.
C.1 One-Neuron Network
Empirically, gradient flow converges to the point where all student-teacher correlations are  $\frac{1}{k}^4$ .
We know from Theorem 5.1 that at the non-trivial critical point all correlations are equal at correlation  $\frac{1}{\sqrt{k}}$  and there is possibly another critical point at correlation  $-\frac{1}{\sqrt{k}}$ . Depending on the activation function, the point where correlations are  $-\frac{1}{\sqrt{k}}$  might be
- either an equivalent of the optimum solution (for odd activation functions),
- or a saddle (for ReLU),
- or does not exist (for softplus).
The details can be found in the proofs for individual cases.
C.2 Erf Experiments
In this section, we first numerically investigate whether the CA-optimal critical point is a saddle or minimum in Figure 6. Surprisingly, we find that the point turns from a saddle point to a minimum
For odd activation functions, there are two solutions that are sign-symmetric: the first one where all correlations are  $\frac{1}{\sqrt{k}}$  and its equivalent where all correlations are  $-\frac{1}{\sqrt{k}}$ . The first solution is plotted in Fig. 5 for a fine comparison on the positive scale.
Figure 6: The minimum eigenvalue of the Hessian at an optimal-CA point. We numerically investigate whether a CA-optimal critical point is a strict saddle (min. eig. of the Hessian is negative) or a minimum (min. eig. of the Hessian is non-negative). Interestingly, the minimum eigenvalue turns from positive to negative as  $n$  grows for  $k = n + h$  for fixed  $h = 1,2,3$  (left panel). Therefore in this regime, the CA-optimal cannot be the optimal solution of the non-convex problem for large  $n$ . For  $k \gg n$ , for example for  $k = n,2n,3n$  (right panel), the min. eigenvalue is positive and it approaches zero as  $n$  increases.
Figure 7: Evolution of order parameters during convergence to a  $(n - 1)$ -C-1-A solution: top  $n = 2$ , middle  $n = 4$ , bottom  $n = 8$ ; and  $k = 64$ ; representative seeds. We can distinguish 3 phases: before iteration 20, after iteration 20, and beyond iteration 40. We observe that in the first phase of training (less than 5 iterations), the student neurons do not specialize into teacher neurons but approach the one-neuron solution. For  $n = 2$  (top row), in the second phase, we observe that the first neuron implements an average of teacher neurons and the second neuron implements a copy of the remaining teacher neuron. In the second phase, in general,  $n - 1$  neurons specialize to match one teacher neuron each (or its negative equivalent) and the  $n$ -th neuron splits its correlations into two groups: those that correspond to the teacher neurons being matched become negative and the others collapse on each other. Finally, in the third phase, the negative correlations converge to zero correlation, decoupling the student neurons from each other. All student neuron correlation signs can be flipped as long as the corresponding outgoing weight signs are flipped since the erf activation is odd. These examples illustrate the green regime (see Fig. 4 in the main text).
point in some regimes of  $(n, k)$ , despite that the configuration has the same structure of copying  $n - 1$  teacher neurons and taking an average of the remaining teacher neurons. When  $k - n$  is fixed, and for large  $n$ , Figure 6 explains why gradient flow does not converge to the CA-optimal point.
Figure 8: Evolution of order parameters;  $P$ - $n$ - $C$  solution: for  $n = 8$  and  $k = 9$ . We observe that all student neurons match one teacher neuron in this case, however not perfectly at the end of training. This is an example of the red regime (see Fig. 4 in the main text), where the students converge to a perturbation of the  $n$ -copy configuration.
A1  $n = 2,k = 5$
A2  $n = 2, k = 10$
A3  $n = 2,k = 20$
B1  $n = 4,k = 5$
B2  $n = 4, k = 8$
B3  $n = 4,k = 16$
C1  $n = 12, k = 16$
C2  $n = 12, k = 24$
C3  $n = 12, k = 32$
Figure 9: Optimal configuration found by gradient flow for ReLU activation function for  $n = 2$  (top);  $n = 4$  (middle);  $n = 12$  (bottom). The last row of the correlation matrices represents  $1 - \| u_i \|$  which is zero for all student neurons in all cases. Top row,  $n = 2$ ; the optimal point is composed of a copy and an average neuron: the copy neuron is very close to one of the teacher neurons, and the average neuron is close to the average of the remaining teacher neurons while negatively correlating with the copied teacher neuron. The negative correlation increases in magnitude as  $k$  increases. Middle row,  $n = 4$ ; for  $k = 5$ , the optimal point is a perturbation of the all-copy configuration; for  $k = 8, 16$ , it is close to the  $(n - 1)$ -copy-1-average configuration. Bottom row,  $n = 12$ ; for  $k = 16, 24$ , the optimal point is a perturbation of the all-copy configuration; for  $k = 32$ , it is close to the  $(n - 1)$ -copy-1-average configuration. In all regimes, the norms and outgoing weights of all student neurons are close to each other (for the bottom row only the first four neurons are shown).
We show some representative trajectories of gradient flow, in the regime when the CA-optimal critical point is a minimum in Figure 7 and in the regime when it is a saddle point in Figure 8.
C.3 ReLU Experiments
In this subsection, we will present the structure of the minimum loss configuration found by gradient flow. In the regime  $n \ll k$ , the minimum loss configuration is qualitatively similar to the optimal-CA solution but without perfect decoupling. For  $k$  that is slightly bigger than  $n$ , the gradient flow finds an "all-copy" configuration for  $n = 4, 12$ . The overall trend of the minimum loss configuration is
similar to the case of erf activation; however, as we do not have the analytic formula of correlations, a theoretical prediction for the optimal solution is left for future work.
D Constrained Optimization Formulation
Expressing the loss function in terms of the order parameters yields the 'projected' loss function
$$
L _ {\text {p r o j}} ^ {n, k} = \sum_ {i = 1} ^ {n} \underbrace {a _ {i} ^ {2} g _ {\sigma} \left(r _ {i} , r _ {i} , 1\right)} _ {\text {s t u d e n t m a g n i t u d e t e r m}} + 2 \sum_ {i \neq i ^ {\prime}} \underbrace {a _ {i} a _ {i ^ {\prime}} g _ {\sigma} \left(r _ {i} , r _ {i ^ {\prime}} , \rho_ {i i ^ {\prime}}\right)} _ {\text {s t u d e n t - s t u d e n t i n t e r a c t i o n}} - 2 \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {k} \underbrace {a _ {i} b _ {j} g _ {\sigma} \left(r _ {i} , \| v _ {j} \| , u _ {i j}\right)} _ {\text {s t u d e n t - t e a c h e r i n t e r a c t i o n}} + C \tag {25}
$$
where the constant term  $C = \mathbb{E}_{x\sim \mathcal{D}}[f^{*}(x)^{2}]$ .  $L_{\mathrm{proj}}$  has  $n(k + 2) + \binom{n}{2}$  parameters, that is  $n$  output weights and  $n(k + 1) + \binom{n}{2}$  order parameters, instead of  $n(d + 1)$  parameters of the original loss function. For  $d \gg k + 1 + \frac{n - 1}{2}$ ,  $L_{\mathrm{proj}}$  has significantly less number of parameters.
In the special case  $d = k$ , the incoming vectors can be expressed as a linear combination of the teacher's incoming vectors, hence the correlations between them are not free (see Appendix D.1).
Each normalized incoming vector can be expressed as a sum of its projection on the span of the teacher's incoming vectors and an orthogonal component
$$
\frac {w _ {i}}{r _ {i}} = \sum_ {j = 1} ^ {k} u _ {i j} v _ {j} + v _ {i} ^ {\perp}, \quad \sum_ {j = 1} ^ {k} u _ {i j} ^ {2} \leq 1, \tag {26}
$$
and the inequality constraint pops up since  $\| u_i\|^2 = 1 - \| v_i^\perp\|^2$  where  $u_i = (u_{i1},\ldots ,u_{ik})$ . For the correlations between the incoming vectors, we get
$$
\rho_ {i i ^ {\prime}} = u _ {i} \cdot u _ {i ^ {\prime}} + v _ {i} ^ {\perp} \cdot v _ {i ^ {\prime}} ^ {\perp}, \tag {27}
$$
which yields the second set of constraints on the optimization problem
$$
\left| \rho_ {i i ^ {\prime}} - u _ {i} \cdot u _ {i ^ {\prime}} \right| \leq \sqrt {1 - \| u _ {i} \| ^ {2}} \sqrt {1 - \| u _ {i ^ {\prime}} \| ^ {2}} \quad \forall i ^ {\prime} \neq i \in [ n ], \tag {28}
$$
since we have  $|v_i^\perp \cdot v_{i'}^\perp| \leq \| v_i^\perp\| \| v_{i'}^\perp\|$  due to the Cauchy-Schwarz inequality.
We note if all incoming vectors are in the span of the teacher's incoming vectors, we have that  $\| u_{i}\| = 1$ . As a result, the second set of inequalities in Eq. 28 collapses onto equalities, hence the secondary constraints are in fact equality constraints (see Appendix Eq. 34). We show that this is indeed the case for the non-trivial critical points of the one-neuron network in Section G.
In general, the constrained optimization formulation is possible for non-orthogonal teacher networks.
Remark D.1. (Non-orthogonal teacher network) We can relax the assumption of orthogonality between  $v_{1}, \ldots, v_{k}$  to linear independence. Let us collect the incoming vectors into a matrix  $V = [v_{1}, \ldots, v_{k}] \in \mathbb{R}^{d \times k}$ . The expansion in Eq. 26 can be rewritten as
$$
\frac {w _ {i}}{r _ {i}} = \sum_ {j = 1} ^ {k} \gamma_ {i j} v _ {j} + v _ {i} ^ {\perp} = V \Gamma_ {i} + v _ {i} ^ {\perp} \tag {29}
$$
where  $\Gamma_{i} = (\gamma_{i1},\dots ,\gamma_{ik})\in \mathbb{R}^{k}$  and  $v_{i}^{\perp}\cdot v_{j} = 0$  for all  $j\in [k]$ . The normalized vector has a unit norm, hence we have
$$
\left\| V \Gamma_ {i} + v _ {i} ^ {\perp} \right\| ^ {2} = \Gamma_ {i} ^ {T} V ^ {T} V \Gamma_ {i} + \left\| v _ {i} ^ {\perp} \right\| ^ {2} = 1. \tag {30}
$$
The correlation vector can be written as  $u_{i} = V^{T}V\Gamma_{i}\in \mathbb{R}^{k}$  which yields the following constraint
$$
u _ {i} ^ {T} \left(V ^ {T} V\right) ^ {- 1} u _ {i} \leq 1. \tag {31}
$$
D.1 Equality Constraints in the Case  $d = k$
In this case, the teacher incoming vectors  $v_{1}, \ldots, v_{k}$  span the input domain  $\mathbb{R}^{d}$ . Each incoming vector (of the student) can be expressed as a linear combination of the teacher's incoming vectors
$$
\frac {w _ {i}}{r _ {i}} = \sum_ {j = 1} ^ {k} u _ {i j} v _ {j}, \quad \sum_ {j = 1} ^ {k} u _ {i j} ^ {2} = 1, \tag {32}
$$
and the equality constraint pops up since the normalized vector has a unit  $\ell_2$  norm. The correlations between the incoming vectors are then expressed in terms of the student-teacher correlations
$$
\rho_ {i i ^ {\prime}} = u _ {i} \cdot u _ {i ^ {\prime}}. \tag {33}
$$
Therefore the optimization problem is equivalent to
$$
\min \sum_ {i = 1} ^ {n} a _ {i} ^ {2} g _ {\sigma} (r _ {i}, r _ {i}, 1) + 2 \sum_ {i \neq i ^ {\prime}} a _ {i} a _ {i ^ {\prime}} g _ {\sigma} (r _ {i}, r _ {i ^ {\prime}}, \sum_ {j = 1} ^ {k} u _ {i j} u _ {i ^ {\prime} j}) - 2 \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {k} a _ {i} b _ {j} g _ {\sigma} (r _ {i}, \| v _ {j} \|, u _ {i j})
$$
$$
\text {s u b j e c t} \quad \sum_ {j = 1} ^ {k} u _ {i j} ^ {2} = 1, r _ {i} \geq 0, \quad \text {f o r a l l} i \in [ n ]. \tag {34}
$$
Since  $\binom{n}{2}$  student-student correlations terms are not free, the problem has only  $n(k + 2)$  free parameters and  $k$  equality constraints, yielding  $n(k + 1)$  effective parameters, which is the same number as the number of parameters of the original problem in the weight-space.
D.2 Binary-Equality Constraints in the Case  $d = k + 1$
In this case, there is only one direction orthogonal to the span of the teacher's incoming vectors (i.e.  $v_{i}^{\perp} \parallel v_{i'}^{\perp}$ ). Therefore the general inequality constraint on  $\rho_{ii'}$  reduces to
$$
\rho_ {i i ^ {\prime}} = u _ {i} \cdot u _ {i ^ {\prime}} \pm \sqrt {1 - \| u _ {i} \| ^ {2}} \sqrt {1 - \| u _ {i ^ {\prime}} \| ^ {2}}. \tag {35}
$$
D.3 Three-Neuron Network
We present a case for the three-neuron network where the optimal solution of the constrained optimization problem may not be projected back to the weight space. In particular, let us consider a positive and monotonic activation function, i.e.  $\sigma (x) > 0$ . This implies that  $g_{\sigma} > 0$  and that  $g_{\sigma}$  is increasing in correlation. It is natural to expect that all outgoing weights are positive since this would bring the network function closer to the target function. If the network is overparameterized, some outgoing weights may be zero or even negative (balanced by a positive outgoing weight corresponding to the same incoming vector). Let us pick three positive outgoing weights  $a_1,a_2,a_3 > 0$  and three corresponding student-student interaction terms
$$
\text {m i n i m i z e} a _ {1} a _ {2} g _ {\sigma} \left(r _ {1}, r _ {2}, \rho_ {1 2}\right) + a _ {1} a _ {3} g _ {\sigma} \left(r _ {1}, r _ {3}, \rho_ {1 3}\right) + a _ {2} a _ {3} g _ {\sigma} \left(r _ {2}, r _ {3}, \rho_ {2 3}\right) + \dots ,
$$
$$
\text {s u b j e c t} \left| \rho_ {i i ^ {\prime}} - u _ {i} \cdot u _ {i ^ {\prime}} \right| \leq \sqrt {1 - \| u _ {i} \| ^ {2}} \sqrt {1 - \| u _ {i ^ {\prime}} \| ^ {2}}. \tag {36}
$$
Note that each  $\rho_{ii'}$  is decoupled from each other. Since  $g_{\sigma}$  is increasing in correlation, the minimum of each term above is achieved when
$$
\rho_ {i i ^ {\prime}} = u _ {i} \cdot u _ {i ^ {\prime}} - \sqrt {1 - \| u _ {i} \| ^ {2}} \sqrt {1 - \| u _ {i ^ {\prime}} \| ^ {2}}. \tag {37}
$$
This implies that the inequality is tight and therefore  $v_{i}^{\perp} \parallel v_{i'}^{\perp}$  moreover,
$$
v _ {i} ^ {\perp} = \sqrt {1 - \| u _ {i} \| ^ {2}} v ^ {\perp} \quad \text {a n d} \quad v _ {i ^ {\prime}} ^ {\perp} = - \sqrt {1 - \| u _ {i ^ {\prime}} \| ^ {2}} v ^ {\perp} \tag {38}
$$
up to a sign flip. However, it is not possible that the three vectors all have pairwise flipped directions to each other as we would have  $(-)\cdot (-) = (+)$ . If the optimal solution of the constrained optimization problem verifies  $\| u_{i}\|^{2} < 1$ , then we conclude that it cannot be mapped back to the weight space; in this case, the optimal  $L_{\mathrm{proj}}$  would only give a lower bound on the optimal loss of the weight-space.
E Copy-Average Critical Points
Theorem E.1. Assume that  $\sigma$  is the erf activation function. We pick a copy-average parameter vector
$$
\theta^ {*} = \left(w _ {1} ^ {*}, a _ {1} ^ {*}\right) \oplus \dots \oplus \left(w _ {n} ^ {*}, a _ {n} ^ {*}\right) \tag {39}
$$
Proof. Let us write down the partial derivatives with respect to the outgoing weights and incoming vectors
$$
\frac {d}{d a _ {i}} L ^ {n, k} \left(\theta^ {*}\right) = 2 \mathbb {E} _ {x \sim \mathcal {D}} \left[ \sigma \left(w _ {i} ^ {*} \cdot x\right) \left(\sum_ {j = 1} ^ {n} a _ {j} ^ {*} \sigma \left(w _ {j} ^ {*} \cdot x\right) - f ^ {*} (x)\right) \right],
$$
$$
\frac {d}{d w _ {i}} L ^ {n, k} \left(\theta^ {*}\right) = 2 a _ {i} ^ {*} \mathbb {E} _ {x \sim \mathcal {D}} \left[ \sigma^ {\prime} \left(w _ {i} ^ {*} \cdot x\right) x \left(\sum_ {j = 1} ^ {n} a _ {j} ^ {*} \sigma \left(w _ {j} ^ {*} \cdot x\right) - f ^ {*} (x)\right) \right]. \tag {40}
$$
We will show that they are equivalent to the following
$$
\frac {d}{d a _ {i}} L ^ {n, k} \left(\theta^ {*}\right) = 2 \mathbb {E} _ {x \sim \mathcal {D}} \left[ \sigma \left(w _ {i} ^ {*} \cdot x\right) \left(a _ {i} ^ {*} \sigma \left(w _ {i} ^ {*} \cdot x\right) - f _ {i} ^ {*} (x)\right) \right],
$$
$$
\frac {d}{d w _ {i}} L ^ {n, k} \left(\theta^ {*}\right) = 2 a _ {i} ^ {*} \mathbb {E} _ {x \sim \mathcal {D}} \left[ \sigma^ {\prime} \left(w _ {i} ^ {*} \cdot x\right) x \left(a _ {i} ^ {*} \sigma \left(w _ {i} ^ {*} \cdot x\right) - f _ {i} ^ {*} (x)\right) \right], \tag {41}
$$
which implies that the partial derivatives are zero, since  $(w_{i}^{*},a_{i}^{*})$  is a critical point of the loss
$$
L ^ {1, \ell_ {i}} = \mathbb {E} _ {x \sim \mathcal {D}} \left[ \left(a \sigma (w \cdot x) - f _ {i} ^ {*} (x)\right) ^ {2} \right]. \tag {42}
$$
Since  $(w_{i}^{*},a_{i}^{*})$  is the optimal solution of the teacher network generated by  $v_{s_{i - 1} + 1},\ldots ,v_{s_i}$ , from Theorem 5.1, we have that  $w_{i}^{*}$  is in the span of  $v_{s_{i - 1} + 1},\ldots ,v_{s_i}$ . We have that
$$
w _ {i} ^ {*} \cdot w _ {i ^ {\prime}} ^ {*} = 0 \quad \text {a n d} \quad w _ {i} ^ {*} \cdot v _ {j} = 0 \quad \text {f o r} \quad j \in [ k ] \backslash [ s _ {i - 1} + 1, s _ {i} ] \tag {43}
$$
since the two incoming vectors are in the span of two orthogonal subspaces respectively and  $w_{i}^{*}$  is orthogonal to all other teacher incoming vectors that are outside of the span of  $v_{s_{i-1}+1}, \ldots, v_{s_i}$ . For two orthogonal vectors say  $w_{i}^{*}$  and  $v$ , we have that
$$
\mathbb {E} _ {x \sim \mathcal {D}} [ \sigma_ {1} (w _ {i} ^ {*} \cdot x) \sigma_ {2} (v \cdot x) ] = \mathbb {E} _ {x \sim \mathcal {D}} [ \sigma_ {1} (w _ {i} ^ {*} \cdot x) ] \mathbb {E} _ {x \sim \mathcal {D}} [ \sigma_ {2} (v \cdot x) ] \tag {44}
$$
which is zero if at least one of  $\sigma_{1}$  and  $\sigma_{2}$  is odd. This implies the first equation in 41 for the partial derivatives with respect to the outgoing weights. In order to show the second equation for the partial derivatives with respect to the incoming vectors, let us define  $S_{j} = [s_{j - 1} + 1,s_{j}]$ ,
$$
W _ {i, j} = \mathbb {E} _ {x \sim \mathcal {D}} \left[ \sigma^ {\prime} \left(w _ {i} ^ {*} \cdot x\right) x \left(a _ {j} ^ {*} \sigma \left(w _ {j} ^ {*} \cdot x\right) - \sum_ {k \in S _ {j}} \sigma \left(v _ {k} \cdot x\right)\right) \right], \tag {45}
$$