SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Finding the optimal solution for the multi-neuron network is challenging. Natural candidates are concatenation of student neurons where each one of them is a critical point of the loss function  $L^{1,\ell_i}$  where  $\ell_i$  is the number of the subgroup of teacher neurons. More precisely, let us pick a partition  $\ell_1 + \ldots + \ell_n \leq k$  such that  $\ell_i \geq 1$ , and define  $s_m = \sum_{i=1}^m \ell_i$  for  $m \leq n$  and  $s_0 = 0$ . We denote a one-neuron critical point by  $\theta_i^* = (w_i^*, a_i^*)$ , when learning from a part of the teacher network
$$
f _ {i} ^ {*} (x) = \sum_ {j = s _ {i - 1} + 1} ^ {s _ {i}} \sigma \left(v _ {j} \cdot x\right). \tag {11}
$$
Since  $f_{i}^{*}$  is a unit-orthonormal teacher,  $w_{i}^{*}$  is in the span of  $v_{s_{i - 1} + 1},\ldots ,v_{s_i}$  due to Proposition 4.1.
We use the term copy-average (CA) point or configuration to refer to the concatenation of such one-neuron critical points in the student network with  $n$  neurons: if  $\ell_i = 1$ , the student neuron copies one of the teacher neurons  $(v_j, 1)$ ; if  $\ell_i > 1$ , it averages a group of teacher neurons in the sense of approximating their sum with one neuron. For odd activation functions, the one-neuron network problems decouple from each other, as the cross-terms  $\mathbb{E}[\sigma(w_1 \cdot x) \sigma(w_2 \cdot x)]$  vanish for  $w_1 \perp w_2$ . For the specific case of erf, we prove that all the copy-average configurations are critical points.
Theorem 4.2. Assume that  $\sigma(x) = \sigma_{\text{erf}}(x) = \frac{2}{\sqrt{\pi}} \int_{0}^{\frac{x}{\sqrt{2}}} e^{-t^2} dt$ . We pick a copy-average parameter
$$
\theta^ {*} = \left(w _ {1} ^ {*}, a _ {1} ^ {*}\right) \oplus \dots \oplus \left(w _ {n} ^ {*}, a _ {n} ^ {*}\right) \tag {12}
$$
where  $(w_{i}^{*},a_{i}^{*})$  is a non-trivial critical point when learning from a unit-orthonormal teacher  $f_{i}^{*}$  with the incoming vectors  $v_{s_{i - 1} + 1},\ldots ,v_{s_i}$  shown in Eq. 11. Then  $\theta^*$  is a critical point of the loss function  $L^{n,k}$  where the target function is  $f^{*}(x) = \sum_{j = 1}^{k}\sigma (v_{j}\cdot x)$ .
In particular, all neurons are equivalent to each other in a unit-orthonormal teacher network. Therefore, the copy-average configurations where  $n - 1$  student neurons each copy a distinct teacher neuron and the  $n$ -th student neuron takes an average are also equivalent and called  $(n - 1)$ -copy-1-average, or  $(n - 1)$ -C-1-A in short. Another interesting configuration is where  $n$  student neurons each copy a distinct teacher neuron, which is called  $n$ -copy, or  $n$ -C in short.
For general activation functions the copy-average parameter vectors are not critical points (see Eq. 56). Nevertheless, we numerically find that the gradient flow converges to similar configurations for the ReLU activation function (see Figure 1, see Appendix C.3 for more experiments).
5 Approximation Error of Underparameterized Networks
The target function is assumed to be a unit-orthonormal teacher network in this section. In Subsection 5.1, we show for the one-neuron network that there is a unique non-trivial critical point up to symmetries, which is necessarily the global minimum (Theorem 5.1). Furthermore, we give the analytic expression of the optimal solution and its loss for erf (Corollary 5.2) and ReLU (Corollary G.5) activation functions. In Subsection 5.2, we provide for the under-parameterized student with  $n > 1$  neurons the exact loss of copy-average critical points for the erf activation function and show that the  $(n - 1)$ -copy-1-average configurations reach the lowest loss among CA-critical points (see also Appendix E.1 for the combinatorial number of the equivalent copy-average configurations related to the landscape complexity calculations [39]).
5.1 One-Neuron Network
Using the constrained optimization formulation in 10, we first prove that at any non-trivial critical point of the one-neuron network, the incoming vector aligns equally with all teacher's incoming vectors for unit-orthonormal teachers for activation functions satisfying Assumption 3.1 (see Theorem 5.1). This is related to the symmetric solution visited during the learning plateaus studied in Saad and Solla [2] for erf activation and in Tian [5] for ReLU activation (see Appendix B for a detailed comparison). Our proof works for a broad class of activation functions and does not use the analytic expression of the interaction function.
Theorem 5.1. Assume that the activation function satisfies Assumptions 3.1 (i) and (ii). At any non-trivial critical point  $(w^{*},a^{*})$  of the loss  $L^{1,k}$  for the unit-orthonormal teacher network, the incoming vector satisfies
$$
\frac {w ^ {*}}{\| w ^ {*} \|} = u ^ {*} \sum_ {j = 1} ^ {k} v _ {j} \tag {13}
$$
where  $u^{*}$  is either  $\frac{1}{\sqrt{k}}$  or  $-\frac{1}{\sqrt{k}}$
Proof Sketch. There is no student-student interaction term since we have a single neuron; therefore we write  $u_{j}$  instead of  $u_{1j}$  and the constrained optimization problem in 10 simplifies to
$$
\text {m i n i m i z e} a ^ {2} g _ {\sigma} (r, r, 1) - 2 a \sum_ {j = 1} ^ {k} g _ {\sigma} (r, 1, u _ {j}) + \text {c o n s t}, \quad \text {s u b j e c t t o} \| u \| \leq 1, r \geq 0. \tag {14}
$$
A erf
Figure 3: One-neuron network solutions. A Network output (color coded) as a function of input in  $d = 2$  for (left) a unit-orthonormal network with  $k = 2$  neurons (incoming vectors  $v_{1}$  and  $v_{2}$  are shown as black dots) and (right) the student network function generated by the optimal solution (incoming vector shown in red) for the erf activation function. B Same for the softplus activation function. C Approximation error of a student with  $n = 1$  neurons as a function of the number of  $k$  teacher neurons. For large  $k$ , the approximation error for  $n = 1$  grows near-linearly for the differentiable activation functions studied in this paper (erf, sigmoid, tanh, and softplus with  $\beta = 1$ ); however the growth is quadratic for ReLU (see Appendix Corollary G.5).
Bsoftplus
C one-neuron approx. error
From Proposition 4.1, we have that  $\| u \| = 1$  for any non-trivial critical point. Therefore, the constraint of 14 on the correlations  $u = (u_{1}, \ldots, u_{k})$  is satisfied. The mapping of any non-trivial critical point to the order-parameter space is a critical point of the Lagrangian loss (see Appendix Lemma G.2). Hence every  $u_{j}$  satisfies
$$
- 2 a \frac {d}{d u _ {j}} g _ {\sigma} (r, 1, u _ {j}) + 2 \lambda u _ {j} = 0 \tag {15}
$$
for fixed  $(r, a)$ . Assumption 3.1-(ii) implies that  $\frac{1}{u} \partial_{u} g_{\sigma}(r, 1, u)$  is one-to-one hence all  $u_{j}$  are equal. End of Proof Sketch.
We show in Lemma F.2 that the interactions of the common activation functions such as erf, tanh, sigmoid, and softplus (respectively)
$$
\sigma_ {\operatorname {e r f}} (x) = \frac {2}{\sqrt {\pi}} \int_ {0} ^ {\frac {x}{\sqrt {2}}} e ^ {- t ^ {2}} d t, \sigma_ {\tanh } (x) = \frac {1 - e ^ {- x}}{1 + e ^ {- x}}, \sigma_ {\operatorname {s i g}} (x) = \frac {1}{1 + e ^ {- x}}, \sigma_ {\operatorname {s o f t}} ^ {\beta} (x) = \frac {1}{\beta} \log (e ^ {\beta x} + 1),
$$
with  $\beta \in (0,2]$  satisfy Assumption 3.1-(ii). The interaction of the ReLU activation function, i.e.  $\sigma_{\mathrm{relu}}(x) = \max (0,x)$  also satisfies Assumptions 3.1 (with a slight modification in the domain for (ii); see the proof of Corollary G.5).
Thanks to Theorem 5.1, the loss in Eq. 14 can be reduced to a two-dimensional loss in  $a$  and  $r$ , which can be solved explicitly for ReLU (Corollary G.5) and erf.
Corollary 5.2. Assume that the activation function is  $\sigma_{\text{erf}}$ . The optimal solution  $(w^{*}, a^{*})$  is given by
$$
\| w ^ {*} \| = \sqrt {\frac {1}{2 k - 1}}, \quad a ^ {*} = k, \quad \frac {w ^ {*}}{\| w ^ {*} \|} = \frac {1}{\sqrt {k}} \sum_ {i = 1} ^ {k} v _ {i},
$$
or, equivalently, by  $(-w^{*}, -a^{*})$ . The optimal loss is then given by
$$
L _ {e r f} ^ {*} (k) = \frac {2}{\pi} \left(k \arcsin \left(\frac {1}{2}\right) - k ^ {2} \arcsin \left(\frac {1}{2 k}\right)\right). \tag {16}
$$
The proof of Corollary 5.2 is presented in Section G.3. For general activation functions, the two-dimensional loss does not admit an analytical expression. For this case, from the partial derivatives, we obtain a fixed-point equation in  $r$  which we solve numerically for the activation functions listed above (see Appendix Section G.2.2). For softplus, specifically, we prove in addition the following
Theorem 5.3. Assume that the activation function is  $\sigma_{\text{soft}}^{\beta}(x)$  with  $\beta \leq 2$ . The optimal solution  $(w^{*}, a^{*})$  satisfies  $\| w^{*} \| \leq 1 / \sqrt{k}$  and  $a^{*} \geq k$ .
We use the FKG inequality to prove Theorem 5.3 (see Appendix G.5). Although the proof requires specific properties of softplus, we show that the above bounds hold also for tanh and sigmoid by numerically solving the fixed point equation (Figure 10, also Figure 5). Note that the incoming vector norm is bounded above by  $1 / \sqrt{k}$ , hence  $w^{*}$  is a damped average of the teacher incoming vectors.
A Solutions of Gradient flow (GF)
B1 In all seeds, GF finds CA-optimal
B2 In some seeds, GF finds CA-optimal
Figure 4: Under-parameterized student networks of width  $n$  with erf activation function learning (via gradient flow) from a unit-orthonormal teacher network of width  $k$ . A Each dot is the mean error at convergence for 20 seeds of random initializations; black-dashed lines are the theory predictions  $L_{\mathrm{eff}}^* (k - n + 1)$ , see Eq. 20. Standard deviations do not show on the figure as they are too small. We identify four regimes indicated by colors (green-gray-blue-red) depending on the type of solution found by gradient flow (GF). In the green regime, GF converges to an optimal  $(n - 1)$ -C-1-A solution for all 20 initializations (Fig. 4-B1). In the gray regime, GF converges either to  $(n - 1)$ -C-1-A solution or to a "Perturbation of the all-copy solution" that we call P-n-C (Fig. 4-B2). In the blue and red regimes, for  $n > \gamma_2k$  where  $n = 8, 12, 16$  the gradient flow converges to a P-n-C solution from all seeds (Fig. 4-B3). Moreover, in the red regime, for  $n > \gamma_3k$  where  $n = 8, 12, 16$  and  $\gamma_3$  is near 0.75, the P-n-C solutions achieve lower loss than the  $(n - 1)$ -C-1-A solutions (Fig. 4-B4). B1-B4 Examples of loss at convergence (vertical axis) for all 20 different initialization seeds (horizontal axis); theory is shown by the red-dashed horizontal line. Insets show examples of correlation matrices  $u_{ij}$  ( $k$  lines,  $n$  columns) between student and teacher incoming vectors at convergence after reordering neurons. In the gray regime (for ex. B2) the gradient flow converges to either one of the two types of minima with correlations shown in the inset; in the other regimes, it consistently converges to the same minimum up to permutations.
B3 In all seeds, GF finds P-n-C solution
B4 In all seeds, GF finds P-n-C solution
Remark 5.4. We do not impose either of the two reductions that are common in literature: (i) incoming vector  $w$  is constrained to be on the unit sphere [9, 11, 14, 40], (ii) the outgoing weight  $a$  is constrained to be one [7, 18, 20]. An important step in our analysis is related to the norm  $r$  of the incoming vector which we discuss in Appendix Section G.2.2.
5.2 Multi-Neuron Network
In this subsection, we assume that the activation function is erf such that CA-configurations are critical points (see Theorem 4.2). For a student network with  $n = 2$  and a partition  $(\ell_1, \ell_2)$  we can decompose the loss of a CA critical point as
$$
L _ {\text {e r f}} ^ {*} \left(\ell_ {1}\right) + L _ {\text {e r f}} ^ {*} \left(\ell_ {2}\right) + L _ {\text {e r f}} ^ {*} \left(0, k - \left(\ell_ {1} + \ell_ {2}\right)\right) \tag {17}
$$
where  $L_{\mathrm{erf}}^{*}(0,\ell_{0}) \coloneqq \mathbb{E}_{x\sim \mathcal{D}}[f_{\ell_{0}}^{*}(x)^{2}]$  is the error made by a student with vanishing output when representing a reduced unit-orthonormal teacher network with  $\ell_0$  neurons. This decomposition is possible because the cross-terms between orthogonal vectors are zero for odd activation functions. Furthermore, for the erf activation function, we show that
$$
L _ {\text {e r f}} ^ {*} (\ell_ {1}) + L _ {\text {e r f}} ^ {*} (0, \ell_ {0}) > L _ {\text {e r f}} ^ {*} (\ell_ {1} + \ell_ {0}) \tag {18}
$$
(see the proof of Lemma E.2). Therefore, we should search for the minimum loss configuration among the partitions with  $\ell_1 + \ell_2 = k$ . Among such partitions, Lemma E.2 shows that the optimum CA-point has the partition  $(1,k - 1)$ . In words, the optimum is a 1-copy-1-average point.
For general  $n$ , using Lemma E.2 and a small trick, we prove the following.
Theorem 5.5. Consider a unit-orthonormal teacher network  $f^{*}(x) = \sum_{j=1}^{k} \sigma(v_{j} \cdot x)$  and the erf activation function. For an under-parameterized student network with  $n$  neurons, the minimum-loss copy-average configuration up to permutations (of the student and teacher neurons) is
$$
\theta = \left(\epsilon_ {1} v _ {1}, \epsilon_ {1}\right) \oplus \dots \oplus \left(\epsilon_ {n - 1} v _ {n - 1}, \epsilon_ {n - 1}\right) \oplus \left(\epsilon_ {n} w _ {n} ^ {*}, \epsilon_ {n} a _ {n} ^ {*}\right) \tag {19}
$$
where  $\epsilon_{i}\in \{\pm 1\}$  and  $(w_{n}^{*},a_{n}^{*})$  is given by Corollary 5.2 after substituting  $k$  with  $k - n + 1$
See Appendix E.3 for the proof. Because copy-average critical points are not necessarily the only critical points of the loss function for students with  $n > 1$ , we investigate in simulations, if they are found by gradient flow where the weights are initialized as Gaussian with a fixed standard deviation (see Fig. 4).
Interestingly, gradient flow converges to the CA-optimal solution for all random seeds in a broad regime of under-parameterization (green in Fig. 4). Only when  $n > \gamma_1 k$  for  $n = 8, 12, 16$  and  $\gamma_1 \sim 0.46$ , gradient flow converged in some seeds to points close to the  $n$ -copy critical point. We call these newly found points "perturbed  $n$ -copy" (P- $n$ -C) points. In gray-blue regimes, the P- $n$ -C points have higher loss than the optimal CA critical point (Fig. 4).
However, this is not always the case: when the student width is close to the teacher width (low compression regime), the P- $n$ -C point has a slightly lower loss the lowest amongst the CA critical points (red in Fig. 4). When  $k - n$  is fixed, we found that the  $(n - 1)$ -C-1-A solution turns from a minimum for small  $n$  to a saddle for large  $n$  (see App. Fig. 6); which explains why the gradient flow escapes it in this regime and converges to another minimum at a lower loss.
Finally, based on our theory and experiments, we conjecture that there exists a  $\gamma_0\in (0,\gamma_3)$  such that when  $n < \gamma_0k$  and when the activation function is erf, the global optimum of the non-convex loss in Eq. 5 is a  $(n - 1)$ -C-1-A configuration. Therefore, if our conjecture holds, the exact approximation error, i.e. the optimal loss, is identical to that of a one-neuron network approximating a teacher with  $k - n + 1$  neurons and is given by
$$
L _ {\text {e r f}} ^ {*} (n, k) = L _ {\text {e r f}} ^ {*} (k - n + 1). \tag {20}
$$
6 Conclusion & Future Directions
We studied the learning of under-parameterized student networks from orthogonal teacher networks for standard Gaussian input data and vanishing thresholds. For erf activation function, we introduced a new family of critical points that arise from the decoupling of the problem into one-neuron networks that can be solved separately. Moreover, the exact parameters of copy-average (CA) critical points are given which can be used to study escape behavior near saddles and to determine convergence of first and second-order optimization algorithms [16, 41].
Furthermore, we showed that the optimal CA point is that  $n - 1$  neurons copy teacher neurons and the  $n$ -th neuron averages the remaining  $k - n + 1$  neurons. In simulations, gradient flow converges to a CA-optimal solution for  $n < \gamma_1k$  where  $\gamma_{1}$  is near 0.46. However, for  $n > \gamma_2k$  where  $\gamma_{2}$  is near 0.6, we observe another phase where the gradient flow finds a perturbed copy solution. For the ReLU activation function and the onset of under-parameterization, gradient flow converges to qualitatively similar solutions; however, at the crossing point from under-parameterization to over-parameterization (i.e.  $n = k$ ), gradient flow is known to get stuck in spurious local minima [27]. On another note, determining the CA-optimal solution of the two-neuron network plays a critical role in our analysis. Still, there is only little literature on two-neuron networks [5, 42] compared to the well-studied one-neuron case [5-7, 22, 23]. The two-neuron network is possibly the simplest model with interactions between neurons, hence it is important to understand the global minimum and gradient flow dynamics of this challenging problem.
On the practical side, our analysis of under-parameterized networks gives a recipe for how to warm-start smaller neural networks for distilling unit-orthonormal teacher networks. If one desires low compression ( $n > \gamma_3k$ ), then we recommend initializing the student network in a configuration where each neuron copies a different teacher neuron, to be close to a P- $n$ -C point. However, for higher compression, we recommend initializing the student network in a configuration where  $n - 1$  neurons are each copied and the  $n$ -th neuron is initialized as an average neuron to be close to a ( $n - 1$ )-copy-1-average point. It remains an open question whether this recipe applies to non-idealized scenarios such as non-isotropic input distribution, teacher networks with non-orthogonal incoming vectors, or non-unit outgoing weights. More generally, it is natural to expect that the optimal distillation strategy changes from low compression levels to high compression levels. How exactly and where this change happens is a very intriguing question of theory and practice.
The authors thank Lenka Zdeborova for the discussions and encouragement at the beginning of this project and Clement Hongler for many discussions and valuable feedback. This work was supported by the Swiss National Science Foundation (no. 200020_207426).
[1] Ken-Ichi Funahashi. On the approximate realization of continuous mappings by neural networks. Neural networks, 2(3):183-192, 1989.
[2] David Saad and Sara A Solla. On-line learning in soft committee machines. Physical Review E, 52(4):4225, 1995.
[3] Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborova. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. Advances in neural information processing systems, 32, 2019.
[4] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances in neural information processing systems, 22, 2009.
[5] Yuandong Tian. An analytical formula of population gradient for two-layered relu network and its applications in convergence and critical point analysis. In International conference on machine learning, pages 3404-3413. PMLR, 2017.
[6] Song Mei, Yu Bai, and Andrea Montanari. The landscape of empirical risk for nonconvex losses. The Annals of Statistics, 46(6A):2747-2774, 2018.
[7] Gilad Yehudai and Shamir Ohad. Learning a single neuron with gradient methods. In Conference on Learning Theory, pages 3756-3786. PMLR, 2020.
[8] Charles M Stein. Estimation of the mean of a multivariate normal distribution. The annals of Statistics, pages 1135-1151, 1981.
[9] Rishabh Dudeja and Daniel Hsu. Learning single-index models in gaussian space. In Conference On Learning Theory, pages 1887-1930. PMLR, 2018.
[10] Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference. The Journal of Machine Learning Research, 22(1):4788–4838, 2021.
[11] Alberto Bietti, Joan Bruna, Clayton Sanford, and Min Jae Song. Learning single-index models with shallow neural networks. Advances in Neural Information Processing Systems, 35:9768-9783, 2022.
[12] Raphaël Berthier, Andrea Montanari, and Kangjie Zhou. Learning time-scales in two-layers neural networks. arXiv preprint arXiv:2303.00055, 2023.
[13] Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. Learning two-layer neural networks, one (giant) step at a time. arXiv preprint arXiv:2305.18270, 2023.
[14] Alex Damian, Eshaan Nichani, Rong Ge, and Jason D Lee. Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index models. arXiv preprint arXiv:2305.10633, 2023.
[15] Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. Advances in neural information processing systems, 29, 2016.
[16] Johanni Brea, Flavio Martinelli, Berfin Şimsek, and Wulfram Gerstner. Mlpgradientflow: going with the flow of multilayer perceptrons (and finding minima fast and accurately). arXiv preprint arXiv:2301.10638, 2023.
[37] Itay M Safran, Gilad Yehudai, and Ohad Shamir. The effects of mild over-parameterization on the optimization landscape of shallow relu neural networks. In Conference on Learning Theory, pages 3889-3934. PMLR, 2021.
[38] Alireza Mousavi-Hosseini, Sejun Park, Manuela Girotti, Ioannis Mitliagkas, and Murat A Erdogdu. Neural networks efficiently learn low-dimensional representations with sgd. arXiv preprint arXiv:2209.14863, 2022.
[39] Berfin Şimşek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning, pages 9722-9732. PMLR, 2021.
[40] Joan Bruna, Loucas Pillaud-Vivien, and Aaron Zweig. On single index models beyond gaussian data. arXiv preprint arXiv:2307.15804, 2023.
[41] Charles G Frye, James Simon, Neha S Wadia, Andrew Ligeralde, Michael R DeWeese, and Kristofer E Bouchard. Critical point-finding methods reveal gradient-flat regions of deep network losses. Neural computation, 33(6):1469-1497, 2021.
[42] Chenwei Wu, Jiajun Luo, and Jason D Lee. No spurious local minima in a two hidden unit relu network. Workshop track - ICLR, 2018.
[43] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[44] Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
A Summary of Results 14
B Further Comparison to Literature 14
C Further Experiments 16
C.1 One-Neuron Network 16
C.2 Erf Experiments 16
C.3 ReLU Experiments 18
D Constrained Optimization Formulation 19
D.1 Equality Constraints in the Case  $d = k$  20
D.2 Binary-Equality Constraints in the Case  $d = k + 1$  20
D.3 Three-Neuron Network 20
E Copy-Average Critical Points 21
E.1 Number of CA Critical Points 24
F General Properties of the Interaction Function 24
G The One-Neuron Network 27
G.1 Any Non-Trivial Critical Point Satisfies the Lagrangian Condition 27
G.2 General Activation Functions 28
G.3 Closed-Form Solution for Erf Activation 31
G.4 Closed-Form Solution for ReLU Activation 32
G.5 Bounds on Incoming Vector Norm and Outgoing Weight for Softplus 34
A Summary of Results
Table 1: Summary of Results
columns=conditions: lines=results:	Orthogonal	UO & erf	UO & σ satisfying gσ assumptions
n = 1 average is the optimal solution	n.i.g.	yes*	yes
n > 1 CA points are critical points	n.i.g.	yes	n.i.g. maybe for odd
n > 1 (n - 1)-C-1-A is the optimal-CA solution	n.i.g.	yes	n.i.g. maybe for some odd
n = 1 w* is in the span of {v1, ..., vk}	yes	yes*	yes*
In the table above, UO means unit-orthonormal, n.i.g. stands for 'not in general' and yes* follows as a special case from the results with yes on the same row.
B Further Comparison to Literature