SlowGuess/NeurIPS_2023_OCR huggingface dataset:
and note that it suffices to show that  $W_{i,j} = 0$  for all  $i \neq j$ . This is true if and only if  $W_{i,j} \cdot \bar{v}_{\ell} = 0$  where  $\{\bar{v}_1, \dots, \bar{v}_d\}$  form an orthogonal basis of  $\mathbb{R}^d$ . Let us choose  $\bar{v}_1 = v_1, \dots, \bar{v}_k = v_k$  and that  $\bar{v}_{k+1}, \dots, \bar{v}_d$  completes the basis if  $d > k$ . One can observe that  $W_{i,j} \cdot \bar{v}_{\ell} = 0$  for  $k + 1 \leq \ell \leq d$  since  $x \cdot \bar{v}_{\ell}$  is an independent Gaussian from the others. Hence, the expectation, i.e.  $W_{i,j} \cdot \bar{v}_{\ell}$ , factorizes with a factor of  $\mathbb{E}[x \cdot \bar{v}_{\ell}]$  which is zero.
It remains to check  $W_{i,j} \cdot v_l = 0$  for all  $v_l$ 's. We split the analysis into two cases. If  $v_l \notin S_j$ , then  $x \cdot v_l$  is independent from  $x \cdot w_j^*$  and from  $x \cdot v_k$  for  $k \in S_j$ . Hence  $W_{i,j}$  splits into
$$
W _ {i, j} \cdot v _ {l} = \mathbb {E} _ {x \sim \mathcal {D}} \left[ \sigma^ {\prime} \left(w _ {i} ^ {*} \cdot x\right) x \cdot v _ {l} \right] \mathbb {E} _ {x \sim \mathcal {D}} \left[ \left(a _ {j} ^ {*} \sigma \left(w _ {j} ^ {*} \cdot x\right) - \sum_ {k \in S _ {j}} \sigma \left(v _ {k} \cdot x\right)\right) \right] = 0, \tag {46}
$$
where the second term in the product is zero because  $w_{j}^{*}\cdot x$  and  $v_{k}\cdot x$  are centered Gaussian and  $\sigma$  is odd. For the second case, where  $v_{l}\in S_{j}$ , using the fact that  $x\cdot v_{l}$  is independent from  $w_{i}^{*}\cdot x$  and from  $x\cdot v_{k}$  for  $l\neq k$ , we have
$$
\begin{array}{l} W _ {i, j} \cdot v _ {l} = \mathbb {E} _ {x \sim \mathcal {D}} \left[ \sigma^ {\prime} \left(w _ {i} ^ {*} \cdot x\right) \right] \mathbb {E} _ {x \sim \mathcal {D}} \left[ x \cdot v _ {l} \left(a _ {j} ^ {*} \sigma \left(w _ {j} ^ {*} \cdot x\right) - \sum_ {k \in S _ {j}} \sigma \left(v _ {k} \cdot x\right)\right) \right] (47) \\ = \mathbb {E} _ {x \sim \mathcal {D}} \left[ \sigma^ {\prime} \left(w _ {i} ^ {*} \cdot x\right) \right] \mathbb {E} _ {x \sim \mathcal {D}} \left[ x \cdot v _ {l} \left(a _ {j} ^ {*} \sigma \left(w _ {j} ^ {*} \cdot x\right) - \sigma \left(v _ {l} \cdot x\right)\right) \right] (48) \\ = \mathbb {E} _ {x \sim \mathcal {D}} \left[ \sigma^ {\prime} \left(w _ {i} ^ {*} \cdot x\right) \right] \left(\mathbb {E} _ {x \sim \mathcal {D}} \left[ a _ {j} ^ {*} \sigma \left(w _ {j} ^ {*} \cdot x\right) x \cdot v _ {l} \right] - \mathbb {E} _ {x \sim \mathcal {D}} \left[ \sigma \left(v _ {l} \cdot x\right) x \cdot v _ {l} \right]\right). (49) \\ \end{array}
$$
Applying Stein's Lemma to both terms on the right we have
$$
\left. W _ {i, j} \cdot v _ {l} = \mathbb {E} _ {x \sim \mathcal {D}} \left[ \sigma^ {\prime} \left(w _ {i} ^ {*} \cdot x\right) \right] \left(\mathbb {E} \left[ a _ {j} ^ {*} r _ {j} ^ {*} u ^ {*} \sigma^ {\prime} \left(r _ {j} ^ {*} Z\right) \right] - \mathbb {E} \left[ \sigma^ {\prime} (Z) \right]\right)\right), \tag {50}
$$
where  $Z$  is standard Gaussian and  $r_j^* = \sqrt{\frac{1}{2k - 1}}$ ,  $u^* = \sqrt{\frac{1}{k}}$ ,  $a_j^* = k$  (parameters of erf). Hence we want to show that
$$
a _ {j} ^ {*} r _ {j} ^ {*} u ^ {*} \mathbb {E} \left[ \sigma^ {\prime} \left(r _ {j} ^ {*} Z\right) \right] = \mathbb {E} \left[ \sigma^ {\prime} (Z) \right]. \tag {51}
$$
To show this, we use the following relation [2, 3]
$$
g _ {\text {e r f}} (r, r, u) = \mathbb {E} [ \sigma (r x) \sigma (r y) ] = \frac {2}{\pi} \arcsin \left(\frac {r ^ {2} u}{r ^ {2} + 1}\right). \tag {52}
$$
Differentiating with respect to the correlation  $u$  we have
$$
\frac {d}{d u} g _ {\text {e r f}} (r, r, u) = r ^ {2} \mathbb {E} [ \sigma^ {\prime} (r x) \sigma^ {\prime} (r y) ] = \frac {2}{\pi} \frac {1}{\sqrt {1 - u ^ {2} \left(\frac {r ^ {2}}{r ^ {2} + 1}\right) ^ {2}}} \frac {r ^ {2}}{(r ^ {2} + 1)}. \tag {53}
$$
In particular, at correlation zero, we get
$$
\mathbb {E} \left[ \sigma^ {\prime} (r x) \right] = \sqrt {\frac {2}{\pi} \frac {1}{\left(r ^ {2} + 1\right)}} \tag {54}
$$
$$
\mathbb {E} \left[ \sigma^ {\prime} \left(r _ {j} ^ {*} x\right) \right] = \sqrt {\frac {2}{\pi} \left(\frac {2 k - 1}{2 k}\right)}, \quad \mathbb {E} \left[ \sigma^ {\prime} (x) \right] = \sqrt {\frac {1}{\pi}}, \tag {55}
$$
which implies 51 and the proof is complete.
For general activation functions, using the substitution in Eq.41, the first partial derivatives in Eq. 40 reduce to
$$
\frac {d}{d a _ {i}} L ^ {n, k} \left(\theta^ {*}\right) = 2 \sum_ {i \neq i ^ {\prime}} a _ {i ^ {\prime}} ^ {*} g _ {\sigma} \left(\| w _ {i} ^ {*} \|, \| w _ {i ^ {\prime}} ^ {*} \|, 0\right) - 2 \left(k - \ell_ {i}\right) g _ {\sigma} \left(\| w _ {i} ^ {*} \|, 1, 0\right) \tag {56}
$$
which is in general non-zero if  $\sigma$  is not odd.
Lemma E.2. Assume  $\ell_2 > \ell_1\geq 1$  . We have that
$$
L _ {e r f} ^ {*} (\ell_ {2} + 1) - L _ {e r f} ^ {*} (\ell_ {2}) <   L _ {e r f} ^ {*} (\ell_ {1} + 1) - L _ {e r f} ^ {*} (\ell_ {1}). \tag {57}
$$
Proof. We first show that the function  $x^{2} \arcsin \left( \frac{1}{2x} \right)$  is increasing for  $x \geq 1$  and convex for  $x > 0$ . Using the Taylor expansion of  $\arcsin$ , we have that
$$
f (x) = x ^ {2} \arcsin \left(\frac {1}{2 x}\right) = \frac {x}{2} + \frac {1}{2 \cdot 3} \frac {1}{2 ^ {3} x} + \frac {1 \cdot 3}{2 \cdot 4 \cdot 5} \frac {1}{2 ^ {5} x ^ {3}} + \dots \tag {58}
$$
where the higher-order terms all have positive coefficients. The first derivative is
$$
f ^ {\prime} (x) = \frac {1}{2} - \frac {1}{2 \cdot 3} \frac {1}{2 ^ {3} x ^ {2}} - \frac {1 \cdot 3 \cdot 3}{2 \cdot 4 \cdot 5} \frac {1}{2 ^ {5} x ^ {4}} + \dots \tag {59}
$$
which is positive for  $x \geq 1$  since we have
$$
\frac {1}{2 \cdot 3} \frac {1}{2 ^ {3} x ^ {2}} + \frac {1 \cdot 3 \cdot 3}{2 \cdot 4 \cdot 5} \frac {1}{2 ^ {5} x ^ {4}} + \dots <   \frac {1}{2 ^ {2} x ^ {2}} + \frac {1}{2 ^ {4} x ^ {4}} + \dots \leq \frac {1}{4} + \frac {1}{4 ^ {2}} + \frac {1}{4 ^ {3}} + \dots = \frac {1}{3}. \tag {60}
$$
The second derivative is
$$
f ^ {\prime \prime} (x) = 2 \frac {1}{2 \cdot 3} \frac {1}{2 ^ {3} x ^ {3}} + 4 \frac {1 \cdot 3 \cdot 3}{2 \cdot 4 \cdot 5} \frac {1}{2 ^ {5} x ^ {3}} + \dots \tag {61}
$$
which is positive for positive  $x$ .
First, let us show that Eq. 18 holds. Plugging in the analytic expressions for  $L_{\mathrm{erf}}^{*}(0,\cdot)$  and  $L_{\mathrm{erf}}^{*}(\cdot)$ , it is equivalent to
$$
\left. \ell_ {0} \frac {2}{\pi} \arcsin \left(\frac {1}{2}\right) > \ell_ {0} \frac {2}{\pi} \arcsin \left(\frac {1}{2}\right) - \frac {2}{\pi} \left((\ell_ {1} + \ell_ {0}) ^ {2} \arcsin \left(\frac {1}{2 (\ell_ {1} + \ell_ {0})}\right) - \ell_ {1} ^ {2} \arcsin \left(\frac {1}{2 \ell_ {1}}\right)\right). \right. \tag {62}
$$
Since  $f(x)$  is increasing for  $x \geq 1$ , the second term inside the parenthesis is positive, hence the inequality holds.
We will now prove the statement of the Lemma. It suffices to show the following for all  $\ell = \ell_2 \geq 2$
$$
L _ {\text {e r f}} ^ {*} (\ell + 1) - L _ {\text {e r f}} ^ {*} (\ell) <   L _ {\text {e r f}} ^ {*} (\ell) - L _ {\text {e r f}} ^ {*} (\ell - 1) \tag {63}
$$
since then we can continue to decrease  $\ell$  by one, i.e.  $\ell - 1, \ell - 2, \ldots$ , until we reach  $\ell_1$ . Eq. 63 is equivalent to
$$
L _ {\text {e r f}} ^ {*} (\ell + 1) - 2 L _ {\text {e r f}} ^ {*} (\ell) + L _ {\text {e r f}} ^ {*} (\ell - 1) <   0, \tag {64}
$$
that is the second-order finite difference, similar to the second-derivative of a continuous function. The proof is completed by observing that  $L_{\mathrm{erf}}^{*}(\cdot)$  is a discrete-concave function since its continuous interpolation
$$
L _ {\operatorname {e r f}} ^ {*} (x) = x \arcsin \left(\frac {1}{2}\right) - x ^ {2} \arcsin \left(\frac {1}{2 x}\right) \tag {65}
$$
is concave for  $x > 0$  since it can be written as  $L_{\mathrm{erf}}^{*}(x) = \alpha x - f(x)$  where  $f$  is convex for  $x > 0$ .
Lemma E.2 tells us that if we add one neuron to the teacher, then it is better to approximate it by the student neuron that already approximates many teacher neurons. Applying Lemma E.2 iteratively, we get
$$
L _ {\mathrm {e r f}} ^ {*} (k - 1) + L _ {\mathrm {e r f}} ^ {*} (1) <   L _ {\mathrm {e r f}} ^ {*} (k - 2) + L _ {\mathrm {e r f}} ^ {*} (2) <   \ldots <   L _ {\mathrm {e r f}} ^ {*} (\ell_ {2} + 1) + L _ {\mathrm {e r f}} ^ {*} (\ell_ {1}) <   L _ {\mathrm {e r f}} ^ {*} (\ell_ {2}) + L _ {\mathrm {e r f}} ^ {*} (\ell_ {1} + 1)
$$
where  $\ell_2 = \frac{k}{2},\ell_1 = \frac{k}{2} -1$  if  $k$  is even and  $\ell_2 = \frac{k + 1}{2},\ell_1 = \frac{k - 3}{2}$  if  $k$  is odd.
Theorem E.3. Consider a unit-orthonormal teacher network  $f^{*}(x) = \sum_{j=1}^{k} \sigma(v_{j} \cdot x)$  and the erf activation function. For an under-parameterized student network with  $n$  neurons, the minimum-loss copy-average configuration up to permutations (of the student and teacher neurons) is
$$
\theta = \left(\epsilon_ {1} v _ {1}, \epsilon_ {1}\right) \oplus \dots \oplus \left(\epsilon_ {n - 1} v _ {n - 1}, \epsilon_ {n - 1}\right) \oplus \left(\epsilon_ {n} w _ {n} ^ {*}, \epsilon_ {n} a _ {n} ^ {*}\right) \tag {66}
$$
Proof. We will conclude with a simple argument that the minimum-loss CA configuration for a multi-neuron network with  $n$  neurons is  $(n - 1)$ -C-1-A. In particular, if there are two averaging neurons inside the student network, we can redistribute the teacher neurons shared between these two to a lower-loss CA configuration by ensuring that one student neuron copies and the other student neuron averages (see Lemma E.2). The minimum-loss CA point is then achieved among CA configurations where at least  $n - 1$  neurons each copy a single teacher neuron (of the  $k$  possible ones). The remaining student neuron can be treated as a single-neuron network learning from a teacher with  $k - n + 1$  neurons – for which we know the optimal solution is to average (Theorem 5.1).
E.1 Number of CA Critical Points
There is a combinatorial number of  $(\ell_1,\dots ,\ell_n)$ -CAC critical points, that is
$$
c \left(\ell_ {1}, \dots , \ell_ {n}\right) \binom {k} {\ell_ {1}} \dots \binom {k - \left(\ell_ {1} + \dots + \ell_ {n}\right)} {\ell_ {n}} \tag {67}
$$
where  $c_{n} \coloneqq c(\ell_{1},\dots,\ell_{n})$  counts distinguishable permutations between the neurons of the student network, and the binomial coefficients stand for grouping teacher neurons into  $n$  non-empty buckets.
If  $\ell_1 = \ldots = \ell_n$ , permutation between the student neurons is already counted when distributing the teacher neurons, hence  $c_{n} = 1$ . If all  $\ell_1,\dots,\ell_n$  are distinct from each other, we have that  $c_{n} = n!$  since we swap all pairs of student neurons after assigning groups of teacher neurons. In general, let  $c_{i}$  denote the number of  $i$ 's among  $\ell_1,\ldots ,\ell_n$  for all  $i = 1,\ldots ,k$ ; the formula for the permutation-factor is given by
$$
c \left(\ell_ {1}, \dots , \ell_ {n}\right) = \frac {n !}{c _ {1} ! \dots c _ {k} !}. \tag {68}
$$
F General Properties of the Interaction Function
In this Section, we introduce some general properties of the interactions. We use these only for the one-neuron network in this paper (see Section G), however, these properties are likely to play a role in studying the networks with two or more neurons.
We first present the partial derivative of a general interaction function, i.e. two activation functions may be different, for example, if the student activation function does not match the teacher, with respect to the correlation in a simple expression in Lemma F.1. In the second part, we present a property of the activation function sufficient for Assumption 3.1 (ii), and show that the differentiable activation functions studied in this paper satisfy this property in Lemma F.2.
Lemma F.1. Assume that functions  $\sigma_{1}$  and  $\sigma_{2}$  are differentiable. The partial derivative of the following Gaussian integral term  $\mathbb{E}[\sigma_1(r_1x)\sigma_2(r_2y)]$  with respect to the correlation  $\mathbb{E}[xy] = u$  is
$$
\frac {d}{d u} \mathbb {E} \left[ \sigma_ {1} \left(r _ {1} x\right) \sigma_ {2} \left(r _ {2} y\right) \right] = r _ {1} r _ {2} \mathbb {E} \left[ \sigma_ {1} ^ {\prime} \left(r _ {1} x\right) \sigma_ {2} ^ {\prime} \left(r _ {2} y\right) \right]. \tag {69}
$$
We apply the Lemma for  $\sigma_{1} = \sigma_{2} = \sigma$  in the main text in Eq. 9.
Proof. We compute the derivative of  $\mathbb{E}[\sigma_1(r_1x)\sigma_2(r_2y)]$  by making the correlation  $u$  explicit. Denote  $u' = \sqrt{1 - u^2}$  and  $y = ux + u'z$ . After the computation, we use Stein's lemma to reach the desired formula.
$$
\partial_ {u} \mathbb {E} \left[ \sigma_ {1} \left(r _ {1} x\right) \sigma_ {2} \left(r _ {2} y\right) \right] = r _ {2} \mathbb {E} \left[ \sigma_ {1} \left(r _ {1} x\right) \sigma_ {2} ^ {\prime} \left(r _ {2} y\right) x \right] - \frac {r _ {2} u}{u ^ {\prime}} \mathbb {E} \left[ \sigma_ {1} \left(r _ {1} x\right) \sigma_ {2} ^ {\prime} \left(r _ {2} y\right) z \right] \tag {70}
$$
where  $x$  and  $z$  are independent standard Gaussians. Here is a reminder for Stein's Lemma for a standard Gaussian  $z$
$$
\mathbb {E} [ v (z) z ] = \mathbb {E} [ v ^ {\prime} (z) ]. \tag {71}
$$
To remove  $x$  in the first term, we apply Stein's formula for  $v(x) = \sigma_1(r_1x)\sigma_2'(r_2(ux + u'z))$  yielding
$$
r _ {1} r _ {2} \mathbb {E} \left[ \sigma_ {1} ^ {\prime} (r _ {1} x) \sigma_ {2} ^ {\prime} (r _ {2} y) \right] + r _ {2} ^ {2} u \mathbb {E} \left[ \sigma_ {1} (r _ {1} x) \sigma_ {2} ^ {\prime \prime} (r _ {2} y) \right]. \tag {72}
$$
To remove  $z$  in the second term, we apply Stein's formula for  $v(z) = \sigma_2'(r_2(ux + u'z))$  by considering fixed  $x$  which yields
$$
- r _ {2} ^ {2} u \mathbb {E} \left[ \sigma_ {1} \left(r _ {1} x\right) \sigma_ {2} ^ {\prime \prime} \left(r _ {2} y\right) \right]. \tag {73}
$$
Summing up the two terms completes the proof.
For softplus that is increasing and convex, using Lemma F.1 for  $\sigma_{1} = \sigma_{2} = \sigma$  twice, we infer that the interaction  $g$  is also increasing and convex in  $u$ . Hence, for  $u < 0$ , Assumption 3.1 (ii) holds for softplus. However, for the other activation functions, using second-order derivatives does not suffice to show the assumption. We will propose a new property of the activation function that implies that the interaction satisfies Assumption 3.1 (ii) and prove that softplus with  $\beta \leq 2$ , sigmoid, tanh, and erf satisfy this property.
Lemma F.2. If the activation function  $\sigma$  is thrice-differentiable and it satisfies
$$
\sigma^ {\prime} (x) - x \sigma^ {\prime \prime} (x) + \sigma^ {\prime \prime \prime} (x) > 0, \tag {74}
$$
then its interaction satisfies Assumption 3.1 (ii) for all  $u \in (-1,1)$ . Softplus with  $\beta \in (0,2]$ , sigmoid, tanh, and erf activation functions satisfy the above inequality.
Proof. Let us first write out Assumption 3.1 (ii) explicitly using Lemma F.1
$$
r _ {1} u \mathbb {E} \left[ \bar {\sigma} ^ {\prime} \left(r _ {1} x\right) \bar {\sigma} ^ {\prime} (y) \right] <   \mathbb {E} \left[ \bar {\sigma} \left(r _ {1} x\right) \bar {\sigma} (y) \right]. \tag {75}
$$
where  $\bar{\sigma} (x) = \sigma '(x)$ . Using Stein's Lemma for  $v(x) = \bar{\sigma} (r_1x)\bar{\sigma}'(y)$ , we get
$$
\mathbb {E} \left[ \bar {\sigma} \left(r _ {1} x\right) \bar {\sigma} ^ {\prime} (y) x \right] = \mathbb {E} \left[ \bar {\sigma} ^ {\prime} \left(r _ {1} x\right) \bar {\sigma} ^ {\prime} (y) \right] r _ {1} + \mathbb {E} \left[ \bar {\sigma} \left(r _ {1} x\right) \bar {\sigma} ^ {\prime \prime} (y) \right] u. \tag {76}
$$
The desired inequality is equivalent to
$$
\mathbb {E} \left[ \bar {\sigma} \left(r _ {1} x\right) (\bar {\sigma} (y) - \bar {\sigma} ^ {\prime} (y) x u + \bar {\sigma} ^ {\prime \prime} (y) u ^ {2}) \right] > 0. \tag {77}
$$
Let us introduce  $f(x) = \bar{\sigma}(x) - x\bar{\sigma}'(x) + \bar{\sigma}''(x)$ . For  $y = ux + u'z$  where  $u' = \sqrt{1 - u^2}$ , we have the conditional average of  $y$  fixing  $x$  (we drop conditioning on the right-hand terms for convenience)
$$
\begin{array}{l} \mathbb {E} [ f (y) | x ] = \mathbb {E} [ \bar {\sigma} (y) ] - \mathbb {E} [ y \bar {\sigma} ^ {\prime} (y) ] + \mathbb {E} [ \bar {\sigma} ^ {\prime \prime} (y) ] \\ = \mathbb {E} [ \bar {\sigma} (y) ] - u x \mathbb {E} [ \bar {\sigma} ^ {\prime} (y) ] - \mathbb {E} [ u ^ {\prime} z \bar {\sigma} ^ {\prime} (y) ] + \mathbb {E} [ \bar {\sigma} ^ {\prime \prime} (y) ] \\ = \mathbb {E} [ \bar {\sigma} (y) ] - u x \mathbb {E} [ \bar {\sigma} ^ {\prime} (y) ] - \left(u ^ {\prime}\right) ^ {2} \mathbb {E} [ \bar {\sigma} ^ {\prime \prime} (y) ] + \mathbb {E} [ \bar {\sigma} ^ {\prime \prime} (y) ] \\ = \mathbb {E} [ \bar {\sigma} (y) ] - u x \mathbb {E} [ \bar {\sigma} ^ {\prime} (y) ] + u ^ {2} \mathbb {E} [ \bar {\sigma} ^ {\prime \prime} (y) ], \tag {78} \\ \end{array}
$$
where second last equality comes from Stein's Lemma for  $v(z) = \bar{\sigma}'(ux + u'z)$ . Hence the desired inequality is equivalent to
$$
\mathbb {E} \left[ \bar {\sigma} \left(r _ {1} x\right) f (y) \right] > 0. \tag {79}
$$
By straightforward calculus, we will show that  $f(x) > 0$ , or that  $f(x) \geq 0$  and  $f(x) = 0$  if and only if  $x = 0$ . In the latter case, the expectation in Eq. 79 is positive since  $f(y) > 0$  for some  $y$  values of the integrand. First, for the sigmoid and tanh activation functions, for which we have
$$
\bar {\sigma} (x) = \frac {e ^ {x}}{\left(e ^ {x} + 1\right) ^ {2}}, \bar {\sigma} ^ {\prime} (x) = \frac {e ^ {x} \left(1 - e ^ {x}\right)}{\left(e ^ {x} + 1\right) ^ {3}}, \bar {\sigma} ^ {\prime \prime} (x) = \frac {e ^ {x} \left(e ^ {2 x} - 4 e ^ {x} + 1\right)}{\left(e ^ {x} + 1\right) ^ {4}}. \tag {80}
$$
Hence, we can explicitly write  $f$  as
$$
\begin{array}{l} f (x) = \frac {e ^ {x}}{\left(e ^ {x} + 1\right) ^ {2}} - x \frac {e ^ {x} \left(1 - e ^ {x}\right)}{\left(e ^ {x} + 1\right) ^ {3}} + \frac {e ^ {x} \left(e ^ {2 x} - 4 e ^ {x} + 1\right)}{\left(e ^ {x} + 1\right) ^ {4}} (81) \\ = \frac {e ^ {x}}{\left(e ^ {x} + 1\right) ^ {4}} \left(\left(e ^ {x} + 1\right) ^ {2} - x \left(1 - e ^ {x}\right) \left(e ^ {x} + 1\right) + \left(e ^ {2 x} - 4 e ^ {x} + 1\right)\right). (82) \\ \end{array}
$$
Therefore showing  $f(x) > 0$  is equivalent to showing that the factor on the right, that is,
$$
2 e ^ {x} \left(e ^ {x} - 1\right) + 2 - x \left(1 - e ^ {2 x}\right) \tag {83}
$$
is positive. For  $x < 0$ , we have  $e^x < 1$  which implies  $-x(1 - e^{2x}) > 0$  and  $(1 - e^x)e^x \leq 1/4$  due to the inequality of arithmetic and geometric means hence the first term is upper bounded by  $-1/2$  and since we have  $+2$ , the whole term is positive. For  $x \geq 0$ , we have  $e^x \geq 1$ , hence we can rewrite the inequality as a sum of non-negative terms
$$
2 e ^ {x} \left(e ^ {x} - 1\right) + 2 + x \left(e ^ {2 x} - 1\right) > 0. \tag {84}
$$
Let us now handle the case of erf. Its first three derivatives are given by
$$
\bar {\sigma} (x) = \frac {2}{\sqrt {\pi}} e ^ {- x ^ {2} / 2}, \bar {\sigma} ^ {\prime} (x) = - \frac {2}{\sqrt {\pi}} x e ^ {- x ^ {2} / 2}, \bar {\sigma} ^ {\prime \prime} (x) = \frac {2}{\sqrt {\pi}} \left(x ^ {2} e ^ {- x ^ {2} / 2} - e ^ {- x ^ {2} / 2}\right) \tag {85}
$$
$$
f (x) = \frac {2}{\sqrt {\pi}} e ^ {- x ^ {2} / 2} \left(1 + x x + x ^ {2} - 1\right) = \frac {4}{\sqrt {\pi}} e ^ {- x ^ {2} / 2} x ^ {2} \tag {86}
$$
that is non-negative for all  $x$  and zero iff  $x = 0$ .
Finally, for the softplus activation function with  $\beta \in (0,2]$ , we have the following derivatives
$$
\bar {\sigma} (x) = \frac {e ^ {\beta x}}{\left(e ^ {\beta x} + 1\right)}, \bar {\sigma} ^ {\prime} (x) = \frac {\beta e ^ {\beta x}}{\left(e ^ {\beta x} + 1\right) ^ {2}}, \bar {\sigma} ^ {\prime \prime} (x) = \frac {\beta^ {2} e ^ {\beta x} \left(1 - e ^ {\beta x}\right)}{\left(e ^ {\beta x} + 1\right) ^ {3}}. \tag {87}
$$
Plugging in the function  $f$ , we get
$$
\begin{array}{l} f (x) = \frac {e ^ {\beta x}}{\left(e ^ {\beta x} + 1\right)} - x \frac {\beta e ^ {\beta x}}{\left(e ^ {\beta x} + 1\right) ^ {2}} + \frac {\beta^ {2} e ^ {\beta x} \left(1 - e ^ {\beta x}\right)}{\left(e ^ {\beta x} + 1\right) ^ {3}} (88) \\ = \frac {e ^ {\beta x}}{\left(e ^ {\beta x} + 1\right) ^ {3}} \left(\left(e ^ {\beta x} + 1\right) ^ {2} - x \beta \left(e ^ {\beta x} + 1\right) + \beta^ {2} \left(1 - e ^ {\beta x}\right)\right) (89) \\ \end{array}
$$
$$
e ^ {2 \beta x} + e ^ {\beta x} (2 - x \beta - \beta^ {2}) + 1 - x \beta + \beta^ {2} \tag {90}
$$
is positive. For  $x \leq 0$ , we have that  $-x\beta > 0$  and  $2 - \beta^2 \geq -2$  since  $\beta \leq 2$ , hence it is sufficient to show that the following is positive
$$
e ^ {2 \beta x} - 2 e ^ {\beta x} + 1 + \beta^ {2} = (e ^ {\beta x} - 1) ^ {2} + \beta^ {2} \tag {91}
$$
which is a sum of squares. For  $x > 0$ , in the rest of the proof we will show that
$$
e ^ {\beta x} \left(e ^ {\beta x} + 2 - x \beta - \beta^ {2}\right) + 1 - x \beta + \beta^ {2} > 0, \tag {92}
$$
for  $\beta \in (0,2]$ . Using  $e^{\beta x} \geq (\beta x)^{2}/2 + \beta x + 1$ , it suffices to show that
$$
e ^ {\beta x} \left(\left(\beta x\right) ^ {2} / 2 + 3 - \beta^ {2}\right) + 1 - x \beta + \beta^ {2} > 0. \tag {93}
$$
If  $(\beta x)^2 / 2 + 3 - \beta^2 \geq 1$ , then the first term is bigger than  $\beta x + 1$  hence the above term is positive. The remaining possibility is that we have
$$
\frac {x ^ {2}}{2} <   1 - \frac {2}{\beta^ {2}}. \tag {94}
$$
$\beta \leq 2$  implies  $x < 1$  and  $x^2 > 0$  implies  $\beta > \sqrt{2}$ . Hence we have  $-x\beta + \beta^2 > 0$  since  $\beta > x$ . Therefore, if we have  $(\beta x)^2 / 2 + 3 - \beta^2 \geq 0$ , Eq. 92 is positive. Assuming the opposite, we get
$$
\frac {x ^ {2}}{2} <   1 - \frac {3}{\beta^ {2}}, \tag {95}
$$
$\beta \leq 2$  implies  $x <   1 / \sqrt{2}$  and  $x^{2} > 0$  implies  $\beta >\sqrt{3}$
Going back to Eq. 92, what remains to show is that it is positive in the domain  $x < 1 / \sqrt{2}, \beta \in (\sqrt{3}, 2]$ . It suffices to show that  $e^{\beta x} + 2 - x\beta - \beta^2 > 0$ . Assuming the contrary implies  $e^{\beta x} < x\beta + 2$  since  $\beta \leq 2$ . We can then deduce that  $x\beta < c = 1.2$  since otherwise we would have
$$
\begin{array}{l} e ^ {\beta x} = 1 + \beta x + \frac {(\beta x) ^ {2}}{2 !} + \frac {(\beta x) ^ {3}}{3 !} + \dots (96) \\ \geq 1 + \beta x + \frac {c ^ {2}}{2 !} + \frac {c ^ {3}}{3 !} + \dots = 1 + \beta x + (e ^ {c} - c - 1) > 1 + \beta x + 1 (97) \\ \end{array}
$$
which implies a contradiction.  $c$  can be chosen smaller but this will be enough for our purposes.
Assuming  $e^{\beta x} + 2 - x\beta - \beta^2 \leq 0$ , let us expand Eq. 92
$$
e ^ {\beta x} \left(e ^ {\beta x} + 2 - x \beta - \beta^ {2}\right) + 1 - x \beta + \beta^ {2} \geq (\text {u s i n g} e ^ {\beta x} <   \beta x + 2) \tag {98}
$$
$$
(x \beta + 2) e ^ {\beta x} + (x \beta + 2) \left(2 - x \beta - \beta^ {2}\right) + 1 - x \beta + \beta^ {2} = \tag {99}
$$
$$
(x \beta + 2) e ^ {\beta x} - (x \beta) ^ {2} - (1 + \beta^ {2}) x \beta + 5 - \beta^ {2} > (\text {u s i n g} e ^ {\beta x} > \beta x + 1) \tag {100}
$$
$$
7 - \beta^ {2} + (2 - \beta^ {2}) x \beta \geq 3 - 2 x \beta > 0 \tag {101}
$$
where in the last inequality we used  $x\beta < 1.2$ . We note that this inequality holds for slightly larger  $\beta$  using the same technique, however, for significantly larger  $\beta$ , the property breaks down.
G The One-Neuron Network
We study the critical points of the following loss function
$$
L ^ {1, k} (w, a) = \mathbb {E} _ {x \sim \mathcal {D}} [ a \sigma (w \cdot x) - \sum_ {j = 1} ^ {k} b _ {j} \sigma (v _ {j} \cdot x) ], \tag {102}
$$
in particular, the optimal solution. For  $n = 1$ , all configurations of order parameters are realizable in the weight space, therefore, the optimal solution of the following loss (repeating Eq. 14)
$$
L _ {\text {p r o j}} ^ {1, k} = a ^ {2} g _ {\sigma} (r, r, 1) - 2 a \sum_ {j = 1} ^ {k} b _ {j} g _ {\sigma} (r, \| v _ {j} \|, u _ {j}) + \text {c o n s t}, \quad \text {s u b j e c t} \quad \sum_ {j = 1} ^ {k} u _ {j} ^ {2} \leq 1, r \geq 0, \tag {103}
$$