SlowGuess/NeurIPS_2023_OCR huggingface dataset:
is equivalent to the optimal solution in the weight space. Let us denote the unit ball by  $B = \{(u_1, \dots, u_k) \mid u_1^2 + \dots + u_k^2 \leq 1\}$ . Its interior is denoted by int  $B$  and its boundary is denoted by  $\partial B$ .
We will present the results for the one-neuron network in five parts
1. In Subsection G.1, we give a proof of Proposition 4.1: any non-trivial critical point  $(w,a)$  of  $L^{1,k}$  satisfies that  $w$  is in the span of the teacher's incoming vectors if the activation function satisfies Assumption 3.1 (i). Moreover, we show in Lemma G.2 that the corresponding order parameters should satisfy a Lagrangian condition (Eq. 105).
2. In Subsection G.2, we characterize the topology of the loss landscape in terms of its critical points for the activation functions studied in this paper and for the unit-orthonormal teacher. Our results for the one-neuron network are strong in the sense that it gives all possible critical points of the loss landscape.
(a) In Subsection G.2.1, we give a proof of Theorem 5.1: for general activation functions satisfying Assumption 3.1, any non-trivial critical point of the one-neuron network attains equal correlations that are either  $\frac{1}{\sqrt{k}}$  or  $-\frac{1}{\sqrt{k}}$ .
(b) In Subsection G.2.2, we study the two-dimensional loss obtained after applying Theorem 5.1. From its derivative constraints, we get a fixed point equation (Eq. 120) that needs to be satisfied by the incoming vector norm  $r$  at any non-trivial critical point with equal correlations  $u$ . Numerically, we show that there is a unique solution of the fixed point equation for  $u > 0$  for differentiable activation functions studied in this paper (Fig. 10). Finally, we give some sufficient conditions in Eq. 121 to prove uniqueness based on log-concavity; numerically, these are shown to be satisfied by softplus and sigmoid activation functions.
3. In Subsection G.3, we give a proof of Corollary 5.2 by solving the two-dimensional loss for the erf activation function. Moreover, from the proof, we conclude that there are exactly two non-trivial critical points identical up to the mirror symmetry of erf (since it is odd); and these are the optimal solutions for the loss landscape.
4. In Subsection G.4, we present and prove Corollary G.5 by solving the two-dimensional loss for the ReLU activation function. We find that there are two non-trivial critical points of the loss function: a saddle 'point' at correlation  $-1/\sqrt{k}$  and the optimal 'solution' at correlation  $1/\sqrt{k}$ . Due to the positive homogeneity of ReLU, these are not two points but two equal-loss hyperbolas in the loss landscape.
5. In Subsection G.5, we study the two-dimensional loss for the softplus activation function and give a proof of Theorem 5.3. Absence of analytical expression for the Gaussian integral terms makes the problem challenging; we use several non-trivial steps in the proof. The proof shows that there is no critical point at correlation  $-1/\sqrt{k}$ ; and a non-trivial critical point  $(w^{*},a^{*})$  at correlation  $1/\sqrt{k}$  satisfies the following bounds:  $a^{*} \geq k$  and  $\| w^{*} \| \leq 1/\sqrt{k}$ . Numerically, we find that these bounds hold for other activation functions studied in this paper (tanh and sigmoid; Fig. 5).
G.1 Any Non-Trivial Critical Point Satisfies the Lagrangian Condition
We add a reminder here for the definition of the non-trivial critical point  $\theta = (w, a)$ : it is a critical point of the loss function that satisfies  $a \neq 0$  and  $\| w \| \neq 0$ .
Proposition G.1. Assume that  $f^{*}$  is an orthogonal teacher network of width  $k$ . If the activation function satisfies Assumption 3.1 (i), any non-trivial critical point  $\theta^{*} = (w^{*}, a^{*})$ , i.e.  $\nabla L^{1,k}(\theta^{*}) = 0$ , satisfies that  $w^{*}$  is in the span of the teacher's incoming vectors.
Proof. We will prove by contradiction. Let us assume that  $w$  is outside of the span of the teacher's incoming vectors. We will show that  $(w, a)$  is not a critical point for any  $a \neq 0$ . Mapping  $(w, a)$  to the order parameter space, we get that  $(r, u, a)$  where  $u = (u_1, \dots, u_k) \in \operatorname{int} B$  which implies  $u_j \in (-1, 1)$ . Since  $u \in \operatorname{int} B$  and  $r > 0$ , we have that  $(r, u, a)$  is a critical point of  $L_{\mathrm{proj}}^{1,k}$  since the boundaries are not seen near the neighborhood of this point. Therefore the partial derivatives of  $L_{\mathrm{proj}}^{1,k}$  are all zero including
$$
b _ {j} \frac {d}{d u _ {j}} g _ {\sigma} (r, \| v _ {j} \|, u _ {j}) = 0. \tag {104}
$$
From Assumption 3.1 (i), we have that  $\partial_{u}g_{\sigma}(r_1,r_2,u) > 0$  for  $u\in (-1,1)$  which yields a contradiction. Thus, each critical point of the projected loss is on the boundary, i.e.  $u\in \partial B$ , which implies that the incoming vector is in the span of the teacher's incoming vectors.
We will next show that any non-trivial critical point satisfies a Lagrangian condition since it is on the boundary of a constrained optimization problem.
Lemma G.2. Let  $\theta = (w, a)$  be a non-trivial critical point of  $L^{1,k}$ . Then the corresponding order parameters  $p = (r, u, a)$  satisfy the following Lagrangian condition
$$
b _ {j} \partial_ {u} g _ {\sigma} (r, \| v _ {j} \|, u _ {j}) = \lambda u _ {j} \quad f o r a l l j \in [ k ]. \tag {105}
$$
Proof. We will first show that for any differentiable path  $(r, \gamma(t), a)$  on the boundary such that  $\gamma(t) \in \partial B$  for  $t \in (-\epsilon, \epsilon)$  for some  $\epsilon > 0$  and  $\gamma(0) = u$ , the following holds
$$
\left. \frac {d}{d t} L _ {\text {p r o j}} ^ {1, k} (\gamma (t)) \right| _ {t = 0} = \nabla_ {u} L _ {\text {p r o j}} ^ {1, k} (p) \cdot \gamma^ {\prime} (0) = 0. \tag {106}
$$
Let us assume the contrary. We construct the corresponding following path in the weight space
$$
\theta (t) = \left(r \left(\sum_ {j = 1} ^ {k} u _ {j} (t) v _ {j} + v _ {\perp}\right), a\right). \tag {107}
$$
Thanks to the equivalence of the losses along the path, we have that
$$
\left. \frac {d}{d t} L ^ {1, k} (\theta (t)) \right| _ {t = 0} = \left. \frac {d}{d t} L _ {\text {p r o j}} ^ {1, k} (\gamma (t)) \right| _ {t = 0} = 0, \tag {108}
$$
since  $\theta(0) = \theta$  is a critical point in the weight space. Therefore, Eq. 106 holds for any differentiable path on the boundary and implies that  $\nabla_u L(p)$  is orthogonal to all  $\gamma'(0)$ . The vector that is orthogonal to all  $\gamma'(0)$  is the gradient of the surface, that is  $2(u_1, \ldots, u_k)$ . Hence we get  $\nabla_u L(p) \parallel u$  which is written explicitly as the Lagrangian condition in Eq. 105. This is equivalent to setting the partial derivatives of the Lagrangian loss with respect to  $u_j$  to zero where the Lagrangian loss is given by
$$
\mathcal {L} (p, \lambda) = - 2 a \sum_ {j = 1} ^ {k} b _ {j} g _ {\sigma} \left(r, \| v _ {j} \|, u _ {j}\right) + \lambda^ {\prime} \left(\sum_ {j = 1} ^ {k} u _ {j} ^ {2} - 1\right). \tag {109}
$$
We set  $\lambda = \lambda' / a$  in Eq. 105.
G.2 General Activation Functions
Before we present our results, let us take a detour to check the applicability of the convex optimization framework. For a convex and twice-differentiable activation function such as softplus, applying Lemma F.1 twice implies that the interaction  $g_{\sigma}(r_1,r_2,\cdot)$  is a convex function of the correlation  $u\in (-1,1)$  for  $r_1,r_2 > 0$ . Let us consider a fixed  $a < 0$  and  $r > 0$  and consider the loss parameterized by  $u_{j}$ 's. It is convex since its Hessian is a diagonal matrix with entries
$$
\frac {d ^ {2}}{d u _ {j} ^ {2}} L = - 2 a \frac {d ^ {2}}{d u _ {j} ^ {2}} g _ {\sigma} (r, \| v _ {j} \|, u _ {j}) > 0. \tag {110}
$$
Since the constraint on the correlations (Eq. 14) is also convex, we get a convex optimization problem that has a unique global minimum (see Boyd et al. [44], Section 4.2). Swapping a pair of  $u_{j}$  does not change the loss, thus it is permutation symmetric. If any two  $u_{j}$  were distinct from each other at the minimum, then its permutation would also be a minimum which would violate the unicity. We conclude that at the unique minimum point, the correlations are equal to each other. However, for the case  $a > 0$ , and for other activation functions, the objective is not convex.
We instead use Lagrange multipliers for proving Theorem 5.1.
G.2.1 Proof of Theorem 5.1
Theorem G.3. Assume that the activation function satisfies Assumption 3.1. At any non-trivial critical point  $(w^{*},a^{*})$  of the loss  $L^{1,k}$  for the unit-orthonormal teacher network, the incoming vector satisfies
$$
\frac {w ^ {*}}{\| w ^ {*} \|} = u \sum_ {j = 1} ^ {k} v _ {j} \tag {111}
$$
where  $u$  is either  $\frac{1}{\sqrt{k}}$  or  $-\frac{1}{\sqrt{k}}$ .
Proof. From Proposition 4.1 and Lemma G.2, we get that any non-trivial critical point should satisfy the Lagrangian condition in Eq. 105. In particular for unit-orthonormal teacher, setting  $\| v_{j}\| = 1$  and  $b_{j} = 1$ , we get the following Lagrangian condition
$$
\partial_ {u} g _ {\sigma} (r, 1, u _ {j}) = \lambda u _ {j} \forall j \in [ k ], \quad \sum_ {j = 1} ^ {k} u _ {j} ^ {2} = 1. \tag {112}
$$
If  $u_{j} = 0$ , we get  $\partial_{u}g_{\sigma}(r,1,0) = 0$  which is not possible since  $g_{\sigma}(r,1,u)$  is increasing due to Assumption 3.1 (i). Hence we have
$$
\frac {\partial_ {u} g _ {\sigma} (r , 1 , u _ {j})}{u _ {j}} = \lambda . \tag {113}
$$
Let us observe that  $\partial_{u}g_{\sigma}(r,1,u) / u$  is decreasing for  $u\in (-1,1)\setminus \{0\}$  if and only if
$$
\frac {d}{d u} \left(\frac {1}{u} \frac {d}{d u} g _ {\sigma} (r, 1, u)\right) = \frac {1}{u} \frac {d ^ {2}}{d u ^ {2}} g _ {\sigma} (r, 1, u) - \frac {1}{u ^ {2}} \frac {d}{d u} g _ {\sigma} (r, 1, u) <   0, \tag {114}
$$
which is equivalent to Assumption 3.1 (ii) for  $u \in (-1,1) \setminus \{0\}$  (we included  $u = 0$  in Assumption 3.1 (ii) for a simpler statement which is already implied from Assumption 3.1 (i) at  $u = 0$ ).
Taken together, we conclude that  $\partial_{u}g_{\sigma}(r,1,u) / u$  is one-to-one in  $u\in (-1,1)\setminus \{0\}$ . We need to consider the remaining case  $u_{i}\in \{-1,1\}$ . For  $k\geq 2$ , necessarily, we have  $u_{j} = 0$  for  $j\neq i$ , which is not possible as we have shown. For  $k = 1$ ,  $u_{i}\in \{-1,1\}$  is the only option that satisfies the boundary condition. For  $k\geq 2$ , Eq. 113 implies that all correlations are equal. Combining it with the boundary condition, we get  $u_{1} = \ldots = u_{k} = u$  with  $ku^{2} = 1$ , which completes the proof.
G.2.2 Two-Dimensional Loss, The Derivative Constraints, Uniqueness
At any non-trivial critical point, we proved in Theorem 5.1 that all correlations are equal and denoted by  $u$  that is either  $1 / \sqrt{k}$  or  $-1 / \sqrt{k}$ . The projected loss at a critical point reduces to
$$
L = a ^ {2} g _ {\sigma} (r, r, 1) - 2 k a g _ {\sigma} (r, 1, u) + C. \tag {115}
$$
Moreover, at a critical point, the partial derivatives with respect to the outgoing weight and norm should also be zero which gives the following two constraints
$$
\partial_ {a} L = 2 a g _ {\sigma} (r, r, 1) - 2 k g _ {\sigma} (r, 1, u) = 0,
$$
$$
\partial_ {r} L = a ^ {2} \partial_ {r} g _ {\sigma} (r, r, 1) - 2 k a \partial_ {r} g _ {\sigma} (r, 1, u) = 0, \tag {116}
$$
Figure 10: The graph of  $f(r, u) = \frac{d}{dr} \left( \frac{1}{2} \log g_{\sigma}(r, r, 1) - \log g_{\sigma}(r, 1, u) \right)$  for activation functions erf, softplus with  $\beta = 1$ , sigmoid, tanh, and gelu, respectively. Zero crossings of  $f$  are shown in red. For softplus and sigmoid, we observe that  $f$  is negative for  $r = 0, u \in (0, 1)$ , positive for  $r = 1, u \in (0, 1)$ , and increasing in  $r \in [0, 1]$  for any fixed  $u$ , thus satisfying the sufficient conditions in Eq. 121. However, for tanh and erf,  $f$  shows non-monotonic behavior in  $r$  when  $u$  is close to 1. For the GeLU activation function  $\sigma(x) = x \Phi(x)$ , which is non-monotonic, we observe that  $f$  does not cross zero for any  $(r, u)$  pair in the plotted domain. It approaches zero from below when  $r \to \infty$  thus showing a very different behavior from the other activation functions.
which can be rearranged into the following (assuming  $g_{\sigma}(r,r,1)\neq 0$  and  $\partial_r g_\sigma (r,r,1)\neq 0$ )
$$
\frac {a}{k} = \frac {g _ {\sigma} (r , 1 , u)}{g _ {\sigma} (r , r , 1)} = \frac {2 \partial_ {r} g _ {\sigma} (r , 1 , u)}{\partial_ {r} g _ {\sigma} (r , r , 1)}. \tag {117}
$$
The second equality between the two ratios of Gaussian integral terms gives a fixed point equation on the norm  $r$ . Writing the interactions in Eq. 117 explicitly and rearranging the ratios, we get
$$
f (r, u) = \frac {\mathbb {E} \left[ \sigma^ {\prime} (r x) \sigma (r x) x \right]}{\mathbb {E} \left[ \sigma (r x) ^ {2} \right]} - \frac {\mathbb {E} \left[ \sigma^ {\prime} (r x) \sigma (y) x \right]}{\mathbb {E} \left[ \sigma (r x) \sigma (y) \right]} = 0, \tag {118}
$$
where  $x$  and  $y$  are standard Gaussians with correlation  $\mathbb{E}[xy] = u$ . Let us define the following helper functions
$$
G (r) = \frac {\mathbb {E} [ \sigma^ {\prime} (r x) \sigma (r x) x ]}{\mathbb {E} [ \sigma (r x) ^ {2} ]} = \frac {1}{2} \frac {d}{d r} \log (\mathbb {E} [ \sigma (r x) ^ {2} ]),
$$
$$
\tilde {G} (u, r) = \frac {\mathbb {E} [ \sigma^ {\prime} (r x) \sigma (y) x ]}{\mathbb {E} [ \sigma (r x) \sigma (y) ]} = \frac {d}{d r} \log (\mathbb {E} [ \sigma (r x) \sigma (y) ]), \tag {119}
$$
$$
f (r, u) = G (r) - \tilde {G} (u, r) = \frac {d}{d r} \log \left(\frac {\mathbb {E} [ \sigma (r x) ^ {2} ] ^ {\frac {1}{2}}}{\mathbb {E} [ \sigma (r x) \sigma (y) ]}\right) = 0. \tag {120}
$$
Let us consider the case  $u > 0$ . We want to show that for any given  $u \in (0,1]$  there is a unique  $r \in (0,1]$  such that  $f(r,u) = 0$ . Under the assumption  $\sigma(0) \neq 0$ , if the following three conditions are satisfied for all  $u \in (0,1]$ ,
(i)  $\frac{\sigma'(0)}{\sigma(0)}\frac{\mathbb{E}[\sigma(y)x]}{\mathbb{E}[\sigma(y)]} > 0,$
(ii)  $\frac{\mathbb{E}[\sigma'(x)\sigma(x)x]}{\mathbb{E}[\sigma(x)^2]} >\frac{\mathbb{E}[\sigma'(x)\sigma(y)x]}{\mathbb{E}[\sigma(x)\sigma(y)]},$
(iii)  $\frac{d^2}{dr^2}\log \left(\frac{\mathbb{E}[\sigma(rx)^2]^{\frac{1}{2}}}{\mathbb{E}[\sigma(rx)\sigma(y)]}\right) > 0,$  (121)
then we have a unique  $r$  solving Eq. 120 as we explain next. Note that the first two conditions are equivalent to  $f(0, u) < 0$  and  $f(1, u) > 0$ , respectively. The tricky part is the third condition which is equivalent to showing that
$$
\frac {\mathbb {E} [ \sigma (r x) \sigma (y) ]}{\mathbb {E} [ \sigma (r x) ^ {2} ] ^ {\frac {1}{2}}} \tag {122}
$$
is log-concave in  $r$ . We note that marginalization properties of log-concave functions may be helpful here. In this paper, we were not able to prove the sufficient conditions listed above for general activation functions that do not admit an analytic formula of the interaction, even for softplus which we studied in detail (see Subsection G.5). Instead, we present the numerical integration results, which show that for any given  $u \in (0,1]$ , there is a unique  $r \in (0,1]$  such that  $f = 0$  (see Fig. 10). Once  $r$  is shown to be unique, then the matching outgoing weight  $a$  follows from Eq. 117.
G.3 Closed-Form Solution for Erf Activation
Corollary G.4. Assume that the activation function is  $\sigma_{\text{erf}}$ . The optimal solution  $(w^{*}, a^{*})$  is given by
or, equivalently, by  $(-w^{*}, -a^{*})$ . The optimal loss is given by
$$
L _ {e r f} ^ {*} (k) = \frac {2}{\pi} \left(k \arcsin \left(\frac {1}{2}\right) - k ^ {2} \arcsin \left(\frac {1}{2 k}\right)\right). \tag {123}
$$
Proof. Since erf is an odd activation function, it suffices to find parameters of the non-trivial critical points satisfying  $u \geq 0$ . For any such critical point  $(w^{*}, a^{*})$ , its mirror symmetry  $(-w^{*}, -a^{*})$  is an equivalent critical point due to Eq. 41.
For  $u = 0$ , we have that  $g_{\mathrm{erf}}(r,1,u) = 0$  which implies  $g_{\mathrm{erf}}(r,r,1) = \mathbb{E}[\sigma (rx)^2 ] = 0$  due to the first derivative constraint in Eq. 116 which holds if and only if  $r = 0$ . This gives a possible trivial critical point yielding the zero predictor function.
For a given  $u = \frac{1}{\sqrt{k}} > 0$ , from Fig. 10, we observe that there is a unique  $r \in (0,1]$  satisfying the fixed point equation in Eq. 120. For uniqueness, we rely on numerical integration. We will find one solution to the derivative constraints given below
$$
a g _ {\text {e r f}} (r, r, 1) = k g _ {\text {e r f}} (r, 1, u), \quad a \partial_ {r} g _ {\text {e r f}} (r, r, 1) - 2 k \partial_ {r} g _ {\text {e r f}} (r, 1, u) = 0, \tag {124}
$$
for a given  $k$ , and equivalently  $u = \frac{1}{\sqrt{k}} > 0$ ; and due to uniqueness, conclude that it is the only non-trivial critical point up to symmetries.
In particular, we will use the analytic formula for the interaction function [2, 3]
$$
g _ {\text {e r f}} \left(r _ {1}, r _ {2}, u\right) = \frac {2}{\pi} \arcsin \left(\frac {r _ {1} r _ {2} u}{\sqrt {r _ {1} ^ {2} + 1} \sqrt {r _ {2} ^ {2} + 1}}\right). \tag {125}
$$
Let us find  $r$  where we have
$$
g _ {\mathrm {e r f}} (r, 1, u) = g _ {\mathrm {e r f}} (r, r, 1)
$$
that is satisfied if we have that the arguments of  $\arcsin$  match, which happens at
$$
\frac {r u}{\sqrt {r ^ {2} + 1} \sqrt {2}} = \frac {r ^ {2}}{r ^ {2} + 1} \quad \Rightarrow \quad r = \sqrt {\frac {u ^ {2}}{2 - u ^ {2}}} = \frac {1}{\sqrt {2 k - 1}}. \tag {126}
$$
Interestingly, at this value of  $r$ , we also have
$$
2 \partial_ {r} g _ {\text {e r f}} (r, 1, u) = \partial_ {r} g _ {\text {e r f}} (r, r, 1)
$$
which can be seen by inserting the guessed values in the following equation
$$
2 \partial_ {r} \Big (\frac {r u}{\sqrt {r ^ {2} + 1} \sqrt {2}} \Big) \arcsin^ {\prime} \Big (\frac {r u}{\sqrt {r ^ {2} + 1} \sqrt {2}} \Big) = \partial_ {r} \Big (\frac {r ^ {2}}{r ^ {2} + 1} \Big) \arcsin^ {\prime} \Big (\frac {r ^ {2}}{r ^ {2} + 1} \Big).
$$
Setting  $a = k$  in Eq. 124 completes the order parameters of the non-trivial critical point. Finally, let us compute the loss at  $r = 1 / \sqrt{2k - 1}$ ,  $u = 1 / \sqrt{k}$ ,  $a = k$ ;
$$
\begin{array}{l} L _ {\text {e r f}} ^ {*} (k) := a ^ {2} g _ {\text {e r f}} (r, r, 1) - 2 a k g _ {\text {e r f}} (r, 1, u) + k g _ {\text {e r f}} (1, 1, 1), \\ = - k ^ {2} g _ {\text {e r f}} (r, r, 1) + k g _ {\text {e r f}} (1, 1, 1), \\ = \frac {2}{\pi} \left(k \arcsin \left(\frac {1}{2}\right) - k ^ {2} \arcsin \left(\frac {1}{2 k}\right)\right). \tag {127} \\ \end{array}
$$
G.4 Closed-Form Solution for ReLU Activation
Corollary G.5. Assume that the activation function is  $\sigma_{\text{relu}}$ . Any optimal solution  $(w^{*}, a^{*})$  satisfies
$$
\| w ^ {*} \| a ^ {*} = \frac {k}{h (1)} h \left(\frac {1}{\sqrt {k}}\right), \quad \frac {w ^ {*}}{\| w ^ {*} \|} = \frac {1}{\sqrt {k}} \sum_ {i = 1} ^ {k} v _ {i}, \tag {128}
$$
forming an equal-loss hyperbola. The optimal loss is given by
$$
L _ {r e l u} ^ {*} (k) = k ^ {2} \left(h (0) - \frac {1}{h (1)} h \left(\frac {1}{\sqrt {k}}\right) ^ {2}\right) + k (h (1) - h (0)). \tag {129}
$$
We will first show that the interaction of ReLU satisfies
(i)  $h^\prime (u) > 0$  for  $u\in (-1,1)$
(ii)  $h''(u)u < h'(u)$  for  $u \in (-1, u_0]$ ,
(iii)  $\frac{h'(u_0)}{u_0} > \frac{h'(u)}{u}$  for  $u \in (u_0, 1)$ ; (130)
where  $u_0 = 1 / \sqrt{2}$ . Note that property (i) is equivalent to Assumption 3.1 (i), and property (ii) is almost equivalent to Assumption 3.1 (ii) except that it holds in the interval  $(-1, u_0]$ ; property (iii) covers up for the missing piece of the interval in the property (ii).
ReLU interaction satisfies Properties 130; Proof. Let us write the first two derivatives of  $h$ :
$$
h ^ {\prime} (u) = \frac {\pi - \operatorname {a r c c o s} (u)}{2 \pi}, \quad h ^ {\prime \prime} (u) = \frac {1}{2 \pi \sqrt {1 - u ^ {2}}}. \tag {131}
$$
Property (i) easily comes from noting that the derivative of  $h$  is positive for  $u \in (-1, 1)$ . Property (ii) holds for  $u \in (-1, 0]$  since both the first and second derivatives are positive. Let us show that Property (ii) holds for  $u \in (0, u_0]$ , that is equivalent to
$$
\frac {u}{\sqrt {1 - u ^ {2}}} <   \pi - \arccos  (u) = \frac {\pi}{2} + \arcsin (u). \tag {132}
$$
Let us note that the left-hand side is smaller than 1 since
$$
\frac {u ^ {2}}{1 - u ^ {2}} \leq 1.
$$
Note that  $\arcsin (u) > 0$  for  $u > 0$ ; and  $\pi /2 > 1$ . This completes the proof of Property (ii).
For Property (iii), we first show that  $h'(u) / u$  is convex in  $u \in (0,1)$ . The first two derivatives are
$$
\frac {d}{d u} \left(\frac {h ^ {\prime} (u)}{u}\right) = \frac {h ^ {\prime \prime} (u)}{u} - \frac {h ^ {\prime} (u)}{u ^ {2}}, \quad \frac {d ^ {2}}{d u ^ {2}} \left(\frac {h ^ {\prime} (u)}{u}\right) = \frac {h ^ {\prime \prime \prime} (u)}{u} - \frac {2 h ^ {\prime \prime} (u)}{u ^ {2}} + \frac {2 h ^ {\prime} (u)}{u ^ {3}}.
$$
Thus, it is equivalent to showing
$$
h ^ {\prime \prime \prime} (u) u - 2 h ^ {\prime \prime} (u) + \frac {2 h ^ {\prime} (u)}{u} = \frac {u ^ {2}}{(1 - u ^ {2}) ^ {3 / 2}} - \frac {2}{(1 - u ^ {2}) ^ {1 / 2}} + \frac {\pi + 2 \arcsin (u)}{u} > 0.
$$
Using the Taylor series of  $\arcsin$  and  $u > 0$ , we have that  $\arcsin (u) > u$ . Hence, it suffices to show
$$
\frac {1}{(1 - u ^ {2}) ^ {1 / 2}} \left(- 3 + \frac {1}{1 - u ^ {2}} + 2 (1 - u ^ {2}) ^ {1 / 2}\right) \geq 0; \tag {133}
$$
where we dropped the positive term  $\frac{\pi}{u}$  which holds due to the inequality of arithmetic and geometric means
$$
\frac {1}{1 - u ^ {2}} + (1 - u ^ {2}) ^ {1 / 2} + (1 - u ^ {2}) ^ {1 / 2} \geq 3.
$$
Let us assume the contrary of Property (iii), that there exists  $u \in (u_0, 1)$  such that
$$
\frac {h ^ {\prime} \left(u _ {0}\right)}{u _ {0}} \leq \frac {h ^ {\prime} (u)}{u}. \tag {134}
$$
Note that  $h'(u_0) / u_0 > h'(1)$  because  $\pi(1 - u_0) - \arccos(u_0) > 0$  holds at  $u_0 = 1 / \sqrt{2}$ . Since  $h'(u) / u$  is left-continuous at  $u = 1$ , there exists  $\epsilon > 0$  such that
$$
\frac {h ^ {\prime} \left(u _ {0}\right)}{u _ {0}} > \frac {h ^ {\prime} (1 - \epsilon)}{1 - \epsilon}. \tag {135}
$$
Finally, there exists  $\alpha \in (0,1)$  such that  $u = \alpha (1 - \epsilon) + (1 - \alpha)u_{0}$  which gives due to the convexity of  $h^\prime (u) / u$  the following
$$
\alpha \frac {h ^ {\prime} (1 - \epsilon)}{1 - \epsilon} + (1 - \alpha) \frac {h ^ {\prime} \left(u _ {0}\right)}{u _ {0}} \geq \frac {h ^ {\prime} (u)}{u}. \tag {136}
$$
This yields a contradiction since the left-hand side is strictly smaller than  $h'(u_0) / u_0$  hence the proof of Property (iii) is complete. ReLU interaction satisfies Properties 130; End of Proof.
Proof. First, we replicate the proof steps of Theorem 5.1 to show that any non-trivial critical point must be on the boundary and attain equal correlations. From Property 130 (i), we get that there is no non-trivial critical point in int  $B$ . For  $k = 1$ , this implies that  $u_{1} = -1$  or  $u_{1} = 1$ .
For general  $k$ , let us recall that we get the Lagrangian condition for non-trivial critical points
$$
r h ^ {\prime} \left(u _ {j}\right) = \lambda u _ {j} \forall j \in [ k ], \quad \sum_ {j = 1} ^ {k} u _ {j} ^ {2} = 1. \tag {137}
$$
which is equivalent to Eq. 112 for ReLU activation function.  $u_{j} = 0$  is not possible since we have  $h^\prime (0)\neq 0$ . Hence, we get
$$
\frac {h ^ {\prime} \left(u _ {j}\right)}{u _ {j}} = \frac {\lambda}{r}, \quad \forall j \in [ k ]. \tag {138}
$$
Property 130 (ii) implies that  $f(u) = h'(u) / u$  is decreasing for  $u \in (-1, u_0) \setminus \{0\}$ . Moreover,  $f$  is negative for  $u < 0$  and positive for  $u > 0$ .