SlowGuess/NeurIPS_2023_OCR huggingface dataset:
1. If  $\lambda / r < 0$ , we get that all  $u_j$  are equal and negative, hence they are equal to  $-1/\sqrt{k}$  due to the boundary condition.
2. If  $\lambda / r = 0$ , we get  $u_j = -1$  for all  $j$  which implies that  $k = 1$  which is already covered above.
3. If  $\lambda / r > 0$ , Property 130 (iii) gives that  $f(u_0) > f(u)$  for  $u \in (u_0, 1)$ . Since  $f$  is decreasing we have also  $f(u) > f(u_0)$  for  $u \in (0, u_0)$ ; hence  $f(u_j)$  are equal only when all  $u_j < u_0$  or  $u_j > u_0$ ; however, the latter case is not possible for  $k \geq 2$  since it breaks the ball constraint, i.e.  $u_1^2 + u_2^2 > 1$ .
Hence, we get that  $u_{j} \in (0, u_{0}]$  and are equal since  $f$  is decreasing in this interval. This completes the proof of replica of Theorem 5.1 for the ReLU activation function.
For the ReLU activation function, there is at least one non-differentiable critical point at  $a = 0$  or  $r = 0$ . The careful analysis of this critical point is beyond the scope of this work. For any such 'trivial' point, the error of zero-function is equivalent to
$$
\mathbb {E} \left[ \left(\sum_ {j = 1} ^ {k} \sigma \left(v _ {j} \cdot x\right)\right) ^ {2} \right] = k h (1) + k (k - 1) h (0). \tag {139}
$$
We will next show that  $(-1 / \sqrt{k})_{j = 1}^{k}$  and  $(1 / \sqrt{k})_{j = 1}^{k}$  are the global minimum and the global maximum of the following loss function
$$
\sum_ {j = 1} ^ {k} h \left(u _ {j}\right), \quad \text {s u b j e c t} \quad \sum_ {j = 1} ^ {k} u _ {j} ^ {2} \leq 1. \tag {140}
$$
Due to the Lagrange condition, there is no other critical point, hence these are the only two critical points of the constrained objective in Eq. 140. The objective then reduces to  $kh(u)$  which is minimized at  $u = -1 / \sqrt{k}$  and maximized at  $u = 1 / \sqrt{k}$ .
Next, we will give the closed-form solution of the remaining order parameters. Plugging in the correlation in the loss and using the factorization of the interaction in Eq. 14, we get
$$
L = a ^ {2} r ^ {2} \cdot h (1) - 2 k a r \cdot h (u) + C.
$$
Let us set  $\tilde{a} = ar$ . The loss is a second-order polynomial in  $\tilde{a}$
$$
L = h (1) \left(\tilde {a} ^ {2} - 2 \tilde {a} k \frac {h (u)}{h (1)} + k + k (k - 1) \frac {h (0)}{h (1)}\right)
$$
where we made the constant explicit. Since the coefficient of the leading term is positive, there is a minimizer and it is the only critical point. Taking the derivative, the minimum is attained at
$$
\tilde {a} _ {*} = k \frac {h (u)}{h (1)} \tag {141}
$$
Finally, plugging in  $\tilde{a}_{*}$ , we get
$$
L (u) = - k ^ {2} \frac {h (u) ^ {2}}{h (1)} + k h (1) + k (k - 1) h (0). \tag {142}
$$
For  $u = 1 / \sqrt{k}$  and  $u = -1 / \sqrt{k}$ ,  $h(u)$  is non-zero; hence  $l(u)$  is smaller than the loss of the zero function (trivial critical points). The smallest loss is attained at  $u = 1 / \sqrt{k}$  which is, therefore, the optimal solution. We conclude that the critical point at  $u = -1 / \sqrt{k}$  is a saddle point since it is a maximum in  $u$  and a minimum in  $\tilde{a}$ .
G.5 Bounds on Incoming Vector Norm and Outgoing Weight for Softplus
Unlike ReLU and erf, the interaction function does not have a known analytic expression for softplus, hence the proof involves some techniques to compare ratios of Gaussian integral terms.
FKG Inequality. We will use a special case of the FKG inequality repeatedly, that is,
$$
\mathbb {E} [ f (x) g (x) ] > \mathbb {E} [ f (x) ] \mathbb {E} [ g (x) ] \tag {143}
$$
if both  $f, g$  are increasing (or decreasing) implying that  $f$  and  $g$  are positively correlated. The inequality changes direction if  $f$  is increasing and  $g$  is decreasing (or vice versa) implying that  $f$  and  $g$  are negatively correlated.
We will rely on some specific properties of the softplus family that are developed in Section G.5.4. Unfortunately, some of these properties do not apply to other activation functions. As a first example of managing interactions that do not have an analytic formula, the proof may inspire generalizations to other activation functions. Below we present the proof sketch for Theorem 5.3. In the following Subsections G.5.1, G.5.2, G.5.3, and G.5.4, the components of the proof are presented in detail.
Proof Sketch. We want to characterize the zero(s) of  $f$  introduced in Section G.2.2 that is
$$
f (r, u) = G (r) - \tilde {G} (u, r) = \frac {\mathbb {E} [ \sigma^ {\prime} (r x) \sigma (r x) x ]}{\mathbb {E} [ \sigma (r x) ^ {2} ]} - \frac {\mathbb {E} [ \sigma^ {\prime} (r x) \sigma (y) x ]}{\mathbb {E} [ \sigma (r x) \sigma (y) ]}. \tag {144}
$$
For  $r \in [0,1]$ , there is a unique correlation  $u \in [0,1]$  such that  $f(r,u) = 0$ . Denoting this correlation by  $h(r)$ , we have a map  $h:[0,1] \to [0,1]$  with boundary conditions  $h(0) = 0$  and  $h(1) = 1$ . For  $r > 1$ , there is no solution of  $f$ . As a consequence, no  $r \geq 0$  solves  $f(r,u) = 0$  for negative  $u$ , hence there is no non-trivial critical point at  $u = -1 / \sqrt{k}$ .
In Section G.5.2, we prove the inequality  $h(r) \geq r$ , which gives us the upper bound on the norm since we have that the correlation at a non-trivial critical point is  $h(r) = 1 / \sqrt{k}$ . Using this inequality and Stein's Lemma, we give a lower bound on the outgoing weight, that is  $a \geq k$ , in Section G.5.3. In summary, any non-trivial critical point of the loss has equal correlations that are  $u = 1 / \sqrt{k}$ , the norm satisfies  $r \leq u$ , and the lower bound on the outgoing weight follows. End of Proof Sketch.
G.5.1 Constraining the Zeros of  $f$
In this subsection, we will describe all zero-crossings of  $f:[0,\infty)\times [-1,1]\to \mathbb{R}$ . We need to check four cases (i)  $r = 0$ , (ii)  $r = 1$ , (iii)  $r > 1$ , and (iv)  $r\in (0,1)$ .
(i)  $r = 0$ : Note that  $G(0) = \tilde{G}(0,0) = 0$ . Since  $\tilde{G}$  is increasing in correlation for  $u \in [0,1]$  and  $\tilde{G}(u,0) < \tilde{G}(0,0)$  for  $u < 0$  (Lemma G.6), the only solution is  $u = 0$ .
(ii)  $r = 1$ : Note that  $G(1) = \tilde{G}(1,1)$  since  $y = x$  due to correlation one in Eq. 119. Since  $\tilde{G}$  is increasing in correlation for  $u \in [0,1]$  and  $\tilde{G}(u,1) < \tilde{G}(0,1)$  for  $u < 0$  (Lemma G.6), the only solution is  $u = 1$ .
(iii)  $r > 1$ : We will show that there is no zero in this case. Let us first show that  $G(r) > \tilde{G}(1, r)$  for  $r > 1$ , which is equivalent to
$$
\mathbb {E} \left[ \sigma^ {\prime} (r x) \sigma (r x) x \right] \mathbb {E} \left[ \sigma (r x) \sigma (x) \right] > \mathbb {E} \left[ \sigma^ {\prime} (r x) \sigma (x) x \right] \mathbb {E} \left[ \sigma (r x) ^ {2} \right]. \tag {145}
$$
Changing the measure of  $x$  from the standard Gaussian  $p(x)$  to  $\tilde{p}(x) = p(x)\sigma(rx)^2/\mathbb{E}[\sigma(rx)^2]$ , we get the following equivalent inequality
$$
\mathbb {E} _ {x \sim \tilde {p}} \left[ \frac {\sigma^ {\prime} (r x) x}{\sigma (r x)} \right] \mathbb {E} _ {x \sim \tilde {p}} \left[ \frac {\sigma (x)}{\sigma (r x)} \right] > \mathbb {E} _ {x \sim \tilde {p}} \left[ \frac {\sigma^ {\prime} (r x) x}{\sigma (r x)} \frac {\sigma (x)}{\sigma (r x)} \right]. \tag {146}
$$
From the property (iv) of Lemma G.8, we have that  $\sigma'(rx)x / \sigma(rx)$  is increasing after a substitution  $x \gets rx$ . We need to show  $\sigma(x) / \sigma(rx)$  is decreasing in  $x$  for  $r > 1$ . We take the derivative
$$
\frac {d}{d x} \frac {\sigma (x)}{\sigma (r x)} = \frac {\sigma^ {\prime} (x) \sigma (r x) - \sigma (x) \sigma^ {\prime} (r x) r}{\sigma (r x) ^ {2}}. \tag {147}
$$
Since  $\sigma'(x)x / \sigma(x)$  is increasing  $\forall x \in \mathbb{R}$ , we have
$$
\frac {\sigma^ {\prime} (x) x}{\sigma (x)} <   \frac {\sigma^ {\prime} (r x) r x}{\sigma (r x)} \text {f o r} x > 0, \text {a n d} \frac {\sigma^ {\prime} (x) x}{\sigma (x)} > \frac {\sigma^ {\prime} (r x) r x}{\sigma (r x)} \text {f o r} x <   0
$$
which yields  $\sigma'(x)\sigma(rx) < \sigma(x)\sigma'(rx)r$ , hence we conclude that  $\sigma(x)/\sigma(rx)$  is decreasing. Thanks to the FKG inequality,  $\sigma'(rx)x/\sigma(rx)$  and  $\sigma(x)/\sigma(rx)$  are negatively correlated which completes the argument. Since from Lemma G.6,  $\tilde{G}$  is increasing in correlation and  $\tilde{G}(0,r) > \tilde{G}(u,r)$  for  $u < 0$ , we have  $\tilde{G}(1,r) > \tilde{G}(u,r)$  for all  $u \in [-1,1)$ , therefore there is no solution of  $f$ .
$(iv) r \in (0,1)$ : We want to show that  $\forall r \in (0,1)$ , there is a unique  $u \in (0,1)$  such that  $f(r,u) = 0$ . It suffices to show
$$
\tilde {G} (0, r) <   G (r) <   \tilde {G} (1, r),
$$
since  $\tilde{G}$  is continuous and increasing in correlation for  $u\in [0,1]$  (Lemma G.6), it then crosses  $G(r)$  at a unique  $u\in (0,1)$ .
First inequality;  $\tilde{G}(0, r) < G(r)$ . In this case,  $x$  and  $y$  are Gaussians with zero correlation, hence independent. We can expand  $\tilde{G}(0, r)$  by factorizing the integrals
$$
\tilde {G} (0, r) = \frac {\mathbb {E} [ \sigma^ {\prime} (r x) x ] \mathbb {E} [ \sigma (y) ]}{\mathbb {E} [ \sigma (r x) ] \mathbb {E} [ \sigma (y) ]} = \frac {\mathbb {E} [ \sigma^ {\prime} (r x) x ]}{\mathbb {E} [ \sigma (r x) ]}.
$$
We want to show
$$
\mathbb {E} \left[ \sigma^ {\prime} (r x) x \right] \mathbb {E} \left[ \sigma (r x) ^ {2} \right] <   \mathbb {E} \left[ \sigma^ {\prime} (r x) \sigma (r x) x \right] \mathbb {E} [ \sigma (r x) ] \tag {148}
$$
which is equivalent to the following inequality after changing the measure from standard Gaussian  $p(x)$  to  $\tilde{p} (x) = p(x)\sigma (rx) / \mathbb{E}[\sigma (rx)]$
$$
\mathbb {E} _ {x \sim \tilde {p}} \left[ \frac {\sigma^ {\prime} (r x) x}{\sigma (r x)} \right] \mathbb {E} _ {x \sim \tilde {p}} [ \sigma (r x) ] <   \mathbb {E} _ {x \sim \tilde {p}} \left[ \frac {\sigma^ {\prime} (r x) x}{\sigma (r x)} \sigma (r x) \right]. \tag {149}
$$
This follows from the FKG inequality since we have that both  $\sigma'(x)x / \sigma(x)$  and  $\sigma(x)$  are increasing from the properties (iv) and (i) of softplus (Lemma G.8).
Second inequality;  $G(r) < \tilde{G}(1, r)$ . This is equivalent to the Ineq. 145, but the direction is reversed since in this case  $r < 1$ . We showed that  $\sigma(x) / \sigma(r'x)$  is decreasing in  $x$  for all  $r' > 1$ , therefore its reciprocal  $\sigma(r'x) / \sigma(x)$  is increasing in  $x$ . Substituting  $x \gets rx$  where  $r = 1 / r' < 1$ , we get that  $\sigma(x) / \sigma(rx)$  is increasing in  $x$  for  $r < 1$ . This yields a positive correlation between  $\sigma'(rx)x / \sigma(rx)$  and  $\sigma(x) / \sigma(rx)$  from the FKG inequality and completes the argument.
Overall, we showed that there are no zeros of  $f$  for  $r > 1$ . For  $r \in [0,1]$ , there is a unique correlation  $u$ , that we will denote by  $h(r)$ , such that  $f(r,h(r)) = 0$ . Furthermore,  $h:[0,1] \to [0,1]$  satisfies the following
i.  $h(0) = 0$  and  $h(1) = 1$
ii. for  $r\in (0,1)$  , we have  $h(r)\in (0,1)$
G.5.2 Bound on the Norm
In this subsection, we will show that  $h(r) \geq r$  for all  $r \in (0,1)$ . Let us assume the contrary, which implies
$$
\tilde {G} (h (r), r) <   \tilde {G} (r, r)
$$
due to Lemma G.6. It suffices to show that for all  $r \in (0,1)$ , we have
$$
\tilde {G} (r, r) \leq G (r), \tag {150}
$$
which yields a contradiction since  $G(r) = \tilde{G}(h(r), r)$ . Showing this is equivalent to
$$
\mathbb {E} \left[ \sigma^ {\prime} (r x) \sigma \left(r x + r ^ {\prime} z\right) x \right] \mathbb {E} \left[ \sigma (r x) ^ {2} \right] \leq \mathbb {E} \left[ \sigma^ {\prime} (r x) \sigma (r x) x \right] \mathbb {E} \left[ \sigma (r x) \sigma \left(r x + r ^ {\prime} z\right) \right] \tag {151}
$$
where  $r' = \sqrt{1 - r^2}$ . After a change of measure from standard Gaussian  $p(x)$  to
$$
\tilde {p} (x) = p (x) \frac {\mathbb {E} [ \sigma (r x + r ^ {\prime} z) | x ] \sigma (r x)}{\mathbb {E} [ \sigma (r x + r ^ {\prime} z) \sigma (r x) ]},
$$
this is equivalent to the following inequality
$$
\mathbb {E} _ {x \sim \tilde {p}} \left[ \frac {\sigma^ {\prime} (r x) x}{\sigma (r x)} \right] \mathbb {E} _ {x \sim \tilde {p}} \left[ \frac {\sigma (r x)}{\mathbb {E} [ \sigma (r x + r ^ {\prime} z) | x ]} \right] \leq \mathbb {E} _ {x \sim \tilde {p}} \left[ \frac {\sigma^ {\prime} (r x) x}{\sigma (r x)} \frac {\sigma (r x)}{\mathbb {E} [ \sigma (r x + r ^ {\prime} z) | x ]} \right]. \tag {152}
$$
What remains to show is that
$$
\frac {\mathbb {E} [ \sigma (r x + r ^ {\prime} z) | x ]}{\sigma (r x)}
$$
is non-increasing in  $x$  since then we can conclude by the FKG inequality. Since  $r > 0$  we can drop it up to a change in the standard deviation of  $x$ . We want to show that its derivative is non-positive:
$$
\sigma (x) \mathbb {E} [ \sigma^ {\prime} (x + r ^ {\prime} z) | x ] \leq \sigma^ {\prime} (x) \mathbb {E} [ \sigma (x + r ^ {\prime} z) | x ] \Leftrightarrow \frac {\sigma (x)}{\sigma^ {\prime} (x)} \leq \frac {\mathbb {E} [ \sigma (x + r ^ {\prime} z) | x ]}{\mathbb {E} [ \sigma^ {\prime} (x + r ^ {\prime} z) | x ]}. \tag {153}
$$
From the property (iii) of softplus (Lemma G.8), we have that  $R(x) = \sigma(x) / \sigma'(x)$  is convex. Applying Jensen, we get
$$
\frac {\sigma (x)}{\sigma^ {\prime} (x)} \leq \mathbb {E} \left[ \frac {\sigma (x + r ^ {\prime} z)}{\sigma^ {\prime} (x + r ^ {\prime} z)} \Big | x \right].
$$
$$
\mathbb {E} \left[ \frac {\sigma \left(x + r ^ {\prime} z\right)}{\sigma^ {\prime} \left(x + r ^ {\prime} z\right)} | x \right] \mathbb {E} \left[ \sigma^ {\prime} \left(x + r ^ {\prime} z\right) | x \right] \leq \mathbb {E} \left[ \sigma \left(x + r ^ {\prime} z\right) | x \right]. \tag {154}
$$
Note that  $\mathbb{E}[\sigma'(x + r'z)|x]$  is increasing in  $x$  since  $\sigma'$  is increasing. Moreover, the function
$$
\mathbb {E} \left[ \frac {\sigma (x + r ^ {\prime} z)}{\sigma^ {\prime} (x + r ^ {\prime} z)} \Big | x \right]
$$
is increasing in  $x$  since its integrand  $R$  is increasing from the property (ii) of softplus (Lemma G.8). Then we conclude by the FKG inequality that Eq. 154 holds. Therefore, for a solution  $(r,u)$  of the fixed point Eq. 117, we have  $r\leq u = \frac{1}{k}$ .
G.5.3 Bounding the Outgoing Weight
To get a bound on  $a$ , let us analyze the ratio of interactions in Eq. 117
$$
\frac {g _ {\sigma} (r , 1 , u)}{g _ {\sigma} (r , r , 1)} = \frac {a}{k}. \tag {155}
$$
Using the convexity of softplus (property (i) of Lemma G.8), we get
$$
\begin{array}{l} \frac {\mathbb {E} [ \sigma (r x) \sigma (u x + u ^ {\prime} z) ]}{\mathbb {E} [ \sigma (r x) ^ {2} ]} \geq \frac {\mathbb {E} [ \sigma (r x) \sigma (r x) ] + \mathbb {E} [ \sigma (r x) ((u - r) x + u ^ {\prime} z) \sigma^ {\prime} (r x) ]}{\mathbb {E} [ \sigma (r x) ^ {2} ]} \\ = 1 + (u - r) \frac {\mathbb {E} [ \sigma^ {\prime} (r x) \sigma (r x) x ]}{\mathbb {E} [ \sigma (r x) ^ {2} ]}. \tag {156} \\ \end{array}
$$
We can transform the numerator using Stein's lemma with  $v(x) = \sigma(rx)\sigma'(rx)$ .
$$
\mathbb {E} [ \sigma (r x) \sigma^ {\prime} (r x) x ] = r \mathbb {E} [ \sigma^ {\prime} (r x) ^ {2} + \sigma (r x) \sigma^ {\prime \prime} (r x) ] \tag {157}
$$
which is positive since softplus is positive, increasing, and convex. Combining it with  $u \geq r$ , we get that the ratio is bounded below by 1 which yields  $a \geq k$ .
G.5.4 Helper Lemmas
In this subsection, we provide helper lemmas used in the proof of Theorem 5.3. We present Lemma G.6 which shows that  $\tilde{G}$  is increasing in correlation and Lemma G.7 used in the proof of the former. Finally, we present several properties of the softplus family in Lemma G.8 that are used throughout the proof.
Lemma G.6. The following function is increasing in  $u \in [0,1]$
$$
\tilde {G} (u, r) = \frac {\mathbb {E} [ \sigma^ {\prime} (r x) \sigma (y) x ]}{\mathbb {E} [ \sigma (r x) \sigma (y) ]} \tag {158}
$$
for any  $r \geq 0$ , where  $x$  and  $y$  are standard Gaussians with correlation  $\mathbb{E}[xy] = u$ . Moreover,  $\tilde{G}(u,r) < \tilde{G}(0,r)$  for  $u < 0$ .
Proof. Let us assume  $0 \leq u_{1} < u_{2} \leq 1$ . For the first part of the statement, we want to show
$$
\frac {\mathbb {E} \left[ \sigma^ {\prime} (r x) \sigma \left(y _ {1}\right) x \right]}{\mathbb {E} \left[ \sigma (r x) \sigma \left(y _ {1}\right) \right]} <   \frac {\mathbb {E} \left[ \sigma^ {\prime} (r x) \sigma \left(y _ {2}\right) x \right]}{\mathbb {E} \left[ \sigma (r x) \sigma \left(y _ {2}\right) \right]} \tag {159}
$$
where  $\mathbb{E}[xy_1] = u_1$  and  $\mathbb{E}[xy_2] = u_2$ . Changing the measure from the standard Gaussian  $p(x)$  to
$$
\tilde {p} (x) = p (x) \frac {\sigma (r x) \mathbb {E} [ \sigma (y _ {2}) | x ]}{\mathbb {E} [ \sigma (r x) \sigma (y _ {2}) ]},
$$
we get the following equivalent inequality
$$
\mathbb {E} \left[ \frac {\sigma^ {\prime} (r x) x}{\sigma (r x)} \frac {\mathbb {E} [ \sigma \left(y _ {1}\right) | x ]}{\mathbb {E} [ \sigma \left(y _ {2}\right) | x ]} \right] <   \mathbb {E} \left[ \frac {\sigma^ {\prime} (r x) x}{\sigma (r x)} \right] \mathbb {E} \left[ \frac {\mathbb {E} [ \sigma \left(y _ {1}\right) | x ]}{\mathbb {E} [ \sigma \left(y _ {2}\right) | x ]} \right]. \tag {160}
$$
Thanks to the property (iv) of softplus (Lemma G.8), we have that  $\sigma'(rx)x / \sigma(rx)$  is increasing in  $x$  after a substitution  $x \gets rx$  for  $r > 0$ . For  $r = 0$ , the function reduces to  $\gamma x$  with some  $\gamma > 0$ , hence increasing. We will next show that (all integrations are w.r.t  $z$  hereafter, hence we drop the conditioning on  $x$ )
$$
\frac {\mathbb {E} [ \sigma (y _ {1}) ]}{\mathbb {E} [ \sigma (y _ {2}) ]} \tag {161}
$$
is decreasing in  $x$ . Computing the derivative w.r.t  $x$ , we want to show that it is negative
$$
\frac {\mathbb {E} \left[ \sigma^ {\prime} \left(y _ {1}\right) \right] u _ {1}}{\mathbb {E} \left[ \sigma \left(y _ {2}\right) \right]} - \frac {\mathbb {E} \left[ \sigma \left(y _ {1}\right) \right] \mathbb {E} \left[ \sigma^ {\prime} \left(y _ {2}\right) \right] u _ {2}}{\mathbb {E} \left[ \sigma \left(y _ {2}\right) \right] ^ {2}} <   0 \Leftrightarrow \frac {\mathbb {E} \left[ \sigma^ {\prime} \left(y _ {1}\right) \right] u _ {1}}{\mathbb {E} \left[ \sigma \left(y _ {1}\right) \right]} <   \frac {\mathbb {E} \left[ \sigma^ {\prime} \left(y _ {2}\right) \right] u _ {2}}{\mathbb {E} \left[ \sigma \left(y _ {2}\right) \right]}. \tag {162}
$$
Note that this is equivalent to showing
$$
\frac {d}{d u} \frac {\mathbb {E} [ \sigma^ {\prime} (y) ] u}{\mathbb {E} [ \sigma (y) ]} = \frac {d ^ {2}}{d u d x} \log (\mathbb {E} [ \sigma (y) ]) > 0
$$
for all  $u \in [0,1)$  and  $x \in \mathbb{R}$ . Changing the order of derivatives, it is sufficient to show
$$
\frac {d}{d x} \left(\frac {\mathbb {E} \left[ \sigma^ {\prime} (y) \right] x}{\mathbb {E} [ \sigma (y) ]} - \left(\frac {u}{1 - u ^ {2}}\right) \frac {\mathbb {E} \left[ u ^ {\prime} z \sigma^ {\prime} (y) \right]}{\mathbb {E} [ \sigma (y) ]}\right) > 0. \tag {163}
$$
The first function
$$
s _ {1} (x) = \frac {\mathbb {E} [ \sigma^ {\prime} (y) ] x}{\mathbb {E} [ \sigma (y) ]} \tag {164}
$$
is shown to be increasing in  $x$  in Lemma G.7 where we need to substitute  $x \to xu_{1}$  for  $u_{1} > 0$ , and for  $u_{1} = 0$ , we have  $s_1(x) = \gamma x$  for some  $\gamma > 0$  hence it is increasing. The remaining part is to show that the second function
$$
s _ {2} (x) = \frac {\mathbb {E} \left[ u ^ {\prime} z \sigma^ {\prime} (y) \right]}{\mathbb {E} [ \sigma (y) ]} \tag {165}
$$
is decreasing. We will consider  $z \gets u'z$  and  $x \gets ux$  in what follows. We have
$$
\frac {d}{d x} \frac {\mathbb {E} [ z \sigma^ {\prime} (x + z) ]}{\mathbb {E} [ \sigma (x + z) ]} <   0 \quad \Leftrightarrow \quad \frac {d}{d x} \frac {\mathbb {E} [ \sigma (x + z) ]}{\mathbb {E} [ \sigma^ {\prime \prime} (x + z) ]} > 0
$$
due to first applying Stein's Lemma to the numerator and then inverting the ratio. Using the chain rule, it is sufficient to show that
$$
f _ {1} (x) = \frac {\mathbb {E} [ \sigma (x + z) ]}{\mathbb {E} [ \sigma^ {\prime} (x + z) ]}, \quad \text {a n d} \quad f _ {2} (x) = \frac {\mathbb {E} [ \sigma^ {\prime} (x + z) ]}{\mathbb {E} [ \sigma^ {\prime \prime} (x + z) ]} \tag {166}
$$
are increasing, since both functions are positive.
Interestingly,  $f_{1}$  is increasing in  $x$  if  $\sigma$  is a log-concave function. Because its derivative is positive
$$
\frac {d}{d x} f _ {1} (x) = 1 - \frac {\mathbb {E} [ \sigma (x + z) ] \mathbb {E} [ \sigma^ {\prime \prime} (x + z) ]}{\mathbb {E} [ \sigma^ {\prime} (x + z) ] ^ {2}} > 0
$$
if  $\mathbb{E}[\sigma (x + z)]$  is log-concave. This is the case since a centered Gaussian distribution  $p(z)$  is log-concave, therefore  $\sigma (x + z)p(z)$  is jointly log-concave, and marginalization preserves log-concavity.
Similarly,  $f_{2}$  is increasing since  $\sigma'$  is also log-concave due to property (v) of softplus (Lemma G.8). Hence we showed that
$$
r (u) = \frac {\mathbb {E} \left[ \sigma^ {\prime} (y) \right] x}{\mathbb {E} [ \sigma (y) ]} \tag {167}
$$
is increasing for  $u \in [0,1)$ . The derivative of  $r$  explodes at 1, however, we can conclude by contradiction that  $r(1) > r(u)$  for  $u < 1$ : if  $r(u) \geq r(1)$  for some  $0 \leq u < 1$ , then there exists  $u_0 \in (u,1)$  where the function is decreasing. Therefore,  $r$  is increasing for  $u \in [0,1]$ . We can conclude the first part of the proof by the FKG inequality  $\sigma'(rx)x / \sigma(rx)$  and  $\mathbb{E}[\sigma(y_1)] / \mathbb{E}[\sigma(y_2)]$  are negatively correlated.
For the second part of the statement, we need to show
$$
\mathbb {E} \left[ \sigma^ {\prime} (r x) \sigma \left(u x + u ^ {\prime} z\right) x \right] \mathbb {E} \left[ \sigma (r x) \right] <   \mathbb {E} \left[ \sigma^ {\prime} (r x) x \right] \mathbb {E} \left[ \sigma (r x) \sigma \left(u x + u ^ {\prime} z\right) \right] \tag {168}
$$
for  $u < 0$ . Changing the measure from standard Gaussian  $p(x)$  to
$$
\tilde {p} (x) = p (x) \frac {\sigma (r x)}{\mathbb {E} [ \sigma (r x) ]}, \tag {169}
$$
the above inequality is equivalent to
$$
\mathbb {E} _ {x \sim \tilde {p}} \left[ \frac {\sigma^ {\prime} (r x) x}{\sigma (r x)} \sigma \left(u x + u ^ {\prime} z\right) \right] <   \mathbb {E} _ {x \sim \tilde {p}} \left[ \frac {\sigma^ {\prime} (r x) x}{\sigma (r x)} \right] \mathbb {E} _ {x \sim \tilde {p}} [ \sigma \left(u x + u ^ {\prime} z\right) ]. \tag {170}
$$
This holds since  $\sigma'(rx)x / \sigma(rx)$  is increasing in  $x$ , however,  $\sigma(ux + u'z)$  is decreasing in  $x$  since  $u$  is negative which implies a negative correlation due to the FKG inequality.
Lemma G.7. The following function
$$
\frac {\mathbb {E} [ \sigma^ {\prime} (x + z) | x ] x}{\mathbb {E} [ \sigma (x + z) | x ]}
$$
is increasing in  $x$  where the integrations are w.r.t a centered Gaussian  $z$ .
Proof. Since all integrals are w.r.t  $z$ , we drop the conditioning with respect to  $x$  in the proof. Taking the derivative w.r.t  $x$ , and arranging the terms, it suffices to show
$$
\left(\frac {\mathbb {E} \left[ \sigma^ {\prime \prime} (x + z) \right] x}{\mathbb {E} \left[ \sigma^ {\prime} (x + z) \right]} + 1\right) \mathbb {E} [ \sigma (x + z) ] > \mathbb {E} [ \sigma^ {\prime} (x + z) ] x \tag {171}
$$
which is equivalent to the following due to the property  $\sigma''(z) = \beta \sigma'(z)(1 - \sigma'(z))$
$$
\left(\beta x \left(1 - \frac {\mathbb {E} \left[ \sigma^ {\prime} (x + z) ^ {2} \right]}{\mathbb {E} \left[ \sigma^ {\prime} (x + z) \right]}\right) + 1\right) \mathbb {E} [ \sigma (x + z) ] > \mathbb {E} [ \sigma^ {\prime} (x + z) ] x. \tag {172}
$$
In the case  $x \geq 0$ , the LHS is bigger than  $\mathbb{E}[\sigma(x + z)]$  since  $\sigma'(\cdot)$  is upper bounded by 1. Moreover, since  $\sigma(x) > x$  and from the convexity of  $\text{softplus}$ , we get  $\mathbb{E}[\sigma(x + z)] > x$ . This yields the above inequality by again noting that  $\mathbb{E}[\sigma'(x + z)]$  is upper bounded by 1.
In the case  $x < 0$ , we need another strategy. We have thanks to Cauchy-Schwartz
$$
\frac {\mathbb {E} \left[ \sigma^ {\prime} (x + z) ^ {2} \right]}{\mathbb {E} \left[ \sigma^ {\prime} (x + z) \right]} \geq \mathbb {E} \left[ \sigma^ {\prime} (x + z) \right], \tag {173}
$$
thus it suffices to show