SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left(\beta x - \beta x \mathbb {E} \left[ \sigma^ {\prime} (x + z) \right] + 1\right) \mathbb {E} \left[ \sigma (x + z) \right] > \mathbb {E} \left[ \sigma^ {\prime} (x + z) \right] x. \tag {174}
$$
We will now show the following
$$
\mathbb {E} \left[ \sigma^ {\prime} (x + z) \right] \geq \sigma^ {\prime} (x) \tag {175}
$$
for which it suffices to show that  $v(z) \coloneqq \sigma'(x + z) + \sigma'(x - z) \geq 2\sigma'(x)$  for all  $z$  since the centered Gaussian measure  $p(z)$  is even and the integration can be done over the integrand  $v(z)$ . We have
$$
v ^ {\prime} (z) = \sigma^ {\prime \prime} (x + z) - \sigma^ {\prime \prime} (x - z) \tag {176}
$$
that is zero iff either  $x + z = x - z$  or  $x + z = -x + z$  where the latter is not possible since  $x < 0$ . Hence we get that a critical point of  $v(z)$  at  $z = 0$  which is a minimizer since  $v''(0) = 2\sigma'''(x) > 0$  for  $x < 0$ . Hence  $v(z) \geq v(0) = 2\sigma'(x)$  for all  $z$  which completes the argument.
Finally, it remains to show
$$
\left(\frac {\beta x}{e ^ {\beta x} + 1} + 1\right) \mathbb {E} [ \sigma (x + z) ] > \mathbb {E} [ \sigma^ {\prime} (x + z) ] x. \tag {177}
$$
From the proof of Lemma G.8, we have that  $\beta \sigma(x) > \sigma'(x)$ , which in combination with the following trivial observation for all  $x < 0$  (note that  $+1$  is not needed for the following to hold)
$$
\frac {\beta x}{e ^ {\beta x} + 1} + 1 > \beta x \tag {178}
$$
shows that Eq. 177 holds, hence the proof is complete.
Lemma G.8. The softplus family has the following properties
i.  $\sigma (x)$  is increasing and convex,
ii.  $\sigma (x)$  is log-concave (equivalently,  $\sigma (x) / \sigma '(x)$  is increasing),
iii.  $\sigma (x) / \sigma '(x)$  is convex,
iv.  $\sigma'(x)x / \sigma(x)$  is increasing,
$\nu .\sigma^{\prime}(x)$  is log-concave.
Proof. For the property (i), see the formulas of  $\sigma'$  and  $\sigma''$  in the proof of Lemma F.2. We next prove each one of the properties one after the other. Let us start with property (ii). First note that  $\sigma(x)$  is log-concave if and only if  $\sigma(x) / \sigma'(x)$  is increasing since
$$
\frac {d}{d x} \frac {\sigma (x)}{\sigma^ {\prime} (x)} = 1 - \frac {\sigma (x) \sigma^ {\prime \prime} (x)}{\sigma^ {\prime} (x) ^ {2}} > 0 \Leftrightarrow \sigma^ {\prime} (x) ^ {2} > \sigma (x) \sigma^ {\prime \prime} (x) \tag {179}
$$
where the second inequality is a characterization of log-concavity. We will prove that  $R(x) \coloneqq \sigma(x) / \sigma'(x)$  is increasing.
Let us write out the ratio explicitly
$$
R (x) = \frac {1}{\beta} \left(\log \left(e ^ {\beta x} + 1\right) + \frac {\log \left(e ^ {\beta x} + 1\right)}{e ^ {\beta x}}\right). \tag {180}
$$
The first derivative of  $R$  is given by
$$
R ^ {\prime} (x) = \sigma^ {\prime} (x) + \frac {\sigma^ {\prime} (x) - \beta \sigma (x)}{e ^ {\beta x}} = \frac {e ^ {\beta x} - \beta \sigma (x)}{e ^ {\beta x}}. \tag {181}
$$
Since  $\log$  is concave, expanding it around 1 we get  $\log (y + 1) < y$  for all  $y > 0$ . Substituting  $y = e^{\beta x}$ , we get that the numerator of  $R'$  is positive, thus  $R$  is increasing. This completes the proof of property (ii). Computing the second derivative of  $R$ , we get
$$
R ^ {\prime \prime} (x) = \frac {\sigma^ {\prime \prime} (x) \left(e ^ {\beta x} + 1\right) - 2 \beta \sigma^ {\prime} (x) + \beta^ {2} \sigma (x)}{e ^ {\beta x}} = \beta \left(\frac {- \sigma^ {\prime} (x) + \beta \sigma (x)}{e ^ {\beta x}}\right). \tag {182}
$$
What remains to show is that  $\beta \sigma(x) > \sigma'(x)$ . Using the fundamental theorem of calculus, we get
$$
\frac {\log (y + 1)}{y} = \frac {1}{y} \int_ {0} ^ {y} \frac {1}{t + 1} d t > \frac {1}{y + 1} \tag {183}
$$
since  $1 / (y + 1)$  is a lower bound of the integrand which completes the proof of the property (iii). Let us prove the property (iv) by taking the derivative of the function of interest
$$
\frac {d}{d x} \frac {\sigma^ {\prime} (x) x}{\sigma (x)} = \frac {\left(\sigma^ {\prime \prime} (x) x + \sigma^ {\prime} (x)\right) \sigma (x) - \sigma^ {\prime} (x) ^ {2} x}{\sigma (x) ^ {2}} \tag {184}
$$
Using  $\sigma''(x) = \beta \sigma'(x)(1 - \sigma'(x))$  and dropping the positive term  $\sigma'(x)$ , the numerator of the derivative is
$$
\begin{array}{l} \left((1 - \sigma^ {\prime} (x)) \beta x + 1\right) \sigma (x) - \sigma^ {\prime} (x) x = \left(\frac {\beta x}{e ^ {\beta x} + 1} + 1\right) \sigma (x) - \frac {e ^ {\beta x}}{e ^ {\beta x} + 1} x (185) \\ = \frac {e ^ {\beta x}}{e ^ {\beta x} + 1} \left(\frac {1}{e ^ {\beta x}} \left(\beta x + e ^ {\beta x} + 1\right) \sigma (x) - x\right) (186) \\ \end{array}
$$
For the case  $x \geq 0$ , we have  $\sigma(x) > x$  and  $(\beta x + 1) / e^{\beta x} > 0$ , hence the derivative is positive. For the case  $x < 0$ , we want to show
$$
\left(e ^ {\beta x} + \beta x + 1\right) \frac {\log \left(e ^ {\beta x} + 1\right)}{e ^ {\beta x}} > \beta x. \tag {187}
$$
If  $e^{\beta x} + \beta x + 1 > 0$ , it is done since the LHS is positive. If  $e^{\beta x} + \beta x + 1 \leq 0$ , we have
$$
\left(e ^ {\beta x} + \beta x + 1\right) \frac {\log \left(e ^ {\beta x} + 1\right)}{e ^ {\beta x}} \geq \left(e ^ {\beta x} + \beta x + 1\right) \sup  \frac {\log \left(e ^ {\beta x} + 1\right)}{e ^ {\beta x}} \tag {188}
$$
since  $\log (e^{\beta x} + 1) / e^{\beta x}$  is positive. We will next show that  $\log (e^{\beta x} + 1) / e^{\beta x}$  is a decreasing function therefore its supremum is achieved at  $x\to -\infty$ . From the integral expression in Eq. 183, we deduce that  $\log (y + 1) / y$  is a decreasing function since adding smaller terms in the average decreases it. Thus the following limit gives us the supremum using L'Hôpital's rule
$$
\lim  _ {y \rightarrow 0} \frac {\log (y + 1)}{y} = \lim  _ {y \rightarrow 0} \frac {1}{y + 1} = 1. \tag {189}
$$
Combining it with the Eq. 188 after the substitution  $y = e^{\beta x}$ , we get the desired Ineq. 187 which implies that the derivative is positive in this case too. This completes the proof of property (iv).
For the property (v), we first give a formula for the third derivative of softplus
$$
\sigma^ {\prime \prime \prime} (x) = \beta \sigma^ {\prime \prime} (x) \left(1 - 2 \sigma^ {\prime} (x)\right). \tag {190}
$$
$\sigma^{\prime}$  is log-concave if and only if we have
$$
\sigma^ {\prime \prime \prime} (x) \sigma (x) <   \sigma^ {\prime \prime} (x) \sigma^ {\prime} (x) \Leftrightarrow
$$
$$
\beta \sigma^ {\prime \prime} (x) \left(1 - 2 \sigma^ {\prime} (x)\right) \sigma (x) <   \sigma^ {\prime \prime} (x) \sigma^ {\prime} (x) \tag {191}
$$
$$
(1 - e ^ {\beta x}) \log \left(e ^ {\beta x} + 1\right) <   e ^ {\beta x}. \tag {192}
$$
This is equivalent to  $(1 - y)\log (y + 1) < \log (y + 1) < y$  where  $y = e^{\beta x} > 0$ ; the second inequality holds due to  $y + 1 < e^{y}$ .