SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition E.4. Define  $f(\mathbf{W}, \mathbf{w}) = \mathbf{w}^\top \sigma (\mathbf{W}^\top \mathbf{x})$ , where  $\sigma$  is the activation function. When  $\| \mathbf{x} \|_1 \leq r_1$ ,  $\| \mathbf{w} \| \leq r_2$  and  $\sigma''(x) \leq \alpha$ , we have  $\operatorname{tr}(\nabla^2 f(\mathbf{W}, \mathbf{w})) \leq \alpha r_1 r_2$ .
Proof. By direct computation, we have
$$
\frac {\partial f}{\partial \mathbf {w}} = \sigma (\mathbf {W} ^ {\top} \mathbf {x}), \quad \frac {\partial f}{\partial \mathbf {W}} = \left(\sigma^ {\prime} (\mathbf {W} ^ {\top} \mathbf {x}) \odot \mathbf {w}\right) \otimes \mathbf {x},
$$
$$
\frac {\partial^ {2} f}{\partial \mathbf {w} ^ {2}} = \mathbf {0}, \quad \frac {\partial^ {2} f}{\partial \mathbf {W} ^ {2}} = \operatorname {D i a g} (\sigma^ {\prime \prime} (\mathbf {W} ^ {\top} \mathbf {x}) \odot \mathbf {w}) \otimes \mathbf {x} \otimes \mathbf {x}.
$$
$$
\begin{array}{l} \operatorname {t r} \left(\nabla^ {2} f (\mathbf {W}, \mathbf {w})\right)) = \| \mathbf {x} \| ^ {2} \cdot \operatorname {t r} \left(\operatorname {D i a g} \left(\sigma^ {\prime \prime} \left(\mathbf {W} ^ {\top} \mathbf {x}\right) \odot \mathbf {w}\right)\right) \\ \leq r _ {1} ^ {2} \cdot \left\langle \sigma^ {\prime \prime} \left(\mathbf {W} ^ {\top} \mathbf {x}\right), \mathbf {x} \right\rangle \tag {E.2} \\ \leq \alpha r _ {1} r _ {2}. \\ \end{array}
$$