SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| I _ {3} \right| \lesssim \frac {R _ {\max }}{1 - \gamma} \cdot | \mathcal {A} | \cdot \left((1 - \varepsilon) C _ {t} + \varepsilon \cdot C _ {t}\right) \tag {113} \\ = C _ {d} \cdot \left(C _ {t} + (1 - C _ {t}) \varepsilon\right) \cdot \frac {R _ {\max}}{1 - \gamma}, \\ \end{array}
$$
where  $C_d = |\mathcal{A}|$ .
In conclusion, let  $\pmb{\alpha} \in \mathbb{R}^{Kd}$  and  $\pmb{\alpha}_j \in \mathbb{R}^d$  with  $\pmb{\alpha} = [\pmb{\alpha}_1^T, \pmb{\alpha}_2^T, \dots, \pmb{\alpha}_K^T]^T$ , we have
$$
\begin{array}{l} \left\| g _ {t} (\boldsymbol {W}) - \nabla f _ {t} (\boldsymbol {W}) \right\| _ {2} \\ = \left| \boldsymbol {\alpha} ^ {T} \left(g _ {t} (\boldsymbol {W}) - \nabla f _ {t} (\boldsymbol {W})\right) \right| \\ \leq \sum_ {k = 1} ^ {K} \left| \boldsymbol {\alpha} _ {k} ^ {T} \left(g _ {t} \left(\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}\right) - \frac {\partial f}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W})\right) \right| \\ \leq \sum_ {k = 1} ^ {K} \left\| g _ {t} \left(\boldsymbol {w} _ {\ell , k}; \boldsymbol {W}\right) - \frac {\partial f}{\partial \boldsymbol {w} _ {\ell , k}} (\boldsymbol {W}) \right\| _ {2} \cdot \| \boldsymbol {\alpha} _ {k} \| _ {2} \tag {114} \\ \leq \sum_ {k = 1} ^ {K} \left(\| \boldsymbol {I} _ {1} \| _ {2} + \| \boldsymbol {I} _ {2} \| _ {2} + \| \boldsymbol {I} _ {3} \| _ {2}\right) \cdot \| \boldsymbol {\alpha} _ {k} \| _ {2} \\ \leq \frac {2 - \varepsilon}{K} \sqrt {\frac {d \log q}{N}} \cdot \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} + \frac {(1 - \varepsilon / 2) \gamma}{K} \cdot \| \boldsymbol {W} ^ {(t, 0)} - \boldsymbol {W} ^ {\star} \| _ {2} \\ + C _ {d} \cdot \left(C _ {t} + (1 - C _ {t}) \varepsilon\right) \cdot \frac {R _ {\max}}{1 - \gamma} \\ \end{array}
$$
E Additional proof of the lemmas in Appendix C
E.1 Proof of Lemma 6
The distance of the second order derivatives of the population risk function  $f(\cdot)$  at point  $W$  and  $W^{\star}$  can be converted into bounding  $P_{1}, P_{2}$ , which are defined in (116). The major idea in proving  $P_{1}$  is to connect the error bound to the angle between  $W$  and  $W^{\star}$  given  $h^{(\ell)}$  belongs to the sub-Gaussian distribution.
Proof of Lemma 6. From the definition of  $f$  in (31), we have
$$
\frac {\partial^ {2} f}{\partial \boldsymbol {w} _ {\ell , j _ {1}} \partial \boldsymbol {w} _ {\ell , j _ {2}}} (\boldsymbol {W} ^ {\star}) = \frac {1}{K ^ {2}} \mathbb {E} _ {\boldsymbol {x}} \mathcal {I} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {j _ {1}} ^ {\star \top} \boldsymbol {h}\right) \cdot \mathcal {I} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {j _ {2}} ^ {\star \top} \boldsymbol {h}\right) \cdot \boldsymbol {h} ^ {\star} \boldsymbol {h} ^ {\star \top}, \tag {115}
$$
and  $\frac{\partial^2f}{\partial\boldsymbol{w}_{\ell,j_1}\partial\boldsymbol{w}_{\ell,j_2}} (\boldsymbol {W}) = \frac{1}{K^2}\mathbb{E}_{\boldsymbol{x}}\phi '\mathcal{J}_{\ell ,k}^\star (\boldsymbol{w}_{\ell ,j_1}^\top \boldsymbol {h})\cdot \mathcal{J}_{\ell ,k}^\star \phi '(\boldsymbol{w}_{\ell ,j_2}^\top \boldsymbol {h})\cdot \boldsymbol {h}\boldsymbol{h}^\top,$
where  $\pmb {h} = \pmb{h}^{(\ell)}(\pmb {W})$  and  $h^\star = h^{(\ell)}(W^{\star})$
$$
\begin{array}{l} \frac {\partial^ {2} f}{\partial \boldsymbol {w} _ {\ell , j _ {1}} \partial \boldsymbol {w} _ {\ell , j _ {2}}} (\boldsymbol {W} ^ {*}) - \frac {\partial^ {2} f}{\partial \boldsymbol {w} _ {\ell , j _ {1}} \partial \boldsymbol {w} _ {\ell , j _ {2}}} (\boldsymbol {W}) \\ = \frac {1}{K ^ {2}} \mathbb {E} _ {\boldsymbol {x}} \left[ \mathcal {J} _ {\ell , k} ^ {\star} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {1}} ^ {\star T} \boldsymbol {h} ^ {\star}\right) \mathcal {J} _ {\ell , k} ^ {\star} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \boldsymbol {h} ^ {\star}\right) \boldsymbol {h} ^ {\star} \boldsymbol {h} ^ {\star \top} - \mathcal {J} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {1}} ^ {\top} \boldsymbol {h}\right) \mathcal {J} _ {\ell , k} \mathcal {J} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \boldsymbol {h}\right) \boldsymbol {h h} ^ {\top} \right] \\ = \frac {1}{K ^ {2}} \mathbb {E} _ {\boldsymbol {x}} \left[ \mathcal {J} _ {\ell , k} ^ {\star} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {1}} ^ {\star T} \boldsymbol {h} ^ {\star}\right) \left(\mathcal {J} _ {\ell , k} ^ {\star} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \boldsymbol {h} ^ {\star}\right) \boldsymbol {h} ^ {\star} \boldsymbol {h} ^ {\star \top} - \mathcal {J} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \boldsymbol {h}\right) \boldsymbol {h} \boldsymbol {h} ^ {\top}\right) \right. \\ \left. \right.\left. + \mathcal {J} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \boldsymbol {h}\right)\left(\mathcal {J} _ {\ell , k} ^ {\star} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {1}} ^ {\star T} \boldsymbol {h}\right) \boldsymbol {h} ^ {\star} \boldsymbol {h} ^ {\star \top} - \mathcal {J} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {1}} ^ {\top} \boldsymbol {h}\right) \boldsymbol {h} \boldsymbol {h} ^ {\top}\right)\right] \\ := \frac {1}{K ^ {2}} \left(\boldsymbol {P} _ {1} + \boldsymbol {P} _ {2}\right). \tag {116} \\ \end{array}
$$
For any  $\pmb{a} \in \mathbb{R}^{K_{\ell}}$  with  $\| \pmb{a} \|_2 = 1$ , we have
$$
\boldsymbol {a} ^ {\top} \boldsymbol {P} _ {1} \boldsymbol {a} = \mathbb {E} _ {\boldsymbol {x}} \mathcal {J} _ {\ell , k} ^ {\star} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {1}} ^ {\star T} \boldsymbol {h} ^ {\star}\right) \left(\mathcal {J} _ {\ell , k} ^ {\star} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \boldsymbol {h} ^ {\star}\right) \left(\boldsymbol {a} ^ {\top} \boldsymbol {h} ^ {\star}\right) ^ {2} - \mathcal {J} _ {\ell , k} \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \boldsymbol {h}\right) \left(\boldsymbol {a} ^ {\top} \boldsymbol {h}\right) ^ {2}\right). \tag {117}
$$
Utilizing the Gram-Schmidt process, we can demonstrate the existence of a set of normalized orthonormal vectors denoted as  $\mathcal{B} = \{a,b,c,a_4^\perp ,\dots ,a_d^\perp \} \in \mathbb{R}^d$ . This set forms an orthogonal and normalized basis for  $\mathbb{R}^d$ , wherein the subspace spanned by  $a,b,c$  includes  $a,w_{\ell,j_2}$ , and  $w_{\ell,j_2}^*$ .
Then, for any  $\pmb{x} \in \mathbb{R}^d$ , we have a unique  $\pmb{z} = [z_1, z_2, \dots, z_d]^\top$  such that
$$
\boldsymbol {h} = z _ {1} \boldsymbol {a} + z _ {2} \boldsymbol {b} + z _ {3} \boldsymbol {c} + \dots + z _ {d} \boldsymbol {a} _ {d} ^ {\perp}.
$$
Because (i)  $\pmb{a}, \pmb{w}_{\ell,j_2}$ , and  $\pmb{w}_{\ell,j_2}^*$  belongs to the subspace spanned by vectors  $\{\pmb{a}, \pmb{b}, \pmb{c}\}$  and (ii)  $\pmb{a}_4^\perp, \dots, \pmb{a}_d^\perp, \dots$  are orthogonal to  $\pmb{a}, \pmb{b}$ , and  $\pmb{c}$ . Then, we know that
$$
\begin{array}{l} \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \boldsymbol {h} = \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \left(z _ {1} \boldsymbol {a} + z _ {2} \boldsymbol {b} + z _ {3} \boldsymbol {c} + \dots + z _ {d} \boldsymbol {a} _ {d} ^ {\perp}\right) \\ = z _ {1} \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \boldsymbol {a} + z _ {2} \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \boldsymbol {b} + z _ {3} \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \boldsymbol {c} + \dots + z _ {d} \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \boldsymbol {a} _ {d} ^ {\perp} \\ = z _ {1} \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \boldsymbol {a} + z _ {2} \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \boldsymbol {b} + z _ {3} \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \boldsymbol {c} + 0 \tag {119} \\ = \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \left(z _ {1} \boldsymbol {a} + z _ {2} \boldsymbol {b} + z _ {3} \boldsymbol {c}\right) \\ \boldsymbol {:=} \boldsymbol {\mathcal {W}} _ {\ell , j} ^ {\top} \widetilde {\boldsymbol {h}}. \\ \end{array}
$$
where  $\widetilde{\pmb{h}} = z_1\pmb {a} + z_2\pmb {b} + z_3\pmb{c}$ . Similar to (119), we have  $\pmb{w}_{\ell ,j_2}^{\top}\pmb {h} = \pmb{w}_{\ell ,j_2}^{\top}\widetilde{\pmb{h}}$  and  $\pmb{a}^{\top}\pmb {h} = \pmb{a}^{\top}\widetilde{\pmb{h}}$
Then, we define  $I_4$  as
$$
\begin{array}{l} I _ {4} := \mathbb {E} _ {\boldsymbol {h}} \left| \left(\phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \boldsymbol {h}) - \phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \boldsymbol {h})\right) \cdot \left(\boldsymbol {a} ^ {\top} \boldsymbol {h}\right) \right| \\ = \int_ {\mathcal {R} _ {h}} \left| \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \boldsymbol {h}\right) - \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \boldsymbol {h}\right) \right| \cdot \left| \boldsymbol {a} ^ {\top} \boldsymbol {h} \right| ^ {2} \cdot f _ {H} (\boldsymbol {h}) d \boldsymbol {h} \tag {120} \\ = \int_ {\mathcal {R} _ {z}} \left| \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \boldsymbol {h}\right) - \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {* T} \boldsymbol {h}\right) \right| \cdot \left| \boldsymbol {a} ^ {\top} \boldsymbol {h} \right| ^ {2} \cdot f _ {Z} (\boldsymbol {z}) \cdot | J _ {\boldsymbol {h}} (\boldsymbol {z}) | d \boldsymbol {z} \\ \end{array}
$$
where  $|J_h(z)|$  is the determinant of the Jacobian matrix  $\frac{\partial h}{\partial z}$ . Since  $z$  is a representation of  $h$  based on an orthogonal and normalized basis, we have  $|J_h(z)| = 1$ . According to (119),  $I_4$  can be rewritten as
$$
\begin{array}{l} I _ {4} = \int_ {\mathcal {R} _ {z}} \left| \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \widetilde {\boldsymbol {h}}\right) - \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \widetilde {\boldsymbol {h}}\right) \right| \cdot \left| \boldsymbol {a} ^ {\top} \widetilde {\boldsymbol {h}} \right| ^ {2} \cdot f _ {Z} (z) d z \tag {121} \\ = \int_ {\mathcal {R} _ {z}} \left| \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \widetilde {\boldsymbol {h}}\right) - \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \widetilde {\boldsymbol {h}}\right) \right| \cdot \left| \boldsymbol {a} ^ {\top} \widetilde {\boldsymbol {h}} \right| ^ {2} \cdot f _ {Z} \left(z _ {1}, z _ {2}, z _ {3}\right) d z _ {1} d z _ {2} d z _ {3} \\ \end{array}
$$
where in the last equality we abuse  $f_{Z}(z_{1},z_{2},z_{3})$  to represent the probability density function of  $(z_{1},z_{2},z_{3})$  defined in region  $\mathcal{R}_z$ .
Next, we show that  $\mathbf{z}$  is rotational invariant over  $\mathcal{R}_z$ . Let  $\mathbf{R} = [a b c \cdots a_d^\perp]$ , we have  $\mathbf{h} = \mathbf{R} \mathbf{z}$ . For any  $\mathbf{z}^{(1)}$  and  $\mathbf{z}^{(2)}$  with  $\| \mathbf{z}^{(1)} \|_2 = \| \mathbf{z}^{(2)} \|_2$ . We define  $\mathbf{h}^{(1)} = R \mathbf{z}^{(1)}$  and  $\mathbf{h}^{(2)} = R \mathbf{z}^{(2)}$ . Since  $\mathbf{x}$  is rotational invariant and  $\| \mathbf{h}^{(1)} \|_2 = \| \mathbf{h}^{(2)} \|_2 = \| \mathbf{z}^{(1)} \|_2 = \| \mathbf{z}^{(2)} \|_2$ , then we know  $\mathbf{h}^{(1)}$  and  $\mathbf{h}^{(2)}$  has the same distribution density. Then,  $\mathbf{z}^{(1)}$  and  $\mathbf{z}^{(2)}$  has the same distribution density as well. Therefore,  $\mathbf{z}$  is rotational invariant over  $\mathcal{R}_z$ .
Then, we consider spherical coordinates with  $z_{1} = R\cos \phi_{1}, z_{2} = R\sin \phi_{1}\sin \phi_{2}, z_{3} = R\sin \phi_{1}\cos \phi_{2}$ . Hence, we have
$$
I _ {4} = \int | \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \widetilde {\boldsymbol {h}}\right) - \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \widetilde {\boldsymbol {h}}\right) | \cdot | R \cos \phi_ {1} | ^ {2} \cdot f _ {Z} (R, \phi_ {1}, \phi_ {2}) \cdot R ^ {2} \sin \phi_ {1} \cdot d R d \phi_ {1} d \phi_ {2}. \tag {122}
$$
Since  $z$  is rotational invariant, we have that
$$
f _ {Z} \left(R, \phi_ {1}, \phi_ {2}\right) = f _ {Z} (R). \tag {123}
$$
$$
\begin{array}{l} I _ {4} = \int | \phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} (\widetilde {\boldsymbol {h}} / R)) - \phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} (\widetilde {\boldsymbol {h}} / R)) | \cdot | R \cos \phi_ {1} | ^ {2} \cdot f _ {Z} (R) R ^ {2} \sin \phi_ {1} d R d \phi_ {1} d \phi_ {2} \\ = \int_ {0} ^ {\infty} R ^ {4} f _ {z} (R) d R \int_ {0} ^ {\psi_ {1} (R)} \int_ {0} ^ {\psi_ {2} (R)} | \cos \phi_ {1} | ^ {2} \cdot \sin \phi_ {1} \tag {124} \\ \cdot \left| \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} (\widetilde {\boldsymbol {h}} / R)\right) - \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} (\widetilde {\boldsymbol {h}} / R)\right) \right| d \phi_ {1} d \phi_ {2} \\ \leq \int_ {0} ^ {\infty} R ^ {4} f _ {z} (R) d R \int_ {0} ^ {\pi} \int_ {0} ^ {2 \pi} \sin \phi_ {1} \cdot | \phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \bar {\boldsymbol {x}}) - \phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \bar {\boldsymbol {x}}) | d \phi_ {1} d \phi_ {2}, \\ \end{array}
$$
where the first equality holds because  $\phi^{\prime}(\pmb{w}_{i,..,j_2}^{\top}\pmb{h})$  only depends on the direction of  $\pmb{h}$ , and  $\bar{x} := h / R = (\cos \phi_1, \sin \phi_1 \sin \phi_2, \sin \phi_1 \cos \phi_2)$  in the last inequality.
Because  $z$  belongs to the sub-Gaussian distribution, we have  $F_{z}(R) \geq 1 - 2e^{-\frac{R^{2}}{\sigma^{2}}}$  for some constant  $\sigma > 0$ . Then, the integration of  $R$  can be represented as
$$
\begin{array}{l} \int_ {0} ^ {\infty} R ^ {4} f _ {Z} (R) d R = \int_ {0} ^ {\infty} R ^ {4} d \left(1 - F _ {z} (R)\right) \\ \leq \int_ {0} ^ {\infty} 4 R ^ {3} (1 - F _ {z} (R)) d R \\ \leq \int_ {0} ^ {\infty} 8 R ^ {3} e ^ {- \frac {R ^ {2}}{\sigma^ {2}}} d R \tag {125} \\ \leq \frac {3 2}{\sqrt {2 \pi}} \sigma \int_ {0} ^ {\infty} R ^ {2} e ^ {- \frac {R ^ {2}}{\sigma^ {2}}} d R \\ = 3 2 \sigma^ {2} \int_ {0} ^ {\infty} R ^ {2} \frac {1}{\sqrt {2 \pi \sigma^ {2}}} e ^ {- \frac {R ^ {2}}{\sigma^ {2}}} d R, \\ \end{array}
$$
where the last inequality comes from the calculation that
$$
\begin{array}{l} \int_ {0} ^ {\infty} 2 R ^ {2} e ^ {- \frac {R ^ {2}}{\sigma^ {2}}} d R = \sqrt {2 \pi} \sigma^ {3}, \tag {126} \\ \int_ {0} ^ {\infty} 2 R ^ {3} e ^ {- \frac {R ^ {2}}{\sigma^ {2}}} d R = 4 \sigma^ {4}. \\ \end{array}
$$
Then, we define  $\widetilde{\pmb{x}}\in \mathbb{R}^{K_{\ell}}$  belongs to Gaussian distribution as  $\widetilde{\pmb{x}}\sim \mathcal{N}(\mathbf{0},\sigma^2\pmb {I})$ . Therefore, we have
$$
\begin{array}{l} I _ {4} \leq 3 2 \sigma^ {2} \cdot \int_ {0} ^ {\infty} R ^ {2} \frac {1}{\sqrt {2 \pi \sigma^ {2}}} e ^ {- \frac {R ^ {2}}{\sigma^ {2}}} d R \int_ {0} ^ {\pi} \int_ {0} ^ {2 \pi} \sin \phi_ {1} \cdot | \phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \bar {\boldsymbol {x}}) - \phi^ {\prime} (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \bar {\boldsymbol {x}}) | d \phi_ {1} d \phi_ {2} \\ = 3 2 \sigma^ {2} \cdot \mathbb {E} _ {z _ {1}, z _ {2}, z _ {3}} \left| \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \widetilde {\boldsymbol {x}}\right) - \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \widetilde {\boldsymbol {x}}\right) \right| \tag {127} \\ \overleftarrow {\mathbb {E} _ {\widetilde {\boldsymbol {x}}}} \big | \phi^ {\prime} \big (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \widetilde {\boldsymbol {x}} \big) - \phi^ {\prime} \big (\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \widetilde {\boldsymbol {x}} \big) \big |, \\ \end{array}
$$
where  $\widetilde{x}$  belongs to Gaussian distribution.
Therefore, the inequality bound over a sub-Gaussian distribution is bounded by the one over a Gaussian distribution. In the following contexts, we provide the upper bound of  $\mathbb{E}_{\widetilde{\boldsymbol{x}}}\big|\phi^{\prime}\big(\boldsymbol{w}_{\ell ,j_2}^{\top}\widetilde{\boldsymbol{x}}\big) - \phi^{\prime}\big(\boldsymbol{w}_{\ell ,j_2}^{\star T}\widetilde{\boldsymbol{x}}\big)\big|$ .
Define a set  $\mathcal{A}_1 = \{\pmb{x} | (\pmb{w}_{\ell,j_2}^{\star \top} \widetilde{\pmb{x}})(\pmb{w}_{\ell,j_2}^\top \widetilde{\pmb{x}}) < 0\}$ . If  $\widetilde{\pmb{x}} \in \mathcal{A}_1$ , then  $\pmb{w}_{\ell,j_2}^{\star \top} \widetilde{\pmb{x}}$  and  $\pmb{w}_{\ell,j_2}^\top \widetilde{\pmb{x}}$  have different signs, which means the value of  $\phi'(\pmb{w}_{\ell,j_2}^\top \widetilde{\pmb{x}})$  and  $\phi'(\pmb{w}_{\ell,j_2}^{\star \top} \widetilde{\pmb{x}})$  are different. This is equivalent to say that
$$
\left| \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \widetilde {\boldsymbol {x}}\right) - \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star \top} \widetilde {\boldsymbol {x}}\right) \right| = \left\{ \begin{array}{l l} 1, & \text {i f} \widetilde {\boldsymbol {x}} \in \mathcal {A} _ {1} \\ 0, & \text {i f} \widetilde {\boldsymbol {x}} \in \mathcal {A} _ {1} ^ {c}. \end{array} \right. \tag {128}
$$
Moreover, if  $\widetilde{\pmb{x}}\in \mathcal{A}_1$  , then we have
$$
\left| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \widetilde {\boldsymbol {x}} \right| \leq \left| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \widetilde {\boldsymbol {x}} - \boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \widetilde {\boldsymbol {x}} \right| \leq \left\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star} - \boldsymbol {w} _ {\ell , j _ {2}} \right\| _ {2} \cdot \| \widetilde {\boldsymbol {x}} \| _ {2}. \tag {129}
$$
Let us define a set  $\mathcal{A}_2$  such that
$$
\begin{array}{l} \mathcal {A} _ {2} = \left\{\widetilde {\boldsymbol {x}} \Big | \frac {\left| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \widetilde {\boldsymbol {x}} \right|}{\left\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {*} \right\| _ {2} \left\| \widetilde {\boldsymbol {x}} \right\| _ {2}} \leq \frac {\left\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {*} - \boldsymbol {w} _ {\ell , j _ {2}} \right\| _ {2}}{\left\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {*} \right\| _ {2}} \right\} \tag {130} \\ = \Bigl \{\theta_ {\widetilde {\boldsymbol {x}}, \boldsymbol {w} _ {\ell , j _ {2}} ^ {*}} \bigg | | \cos \theta_ {\widetilde {\boldsymbol {x}}, \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star}} | \leq \frac {\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star} - \boldsymbol {w} _ {\ell , j _ {2}} \| _ {2}}{\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star} \| _ {2}} \Bigr \}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {\widetilde {\boldsymbol {x}}} \left| \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \widetilde {\boldsymbol {x}}\right) - \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \widetilde {\boldsymbol {x}}\right)\right| ^ {2} = \mathbb {E} _ {\widetilde {\boldsymbol {x}}} \left| \right. \phi^ {\prime} \left(\boldsymbol {w} _ {\ell , j _ {2}} ^ {\top} \widetilde {\boldsymbol {x}}\right) - \phi^ {\prime} \left( \right.\boldsymbol {w} _ {\ell , j _ {2}} ^ {\star T} \widetilde {\boldsymbol {x}}\big) \left. \right| \\ = \operatorname {P r o b} (\widetilde {\boldsymbol {x}} \in \mathcal {A} _ {1}) \tag {131} \\ \leq \operatorname {P r o b} (\widetilde {\boldsymbol {x}} \in \mathcal {A} _ {2}). \\ \end{array}
$$
Since  $\widetilde{\pmb{x}}\sim \mathcal{N}(\mathbf{0},\| \pmb {a}\| _2^2\pmb {I})$ $\theta_{\widetilde{\boldsymbol{x}},\boldsymbol{w}_{\ell ,j_2}^{\star}}$  belongs to the uniform distribution on  $[- \pi ,\pi ]$  , we have
$$
\begin{array}{l} \operatorname {P r o b} (\widetilde {\boldsymbol {x}} \in \mathcal {A} _ {2}) = \frac {\pi - \operatorname {a r c c o s} \frac {\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star} - \boldsymbol {w} _ {\ell , j _ {2}} \| _ {2}}{\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star} \| _ {2}}}{\pi} \leq \frac {1}{\pi} \tan (\pi - \operatorname {a r c c o s} \frac {\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star} - \boldsymbol {w} _ {\ell , j _ {2}} \| _ {2}}{\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star} \| _ {2}}) \\ = \frac {1}{\pi} \cot (\operatorname {a r c c o s} \frac {\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star} - \boldsymbol {w} _ {\ell , j _ {2}} \| _ {2}}{\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star} \| _ {2}}) \tag {132} \\ \leq \frac {2}{\pi} \frac {\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star} - \boldsymbol {w} _ {\ell , j _ {2}} \| _ {2}}{\| \boldsymbol {w} _ {\ell , j _ {2}} ^ {\star} \| _ {2}} \\ \leq \left\| \boldsymbol {W} _ {\ell} ^ {\star} - \boldsymbol {W} _ {\ell} \right\| _ {2} \\ \end{array}
$$
Hence, (124) and (132) suggest that
$$
I _ {4} \lesssim \| \boldsymbol {W} _ {i} - \boldsymbol {W} _ {i} ^ {\star} \| _ {2} \cdot \| \boldsymbol {a} \| _ {2} ^ {2}, \tag {133}
$$
$$
\text {a n d} \quad \| P _ {1} \| _ {2} \leq \| W - W ^ {\star} \| _ {2} + I _ {4} \lesssim \| W - W ^ {\star} \| _ {2},
$$
The same bound that is shown in (133) holds for  $P_{2}$  as well.
$$
\begin{array}{l} \left\| \nabla_ {\ell} ^ {2} f (\boldsymbol {W} ^ {\star}) - \nabla_ {\ell} ^ {2} f (\boldsymbol {W}) \right\| _ {2} = \max  _ {\| \boldsymbol {\alpha} \| _ {2} \leq 1} \left| \boldsymbol {\alpha} ^ {\top} \left(\nabla_ {\ell} ^ {2} f (\boldsymbol {W} ^ {\star}) - \nabla_ {\ell} ^ {2} f (\boldsymbol {W})\right) \boldsymbol {\alpha} \right| \\ \leq \frac {1}{K ^ {2}} \sum_ {j _ {1} = 1} ^ {K} \sum_ {j _ {2} = 1} ^ {K} \| P _ {1} + P _ {2} \| _ {2} \cdot \| \alpha_ {j _ {1}} \| _ {2} \cdot \| \alpha_ {j _ {2}} \| _ {2} \\ \lesssim \frac {1}{K ^ {2}} \cdot \sum_ {j _ {1} = 1} ^ {K} \sum_ {j _ {2} = 1} ^ {K} \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} \cdot \| \boldsymbol {\alpha} _ {j _ {1}} \| _ {2} \| \boldsymbol {\alpha} _ {j _ {2}} \| _ {2} \tag {134} \\ \lesssim \frac {1}{K ^ {2}} \cdot \sum_ {j _ {1} = 1} ^ {K} \sum_ {j _ {2} = 1} ^ {K} \| \boldsymbol {W} - \boldsymbol {W} ^ {\star} \| _ {2} \cdot \left(\frac {\| \boldsymbol {\alpha} _ {j _ {1}} \| _ {2} ^ {2} + \| \boldsymbol {\alpha} _ {j _ {2}} \| _ {2} ^ {2}}{2}\right) \\ \lesssim \frac {1}{K} \cdot \| \boldsymbol {W} ^ {\star} - \boldsymbol {W} \| _ {2}, \\ \end{array}
$$
where  $\pmb {\alpha}\in \mathbb{R}^{Kd}$  and  $\pmb {\alpha}_j\in \mathbb{R}^{K_\ell}$  with  $\pmb {\alpha} = [\pmb{\alpha}_{1}^{\top},\pmb{\alpha}_{2}^{\top},\dots ,\pmb{\alpha}_{K}^{\top}]^{\top}$ .
E.2 Proof of Lemma 7
We aim to prove that  $\int_{\mathcal{R}}\left(\sum_{j = 1}^{K}\pmb{\alpha}^{\top}\pmb{h}\phi^{\prime}(\pmb{w}_{\ell ,j}^{\top}\pmb {h})\right)^{2}p_{H}(\pmb {h})\cdot d\pmb {h}$  is strictly greater than zero for any  $\alpha$ . Therefore, the  $\rho$  in (2) is strictly greater than zero. The proof is inspired by Theorem 3.1 in [22]. It is obviously that  $\left(\sum_{j = 1}^{K}\pmb{\alpha}^{\top}\pmb{h}\phi^{\prime}(\pmb{w}_{\ell ,j}^{\top}\pmb {h})\right)^{2}$  is greater or equal to zero. Given  $\left(\sum_{j = 1}^{K}\pmb{\alpha}^{\top}\pmb{h}\phi^{\prime}(\pmb{w}_{\ell ,j}^{\top}\pmb {h})\right)^{2}$  is continuous, we only need to show that  $\alpha$  such that  $\sum_{j = 1}^{K}\pmb{\alpha}^{\top}\pmb{h}\phi^{\prime}(\pmb{w}_{\ell ,j}^{\top}\pmb {h})\neq 0$  for any  $\alpha$ , namely,  $\{\pmb{h}\phi^{\prime}(\pmb{w}_{\ell ,j}^{\top}\pmb {h})\}_{j = 1}^{K}$  are linear independent. Compared with Theorem 3.1 in [22], we need to address two challenges: (1) the neuron weights  $\pmb{w}$  is the random variable in [22] while the input  $\pmb{h}$  is the random variable in this paper and (2) the random variable belongs to Gaussian distribution in [22] while the random variable belongs to sub-Gaussian distribution in this paper.
Proof of Lemma 7. Let  $\mathcal{H}$  be a Hilbert space on  $\mathbb{R}^{K_{\ell}}$ , and the inner product of  $\mathcal{H}$  is defined as
$$
\langle f, g \rangle = \int_ {\mathcal {R}} f (\boldsymbol {h}) ^ {\top} g (\boldsymbol {h}) f _ {H} (\boldsymbol {h}) \cdot d \boldsymbol {h}, \quad \forall f, g \in \mathcal {H}, \tag {135}
$$