SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left| \widetilde {\mathcal {L}} _ {\mathcal {Z}} \left(f _ {\Xi}\right) - \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi}\right) \right| (141) \\ = \left| \frac {1}{n} \sum_ {l = 1} ^ {n} \left(\ell \left(y ^ {(l)} \mathbf {a} ^ {\top} \left[ \sigma \left(\mathbf {W} ^ {\top} \mathbf {x} ^ {(l)} - \mathbf {b}\right) \right]\right) - \mathbb {E} _ {(\mathbf {x}, y) \sim \mathcal {D}} \left[ \ell \left(y \mathbf {a} ^ {\top} \left[ \sigma \left(\mathbf {W} ^ {\top} \mathbf {x} - \mathbf {b}\right) \right]\right) \right]\right) \right|. (142) \\ \end{array}
$$
By Assumption 3.1, we have  $\ell \left(y^{(l)}\mathbf{a}^{\top}\left[\sigma (\mathbf{W}^{\top}\mathbf{x}^{(l)} - \mathbf{b})\right]\right) - \mathbb{E}_{(\mathbf{x},y)\sim \mathcal{D}}\left[\ell \left(y\mathbf{a}^{\top}\left[\sigma (\mathbf{W}^{\top}\mathbf{x} - \mathbf{b})\right]\right)\right]$  is a zero-mean random variable, with bound  $2\| \mathbf{a}\| _0\| \mathbf{a}\|_{\infty}(\max_{i\in [4m]}\| \mathbf{w}_i\| _2B_x + \tilde{b}) + 2$  . By Hoeffding's inequality, for all  $z > 0$  , we have
$$
\Pr \left(\left| \widetilde {\mathcal {L}} _ {\mathcal {Z}} (f _ {\Xi}) - \mathcal {L} _ {\mathcal {D}} (f _ {\Xi}) \right| \geq z\right) \leq 2 \exp \left(- \frac {z ^ {2} n}{(\| \mathbf {a} \| _ {0} \| \mathbf {a} \| _ {\infty} (\max _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} \| _ {2} B _ {x} + \tilde {b}) + 1) ^ {2}}\right).
$$
Thus, with probability at least  $1 - O\left(\frac{1}{n}\right)$ , we have
$$
\left| \widetilde {\mathcal {L}} _ {\mathcal {Z}} \left(f _ {\Xi}\right) - \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi}\right) \right| \leq O \left(\frac {\left(\| \mathbf {a} \| _ {0} \| \mathbf {a} \| _ {\infty} \left(\max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} \| _ {2} B _ {x} + \tilde {b}\right) + 1\right) \sqrt {\log n}}{n ^ {\frac {1}{2}}}\right). \tag {143}
$$
The gradients allow for obtaining a set of neurons approximating the "ground-truth" network with comparable loss.
Lemma D.13 (Existence of Good Networks under Empirical Risk). Suppose  $\frac{n}{\log n} > \Omega \left(\frac{B_x^2}{B_{x2}} + \frac{1}{p} + \frac{B_{x2}}{B_G^2|\ell'(0)|^2}\right)$ . Let  $\lambda^{(1)} = \frac{1}{\eta^{(1)}}$ . For any  $B_{\epsilon} \in (0, B_b)$ , let  $\sigma_a = \Theta \left(\frac{\tilde{b}}{-|\ell'(0)|\eta^{(1)}B_GB_\epsilon}\right)$  and  $\delta = 2re^{-\sqrt{\frac{mp}{2}}}$ . Then, with probability at least  $1 - \delta$  over the initialization and training samples, there exists  $\tilde{\mathbf{a}}_i$ 's such that  $f_{(\tilde{\mathbf{a}},\mathbf{W}^{(1)},\mathbf{b})}(\mathbf{x}) = \sum_{i=1}^{4m} \tilde{\mathbf{a}}_i \sigma\left(\left\langle \mathbf{w}_i^{(1)}, \mathbf{x} \right\rangle - \mathbf{b}_i\right)$  satisfies
$$
\mathcal {L} _ {\mathcal {D}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) \tag {144}
$$
$$
\leq r B _ {a 1} \left(\frac {2 B _ {x 1} ^ {2} B _ {b}}{\sqrt {m p} B _ {G} B _ {\epsilon}} + B _ {x 1} \sqrt {2 \gamma + O \left(\frac {\sqrt {B _ {x 2} \log n}}{B _ {G} \left| \ell^ {\prime} (0) \right| n ^ {\frac {1}{2}}}\right)} + B _ {\epsilon}\right) + \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}}, \tag {145}
$$
and  $\| \tilde{\mathbf{a}} \|_0 = O\left(r(mp)^{\frac{1}{2}}\right)$ ,  $\| \tilde{\mathbf{a}} \|_2 = O\left(\frac{B_{a2}B_b}{\tilde{b}(mp)^{\frac{1}{4}}}\right)$ ,  $\| \tilde{\mathbf{a}} \|_\infty = O\left(\frac{B_{a1}B_b}{\tilde{b}(mp)^{\frac{1}{2}}}\right)$ .
Proof of Lemma D.13. Denote  $\rho = O\left(\frac{1}{n}\right)$  and  $\beta = O\left(\frac{\sqrt{B_{x2}\log n}}{n^{\frac{1}{2}}}\right)$ . Note that by symmetric initialization, we have  $\ell'(yf_{\Xi^{(0)}}(\mathbf{x})) = |\ell'(0)|$  for any  $\mathbf{x} \in \mathcal{X}$ , so that, by Lemma D.12, we have  $\left\| \widetilde{G}(\mathbf{w}_i^{(0)}, \mathbf{b}_i) - G(\mathbf{w}_i^{(0)}, \mathbf{b}_i) \right\|_2 \leq \frac{\beta}{|\ell'(0)|}$  with probability at least  $1 - \rho$ . Thus, by union bound, we can see that  $S_{p,\gamma,B_G} \subseteq \widetilde{S}_{p-\rho,\gamma+\frac{\beta}{BG|\ell'(0)|}, B_G-\frac{\beta}{|\ell'(0)|}}$ . Consequently, we have  $\mathrm{OPT}_{d,r,B_F,\widetilde{S}_{p-\rho,\gamma+\frac{\beta}{BG|\ell'(0)|}, B_G-\frac{\beta}{|\ell'(0)|}}} \leq \mathrm{OPT}_{d,r,B_F,S_{p,\gamma,B_G}}$ . Exactly follow the proof in Lemma D.4 by replacing  $S_{p,\gamma,B_G}$  to  $\widetilde{S}_{p-\rho,\gamma+\frac{\beta}{BG|\ell'(0)|}, B_G-\frac{\beta}{|\ell'(0)|}}$ . Then, we finish the proof by  $\rho \leq \frac{p}{2}, \frac{\beta}{|\ell'(0)|} \leq (1 - 1/\sqrt{2})B_G$ .
We will use Theorem D.5 to prove that gradient descent learns a good classifier (Theorem D.17). Theorem 3.12 is simply a direct corollary of Theorem D.17. To apply the theorem we first present a few lemmas bounding the change in the network during steps.
Lemma D.14 (Bound of  $\Xi^{(0)},\Xi^{(1)}$  under Empirical Risk). Assume the same conditions as in Lemma D.13, and  $d\geq \log m$  , with probability at least  $1 - \delta -\frac{1}{m^2} -O\left(\frac{m}{n}\right)$  over the initialization and training samples,  $\| \mathbf{a}^{(0)}\|_{\infty} = O\left(\frac{\tilde{b}\sqrt{\log m}}{|\ell'(0)|\eta^{(1)}B_{G}B_{\epsilon}}\right)$  , and for all  $i\in [4m]$  , we have  $\| \mathbf{w}_i^{(0)}\| _2 = O\left(\sigma_w\sqrt{d}\right)$  . Finally,  $\| \mathbf{a}^{(1)}\|_{\infty} = O\left(\eta^{(1)}|\ell '(0)|(B_{x1}\sigma_w\sqrt{d} +\tilde{b}) + \eta^{(1)}\frac{\sigma_w\sqrt{dB_{x2}\log n}}{n^{\frac{1}{2}}}\right)$  , and for all  $i\in [4m]$  ,  $\| \mathbf{w}_i^{(1)}\| _2 = O\left(\frac{\tilde{b}\sqrt{\log m}B_{x1}}{B_GB_\epsilon} +\frac{\tilde{b}\sqrt{\log m}B_{x2}\log n}{|\ell'(0)|B_GB_\epsilon n^{\frac{1}{2}}}\right)$
Proof of Lemma D.14. The proof exactly follows the proof of Lemma D.6 with Lemma D.12.  $\square$
Lemma D.15 (Bound of  $\Xi^{(t)}$  under Empirical Risk). Assume the same conditions as in Lemma D.14, and let  $\eta = \eta^{(t)}$  for all  $t\in \{2,3,\ldots ,T\}$ ,  $0 < T\eta B_{x1}\leq o(1)$ , and  $0 = \lambda = \lambda^{(t)}$  for all  $t\in \{2,3,\dots ,T\}$ . With probability at least  $1 - O\left(\frac{Tm}{n}\right)$  over training samples, for all  $i\in [4m]$ , for all  $t\in \{2,3,\dots ,T\}$ , we have
$$
\left| \mathbf {a} _ {i} ^ {(t)} \right| \leq O \left(\left| \mathbf {a} _ {i} ^ {(1)} \right| + \left\| \mathbf {w} _ {i} ^ {(1)} \right\| _ {2} + \frac {\tilde {b}}{\left(B _ {x 1} + \frac {\sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right)} + \eta \tilde {b}\right) \tag {146}
$$
$$
\begin{array}{l} \left\| \mathbf {w} _ {i} ^ {(t)} - \mathbf {w} _ {i} ^ {(1)} \right\| _ {2} \leq O \left(t \eta \left(B _ {x 1} + \frac {\sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right) \left| \mathbf {a} _ {i} ^ {(1)} \right| + t \eta^ {2} \left(B _ {x 1} + \frac {\sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right) ^ {2} \left\| \mathbf {w} _ {i} ^ {(1)} \right\| _ {2} \right. \\ + t \eta^ {2} \left(B _ {x 1} + \frac {\sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right) \tilde {b}). \tag {147} \\ \end{array}
$$
Proof of Lemma D.15. The proof exactly follows the proof of Lemma D.7 with Lemma D.12. Note that, we have
$$
\begin{array}{l} \left| \mathbf {a} _ {i} ^ {(t)} \right| \leq \left| \mathbf {a} _ {i} ^ {(t - 1)} \right| + \eta \left(B _ {x 1} \| \mathbf {w} _ {i} ^ {(t - 1)} \| _ {2} + \tilde {b}\right) + \eta \frac {\left\| \mathbf {w} _ {i} ^ {(t - 1)} \right\| _ {2} \sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}} (148) \\ \leq \left| \mathbf {a} _ {i} ^ {(t - 1)} \right| + \eta \left(B _ {x 1} + \frac {\sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right) \| \mathbf {w} _ {i} ^ {(t - 1)} - \mathbf {w} _ {i} ^ {(1)} \| _ {2} + \eta Z _ {i}, (149) \\ \end{array}
$$
where we denote  $Z_{i} = \left(B_{x1} + \frac{\sqrt{B_{x2}\log n}}{n^{\frac{1}{2}}}\right)\| \mathbf{w}_{i}^{(1)}\|_{2} + \tilde{b}$ . Similarly, we have
$$
\left\| \mathbf {w} _ {i} ^ {(t)} - \mathbf {w} _ {i} ^ {(1)} \right\| _ {2} \leq \left\| \mathbf {w} _ {i} ^ {(t - 1)} - \mathbf {w} _ {i} ^ {(1)} \right\| _ {2} + \eta \left(B _ {x 1} + \frac {\sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right) | \mathbf {a} _ {i} ^ {(t - 1)} |. \tag {150}
$$
We finish the proof by following the same arguments in the proof of Lemma D.7 and union bound.
Lemma D.16 (Bound of Loss Gap and Gradient under Empirical Risk). Assume the same conditions as in Lemma D.15. With probability at least  $1 - O\left(\frac{T}{n}\right)$ , for all  $t \in [T]$ , we have
$$
\begin{array}{l} \left| \widetilde {\mathcal {L}} _ {\mathcal {Z} ^ {(t)}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(t)}, \mathbf {b})}\right) - \mathcal {L} _ {\mathcal {D}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) \right| (151) \\ \leq O \left(\frac {\left(\| \tilde {\mathbf {a}} \| _ {0} \| \tilde {\mathbf {a}} \| _ {\infty} \left(\max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(t)} \| _ {2} B _ {x} + \tilde {b}\right) + 1\right) \sqrt {\log n}}{n ^ {\frac {1}{2}}}\right) (152) \\ + B _ {x 1} \| \tilde {\mathbf {a}} \| _ {2} \sqrt {\| \tilde {\mathbf {a}} \| _ {0}} \max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(t)} - \mathbf {w} _ {i} ^ {(1)} \| _ {2}. (153) \\ \end{array}
$$
With probability at least  $1 - O\left(\frac{T}{n}\right)$ , for all  $t \in [T]$ ,  $i \in [4m]$  we have
$$
\left| \frac {\partial \widetilde {\mathcal {L}} _ {\mathcal {Z} ^ {(t)}} \left(f _ {\Xi^ {(t)}}\right)}{\partial \mathbf {a} _ {i} ^ {(t)}} \right| \leq B _ {x 1} \left(\left\| \mathbf {w} _ {i} ^ {(t)} - \mathbf {w} _ {i} ^ {(1)} \right\| _ {2} + \left\| \mathbf {w} _ {i} ^ {(1)} \right\| _ {2}\right) + \tilde {b} + O \left(\frac {\left\| \mathbf {w} _ {i} ^ {(t)} \right\| _ {2} \sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right). \tag {154}
$$
Proof of Lemma D.16. By Lemma D.8 and Lemma D.12, with probability at least  $1 - O\left(\frac{T}{n}\right)$ , for all  $t \in [T]$ , we have
$$
\begin{array}{l} \left| \widetilde {\mathcal {L}} _ {\mathcal {Z} ^ {(t)}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(t)}, \mathbf {b})}\right) - \mathcal {L} _ {\mathcal {D}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) \right| (155) \\ \leq \left| \widetilde {\mathcal {L}} _ {\mathcal {Z} ^ {(t)}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(t)}, \mathbf {b})}\right) - \mathcal {L} _ {\mathcal {D}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(t)}, \mathbf {b})}\right) \right| + \left| \mathcal {L} _ {\mathcal {D}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(t)}, \mathbf {b})}\right) - \mathcal {L} _ {\mathcal {D}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) \right| (156) \\ \leq O \left(\frac {\left(\| \tilde {\mathbf {a}} \| _ {0} \| \tilde {\mathbf {a}} \| _ {\infty} \left(\max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(t)} \| _ {2} B _ {x} + \tilde {b}\right) + 1\right) \sqrt {\log n}}{n ^ {\frac {1}{2}}}\right) (157) \\ + B _ {x 1} \| \tilde {\mathbf {a}} \| _ {2} \sqrt {\| \tilde {\mathbf {a}} \| _ {0}} \max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(t)} - \mathbf {w} _ {i} ^ {(1)} \| _ {2}. (158) \\ \end{array}
$$
By Lemma D.8 and Lemma D.12, with probability at least  $1 - O\left(\frac{T}{n}\right)$ , for all  $t \in [T]$ ,  $i \in [4m]$  we have
$$
\left| \frac {\partial \widetilde {\mathcal {L}} _ {\mathcal {Z} ^ {(t)}} \left(f _ {\Xi^ {(t)}}\right)}{\partial \mathbf {a} _ {i} ^ {(t)}} \right| \leq B _ {x 1} \left(\left\| \mathbf {w} _ {i} ^ {(t)} - \mathbf {w} _ {i} ^ {(1)} \right\| _ {2} + \left\| \mathbf {w} _ {i} ^ {(1)} \right\| _ {2}\right) + \tilde {b} + O \left(\frac {\left\| \mathbf {w} _ {i} ^ {(t)} \right\| _ {2} \sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right). \tag {159}
$$
Theorem D.17 (Online Convex Optimization under Empirical Risk. Full Statement of Theorem 3.12). Consider training by Algorithm 1, and any  $\delta \in (0,1)$ . Assume  $d\geq \log m$ . Set
$$
\sigma_ {w} > 0, \quad \tilde {b} > 0, \quad \eta^ {(t)} = \eta , \lambda^ {(t)} = 0 \text {f o r a l l} t \in \{2, 3, \dots , T \},
$$
$$
\eta^ {(1)} = \Theta \left(\frac {\operatorname* {m i n} \{O (\eta) , O (\tilde {\eta b}) \}}{- \ell^ {\prime} (0) (B _ {x 1} \sigma_ {w} \sqrt {d} + \tilde {b})}\right), \lambda^ {(1)} = \frac {1}{\eta^ {(1)}}, \sigma_ {a} = \Theta \left(\frac {\tilde {b} (m p) ^ {\frac {1}{4}}}{- \ell^ {\prime} (0) \eta^ {(1)} B _ {x 1} \sqrt {B _ {G} B _ {b}}}\right).
$$
Let  $0 < T\eta B_{x1} \leq o(1)$ ,  $m = \Omega \left(\frac{1}{\sqrt{\delta}} + \frac{1}{p} \left( \log \left( \frac{r}{\delta} \right) \right)^2 \right)$  and  $\frac{n}{\log n} > \Omega \left( \frac{B_x^2}{B_{x2}} + \frac{1}{p} + \left( \frac{1}{B_G^2} + \frac{1}{B_{x1}^2} \right) \frac{B_{x2}}{|l'(0)|^2} + \frac{Tm}{\delta} \right)$ . With probability at least  $1 - \delta$  over the initialization and training samples, there exists  $t \in [T]$  such that
$$
\begin{array}{l} \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi^ {(t)}}\right) (160) \\ \leq \operatorname {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}} + r B _ {a 1} \left(\frac {2 \sqrt {2} B _ {x 1}}{(m p) ^ {\frac {1}{4}}} \sqrt {\frac {B _ {b}}{B _ {G}}} + B _ {x 1} \sqrt {2 \gamma + O \left(\frac {\sqrt {B _ {x 2} \log n}}{B _ {G} \left| \ell^ {\prime} (0) \right| n ^ {\frac {1}{2}}}\right)}\right) (161) \\ + \eta \left(\sqrt {r} B _ {a 2} B _ {b} T \eta B _ {x 1} ^ {2} + m \tilde {b}\right) O \left(\frac {\sqrt {\log m} B _ {x 1} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + 1\right) + O \left(\frac {B _ {a 2} ^ {2} B _ {b} ^ {2}}{\eta T \tilde {b} ^ {2} (m p) ^ {\frac {1}{2}}}\right) (162) \\ + \frac {\sqrt {\log n}}{n ^ {\frac {1}{2}}} O \left(\left(\frac {r B _ {a 1} B _ {b}}{\tilde {b}} + m \left(\frac {\tilde {b} \sqrt {\log m} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + \frac {\tilde {b}}{B _ {x 1}}\right)\right) \right. (163) \\ \left. \cdot \left(\left(\frac {\tilde {b} \sqrt {\log m} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + T \eta^ {2} B _ {x 1} \tilde {b}\right) B _ {x} + \tilde {b}\right) + 2\right) (164) \\ + \frac {\sqrt {\log n}}{n ^ {\frac {1}{2}}} O \left(m \eta \left(\frac {\tilde {b} \sqrt {\log m} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + T \eta^ {2} B _ {x 1} \tilde {b}\right) \sqrt {B _ {x 2}}\right). (165) \\ \end{array}
$$
$$
\tilde {b} = \Theta \left(\frac {B _ {G} ^ {\frac {1}{4}} B _ {a 2} B _ {b} ^ {\frac {3}{4}}}{\sqrt {r B _ {a 1}}}\right), \quad m = \Omega \left(\frac {1}{p \epsilon^ {4}} \left(r B _ {a 1} B _ {x 1} \sqrt {\frac {B _ {b}}{B _ {G}}}\right) ^ {4} + \frac {1}{\sqrt {\delta}} + \frac {1}{p} \left(\log \left(\frac {r}{\delta}\right)\right) ^ {2}\right),
$$
$$
\eta = \Theta \left(\frac {\epsilon}{\left(\frac {\sqrt {r} B _ {a 2} B _ {b} B _ {x 1}}{(m p) ^ {\frac {1}{4}}} + m \tilde {b}\right) \left(\frac {\sqrt {\log m} B _ {x 1} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + 1\right)}\right), T = \Theta \left(\frac {1}{\eta B _ {x 1} (m p) ^ {\frac {1}{4}}}\right),
$$
$$
\frac {n}{\log n} = \Omega \left(\frac {m ^ {3} p B _ {x} ^ {2} B _ {a 2} ^ {4} B _ {b} (\log m) ^ {2}}{\epsilon^ {2} r ^ {2} B _ {a 1} ^ {2} B _ {G}} + \frac {(m p) ^ {\frac {1}{2}} B _ {x 2} \log m}{B _ {b} B _ {G}} + \frac {B _ {x} ^ {2}}{B _ {x 2}} + \frac {1}{p} + \left(\frac {1}{B _ {G} ^ {2}} + \frac {1}{B _ {x 1} ^ {2}}\right) \frac {B _ {x 2}}{| \ell^ {\prime} (0) | ^ {2}} + \frac {T m}{\delta}\right),
$$
$$
\begin{array}{l} \Pr \left[ \operatorname {s i g n} \left(f _ {\Xi^ {(t)}}\right) (\mathbf {x}) \neq y \right] \leq \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi^ {(t)}}\right) (166) \\ \leq \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}} + r B _ {a 1} B _ {x 1} \sqrt {2 \gamma + O \left(\frac {\sqrt {B _ {x 2} \log n}}{B _ {G} | \ell^ {\prime} (0) | n ^ {\frac {1}{2}}}\right)} + \epsilon . (167) \\ \end{array}
$$
Proof of Theorem D.17. We follow the proof in Theorem D.9. By  $m = \Omega\left(\frac{1}{\sqrt{\delta}} + \frac{1}{p}\left(\log \left(\frac{r}{\delta}\right)\right)^2\right)$  and  $\frac{n}{\log n} > \Omega\left(\frac{B_x^2}{B_{x2}} + \frac{1}{p} + \frac{B_{x2}}{B_G^2|\ell'(0)|^2} + \frac{Tm}{\delta}\right)$ , we have  $2re^{-\sqrt{\frac{mp}{2}}} + \frac{1}{m^2} + O\left(\frac{Tm}{n}\right) \leq \delta$ . For any  $B_\epsilon \in (0, B_b)$ , when  $\sigma_a = \Theta\left(\frac{\tilde{b}}{-\ell'(0)\eta^{(1)}B_GB_\epsilon}\right)$ , by Theorem D.5, Lemma D.12, Lemma D.13,
$$
\begin{array}{l} \frac {1}{T} \sum_ {t = 1} ^ {T} \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi^ {(t)}}\right) (168) \\ \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \left| \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi^ {(t)}}\right) - \widetilde {\mathcal {L}} _ {\mathcal {Z} ^ {(t)}} \left(f _ {\Xi^ {(t)}}\right) \right| + \frac {1}{T} \sum_ {t = 1} ^ {T} \widetilde {\mathcal {L}} _ {\mathcal {Z} ^ {(t)}} \left(f _ {\Xi^ {(t)}}\right) (169) \\ \leq \frac {1}{T} \sum_ {t = 1} ^ {T} \left| \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi^ {(t)}}\right) - \widetilde {\mathcal {L}} _ {\mathcal {Z} ^ {(t)}} \left(f _ {\Xi^ {(t)}}\right) \right| + \frac {1}{T} \sum_ {t = 1} ^ {T} \left| \widetilde {\mathcal {L}} _ {\mathcal {Z} ^ {(t)}} \left(f _ {(\bar {\mathbf {a}}, \mathbf {W} ^ {(t)}, \mathbf {b})}\right) - \mathcal {L} _ {\mathcal {D}} \left(f _ {(\bar {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) \right| (170) \\ + \mathcal {L} _ {\mathcal {D}} \left(f _ {(\tilde {\mathbf {a}}, \mathbf {W} ^ {(1)}, \mathbf {b})}\right) + \frac {\| \tilde {\mathbf {a}} \| _ {2} ^ {2}}{2 \eta T} + \left(2 \| \mathbf {a} ^ {(1)} \| _ {2} \sqrt {m} + 4 \eta m\right) \max  _ {t \in [ T ], i \in [ 4 m ]} \left| \frac {\partial \widetilde {\mathcal {L}} _ {\mathcal {Z} ^ {(t)}} \left(f _ {\Xi^ {(t)}}\right)}{\partial \mathbf {a} _ {i} ^ {(t)}} \right| (171) \\ \leq B _ {x 1} \| \tilde {\mathbf {a}} \| _ {2} \sqrt {\| \tilde {\mathbf {a}} \| _ {0}} \max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(T)} - \mathbf {w} _ {i} ^ {(1)} \| _ {2} (172) \\ + O \left(\frac {\left(\| \tilde {\mathbf {a}} \| _ {0} \| \tilde {\mathbf {a}} \| _ {\infty} + m \| \mathbf {a} ^ {(T)} \| _ {\infty}\right) \left(\max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(T)} \| _ {2} B _ {x} + \tilde {b}\right) + 2}{n ^ {\frac {1}{2}}}\right) (173) \\ + \mathrm {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}} + r B _ {a 1} \left(\frac {2 B _ {x 1} ^ {2} B _ {b}}{\sqrt {m p} B _ {G} B _ {\epsilon}} + B _ {x 1} \sqrt {2 \gamma + O \left(\frac {\sqrt {B _ {x 2} \log n}}{B _ {G} | \ell^ {\prime} (0) | n ^ {\frac {1}{2}}}\right)} + B _ {\epsilon}\right) (174) \\ + \frac {\| \tilde {\mathbf {a}} \| _ {2} ^ {2}}{2 \eta T} + 4 m B _ {x 1} \left(\| \mathbf {a} ^ {(1)} \| _ {\infty} + \eta\right) (175) \\ \cdot \left(\max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(T)} - \mathbf {w} _ {i} ^ {(1)} \| _ {2} + \max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(1)} \| _ {2} + \frac {\tilde {b}}{B _ {x 1}} + O \left(\frac {\max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(T)} \| _ {2} \sqrt {B _ {x 2} \log n}}{B _ {x 1} n ^ {\frac {1}{2}}}\right)\right). \\ \end{array}
$$
Set  $B_{\epsilon} = \frac{B_{x1}}{(mp)^{\frac{1}{4}}}\sqrt{\frac{2B_b}{B_G}}$ , we have  $\sigma_{a} = \Theta \left(\frac{\tilde{b}(mp)^{\frac{1}{4}}}{-\ell'(0)\eta^{(1)}B_{x1}\sqrt{B_{G}B_{b}}}\right)$  which satisfy the requirements. By Lemma D.13, Lemma D.14, Lemma D.15,  $\frac{n}{\log n} > \Omega\left(\frac{B_{x2}}{B_{x1}^2|\ell'(0)|^2}\right)$ , when  $\eta^{(1)} = \Theta\left(\frac{\min\{O(\eta),O(\eta\tilde{b})\}}{-\ell'(0)(B_{x1}\sigma_w\sqrt{d} + \tilde{b})}\right)$ , we have
$$
\begin{array}{l} \left\| \tilde {\mathbf {a}} \right\| _ {0} = O \left(r (m p) ^ {\frac {1}{2}}\right), \quad \left\| \tilde {\mathbf {a}} \right\| _ {2} = O \left(\frac {B _ {a 2} B _ {b}}{\tilde {b} (m p) ^ {\frac {1}{4}}}\right), \quad \left\| \tilde {\mathbf {a}} \right\| _ {\infty} = O \left(\frac {B _ {a 1} B _ {b}}{\tilde {b} (m p) ^ {\frac {1}{2}}}\right) (176) \\ \left\| \mathbf {a} ^ {(1)} \right\| _ {\infty} = O \left(\eta^ {(1)} \left| \ell^ {\prime} (0) \right| \left(B _ {x 1} \sigma_ {w} \sqrt {d} + \tilde {b}\right) + \eta^ {(1)} \frac {\sigma_ {w} \sqrt {d B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right) (177) \\ = \min  \left\{O (\eta), O (\eta \tilde {b}) \right\} (178) \\ \end{array}
$$
$$
\begin{array}{l} \left\| \mathbf {a} ^ {(T)} \right\| _ {\infty} \leq O \left(\left\| \mathbf {a} ^ {(1)} \right\| _ {\infty} + \max  _ {i \in [ 4 m ]} \left\| \mathbf {w} _ {i} ^ {(1)} \right\| _ {2} + \frac {\tilde {b}}{\left(B _ {x 1} + \frac {\sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right)} + \eta \tilde {b}\right) (179) \\ \leq O \left(\max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(1)} \| _ {2} + \frac {\tilde {b}}{B _ {x 1}}\right) (180) \\ \end{array}
$$
Lemma D.16, with probability at least  $1 - \delta$  over the initialization and training samples, we have
$$
\max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(1)} \| _ {2} = O \left(\frac {\tilde {b} \sqrt {\log m} B _ {x 1}}{B _ {G} B _ {\epsilon}} + \frac {\tilde {b} \sqrt {\log m} \sqrt {B _ {x 2} \log n}}{\left| \ell^ {\prime} (0) \right| B _ {G} B _ {\epsilon} n ^ {\frac {1}{2}}}\right) \tag {181}
$$
$$
= O \left(\frac {\tilde {b} \sqrt {\log m} B _ {x 1}}{B _ {G} B _ {\epsilon}}\right) \tag {182}
$$
$$
= O \left(\frac {\tilde {b} \sqrt {\log m} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}}\right) \tag {183}
$$
$$
\max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(T)} - \mathbf {w} _ {i} ^ {(1)} \| _ {2} = O \left(T \eta \left(B _ {x 1} + \frac {\sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right) | \mathbf {a} _ {i} ^ {(1)} |\right. \tag {184}
$$
$$
+ T \eta^ {2} \left(B _ {x 1} + \frac {\sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right) ^ {2} \| \mathbf {w} _ {i} ^ {(1)} \| _ {2} \tag {185}
$$
$$
+ T \eta^ {2} \left(B _ {x 1} + \frac {\sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right) \tilde {b}) \tag {186}
$$
$$
= O \left(T \eta^ {2} B _ {x 1} ^ {2} \left(\max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(1)} \| _ {2} + \frac {\tilde {b}}{B _ {x 1}}\right)\right). \tag {187}
$$
Then, following the proof in Theorem D.9, we have
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \mathcal {L} _ {\mathcal {D}} \left(f _ {\Xi^ {(t)}}\right) \tag {188}
$$
$$
\leq \operatorname {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}} + r B _ {a 1} \left(\frac {2 \sqrt {2} B _ {x 1}}{(m p) ^ {\frac {1}{4}}} \sqrt {\frac {B _ {b}}{B _ {G}}} + B _ {x 1} \sqrt {2 \gamma + O \left(\frac {\sqrt {B _ {x 2} \log n}}{B _ {G} \left| \ell^ {\prime} (0) \right| n ^ {\frac {1}{2}}}\right)}\right) \tag {189}
$$
$$
+ \eta \left(\sqrt {r} B _ {a 2} B _ {b} T \eta B _ {x 1} ^ {2} + m \tilde {b}\right) O \left(\frac {\sqrt {\log m} B _ {x 1} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + 1\right) + O \left(\frac {B _ {a 2} ^ {2} B _ {b} ^ {2}}{\eta T \tilde {b} ^ {2} (m p) ^ {\frac {1}{2}}}\right) \tag {190}
$$
$$
+ O \left(\frac {\left(\left(\| \tilde {\mathbf {a}} \| _ {0} \| \tilde {\mathbf {a}} \| _ {\infty} + m \| \mathbf {a} ^ {(T)} \| _ {\infty}\right) \left(\max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(T)} \| _ {2} B _ {x} + \tilde {b}\right) + 2\right) \sqrt {\log n}}{n ^ {\frac {1}{2}}}\right) \tag {191}
$$
$$
+ O \left(\frac {m \eta \max  _ {i \in [ 4 m ]} \| \mathbf {w} _ {i} ^ {(T)} \| _ {2} \sqrt {B _ {x 2} \log n}}{n ^ {\frac {1}{2}}}\right) \tag {192}
$$
$$
\leq \operatorname {O P T} _ {d, r, B _ {F}, S _ {p, \gamma , B _ {G}}} + r B _ {a 1} \left(\frac {2 \sqrt {2} B _ {x 1}}{(m p) ^ {\frac {1}{4}}} \sqrt {\frac {B _ {b}}{B _ {G}}} + B _ {x 1} \sqrt {2 \gamma + O \left(\frac {\sqrt {B _ {x 2} \log n}}{B _ {G} \left| \ell^ {\prime} (0) \right| n ^ {\frac {1}{2}}}\right)}\right) \tag {193}
$$
$$
+ \eta \left(\sqrt {r} B _ {a 2} B _ {b} T \eta B _ {x 1} ^ {2} + m \tilde {b}\right) O \left(\frac {\sqrt {\log m} B _ {x 1} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + 1\right) + O \left(\frac {B _ {a 2} ^ {2} B _ {b} ^ {2}}{\eta T \tilde {b} ^ {2} (m p) ^ {\frac {1}{2}}}\right) \tag {194}
$$
$$
+ \frac {\sqrt {\log n}}{n ^ {\frac {1}{2}}} O \left(\left(\frac {r B _ {a 1} B _ {b}}{\tilde {b}} + m \left(\frac {\tilde {b} \sqrt {\log m} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + \frac {\tilde {b}}{B _ {x 1}}\right)\right) \right. \tag {195}
$$
$$
\left. \cdot \left(\left(\frac {\tilde {b} \sqrt {\log m} (m p) ^ {\frac {1}{4}}}{\sqrt {B _ {b} B _ {G}}} + T \eta^ {2} B _ {x 1} \tilde {b}\right) B _ {x} + \tilde {b}\right) + 2\right) \tag {196}
$$