SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {\mathcal {R}} \left(\boldsymbol {a} ^ {T ^ {\prime}}\right) \leq \hat {\mathcal {R}} ^ {*} + \varepsilon \quad \text {a n d} \quad \| \boldsymbol {a} ^ {T ^ {\prime}} \| \leq 2 \| \boldsymbol {a} ^ {*} \| \lesssim \tilde {\mathcal {O}} (1 / \sqrt {m}).
$$
$$
\hat {\mathcal {R}} \left(\boldsymbol {a} ^ {T ^ {\prime}}\right) \lesssim \mathbb {E} \left[ \epsilon^ {2} \right] + \tilde {\mathcal {O}} (1 / m + \varepsilon).
$$
$$
\hat {\mathcal {R}} (\pmb {a} ^ {T ^ {\prime}}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \Bigl (\hat {y} (\widetilde {\pmb {z}} ^ {(i)}; \overline {{\pmb {w}}}) - y ^ {(i)} \Bigr) ^ {2},
$$
is the final training error which we also denoted by  $\hat{\mathcal{R}} (\overline{w})$  earlier in this section when were not focusing on the second layer. From the previous two steps, we know how to bound  $\mathcal{R}(\overline{w})$  via  $\hat{R} (\overline{u})$ . Thus the last step is to upper bound  $\hat{R} (\overline{u})$  via  $\hat{\mathcal{R}} (\overline{w})$ . To that end, via Jensen's inequality
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\hat {y} \left(\boldsymbol {z} ^ {(i)}; \overline {{\boldsymbol {u}}}\right) - y ^ {(i)}\right) ^ {2} \leq \frac {3}{n} \sum_ {i = 1} ^ {n} \left(\hat {y} \left(\tilde {\boldsymbol {z}} ^ {(i)}; \overline {{\boldsymbol {w}}}\right) - y ^ {(i)}\right) ^ {2} \\ + \frac {3}{n} \sum_ {i = 1} ^ {n} \left(\hat {y} \left(\boldsymbol {z} ^ {(i)}; \overline {{\boldsymbol {w}}}\right) - \hat {y} \left(\tilde {\boldsymbol {z}} ^ {(i)}; \overline {{\boldsymbol {w}}}\right)\right) ^ {2} \\ + \frac {3}{n} \sum_ {i = 1} ^ {n} \left(\hat {y} \left(\boldsymbol {z} ^ {(i)}; \overline {{\boldsymbol {w}}}\right) - \hat {y} \left(\boldsymbol {z} ^ {(i)}; \overline {{\boldsymbol {u}}}\right)\right) ^ {2}. \\ \end{array}
$$
The first term on the RHS is  $\hat{\mathcal{R}} (\overline{w})$  for which we developed a bound earlier in this step. Bounding the latter two terms can be performed similarly to the arguments in the previous sections. In particular,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \Big (\hat {y} (\boldsymbol {z} ^ {(i)}; \overline {{\boldsymbol {w}}}) - \hat {y} (\tilde {\boldsymbol {z}} ^ {(i)}; \overline {{\boldsymbol {w}}}) \Big) ^ {2} \leq r _ {a} ^ {2} \| \mathbf {I} _ {d} - \hat {\boldsymbol {\Sigma}} ^ {- 1 / 2} \boldsymbol {\Sigma} ^ {1 / 2} \| ^ {2} \| \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {z} ^ {(i)} \boldsymbol {z} ^ {(i)} ^ {\top} \| ^ {2} \leq \tilde {\mathcal {O}} (d / n ^ {\prime}),
$$
where the last inequality holds with probability at least  $1 - 2e^{-d}$  (over the event of Lemma 24). Similarly,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \left(\hat {y} (\boldsymbol {z} ^ {(i)}; \overline {{\boldsymbol {w}}}) - \hat {y} (\boldsymbol {z} ^ {(i)}; \overline {{\boldsymbol {u}}})\right) ^ {2} \leq r _ {a} \| \overline {{\boldsymbol {w}}} - \overline {{\boldsymbol {u}}} \| ^ {2} \leq \tilde {\mathcal {O}} (\varepsilon).
$$
Putting the bounds back together (recall  $n' \geq n \gtrsim d\varepsilon^{-2}$ ), we arrive at
$$
\hat {R} (\overline {{\boldsymbol {u}}}) \lesssim \hat {\mathcal {R}} (\overline {{\boldsymbol {w}}}) + \tilde {\mathcal {O}} (\varepsilon + d / n ^ {\prime}) \lesssim \hat {\mathcal {R}} (\overline {{\boldsymbol {w}}}) + \tilde {\mathcal {O}} (\varepsilon).
$$
Combining the result of this step with the two previous steps implies
$$
\mathcal {R} (\overline {{\boldsymbol {w}}}) \lesssim \mathbb {E} \big [ \epsilon^ {2} \big ] + \tilde {\mathcal {O}} (1 / m + \varepsilon),
$$
with probability at least  $1 - \mathcal{O}(d^{-q})$  (when conditioned on  $\langle \overline{\boldsymbol{w}}^0,\overline{\boldsymbol{u}}\rangle >0)$  which completes the proof of Theorem 9.
F Auxiliary Lemmas
In this section, we recall a number of standard lemmas which we employ in various parts of our proofs.
Lemma 23. [Wai19, Theorem 6.1]. Suppose  $\{\pmb{x}^{(i)}\}_{i=1}^{n'} \stackrel{i.i.d.}{\sim} \mathcal{N}(0, \Sigma)$ . Let  $\hat{\Sigma} \coloneqq \frac{1}{n'} \sum_{i=1}^{n'} \pmb{x}^{(i)} \pmb{x}^{(i)}^\top$ . Then, for  $n' \geq \operatorname{tr}(\pmb{\Sigma}) / \lambda_{\max}(\pmb{\Sigma})$ ,
$$
\lambda_ {\max } \left(\hat {\boldsymbol {\Sigma}}\right) \leq \lambda_ {\max } (\boldsymbol {\Sigma}) \left(4 + 5 \sqrt {\frac {\operatorname {t r} (\boldsymbol {\Sigma})}{n ^ {\prime} \lambda_ {\max } (\boldsymbol {\Sigma})}}\right)
$$
with probability at least  $1 - e^{-n' / 2}$ . Furthermore, for  $n' \geq d$ ,
$$
\lambda_ {\min } \left(\hat {\boldsymbol {\Sigma}}\right) \geq \lambda_ {\min } (\boldsymbol {\Sigma}) \left(\frac {1}{4} - \sqrt {\frac {\operatorname {t r} (\boldsymbol {\Sigma})}{n \lambda_ {\min } (\boldsymbol {\Sigma})}}\right)
$$
with probability at least  $1 - e^{-n'/8}$ .
Lemma 24. Suppose  $\left\{\pmb{z}^{(i)}\right\}_{i = 1}^{n'} \stackrel{i.i.d.}{\sim} \mathcal{N}(0, \mathbf{I}_d)$ , let  $\pmb{x}^{(i)} \coloneqq \pmb{\Sigma}^{1/2}\pmb{z}^{(i)}$  for some invertible  $\pmb{\Sigma}$ , and define
$$
\hat {\boldsymbol {\Sigma}} := \frac {1}{n ^ {\prime}} \sum_ {i = 1} ^ {n ^ {\prime}} \boldsymbol {x} ^ {(i)} \boldsymbol {x} ^ {(i) ^ {\top}}.
$$
$$
\left\| \mathbf {I} _ {d} - \hat {\boldsymbol {\Sigma}} ^ {1 / 2} \boldsymbol {\Sigma} ^ {- 1 / 2} \right\| \vee \left\| \mathbf {I} _ {d} - \hat {\boldsymbol {\Sigma}} ^ {- 1 / 2} \boldsymbol {\Sigma} ^ {1 / 2} \right\| \lesssim \sqrt {\frac {d}{n ^ {\prime}}}
$$
$$
\begin{array}{l} \| \mathbf {I} _ {d} - \hat {\boldsymbol {\Sigma}} ^ {- 1 / 2} \boldsymbol {\Sigma} ^ {1 / 2} \| = \left\{\lambda_ {\max } \left(\hat {\boldsymbol {\Sigma}} ^ {- 1 / 2} \boldsymbol {\Sigma} ^ {1 / 2}\right) - 1 \right\} \vee \left\{1 - \lambda_ {\min } \left(\hat {\boldsymbol {\Sigma}} ^ {- 1 / 2} \boldsymbol {\Sigma} ^ {1 / 2}\right) \right\} \\ = \left\{\lambda_ {\max } \left(\boldsymbol {\Sigma} ^ {1 / 2} \hat {\boldsymbol {\Sigma}} ^ {- 1} \boldsymbol {\Sigma} ^ {1 / 2}\right) ^ {1 / 2} - 1 \right\} \vee \left\{1 - \lambda_ {\min } \left(\boldsymbol {\Sigma} ^ {1 / 2} \hat {\boldsymbol {\Sigma}} ^ {- 1} \boldsymbol {\Sigma} ^ {1 / 2}\right) \right\} \\ = \left\{\lambda_ {\min } \left(\boldsymbol {\Sigma} ^ {- 1 / 2} \hat {\boldsymbol {\Sigma}} \boldsymbol {\Sigma} ^ {- 1 / 2}\right) ^ {- 1 / 2} - 1 \right\} \vee \left\{1 - \lambda_ {\max } \left(\boldsymbol {\Sigma} ^ {- 1 / 2} \hat {\boldsymbol {\Sigma}} \boldsymbol {\Sigma} ^ {- 1 / 2}\right) ^ {- 1 / 2} \right\} \\ = \left\{\lambda_ {\min } \left(\frac {1}{n ^ {\prime}} \sum_ {i = 1} ^ {n ^ {\prime}} \boldsymbol {z} ^ {(i)} \boldsymbol {z} ^ {(i) ^ {\top}}\right) ^ {- 1 / 2} - 1 \right\} \vee \left\{1 - \lambda_ {\max } \left(\frac {1}{n ^ {\prime}} \sum_ {i = 1} ^ {n ^ {\prime}} \boldsymbol {z} ^ {(i)} \boldsymbol {z} ^ {(i) ^ {\top}}\right) ^ {- 1 / 2} \right\}. \\ \end{array}
$$
$$
\| \mathbf {I} _ {d} - \hat {\boldsymbol {\Sigma}} ^ {1 / 2} \boldsymbol {\Sigma} ^ {- 1 / 2} \| = \left\{\lambda_ {\max} \left(\frac {1}{n ^ {\prime}} \sum_ {i = 1} ^ {n ^ {\prime}} \boldsymbol {z} ^ {(i)} \boldsymbol {z} ^ {(i) ^ {\top}}\right) ^ {1 / 2} - 1 \right\} \vee \left\{1 - \lambda_ {\min} \left(\frac {1}{n ^ {\prime}} \sum_ {i = 1} ^ {n ^ {\prime}} \boldsymbol {z} ^ {(i)} \boldsymbol {z} ^ {(i) ^ {\top}}\right) ^ {1 / 2} \right\}
$$
Moreover, by [Wai19, Example 6.2], we have with probability at least  $1 - 2e^{-d}$
$$
\lambda_ {\max } \left(\frac {1}{n ^ {\prime}} \sum_ {i = 1} ^ {n ^ {\prime}} \boldsymbol {z} ^ {(i)} \boldsymbol {z} ^ {(i)} ^ {\top}\right) \leq 1 + (\sqrt {2} + 1) \sqrt {\frac {d}{n ^ {\prime}}} \quad \text {a n d} \quad \lambda_ {\min } \left(\frac {1}{n ^ {\prime}} \sum_ {i = 1} ^ {n ^ {\prime}} \boldsymbol {z} ^ {(i)} \boldsymbol {z} ^ {(i)} ^ {\top}\right) \geq 1 - (\sqrt {2} + 1) \sqrt {\frac {d}{n ^ {\prime}}}.
$$
Thus, for  $n' \gtrsim d$  (with a sufficiently large absolute constant), we have
$$
\| \mathbf {I} _ {d} - \hat {\boldsymbol {\Sigma}} ^ {1 / 2} \hat {\boldsymbol {\Sigma}} ^ {- 1 / 2} \| \vee \| \mathbf {I} _ {d} - \hat {\boldsymbol {\Sigma}} ^ {- 1 / 2} \boldsymbol {\Sigma} ^ {1 / 2} \| \lesssim \sqrt {\frac {d}{n ^ {\prime}}}
$$
Lemma 25 (Chernoff's Inequality). Suppose  $X_{1},\ldots ,X_{n}$  are i.i.d. Bernoulli random variables, and further assume that  $\mathbb{E}[\sum_i X_i]\leq \mu$  . Then, for any  $\delta \geq 1$
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {n} X _ {i} \geq \mu (1 + \delta)\right) \leq e ^ {- \mu \delta / 3}. \tag {F.1}
$$
Proof. The proof follows from a standard Chernoff bound. From [Ver18, Theorem 2.3.1]
$$
\mathbb {P} \left(\sum_ {i} X _ {i} \geq \mu (1 + \delta)\right) \leq e ^ {\mu (\delta - (1 + \delta) \ln (1 + \delta))},
$$
(notice that the statement of [Ver18, Theorem 2.3.1] holds true even when  $\mathbb{E}[\sum_i X_i] = \mu$  is replaced with  $\mathbb{E}[\sum_i X_i] \leq \mu$ ). We conclude by remarking that  $\delta - (1 + \delta) \ln(1 + \delta) \leq -\delta / 3$  for  $\delta \geq 1$ .