SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. The first and second statements of the lemma follow from Lemmas 16 and 17 respectively. The third and fourth statements are standard Gaussian covariance concentration bounds (see e.g. [Wai19, Theorem 6.1] where both statements hold with probability at least  $1 - 2e^{-d}$ ).  $\square$
D.3 Proof of Theorem 5
We begin by recalling the definition  $\overline{\boldsymbol{w}} \coloneqq \frac{\hat{\boldsymbol{\Sigma}}^{1/2}\boldsymbol{w}}{\|\hat{\boldsymbol{\Sigma}}^{1/2}\boldsymbol{w}\|}$  and the finite-samples dynamics (3.5), which we copy here for the reader's convenience,
$$
\frac {\mathrm {d} \boldsymbol {w} ^ {t}}{\mathrm {d} t} = \eta (\boldsymbol {w} ^ {t}) \nabla_ {\boldsymbol {w} ^ {t}} \left\{\frac {1}{n} \sum_ {i = 1} ^ {n} \phi \left(\frac {\left\langle \boldsymbol {w} ^ {t} , \boldsymbol {x} ^ {(i)} \right\rangle}{\| \hat {\boldsymbol {\Sigma}} ^ {1 / 2} \boldsymbol {w} ^ {t} \|}\right) y ^ {(i)} \right\},
$$
where  $\eta (\pmb{w}^t) = \| \hat{\pmb{\Sigma}}^{1 / 2}\pmb{w}^t\| ^2$  . Moreover, via chain rule, we obtain
$$
\frac {\mathrm {d} \overline {{\boldsymbol {w}}} ^ {t}}{\mathrm {d} t} = \frac {\left(\mathbf {I} _ {d} - \overline {{\boldsymbol {w}}} ^ {t} \overline {{\boldsymbol {w}}} ^ {t ^ {\top}}\right) \hat {\boldsymbol {\Sigma}} ^ {1 / 2}}{\| \hat {\boldsymbol {\Sigma}} ^ {1 / 2} \boldsymbol {w} ^ {t} \|} \frac {\mathrm {d} \boldsymbol {w} ^ {t}}{\mathrm {d} t}. \tag {D.5}
$$
Let  $\tilde{\pmb{z}}^{(i)}\coloneqq \hat{\pmb{\Sigma}}^{-1 / 2}\pmb{x}^{(i)} = \hat{\pmb{\Sigma}}^{-1 / 2}\pmb{\Sigma}^{1 / 2}\pmb{z}^{(i)}$  . Then,
$$
\frac {\mathrm {d} \overline {{\boldsymbol {w}}} ^ {t}}{\mathrm {d} t} = (\mathbf {I} _ {d} - \overline {{\boldsymbol {w}}} ^ {t} \overline {{\boldsymbol {w}}} ^ {t ^ {\top}}) \hat {\boldsymbol {\Sigma}} (\mathbf {I} _ {d} - \overline {{\boldsymbol {w}}} ^ {t} \overline {{\boldsymbol {w}}} ^ {t ^ {\top}}) \left\{\frac {1}{n} \sum_ {i = 1} ^ {n} \phi^ {\prime} \left(\frac {\left\langle \boldsymbol {w} ^ {t} , \boldsymbol {x} ^ {(i)} \right\rangle}{\| \hat {\boldsymbol {\Sigma}} ^ {1 / 2} \boldsymbol {w} ^ {t} \|}\right) y ^ {(i)} \tilde {\boldsymbol {z}} ^ {(i)} \right\}
$$
$$
\boldsymbol {\nu} (\overline {{\boldsymbol {w}}}) := \left(\mathbf {I} _ {d} - \overline {{\boldsymbol {w}}} \overline {{\boldsymbol {w}}} ^ {\top}\right) \hat {\boldsymbol {\Sigma}} \left(\mathbf {I} _ {d} - \overline {{\boldsymbol {w}}} \overline {{\boldsymbol {w}}} ^ {\top}\right) \overline {{\boldsymbol {u}}}.
$$
$$
\frac {\mathrm {d} \left\langle \overline {{\boldsymbol {w}}} ^ {t} , \overline {{\boldsymbol {u}}} \right\rangle}{\mathrm {d} t} = \left\langle \boldsymbol {\nu} (\overline {{\boldsymbol {w}}} ^ {t}), \frac {1}{n} \sum_ {i = 1} ^ {n} \phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t}, \tilde {\boldsymbol {z}} ^ {(i)} \right\rangle\right) y ^ {(i)} \tilde {\boldsymbol {z}} ^ {(i)} \right\rangle .
$$
We can decompose the above dynamics into a population term and three different error terms in the following manner:
$$
\begin{array}{l} \frac {\mathrm {d} \left\langle \overline {{\boldsymbol {w}}} ^ {t} , \overline {{\boldsymbol {u}}} \right\rangle}{\mathrm {d} t} = \left\langle \boldsymbol {\nu} (\overline {{\boldsymbol {w}}} ^ {t}), \mathbb {E} _ {\boldsymbol {z}, y} \left[ \phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t}, \boldsymbol {z} \right\rangle\right) y \boldsymbol {z} \right] \right\rangle \\ + \underbrace {\left\langle \boldsymbol {\nu} \left(\overline {{\boldsymbol {w}}} ^ {t}\right) , \frac {1}{n} \sum_ {i = 1} ^ {n} \phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t} , \boldsymbol {z} ^ {(i)} \right\rangle\right) y ^ {(i)} \boldsymbol {z} ^ {(i)} - \mathbb {E} _ {\boldsymbol {z} , y} \left[ \phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t} , \boldsymbol {z} \right\rangle\right) y \boldsymbol {z} \right] \right\rangle} _ {=: \mathcal {E} _ {1}} \\ + \underbrace {\frac {1}{n} \sum_ {i = 1} ^ {n} \left\{\phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t} , \tilde {\boldsymbol {z}} ^ {(i)} \right\rangle\right) - \phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t} , \boldsymbol {z} ^ {(i)} \right\rangle\right) \right\} y ^ {(i)} \left\langle \boldsymbol {z} ^ {(i)} , \boldsymbol {\nu} (\overline {{\boldsymbol {w}}} ^ {t}) \right\rangle} _ {=: \mathcal {E} _ {2}} \\ + \underbrace {\frac {1}{n} \sum_ {i = 1} ^ {n} \phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}}, \tilde {\boldsymbol {z}} ^ {(i)} \right\rangle\right) y ^ {(i)} \left\langle \tilde {\boldsymbol {z}} ^ {(i)} - \boldsymbol {z} ^ {(i)} , \boldsymbol {\nu} (\overline {{\boldsymbol {w}}} ^ {t}) \right\rangle} _ {=: \mathcal {E} _ {3}}. \\ \end{array}
$$
We will proceed in three steps. In the first, we bound  $\mathcal{E}_1$ , the concentration error. In the second, we bound  $\mathcal{E}_2$  and  $\mathcal{E}_3$ , the errors due to estimating  $\boldsymbol{\Sigma}$  with  $\hat{\boldsymbol{\Sigma}}$  (i.e. replacing  $z^{(i)}$  with  $\tilde{z}^{(i)}$ ). Finally, we will analyze the convergence time similar to that of Proposition 4. Throughout the proof, we will assume that the event  $\mathcal{G}$  of Lemma 18 occurs.
Step 1. Controlling the concentration error  $\mathcal{E}_1$ . Let  $K \asymp \ln (nd^q)^{p/2}$ , and notice that on event  $\mathcal{G}$  we have  $|y^{(i)}| \lesssim K$  for all  $i$ . Let  $y_K := y\mathbf{1}(|y| \leq K)$ . On the event  $\mathcal{G}$ , we have  $y_K^{(i)} = y^{(i)}$  for all  $i$ , and
$$
\mathcal {E} _ {1} = \left\langle \nu (\overline {{\boldsymbol {w}}} ^ {t}), \boldsymbol {\Delta} _ {n} \right\rangle \geq - \| \boldsymbol {\Delta} _ {n} \| \| \nu (\overline {{\boldsymbol {w}}} ^ {t}) \|,
$$
$$
\pmb {\Delta} _ {n} := \frac {1}{n} \sum_ {i = 1} ^ {n} \phi^ {\prime} \Big (\big <   \overline {{\pmb {w}}}, \pmb {z} ^ {(i)} \big > \Big) y _ {K} ^ {(i)} \pmb {z} ^ {(i)} - \mathbb {E} _ {\pmb {z}, y} [ \phi^ {\prime} (\langle \overline {{\pmb {w}}}, \pmb {z} \rangle) y \pmb {z} ].
$$
Thus, our objective is to bound  $\| \pmb{\Delta}_n\|$  uniformly for all  $\overline{w}\in \mathbb{S}^{d - 1}$ . To that end, we first modify the expectation in the above definition so that the empirical average and expected value match in terms of their random variables. Specifically,
$$
\begin{array}{l} \sup  _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v} \in \mathbb {S} ^ {d - 1}} \langle \boldsymbol {\Delta} _ {n}, \boldsymbol {v} \rangle = \sup  _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v} \in \mathbb {S} ^ {d - 1}} \frac {1}{n} \sum_ {i = 1} ^ {n} \phi^ {\prime} (\left\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} ^ {(i)} \right\rangle) y _ {K} ^ {(i)} \left\langle \boldsymbol {z} ^ {(i)}, \boldsymbol {v} \right\rangle - \mathbb {E} _ {\boldsymbol {z}, y} [ \phi^ {\prime} (\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle) y _ {K} \langle \boldsymbol {z}, \boldsymbol {v} \rangle ] \\ - \mathbb {E} _ {\boldsymbol {z}, y} [ \phi^ {\prime} (\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle) y \langle \boldsymbol {z}, \boldsymbol {v} \rangle \mathbf {1} (| y | > K) ]. \\ \end{array}
$$
By the Cauchy-Schwartz inequality,
$$
\begin{array}{l} \left. \right.\left| \mathbb {E} _ {\boldsymbol {z}, y} \left[ \phi^ {\prime} \left(\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle\right) y \langle \boldsymbol {z}, \boldsymbol {v} \rangle \mathbf {1} (| y | > K) \right]\right| \leq \mathbb {E} _ {\boldsymbol {z}, y} \left[ \phi^ {\prime} \left(\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle\right) ^ {2} y ^ {2} \langle \boldsymbol {z}, \boldsymbol {v} \rangle^ {2} \right] ^ {1 / 2} \mathbb {E} [ \mathbf {1} (| y | > K) ] ^ {1 / 2} \\ \lesssim \mathbb {E} \left[ y ^ {4} \right] ^ {1 / 4} \mathbb {E} _ {\boldsymbol {z}} \left[ \langle \boldsymbol {z}, \boldsymbol {v} \rangle^ {4} \right] ^ {1 / 4} \mathbb {P} (| y | > K) ^ {1 / 2} \\ \lesssim d ^ {- q / 2}, \\ \end{array}
$$
where the last inequality follows from Lemma 16. Hence,
$$
\begin{array}{l} \sup  _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v} \in \mathbb {S} ^ {d - 1}} \left\langle \boldsymbol {\Delta} _ {n}, \boldsymbol {v} \right\rangle \leq \sup  _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v} \in \mathbb {S} ^ {d - 1}} \frac {1}{n} \sum_ {i = 1} ^ {n} \phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} ^ {(i)} \right\rangle\right) y _ {K} ^ {(i)} \left\langle \boldsymbol {z} ^ {(i)}, \boldsymbol {v} \right\rangle - \mathbb {E} [ \phi^ {\prime} (\left\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \right\rangle) y _ {K} \left\langle \boldsymbol {z}, \boldsymbol {v} \right\rangle ] \\ + \mathcal {O} \left(d ^ {- q / 2}\right). \\ \end{array}
$$
Next, we need to establish high-probability bounds via a covering argument. To simplify the exposition, define the stochastic process indexed by  $\overline{\boldsymbol{w}}\in \mathbb{S}^{d - 1}$  and  $\pmb {v}\in \mathbb{S}^{d - 1}$  via
$$
X _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v}} ^ {(i)} := \phi^ {\prime} \Big (\left\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} ^ {(i)} \right\rangle \Big) y _ {K} ^ {(i)} \Big \langle \boldsymbol {z} ^ {(i)}, \boldsymbol {v} \Big \rangle .
$$
Fix some  $\epsilon_w, \epsilon_v > 0$ . Let  $\Theta_w$  and  $\Theta_v$  be  $\epsilon_w$  and  $\epsilon_v$  coverings of  $\mathbb{S}^{d-1}$ , and let  $\hat{\boldsymbol{w}}$  and  $\hat{\boldsymbol{v}}$  denote the projection of  $\overline{\boldsymbol{w}}$  onto  $\Theta_w$  and of  $\boldsymbol{v}$  onto  $\Theta_v$  respectively, then
$$
\begin{array}{l} \sup  _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v} \in \mathbb {S} ^ {d - 1}} \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v}} ^ {(i)} - \mathbb {E} [ X _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v}} ] = \sup  _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v} \in \mathbb {S} ^ {d - 1}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(X _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v}} ^ {(i)} - X _ {\overline {{\boldsymbol {w}}}, \hat {\boldsymbol {v}}} ^ {(i)}\right) + \frac {1}{n} \sum_ {i = 1} ^ {n} \left(X _ {\overline {{\boldsymbol {w}}}, \hat {\boldsymbol {v}}} ^ {(i)} - X _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} ^ {(i)}\right) \\ + \mathbb {E} _ {\boldsymbol {z}, y} \left[ X _ {\overline {{\boldsymbol {w}}}, \hat {\boldsymbol {v}}} - X _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v}} \right] + \mathbb {E} _ {\boldsymbol {z}, y} \left[ X _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} - X _ {\overline {{\boldsymbol {w}}}, \hat {\boldsymbol {v}}} \right] \\ + \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} ^ {(i)} - \mathbb {E} _ {\boldsymbol {z}, y} [ X _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} ]. \\ \end{array}
$$
We bound each of the terms using Cauchy-Schwartz. Specifically,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \Big (X _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v}} ^ {(i)} - X _ {\overline {{\boldsymbol {w}}}, \hat {\boldsymbol {v}}} ^ {(i)} \Big) \leq \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \phi^ {\prime} (\langle \overline {{\boldsymbol {w}} , \boldsymbol {z} ^ {(i)} \rangle) ^ {2} y _ {K} ^ {(i)} ^ {2}} \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \big <   \boldsymbol {z} ^ {(i)} , \boldsymbol {v} - \hat {\boldsymbol {v}} \big > ^ {2}}} \lesssim K \epsilon_ {v},
$$
where we used the upper bound on the operator norm of  $\frac{1}{n}\sum_{i=1}^{n}z^{(i)}z^{(i)^{\top}}$  from Lemma 18 together with the fact that  $n\stackrel{>}{\sim}d$ . Similarly,
$$
\mathbb {E} _ {\boldsymbol {z}, \boldsymbol {y}} \left[ X _ {\overline {{\boldsymbol {w}}}, \hat {\boldsymbol {v}}} - X _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v}} \right] \leq \mathbb {E} _ {\boldsymbol {z}, \boldsymbol {y}} \left[ \phi^ {\prime} (\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle) ^ {2} y _ {K} ^ {2} \right] ^ {1 / 2} \mathbb {E} _ {\boldsymbol {z}} \left[ \langle \boldsymbol {z}, \boldsymbol {v} - \hat {\boldsymbol {v}} \rangle^ {2} \right] ^ {1 / 2} \lesssim K \epsilon_ {v}.
$$
To bound the differences when we replace  $\overline{w}$  with  $\hat{w}$ , we need to make a distinction between ReLU and smooth activations as the respective arguments are to some extent different. When  $\phi'$  is Lipschitz,
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \Big (X _ {\overline {{\boldsymbol {w}}}, \widehat {\boldsymbol {v}}} ^ {(i)} - X _ {\widehat {\boldsymbol {w}}, \widehat {\boldsymbol {v}}} ^ {(i)} \Big) \leq \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} (y _ {K} ^ {(i)}) ^ {2} \big (\phi^ {\prime} (\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} ^ {(i)} \rangle) - \phi^ {\prime} (\langle \widehat {\boldsymbol {w}}, \boldsymbol {z} ^ {(i)} \rangle) \big) ^ {2}} \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \langle \boldsymbol {z} ^ {(i)} , \widehat {\boldsymbol {v}} ^ {2} \rangle} \lesssim K \epsilon_ {w},
$$
$$
\mathbb {E} [ X _ {\widehat {\boldsymbol {w}}, \widehat {\boldsymbol {v}}} - X _ {\overline {{\boldsymbol {w}}}, \widehat {\boldsymbol {v}}} ] \leq \mathbb {E} \Big [ y _ {K} ^ {2} (\phi^ {\prime} (\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle) - \phi^ {\prime} (\langle \widehat {\boldsymbol {w}}, \boldsymbol {z} \rangle)) ^ {2} \Big ] ^ {1 / 2} \mathbb {E} \Big [ \langle \boldsymbol {z}, \widehat {\boldsymbol {v}} \rangle^ {2} \Big ] ^ {1 / 2} \lesssim K \epsilon_ {w}.
$$
Therefore, for a smooth activation  $\phi$  we choose  $\epsilon_v = \epsilon_w = \sqrt{d / n}$ , and obtain
$$
\sup _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v} \in \mathbb {S} ^ {d - 1}} \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v}} ^ {(i)} - \mathbb {E} _ {\boldsymbol {z}, y} [ X _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v}} ] \leq \sup _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} ^ {(i)} - \mathbb {E} _ {\boldsymbol {z}, y} [ X _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} ] + \tilde {\mathcal {O}} (\sqrt {d / n}).
$$
When  $\phi$  is the ReLU activation, we need to show that the sign of the preactivation changes only for a small number of samples when we change the weight  $\overline{w}$  to  $\hat{w}$ . Notice that
$$
\begin{array}{l} \operatorname {s i g n} \left(\left\langle \bar {w}, z ^ {(i)} \right\rangle\right) \neq \operatorname {s i g n} \left(\left\langle \hat {w}, z ^ {(i)} \right\rangle\right) \Rightarrow \left| \left\langle \bar {w}, z ^ {(i)} \right\rangle \right| \leq \left| \left\langle \hat {w} - \bar {w}, z ^ {(i)} \right\rangle \right| \\ \Rightarrow \left| \left\langle \overline {{\boldsymbol {w}}}, \overline {{\boldsymbol {z}}} ^ {(i)} \right\rangle \right| \leq \epsilon_ {w}. \\ \end{array}
$$
Recall that  $\overline{z}^{(i)}\coloneqq z^{(i)} / \| z^{(i)}\|$ . Choose  $\epsilon_w\succ \sqrt{d} /n$ . On event  $\mathcal{G}$ , we know from Lemma 18 that at most  $\mathcal{O}(d\ln (n / \sqrt{d}))$  samples can satisfy the above condition. Therefore,
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(X _ {\widehat {\boldsymbol {w}}, \widehat {\boldsymbol {v}}} ^ {(i)} - X _ {\widehat {\boldsymbol {w}}, \widehat {\boldsymbol {v}}} ^ {(i)}\right) \leq \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {K} ^ {(i)}\right) ^ {2} \left(\phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} ^ {(i)} \right\rangle\right) - \phi^ {\prime} \left(\left\langle \widehat {\boldsymbol {w}}, \boldsymbol {z} ^ {(i)} \right\rangle\right)\right) ^ {2}} \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle \boldsymbol {z} ^ {(i)}, \widehat {\boldsymbol {v}} \right\rangle^ {2}} \\ \lesssim K \sqrt {\frac {d \ln (n / \sqrt {d})}{n}}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} \left[ X _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} - X _ {\overline {{\boldsymbol {w}}}, \hat {\boldsymbol {v}}} \right] \leq \mathbb {E} \left[ y _ {K} ^ {2} \left(\phi^ {\prime} (\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle) - \phi^ {\prime} (\langle \hat {\boldsymbol {w}}, \boldsymbol {z} \rangle)\right) ^ {2} \right] ^ {1 / 2} \mathbb {E} \left[ \langle \boldsymbol {z}, \hat {\boldsymbol {v}} \rangle^ {2} \right] ^ {1 / 2} \\ \leq K \mathbb {P} (\operatorname {s i g n} (\langle \bar {w}, z \rangle) \neq \operatorname {s i g n} (\langle \hat {w}, z \rangle)) ^ {1 / 2} \\ \leq K \mathbb {P} (| \langle \overline {{w}}, z \rangle | \leq \epsilon_ {w}) ^ {1 / 2} \\ \leq 2 K \sqrt {d ^ {1 / 2} \epsilon_ {w}}, \\ \end{array}
$$
where the last inequality follows from the anti-concentration on the sphere [BBSS22, Lemma A.7]. Thus, for ReLU we choose  $\epsilon_v \asymp \sqrt{d / n}$  and  $\epsilon_w \asymp \sqrt{d} / n$ , and once again obtain
$$
\sup  _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v} \in \mathbb {S} ^ {d - 1}} \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v}} ^ {(i)} - \mathbb {E} _ {\boldsymbol {z}, y} [ X _ {\overline {{\boldsymbol {w}}}, \boldsymbol {v}} ] \leq \sup  _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} ^ {(i)} - \mathbb {E} _ {\boldsymbol {z}, y} [ X _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} ] + \tilde {\mathcal {O}} (\sqrt {d / n}).
$$
It remains to bound the term
$$
\sup  _ {\hat {w}, \hat {v}} \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {\hat {w}, \hat {v}} ^ {(i)} - \mathbb {E} [ X _ {\hat {w}, \hat {v}} ].
$$
Notice that for fixed  $\hat{w},\hat{v}$ $X_{\hat{w},\hat{v}}$  is sub-Gaussian with sub-Gaussian norm  $\mathcal{O}(K)$ . Thus, via the sub-Gaussian maximal inequality [VH16, Lemma 5.2],
$$
\sup _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} \frac {1}{n} \sum_ {i = 1} ^ {n} X _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} ^ {(i)} - \mathbb {E} [ X _ {\hat {\boldsymbol {w}}, \hat {\boldsymbol {v}}} ] \lesssim \sqrt {K ^ {2} d / n \ln (1 / (\epsilon_ {w} \epsilon_ {v}))},
$$
with probability at least  $1 - e^{-d}$ . Consequently, we have
$$
\sup _ {\overline {{\boldsymbol {w}}} \in \mathbb {S} ^ {d - 1}} \| \boldsymbol {\Delta} _ {n} \| \leq \tilde {\mathcal {O}} (\sqrt {d / n} + d ^ {- q / 2}),
$$
with probability at least  $1 - \mathcal{O}(d^{-q})$ . Assuming that  $n$  grows at most polynomially in dimension and choosing a sufficiently large  $q$ , we have  $\sup_{\overline{w} \in \mathbb{S}^{d-1}} \| \Delta_n \| \leq \tilde{\mathcal{O}}(\sqrt{d/n})$  with probability at least  $1 - \mathcal{O}(d^{-q})$ .
Finally, by Lemma 23,
$$
\left\| \boldsymbol {\nu} \left(\overline {{\boldsymbol {w}}} ^ {t}\right) \right\| \leq \lambda_ {\max } \left(\hat {\boldsymbol {\Sigma}}\right) \lesssim \lambda_ {\max } (\boldsymbol {\Sigma}), \tag {D.6}
$$
with probability at least  $1 - e^{-n'/2}$ . Combining the above with the bound on  $\|\pmb{\Delta}_n\|$ , we have  $\mathcal{E}_1 \geq -\lambda_{\max}(\pmb{\Sigma}) \tilde{\mathcal{O}}(\sqrt{d/n})$  with probability at least  $1 - \mathcal{O}(d^{-q})$ , which concludes the first step of the proof.
Step 2. Bounding the error due to the estimation of  $\Sigma$ , i.e.  $\mathcal{E}_2$  and  $\mathcal{E}_3$ . Recall that we are considering the event  $\mathcal{G}$ , thus  $y^{(i)} = y_K^{(i)}$ . We can control each of the error terms separately. We begin by  $\mathcal{E}_2$ , where by Cauchy-Schwartz
$$
\begin{array}{l} \mathcal {E} _ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left\{\phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t}, \tilde {\boldsymbol {z}} ^ {(i)} \right\rangle\right) - \phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t}, \boldsymbol {z} ^ {(i)} \right\rangle\right) \right\} y _ {K} ^ {(i)} \left\langle \boldsymbol {z} ^ {(i)}, \boldsymbol {\nu} (\overline {{\boldsymbol {w}}} ^ {t}) \right\rangle \\ \geq - \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \left\{\phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t} , \boldsymbol {z} ^ {(i)} \right\rangle\right) - \phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t} , \tilde {\boldsymbol {z}} ^ {(i)} \right\rangle\right) \right\} ^ {2}} \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} y _ {K} ^ {(i) 2} \left\langle \boldsymbol {z} ^ {(i)} , \boldsymbol {\nu} (\overline {{\boldsymbol {w}}} ^ {t}) \right\rangle^ {2}} \\ \geq - K \| \boldsymbol {\nu} (\overline {{\boldsymbol {w}}} ^ {t}) \| \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \left\{\phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t} , \boldsymbol {z} ^ {(i)} \right\rangle\right) - \phi^ {\prime} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t} , \tilde {\boldsymbol {z}} ^ {(i)} \right\rangle\right) \right\} ^ {2}}, \\ \end{array}
$$
where the last line follows from Lemma 18 and the fact that  $n \gtrsim d$ . When  $\phi'$  is additionally Lipschitz, we have
$$
\mathcal {E} _ {2} \gtrsim - K \| \boldsymbol {\nu} (\overline {{\boldsymbol {w}}} ^ {t}) \| \sqrt {\frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle \overline {{\boldsymbol {w}}} ^ {t} , \boldsymbol {z} ^ {(i)} - \tilde {\boldsymbol {z}} ^ {(i)} \right\rangle^ {2}}.
$$
Moreover, for any  $\overline{\boldsymbol{w}}\in \mathbb{S}^{d - 1}$
$$
\begin{array}{l} \frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} ^ {(i)} - \tilde {\boldsymbol {z}} ^ {(i)} \right\rangle^ {2} \leq \| \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {z} ^ {(i)} \boldsymbol {z} ^ {(i) ^ {\top}} \| ^ {2} \| (\mathbf {I} _ {d} - \hat {\boldsymbol {\Sigma}} ^ {- 1 / 2} \boldsymbol {\Sigma} ^ {1 / 2}) \overline {{\boldsymbol {w}}} \| ^ {2} \\ \leq \| \frac {1}{n} \sum_ {i = 1} ^ {n} \boldsymbol {z} ^ {(i)} \boldsymbol {z} ^ {(i)} ^ {\top} \| ^ {2} \| \mathbf {I} _ {d} - \hat {\boldsymbol {\Sigma}} ^ {- 1 / 2} \boldsymbol {\Sigma} ^ {1 / 2} \| ^ {2} \\ \lesssim \frac {d}{n ^ {\prime}}, \\ \end{array}
$$
where the last inequality holds with probability at least  $1 - 2e^{-d}$  on the event of Lemma 24. Hence for smooth activations we conclude
$$
\mathcal {E} _ {2} \gtrsim - K \| \boldsymbol {\nu} (\overline {{\boldsymbol {w}}} ^ {t}) \| \sqrt {d / n ^ {\prime}}.
$$
When  $\phi$  is the ReLU activation, we need a more involved argument to control  $\mathcal{E}_2$ . In particular, we will show that for any  $\overline{\boldsymbol{w}}$ , at most only  $\tilde{\mathcal{O}}(d)$  datapoints can have  $\mathrm{sign}\bigl(\langle \overline{\boldsymbol{w}}, \boldsymbol{z}^{(i)} \rangle \bigr) \neq \mathrm{sign}\biggl(\langle \overline{\boldsymbol{w}}, \tilde{\boldsymbol{z}}^{(i)} \rangle \biggr)$ . Notice that
$$
\begin{array}{l} \left. \right. \operatorname {s i g n} \left(\left\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} ^ {(i)} \right\rangle\right) \neq \operatorname {s i g n} \left(\left\langle \overline {{\boldsymbol {w}}}, \tilde {\boldsymbol {z}} ^ {(i)} \right\rangle\right) \Rightarrow \left|\left\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} ^ {(i)} \right\rangle\right| \leq \left|\left\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} ^ {(i)} - \tilde {\boldsymbol {z}} ^ {(i)} \right\rangle\right| \\ \Rightarrow \left| \left\langle \bar {\boldsymbol {w}}, \bar {\boldsymbol {z}} ^ {(i)} \right\rangle \right| \leq \| \mathbf {I} _ {d} - \hat {\boldsymbol {\Sigma}} ^ {- 1 / 2} \boldsymbol {\Sigma} ^ {1 / 2} \| \tag {D.7} \\ \end{array}
$$