SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
g ^ {\prime} = \sum_ {j \geq 1} \sqrt {j} \alpha_ {j} h _ {j - 1} \quad \text {a n d} \quad \phi^ {\prime} = \sum_ {j \geq 1} \sqrt {j} \beta_ {j} h _ {j - 1}.
$$
Since we additionally assume  $\mathbb{E}[g] = \alpha_0 = 0$ , by the definition of  $\tilde{\psi}_{\phi,g}$  in Lemma 12 and the properties of Hermite expansion discussed in Appendix B, we have
$$
\tilde {\psi} _ {\phi , g} (\boldsymbol {w} ^ {t}) = \frac {\sum_ {j \geq s} (j - 1) \alpha_ {j} \beta_ {j} \left\langle \overline {{\boldsymbol {w}}} ^ {t} , \boldsymbol {u} \right\rangle^ {j}}{\| \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {w} \|} + \frac {1}{2},
$$
$$
\tilde {\zeta} _ {\phi , g} (\boldsymbol {w} ^ {t}) := - \sum_ {j \geq s} j \alpha_ {j} \beta_ {j} \left\langle \overline {{\boldsymbol {w}}} ^ {t}, \boldsymbol {u} \right\rangle^ {j - 1}.
$$
Thus we obtain
$$
\frac {\mathrm {d} \left\langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \right\rangle^ {2}}{\mathrm {d} t} = 2 F (\boldsymbol {w} ^ {t}) (1 - \left\langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \right\rangle^ {2}),
$$
$$
F (\boldsymbol {w} ^ {t}) := \sum_ {j \geq s} j \alpha_ {j} \beta_ {j} \langle \overline {{\boldsymbol {w}}} ^ {t}, \boldsymbol {u} \rangle^ {j - 1} \langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \rangle - \frac {\kappa}{1 + \kappa} \sum_ {j \geq s} (j - 1) \alpha_ {j} \beta_ {j} \langle \overline {{\boldsymbol {w}}} ^ {t}, \boldsymbol {u} \rangle^ {j + 1} \langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \rangle - \frac {\kappa \langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle^ {2}}{2 (1 + \kappa)}.
$$
We proceed by upper bounding  $F$ . To do so, first note that
$$
\| \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {w} \| = \sqrt {\frac {1 + \kappa \langle \boldsymbol {w} , \boldsymbol {u} \rangle^ {2}}{1 + \kappa}} \geq \sqrt {\frac {1}{1 + \kappa}}.
$$
To bound the first term of  $F$ , we have
$$
\begin{array}{l} \sum_ {j \geq s} j \alpha_ {j} \beta_ {j} \left\langle \overline {{\boldsymbol {w}}} ^ {t}, \boldsymbol {u} \right\rangle^ {j - 1} \left\langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \right\rangle \leq | \left\langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \right\rangle | | \left\langle \overline {{\boldsymbol {w}}} ^ {t}, \boldsymbol {u} \right\rangle | ^ {s - 1} \sum_ {j \geq s} j | \alpha_ {j} \beta_ {j} | | \left\langle \overline {{\boldsymbol {w}}} ^ {t}, \boldsymbol {u} \right\rangle | ^ {j - s} \\ \leq \| \phi^ {\prime} \| _ {\gamma} \| g ^ {\prime} \| _ {\gamma} | \langle \overline {{\boldsymbol {w}}} ^ {t}, \boldsymbol {u} \rangle | ^ {s - 1} | \langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \rangle | \\ \leq \| \phi^ {\prime} \| _ {\gamma} \| g ^ {\prime} \| _ {\gamma} (1 + \kappa) ^ {(s - 1) / 2} | \langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \rangle | ^ {s}. \\ \end{array}
$$
$$
- \frac {\kappa}{1 + \kappa} \sum_ {j \geq s} (j - 1) \alpha_ {j} \beta_ {j} \left\langle \overline {{\boldsymbol {w}}} ^ {t}, \boldsymbol {u} \right\rangle^ {j + 1} \left\langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \right\rangle \leq \| \phi^ {\prime} \| _ {\gamma} \| g ^ {\prime} \| _ {\gamma} (1 + \kappa) ^ {(s + 1) / 2} \left| \left\langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \right\rangle \right| ^ {s + 2}.
$$
Hence, for  $\kappa \geq 1$
$$
F (\boldsymbol {w} ^ {t}) \leq \| \phi^ {\prime} \| _ {\gamma} \| g ^ {\prime} \| _ {\gamma} (1 + \kappa) ^ {(s - 1) / 2} | \langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \rangle | ^ {s} \left(1 + (1 + \kappa) \langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \rangle^ {2}\right) - \frac {\langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle^ {2}}{4}.
$$
Suppose  $\kappa <  d / C^2 -1$  , then
$$
\begin{array}{l} F (\boldsymbol {w} ^ {t}) \leq \left\langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \right\rangle^ {2} \left(2 \| \phi^ {\prime} \| _ {\gamma} \| g ^ {\prime} \| _ {\gamma} (1 + \kappa) ^ {(s - 1) / 2} | \left\langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \right\rangle | ^ {s - 2} - 1 / 4\right) \\ \leq \left\langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \right\rangle^ {2} \left(2 \| \phi^ {\prime} \| _ {\gamma} \| g ^ {\prime} \| _ {\gamma} C ^ {s - 2} \sqrt {\frac {(1 + \kappa) ^ {s - 1}}{d ^ {s - 2}}} - 1 / 4\right). \\ \end{array}
$$
Thus, for any  $\kappa$  such that
$$
1 \leq \kappa \leq \left\{\frac {d ^ {\frac {s - 2}{s - 1}}}{\left(8 C ^ {s - 2} \| \phi^ {\prime} \| _ {\gamma} \| g ^ {\prime} \| _ {\gamma}\right) ^ {\frac {2}{s - 1}}} \wedge \frac {d}{C ^ {2}} \right\} - 1
$$
and any  $\pmb{w}^t$  such that  $|\langle \pmb{w}^t, \pmb{u} \rangle| \leq C / \sqrt{d}$ , we have  $\frac{\mathrm{d}\langle \pmb{w}^t, \pmb{u} \rangle^2}{\mathrm{d}t} \leq 0$ , hence  $\sup_{t \geq 0} |\langle \pmb{w}^t, \pmb{u} \rangle| \leq C / \sqrt{d}$  as long as the above holds true at initialization.
Finally, we will show  $\left|\langle \pmb{w}^0, \pmb{u} \rangle \right| \leq C / \sqrt{d}$  with probability at least 0.99 for a suitable choice of constant  $C$ . Indeed, this is an elementary concentration of measure result on the unit sphere. For simplicity, we avoid performing sharp probability of failure analysis and only remark that  $\mathbb{E}\left[\langle \pmb{w}^0, \pmb{u} \rangle^2\right] = 1 / d$ , thus by the Markov inequality
$$
\mathbb {P} \left(\left\langle \boldsymbol {w} ^ {0}, \boldsymbol {u} \right\rangle^ {2} \geq C ^ {2} / d\right) \leq 1 / C ^ {2},
$$
hence a choice of  $C \geq 10$  suffices, and the proof is complete.
C.2 Extremely Large Spike
In this section, we will show that under extremely large spike, the spherical gradient flow (2.5) can potentially recover the true direction. Namely, we will prove the following proposition.
Proposition 14. Suppose we initialize the spherical population gradient flow (2.5) from  $\pmb{w}^{0}$ . Let  $\phi$  be the ReLU activation and assume
$$
\langle \phi , g \rangle_ {\gamma} := \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} [ \phi (z) g (z) ] = \alpha > 1 / 2,
$$
and  $\kappa \geq \frac{C}{\langle \pmb{w}^0, \pmb{u} \rangle^2}$  for a sufficiently large constant  $C > 0$  depending only on  $g$ . Then, the gradient flow on the sphere satisfies
$$
\left\{ \begin{array}{l l} {\left\langle \boldsymbol {w} ^ {T}, \boldsymbol {u} \right\rangle \geq 1 - \varepsilon} & {i f \quad \left\langle \boldsymbol {w} ^ {0}, \boldsymbol {u} \right\rangle > 0} \\ {\left\langle \boldsymbol {w} ^ {T}, \boldsymbol {u} \right\rangle \leq - 1 + \varepsilon} & {i f \quad \left\langle \boldsymbol {w} ^ {0}, \boldsymbol {u} \right\rangle <   0} \end{array} \right. \tag {C.3}
$$
$$
T \geq \frac {1}{\alpha - 1 / 2} \ln (2 / \varepsilon). \tag {C.4}
$$
Before proceeding to the proof, we notice that if we uniformly initialize  $\pmb{w}^0$  over  $\mathbb{S}^{d - 1}$ , then the typical value for  $\langle \pmb{w}^0, \pmb{u} \rangle$  is of order  $d^{-1/2}$ , meaning that the above proposition asks for  $\kappa = \Omega(d)$ . This is a regime where lower bounds for the sample complexity of kernel methods are  $\Omega(1)$  [DWY21], thus no meaningful separation in terms of dimension dependency of the sample complexity between neural networks and kernel methods is possible, as the problem becomes effectively one-dimensional.
Proof. The cases where  $\langle \pmb{w}^0, \pmb{u} \rangle > 0$  and  $\langle \pmb{w}^0, \pmb{u} \rangle < 0$  are symmetric, thus we only present the proof for the former. Using (C.2), we can write the dynamics on the sphere more explicitly as
$$
\frac{\mathrm{d}\langle\boldsymbol{w}^{t},\boldsymbol{u}\rangle}{\mathrm{d}t} = \left\{\underbrace{\frac{\mathbb{E}\big[\phi^{\prime}(\langle\overline{\boldsymbol{w}}^{t},\boldsymbol{z}\rangle)g^{\prime}(\langle\boldsymbol{u},\boldsymbol{z}\rangle)\big]}{1 + \kappa\langle\boldsymbol{w}^{t},\boldsymbol{u}\rangle^{2}}} _ {=: B_{1}}}_{=: B_{1}} + \underbrace{\frac{\kappa\langle\boldsymbol{w}^{t},\boldsymbol{u}\rangle\mathbb{E}\big[\phi(\langle\overline{\boldsymbol{w}}^{t},\boldsymbol{z}\rangle)g(\langle\boldsymbol{u},\boldsymbol{z}\rangle)\big]}{\sqrt{1 + \kappa}\sqrt{1 + \kappa\langle\boldsymbol{w}^{t},\boldsymbol{u}\rangle^{2}}}} _ {=: B_{2}} - \frac{\kappa\langle\boldsymbol{w}^{t},\boldsymbol{u}\rangle}{2(1 + \kappa)} \right\} (1 - \langle \boldsymbol{w}^{t},\boldsymbol{u}\rangle^{2}).
$$
Our goal is to study the regime of large  $\kappa$ , therefore we will bound how much  $B_{1}$  and  $B_{2}$  can deviate from their corresponding  $\kappa = \infty$  values. In particular, we have
$$
B _ {1} \geq - \frac {\| \phi^ {\prime} \| _ {\gamma} \| g ^ {\prime} \| _ {\gamma}}{1 + \kappa \langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle^ {2}} = \frac {- \| g ^ {\prime} \| _ {\gamma} / 2}{1 + \kappa \langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle^ {2}}.
$$
Furthermore, assuming  $\langle \pmb{w}^t, \pmb{u} \rangle > 0$  and  $\langle \phi, g \rangle_\gamma > 0$ , let  $c_{\kappa}(\pmb{w}^t) \coloneqq \frac{\kappa \langle \pmb{w}^t, \pmb{u} \rangle}{\sqrt{1 + \kappa} \sqrt{1 + \kappa \langle \pmb{w}^t, \pmb{u} \rangle^2}}$ . Then, by the Lipschitzness of  $\phi$ ,
$$
\begin{array}{l} B _ {2} = c _ {\kappa} \left(\boldsymbol {w} ^ {t}\right) \langle \phi , g \rangle_ {\gamma} + c _ {\kappa} \left(\boldsymbol {w} ^ {t}\right) \mathbb {E} \left[ \left(\phi (\left\langle \overline {{\boldsymbol {w}}} ^ {t}, \boldsymbol {z} \right\rangle) - \phi (\langle \boldsymbol {u}, \boldsymbol {z} \rangle)\right) g (\langle \boldsymbol {u}, \boldsymbol {z} \rangle) \right] \\ \geq c _ {\kappa} \left(\boldsymbol {w} ^ {t}\right) \langle \phi , g \rangle_ {\gamma} - \left| c _ {\kappa} \left(\boldsymbol {w} ^ {t}\right) \right| \| g \| _ {\gamma} \| \overline {{\boldsymbol {w}}} ^ {t} - \boldsymbol {u} \| \\ \geq c _ {\kappa} (\boldsymbol {w} ^ {t}) \langle \phi , g \rangle_ {\gamma} - \left| c _ {\kappa} (\boldsymbol {w} ^ {t}) \right| \| g \| _ {\gamma} \sqrt {2 (1 - \left\langle \overline {{\boldsymbol {w}}} ^ {t} , \boldsymbol {u} \right\rangle^ {2})} \\ \geq c _ {\kappa} (\boldsymbol {w} ^ {t}) \langle \phi , g \rangle_ {\gamma} - \frac {\sqrt {2 \kappa} \| g \| _ {\gamma} | \langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle |}{1 + \kappa \langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle^ {2}}. \\ \end{array}
$$
where we used  $\overline{\boldsymbol{w}}^t = \frac{\boldsymbol{\Sigma}^{1/2}\boldsymbol{w}^t}{\|\boldsymbol{\Sigma}^{1/2}\boldsymbol{w}^t\|}$  in the last step. Suppose  $\langle \boldsymbol{w}^t, \boldsymbol{u} \rangle > 0$  (which holds at least on a neighborhood around initialization, and as we will see below holds for all  $t > 0$ ), then,
$$
c _ {\kappa} (\boldsymbol {w} ^ {t}) \geq \frac {\kappa \langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle^ {2}}{1 + \kappa \langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle^ {2}}.
$$
As a result, we obtain
$$
B _ {1} + B _ {2} - \frac {\kappa \langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle}{2 (1 + \kappa)} \geq \langle \phi , g \rangle_ {\gamma} \left(1 - \frac {1}{\kappa \langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle^ {2}}\right) - \frac {\| g ^ {\prime} \| _ {\gamma} / 2 + \sqrt {2} \| g \| _ {\gamma}}{\sqrt {\kappa \langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle^ {2}}} - \frac {1}{2}.
$$
Consequently, the lower bound of the time derivative of  $\langle \pmb{w}^t, \pmb{u} \rangle$  becomes larger as  $\langle \pmb{w}^t, \pmb{u} \rangle$  increases. Therefore, assuming  $\langle \pmb{w}^0, \pmb{u} \rangle > 0$ , we only need to control this lower bound at initialization. Assume
$$
\kappa \geq \frac {4}{\langle \boldsymbol {w} ^ {0} , \boldsymbol {u} \rangle^ {2} (\alpha - 1 / 2)} \left\{\alpha \vee \frac {\left(\| g ^ {\prime} \| _ {\gamma} + \sqrt {8} \| g \| _ {\gamma}\right) ^ {2}}{\alpha - 1 / 2} \right\}.
$$
From this, we conclude that when  $\langle \pmb{w}^t, \pmb{u} \rangle > 0$  and  $\langle \phi, g \rangle_\gamma = \alpha > 1/2$ , we have
$$
\frac {\mathrm {d} \langle \boldsymbol {w} ^ {t} , \boldsymbol {u} \rangle}{\mathrm {d} t} \geq \frac {\alpha - 1 / 2}{2} (1 - \langle \boldsymbol {w} ^ {t}, \boldsymbol {u} \rangle^ {2}),
$$
integration yields the desired result.
D Proofs of Section 3
We begin by stating the closed-form expression for the population gradient, i.e. the counterpart of Lemma 12 in the normalized setting.
Lemma 15. Consider the population risk  $\mathcal{R}(\boldsymbol{w})$  defined by (3.1), recall that
$$
\mathcal {R} (\boldsymbol {w}) = \mathbb {E} \left[ - \phi \left(\frac {\langle \boldsymbol {w} , \boldsymbol {x} \rangle}{\| \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {w} \|}\right) g (\langle \overline {{\boldsymbol {u}}}, \boldsymbol {z} \rangle) \right] + \frac {1}{2} \| \phi \| _ {\gamma} ^ {2} + \frac {1}{2} \mathbb {E} [ y ^ {2} ].
$$
$$
\nabla \mathcal {R} (\boldsymbol {w}) = \frac {\boldsymbol {\Sigma} ^ {1 / 2} \left(\mathbf {I} _ {d} - \overline {{\boldsymbol {w}}} \overline {{\boldsymbol {w}}} ^ {\top}\right) \zeta_ {\phi , g} \left(\left\langle \overline {{\boldsymbol {w}}}, \overline {{\boldsymbol {u}}} \right\rangle\right) \overline {{\boldsymbol {u}}}}{\| \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {w} \|}, \tag {D.1}
$$
$$
\zeta_ {\phi , g} (\langle \overline {{\boldsymbol {w}}}, \overline {{\boldsymbol {u}}} \rangle) := - \mathbb {E} [ \phi^ {\prime} (\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle) g ^ {\prime} (\langle \overline {{\boldsymbol {u}}}, \boldsymbol {z} \rangle) ] = - \sum_ {j \geq s} j \alpha_ {j} \beta_ {j} \langle \overline {{\boldsymbol {w}}}, \overline {{\boldsymbol {u}}} \rangle^ {j - 1}. \tag {D.2}
$$
Proof. Recall from (3.1) that
$$
\begin{array}{l} \nabla \mathcal {R} (\boldsymbol {w}) = \nabla_ {\boldsymbol {w}} \mathbb {E} \left[ - \phi \left(\frac {\langle \boldsymbol {w} , \boldsymbol {x} \rangle}{\| \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {w} \|}\right) g (\langle \overline {{\boldsymbol {u}}}, \boldsymbol {z} \rangle) \right] \\ = - \frac {1}{\| \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {w} \|} \left(\mathbf {I} _ {d} - \frac {\boldsymbol {\Sigma} \boldsymbol {w} \boldsymbol {w} ^ {\top}}{\| \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {w} \| ^ {2}}\right) \boldsymbol {\Sigma} ^ {1 / 2} \mathbb {E} [ \phi^ {\prime} (\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle) g (\langle \overline {{\boldsymbol {u}}}, \boldsymbol {z} \rangle) \boldsymbol {z} ] \\ = \frac {\boldsymbol {\Sigma} ^ {1 / 2} \left(\mathbf {I} _ {d} - \overline {{\boldsymbol {w}}} \overline {{\boldsymbol {w}}} ^ {\top}\right)}{\| \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {w} \|} \left\{\zeta_ {\phi , g} (\langle \overline {{\boldsymbol {w}}}, \overline {{\boldsymbol {u}}} \rangle) \overline {{\boldsymbol {u}}} + \psi_ {\phi , g} (\langle \overline {{\boldsymbol {w}}}, \overline {{\boldsymbol {u}}} \rangle) \overline {{\boldsymbol {w}}} \right\} \quad (b y \\ = \frac {\pmb {\Sigma} ^ {1 / 2} (\mathbf {I} _ {d} - \overline {{\pmb {w}}} \overline {{\pmb {w}}} ^ {\top}) \zeta_ {\phi , g} (\langle \overline {{\pmb {w}}}, \overline {{\pmb {u}}} \rangle) \overline {{\pmb {u}}}}{\| \pmb {\Sigma} ^ {1 / 2} \pmb {w} \|}, \\ \end{array}
$$
$$
\psi_ {\phi , g} (\langle \overline {{\boldsymbol {w}}}, \overline {{\boldsymbol {u}}} \rangle) := - \mathbb {E} [ \phi^ {\prime} (\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle) g (\langle \overline {{\boldsymbol {u}}}, \boldsymbol {z} \rangle) \langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle - \phi^ {\prime} (\langle \overline {{\boldsymbol {w}}}, \boldsymbol {z} \rangle) g ^ {\prime} (\langle \overline {{\boldsymbol {u}}}, \boldsymbol {z} \rangle) \langle \overline {{\boldsymbol {w}}}, \overline {{\boldsymbol {u}}} \rangle ] \tag {D.3}
$$
(the above is only a function of  $\langle \overline{w},\overline{u}\rangle$  due to the Hermite expansion).
Given the closed form of the population gradient, the proof of Lemma 3 is immediate by noticing that
$$
\frac {\mathrm {d} \overline {{\boldsymbol {w}}} ^ {t}}{\mathrm {d} t} = \frac {\left(\mathbf {I} _ {d} - \overline {{\boldsymbol {w}}} ^ {t} \overline {{\boldsymbol {w}}} ^ {t} ^ {\top}\right)}{\| \boldsymbol {\Sigma} ^ {1 / 2} \boldsymbol {w} \|} \boldsymbol {\Sigma} ^ {1 / 2} \frac {\mathrm {d} \boldsymbol {w} ^ {t}}{\mathrm {d} t}. \tag {D.4}
$$
Next, we move on to prove Proposition 4.
D.1 Proof of Proposition 4
From Lemma 3, we have
$$
\begin{array}{l} \frac {\mathrm {d} \left\langle \overline {{\boldsymbol {w}}} ^ {t} , \overline {{\boldsymbol {u}}} \right\rangle}{\mathrm {d} t} = - \zeta_ {\phi , g} \left(\left\langle \overline {{\boldsymbol {w}}} ^ {t}, \overline {{\boldsymbol {u}}} \right\rangle\right) \| \boldsymbol {\Sigma} ^ {1 / 2} \overline {{\boldsymbol {u}}} _ {\perp} ^ {t} \| ^ {2} \\ \geq c \lambda_ {\min } (\boldsymbol {\Sigma}) \left\langle \overline {{\boldsymbol {w}}} ^ {t}, \overline {{\boldsymbol {u}}} \right\rangle^ {s - 1} \left(1 - \left\langle \overline {{\boldsymbol {w}}} ^ {t}, \overline {{\boldsymbol {u}}} \right\rangle^ {2}\right), \\ \end{array}
$$
where  $\overline{\pmb{u}}_{\perp}^{t} \coloneqq \overline{\pmb{u}} - \langle \overline{\pmb{w}}^{t}, \overline{\pmb{u}} \rangle \overline{\pmb{w}}^{t}$ . The above inequality and the fact that  $\langle \overline{\pmb{w}}^{0}, \overline{\pmb{u}} \rangle > 0$  imply that  $\langle \overline{\pmb{w}}^{t}, \overline{\pmb{u}} \rangle$  is non-decreasing in time. Let
$$
T _ {1} := \sup  \left\{t > 0: \left\langle \overline {{\boldsymbol {w}}} ^ {t}, \overline {{\boldsymbol {u}}} \right\rangle <   1 / 2 \right\}.
$$
Then, on  $t\in [0,T_1]$  , we have
$$
\frac {\mathrm {d} \langle \overline {{\boldsymbol {w}}} ^ {t} , \overline {{\boldsymbol {u}}} \rangle}{\mathrm {d} t} \geq \frac {3 c \lambda_ {\min } (\boldsymbol {\Sigma})}{4} \langle \overline {{\boldsymbol {w}}} ^ {t}, \overline {{\boldsymbol {u}}} \rangle^ {s - 1},
$$
and integration yields
$$
T _ {1} \leq 0 \vee \frac {4}{3 c \lambda_ {\min } (\boldsymbol {\Sigma})} \left\{ \begin{array}{l l} 1 / 2 - \langle \overline {{\boldsymbol {w}}} ^ {0}, \overline {{\boldsymbol {u}}} \rangle & s = 1 \\ \ln (1 / (2 \langle \overline {{\boldsymbol {w}}} ^ {0}, \overline {{\boldsymbol {u}}} \rangle)) & s = 2. \\ \frac {1}{s - 2} \big ((1 / \langle \overline {{\boldsymbol {w}}} ^ {0}, \overline {{\boldsymbol {u}}} \rangle) ^ {s - 2} - 2 ^ {s - 2} \big) & s > 2 \end{array} \right.
$$
Therefore,  $T_{1}\lesssim \tau_{s}(\langle \overline{\boldsymbol{w}}^{0},\overline{\boldsymbol{u}}\rangle) / \lambda_{\min}(\boldsymbol {\Sigma})$
For  $t > T_{1}$ , we have
$$
\frac {\mathrm {d} \left\langle \overline {{\boldsymbol {w}}} ^ {t} , \overline {{\boldsymbol {u}}} \right\rangle}{\mathrm {d} t} \geq \frac {c \lambda_ {\min } (\boldsymbol {\Sigma})}{2 ^ {s - 1}} (1 - \left\langle \overline {{\boldsymbol {w}}} ^ {t}, \overline {{\boldsymbol {u}}} \right\rangle^ {2}).
$$
$$
T _ {2} = \sup  \left\{t > 0: \left\langle \overline {{\boldsymbol {w}}} ^ {t}, \overline {{\boldsymbol {u}}} \right\rangle <   1 - \varepsilon \right\}.
$$
Once again, integration implies
$$
T _ {2} \leq T _ {1} + \frac {2 ^ {s - 2}}{c \lambda_ {\min } (\Sigma)} \ln (2 / (3 \varepsilon)),
$$
D.2 Preliminary Lemmas for proving Theorem 5
We first introduce a number of concentration (and anti-concentration) lemmas that will be useful for proving Theorem 5.
Lemma 16. Suppose  $\{\pmb{z}^{(i)}\}_{i = 1}^n\stackrel {i.i.d.}{\sim}\mathcal{N}(0,\mathbf{I}_d)$ , and  $g:\mathbb{R}\to \mathbb{R}$  satisfies  $|g(\cdot)|\leq C(1 + |\cdot |^p)$  for some  $C > 0$  and  $p\geq 1$ . Additionally, suppose  $\{\epsilon^{(i)}\}_{i = 1}$  are i.i.d.  $\sigma$ -sub-Gaussian zero-mean noise independent of  $\{\pmb{z}^{(i)}\}_{i = 1}^n$ . Let  $y^{(i)}\coloneqq g(\langle \overline{\pmb{u}},\pmb{z}^{(i)}\rangle) + \epsilon^{(i)}$  for some  $\overline{\mathbf{u}}\in \mathbb{S}^{d - 1}$ . Then, for any  $q > 0$  with probability at least  $1 - 4d^{-q}$ , we have
$$
\left| y ^ {(i)} \right| \leq C + C (2 \ln (n d ^ {q})) ^ {p / 2} + \sigma \sqrt {2 \ln (n d ^ {q})} \lesssim \ln (n d ^ {q}) ^ {p / 2},
$$
for all  $i$ .
Proof. Notice that  $\langle \overline{\mathbf{u}}, z^{(i)} \rangle \sim \mathcal{N}(0, 1)$ , thus
$$
\mathbb {P} \left(\left| \left\langle \overline {{\boldsymbol {u}}}, \boldsymbol {z} ^ {(i)} \right\rangle \right| \geq \sqrt {2 \ln (n d ^ {q})}\right) \leq 2 n ^ {- 1} d ^ {- q}.
$$
Similarly, by the sub-Gaussian and zero-mean property of  $\epsilon^{(i)}$ ,
$$
\mathbb {P} \left(\left| \epsilon^ {(i)} \right| \geq \sigma \sqrt {2 \ln (n d ^ {q})}\right) \leq 2 n ^ {- 1} d ^ {- q}.
$$
Thus, by a union bound, we have
$$
\left| \left\langle \overline {{u}}, z ^ {(i)} \right\rangle \right| \leq \sqrt {2 \ln (n d ^ {q})} \quad \text {a n d} \quad \left| \epsilon^ {(i)} \right| \leq \sigma \sqrt {2 \ln (n d ^ {q})}, \quad \text {f o r a l l} 1 \leq i \leq n,
$$
with probability at least  $1 - 4d^{-q}$ . Using the upper bound on  $|g|$  finishes the proof.
Lemma 17. Suppose  $\{\overline{z}^{(i)}\}_{i = 1}^n$  are i.i.d. samples drawn uniformly from  $\mathbb{S}^{d - 1}$ . Then,
$$
\mathbb {P} \left(\sup  _ {\overline {{\boldsymbol {w}}} \in \mathbb {S} ^ {d - 1}} \sum_ {i = 1} ^ {n} \mathbf {1} \left(\left| \left\langle \overline {{\boldsymbol {w}}}, \overline {{\boldsymbol {z}}} ^ {(i)} \right\rangle \right| \leq \frac {3 \sqrt {d}}{8 n}\right) \geq 3 d \left(2 + \ln (8 n / \sqrt {d})\right)\right) \leq e ^ {- d}.
$$
Proof. Fix some  $\epsilon \in (0,1)$ . Let  $N_{\epsilon}$  be a minimal  $\epsilon$ -covering of  $\mathbb{S}^{d-1}$ . Let  $\hat{\boldsymbol{w}}$  be the projection of  $\overline{\boldsymbol{w}}$  onto  $N_{\epsilon}$ . Notice that by the triangle inequality and the union bound
$$
\begin{array}{l} \mathbb {P} \left(\sup  _ {\overline {{\boldsymbol {w}}} \in \mathbb {S} ^ {d - 1}} \sum_ {i = 1} ^ {n} \mathbf {1} \left(\left| \left\langle \overline {{\boldsymbol {w}}}, \overline {{\boldsymbol {z}}} ^ {(i)} \right\rangle \right| \leq \epsilon\right) \geq \alpha\right) \leq \mathbb {P} \left(\sup  _ {\hat {\boldsymbol {w}} \in N _ {\epsilon}} \sum_ {i = 1} ^ {n} \mathbf {1} \left(\left| \left\langle \hat {\boldsymbol {w}}, \overline {{\boldsymbol {z}}} ^ {(i)} \right\rangle \right| \leq 2 \epsilon\right) \geq \alpha\right) \\ \leq | N _ {\epsilon} | \mathbb {P} \left(\sum_ {i = 1} ^ {n} \mathbf {1} \left(\left| \left\langle \hat {\boldsymbol {w}}, \overline {{\boldsymbol {z}}} ^ {(i)} \right\rangle \right| \leq 2 \epsilon\right) \geq \alpha\right) \\ \leq (3 / \epsilon) ^ {d} \mathbb {P} \left(\sum_ {i = 1} ^ {n} \mathbf {1} \left(\left| \left\langle \hat {\boldsymbol {w}}, \overline {{\boldsymbol {z}}} ^ {(i)} \right\rangle \right| \leq 2 \epsilon\right) \geq \alpha\right). \\ \end{array}
$$
Moreover, due to [BBSS22, Lemma A.7],
$$
\mathbb {E} [ \mathbf {1} (\left| \langle \hat {\boldsymbol {w}}, \bar {z} \rangle \right| \leq 2 \epsilon) ] = \mathbb {P} (\langle \hat {\boldsymbol {w}}, \bar {z} \rangle \leq 2 \epsilon) \leq 8 \sqrt {d} \epsilon .
$$
Choose  $\epsilon = \frac{3\sqrt{d}}{8n}$ . By Lemma 25
$$
\mathbb {P} \left(\sum_ {i = 1} ^ {n} \mathbf {1} \left(\left| \left\langle \hat {\boldsymbol {w}}, \bar {\boldsymbol {z}} ^ {(i)} \right\rangle \right| \leq 2 \epsilon\right) \geq 3 d \left(2 + \ln (8 n / \sqrt {d})\right)\right) \leq (3 / \epsilon) ^ {d} e ^ {- d},
$$
We summarize the above statements into a "good event", as characterized by the following lemma.
Lemma 18. Let  $\{\pmb{z}^{(i)}\}_{i=1}^n \stackrel{i.i.d.}{\sim} \mathcal{N}(0, \mathbf{I}_d)$ , and  $\overline{\pmb{z}}^{(i)} \coloneqq \pmb{z}^{(i)} / \| \pmb{z}^{(i)}\|$  for every  $i$ . We say event  $\mathcal{G}$  occurs whenever:
1.  $|y|^{(i)} \lesssim \ln (nd^q)^{p/2}$  for all  $1 \leq i \leq n$ .
2.  $\sup_{\overline{\mathbf{w}}\in \mathbb{S}^{d - 1}}\sum_{i = 1}^{n}\mathbf{1}\Big(\big{\langle}\overline{\mathbf{w}},\overline{\mathbf{z}}^{(i)}\big{\rangle}\lesssim \sqrt{d} /n\Big)\lesssim d\ln (n / \sqrt{d}).$
3.  $\lambda_{\max}\left(\frac{1}{n}\sum_{i = 1}^{n}z^{(i)}z^{(i)^{\top}}\right) - 1\lesssim \sqrt{d / n}.$
4.  $1 - \lambda_{\min}\Big(\frac{1}{n}\sum_{i = 1}^{n}\boldsymbol{z}^{(i)}\boldsymbol{z}^{(i)^{\top}}\Big)\lesssim \sqrt{d / n}.$
For  $n \gtrsim d$ , event  $\mathcal{G}$  occurs with probability at least  $1 - \mathcal{O}(d^{-q})$ .