SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| M _ {1} - S _ {1} \right| \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2}. \tag {F.56}
$$
In order to bound  $|M_2 - S_2|$ , an almost identical, but simpler, argument can be used, with  $y(x)$  in the place of  $f_{\tilde{\rho}_t}(x)$  or  $f_{\hat{\rho}_{\eta t}}(x)$ . In the modified argument,  $\tilde{u}_t^\prime$ ,  $\hat{u}_{\eta t}^{\prime}$  and  $\bar{u}_{\eta t}^{\prime}$  will all be replaced with  $e_1$ . Thus, in the analysis of Eq. (F.40), we can replace  $\bar{u}_{\eta t}^{\prime}$  with  $e_1$  and assume that  $i_2 = 0$  (since otherwise this term becomes 0 due to the  $\delta_{\eta t}^{\prime}\cdot x$  factor) — we can then proceed similarly to the subcase where  $i_2\leq 1$  in the analysis of Eq. (F.40). Additionally, the term analogous to Eq. (F.47) does not arise in the case where  $f_{\tilde{\rho}_t}(x)$  and  $f_{\hat{\rho}_{\eta t}}(x)$  are replaced by  $y(x)$ , since  $\tilde{u}_t^\prime$  and  $\hat{u}_{\eta t}^{\prime}$  are both replaced by  $e_1$ , so the corresponding term is 0. We also note that the Legendre coefficients of  $y(x)$  have absolute values which are less than those of  $\sigma$ . Thus, we have
$$
\left| M _ {1} - S _ {1} \right| + \left| M _ {2} - S _ {2} \right| \leq \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2}. \tag {F.57}
$$
Bounding the Expectation To complete the proof, it suffices to bound  $|M_1 - M_2|$ . We have that
$$
M _ {1} - M _ {2} = \nabla_ {\tilde {u} _ {t}} L (\tilde {\rho} _ {t}) \cdot \tilde {\delta} _ {t} - \nabla_ {\hat {u} _ {\eta t}} L (\hat {\rho} _ {\eta t}) \cdot \tilde {\delta} _ {t} \tag {F.58}
$$
and by Lemma F.1, we have
$$
\begin{array}{l} \left| M _ {1} - M _ {2} \right| \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \Big (\underset {\left. \left(\tilde {u} _ {t} ^ {\prime}, \hat {u} _ {\eta t} ^ {\prime}\right) \sim \left(\tilde {\rho} _ {t}, \hat {\rho} _ {\eta t}\right)\right)} {\mathbb {E}} \left[ \| \tilde {u} _ {t} ^ {\prime} - \hat {u} _ {\eta t} ^ {\prime} \| _ {2} ^ {2} \right] ^ {1 / 2} + \| \tilde {u} _ {t} - \hat {u} _ {\eta t} \| _ {2} \Big) \| \tilde {\delta} _ {t} \| _ {2} (F.59) \\ \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2} (F.60) \\ \end{array}
$$
Combining this with our bound on  $|M_1 - S_1| + |M_2 - S_2|$ , we find that
$$
\left| \left(\operatorname {g r a d} _ {\tilde {u} _ {t}} \widehat {L} \left(\tilde {\rho} _ {t}\right) - \operatorname {g r a d} _ {\hat {u} _ {\eta t}} \widehat {L} \left(\hat {\rho} _ {\eta t}\right)\right) \cdot \tilde {\delta} _ {t} \right| \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2} \tag {F.61}
$$
Lemma F.3. Suppose we are in the setting of Theorem 3.5. Suppose  $s, t \geq 0$ . Then,
$$
\left\| \operatorname {g r a d} _ {\hat {u} _ {t}} \widehat {L} \left(\hat {\rho} _ {t}\right) - \operatorname {g r a d} _ {\hat {u} _ {s}} \widehat {L} \left(\hat {\rho} _ {s}\right) \right\| _ {2} \leq \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) d ^ {4} \cdot \max  _ {i \in [ m ]} \| \hat {u} _ {t} (\chi_ {i}) - \hat {u} _ {s} (\chi_ {i}) \| _ {2} \tag {F.62}
$$
$$
\begin{array}{l} \left\| \operatorname {g r a d} _ {\hat {u} _ {t}} \widehat {L} \left(\hat {\rho} _ {t}\right) - \operatorname {g r a d} _ {\hat {u} _ {s}} \widehat {L} \left(\hat {\rho} _ {s}\right) \right\| _ {2} (F.63) \\ = \| \left(I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}\right) \nabla_ {\hat {u} _ {t}} \widehat {L} (\hat {\rho} _ {t}) - \left(I - \hat {u} _ {s} \hat {u} _ {s} ^ {\top}\right) \nabla_ {\hat {u} _ {s}} \widehat {L} (\hat {\rho} _ {s}) \| _ {2} (F.64) \\ = \left\| \left(I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}\right) \cdot \frac {1}{n} \sum_ {i = 1} ^ {n} \left(f _ {\hat {\rho} _ {t}} \left(x _ {i}\right) - y \left(x _ {i}\right)\right) \sigma^ {\prime} \left(\hat {u} _ {t} ^ {\top} x _ {i}\right) x _ {i} \right. (F.65) \\ \left. - \left(I - \hat {u} _ {s} \hat {u} _ {s} ^ {\top}\right) \cdot \frac {1}{n} \sum_ {i = 1} ^ {n} \left(f _ {\hat {\rho} _ {s}} \left(x _ {i}\right) - y \left(x _ {i}\right)\right) \sigma^ {\prime} \left(\hat {u} _ {s} ^ {\top} x _ {i}\right) x _ {i} \right\| _ {2} (F.66) \\ \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \| \left(f _ {\hat {\rho} _ {t}} \left(x _ {i}\right) - y \left(x _ {i}\right)\right) \sigma^ {\prime} \left(\hat {u} _ {t} ^ {\top} x _ {i}\right) \left(I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}\right) x _ {i} (F.67) \\ - \left(f _ {\hat {\rho} _ {s}} \left(x _ {i}\right) - y \left(x _ {i}\right)\right) \sigma^ {\prime} \left(\hat {u} _ {s} ^ {\top} x _ {i}\right) \left(I - \hat {u} _ {s} \hat {u} _ {s} ^ {\top}\right) x _ {i} \| _ {2}. (F.68) \\ \end{array}
$$
We bound the right-hand side by several applications of the triangle inequality. First, observe that for any  $x \in \mathbb{S}^{d-1}$ , we have
$$
\begin{array}{l} \left| f _ {\hat {\rho} _ {t}} (x) - f _ {\hat {\rho} _ {s}} (x) \right| \leq \frac {1}{m} \sum_ {j = 1} ^ {m} \left| \sigma \left(\hat {u} _ {t} \left(\chi_ {i}\right) \cdot x\right) - \sigma \left(\hat {u} _ {s} \left(\chi_ {i}\right) \cdot x\right) \right| \tag {F.69} \\ \lesssim \frac {1}{m} \sum_ {j = 1} ^ {m} \left(\left| \hat {\sigma} _ {2, d} \right| \sqrt {N _ {2 , d}} + \left| \hat {\sigma} _ {4, d} \right| \sqrt {N _ {4 , d}}\right) \left| \hat {u} _ {t} \left(\chi_ {i}\right) \cdot x - \hat {u} _ {s} \left(\chi_ {i}\right) \cdot x \right| \\ \end{array}
$$
(By Lemma E.12)
$$
\begin{array}{l} \lesssim \frac {\left(\left| \hat {\sigma} _ {2 , d} \right| \sqrt {N _ {2 , d}} + \left| \hat {\sigma} _ {4 , d} \right| \sqrt {N _ {4 , d}}\right)}{m} \sum_ {j = 1} ^ {m} \| \hat {u} _ {t} (\chi_ {i}) - \hat {u} _ {s} (\chi_ {i}) \| _ {2} (F.70) \\ \lesssim \left(| \hat {\sigma} _ {2, d} | \sqrt {N _ {2 , d}} + | \hat {\sigma} _ {4, d} | \sqrt {N _ {4 , d}}\right) \max  _ {i \in [ m ]} \| \hat {u} _ {t} (\chi_ {i}) - \hat {u} _ {s} (\chi_ {i}) \| _ {2}. (F.71) \\ \end{array}
$$
Additionally, for  $\hat{u}_t$  and  $\hat{u}_s$  with the same initialization  $\chi$ , we have
$$
\left| \sigma^ {\prime} \left(\hat {u} _ {t} ^ {\top} x\right) - \sigma^ {\prime} \left(\hat {u} _ {s} ^ {\top} x\right) \right| \lesssim \left(\left| \hat {\sigma} _ {2, d} \right| \sqrt {N _ {2 , d}} + \left| \hat {\sigma} _ {4, d} \right| \sqrt {N _ {4 , d}}\right) \cdot \left| \hat {u} _ {t} ^ {\top} x - \hat {u} _ {s} ^ {\top} x \right| \tag {By Lemma E.12}
$$
$$
\lesssim \left(| \hat {\sigma} _ {2, d} | \sqrt {N _ {2 , d}} + | \hat {\sigma} _ {4, d} | \sqrt {N _ {4 , d}}\right) \| \hat {u} _ {t} - \hat {u} _ {s} \| _ {2}. \tag {F.72}
$$
$$
\lesssim \left(\left| \hat {\sigma} _ {2, d} \right| \sqrt {N _ {2 , d}} + \left| \hat {\sigma} _ {4, d} \right| \sqrt {N _ {4 , d}}\right) \cdot \max  _ {i \in [ m ]} \| \hat {u} _ {t} (\chi_ {i}) - \hat {u} _ {s} (\chi_ {i}) \| _ {2}. \tag {F.73}
$$
Also, for  $\hat{u}_t$  and  $\hat{u}_s$  with the same initialization  $\chi$ , we have
$$
\begin{array}{l} \left\| \left(I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}\right) x _ {i} - \left(I - \hat {u} _ {s} \hat {u} _ {s} ^ {\top}\right) x _ {i} \right\| _ {2} = \left\| \left(\hat {u} _ {t} \cdot x _ {i}\right) \hat {u} _ {t} - \left(\hat {u} _ {s} \cdot x _ {i}\right) \hat {u} _ {s} \right\| _ {2} (F.74) \\ \leq \left\| \left(\hat {u} _ {t} \cdot x _ {i} - \hat {u} _ {s} \cdot x _ {i}\right) \hat {u} _ {t} \right\| _ {2} + \left| \hat {u} _ {s} \cdot x _ {i} \right| \| \hat {u} _ {t} - \hat {u} _ {s} \| _ {2} (F.75) \\ \lesssim \max  _ {i \in [ m ]} \| \hat {u} _ {t} \left(\chi_ {i}\right) - \hat {u} _ {s} \left(\chi_ {i}\right) \| _ {2}. (F.76) \\ \end{array}
$$
Finally, for  $x\in \mathbb{S}^{d - 1}$  and any distribution  $\rho$  on  $\mathbb{S}^{d - 1}$ , we have
$$
\left| f _ {\rho} (x) - y (x) \right| \lesssim \left| \hat {\sigma} _ {2, d} \right| \sqrt {N _ {2 , d}} + \left| \hat {\sigma} _ {4, d} \right| \sqrt {N _ {4 , d}} \tag {F.77}
$$
since we can bound  $|f_{\rho}(x)|$  using Lemma E.12, and because (ignoring  $\hat{h}_{0,d}$  since it is equal to  $\hat{\sigma}_{0,d}$ )  $y(x) = \gamma_2\hat{\sigma}_{2,d}P_{2,d} + \gamma_4\hat{\sigma}_{4,d}P_{4,d}$ , and using Assumption 3.2.
Thus, we can simplify the right-hand side of Eq. (F.68) using the triangle inequality, and combining Eq. (F.71), Eq. (F.73), Eq. (F.76) and Eq. (F.77):
$$
\left\| \left(f _ {\hat {\rho} _ {t}} \left(x _ {i}\right) - y \left(x _ {i}\right)\right) \sigma^ {\prime} \left(\hat {u} _ {t} ^ {\top} x _ {i}\right) \left(I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}\right) x _ {i} - \left(f _ {\hat {\rho} _ {s}} \left(x _ {i}\right) - y \left(x _ {i}\right)\right) \sigma^ {\prime} \left(\hat {u} _ {s} ^ {\top} x _ {i}\right) \left(I - \hat {u} _ {s} \hat {u} _ {s} ^ {\top}\right) x _ {i} \right\| _ {2} \tag {F.78}
$$
$$
\begin{array}{l} \leq \left| f _ {\hat {\rho} _ {t}} \left(x _ {i}\right) - f _ {\hat {\rho} _ {s}} \left(x _ {i}\right) \right| \| \sigma^ {\prime} \left(\hat {u} _ {t} ^ {\top} x _ {i}\right) \left(I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}\right) x _ {i} \| _ {2} (F.79) \\ + \left| f _ {\hat {\rho} _ {s}} \left(x _ {i}\right) - y \left(x _ {i}\right) \right| \left| \sigma^ {\prime} \left(\hat {u} _ {t} ^ {\top} x _ {i}\right) - \sigma^ {\prime} \left(\hat {u} _ {s} ^ {\top} x _ {i}\right) \right| \| \left(I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}\right) x _ {i} \| _ {2} (F.80) \\ + \left| f _ {\hat {\rho} _ {s}} \left(x _ {i}\right) - y \left(x _ {i}\right) \right| \left| \sigma^ {\prime} \left(\hat {u} _ {s} ^ {\top} x _ {i}\right) \right| \| \left(I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}\right) x _ {i} - \left(I - \hat {u} _ {s} \hat {u} _ {s} ^ {\top}\right) x _ {i} \| _ {2} (F.81) \\ \end{array}
$$
$$
\begin{array}{l} \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \cdot \max  _ {i \in [ m ]} \| \hat {u} _ {t} (\chi_ {i}) - \hat {u} _ {s} (\chi_ {i}) \| _ {2} (F.82) \\ + \left| f _ {\hat {\rho} _ {s}} \left(x _ {i}\right) - y \left(x _ {i}\right) \right| \left| \sigma^ {\prime} \left(\hat {u} _ {t} ^ {\top} x _ {i}\right) - \sigma^ {\prime} \left(\hat {u} _ {s} ^ {\top} x _ {i}\right) \right| \| \left(I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}\right) x _ {i} \| _ {2} (F.83) \\ + \left| f _ {\hat {\rho} _ {s}} \left(x _ {i}\right) - y \left(x _ {i}\right) \right| \left| \sigma^ {\prime} \left(\hat {u} _ {s} ^ {\top} x _ {i}\right) \right| \| \left(I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}\right) x _ {i} - \left(I - \hat {u} _ {s} \hat {u} _ {s} ^ {\top}\right) x _ {i} \| _ {2} (ByEq.(F.71), bounding) \\ \end{array}
$$
$$
\begin{array}{l} \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \cdot \max  _ {i \in [ m ]} \| \hat {u} _ {t} (\chi_ {i}) - \hat {u} _ {s} (\chi_ {i}) \| _ {2} (F.84) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \cdot \max  _ {i \in [ m ]} \| \hat {u} _ {t} (\chi_ {i}) - \hat {u} _ {s} (\chi_ {i}) \| _ {2} (F.85) \\ + | f _ {\hat {\rho} _ {s}} (x _ {i}) - y (x _ {i}) | | \sigma^ {\prime} (\hat {u} _ {s} ^ {\top} x _ {i}) | | | (I - \hat {u} _ {t} \hat {u} _ {t} ^ {\top}) x _ {i} - (I - \hat {u} _ {s} \hat {u} _ {s} ^ {\top}) x _ {i} | | _ {2} \quad \text {(B y E q . (F . 7 7) a n d E q . (F . 7 3))} \\ \end{array}
$$
$$
\begin{array}{l} \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \cdot \max  _ {i \in [ m ]} \| \hat {u} _ {t} \left(\chi_ {i}\right) - \hat {u} _ {s} \left(\chi_ {i}\right) \| _ {2} (F.86) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \cdot \max  _ {i \in [ m ]} \| \hat {u} _ {t} (\chi_ {i}) - \hat {u} _ {s} (\chi_ {i}) \| _ {2} (F.87) \\ + \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \cdot \max  _ {i \in [ m ]} \| \hat {u} _ {t} (\chi_ {i}) - \hat {u} _ {s} (\chi_ {i}) \| _ {2} \\ \end{array}
$$
(By Eq. (F.76), Eq. (F.77) and Lemma E.12)
$$
\lesssim \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \cdot \max  _ {i \in [ m ]} \| \hat {u} _ {t} \left(\chi_ {i}\right) - \hat {u} _ {s} \left(\chi_ {i}\right) \| _ {2}. \tag {F.88}
$$
Therefore, Eq. (F.68) gives
$$
\left\| \operatorname {g r a d} _ {\hat {u} _ {t}} \widehat {L} \left(\hat {\rho} _ {t}\right) - \operatorname {g r a d} _ {\hat {u} _ {s}} \widehat {L} \left(\hat {\rho} _ {s}\right) \right\| _ {2} \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) \cdot \max  _ {i \in [ m ]} \| \hat {u} _ {t} (\chi_ {i}) - \hat {u} _ {s} (\chi_ {i}) \| _ {2} \tag {F.89}
$$
Proof of Theorem 3.5. To show that projected gradient descent achieves low population loss, we show that it does not diverge far from projected gradient descent. Suppose  $\hat{\rho}_0$  and  $\tilde{\rho}_0$  are both equal to  $\mathrm{unif}(\chi_1,\dots ,\chi_m)$ . Then, by induction over  $t$ , we bound the difference between  $\tilde{u}_t(\chi)$  and  $\hat{u}_{\eta t}(\chi)$  for all  $\chi \in \{\chi_1,\ldots ,\chi_m\}$ . At  $t = 0$ , we have  $\tilde{u}_0(\chi) = \hat{u}_0(\chi)$ . Let  $t$  be a nonnegative integer, and assume that for all  $s\leq t$  we have  $\max_{\chi}\| \tilde{\delta}_t(\chi)\| _2\leq \frac{1}{\sqrt{d}}$ .
By the definition of projected gradient descent, we have
$$
\tilde {u} _ {t + 1} (\chi) = \frac {\tilde {u} _ {t} (\chi) - \eta \cdot \operatorname {g r a d} _ {\tilde {u} _ {t} (\chi)} \widehat {L} \left(\tilde {\rho} _ {t}\right)}{\| \tilde {u} _ {t} (\chi) - \eta \cdot \operatorname {g r a d} _ {\tilde {u} _ {t} (\chi)} \widehat {L} \left(\tilde {\rho} _ {t}\right) \| _ {2}}. \tag {F.90}
$$
Note that  $\| \tilde{u}_t - \eta \cdot \mathrm{grad}_{\tilde{u}_t} \widehat{L}(\tilde{\rho}_t) \|_2 \geq \| \tilde{u}_t \|_2$ , since  $\mathrm{grad}_{\tilde{u}_t} \widehat{L}(\tilde{\rho}_t)$  is orthogonal to  $\tilde{u}_t$ . Thus, letting  $\Pi$  denote the projection onto the  $\ell_2$  unit ball, we have
$$
\begin{array}{l} \left\| \tilde {u} _ {t + 1} - \hat {u} _ {\eta (t + 1)} \right\| _ {2} = \left\| \Pi \left(\tilde {u} _ {t} - \eta \cdot \operatorname {g r a d} _ {\tilde {u} _ {t}} \widehat {L} \left(\tilde {\rho} _ {t}\right)\right) - \Pi \left(\hat {u} _ {\eta (t + 1)}\right) \right\| _ {2} (F.91) \\ \leq \left\| \tilde {u} _ {t} - \eta \cdot \operatorname {g r a d} _ {\tilde {u} _ {t}} \widehat {L} \left(\tilde {\rho} _ {t}\right) - \hat {u} _ {\eta (t + 1)} \right\| _ {2}. (F.92) \\ \end{array}
$$
$$
\hat {u} _ {\eta (t + 1)} (\chi) = \hat {u} _ {\eta t} (\chi) - \int_ {\eta t} ^ {\eta (t + 1)} \operatorname {g r a d} _ {\hat {u} _ {s} (\chi)} \widehat {L} (\hat {\rho} _ {s}) d s. \tag {F.93}
$$
$$
\begin{array}{l} \left\| \tilde {u} _ {t} - \eta \cdot \operatorname {g r a d} _ {\tilde {u} _ {t}} \widehat {L} \left(\tilde {\rho} _ {t}\right) - \hat {u} _ {\eta (t + 1)} \right\| _ {2} ^ {2} = \left\| \tilde {u} _ {t} - \eta \cdot \operatorname {g r a d} _ {\tilde {u} _ {t}} \widehat {L} \left(\tilde {\rho} _ {t}\right) - \hat {u} _ {\eta t} + \int_ {\eta t} ^ {\eta (t + 1)} \operatorname {g r a d} _ {\hat {u} _ {s}} \widehat {L} \left(\hat {\rho} _ {s}\right) d s \right\| _ {2} ^ {2} (F.94) \\ = \| \tilde {u} _ {t} - \hat {u} _ {\eta t} \| _ {2} ^ {2} + \left\| \eta \cdot \operatorname {g r a d} _ {\tilde {u} _ {t}} \widehat {L} \left(\tilde {\rho} _ {t}\right) - \int_ {\eta t} ^ {\eta (t + 1)} \operatorname {g r a d} _ {\hat {u} _ {s}} \widehat {L} \left(\hat {\rho} _ {s}\right) d s \right\| _ {2} ^ {2} (F.95) \\ + 2 \left(\tilde {u} _ {t} - \hat {u} _ {\eta t}\right) \cdot \int_ {\eta t} ^ {\eta (t + 1)} \left(\operatorname {g r a d} _ {\tilde {u} _ {t}} \widehat {L} \left(\tilde {\rho} _ {t}\right) - \operatorname {g r a d} _ {\hat {u} _ {s}} \widehat {L} \left(\hat {\rho} _ {s}\right)\right) d s (F.96) \\ = \| \tilde {\delta} _ {t} \| _ {2} ^ {2} + O \left(\eta^ {2} \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) ^ {2} d ^ {8}\right) (F.97) \\ + 2 \eta \tilde {\delta} _ {t} \cdot \left(\operatorname {g r a d} _ {\tilde {u} _ {t}} \widehat {L} \left(\tilde {\rho} _ {t}\right) - \operatorname {g r a d} _ {\hat {u} _ {\eta t}} \widehat {L} \left(\hat {\rho} _ {\eta t}\right)\right) (F.98) \\ + 2 \tilde {\delta} _ {t} \cdot \int_ {\eta t} ^ {\eta (t + 1)} \left(\operatorname {g r a d} _ {\hat {u} _ {\eta t}} \widehat {L} \left(\hat {\rho} _ {\eta t}\right) - \operatorname {g r a d} _ {\hat {u} _ {s}} \widehat {L} \left(\hat {\rho} _ {s}\right) d s \right. (By Lemma E.14) \\ \leq \left\| \tilde {\delta} _ {t} \right\| _ {2} ^ {2} + O \left(\eta^ {2} \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) ^ {2} d ^ {8}\right) (F.99) \\ + O \left(\eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2}\right) (F.100) \\ + 2 \tilde {\delta} _ {t} \cdot \int_ {\eta t} ^ {\eta (t + 1)} \left(\operatorname {g r a d} _ {\hat {u} _ {\eta t}} \widehat {L} (\hat {\rho} _ {\eta t}) - \operatorname {g r a d} _ {\hat {u} _ {s}} \widehat {L} (\hat {\rho} _ {s}) d s\right) (By Lemma F.2) \\ \end{array}
$$
Further simplifying using the Cauchy-Schwarz inequality, we have
$$
\left\| \tilde {u} _ {t} - \eta \cdot \operatorname {g r a d} _ {\tilde {u} _ {t}} \widehat {L} \left(\tilde {\rho} _ {t}\right) - \hat {u} _ {\eta (t + 1)} \right\| _ {2} ^ {2} \leq \left\| \tilde {\delta} _ {t} \right\| _ {2} ^ {2} + O \left(\eta^ {2} \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) ^ {2} d ^ {8}\right) \tag {F.101}
$$
$$
+ O \left(\eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2}\right) \tag {F.102}
$$
$$
+ 2 \eta \| \tilde {\delta} _ {t} \| _ {2} \max  _ {s \in [ \eta t, \eta (t + 1) ]} \| \operatorname {g r a d} _ {\hat {u} _ {\eta t}} \widehat {L} (\hat {\rho} _ {\eta t}) - \operatorname {g r a d} _ {\hat {u} _ {s}} \widehat {L} (\hat {\rho} _ {s}) \| _ {2} \tag {F.103}
$$
$$
\leq \left\| \tilde {\delta} _ {t} \right\| _ {2} ^ {2} + O \left(\eta^ {2} \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) ^ {2} d ^ {8}\right) \tag {F.104}
$$
$$
+ O \left(\eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2}\right) \tag {F.105}
$$
$$
+ 2 \eta \| \tilde {\delta} _ {t} \| _ {2} \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) d ^ {4} \tag {F.106}
$$
$$
\cdot \max  _ {s \in [ \eta t, \eta (t + 1) ]} \max  _ {i \in [ m ]} \| \hat {u} _ {s} (\chi_ {i}) - \hat {u} _ {\eta t} (\chi_ {i}) \| _ {2}
$$
(By Lemma F.3)
$$
\leq \left\| \tilde {\delta} _ {t} \right\| _ {2} ^ {2} + O \left(\eta^ {2} \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) ^ {2} d ^ {8}\right) \tag {F.107}
$$
$$
+ O \left(\eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2}\right) \tag {F.108}
$$
$$
+ 2 \eta \| \bar {\delta} _ {t} \| _ {2} (\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}) d ^ {4} \cdot \eta (\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}) d ^ {4}
$$
(By Lemma E.14)
$$
\leq \left\| \tilde {\delta} _ {t} \right\| _ {2} ^ {2} + O \left(\eta^ {2} \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) ^ {2} d ^ {8}\right) \tag {F.109}
$$
$$
+ O \left(\eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2}\right). \tag {F.110}
$$
$$
\max  _ {\chi} \| \tilde {\delta} _ {t + 1} (\chi) \| _ {2} ^ {2} \leq \left(1 + O \left(\eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right)\right)\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2} + O \left(\eta^ {2} \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) ^ {2} d ^ {8}\right) (F. 1 1 1)
$$
which means that for some universal constant  $C$ ,
$$
\max  _ {\chi} \| \tilde {\delta} _ {t + 1} (\chi) \| _ {2} ^ {2} \lesssim \sum_ {i = 0} ^ {t} \left(1 + C \eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right)\right) ^ {i} \cdot \eta^ {2} \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) ^ {2} d ^ {8} \tag {F.112}
$$
$$
\lesssim \eta^ {2} \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) ^ {2} d ^ {8} \cdot \frac {\left(1 + C \eta \left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right)\right) ^ {t + 1} - 1}{\left(1 + C \eta \left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right)\right) - 1} \tag {F.113}
$$
$$
\lesssim \eta^ {2} \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) ^ {2} d ^ {8} \cdot \frac {\left(1 + C \eta \left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right)\right) ^ {t + 1} - 1}{C \eta \left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right)} \tag {F.114}
$$
$$
\lesssim \eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) d ^ {8} \cdot \left(1 + C \eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right)\right) ^ {t + 1} \tag {F.115}
$$
$$
\lesssim \eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) d ^ {8} \cdot e ^ {C \eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) (t + 1)}. \tag {F.116}
$$
Thus, as long as  $\eta = \frac{1}{(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)d^B}$  for a sufficiently large universal constant  $B$ , and choosing  $\eta$  so that  $T_{*,\epsilon}$  is an integer multiple of  $\eta$ , we have  $\max_{\chi} \| \tilde{\delta}_{t+1}(\chi) \|_2 \leq \frac{1}{\sqrt{d}}$ , because if  $t \leq \frac{T_{*,\epsilon}}{\eta}$ , then
$$
\max  _ {\chi} \| \tilde {\delta} _ {t + 1} (\chi) \| _ {2} ^ {2} \leq \eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) d ^ {8} \cdot e ^ {C \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) T _ {* , \epsilon}} \tag {F.117}
$$
$$
\leq \eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) d ^ {8} \cdot e ^ {O (\log d)} \tag {F.118}
$$
$$
\leq \eta \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) d ^ {O (1)}. \tag {F.119}
$$
This completes the induction, and therefore for all  $t \leq \frac{T_{*,\epsilon}}{\eta}$ , we have  $\max_{\chi} \| \tilde{\delta}_t(\chi) \|_2 \leq \frac{1}{\sqrt{d}}$ . Additionally, by Lemma E.16, and using the coupling between  $\hat{\rho}_t$  and  $\hat{\rho}_{\eta t}$  which is the joint distribution of  $(\tilde{u}_t(\chi), \hat{u}_{\eta t}(\chi))$ , we have
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (f _ {\tilde {\rho} _ {t}} (x) - f _ {\hat {\rho} _ {\eta t}} (x)) ^ {2} ] \lesssim (\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2} \lesssim \frac {\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}}{d} \lesssim (\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}) \epsilon \tag {F.120}
$$
G Missing Proofs in Section 3
G.1 Proof of Lemma 3.1
In this section, we prove sufficient and necessary conditions for  $\gamma_{2}$  and  $\gamma_{4}$  when the population loss is 0.
Proposition G.1 (Necessary condition). Let  $f_{\rho}(x) = \mathbb{E}_{u \sim \rho}[\sigma(u^{\top}x)]$  be a (not necessarily symmetric nor rotational invariant) two-layer neural network and  $y(x) = h(e_1^{\top}x)$  the target function. Define  $\gamma_k = \frac{\hat{h}_{k,d}}{\hat{\sigma}_{k,d}}, \forall k \geq 0$ . Suppose  $\mathbb{E}_{x \sim \mathbb{S}^{d-1}}(f_{\rho}(x) - y(x))^2 = 0$  and  $\hat{\sigma}_{2,d} \neq 0, \hat{\sigma}_{4,d} \neq 0$ , we have
$$
\gamma_ {2} ^ {2} \leq \gamma_ {4} + O \left(\frac {1}{\sqrt {d}}\right), \tag {G.1}
$$