SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Therefore, it suffices to show that  $\mathbb{E}_{x\sim \mathbb{S}^{d - 1}}[(\tilde{f}_\rho (x) - \tilde{f}_\bar{\rho} (x))^2 ]$  is small with high probability over  $u_{1},\ldots ,u_{m}$ . Observe that for  $|t|\leq B$ , we have  $|P_{2,d}(t)|\leq B^2 +O(1 / d)$ , and  $|P_{4,d}(t)|\leq B^4 + O(B^2 /d) + O(1 / d^2)$ . Thus, if we choose  $B\geq 1 / \sqrt{d}$ , then for  $|t|\leq B$  we have  $|P_{2,d}(t)|\lesssim B^2$  and  $|P_{4,d}(t)|\lesssim B^4$ . In particular, we have
$$
| \sigma (t) | \lesssim | \hat {\sigma} _ {2, d} | \sqrt {N _ {2 , d}} B ^ {2} + | \hat {\sigma} _ {4, d} | \sqrt {N _ {4 , d}} B ^ {4}. \tag {E.197}
$$
Now, letting  $x \in \mathbb{S}^{d - 1}$ , and using Hoeffding's inequality, we find that if  $u_{1},\ldots ,u_{m}$  are i.i.d. samples from  $\rho$ , then
$$
\left| \right. \frac {1}{m} \sum_ {i = 1} ^ {m} \sigma \left( \right.\varphi_ {B} \left(\langle u _ {1}, x \rangle\right) - \underset {u \sim \rho} {\mathbb {E}} \left[ \sigma \left(\varphi_ {B} \left(\langle u, x \rangle\right)\right)\right]\left. \right| \leq t \tag {E.198}
$$
with probability at least  $1 - 2\exp \left(-\frac{Cmt^2}{\hat{\sigma}_{2,d}^2N_{2,d}B^4 + \hat{\sigma}_{4,d}^2N_{4,d}B^8}\right)$  for some universal constant  $C$ . Now, by Corollary 4.2.13 of Vershynin [74], for any  $\gamma >0$ , there exists  $\mathcal{N}\subset \mathbb{S}^{d - 1}$  of size at most  $(\frac{C}{\gamma})^d$  such that for any  $x\in \mathbb{S}^{d - 1}$ , there exists  $y\in \mathbb{S}^{d - 1}$  such that  $\| x - y\| _2\leq \gamma$ . Thus, with probability at least  $1 - 2\Big(\frac{C}{\gamma}\Big)^{d}\exp \left(-\frac{Cmt^{2}}{\hat{\sigma}_{2,d}^{2}N_{2,d}B^{4} + \hat{\sigma}_{4,d}^{2}N_{4,d}B^{8}}\right)$ , for all  $y\in \mathcal{N}$ , we have
$$
\left| \frac {1}{m} \sum_ {i = 1} ^ {m} \sigma \left(\varphi_ {B} \left(\langle u _ {1}, y \rangle\right) - \underset {u \sim \rho} {\mathbb {E}} \left[ \sigma \left(\varphi_ {B} \left(\langle u, y \rangle\right)\right) \right] \right| \leq t. \right. \tag {E.199}
$$
If Eq. (E.199) holds for any  $y \in \mathcal{N}$ , then for any  $x \in \mathbb{S}^{d-1}$ , since  $|\sigma'(t)| \lesssim |\hat{\sigma}_{2,d}| \sqrt{N_{2,d}} + |\hat{\sigma}_{4,d}| \sqrt{N_{4,d}}$  for any  $t \in [-1,1]$  (see Lemma E.12), if we let  $y \in \mathcal{N}$  such that  $\|x - y\|_2 \leq \gamma$ , then we have
$$
\left| \frac {1}{m} \sum_ {i = 1} ^ {m} \sigma \left(\varphi_ {B} \left(\langle u _ {1}, x \rangle\right)\right) - \underset {u \sim \rho} {\mathbb {E}} \left[ \sigma \left(\varphi_ {B} \left(\langle u, x \rangle\right)\right) \right] \right| \leq t + \left(| \hat {\sigma} _ {2, d} | \sqrt {N _ {2 , d}} + | \hat {\sigma} _ {4, d} | \sqrt {N _ {4 , d}}\right) \gamma . \tag {E.200}
$$
In summary, with probability at least  $1 - 2\left(\frac{C}{\gamma}\right)^d\exp \left(-\frac{Cmt^2}{\hat{\sigma}_{2,d}^2N_{2,d}B^4 + \hat{\sigma}_{4,d}^2N_{4,d}B^8}\right)$ , we have
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (\tilde {f} _ {\rho} (x) - \tilde {f} _ {\bar {\rho}} (x)) ^ {2} ] \leq t ^ {2} + (\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}) \gamma^ {2}. \tag {E.201}
$$
Combining Eq. (E.201), Eq. (E.195) and Eq. (E.196), we have that
$$
\begin{array}{l} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (f _ {\rho} (x) - f _ {\bar {\rho}} (x)) ^ {2} ] \lesssim t ^ {2} + (\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}) \gamma^ {2} (E.202) \\ + \exp \left(- \frac {B ^ {2} d}{2}\right) \cdot \left(\hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d}\right) (E.203) \\ \end{array}
$$
with probability at least  $1 - 2\left(\frac{C}{\gamma}\right)^d\exp \left(-\frac{Cmt^2}{\hat{\sigma}_{2,d}^2N_{2,d}B^4 + \hat{\sigma}_{4,d}^2N_{4,d}B^8}\right)$ . Choosing  $B = \frac{C'(\log d)^2}{\sqrt{d}}$  for a sufficiently large universal constant  $C'$ , and using the fact that  $N_{2,d} \lesssim d^2$  and  $N_{4,d} \lesssim d^4$  (by Eq. (2.10) of Atkinson and Han [12]) we have that
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho} (x) - f _ {\bar {\rho}} (x)\right) ^ {2} \right] \lesssim t ^ {2} + \left(\hat {\sigma} _ {2, d} ^ {2} d ^ {2} + \hat {\sigma} _ {4, d} ^ {2} d ^ {4}\right) \gamma^ {2} + \frac {\hat {\sigma} _ {2 , d} ^ {2} d ^ {2} + \hat {\sigma} _ {4 , d} ^ {2} d ^ {4}}{d ^ {\Omega (\log d)}} \tag {E.204}
$$
with probability at least  $1 - 2\left(\frac{C}{\gamma}\right)^d\exp \left(-\Omega \left(\frac{mt^2}{(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)(\log d)^{16}}\right)\right)$ . We can rearrange the failure probability to find that, up to constant factors, it is at most  $\exp \left(-\Omega \left(\frac{mt^2}{(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)(\log d)^{16}}\right) + d\log \frac{C}{\gamma}\right)$ . Finally, choosing  $\gamma = \frac{1}{d^{\Omega(\log d)}}$ , and choosing  $t$  so that the failure probability is at most  $\delta$ , we obtain
$$
\begin{array}{l} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho} (x) - f _ {\bar {\rho}} (x)\right) ^ {2} \right] \lesssim \frac {\left(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}\right) (\log d) ^ {1 6} \left(d \log \frac {C}{\gamma} + \log \frac {1}{\delta}\right)}{m} + \frac {\hat {\sigma} _ {2 , d} ^ {2} d ^ {2} + \hat {\sigma} _ {4 , d} ^ {2} d ^ {4}}{d ^ {\Omega (\log d)}} \tag {E.205} \\ \begin{array}{r l} & {= \frac {(\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}) (\log d) ^ {1 6} (d ^ {2} \log d + \log \frac {1}{\delta})}{m} + \frac {\hat {\sigma} _ {2 , d} ^ {2} + \hat {\sigma} _ {4 , d} ^ {2}}{d ^ {\Omega (\log d)}}} \\ & {\qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \mathrm {(B . c .} \gamma = \frac {1}{d ^ {\Omega (\log d)}}) \end{array} \\ \end{array}
$$
with probability at least  $1 - \delta$ , as desired.
Lemma E.14 (Bound on Gradient). Let  $u \in \mathbb{S}^{d-1}$  and  $\rho$  be a distribution with support on  $\mathbb{S}^{d-1}$ . In addition, let  $\nabla_u L(\rho)$  and  $\mathrm{grad}_u L(\rho)$  be defined as in Eq. (3.4). Then, we have  $\| \nabla_u L(\rho) \|_2 \lesssim (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2) d^4$  and  $\| \mathrm{grad}_u L(\rho) \|_2 \lesssim (\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2) d^4$ .
Similarly, let  $\nabla_u\widehat{L} (\rho)$  and  $\mathrm{grad}_u\widehat{L} (\rho)$  be defined as in Eq. (3.5). Then, we have  $\| \nabla_u\widehat{L} (\rho)\| _2\lesssim$ $(\hat{\sigma}_{2,d}^2 +\hat{\sigma}_{4,d}^2)d^4$  and  $\| \mathrm{grad}_u\widehat{L} (\rho)\| _2\lesssim (\hat{\sigma}_{2,d}^2 +\hat{\sigma}_{4,d}^2)d^4$
Proof of Lemma E.14. First, we will bound  $\nabla_u L(\rho)$  for any  $u \in \mathbb{S}^{d-1}$  and distribution  $\rho$  with support in  $\mathbb{S}^{d-1}$ :
$$
\begin{array}{l} \left\| \nabla_ {u} L (\rho) \right\| _ {2} = \left\| \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (f _ {\rho} (x) - y (x)) \sigma^ {\prime} (\langle u, x \rangle) x ] \right\| _ {2} (E.206) \\ \leq \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} | f _ {\rho} (x) - y (x) | | \sigma^ {\prime} (\langle u, x \rangle) | (E.207) \\ \end{array}
$$
By Lemma E.12, for any  $u,x\in \mathbb{S}^{d - 1}$ , we have  $|\sigma^{\prime}(\langle u,x\rangle)|\lesssim |\hat{\sigma}_{2,d}|\sqrt{N_{2,d}} +|\hat{\sigma}_{4,d}|\sqrt{N_{4,d}}$ . Additionally, by Eq. (3.1) and by the definition of  $y(x)$ , we have  $|f_{\rho}(x) - y(x)|\leq |\hat{\sigma}_{2,d}|\sqrt{N_{2,d}} +|\hat{\sigma}_{4,d}|\sqrt{N_{4,d}}$  by Lemma E.12. Thus,
$$
\left\| \nabla_ {u} L (\rho) \right\| _ {2} \lesssim \hat {\sigma} _ {2, d} ^ {2} N _ {2, d} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d} \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) d ^ {4} \tag {E.208}
$$
because  $N_{2,d} \lesssim d^2$  and  $N_{4,d} \lesssim d^4$  by Eq. (2.10) of Atkinson and Han [12]. By the same argument, for any  $u \in \mathbb{S}^{d-1}$ ,
$$
\left\| \left(I - u u ^ {\top}\right) \nabla_ {u} L (\rho) \right\| _ {2} \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) d ^ {4} \tag {E.209}
$$
because  $I - uu^{\top}$  is an orthogonal projection matrix. This completes the proof — the proof of the analogous statements for  $\nabla_u\widehat{L} (\rho)$  and  $\mathrm{grad}_u\widehat{L} (\rho)$  follow from the same arguments.
Lemma E.15. Let  $u, u' \in \mathbb{S}^{d-1}$ . Then, we have
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(\sigma^ {\prime} \left(u ^ {\top} x\right) - \sigma^ {\prime} \left(u ^ {\prime \top} x\right)\right) ^ {2} \left(u ^ {\top} x - u ^ {\prime \top} x\right) ^ {2} \right] \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \| u - u ^ {\prime} \| _ {2} ^ {4} \tag {E.210}
$$
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \sigma^ {\prime} \left(u ^ {\top} x\right) ^ {2} \left(u ^ {\prime \top} x\right) ^ {2} \right] \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right). \tag {E.211}
$$
Proof of Lemma E.15. For convenience, let  $\delta \coloneqq u' - u$ . Then, we have
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(\sigma^ {\prime} \left(u ^ {\top} x\right) - \sigma^ {\prime} \left(u ^ {\prime \top} x\right)\right) ^ {2} \left(u ^ {\top} x - u ^ {\prime \top} x\right) ^ {2} \right] \tag {E.212}
$$
$$
\begin{array}{l} \lesssim \hat {\sigma} _ {2, d} ^ {2} N _ {2, d} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(P _ {2, d} ^ {\prime} \left(u ^ {\top} x\right) - P _ {2, d} ^ {\prime} \left(u ^ {\prime \top} x\right)\right) ^ {2} \left(u ^ {\top} x - u ^ {\prime \top} x\right) ^ {2} \right] \tag {E.213} \\ + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (P _ {4, d} ^ {\prime} (u ^ {\top} x) - P _ {4, d} ^ {\prime} (u ^ {\top} x)) ^ {2} (u ^ {\top} x - u ^ {\top} x) ^ {2} ] \\ \end{array}
$$
(By definition of  $\sigma$  and  $\overline{P}_{k,d}$  and b.c.  $(a + b)^2\lesssim a^2 +b^2)$
$$
\begin{array}{l} \lesssim \hat {\sigma} _ {2, d} ^ {2} N _ {2, d} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (\delta^ {\top} x) ^ {4} ] + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ ((u ^ {\top} x) ^ {3} - (u ^ {\prime \top} x) ^ {3}) ^ {2} (u ^ {\top} x - u ^ {\prime \top} x) ^ {2} ] \tag {E.214} \\ + \frac {\hat {\sigma} _ {4 , d} ^ {2} N _ {4 , d}}{d ^ {2}} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (u ^ {\top} x - u ^ {\prime \top} x) ^ {4} ] \\ \end{array}
$$
(By Eq. (C.4) and expanding the term with  $P_{4,d}^{\prime}$ )
$$
\begin{array}{l} \lesssim \hat {\sigma} _ {2, d} ^ {2} N _ {2, d} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (\delta^ {\top} x) ^ {4} ] + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d} \sum_ {p + q = 2} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (u ^ {\top} x) ^ {2 p} (u ^ {\prime \top} x) ^ {2 q} (\delta^ {\top} x) ^ {4} ] (E.215) \\ + \frac {\hat {\sigma} _ {4 , d} ^ {2} N _ {4 , d}}{d ^ {2}} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (\delta^ {\top} x) ^ {4} ] (E.216) \\ \end{array}
$$
Here the last inequality is due to the identity  $a^3 - b^3 = (a - b)(a^2 + ab + b^2)$ , which gives  $((u'^{\top}x)^3 - (u^\top x)^3)^2 = ((u'^{\top}x)^2 + (u'^{\top}x)(u^\top x) + (u^\top x)^2)^2(u'^{\top}x - u^\top x)^2$  — expanding the first factor fully gives terms of the form  $(u'^{\top}x)^p (u^\top x)^q$  where  $p$  and  $q$  are both even and  $p + q = 4$ . Thus, applying Lemma I.17 to the final expression above gives
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(\sigma^ {\prime} \left(u ^ {\top} x\right) - \sigma^ {\prime} \left(u ^ {\prime \top} x\right)\right) ^ {2} \left(u ^ {\top} x - u ^ {\prime \top} x\right) ^ {2} \right] \tag {E.217}
$$
$$
\lesssim \hat {\sigma} _ {2, d} ^ {2} N _ {2, d} \cdot \frac {1}{d ^ {2}} \| \delta \| _ {2} ^ {4} + \hat {\sigma} _ {4, d} ^ {2} N _ {4, d} \sum_ {p + q = 2} \frac {1}{d ^ {p + q + 2}} \| \delta \| _ {2} ^ {4} + \frac {\hat {\sigma} _ {4 , d} ^ {2} N _ {4 , d}}{d ^ {2}} \cdot \frac {1}{d ^ {2}} \| \delta \| _ {2} ^ {4} \tag {By Lemma I.17}
$$
$$
\lesssim \hat {\sigma} _ {2, d} ^ {2} \| \delta \| _ {2} ^ {2} + \hat {\sigma} _ {4, d} ^ {2} \| \delta \| _ {2} ^ {2}
$$
(B.c.  $N_{2,d} \lesssim d^2$  and  $N_{4,d} \lesssim d^4$  (see Equation 2.10 of Atkinson and Han [12]))
which completes the proof of the first statement. The proof of the second statement is similar — we expand using the definition of  $\sigma$  and then apply Lemma I.17 to each of the terms. We omit the details of the calculation.
Lemma E.16. Let  $\rho$  and  $\rho'$  be two distributions on  $\mathbb{S}^{d-1}$  which both have support size  $m < \infty$ , and consider a coupling  $(u, u') \sim (\rho, \rho')$  of the two distributions. Additionally, let  $f_{\rho}$  and  $f_{\rho'}$  be defined as in Eq. (3.1). Then,
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho} (x) - f _ {\rho^ {\prime}} (x)\right) ^ {2} \right] \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \underset {(u, u ^ {\prime}) \sim (\rho , \rho^ {\prime})} {\mathbb {E}} \| u - u ^ {\prime} \| _ {2} ^ {2} \tag {E.218}
$$
Proof of Lemma E.16. We can write the left-hand side as
$$
\begin{array}{l} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (f _ {\rho} (x) - f _ {\rho^ {\prime}} (x)) ^ {2} ] = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(\underset {(u, u ^ {\prime}) \sim (\rho , \rho^ {\prime})} {\mathbb {E}} [ \sigma (\langle u, x \rangle) - \sigma (\langle u ^ {\prime}, x \rangle) ]\right) ^ {2} \right] (E.219) \\ \leq \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \underset {(u, u ^ {\prime}) \sim (\rho , \rho^ {\prime})} {\mathbb {E}} \left(\sigma \left(\langle u, x \rangle\right) - \sigma \left(\langle u ^ {\prime}, x \rangle\right)\right) ^ {2} (E.220) \\ = \underset {(u, u ^ {\prime}) \sim (\rho , \rho^ {\prime})} {\mathbb {E}} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(\sigma \left(\langle u, x \rangle\right) - \sigma \left(\langle u ^ {\prime}, x \rangle\right)\right) ^ {2} (E.221) \\ \end{array}
$$
For any  $u, u' \in \mathbb{S}^{d-1}$  and  $x \in \mathbb{S}^{d-1}$ , by the mean-value theorem, there exists  $\lambda$  which is a convex combination of  $\langle u, x \rangle$  and  $\langle u', x \rangle$  such that
$$
\sigma \left(\langle u, x \rangle\right) - \sigma \left(\langle u ^ {\prime}, x \rangle\right) = \sigma^ {\prime} (\lambda) \left(\left\langle u, x \right\rangle - \left\langle u ^ {\prime}, x \right\rangle\right). \tag {E.222}
$$
Using a similar argument as in the proof of Lemma E.15, we can show that
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \sigma^ {\prime} (\lambda) ^ {2} \left(\langle u, x \rangle - \langle u ^ {\prime}, x \rangle\right) ^ {2} \right] \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \| u - u ^ {\prime} \| _ {2} ^ {2}. \tag {E.223}
$$
The proof proceeds by expanding  $\sigma'(\lambda)$  into each of its terms and using the inequality  $|a + b|^3 \lesssim |a|^3 + |b|^3$  (which holds for any two real numbers  $a, b$ ) in order to consider only terms which involve one of  $u$  or  $u'$  — the resulting terms are of the same form as those which appear in Eq. (E.216). We omit the details of the calculation. Thus,
$$
\begin{array}{l} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (f _ {\rho} (x) - f _ {\rho^ {\prime}} (x)) ^ {2} ] \leq \underset {(u, u ^ {\prime}) \sim (\rho , \rho^ {\prime})} {\mathbb {E}} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(\sigma (\langle u, x \rangle) - \sigma (\langle u ^ {\prime}, x \rangle)\right) ^ {2} (E.224) \\ \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \underset {(u, u ^ {\prime}) \sim (\rho , \rho^ {\prime})} {\mathbb {E}} \| u - u ^ {\prime} \| _ {2} ^ {2} (E.225) \\ \end{array}
$$
F Proof of Theorem 3.5
In this section, we build on our analysis from Appendix E to show that projected gradient descent with  $\frac{1}{\mathrm{poly}(d)}$  learning rate can achieve low population loss. In the rest of the section, for convenience, we define  $\tilde{\delta}_t = \tilde{u}_t - \hat{u}_{\eta t}$ , and omit  $\chi$  when it is clear from context.
Lemma F.1. Let  $\rho, \rho'$  be two distributions on  $\mathbb{S}^{d-1}$ , and let  $\Gamma$  be a coupling between  $\rho$  and  $\rho'$ . Additionally, let  $u_1, u_2, v \in \mathbb{S}^{d-1}$ . Then, we have
$$
\left| \nabla_ {u _ {1}} L (\rho) \cdot v - \nabla_ {u _ {2}} L \left(\rho^ {\prime}\right) \cdot v \right| \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \left(\underset {u, u ^ {\prime} \sim \Gamma} {\mathbb {E}} \left[ \| u - u ^ {\prime} \| _ {2} ^ {2} \right] ^ {1 / 2} + \| u _ {1} - u _ {2} \| _ {2}\right). \tag {F.1}
$$
Proof. We can expand the left-hand side as
$$
\left| \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(f _ {\rho} (x) - y (x)\right) \sigma^ {\prime} \left(u _ {1} \cdot x\right) (v \cdot x) - \left(f _ {\rho^ {\prime}} (x) - y (x)\right) \sigma^ {\prime} \left(u _ {2} \cdot x\right) (v \cdot x) \right] \right|. \tag {F.2}
$$
We first obtain a bound on
$$
\left| \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ f _ {\rho} (x) \sigma^ {\prime} \left(u _ {1} \cdot x\right) (v \cdot x) - f _ {\rho^ {\prime}} (x) \sigma^ {\prime} \left(u _ {2} \cdot x\right) (v \cdot x) \right] \right|. \tag {F.3}
$$
By the definition of  $f_{\rho}$ , we can rewrite Eq. (F.3) as
$$
\left| \underset {u _ {1} ^ {\prime}, u _ {2} ^ {\prime} \sim \Gamma} {\mathbb {E}} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \sigma \left(u _ {1} ^ {\prime} \cdot x\right) \sigma^ {\prime} \left(u _ {1} \cdot x\right) (v \cdot x) - \sigma \left(u _ {2} ^ {\prime} \cdot x\right) \sigma^ {\prime} \left(u _ {2} \cdot x\right) (v \cdot x) \right] \right| \tag {F.4}
$$
By Eq. (C.4),  $\sigma$  is an even fourth-degree polynomial and  $\sigma'$  is an odd third-degree polynomial. Thus, the inner expectation can be written as a linear combination of a constant number of terms of the form
$$
\left| \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(u _ {1} ^ {\prime} \cdot x\right) ^ {i} \left(u _ {1} \cdot x\right) ^ {j} (v \cdot x) - \left(u _ {2} ^ {\prime} \cdot x\right) ^ {i} \left(u _ {2} \cdot x\right) ^ {j} (v \cdot x) \right] \right| \tag {F.5}
$$
where  $i = 0,2,4$  and  $j = 1,3$ . We can also estimate the coefficient which accompanies each individual term Eq. (F.5). We first recall the facts that  $\sigma(t) = \hat{\sigma}_{2,d}\bar{P}_{2,d}(t) + \hat{\sigma}_{4,d}\bar{P}_{4,d}(t)$ , that  $N_{2,d} = O(d^2)$  and  $N_{4,d} = O(d^4)$  by Equation (2.10) of Atkinson and Han [12], and that the  $i^{\text{th}}$  degree term of  $P_{2,d}$  has a coefficient which is  $O(d^{i/2-1})$  and the  $i^{\text{th}}$  degree term of  $P_{4,d}$  has a coefficient which is  $O(d^{i/2-2})$ . From these facts it follows that the  $i^{\text{th}}$  degree term of  $\bar{P}_{2,d} = \sqrt{N_{2,d}} P_{2,d}$  has a coefficient which is  $O(d^{i/2})$  and the  $i^{\text{th}}$  degree term of  $\bar{P}_{4,d} = \sqrt{N_{4,d}} P_{4,d}$  has a coefficient which is  $O(d^{i/2})$ . Thus, each term of the form given in Eq. (F.5) has a coefficient which is at most  $(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)d^{(i+j+1)/2}$  in absolute value. We can now upper bound each of the terms of the form given in Eq. (F.5) as follows:
$$
\left| \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(u _ {1} ^ {\prime} \cdot x\right) ^ {i} \left(u _ {1} \cdot x\right) ^ {j} (v \cdot x) - \left(u _ {2} ^ {\prime} \cdot x\right) ^ {i} \left(u _ {2} \cdot x\right) ^ {j} (v \cdot x) \right] \right| \tag {F.6}
$$
$$
\begin{array}{l} \leq \left| \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(\left(u _ {1} ^ {\prime} \cdot x\right) ^ {i} - \left(u _ {2} ^ {\prime} \cdot x\right) ^ {i}\right) \left(u _ {1} \cdot x\right) ^ {j} (v \cdot x) \right] \right| (F.7) \\ + \left| \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left(u _ {2} ^ {\prime} \cdot x\right) ^ {i} \left(\left(u _ {1} \cdot x\right) ^ {j} - \left(u _ {2} \cdot x\right) ^ {j}\right) (v \cdot x) \right] \right| (F.8) \\ \end{array}
$$
$$
\begin{array}{l} \leq \sum_ {k = 0} ^ {i - 1} \left| \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \left((u _ {1} ^ {\prime} - u _ {2} ^ {\prime}) \cdot x\right) (u _ {1} ^ {\prime} \cdot x) ^ {k} (u _ {2} ^ {\prime} \cdot x) ^ {i - 1 - k} (u _ {1} \cdot x) ^ {j} (v \cdot x) \right] \right| (F.9) \\ + \sum_ {k = 0} ^ {j - 1} \left| _ {x \sim \mathbb {S} ^ {d - 1}} \mathbb {E} \right. \left[ \left(u _ {2} ^ {\prime} \cdot x\right) ^ {i} \left(\left(u _ {1} - u _ {2}\right) \cdot x\right) \left(u _ {1} \cdot x\right) ^ {k} \left(u _ {2} \cdot x\right) ^ {j - 1 - k} (v \cdot x) \right] | (F.10) \\ \end{array}
$$
$$
\lesssim \sum_ {k = 0} ^ {i - 1} \| u _ {1} ^ {\prime} - u _ {2} ^ {\prime} \| _ {2} \cdot \frac {1}{d ^ {(1 + k + i - 1 - k + j + 1) / 2}} + \sum_ {k = 0} ^ {j - 1} \| u _ {1} - u _ {2} \| _ {2} \cdot \frac {1}{d ^ {(i + 1 + k + j - 1 - k + 1) / 2}} \tag {By Lemma I.17}
$$
$$
\lesssim \| u _ {1} ^ {\prime} - u _ {2} ^ {\prime} \| _ {2} \frac {1}{d ^ {(i + j + 1) / 2}} + \| u _ {1} - u _ {2} \| _ {2} \frac {1}{d ^ {(i + j + 1) / 2}}. \tag {F.11}
$$
Since the coefficient of this term is at most  $(\hat{\sigma}_{2,d}^2 + \hat{\sigma}_{4,d}^2)d^{(i+j+1)/2}$  in absolute value, this term contributes at most
$$
\left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \left(\| u _ {1} ^ {\prime} - u _ {2} ^ {\prime} \| _ {2} + \| u _ {1} - u _ {2} \| _ {2}\right). \tag {F.12}
$$
Thus, taking the expectation over  $u_1', u_2' \sim \Gamma$ , we have
$$
\begin{array}{l} \left| \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ f _ {\rho} (x) \sigma^ {\prime} \left(u _ {1} \cdot x\right) (v \cdot x) - f _ {\rho^ {\prime}} (x) \sigma^ {\prime} \left(u _ {2} \cdot x\right) (v \cdot x) \right] \right| (F.13) \\ \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \left(\underset {u, u ^ {\prime} \sim \Gamma} {\mathbb {E}} \left[ \| u - u ^ {\prime} \| _ {2} \right] + \| u _ {1} - u _ {2} \| _ {2}\right) (F.14) \\ \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \left(\underset {u, u ^ {\prime} \sim \Gamma} {\mathbb {E}} \left[ \| u - u ^ {\prime} \| _ {2} ^ {2} \right] ^ {1 / 2} + \| u _ {1} - u _ {2} \| _ {2}\right). (F.15) \\ \end{array}
$$
A similar but simpler argument can be used to bound
$$
\left. \left| _ {x \sim \mathbb {S} ^ {d - 1}} \mathbb {E} \left[ y (x) \sigma^ {\prime} \left(u _ {1} \cdot x\right) (v \cdot x) - y (x) \sigma^ {\prime} \left(u _ {2} \cdot x\right) (v \cdot x) \right] \right| \right. \tag {F.16}
$$
by expanding  $y(x)$  and  $\sigma^{\prime}(x)$ . Thus, we obtain
$$
\left| \nabla_ {u _ {1}} L (\rho) \cdot v - \nabla_ {u _ {2}} L (\rho) \cdot v \right| \leq \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \left(\underset {u, u ^ {\prime} \sim \Gamma} {\mathbb {E}} \left[ \| u - u ^ {\prime} \| _ {2} ^ {2} \right] \right. ^ {1 / 2} + \| u _ {1} - u _ {2} \| _ {2}) \tag {F.17}
$$
Lemma F.2. Suppose we are in the setting of Theorem 3.5. Assume that  $n \gtrsim d^3 (\log d)^C$  for a sufficiently large universal constant  $C > 0$ . Additionally, assume that  $\max_{\chi} \| \tilde{\delta}_t(\chi) \|_2 \leq \frac{1}{\sqrt{d}}$ . Then, we have
$$
\left| \left(\operatorname {g r a d} _ {\tilde {u} _ {t}} \widehat {L} (\tilde {\rho} _ {t}) - \operatorname {g r a d} _ {\hat {u} _ {\eta t}} \widehat {L} (\hat {\rho} _ {\eta t})\right) \cdot \tilde {\delta} _ {t} \right| \lesssim \left(\hat {\sigma} _ {2, d} ^ {2} + \hat {\sigma} _ {4, d} ^ {2}\right) \max  _ {\chi} \| \tilde {\delta} _ {t} (\chi) \| _ {2} ^ {2} \tag {F.18}
$$
Proof. We can write the left-hand side of Eq. (F.18) as