SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \tilde {B} \right\| _ {F} ^ {2} \lesssim \sum_ {| \lambda_ {i} | \leq \tau} \lambda_ {i} ^ {2} + \frac {1}{d} \left(\sum_ {\lambda_ {i} > \tau} \lambda_ {i}\right) ^ {2} + \left| \sum_ {| \lambda_ {i} | > \tau} \lambda_ {i} u _ {i} ^ {T} A u _ {i} \right| ^ {2} \\ \lesssim \sum_ {| \lambda_ {i} | \leq \tau} \lambda_ {i} ^ {2} + \| A \| _ {o p} ^ {2} \bigg | \sum_ {| \lambda_ {i} | > \tau} \lambda_ {i} \bigg | ^ {2} \\ \lesssim 1 + \tau^ {- 2} \| A \| _ {o p} ^ {2} \\ \lesseqqgtr 1. \\ \end{array}
$$
$$
\begin{array}{l} \left\| \tilde {B} \right\| _ {o p} \leq \tau + \left(\frac {1}{d} + \| A \| _ {o p}\right) \left| \sum_ {| \lambda_ {i} | > \tau} \lambda_ {i} u _ {i} ^ {T} A u _ {i} \right| \\ \lesssim \tau + \| A \| _ {o p} ^ {2} \tau^ {- 1}. \\ \end{array}
$$
Next, since  $\left|\mathbb{E}\big[q(x^T Ax)\big] - \mathbb{E}_z[q(x)]\right| \lesssim L\| A\|_{op}$ , Lemma 26 yields
$$
\begin{array}{l} \left| \mathbb {E} \left[ q \left(x ^ {T} A x\right) \cdot \sum_ {\left| \lambda_ {i} \right| > \tau} \lambda_ {i} \left(u _ {i} u _ {i} ^ {T} - \frac {1}{d} I\right) \right] \right| \leq \sum_ {\left| \lambda_ {i} \right| > \tau} | \lambda_ {i} | \left| \mathbb {E} \left[ q \left(x ^ {T} A x\right) \left(u _ {i} ^ {T} x\right) ^ {2} \right] - \mathbb {E} \left[ q \left(x ^ {T} A x\right) \right] \right| \\ \lesssim \sum_ {| \lambda_ {i} | > \tau} | \lambda_ {i} | \cdot L \| A \| _ {o p} ^ {1 / 3} \log^ {2 / 3} d \\ \lesssim L \tau^ {- 1} \| A \| _ {o p} ^ {1 / 3} \log^ {2 / 3} d. \\ \end{array}
$$
$$
\left| \mathbb {E} \left[ q (x ^ {T} A x) \cdot \frac {1}{\| A \| _ {F} ^ {2}} \sum_ {| \lambda_ {i} | > \tau} \lambda_ {i} u _ {i} ^ {T} A u _ {i} \cdot x ^ {T} A x \right] \right| \lesssim L \left| \sum_ {| \lambda_ {i} | > \tau} \lambda_ {i} u _ {i} ^ {T} A u _ {i} \right| \lesssim L \| A \| _ {o p} \tau^ {- 1}.
$$
$$
\left| \mathbb {E} \big [ q (x ^ {T} A x) x ^ {T} B x \big ] \right| \lesssim L \log d \Big (\| A \| _ {o p} ^ {1 / 2} + \tau + \| A \| _ {o p} ^ {2} \tau^ {- 1} + \tau^ {- 1} \| A \| _ {o p} ^ {1 / 3} + \| A \| _ {o p} \tau^ {- 1} \Big) \lesssim L \| A \| _ {o p} ^ {1 / 6} \log d.
$$
where we set  $\tau = \| A\|_{op}^{1 / 6}$
Finally, we use the fact that  $x^{T}Ax$  is approximately Gaussian to show that  $\mathbb{E}[q(x^{T}Ax)x^{T}Ax] \approx c_{1}$ .
Lemma 28. Let  $x^T Ax$  be a spherical harmonic. Then
$$
\left| \mathbb {E} \left[ q \left(x ^ {T} A x\right) x ^ {T} A x \right] - c _ {1} \right| \lesssim L \| A \| _ {o p} \log d
$$
Proof. Define  $H(z) = q(z)z$ . For truncation radius  $R$ , define  $\overline{H}(z) = H(\max(-R, \min(R, z)))$ . For  $x, y \in [-R, R]$  we can bound
$$
\begin{array}{l} | H (x) - H (y) | = | q (x) x - q (y) y | \\ \leq | x | \| q (x) - q (y) | + | q (y) | \| x - y \| \\ \lesssim R L \| x - y \|. \\ \end{array}
$$
Thus  $\overline{H}$  has Lipschitz constant  $O(RL)$ . Since  $W_{1}(x^{T}Ax,\mathcal{N}(0,1))\lesssim \| A\|_{op}$ , we have
$$
\left| \mathbb {E} \bar {H} \left(x ^ {T} A x\right) - \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \bar {H} (z) \right| \lesssim R L \| A \| _ {o p}.
$$
Furthermore, choosing  $R = C \log d$  for appropriate constant  $C$ , we have that
$$
\begin{array}{l} \left| \mathbb {E} _ {x} [ \bar {H} (x ^ {T} A x) - H (x ^ {T} A x) ] \right| \leq \mathbb {P} (\left| x ^ {T} A x \right| > R) \cdot \mathbb {E} [ H (x ^ {T} A x) ^ {2} ] ^ {1 / 2} \lesssim \frac {L}{d} \\ \left| \mathbb {E} _ {z} [ \overline {{H}} (z) - H (z) ] \right| \leq \mathbb {P} (| z | > R) \cdot \mathbb {E} [ H (z) ^ {2} ] ^ {1 / 2} \lesssim \frac {L}{d}. \\ \end{array}
$$
$$
\left| \mathbb {E} _ {x} \left[ H \left(x ^ {T} A x\right) \right] - \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} [ H (z) ] \right| \lesssim L \| A \| _ {o p} \log d.
$$
Substituting  $H(x^{T}Ax) = q(x^{T}Ax)x^{T}Ax$  and  $\mathbb{E}_{z\sim \mathcal{N}(0,1)}[H(z)] = \mathbb{E}_z[q(z)z] = \mathbb{E}_z[q'(z)] = c_1$  yields the desired bound.
We are now set to prove Lemma 20.
Proof of Lemma 20. Let  $P_{2}f^{*} = x^{T}T_{2}x$ . Then  $\left\| P_2f^* -c_1x^TAx\right\|_{L^2}\lesssim \| T_2 - c_1A\| _F$ .
Write  $T_{2} = \alpha A + A^{\perp}$ , where  $\langle A^{\perp},A\rangle = 0$ . We first have that
$$
\mathbb {E} \left[ f ^ {*} (x) x ^ {T} A ^ {\perp} x \right] = \mathbb {E} \left[ x ^ {T} T _ {2} x \cdot x ^ {T} A ^ {\perp} x \right] = 2 \chi_ {2} \langle T _ {2}, A ^ {\perp} \rangle = 2 \chi_ {2} \| A ^ {\perp} \| _ {F} ^ {2}.
$$
Also, by Lemma 27, we have
$$
\mathbb {E} \left[ f ^ {*} (x) x ^ {T} A ^ {\perp} x \right] = \mathbb {E} \left[ q \left(x ^ {T} A x\right) x ^ {T} A ^ {\perp} x \right] \lesssim \left\| A ^ {\perp} \right\| _ {F} \cdot L \| A \| _ {o p} ^ {1 / 6} \log d.
$$
Therefore  $\left\| A^{\perp}\right\|_{F} \lesssim L\| A\|_{op}^{1/6} \log d$ . Next, see that
$$
\mathbb {E} \left[ f ^ {*} (x) x ^ {T} A x \right] = 2 \alpha \chi_ {2} \| A \| _ {F} ^ {2} = \alpha ,
$$
so by Lemma 28 we have  $|c_{1} - \alpha| \lesssim L \|A\|_{op} \log d$ . Altogether,
$$
\left\| T _ {2} - c _ {1} A \right\| _ {F} \leq | \alpha - c _ {1} | \| A \| _ {F} + \left\| A ^ {\perp} \right\| _ {F} \lesssim L \| A \| _ {o p} ^ {1 / 6} \log d = L \kappa^ {1 / 6} d ^ {- 1 / 1 2} \log d.
$$
E.3 Improved Error Floor for Polynomials
When  $q$  is a polynomial of degree  $p = O(1)$ , we can improve the exponent of  $d$  in the error floor.
Theorem 7. Assume that  $q$  is a degree  $p$  polynomial, where  $p = O(1)$ . Under Assumption 5, Assumption 6, and Assumption 7, with high probability Algorithm 1 satisfies the population loss bound
$$
\mathbb {E} _ {x} \left[ \left(f (x; \hat {\theta}) - f ^ {*} (x)\right) ^ {2} \right] \lesssim \tilde {O} \left(\frac {d ^ {4}}{\min  (n , m _ {1} , m _ {2})} + \frac {1}{\sqrt {n}} + \frac {\kappa^ {2}}{d}\right)
$$
The high level strategy to prove Theorem 7 is similar to that for Theorem 3, as we aim to show  $\mathbb{K}f^{*}$  is approximately proportional to  $x^{T}Ax$ . Rather to passing to universality as in Lemma 20, however, we use an algebraic argument to estimate  $P_{2}f^{*}$ .
The key algebraic lemma is the following:
Lemma 29. Let  $\operatorname{Tr}(A) = 0$ . Then
$$
A ^ {\tilde {\otimes} k} (I ^ {\otimes k - 1}) = \sum_ {s = 1} ^ {k} d _ {k, s} A ^ {s} \cdot A ^ {\tilde {\otimes} k - s} (I ^ {\otimes (k - s)}),
$$
where the constants  $d_{k,s}$  are defined by
$$
d _ {k, s} := 2 ^ {s - 1} \frac {(2 k - 2 s - 1) ! ! (k - 1) !}{(2 k - 1) ! ! (k - s) !}
$$
and we denote  $(-1)!! = 1$ .
Proof. The proof proceeds via a counting argument. We first have that
$$
A ^ {\tilde {\otimes} k} (I ^ {\otimes k - 1}) = \sum_ {(\alpha_ {1}, \ldots , \alpha_ {k - 1}) \in [ d ] ^ {k - 1}} \left(A ^ {\tilde {\otimes} k}\right) _ {\alpha_ {1}, \alpha_ {1}, \ldots , \alpha_ {k - 1}, \alpha_ {k - 1}, i, j}.
$$
Consider any permutation  $\sigma \in S_{2k}$ . We can map this permutation to the graph  $G(\sigma)$  on  $k$  vertices and  $k - 1$  edges as follows: for  $m \in [k - 1]$ , if  $\sigma^{-1}(2m - 1) \in \{2a - 1, 2a\}$  and  $\sigma^{-1}(2m) \in \{2b - 1, 2b\}$ , then we draw an edge  $e(m)$  between  $a$  and  $b$ . In the resulting graph  $G(\sigma)$  each node has degree at most 2, and hence there are either two vertices with degree 1 or one vertex with degree 0. For a vertex  $v$ , let  $e_1(v), e_2(v) \in [k - 1]$  be the two edges  $v$  is incident to if  $v$  has degree 2, and otherwise  $e_1(v)$  be the only edge  $v$  is incident to. For shorthand, let  $(i_1, \ldots, i_{2k}) = (\alpha_1, \alpha_1, \ldots, \alpha_{k - 1}, \alpha_{k - 1}, i, j)$ .
If there are two vertices  $(u_{1}, u_{2})$  with degree 1, we have that
$$
\left(A ^ {\otimes k}\right) _ {i _ {\sigma (1)}, i _ {\sigma (2)}, \dots i _ {\sigma (2 k)}} = A _ {i, \alpha_ {e _ {1} (u _ {1})}} A _ {j, \alpha_ {e _ {2} (u _ {2})}} \prod_ {v \neq u _ {1}, u _ {2}} A _ {\alpha_ {e _ {1} (v)}, \alpha_ {e _ {2} (v)}}
$$
Let  $u_{1}, u_{2}$  be connected to each other via a path of  $s$  total vertices, and let  $\mathcal{P}$  be the ordered set of vertices in this path. Via the matrix multiplication formula, one sees that
$$
\sum_ {(\alpha_ {1}, \ldots , \alpha_ {k - 1}) \in [ d ] ^ {k - 1}} \left(A ^ {\otimes k}\right) _ {i _ {\sigma (1)}, i _ {\sigma (2)}, \dots i _ {\sigma (2 k)}} = (A ^ {s}) _ {i, j} \sum_ {v \not \in \mathcal {P}} A _ {\alpha_ {e _ {1} (v)}, \alpha_ {e _ {2} (v)}},
$$
where the sum is over the  $k - s\alpha$  's that are still remaining in  $\{e_i(v):v\notin \mathcal{P}\}$
Likewise, if there is one vertex  $u_{1}$  with degree 0, we have
$$
\left(A ^ {\otimes k}\right) _ {i _ {\sigma (1)}, i _ {\sigma (2)}, \dots i _ {\sigma (2 k)}} = A _ {i, j} \prod_ {v \neq u _ {1}} A _ {\alpha_ {e _ {1} (v)}, \alpha_ {e _ {2} (v)}}.
$$
and thus, since  $\mathcal{P} = \{u_1\}$
$$
\sum_ {(\alpha_ {1}, \ldots , \alpha_ {k - 1}) \in [ d ] ^ {k - 1}} \left(A ^ {\otimes k}\right) _ {i _ {\sigma (1)}, i _ {\sigma (2)}, \dots i _ {\sigma (2 k)}} = A _ {i, j} \sum_ {(\alpha_ {1}, \ldots , \alpha_ {k - 1})} \prod_ {v \notin \mathcal {P}} A _ {\alpha_ {e _ {1} (v)}, \alpha_ {e _ {2} (v)}}.
$$
Altogether, we have that
$$
A ^ {\tilde {\otimes} k} \left(I ^ {\otimes k - 1}\right) = \frac {1}{(2 k) !} \sum_ {\sigma \in S _ {2 k}} A ^ {s} \sum_ {v \notin \mathcal {P}} A _ {\alpha_ {e _ {1} (v)}, \alpha_ {e _ {2} (v)}}
$$
where  $s, \mathcal{P}$  are defined based on the graph  $\mathcal{G}(\sigma)$ . Consider a graph with fixed path  $\mathcal{P}$ , and let  $S_{\mathcal{P}}$  be the set of permutations which give rise to the path  $\mathcal{P}$ . We have that
$$
A ^ {\otimes k} (I ^ {\otimes k - 1}) = \frac {1}{(2 k) !} \sum_ {\mathcal {P}} A ^ {s} \sum_ {\sigma \in \mathcal {S} _ {\mathcal {P}}} \sum_ {v \notin \mathcal {P}} \prod_ {v \notin \mathcal {P}} A _ {\alpha_ {e _ {1} (v)}, \alpha_ {e _ {2} (v)}}.
$$
There are  $(k - 1)\dots (k - s + 1)$  choices for the  $k$  edges to use in the path, and at each vertex  $v$  there are two choices for which edge should correspond to  $2v$  or  $2v + 1$ . Additionally, there are 2 ways to orient each edge. Furthermore, there are  $\frac{k!}{(k - s)!}$  ways to choose the ordering of the path. Altogether, there are  $2^{2s - 1}\frac{(k - 1)!}{(k - s)!}\frac{k!}{(k - s)!}$  ways to construct a path of length  $s$ . We can thus write
$$
A ^ {\tilde {\otimes} k} (I ^ {\otimes k - 1}) = \frac {1}{(2 k) !} \sum_ {s} A ^ {s} 2 ^ {2 s - 1} \frac {(k - 1) !}{(k - s) !} \frac {k !}{(k - s) !} \sum_ {v \notin \mathcal {P}} A _ {\alpha_ {e _ {1} (v)}, \alpha_ {e _ {2} (v)}},
$$
where this latter sum is over all permutations where the mapping corresponding to vertices not on the path have not been decided, along with the sum over the unused  $\alpha$ 's. Reindexing, this latter sum is (letting  $(i_1, \ldots, i_{2k-2s}) = (\alpha_1, \alpha_1, \ldots, \alpha_{k-s}, \alpha_{k-s}))$
$$
\sum_ {\sigma \in S _ {2 k - 2 s}} \sum_ {(\alpha_ {1}, \dots , \alpha_ {k - s}) \in [ d ] ^ {k - s}} \prod A _ {i _ {\sigma (2 j - 1)}, i _ {\sigma (2 j)}} = (2 k - 2 s)! \tilde {A ^ {\otimes k - s}} (I ^ {\otimes k - s})
$$
Altogether, we obtain
$$
\begin{array}{l} A ^ {\tilde {\otimes} k} \left(I ^ {\otimes k - 1}\right) = \sum_ {s \geq 1} \frac {(2 k - 2 s) !}{(2 k) !} \frac {(k - 1) !}{(k - s) !} \frac {k !}{(k - s) !} 2 ^ {2 s - 1} \cdot A ^ {s} \cdot A ^ {\tilde {\otimes} k - s} \left(I ^ {\otimes k - s}\right) \\ = \sum_ {s \geq 1} \frac {k !}{(2 k) !} \frac {(2 k - 2 s) !}{(k - s) !} \frac {(k - 1) !}{(k - s) !} \cdot A ^ {s} \cdot A ^ {\tilde {\otimes} k - s} \left(I ^ {\otimes k - s}\right) \\ = \sum_ {s \geq 1} \frac {(2 k - 2 s - 1) ! !}{(2 k - 1) ! ! 2 ^ {s}} \frac {(k - 1) !}{(k - s) !} 2 ^ {2 s - 1} \cdot A ^ {s} \cdot A ^ {\tilde {\otimes} k - s} (I ^ {\otimes k - s}) \\ = \sum_ {s \geq 1} d _ {k, s} \cdot A ^ {s} \cdot A ^ {\tilde {\otimes} k - s} (I ^ {\otimes k - s}), \\ \end{array}
$$
Definition 8. Define the operator  $\mathcal{T}:\mathbb{R}^{d\times d}\to \mathbb{R}^{d\times d}$  by  $\mathcal{T}(M) = M - \operatorname {Tr}(M)\cdot \frac{I}{d}$ .
Lemma 30. Let  $P_{2}f^{*}(x) = x^{T}T_{2}x$ . Then  $\left\| T_2 - \mathbb{E}_x[(g^*)'(x^TAx)]\cdot A\right\| _F\lesssim \frac{\kappa}{\sqrt{d}}$
Proof. Throughout, we treat  $p = O(1)$  and thus functions of  $p$  independent of  $d$  as  $O(1)$  quantities. Let  $q$  be of the form  $g^{*}(z) = \sum_{k=0}^{p} \alpha_{k} z^{k}$ . We then have
$$
f ^ {*} (x) = \sum_ {k = 0} ^ {p} \alpha_ {k} A ^ {\tilde {\otimes} k} (x ^ {\otimes 2 k})
$$
Therefore  $P_{2}f^{*}(x) = x^{T}T_{2}x$ , where
$$
T _ {2} := \sum_ {k = 0} ^ {p} \alpha_ {k} (2 k - 1)!! k \frac {\chi_ {k + 1}}{\chi_ {2}} \mathcal {T} \left(A ^ {\tilde {\otimes} k} \left(I ^ {\otimes k - 1}\right)\right).
$$
Applying Lemma 29, one has
$$
T _ {2} = \sum_ {s = 0} ^ {p} \mathcal {T} (A ^ {s}) \cdot \sum_ {k = s} ^ {p} \alpha_ {k} (2 k - 1)!! k \frac {\chi_ {k + 1}}{\chi_ {2}} d _ {k, s} A ^ {\tilde {\otimes} k - s} (I ^ {\otimes (k - s)}).
$$
$$
\beta_ {s} = \sum_ {k = s} ^ {p} \alpha_ {k} (2 k - 1)!! k \frac {\chi_ {k + 1}}{\chi_ {2}} d _ {k, s} A ^ {\otimes k - s} \left(I ^ {\otimes (k - s)}\right)
$$
We first see that
$$
\beta_ {1} = \sum_ {k = 1} ^ {p} (2 k - 3)!! \cdot k \alpha_ {k} \frac {\chi_ {k + 1}}{\chi_ {2}} A ^ {\tilde {\otimes} k - 1} \left(I ^ {\otimes (k - 1)}\right)
$$
$$
\frac {\chi_ {k + 1}}{\chi_ {2}} = \frac {d (d + 2)}{(d + 2 k) (d + 2 k - 2)} \chi_ {k - 1} = \chi_ {k - 1} + O (1 / d).
$$
$$
\begin{array}{l} \beta_ {1} = \sum_ {k = 1} ^ {p} (2 k - 3)!! \cdot k \alpha_ {k} \chi_ {k - 1} A ^ {\tilde {\otimes} k - 1} \left(I ^ {\otimes (k - 1)}\right) + O (1 / d) \cdot \sum_ {k = 1} ^ {p} (2 k - 3)!! \cdot k | \alpha_ {k} | A ^ {\tilde {\otimes} k - 1} \left(I ^ {\otimes (k - 1)}\right) \\ = \sum_ {k = 1} ^ {p} k \alpha_ {k} A ^ {\otimes k - 1} \left(\mathbb {E} \left[ x ^ {\otimes 2 k - 2} \right]\right) + O (1 / d) \\ = \mathbb {E} _ {x} \left[ (g ^ {*}) ^ {\prime} \left(x ^ {T} A x\right) \right] + O (1 / d). \\ \end{array}
$$
$$
\left| A ^ {\tilde {\otimes} k} (I ^ {\otimes k}) \right| \lesssim \mathbb {E} _ {x} \left[ \left(x ^ {T} A x\right) ^ {k} \right] \lesssim \mathbb {E} _ {x} \left[ \left(x ^ {T} A x\right) ^ {2} \right] ^ {k / 2} = O (1)
$$
where  $\operatorname{Tr}(A) = 0$  implies  $\mathbb{E}_x[(x^T Ax)^2] = O(1)$  and we invoke spherical hypercontractivity (Lemma 35). Similarly,  $|\beta_s| = O(1)$ , and thus
$$
\left\| T _ {2} - \mathbb {E} _ {x} \left[ (g ^ {*}) ^ {\prime} (x ^ {T} A x) \right] \cdot A \right\| _ {F} \lesssim \frac {1}{d} + \sum_ {s = 2} ^ {p} \| \mathcal {T} (A ^ {s}) \| _ {F} \lesssim \frac {\kappa}{\sqrt {d}},
$$
where we use the inequality
$$
\left\| \mathcal {T} (X) \right\| _ {F} \leq \left\| X \right\| _ {F} + | \operatorname {T r} (X) | \cdot \frac {1}{\sqrt {d}} \leq 2 \| X \| _ {F},
$$
$$
\left\| A ^ {s} \right\| _ {F} \leq \left\| A ^ {2} \right\| _ {F} \leq \frac {\kappa}{\sqrt {d}}
$$
for  $s \geq 2$ .
Lemma 31. Let  $c_{1} = \mathbb{E}_{x}\big[(g^{*})^{\prime}(x^{T}Ax)\big]$ . Then  $\left\| \mathbb{K}f^{*} - \lambda_{2}^{2}(\sigma)c_{1}x^{T}Ax\right\|_{L^{2}}\lesssim \kappa d^{-5 / 2}$
Additionally, by Lemma 30 we have that
$$
\left\| P _ {2} f ^ {*} - c _ {1} x ^ {T} A x \right\| _ {L ^ {2}} \lesssim \| T _ {2} - c _ {1} \cdot A \| _ {F} \lesssim \frac {\kappa}{\sqrt {d}}.
$$
$$
\left| \left| \mathbb {K} f ^ {*} - \lambda_ {2} ^ {2} (\sigma) c _ {1} x ^ {T} A x \right| \right| _ {L ^ {2}} \lesssim \kappa d ^ {- 5 / 2}.
$$
Corollary 4. Assume  $\kappa = o(\sqrt{d})$ . Then
$$
\left\| x ^ {T} A x - \left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} ^ {- 1} \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} \lesssim \kappa / \sqrt {d}
$$
$$
\begin{array}{l} \left\| x ^ {T} A x - \left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} ^ {- 1} \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} \\ = \left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} ^ {- 1} \left\| x ^ {T} A x \right\| \left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} - \left. \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} \\ \leq \left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} ^ {- 1} \left\| \mathbb {K} f ^ {*} - \lambda_ {2} ^ {2} (\sigma) c _ {1} x ^ {T} A x \right\| _ {L ^ {2}} + \left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} ^ {- 1} \left\| \mathbb {K} f ^ {*} \right\| - \lambda_ {2} ^ {2} (\sigma) | c _ {1} | \\ \lesssim \kappa d ^ {- 5 / 2} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1} \\ \lesssim \kappa / \sqrt {d}. \\ \end{array}
$$
The proof of Theorem 7 follows directly from Corollary 4 in an identical manner to the proof of Theorem 3.
F Preliminaries on Spherical Harmonics
In this section we restrict to  $\mathcal{X}_d = \mathcal{S}^{d - 1}(\sqrt{d})$ , the sphere of radius  $\sqrt{d}$ , and  $\nu$  the uniform distribution on  $\mathcal{X}_d$ .
The moments of  $\nu$  are given by the following [18]:
Lemma 32. Let  $x\sim \nu$  . Then
$$
\mathbb {E} _ {x} \left[ x ^ {\otimes 2 k} \right] = \chi_ {k} \cdot (2 k - 1)!! I ^ {\tilde {\otimes} k}
$$
$$
\chi_ {k} := \prod_ {j = 0} ^ {k - 1} \left(\frac {d}{d + 2 j}\right) = \Theta (1).
$$
For integer  $\ell \geq 0$ , let  $V_{d,\ell}$  be the space of homogeneous harmonic polynomials on  $\mathbb{R}^d$  of degree  $\ell$  restricted to  $\mathcal{X}_d$ . One has that  $V_{d,\ell}$  form an orthogonal decomposition of  $L^2 (\nu)$  [27], i.e
$$
L ^ {2} (\nu) = \bigoplus_ {\ell = 0} ^ {\infty} V _ {d, \ell}
$$
Homogeneous polynomials of degree  $\ell$  can be written as  $T(x^{\otimes \ell})$  for an  $\ell$ -tensor  $T \in (\mathbb{R}^d)^{\otimes \ell}$ . The following lemma characterizes  $V_{d,\ell}$ :
Lemma 33.  $T(x^{\otimes \ell})\in V_{d,\ell}$  if and only if  $T(I) = 0$
Proof. By definition, a degree  $l$  homogeneous polynomial  $p(x) \in V_{d,l}$  if and only if  $\Delta p(x) = 0$  for all  $x \in S^{d-1}$ . Note that  $\nabla^2 p(x) = l(l-1)T(x^{\otimes (l-2)})$  so this is satisfied if and only if
$$
0 = \operatorname {t r} T (x ^ {\otimes (l - 2)}) = T (x ^ {\otimes (l - 2)} \otimes I) = \langle T (I), x ^ {\otimes (l - 2)} \rangle .
$$
As this must hold for all  $x$ , this holds if and only if  $T(I) = 0$ .
From the above characterization, we see that  $\dim(V_{d,k}) = B(d,k)$ , where
$$
B (d, k) = \frac {2 k + d - 2}{k} \binom {k + d - 3} {k - 1} = \frac {(k + d - 3) ! (2 k + d - 2)}{k ! (d - 2) !} = (1 + o _ {d} (1)) \frac {d ^ {k}}{k !}.
$$
Define  $P_{\ell}:L^{2}(\nu)\to L^{2}(\nu)$  to be the orthogonal projection onto  $V_{d,\ell}$ . The action of  $P_0,P_1,P_2$  on a homogeneous polynomial is given by the following lemma:
Lemma 34. Let  $T \in (\mathbb{R}^d)^{\otimes 2k}$  be a symmetric  $2k$  tensor, and let  $p(x) = T(x^{\otimes 2k})$  be a polynomial. Then:
$$
P _ {0} p = \chi_ {k} (2 k - 1)!! T \left(I ^ {\otimes k}\right)
$$
$$
P _ {1} p = 0
$$
$$
P _ {2} p = \left\langle \frac {k (2 k - 1) ! ! \chi_ {k + 1}}{\chi_ {2}} \left(T (I ^ {\otimes k - 1}) - T (I ^ {\otimes k}) \cdot \frac {I}{d}\right), x x ^ {T} \right\rangle
$$
Proof. First, we see
$$
P _ {0} p = \mathbb {E} \left[ T \left(x ^ {\otimes 2 k}\right) \right] = \chi_ {k} (2 k - 1)!! T \left(I ^ {\otimes k}\right).
$$
Next, since  $p$  is even,  $P_{1}p = 0$ . Next, let  $P_{2}p = x^{T}T_{2}x$ . For symmetric  $B$  so that  $\operatorname{Tr}(B) = 0$ , we have that
$$
\mathbb {E} \left[ T (x ^ {\otimes 2 k}) x ^ {T} B x \right] = \mathbb {E} \left[ x ^ {T} T _ {2} x x ^ {T} B x \right].
$$
The LHS is
$$
\begin{array}{l} \mathbb {E} \left[ T \left(x ^ {\otimes 2 k}\right) x ^ {T} B x \right] = (2 k + 1)!! \chi_ {k + 1} (T \tilde {\otimes} B) I ^ {\otimes k + 1} \\ = (2 k + 1)!! \chi_ {k + 1} \frac {2 k}{2 k + 1} \langle T (I ^ {\otimes k - 1}), B \rangle \\ = 2 k \cdot (2 k - 1)! \chi_ {k + 1} \langle T (I ^ {\otimes k - 1}) - T (I ^ {\otimes k}) \cdot \frac {I}{d}, B \rangle , \\ \end{array}
$$
where the last step is true since  $\operatorname{Tr}(B) = 0$ . The RHS is
$$
\begin{array}{l} \mathbb {E} \left[ x ^ {T} T _ {2} x x ^ {T} B x \right] = 3!! \chi_ {2} (T _ {2} \tilde {\otimes} B) (I ^ {\otimes 2}) \\ = 2 \chi_ {2} \langle T _ {2}, B \rangle . \\ \end{array}
$$
Since these two quantities must be equal for all  $B$  with  $\mathrm{Tr}(B) = 0$ , and  $\mathrm{Tr}(T_2) = 0$ , we see that
$$
T _ {2} = \frac {k (2 k - 1) ! ! \chi_ {k + 1}}{\chi_ {2}} \left(T \left(I ^ {\otimes k - 1}\right) - T \left(I ^ {\otimes k}\right) \cdot \frac {I}{d}\right),
$$
Polynomials over the sphere verify hypercontractivity:
Lemma 35 (Spherical hypercontractivity [11, 38]). Let  $f$  be a degree  $p$  polynomial. Then for  $q \geq 2$
$$
\left\| f \right\| _ {L ^ {q} (\nu)} \leq (q - 1) ^ {p / 2} \left\| f \right\| _ {L ^ {2} (\nu)}.
$$
F.1 Gegenbauer Polynomials
For an integer  $d > 1$ , let  $\mu_d$  be the density of  $x \cdot e_1$ , where  $x \sim \mathrm{Unif}(\mathcal{S}^{d-1}(1))$  and  $e_1$  is a fixed unit vector. One can verify that  $\mu_d$  is supported on  $[-1, 1]$  and given by
$$
d \mu_ {d} (x) = \frac {\Gamma (d / 2)}{\sqrt {\pi} \Gamma (\frac {d - 1}{2})} (1 - x ^ {2}) ^ {\frac {d - 3}{2}} d x
$$
where  $\Gamma (n)$  is the Gamma function. For convenience, we let  $Z_{d}\coloneqq \frac{\Gamma(d / 2)}{\sqrt{\pi}\Gamma(\frac{d - 1}{2})} = \frac{1}{\beta(\frac{1}{2},\frac{d - 1}{2})}$  denote the normalizing constant.
The Gegenbauer polynomials  $\left(G_k^{(d)}\right)_{k\in \mathbb{Z}^{\geq 0}}$  are a sequence of orthogonal polynomials with respect to the density  $\mu_d$ , defined as  $G_0^{(d)}(x) = 1, G_1^{(d)}(x) = x$ , and
$$
G _ {k} ^ {(d)} (x) = \frac {d + 2 k - 4}{d + k - 3} x G _ {k - 1} ^ {(d)} (x) - \frac {k - 1}{d + k - 3} G _ {k - 2} ^ {(d)} (x). \tag {24}
$$
By construction,  $G_k^{(d)}$  is a polynomial of degree  $k$ . The  $G_k^{(d)}$  are orthogonal in that
$$
\mathbb {E} _ {x \sim \mu_ {d}} \left[ G _ {k} ^ {(d)} (x) G _ {j} ^ {(d)} (x) \right] = \delta_ {j = k} B (d, k) ^ {- 1}
$$
For a function  $f \in L^{2}(\mu_{d})$ , we can write its Gegenbauer decomposition as
$$
f (x) = \sum_ {k = 0} ^ {\infty} B (d, k) \langle f, G _ {k} ^ {(d)} \rangle_ {L ^ {2} (\mu_ {d})} G _ {k} ^ {(d)} (x),
$$
where convergence is in  $L^2(\mu_d)$ . For an integer  $k$ , we define the operator  $P_k^{(d)}: L^2(\mu_d) \to L^2(\mu_d)$  to be the projection onto the degree  $k$  Gegenbauer polynomial, i.e
$$
P _ {k} ^ {(d)} f = B (d, k) \langle f, G _ {k} ^ {(d)} \rangle_ {L ^ {2} (\mu_ {d})} G _ {k} ^ {(d)}.
$$
We also define the operators  $P_{\leq k}^{(d)} = \sum_{\ell = 0}^{k}P_{\ell}^{(d)}$  and  $P_{\geq k}^{(d)} = \sum_{\ell = k}^{\infty}P_{\ell}^{(d)}$
Recall that  $\nu = \mathrm{Unif}(\mathcal{S}^{d - 1}(\sqrt{d}))$ . Let  $\tilde{\mu}_d$  be the density of  $x\cdot e_1$ , where  $x\sim \nu$ . For a function  $\sigma \in L^2 (\tilde{\mu}_d)$ , we define its Gegenbauer coefficients as
$$
\lambda_ {k} (\sigma) := \mathbb {E} _ {x \sim \nu} [ \sigma (x \cdot e _ {1}) G _ {k} ^ {(d)} (d ^ {- 1 / 2} x \cdot e _ {1}) ] = \langle \sigma (d ^ {1 / 2} e _ {1} \cdot), G _ {k} ^ {(d)} \rangle_ {L ^ {2} (\mu_ {d})}.
$$
By Cauchy, we get that  $|\lambda_k(\sigma)| \leq \| \sigma \|_{L^2 (\tilde{\mu}_d)} B(d,k)^{-1 / 2} = O(\| \sigma \|_{L^2 (\tilde{\mu}_d)} d^{-k / 2})$ .
A key property of the Gegebauer coefficients is that they allow us to express the kernel operator  $\mathbb{K}$  in closed form [27, 39]
Lemma 36. For a function  $g \in L^2(\nu)$ , the operator  $\mathbb{K}$  acts as
$$
\mathbb {K} g = \sum_ {k \geq 0} \lambda_ {k} ^ {2} (\sigma) P _ {k} g,
$$
One key fact about Gegenbauer polynomials is the following derivative formula:
Lemma 37 (Derivative Formula).
$$
\frac {d}{d x} G _ {k} ^ {(d)} = \frac {k (k + d - 2)}{d - 1} G _ {k - 1} ^ {(d + 2)} (x).
$$
Furthermore, the following is a corollary of eq. (24):
Corollary 5.
$$
G _ {2 k} ^ {(d)} (0) = (- 1) ^ {k} \frac {(2 k - 1) ! !}{\prod_ {j = 0} ^ {k - 1} (d + 2 j - 1)}.
$$
G Proofs for Section 5
The proof of Theorem 4 relies on the following lemma, which gives the Gegenbauer decomposition of the ReLU function:
Lemma 38 (ReLU Gegenbauer). Let  $\mathrm{ReLU}(x) = \max(x,0)$ . Then
$$
\begin{array}{l} \mathrm {R e L U} (x) = \frac {1}{\beta (\frac {1}{2} , \frac {d - 1}{2}) (d - 1)} G _ {0} ^ {(d)} (x) + \frac {1}{2 d} G _ {1} ^ {(d)} (x) \\ + \sum_ {k \geq 1} (- 1) ^ {k + 1} \frac {(2 k - 3) ! !}{\beta (\frac {1}{2} , \frac {d - 1}{2}) \prod_ {j = 0} ^ {k} (d + 2 j - 1)} B (d, 2 k) G _ {2 k} ^ {(d)} (x). \\ \end{array}
$$