SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
(\mathbb {K} f) (x) = \mathbb {E} _ {x ^ {\prime}} \left[ f ^ {*} \left(x ^ {\prime}\right) x ^ {\prime} \cdot x / d \right] = \frac {1}{d} \mathbb {E} _ {x ^ {\prime}} \left[ \nabla f ^ {*} \left(x ^ {\prime}\right) \right] \cdot x.
$$
$$
\mathbb {E} _ {x ^ {\prime}} [ \nabla f ^ {*} (x ^ {\prime}) ] = \mathbb {E} _ {x ^ {\prime}} \left[ w ^ {*} g ^ {\prime} \left(w ^ {*} \cdot x ^ {\prime}\right) \right] = w ^ {*} \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \left[ g ^ {\prime} (z) \right].
$$
Altogether, letting  $c_{1} = \mathbb{E}_{z\sim \mathcal{N}(0,1)}[g^{\prime}(z)] = \Omega (1)$ , we have  $(\mathbb{K}f^{*})(x) = \frac{1}{d} c_{1}x^{T}w^{*}$ . Assumption 4 is thus satisfied with  $\chi = 1 / 2$ .
Next, see that  $\| \mathbb{K}f^{*}\|_{L^2} = c_1 / d$ . We select the test function  $q$  to be  $q(z) = g(\overline{\eta}^{-1}d / c_1\cdot z)$ , so that
$$
q (\bar {\eta} (\mathbb {K} f ^ {*}) (x)) = g (x ^ {*} \cdot x) = f ^ {*} (x).
$$
Since  $\overline{\eta} = \Theta (d\iota^{-\chi})$  , we see that
$$
\begin{array}{l} \sup  _ {z \in [ - 1, 1 ]} | q (z) | = \sup  _ {z \in [ - \Theta (\iota^ {x}), \Theta (\iota^ {x}) ]} | g (z) | = \operatorname {p o l y} (\iota) \\ \sup_{z\in [-1,1]}|q^{\prime}(z)| = \overline{\eta}^{-1}d / c_{1}\sup_{z\in [-\Theta (\iota^{x}),\Theta (\iota^{x})]}|g^{\prime}(z)| = \operatorname {poly}(\iota) \\ \sup  _ {z \in [ - 1, 1 ]} | q ^ {\prime \prime} (z) | = \overline {{\eta}} ^ {- 2} d ^ {2} / c _ {1} ^ {2} \sup  _ {z \in [ - \Theta (\iota x), \Theta (\iota x) ]} | g ^ {\prime \prime} (z) | = \operatorname {p o l y} (\iota) \\ \end{array}
$$
Therefore we can bound the population loss as
$$
\mathbb {E} _ {x} \left[ \left(f (x; \hat {\theta}) - f ^ {*} (x)\right) ^ {2} \right] \lesssim \frac {d ^ {2} \mathrm {p o l y} (\iota)}{\min (n , m _ {1} , m _ {2})} + \frac {1}{\sqrt {n}}.
$$
E.2 Quadratic Feature
Throughout this section, we call  $x^T A x$  a degree 2 spherical harmonic if  $A$  is symmetric,  $\mathbb{E}\left[x^T A x\right] = 0$ , and  $\mathbb{E}\left[(x^T A x)^2\right] = 1$ . Then, we have that  $\operatorname{Tr}(A) = 0$ , and also
$$
1 = \mathbb {E} [ x ^ {\otimes 4} ] (A ^ {\otimes 2}) = 3 \chi_ {2} I ^ {\tilde {\otimes} 2} (A ^ {\otimes 2}) = 2 \chi_ {2} \| A \| _ {F} ^ {2} \Longrightarrow \| A \| _ {F} = \sqrt {\frac {1}{2 \chi_ {2}}} = \sqrt {\frac {d + 2}{2 d}} = \Theta (1).
$$
See Appendix F for technical background on spherical harmonics.
Our goal is to prove the following key lemma, which states that the projection of  $f^*$  onto degree 2 spherical harmonics is approximately  $x^T A x$ .
Lemma 20. Let  $q$  be a  $L$ -Lipschitz function with  $|q(0)| \leq L$ , and let the target  $f^{*}$  be of the form  $f^{*}(x) = q(x^{T}Ax)$ , where  $x^{T}Ax$  is a spherical harmonic. Let  $c_{1} = \mathbb{E}_{z \sim \mathcal{N}(0,1)}[q'(z)]$ . Then
$$
\left\| P _ {2} f ^ {*} - c _ {1} x ^ {T} A x \right\| _ {L ^ {2}} \lesssim L \kappa^ {1 / 6} d ^ {- 1 / 1 2} \log d.
$$
We defer the proof of this Lemma to Appendix E.2.1.
As a consequence, the learned feature  $\mathbb{K}f^{*}$  is approximately proportional to  $x^T Ax$ .
Lemma 21. Recall  $c_{1} = \mathbb{E}_{z\sim \mathcal{N}(0,1)}[q'(z)]$ . Then
$$
\left\| \mathbb {K} f ^ {*} - \lambda_ {2} ^ {2} (\sigma) c _ {1} x ^ {T} A x \right\| _ {L ^ {2}} \lesssim L \kappa^ {1 / 6} d ^ {- 2 - 1 / 1 2} \log d
$$
Proof. Since  $\mathbb{E}[f^{*}(x)] = 0$ ,  $P_{0}f^{*} = 0$ . Next, since  $f^{*}$  is an even function,  $P_{k}f^{*} = 0$  for  $k$  odd. Thus
$$
\left| \left| \mathbb {K} f ^ {*} - \lambda_ {2} ^ {2} (\sigma) P _ {2} f ^ {*} \right| \right| _ {L ^ {2}} \lesssim d ^ {- 4}.
$$
Additionally, by Lemma 20 we have that
$$
\left\| P _ {2} f ^ {*} - c _ {1} x ^ {T} A x \right\| _ {L ^ {2}} \lesssim \| T _ {2} - c _ {1} \cdot A \| _ {F} \lesssim L \kappa^ {1 / 6} d ^ {- 1 / 1 2} \log d.
$$
Since  $\lambda_2^2 (\sigma) = \Theta (d^{-2})$ , we have
$$
\left\| \mathbb {K} f ^ {*} - \lambda_ {2} ^ {2} (\sigma) c _ {1} x ^ {T} A x \right\| _ {L ^ {2}} \lesssim L \kappa^ {1 / 6} d ^ {- 2 - 1 / 1 2} \log d.
$$
Corollary 3. Assume  $\kappa = o(\sqrt{d})$ . Then
$$
\left\| x ^ {T} A x - \left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} ^ {- 1} \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} \lesssim L \kappa^ {1 / 6} d ^ {- 1 / 1 2} \log d
$$
$$
\begin{array}{l} \left\| x ^ {T} A x - \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1} \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} \\ = \left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} ^ {- 1} \left\| x ^ {T} A x \left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} - \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} \\ \leq \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1} \left| \left| \mathbb {K} f ^ {*} - \lambda_ {2} ^ {2} (\sigma) c _ {1} x ^ {T} A x \right| \right| _ {L ^ {2}} + \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1} \left| \left| \mathbb {K} f ^ {*} \right| - \lambda_ {2} ^ {2} (\sigma) | c _ {1} | \right| \\ \lesssim L \kappa^ {1 / 6} d ^ {- 2 - 1 / 1 2} \log d \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1} \\ \lesssim L \kappa^ {1 / 6} d ^ {- 1 / 1 2} \log d. \\ \end{array}
$$
Proof of Theorem 3. By our choice of  $\nu$ , we see that Assumption 1 is satisfied. We next verify Assumption 2. Since  $f^{*}$  is 1-Lipschitz, we can bound  $|g^{*}(z)| \leq |g^{*}(0)| + |z|$ , and thus
$$
\begin{array}{l} \mathbb {E} _ {x} \left[ g ^ {*} \left(x ^ {T} A x\right) ^ {q} \right] ^ {1 / q} \leq \left| g ^ {*} (0) \right| + \mathbb {E} _ {x} \left[ \left| x ^ {T} A x \right| ^ {q} \right] ^ {1 / q} \\ \leq \left| g ^ {*} (0) \right| + q \\ \leq (1 + g ^ {*} (0)) q, \\ \end{array}
$$
where we used Lemma 35. Thus Assumption 2 holds with  $\ell = 1$ .
$$
\mathbb {K} f ^ {*} = \sum_ {k \geq 2} \lambda_ {k} ^ {2} (\sigma_ {2}) P _ {k} f ^ {*}.
$$
By Lemma 21 we have  $\| \mathbb{K}f^{*}\|_{L^{2}}\geq \frac{1}{2}\lambda_{2}^{2}(\sigma)c_{1}$  for  $d$  larger than some absolute constant. Next, by Lemma 35 we have for any  $q\leq \frac{1}{4} d^2$
$$
\begin{array}{l} \| \mathbb {K} f ^ {*} \| _ {q} \leq \sum_ {k \geq 2} \lambda_ {k} ^ {2} (\sigma_ {2}) \| P _ {k} f ^ {*} \| _ {q} \\ \lesssim \sum_ {k \geq 2} d ^ {- k} q ^ {k / 2} \| P _ {k} f ^ {*} \| _ {L ^ {2}} \\ \lesssim \sum_ {k > 2} (\sqrt {q} / d) ^ {k} \\ = \frac {q}{d ^ {2}} \cdot \frac {1}{1 - \sqrt {q} d} \\ \leq 2 q d ^ {- 2} \\ \end{array}
$$
$$
\| \mathbb {K} f ^ {*} \| _ {q} \lesssim \frac {4}{d ^ {2} \lambda_ {2} ^ {2} (\sigma_ {2}) c _ {1}} q \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} \lesssim q \| \mathbb {K} f ^ {*} \| _ {L ^ {2}},
$$
since  $\lambda_2^2 (\sigma_2) = \Omega (d^{-2})$ . Thus Assumption 4 holds with  $\ell = 1$ .
Next, observe that  $\| \mathbb{K}f^{*}\|_{L^2}\lesssim d^{-2}$ . We select the test function  $q$  to be  $q(z) = g^{*}(\overline{\eta}^{-1}\| \mathbb{K}f^{*}\|_{L^2}^{-1}\cdot z)$ . We see that
$$
q (\bar {\eta} (\mathbb {K} f ^ {*}) (x)) = g ^ {*} \left(\| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1} (\mathbb {K} f ^ {*}) (x)\right),
$$
$$
\begin{array}{l} \left\| f ^ {*} - q \left(\bar {\eta} \left(\mathbb {K} f ^ {*}\right) (x)\right) \right\| _ {L ^ {2}} = \left\| g ^ {*} \left(x ^ {T} A x\right) - g ^ {*} \left(\left\| \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} ^ {- 1} \left(\mathbb {K} f ^ {*}\right) (x)\right) \right\| _ {L ^ {2}} \\ \lesssim \left\| x ^ {T} A x - \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1} \mathbb {K} f ^ {*} \right\| _ {L ^ {2}} \\ \lesssim \kappa^ {1 / 6} d ^ {- 1 / 1 2} \log d, \\ \end{array}
$$
where the first inequality follows from Lipschitzness of  $g^{*}$ , and the second inequality is Corollary 3. Furthermore since  $\overline{\eta} = \Theta (\| \mathbb{K}f^{*}\|_{L^2}^{-1}\iota^{-\chi})$ , we get that  $\overline{\eta}^{-1}\| \mathbb{K}f^{*}\|_{L^2}^{-1} = \Theta (\iota^{\chi})$ , and thus
$$
\begin{array}{l} \sup  _ {z \in [ - 1, 1 ]} | q (z) | = \sup  _ {z \in [ - \Theta (\iota x), \Theta (\iota x) ]} | g ^ {*} (z) | = \operatorname {p o l y} (\iota) \\ \sup  _ {z \in [ - 1, 1 ]} | q ^ {\prime} (z) | = \overline {{\eta}} ^ {- 1} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1} \sup  _ {z \in [ - \Theta (\iota^ {x}), \Theta (\iota^ {x}) ]} | (g ^ {*}) ^ {\prime} (z) | = \operatorname {p o l y} (\iota) \\ \sup  _ {z \in [ - 1, 1 ]} | q ^ {\prime \prime} (z) | = \left(\overline {{\eta}} ^ {- 1} \| \mathbb {K} f ^ {*} \| _ {L ^ {2}} ^ {- 1}\right) ^ {2} \sup  _ {z \in [ - \Theta (\iota x), \Theta (\iota x) ]} | (g ^ {*}) ^ {\prime \prime} (z) | = \operatorname {p o l y} (\iota) \\ \end{array}
$$
Therefore by Theorem 6 we can bound the population loss as
$$
\mathbb {E} _ {x} \left| f (x; \hat {\theta}) - f ^ {*} (x) \right| \lesssim \left(\frac {d ^ {4} \operatorname {p o l y} (\iota)}{\operatorname* {m i n} (n , m _ {1} , m _ {2})} + \frac {1}{\sqrt {n}} + \kappa^ {1 / 3} d ^ {- 1 / 6} \iota\right).
$$
E.2.1 Proof of Lemma 20
The high level sketch of the proof of Lemma 20 is as follows. Consider a second spherical harmonic  $x^{T}Bx$  satisfying  $\mathbb{E}[(x^{T}Ax)(x^{T}Bx)] = 0$  (a simple computation shows that this is equivalent to  $\mathrm{Tr}(AB) = 0$ ). We appeal to a key result in universality to show that in the large  $d$  limit, the distribution of  $x^{T}Ax$  converges to a standard Gaussian; additionally,  $x^{T}Bx$  converges to an independent mean-zero random variable. As a consequence, we show that
$$
\mathbb {E} [ q (x ^ {T} A x) x ^ {T} A x ] \approx \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} [ q (z) z ] = \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} [ q ^ {\prime} (z) ] = c _ {1}.
$$
$$
\mathbb {E} [ q (x ^ {T} A x) x ^ {T} B x ] \approx \mathbb {E} [ q (x ^ {T} A x) ] \cdot \mathbb {E} [ x ^ {T} B x ] = 0.
$$
From this, it immediately follows that  $P_{2}f^{*}\approx c_{1}x^{T}Ax$
The key universality theorem is the following.
Definition 7. For two probability measures  $\mu, \nu$ , the Wasserstein 1-distance between  $\mu$  and  $\nu$  is defined as
$$
W _ {1} (\mu , \nu) := \sup  _ {\| f \| _ {L i p} \leq 1} | \mathbb {E} _ {z \sim \mu} [ f (z) ] - \mathbb {E} _ {z \sim \nu} [ f (z) ] |,
$$
where  $\| f\|_{Lip}:= \sup_{x\neq y}\frac{|f(x) - f(y)|}{\|x - y\|_2}$  is the Lipschitz norm of  $f$
Lemma 22 ([52][Theorem 9.20). Let  $z \sim \mathcal{N}(0, I_d)$  be a standard Gaussian vector, and let  $f: \mathbb{R}^d \to \mathbb{R}$  satisfy  $\mathbb{E}[f(z)] = 0$ ,  $\mathbb{E}[(f(z))^2] = 1$ . Then
$$
W _ {1} (L a w (f (z)), \mathcal {N} (0, 1)) \lesssim \mathbb {E} \left[ \| \nabla f (z) \| ^ {4} \right] ^ {1 / 4} \mathbb {E} \left[ \| \nabla^ {2} f (z) \| _ {o p} ^ {4} \right] ^ {1 / 4},
$$
where  $W_{1}$  is the Wasserstein 1-distance.
We next apply this lemma to show that the quantities  $x^{T}Ax + x^{T}Bx$  and  $x^{T}Ax + x\cdot u$  are approximately Gaussian, given appropriate operator norm bounds on  $A, B$ .
Lemma 23. Let  $x^T Ax$  and  $x^T Bx$  be orthogonal spherical harmonics. Then, for constants  $c_1, c_2$  with  $c_1^2 + c_2^2 = 1$ , we have that the random variable  $Y = c_1 x^T A x + c_2 x^T B x$  satisfies
$$
W _ {1} (Y, \mathcal {N} (0, 1)) \lesssim \| A \| _ {o p} + \| B \| _ {o p}.
$$
Proof. Define the function  $f(z) = c_{1}d\frac{z^{T}Az}{\|z\|^{2}} +c_{2}d\frac{z^{T}Bz}{\|z\|^{2}}$ , and let  $x = \frac{z\sqrt{d}}{\|z\|}$ . Observe that when  $z \sim \mathcal{N}(0,I)$ , we have  $x \sim \mathrm{Unif}(\mathcal{S}^{d - 1}(\sqrt{d}))$ . Therefore  $f(z)$  is equal in distribution to  $Y$ . Define  $f_{1}(z) = d\frac{z^{T}Az}{\|z\|^{2}}$ . We compute
$$
\nabla f _ {1} (z) = 2 d \left(\frac {A z}{\| z \| ^ {2}} - \frac {z ^ {T} A z \cdot z}{\| z \| ^ {4}}\right)
$$
$$
\nabla^ {2} f _ {1} (z) = 2 d \left(\frac {A}{\| z \| ^ {2}} - \frac {2 A z z ^ {T}}{\| z \| ^ {4}} - \frac {2 z z ^ {T} A}{\| z \| ^ {4}} - 2 \frac {z ^ {T} A z}{\| z \| ^ {4}} I + 4 \frac {z ^ {T} A z z z ^ {T}}{\| z \| ^ {6}}\right).
$$
$$
\| \nabla f _ {1} (z) \| \leq 2 d \left(\frac {\| A z \|}{\| z \| ^ {2}} + \frac {\left| z ^ {T} A z \right|}{\| z \| ^ {3}}\right) \leq \frac {\sqrt {d}}{\| z \|} \cdot \| A x \| + \frac {\left| x ^ {T} A x \right|}{\| z \|}.
$$
$$
\left\| \nabla^ {2} f _ {1} (z) \right\| _ {o p} \lesssim \frac {d}{\| z \| ^ {2}} \| A \| _ {o p}.
$$
$\| z\|^2$  is distributed as a chi-squared random variable with  $d$  degrees of freedom, and thus
$$
\mathbb {E} \left[ \| z \| ^ {- 2 k} \right] = \frac {1}{\prod_ {j = 1} ^ {k} (d - 2 j)}
$$
$$
\mathbb {E} \left[ \left\| \nabla^ {2} f _ {1} (z) \right\| _ {o p} ^ {4} \right] ^ {1 / 4} \lesssim d \| A \| _ {o p} \mathbb {E} \left[ \| z \| ^ {- 8} \right] ^ {1 / 4} \lesssim \| A \| _ {o p}.
$$
and, using the fact that  $x$  and  $\|z\|$  are independent,
$$
\mathbb {E} \left[ \| \nabla f _ {1} (z) \| ^ {4} \right] ^ {1 / 4} \lesssim \sqrt {d} \mathbb {E} \left[ \| z \| ^ {- 4} \right] ^ {1 / 4} \mathbb {E} \left[ \| A x \| ^ {4} \right] ^ {1 / 4} + \mathbb {E} \left[ \| z \| ^ {- 4} \right] ^ {1 / 4} \mathbb {E} \left[ (x ^ {T} A x) ^ {4} \right] ^ {1 / 4} \lesssim 1.
$$
As a consequence, we have
$$
\mathbb {E} \left[ \| \nabla f (z) \| ^ {4} \right] ^ {1 / 4} \lesssim 1 \quad \text {a n d} \quad \mathbb {E} \left[ \left\| \nabla^ {2} f (z) \right\| _ {o p} ^ {4} \right] ^ {1 / 4} \lesssim \| A \| _ {o p} + \| B \| _ {o p}.
$$
Thus by Lemma 22 we have
$$
W _ {1} (Y, \mathcal {N} (0, 1)) = W _ {1} (f (z), \mathcal {N} (0, 1)) \lesssim \| A \| _ {o p} + \| B \| _ {o p}.
$$
Lemma 24. Let  $x^T Ax$  be a spherical harmonic, and  $\| u \| = 1$ . Then, for constants  $c_1, c_2$  with  $c_1^2 + c_2^2 = 1$ , we have that the random variable  $Y = c_1 x^T A x + c_2 x^T u$  satisfies
$$
W _ {1} (Y, \mathcal {N} (0, 1)) \lesssim \| A \| _ {o p}.
$$
where  $W_{1}$  is the 1-Wasserstein distance.
Proof. Define  $f_{2}(z) = \frac{\sqrt{d}z^{T}u}{\|z\|}$ . We have
$$
\nabla f _ {2} (z) = \sqrt {d} \left(\frac {u}{\| z \|} - \frac {z z ^ {T} u}{\| z \| ^ {3}}\right).
$$
$$
\nabla^ {2} f _ {2} (z) = \sqrt {d} \bigg (- \frac {u z ^ {T} + z u ^ {T}}{\| z \| ^ {3}} - \frac {z ^ {T} u}{\| z \| ^ {3}} I + 3 \frac {z ^ {T} u z z ^ {T}}{\| z \| ^ {5}} \bigg).
$$
$$
\| \nabla f _ {2} (z) \| \lesssim \frac {\sqrt {d}}{\| z \|} \quad \text {a n d} \quad \left\| \nabla^ {2} f _ {2} (z) \right\| _ {o p} \lesssim \frac {\sqrt {d}}{\| z \| ^ {2}},
$$
$$
\mathbb {E} \left[ \| \nabla f _ {1} (z) \| ^ {4} \right] ^ {1 / 4} \lesssim 1 \quad \text {a n d} \quad \mathbb {E} \left[ \| \nabla f _ {1} (z) \| ^ {4} \right] ^ {1 / 4} \lesssim \frac {1}{\sqrt {d}}.
$$
We finish using the same argument as above.
Lemma 23 implies that, when  $\| A\|_{op},\| B\|_{op}$  are small,  $(x^{T}Ax,x^{T}Bx)$  is close in distribution to the standard Gaussian in 2-dimensions. As a consequence,  $\mathbb{E}[q(x^T Ax)x^T Bx]\approx \mathbb{E}[q(z_1)]\mathbb{E}[z_2] = 0$  where  $z_{1},z_{2}$  are i.i.d Gaussians. This intuition is made formal in the following lemma.
Lemma 25. Let  $x^T Ax, x^T Bx$  be two orthogonal spherical harmonics. Then
$$
\left| \mathbb {E} \left[ q (x ^ {T} A x) x ^ {T} B x \right] \right| \leq L \left(\sqrt {\| A \| _ {o p} \log d} + \| B \| _ {o p} \log d\right)
$$
Proof. Define the function  $F(t) = \int_0^t q(s)ds$ . Then  $F'(t) = q(t)$ , so by a Taylor expansion we get that
$$
\left| F (x + \epsilon y) - F (x) - \epsilon y q (x) \right| \leq \epsilon^ {2} y ^ {2} L.
$$
$$
\left| \mathbb {E} \left[ q \left(x ^ {T} A x\right) x ^ {T} B x \right] \right| \leq \epsilon L \mathbb {E} \left[ \left(x ^ {T} B x\right) ^ {2} \right] + \epsilon^ {- 1} \left| \mathbb {E} \left[ F \left(x ^ {T} A x + \epsilon x ^ {T} B x\right) \right] - \mathbb {E} \left[ F \left(x ^ {T} A x\right) \right] \right|.
$$
Pick truncation radius  $R$ , and define the function  $\overline{F}(z) = F(\max(-R, \min(R, z))$ .  $\overline{F}$  has Lipschitz constant  $\sup_{z \in [-R, R]} |q(z)|$ , and thus since  $W_1(x^T Ax, \mathcal{N}(0, 1)) \lesssim \|A\|_{op}$ , we have
$$
\left| \mathbb {E} \bar {F} \left(x ^ {T} A x\right) - \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \bar {F} (z) \right| \lesssim \sup  _ {z \in [ - R, R ]} | q (z) | \cdot \| A \| _ {o p}.
$$
$$
\left| \mathbb {E} \left[ \overline {{F}} \left(x ^ {T} A x\right) - F \left(x ^ {T} A x\right) \right] \right| \leq \mathbb {E} \left[ \mathbf {1} _ {\left| x ^ {T} A x \right| > R} \left| F \left(x ^ {T} A x\right) \right| \right] \leq \mathbb {P} \left(\left| x ^ {T} A x \right| > R\right) \cdot \mathbb {E} \left[ F \left(x ^ {T} A x\right) ^ {2} \right] ^ {1 / 2}.
$$
$$
\left| \mathbb {E} _ {z} \big [ \overline {{F}} (z) - F (z) \big ] \right| \leq \mathbb {P} (| z | > R) \cdot \mathbb {E} \big [ F (z) ^ {2} \big ] ^ {1 / 2}.
$$
Since  $q$  is  $L$ -Lipschitz, we can bound  $|F(z)| \leq |z||q(0)| + \frac{1}{2} L|z|^2$ , and thus
$$
\mathbb {E} \left[ F (z) ^ {2} \right] \lesssim L ^ {2} \quad \text {a n d} \quad \mathbb {E} \left[ F (x ^ {T} A x) ^ {2} \right] \lesssim L ^ {2}.
$$
The standard Gaussian tail bound yields  $\mathbb{P}(|z| > R) \lesssim \exp(-C_1R^2)$  for appropriate constant  $C_1$ , and polynomial concentration yields  $\mathbb{P}\big(|x^T Ax| > R\big) \lesssim \exp(-C_2R)$  for appropriate constant  $C_2$ . Thus choosing  $R = C_3$  log  $d$  for appropriate constant  $C_3$ , we get that
$$
\left| \mathbb {E} \left[ \bar {F} \left(x ^ {T} A x\right) - F \left(x ^ {T} A x\right) \right] \right| + \left| \mathbb {E} _ {z} \left[ \bar {F} (z) - F (z) \right] \right| \lesssim \frac {L}{d}.
$$
Altogether, since  $|q(z)| \leq |q(0)| + L|z|$ , we get that
$$
\left| \mathbb {E} F (x ^ {T} A x) - \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} F (z) \right| \lesssim \| A \| _ {o p} L \log d.
$$
By an identical calculation, we have that for  $\epsilon < 1$
$$
\left| \mathbb {E} F (x ^ {T} A x + x ^ {T} B x) - \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} F (z \sqrt {1 + \epsilon^ {2}}) \right| \lesssim \left(\| A \| _ {o p} + \epsilon \| B \| _ {o p}\right) \cdot L \log d
$$
Altogether, we get that
$$
\begin{array}{l} \left| \mathbb {E} \left[ F \left(x ^ {T} A x + \epsilon x ^ {T} B x\right) \right] - \mathbb {E} \left[ F \left(x ^ {T} A x\right) \right] \right| \\ \leq \left(\| A \| _ {o p} + \epsilon \| B \| _ {o p}\right) \cdot L \log d + \left| \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} F (z) - \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} F (z \sqrt {1 + \epsilon^ {2}}) \right|. \\ \end{array}
$$
Via a simple calculation, one sees that
$$
\left| F (z \sqrt {1 + \epsilon^ {2}}) - F (z) \right| \leq | q (0) | | z | \left(\sqrt {1 + \epsilon^ {2}} - 1\right) + \frac {L}{2} z ^ {2} \epsilon^ {2} \lesssim L | z | \epsilon^ {2} + L z ^ {2} \epsilon^ {2}.
$$
$$
\left| \mathbb {E} \left[ F \left(x ^ {T} A x + \epsilon x ^ {T} B x\right) \right] - \mathbb {E} \left[ F \left(x ^ {T} A x\right) \right] \right| \lesssim L \left(\left(\left\| A \right\| _ {o p} + \epsilon \| B \| _ {o p}\right) \log d + \epsilon^ {2}\right),
$$
$$
\left| \mathbb {E} \left[ q \left(x ^ {T} A x\right) x ^ {T} B x \right] \right| \leq L \left(\epsilon^ {- 1} \| A \| _ {o p} \log d + \epsilon + \| B \| _ {o p} \log d\right).
$$
Setting  $\epsilon = \sqrt{\|A\|_{op}\log d}$  yields
$$
\left| \mathbb {E} \left[ q (x ^ {T} A x) x ^ {T} B x \right] \right| \lesssim L \Big (\sqrt {\| A \| _ {o p} \log d} + \| B \| _ {o p} \log d \Big),
$$
Similarly, we use the consequence of Lemma 24 that  $(x^{T}Ax,u^{T}x)$  is close in distribution to a 2d standard Gaussian, and show that  $\mathbb{E}[q(x^T Ax)\big((u^T x)^2 -1\big)]\approx \mathbb{E}[q(z_1)\big(z_2^2 -1\big)] = 0$
Lemma 26.
$$
\left| \mathbb {E} \left[ q (x ^ {T} A x) (u ^ {T} x) ^ {2} \right] - \mathbb {E} _ {z} [ q (z) ] \right| \lesssim L \| A \| _ {o p} ^ {1 / 3} \log^ {2 / 3} d.
$$
Proof. Let  $G(t) = \int_0^t F(s)ds$ . Then  $G'(t) = F(t), G''(t) = q(t)$ , so a Taylor expansion yields
$$
G (x + \epsilon y) = G (x) + \epsilon y F (x) + \epsilon^ {2} y ^ {2} q (x) + O (\epsilon^ {3} | y | ^ {3} L)
$$
$$
G (x - \epsilon y) = G (x) - \epsilon y F (x) + \epsilon^ {2} y ^ {2} q (x) + O (\epsilon^ {3} | y | ^ {3} L).
$$
$$
\epsilon^ {2} y ^ {2} q (x) = \frac {1}{2} (G (x + \epsilon y) + G (x - \epsilon y) - 2 G (x)) + O (\epsilon^ {3} | y | ^ {3} L).
$$
$$
\left| \mathbb {E} \left[ q \left(x ^ {T} A x\right) \left(u ^ {T} x\right) ^ {2} \right] \right| \lesssim \epsilon L + \epsilon^ {- 2} \left| \mathbb {E} \left[ G \left(x ^ {T} A x + \epsilon u ^ {T} x\right) + G \left(x ^ {T} A x - \epsilon u ^ {T} x\right) - 2 G \left(x ^ {T} A x\right) \right] \right|.
$$
For truncation radius  $R$ , define  $\overline{G}(z) = G(\max(-R, \min(R, z)))$ . We get that  $G$  has Lipschitz constant  $\sup_{|z| \leq R} |F(z)| \lesssim LR^2$ . Therefore
$$
\left| \mathbb {E} \overline {{G}} \left(x ^ {T} A x\right) - \mathbb {E} _ {z \sim \mathcal {N} (0, 1)} \overline {{G}} (z) \right| \lesssim L R ^ {2} | A | _ {o p},
$$
and by a similar argument in the previous lemma, setting  $R = C_3 \log d$  yields
$$
\left| \mathbb {E} \left[ \overline {{G}} \left(x ^ {T} A x\right) - G \left(x ^ {T} A x\right) \right] \right| + \left| \mathbb {E} _ {z} \overline {{G}} (z) - G (z) \right| \lesssim \frac {L}{d}.
$$
$$
\left| \mathbb {E} G \left(x ^ {T} A x\right) - \mathbb {E} _ {z} G (z) \right| \lesssim \| A \| _ {o p} L \log^ {2} d.
$$
By an identical calculation,
$$
\left| \mathbb {E} G \left(x ^ {T} A x \pm \epsilon u ^ {T} x\right) - \mathbb {E} _ {z} G \left(z \sqrt {1 + \epsilon^ {2}}\right) \right| \lesssim \| A \| _ {o p} \cdot L \log^ {2} d.
$$
Additionally, letting  $z, w$  be independent standard Gaussians,
$$
\begin{array}{l} \epsilon^ {- 2} \mathbb {E} _ {z} \left[ 2 G (z \sqrt {1 + \epsilon^ {2}} - 2 G (z) \right] = \epsilon^ {- 2} \mathbb {E} _ {z, w} [ G (z + \epsilon w) + G (z - \epsilon w) - 2 G (z) ] \\ = \mathbb {E} _ {z, w} [ q (z) w ^ {2} ] + O (\epsilon L) \\ = \mathbb {E} _ {z} [ q (z) ] + O (\epsilon L). \\ \end{array}
$$
$$
\begin{array}{l} \left| \mathbb {E} \left[ q (x ^ {T} A x) (u ^ {T} x) ^ {2} \right] - \mathbb {E} _ {z} [ q (z) ] \right| \lesssim \epsilon L + \epsilon^ {- 2} \| A \| _ {o p} \cdot L \log^ {2} d \\ \lesssim L \| A \| _ {o p} ^ {1 / 3} \log^ {2 / 3} d, \\ \end{array}
$$
where we set  $\epsilon = \| A\|_{op}^{1 / 3}\log^{2 / 3}d$
Lemma 25 shows that when  $\| B\|_{op}\ll 1$ ,  $\mathbb{E}[q(x^T Ax)x^T Bx]\approx 0$ . However, we need to show this is true for all spherical harmonics, even those with  $\| B\|_{op} = \Theta (1)$ . To accomplish this, we decompose  $B$  into the sum of a low rank component and small operator norm component. We use Lemma 25 to bound the small operator norm component, and Lemma 26 to bound the low rank component. Optimizing over the rank threshold yields the following desired result:
Lemma 27. Let  $A, B$  be orthogonal spherical harmonics. Then
$$
\left| \mathbb {E} \left[ q \left(x ^ {T} A x\right) x ^ {T} B x \right] \right| \leq L \| A \| _ {o p} ^ {1 / 6} \log d.
$$
Proof. Let  $\tau >\| A\|_{op}$  be a threshold to be determined later. Decompose  $B$  as follows:
$$
B = \sum_ {i = 1} ^ {d} \lambda_ {i} u _ {i} u _ {i} ^ {T} = \sum_ {| \lambda_ {i} | > \tau} \lambda_ {i} \left(u _ {i} u _ {i} ^ {T} - \frac {1}{d} I\right) - \frac {1}{\| A \| _ {F} ^ {2}} \sum_ {| \lambda_ {i} | > \tau} u _ {i} ^ {T} A u _ {i} \cdot A + \tilde {B},
$$
$$
\tilde {B} = \sum_ {| \lambda_ {i} | \leq \tau} \lambda_ {i} u _ {i} u _ {i} ^ {T} + I \cdot \frac {1}{d} \sum_ {| \lambda_ {i} | > \tau} \lambda_ {i} + \frac {1}{\| A \| _ {F} ^ {2}} \sum_ {| \lambda_ {i} | > \tau} \lambda_ {i} u _ {i} ^ {T} A u _ {i} \cdot A.
$$
By construction, we have,
$$
\operatorname {T r} \left(\tilde {B}\right) = \sum_ {| \lambda_ {i} | \leq \tau} \lambda_ {i} + \sum_ {| \lambda_ {i} | > \tau} \lambda_ {i} = \sum_ {i \in [ d ]} \lambda_ {i} = 0
$$
$$
\langle \tilde {B}, A \rangle = \sum_ {| \lambda_ {i} | \leq \tau} \lambda_ {i} u _ {i} ^ {T} A u _ {i} + \sum_ {| \lambda_ {i} | > \tau} \lambda_ {i} u _ {i} ^ {T} A u _ {i} = \langle A, B \rangle = 0.
$$
Therefore by Lemma 25,
$$
\left| \mathbb {E} \left[ q (x ^ {T} A x) x ^ {T} \tilde {B} x \right] \right| \lesssim L \sqrt {\| A \| _ {o p} \log d} \left\| \tilde {B} \right\| _ {F} + L \left\| \tilde {B} \right\| _ {o p} \log d.
$$
There are at most  $O(\tau^{-2})$  indices  $i$  satisfying  $|\lambda_i| > \tau$ , and thus
$$
\sum_ {| \lambda_ {i} | > \tau} | \lambda_ {i} | \lesssim \sqrt {\tau^ {- 2} \cdot \sum_ {| \lambda_ {i} | > \tau} | \lambda_ {i} | ^ {2}} \leq \tau^ {- 1}.
$$
We thus compute that