SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\gamma_ {4} \leq \gamma_ {2} + O \left(\frac {1}{\sqrt {d}}\right), \tag {G.2}
$$
$$
\gamma_ {2} \leq 1. \tag {G.3}
$$
Proof. Similar to the proof of Lemma 4.2, we have
$$
\begin{array}{l} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ (f _ {\rho} (x) - h (x ^ {\top} e _ {1})) ^ {2} ] = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(\underset {u \sim \rho} {\mathbb {E}} [ \sigma (u ^ {\top} x) ] - h (x ^ {\top} e _ {1})\right) ^ {2} (G.4) \\ = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(\underset {u \sim \rho} {\mathbb {E}} \left[ \sum_ {k = 0} ^ {\infty} \hat {\sigma} _ {k, d} \overline {{P _ {k , d}}} (u ^ {\top} x) \right] - \sum_ {k = 0} ^ {\infty} \hat {h} _ {k, d} \bar {P} _ {k, d} (x ^ {\top} e _ {1})\right) ^ {2} (G.5) \\ = \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(\sum_ {k = 0} ^ {\infty} \left(\hat {\sigma} _ {k, d} \underset {u \sim \rho} {\mathbb {E}} \left[ \bar {P} _ {k, d} \left(u ^ {\top} x\right) \right] - \hat {h} _ {k, d} \bar {P} _ {k, d} \left(x ^ {\top} e _ {1}\right)\right)\right) ^ {2}. (G.6) \\ \end{array}
$$
Continuing the equation by invoking Lemma C.5, we get
$$
\begin{array}{l} \underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left(\sum_ {k = 0} ^ {\infty} \left(\hat {\sigma} _ {k, d} \underset {u \sim \rho} {\mathbb {E}} \left[ \bar {P} _ {k, d} \left(u ^ {\top} x\right) \right] - \hat {h} _ {k, d} \bar {P} _ {k, d} \left(x ^ {\top} e _ {1}\right)\right)\right) ^ {2} (G.7) \\ = \sum_ {k = 0} ^ {\infty} \mathbb {E} _ {x \sim \mathbb {S} ^ {d - 1}} \left(\hat {\sigma} _ {k, d} \underset {u \sim \rho} {\mathbb {E}} \left[ \bar {P} _ {k, d} \left(u ^ {\top} x\right) \right] - \hat {h} _ {k, d} \bar {P} _ {k, d} \left(x ^ {\top} e _ {1}\right)\right) ^ {2} (G.8) \\ = \sum_ {k = 0} ^ {\infty} \hat {\sigma} _ {k, d} ^ {2} \left(\underset {u, u ^ {\prime} \sim \rho} {\mathbb {E}} \left[ P _ {k, d} \left(u ^ {\top} u ^ {\prime}\right) - 2 \gamma_ {k} P _ {k, d} \left(u ^ {\top} e _ {1}\right) + \gamma_ {k} ^ {2} P _ {k, d} \left(e _ {1} ^ {\top} e _ {1} \right]\right)\right) ^ {2}. (G.9) \\ \end{array}
$$
Therefore,  $\mathbb{E}_{x\sim \mathbb{S}^{d - 1}}[(f_{\rho}(x) - h(x^{\top}e_1))^2 ] = 0$  implies that
$$
\underset {u, u ^ {\prime} \sim \rho} {\mathbb {E}} \left[ P _ {2, d} \left(u ^ {\top} u ^ {\prime}\right) - 2 \gamma_ {2} P _ {2, d} \left(u ^ {\top} e _ {1}\right) + \gamma_ {2} ^ {2} P _ {2, d} \left(e _ {1} ^ {\top} e _ {1}\right) \right] = 0, \tag {G.10}
$$
$$
\underset {u, u ^ {\prime} \sim \rho} {\mathbb {E}} \left[ P _ {4, d} \left(u ^ {\top} u ^ {\prime}\right) - 2 \gamma_ {4} P _ {4, d} \left(u ^ {\top} e _ {1}\right) + \gamma_ {4} ^ {2} P _ {4, d} \left(e _ {1} ^ {\top} e _ {1}\right) \right] = 0. \tag {G.11}
$$
First we prove Eq. (G.3). By Lemma C.6, there is a feature mapping  $\phi : \mathbb{S}^{d-1} \to \mathbb{R}^{N_{2,d}}$  such that for every  $u, v \in \mathbb{S}^{d-1}$ ,
$$
\langle \phi (u), \phi (v) \rangle = N _ {2, d} P _ {2, d} \left(u ^ {\top} v\right), \tag {G.12}
$$
$$
\left\| \phi_ {u} \right\| _ {2} ^ {2} = N _ {2, d}. \tag {G.13}
$$
Rewrite Eq. (G.10) using  $\phi_{2}$  we get
$$
\left\| \mathbb {E} _ {u \sim \rho} \left[ \phi_ {2} (u) \right] - \gamma_ {2} \phi_ {2} \left(e _ {1}\right) \right\| _ {2} ^ {2} = 0. \tag {G.14}
$$
$$
\gamma_ {2} \phi_ {2} (e _ {1}) = \underset {u \sim \rho} {\mathbb {E}} [ \phi_ {2} (u) ] \tag {G.15}
$$
$$
\gamma_ {2} = \left\| \frac {\phi_ {2} \left(e _ {1}\right)}{\sqrt {N _ {k , d}}} \right\| _ {2} = \left\| \frac {\mathbb {E} _ {u \sim \rho} [ \phi_ {2} (u) ]}{\sqrt {N _ {k , d}}} \right\| _ {2} \leq \frac {\mathbb {E} _ {u \sim \rho} [ \| \phi_ {2} (u) ] \| _ {2}}{\sqrt {N _ {2 , d}}} = 1. \tag {G.16}
$$
Similarly, we also have  $\gamma_4\leq 1$
Now we prove Eq. (G.1). Recall that
$$
P _ {2, d} (t) = \frac {d}{d - 1} t ^ {2} - \frac {1}{d - 1}, \tag {G.17}
$$
$$
P _ {4, d} (t) = \frac {(d + 2) (d + 4)}{d ^ {2} - 1} t ^ {4} - \frac {6 d + 1 2}{d ^ {2} - 1} t ^ {2} + \frac {3}{d ^ {2} - 1}. \tag {G.18}
$$
Combining Eq. (G.10) and Eq. (G.17) we have
$$
\begin{array}{l} 0 = \frac {d - 1}{d} \underset {u, u ^ {\prime} \sim \rho} {\mathbb {E}} \left[ P _ {2, d} \left(u ^ {\top} u ^ {\prime}\right) - 2 \gamma_ {2} P _ {2, d} \left(u ^ {\top} e _ {1}\right) + \gamma_ {2} ^ {2} P _ {2, d} \left(e _ {1} ^ {\top} e _ {1}\right) \right] (G.19) \\ = \underset {u, u ^ {\prime} \sim \rho} {\mathbb {E}} \left[ \left(u ^ {\top} u ^ {\prime}\right) ^ {2} - 2 \gamma_ {2} \left(u ^ {\top} e _ {1}\right) ^ {2} + \gamma_ {2} ^ {2} \left(e _ {1} ^ {\top} e _ {1}\right) ^ {2} \right] - \frac {1}{d} \left(1 - 2 \gamma_ {2} + \gamma_ {2} ^ {2}\right) (G.20) \\ = \| _ {u \sim \rho} \mathbb {E} [ u ^ {\otimes 2} - \gamma_ {2} e _ {1} ^ {\otimes 2} ] \| _ {F} ^ {2} - \frac {1}{d} (1 - \gamma_ {2}) ^ {2}. (G.21) \\ \end{array}
$$
$$
\left\| _ {u \sim \rho} \mathbb {E} \left[ u ^ {\otimes 2} - \gamma_ {2} e _ {1} ^ {\otimes 2} \right] \right\| _ {F} ^ {2} = \frac {1}{d} (1 - \gamma_ {2}) ^ {2}. \tag {G.22}
$$
Similarly, combining Eq. (G.11) and Eq. (G.18) we get
$$
\begin{array}{l} 0 = \frac {d ^ {2} - 1}{(d + 2) (d + 4)} \underset {u, u ^ {\prime} \sim \rho} {\mathbb {E}} \left[ P _ {4, d} \left(u ^ {\top} u ^ {\prime}\right) - 4 \gamma_ {4} P _ {4, d} \left(u ^ {\top} e _ {1}\right) + \gamma_ {4} ^ {2} P _ {4, d} \left(e _ {1} ^ {\top} e _ {1}\right) \right] (G.23) \\ = \underset {u, u ^ {\prime} \sim \rho} {\mathbb {E}} \left[ \left(u ^ {\top} u ^ {\prime}\right) ^ {4} - 2 \gamma_ {4} \left(u ^ {\top} e _ {1}\right) ^ {4} + \gamma_ {4} ^ {2} \left(e _ {1} ^ {\top} e _ {1}\right) ^ {4} \right] + \frac {3}{(d + 2) (d + 4)} \left(1 - 2 \gamma_ {4} + \gamma_ {4} ^ {2}\right) (G.24) \\ - \frac {6 d + 1 2}{(d + 2) (d + 4)} \underset {u, u ^ {\prime} \sim \rho} {\mathbb {E}} \left[ \left(u ^ {\top} u ^ {\prime}\right) ^ {2} - 2 \gamma_ {4} \left(u ^ {\top} e _ {1}\right) ^ {2} + \gamma_ {4} ^ {2} \left(e _ {1} ^ {\top} e _ {1}\right) ^ {2} \right]. (G.25) \\ \end{array}
$$
$$
\left\| _ {u \sim \rho} \mathbb {E} \left[ u ^ {\otimes 4} - \gamma_ {4} e _ {1} ^ {\otimes 4} \right] \right\| _ {F} ^ {2} \leq \frac {3}{(d + 2) (d + 4)} \left(1 - \gamma_ {4}\right) ^ {2} + \frac {6 d + 1 2}{(d + 2) (d + 4)} (\gamma_ {4} + 1) ^ {2}. \tag {G.26}
$$
Now for any fixed  $\xi \in \mathbb{S}^{d - 1}$ , our key observation is that by Jensen's inequality,
$$
\underset {u \sim \rho} {\mathbb {E}} \left[ \left(u ^ {\top} \xi\right) ^ {2} \right] ^ {2} \leq \underset {u \sim \rho} {\mathbb {E}} \left[ \left(u ^ {\top} \xi\right) ^ {4} \right]. \tag {G.27}
$$
On the one hand, by Eq. (G.22) we have
$$
\begin{array}{l} \left| \mathbb {E} _ {u \sim \rho} [ (u ^ {\top} \xi) ^ {2} ] - \gamma_ {2} \left(e _ {1} ^ {\top} \xi\right) ^ {2} \right| = \left| \mathbb {E} _ {u \sim \rho} [ \langle u ^ {\otimes 2} - \gamma_ {2} e _ {1} ^ {\otimes 2}, \xi^ {\otimes 2} \rangle ] \right| (G.28) \\ \leq \left\| _ {u \sim \rho} \mathbb {E} \left[ u ^ {\otimes 2} - \gamma_ {2} e _ {1} ^ {\otimes 2} \right] \right\| _ {F} \| \xi^ {\otimes 2} \| _ {F} = \sqrt {\frac {(1 - \gamma_ {2}) ^ {2}}{d}}. (G.29) \\ \end{array}
$$
On the other hand, by Eq. (G.26) we have
$$
\begin{array}{l} \left| \underset {u \sim \rho} {\mathbb {E}} \left[ \left(u ^ {\top} \xi\right) ^ {4} \right] - \gamma_ {4} \left(e _ {1} ^ {\top} \xi\right) ^ {4} \right| = \left| \underset {u \sim \rho} {\mathbb {E}} \left[ \left\langle u ^ {\otimes 4} - \gamma_ {4} e _ {1} ^ {\otimes 4}, \xi^ {\otimes 4} \right\rangle \right] \right| (G.30) \\ \leq \| _ {u \sim \rho} \mathbb {E} [ u ^ {\otimes 4} - \gamma_ {4} e _ {1} ^ {\otimes 4} ] \| _ {F} \| \xi^ {\otimes 4} \| _ {F} \leq 2 \sqrt {\frac {6 d + 1 2}{(d + 2) (d + 4)}} (| \gamma_ {4} + 1 | + | \gamma_ {4} - 1 |). (G.31) \\ \end{array}
$$
Finally, combining Eqs (G.27)-(G.31) and set  $\xi = e_1$  we get
$$
\left(\gamma_ {2} - \sqrt {\frac {(1 - \gamma_ {2}) ^ {2}}{d}}\right) ^ {2} \leq \gamma_ {4} + 2 \sqrt {\frac {6 d + 1 2}{(d + 2) (d + 4)}} \left(| \gamma_ {4} + 1 | + | \gamma_ {4} - 1 |\right). \tag {G.32}
$$
$$
\gamma_ {2} ^ {2} \leq \gamma_ {4} + O \left(\frac {1}{\sqrt {d}}\right). \tag {G.33}
$$
Finally we prove Eq. (G.2). For any  $\xi, u \in \mathbb{S}^{d-1}$  we have  $|u^\top \xi| \leq 1$ . As a result,
$$
\underset {u \sim \rho} {\mathbb {E}} \left[ \left(u ^ {\top} \xi\right) ^ {4} \right] \leq \underset {u \sim \rho} {\mathbb {E}} \left[ \left(u ^ {\top} \xi\right) ^ {2} \right]. \tag {G.34}
$$
Combining with Eq. (G.29) and Eq. (G.31) and setting  $\xi = e_1$  we get
$$
\gamma_ {4} \leq O (1 / \sqrt {d}) + \underset {u \sim \rho} {\mathbb {E}} \left[ (u ^ {\top} \xi) ^ {4} \right] \leq O (1 / \sqrt {d}) + \underset {u \sim \rho} {\mathbb {E}} \left[ (u ^ {\top} \xi) ^ {2} \right] \leq O (1 / \sqrt {d}) + \gamma_ {2}. \tag {G.35}
$$
Proposition G.2 (Sufficient condition). Let  $f_{\rho}(x) = \mathbb{E}_{u \sim \rho}[\sigma(u^{\top}x)]$  be a two-layer neural network and  $y(x) = h(e_1^\top x)$  the target function. Let  $\gamma_k = \frac{\hat{h}_{k,d}}{\hat{\sigma}_{k,d}}, \forall k \geq 0$ . Suppose  $\hat{h}_{k,d} = 0, \forall k \notin \{0,2,4\}$ ,  $\hat{h}_{0,d} = \hat{\sigma}_{0,d}$ ,  $\sigma$  is a degree-4 polynomial, and
$$
\gamma_ {2} ^ {2} \leq \gamma_ {4} - O (1 / d), \tag {G.36}
$$
$$
\gamma_ {4} \leq \gamma_ {2} - O (1 / d), \tag {G.37}
$$
$$
\gamma_ {2} \leq 1 - O (1 / d). \tag {G.38}
$$
Then there exists a symmetric and rotational invariant neural network  $\rho$  such that  $\mathbb{E}_{x\sim \mathbb{S}^{d - 1}}(f_{\rho}(x) - y(x))^2 = 0$ .
Proof. Recall that by Lemma 4.2, for every symmetric and rotational invariant neural network  $\rho$  we have
$$
L (\rho) = \frac {\hat {\sigma} _ {2 , d} ^ {2}}{2} \left(\underset {u \sim \rho} {\mathbb {E}} [ P _ {2, d} (w) ] - \gamma_ {2}\right) ^ {2} + \frac {\hat {\sigma} _ {4 , d} ^ {2}}{2} \left(\underset {u \sim \rho} {\mathbb {E}} [ P _ {4, d} (w) ] - \gamma_ {4}\right) ^ {2} \tag {G.39}
$$
where  $u = (w,z)$ ,  $w \in [-1,1]$ , and  $z \in \sqrt{1 - w^2} \mathbb{S}^{d-2}$ . Therefore, we only need to show that there exists a one-dimensional distribution  $\mu$  such that
$$
\underset {w \sim \mu} {\mathbb {E}} [ P _ {2, d} (w) ] = \gamma_ {2}, \tag {G.40}
$$
$$
\underset {w \sim \mu} {\mathbb {E}} \left[ P _ {4, d} (w) \right] = \gamma_ {4}. \tag {G.41}
$$
$$
P _ {2, d} (t) = \frac {d}{d - 1} t ^ {2} - \frac {1}{d - 1}, \tag {G.42}
$$
$$
P _ {4, d} (t) = \frac {(d + 2) (d + 4)}{d ^ {2} - 1} t ^ {4} - \frac {6 d + 1 2}{d ^ {2} - 1} t ^ {2} + \frac {3}{d ^ {2} - 1}. \tag {G.43}
$$
Consequently, Eq. (G.40) and Eq. (G.41) is equivalent to
$$
\underset {w \sim \mu} {\mathbb {E}} [ w ^ {2} ] = \frac {d - 1}{d} \gamma_ {2} + \frac {1}{d}, \tag {G.44}
$$
$$
\underset {w \sim \mu} {\mathbb {E}} \left[ w ^ {4} \right] = \frac {d ^ {2} - 1}{(d + 2) (d + 4)} \gamma_ {4} + \frac {6 d + 1 2}{(d + 2) (d + 4)} \left(\frac {d - 1}{d} \gamma_ {2} + \frac {1}{d}\right) - \frac {3}{(d + 2) (d + 4)}. \tag {G.45}
$$
Let  $\beta_{2} = \frac{d - 1}{d}\gamma_{2} + \frac{1}{d}$  and  $\beta_{4} = \frac{d^{2} - 1}{(d + 2)(d + 4)}\gamma_{4} + \frac{6d + 12}{(d + 2)(d + 4)}\left(\frac{d - 1}{d}\gamma_{2} + \frac{1}{d}\right) - \frac{3}{(d + 2)(d + 4)}$ . Then we have  $|\beta_{2} - \gamma_{2}| \leq O(1 / d)$  and  $|\beta_{4} - \gamma_{4}| \leq O(1 / d)$ . Rewriting Eqs. (G.36)-(G.38) using  $\beta_{2}$  and  $\beta_{4}$ , we have
$$
\beta_ {2} ^ {2} \leq \beta_ {4}, \tag {G.46}
$$
$$
\beta_ {4} \leq \beta_ {2}, \tag {G.47}
$$
$$
\beta_ {2} \leq 1. \tag {G.48}
$$
By Proposition I.14, there exists a distribution  $\mu^{\star}$  such that
$$
\underset {w \sim \mu^ {\star}} {\mathbb {E}} \left[ w ^ {2} \right] = \beta_ {2}, \tag {G.49}
$$
$$
\underset {w \sim \mu^ {\star}} {\mathbb {E}} \left[ w ^ {4} \right] = \beta_ {4}. \tag {G.50}
$$
Hence, by setting the marginal distribution of  $w$  equal to  $\mu^{\star}$ , we have
$$
\underset {w \sim \mu^ {\star}} {\mathbb {E}} [ P _ {2, d} (w) ] = \gamma_ {2}, \quad \underset {w \sim \mu^ {\star}} {\mathbb {E}} [ P _ {4, d} (w) ] = \gamma_ {4}, \tag {G.51}
$$
which proves the desired result.
H Proofs for Sample Complexity Lower Bounds for Kernel Method
In this section, we present a sample complexity lower bound for kernel methods with any inner product kernel. That is, the kernel  $K: \mathbb{S}^{d-1} \times \mathbb{S}^{d-1} \to \mathbb{R}$  can be written as  $K(x,z) = \kappa(x^{\top}z)$  for some one-dimensional function  $\kappa: [-1,1] \to \mathbb{R}$ . Inner product kernels are rotational invariant and do not specialize to any coordinate. In particular, NTK of two-layer neural networks is an inner product kernel (e.g., Du et al. [30], Wei et al. [76]).
To prove Theorem 3.6, we only need to work on the degree- $k$  spherical harmonics component of the target function and its estimation because, by the orthogonality of spherical harmonics, any lower bound for the degree- $k$  components directly translates to the original setting. The following lemma states the lower bound for general  $k \geq 0$ , while Theorem 3.6 only requires the case  $k = 4$ .
Lemma H.1. Fixed any  $k \geq 1$ , let  $g^{\star}(x) = \overline{P}_{k,d}(u^{\top}x)$  for some fixed  $u \in \mathbb{S}^{d-1}$ . When  $d$  is larger than a sufficiently large universal constant and  $n < d^{k}(8k)^{-(3k+1)}(\ln d)^{-(k+2)}$ , with probability at least  $1/2$  over the randomness of  $n$  i.i.d. data  $\{x_{1}, \dots, x_{n}\}$  drawn uniformly from  $\mathbb{S}^{d-1}$ , any estimation  $g$  of the form  $g(x) = \sum_{i=1}^{n} \beta_{i} \overline{P}_{k,d}(x^{\top}x_{i})$  must have a constant error:
$$
\underset {x \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} (g (x) - g (x) ^ {\star}) ^ {2} \geq 3 / 4. \tag {H.1}
$$
Proof of Lemma H.1 is deferred to Appendix H.2, where our key observation is that  $\overline{P}_{k,d}(\langle x_i,\cdot \rangle)$  cannot correlate too much with  $g^{\star}$  when  $x_{i}$  is uniformly randomly drawn from the unit sphere. Indeed,  $\langle \overline{P}_{k,d}(\langle x_i,\cdot \rangle),g^{\star}\rangle = P_{k,d}(x_i^\top u)$  concentrates within  $\pm 1 / N_{k,d}$  (Lemma C.7). Hence, we can upperbound  $\langle g,g^{\star}\rangle = \sum_{i = 1}^{n}\beta_{i}P_{k,d}(x_{i}^{\top}u)\lesssim \| \beta \| _ { 2 }\sqrt { n / N _ { k , d } }$  . However, we also have  $\| g\| _2^2 =$ $\sum_{i,j\in [n]}\beta_i\beta_jP_{k,d}(x_i^\top x_j)\gtrsim \| \beta \| _2^2$  because the matrix  $(P_{k,d}(x_i^\top x_j))_{i,j\in [n]}$  concentrates around  $I$  (Lemma H.3). Since  $\| g - g^{\star}\| _2^2 < 3 / 4$  requires  $\langle g,g^{\star}\rangle >\| g\| _2 / 2$  , we must have  $n\gtrsim N_{k,d}\approx d^{-k}$
H.1 Proof of Theorem 3.6
The following theorem states that for every  $k \geq 0$ , to achieve a constant population error, we need at least  $C_k d^{-k} (\ln d)^{-(k+2)}$  samples where  $C_k = (8k)^{-(3k+1)}$  is a constant that only depends on  $k$ . Hence, Theorem 3.6 is a direct corollary of Theorem H.2 by taking  $k = 4$ .
Theorem H.2. Let  $y: \mathbb{S}^{d-1} \to \mathbb{R}$  be a function of the form  $y(x) = h(q_{\star}^{\top}x)$  where  $q_{\star} \in \mathbb{S}^{d-1}$  is a fixed vector and  $h: [-1,1] \to \mathbb{R}$  is a one-dimensional function. Let  $K: \mathbb{S}^{d-1} \times \mathbb{S}^{d-1} \to \mathbb{R}$  be any inner product kernel. When  $d$  is larger than a universal constant and  $n < d^{k}(8k)^{-(3k+1)}(\ln d)^{-(k+2)}$ , with probability at least  $1/2$  over the randomness of  $n$  i.i.d. data points  $\{x_{i}\}_{i=1}^{n}$ , any estimation  $f$  of the form  $f(x) = \sum_{i=1}^{n} \beta_{i}K(x_{i},x)$  must have a significant error:
$$
\| y - f \| _ {2} ^ {2} \geq \frac {3}{4} (\hat {h} _ {k, d}) ^ {2}.
$$
Proof of Theorem H.2. In the following we first fix a degree  $k \geq 1$ . For every  $l > 0$ , let  $\mathbb{Y}_{l,d}$  be the space of degree-  $l$  spherical harmonics and  $\Pi_{l}$  the projection to  $\mathbb{Y}_{l,d}$ . By the orthogonality of  $\mathbb{Y}_{l,d}$ , we have
$$
\| y - f \| _ {2} ^ {2} = \sum_ {l = 0} ^ {\infty} \| \Pi_ {l} (y - f) \| _ {2} ^ {2} \geq \| \Pi_ {k} (y - f) \| _ {2} ^ {2}. \tag {H.2}
$$
By Atkinson and Han [12, Section 2.3], the projection operator  $\Pi_k$  is given by
$$
\left(\Pi_ {k} f\right) (x) = \sqrt {N _ {k , d}} \underset {\xi \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} \left[ \bar {P} _ {k, d} \left(x ^ {\top} \xi\right) f (\xi) \right]. \tag {H.3}
$$
Consequently, we have  $(\Pi_k y)(x) = \sqrt{N_{k,d}} \mathbb{E}_{\xi \sim \mathbb{S}^{d-1}}[\overline{P}_{k,d}(x^\top \xi)h(q_\star^\top \xi)]$ . Recall that
$$
\mu_ {d} (t) \triangleq (1 - t ^ {2}) ^ {\frac {d - 3}{2}} \frac {\Gamma (d / 2)}{\Gamma ((d - 1) / 2)} \frac {1}{\sqrt {\pi}} \tag {H.4}
$$
is the density of  $u_{1}$  when  $u = (u_{1},\dots ,u_{d})$  is drawn uniformly from  $\mathbb{S}^{d - 1}$ , and  $\hat{h}_{k,d} = \mathbb{E}_{t\sim \mu_d}[\overline{P}_{k,d}(t)h(t)]$ . By Funk-Hecke formula (Theorem C.1) we get
$$
(\Pi_ {k} y) (x) = \hat {h} _ {k, d} \bar {P} _ {k, d} \left(x ^ {\top} q _ {\star}\right), \quad \forall x \in \mathbb {S} ^ {d - 1}. \tag {H.5}
$$
Similarly, we can write the inner product kernel  $K$  as  $K(x,z) = \kappa (x^{\top}z)$ . Define  $\hat{\kappa}_{k,d} = \mathbb{E}_{t\sim \mu_d}[\overline{P}_{k,d}(t)\kappa (t)]$ . Then we have
$$
(\Pi_ {k} f) (x) = \hat {\kappa} _ {k, d} \sum_ {i = 1} ^ {n} \beta_ {i} \bar {P} _ {k, d} \left(x ^ {\top} x _ {i}\right), \quad \forall x \in \mathbb {S} ^ {d - 1}. \tag {H.6}
$$
Now we apply Lemma H.1 to the function  $(\hat{h}_{k,d})^{-1}(\Pi_ky) = \overline{P}_{k,d}(\langle q_\star ,\cdot \rangle)$ . As a result, with probability at least  $1 / 2$  over the randomness of  $\{x_{i}\}_{i = 1}^{n}$ , for any function  $f$  of the form  $f(\cdot) = \sum_{i = 1}^{n}\beta_{i}K(x_{i},\cdot)$ , we have  $\| (\hat{h}_{k,d})^{-1}\Pi_4(y - f)\| _2^2\geq 3 / 4$ , which implies  $\| y - f\| _2^2\geq \frac{3}{4} (\hat{h}_{k,d})^2$ .
H.2 Proof of Lemma H.1
Proof of Lemma H.1. First we prove that when  $\| g^{\star} \|_2 = 1$ ,  $\| g - g^{\star} \|_2^2 < 3/4$  implies  $\langle g, g^{\star} \rangle \geq \| g \|_2 / 2$ . To this end, by basic algebra we get
$$
3 / 4 > \| g - g ^ {\star} \| _ {2} ^ {2} = \| g \| _ {2} ^ {2} + \| g ^ {\star} \| _ {2} ^ {2} - 2 \langle g, g ^ {\star} \rangle . \tag {H.7}
$$
$$
\langle g, g ^ {\star} \rangle \geq \frac {1}{2} \left(\| g \| _ {2} ^ {2} + \frac {1}{4}\right) \geq \frac {1}{2} \| g \| _ {2}, \tag {H.8}
$$
where the last inequality follows from AM-GM. In the following, we prove that with probability at least  $1/2$ , Eq. (H.8) is impossible when  $n < d^{k}(8k)^{-(3k+1)}(\ln d)^{-(k+2)}$ .
Recall that Lemma C.5 states that for every  $x, z \in \mathbb{S}^{d-1}$  and  $k \geq 0$ ,
$$
\left\langle \bar {P} _ {k, d} (\langle x, \cdot \rangle), \bar {P} _ {k, d} (\langle z, \cdot \rangle) \right\rangle = P _ {k, d} (\langle x, z \rangle). \tag {H.9}
$$
$$
\| g \| _ {2} ^ {2} = \sum_ {i, j = 1} ^ {n} \beta_ {i} \beta_ {j} \left\langle \bar {P} _ {k, d} (\langle x _ {i}, \cdot \rangle), \bar {P} _ {k, d} (\langle x _ {j}, \cdot \rangle) \right\rangle = \sum_ {i, j = 1} ^ {n} \beta_ {i} \beta_ {j} P _ {k, d} (\langle x _ {i}, x _ {j} \rangle). \tag {H.10}
$$
Invoking Lemma H.3, with probability at least  $3/4$  over the randomness of  $\{x_{i}\}_{i \in [n]}$ , we have
$$
\| g \| _ {2} ^ {2} = \sum_ {i, j = 1} ^ {n} \beta_ {i} \beta_ {j} P _ {k, d} (\langle x _ {i}, x _ {j} \rangle) \geq \frac {1}{2} \| \beta \| _ {2} ^ {2}. \tag {H.11}
$$
For the LHS of Eq. (H.8),
$$
\begin{array}{l} \langle g, g ^ {\star} \rangle = \sum_ {i = 1} ^ {n} \beta_ {i} \left\langle \bar {P} _ {k, d} (\langle x _ {i}, \cdot \rangle), \bar {P} _ {k, d} (\langle u, \cdot \rangle) \right\rangle = \sum_ {i = 1} ^ {n} \beta_ {i} P _ {k, d} (\langle x _ {i}, u \rangle) (H.12) \\ \leq \left(\sum_ {i = 1} ^ {n} \beta_ {i} ^ {2}\right) ^ {1 / 2} \left(\sum_ {i = 1} ^ {n} P _ {k, d} (\langle x _ {i}, u \rangle) ^ {2}\right) ^ {1 / 2} = \| \beta \| _ {2} \left(\sum_ {i = 1} ^ {n} P _ {k, d} (\langle x _ {i}, u \rangle) ^ {2}\right) ^ {1 / 2}. (H.13) \\ \end{array}
$$
By Lemma C.7, when  $n < d^{k}(8k)^{-(3k + 1)}(\ln d)^{-(k + 2)}$ , with probability at least  $3 / 4$  we have
$$
\begin{array}{l} \langle g, g ^ {\star} \rangle \leq \| \beta \| _ {2} \left(\sum_ {i = 1} ^ {n} P _ {k, d} (\langle x _ {i}, u \rangle) ^ {2}\right) ^ {1 / 2} \leq \| \beta \| _ {2} \left(n \max  _ {i \in [ n ]} P _ {k, d} (\langle x _ {i}, u \rangle) ^ {2}\right) ^ {1 / 2} (H.14) \\ \leq \| \beta \| _ {2} \sqrt {\frac {n (3 2 k \ln (d n)) ^ {k}}{d ^ {k}}} \leq \| \beta \| _ {2} \sqrt {\frac {1}{\ln d}}. (H.15) \\ \end{array}
$$
Combining Eq. (H.11) and Eq. (H.15), there exists a universal constant  $d_0 > 0$  such that when  $d > d_0$ , with probability at least  $1/2$  over the randomness of  $\{x_1, \dots, x_n\}$ ,
$$
\langle g, g ^ {\star} \rangle <   \| \beta \| _ {2} / 4 \leq \| g \| _ {2} / 2. \tag {H.16}
$$
Recall that  $\| g - g^{\star} \|_2^2 < 3/4$  implies  $\langle g, g^{\star} \rangle \geq \| g \|_2 / 2$ . Consequently, with probability at least  $1/2$ , when  $n < d^k (8k)^{-(3k+1)} (\ln d)^{-(k+2)}$  and  $d > d_0$  we have  $\| g - g^{\star} \|_2^2 \geq 3/4$ .
Lemma H.3. Let  $x_{1}, \dots, x_{n}$  be i.i.d. random variables drawn uniformly from  $\mathbb{S}^{d-1}$ , and  $P_{k,d}$  the degree- $k$  Legendre polynomial. For any fixed  $k \geq 1$ , define the matrix  $M_{k} \in \mathbb{R}^{n \times n}$  where  $[M_k]_{i,j} = P_{k,d}(\langle x_i, x_j \rangle)$ . There exists universal constants  $d_0$  such that, when  $n < d^k (8k)^{-(3k+1)} (\ln d)^{-(k+2)}$  and  $d > d_0$ , with probability at least  $3/4$ ,
$$
\lambda_ {\min } \left(M _ {k}\right) \geq 1 / 2. \tag {H.17}
$$
Proof. In the following, we fix  $k \geq 1$ . First, we prove that
$$
\mathbb {E} \left[ \lambda_ {\min } \left(M _ {k}\right) \right] \geq 7 / 8. \tag {H.18}
$$
To this end, we invoke Theorem I.4 by constructing a matrix  $A$  such that  $\frac{1}{N_{k,d}} A^\top A = M_k$ .
By Lemma C.6, there exists a feature mapping  $\phi : \mathbb{S}^{d - 1} \to \mathbb{R}^{N_{k,d}}$  such that for every  $u, v \in \mathbb{S}^{d - 1}$
$$
\langle \phi (u), \phi (v) \rangle = N _ {k, d} P _ {k, d} \left(u ^ {\top} v\right), \tag {H.19}
$$
$$
\| \phi (u) \| _ {2} ^ {2} = N _ {k, d}, \tag {H.20}
$$
$$
\underset {u \sim \mathbb {S} ^ {d - 1}} {\mathbb {E}} [ \phi (u) \phi (u) ^ {\top} ] = I. \tag {H.21}
$$
We set  $A_{i} = \phi (x_{i}),\forall 1\leq i\leq n$  . Consequently,
$$
\frac {1}{N _ {k , d}} A ^ {\top} A = M _ {k}, \tag {H.22}
$$
$$
\left\| A _ {i} \right\| _ {2} ^ {2} = N _ {k, d}, \tag {H.23}
$$
$$
\mathbb {E} \left[ A _ {i} A _ {i} ^ {\top} \right] = I. \tag {H.24}
$$
In the following, we upper bound the incoherence parameter  $p$  defined in Theorem I.4:
$$
p = \frac {1}{N _ {k , d}} \mathbb {E} \left[ \max  _ {i \leq n} \sum_ {j \in [ n ], j \neq i} \left\langle A _ {i}, A _ {j} \right\rangle^ {2} \right]. \tag {H.25}
$$
By Eq. (H.19) we get
$$
\begin{array}{l} \frac {1}{N _ {k , d}} \mathbb {E} \left[ \max  _ {i \leq n} \sum_ {j \in [ n ], j \neq i} \langle A _ {i}, A _ {j} \rangle^ {2} \right] = N _ {k, d} \mathbb {E} \left[ \max  _ {i \leq n} \sum_ {j \in [ n ], j \neq i} P _ {k, d} (\langle x _ {i}, x _ {j} \rangle) ^ {2} \right] (H.26) \\ \leq n N _ {k, d} \mathbb {E} \left[ \max  _ {i \leq n} \max  _ {j \neq i} P _ {k, d} \left(\langle x _ {i}, x _ {j} \rangle\right) ^ {2} \right]. (H.27) \\ \end{array}
$$
Invoking Lemma C.7 we get
$$
N _ {k, d} \mathbb {E} \left[ \max  _ {i \leq n} \max  _ {j \neq i} P _ {k, d} \left(\langle x _ {i}, x _ {j} \rangle\right) ^ {2} \right] \lesssim (3 2 k \ln (d n)) ^ {k}. \tag {H.28}
$$
It follows that  $p \lesssim n(32k\ln (dn))^k$ .
Now that we have established the conditions, by Theorem I.4 we have
$$
\mathbb {E} [ \| M - I \| ] \lesssim \sqrt {\frac {n (\ln n) (3 2 k \ln (d n)) ^ {k}}{N _ {k , d}}}. \tag {H.29}
$$
Note that  $N_{k,d} \geq \binom{k+d-2}{k} \geq (d/k)^{k}$  when  $k \geq 2$ , and clearly  $N_{1,d} \geq d$ . Hence, there exists a universal constant  $d_{0}$  such that, when  $n < d^{k}(8k)^{-(3k+1)}(\ln d)^{-(k+2)}$ .
$$
\frac {n (\ln n) (3 2 k \ln (d n)) ^ {k}}{N _ {k , d}} \leq \frac {n (8 k) ^ {2 k} \ln (d n) ^ {k + 1}}{d ^ {k}} \leq \frac {n (8 k) ^ {2 k} \ln (d ^ {k + 1}) ^ {k + 1}}{d ^ {k}} \leq \frac {1}{\ln d}. \tag {H.30}
$$
Hence, there exists a universal constant  $d_0$  such that when  $d \geq d_0$ ,