SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {\Sigma} = \boldsymbol {I} _ {d} + \theta \boldsymbol {e} _ {1} \boldsymbol {e} _ {1} ^ {\top} \tag {A.2}
$$
and training label  $y_{i} = f_{*}(\pmb{x}_{i})$  only depends on the first entry of the data  $\pmb{x}_{i}$  for  $i \in [n]$ .
A.1 Polynomial Approximation of Kernel Matrix
Denote the concatenation of  $\pmb{x}$  and  $\pmb{X}$  as  $\tilde{\pmb{X}}^{\top} := [\pmb{x},\pmb{X}^{\top}] \in \mathbb{R}^{d \times (n + 1)}$ , and the enlarged nonlinear kernel matrix by  $\tilde{\pmb{K}} = g(\frac{1}{d}\tilde{\pmb{X}}\tilde{\pmb{X}}^{\top}) \in \mathbb{R}^{(n + 1) \times (n + 1)}$ . First we present a preliminary approximation of  $\tilde{\pmb{K}}$  by a proper polynomial kernel.
Lemma 6. Suppose that  $\beta \leq 1 - \frac{1}{p}$  for some  $p \in \mathbb{N}$ . Under the assumptions of Theorem 3, as  $n, d \to \infty$  proportionally, we have
$$
\left\| \tilde {\boldsymbol {H}} _ {p} + \sigma_ {p} \boldsymbol {I} _ {n + 1} - \tilde {\boldsymbol {K}} \right\| = o _ {d, \mathbb {P}} (1),
$$
$$
\tilde {\boldsymbol {H}} _ {p} := P _ {p} \left(\tilde {\boldsymbol {X}} \tilde {\boldsymbol {X}} ^ {\top} / d\right),
$$
and  $P_{p}(x) = \sum_{k = 1}^{2p}g^{(k)}(0)x^{k},\sigma_{p}\coloneqq g(1) - P_{p}(1).$
Proof. For simplicity, we denote  $\pmb{x} = \pmb{x}_0$  and  $\pmb{x}_i^\top = [\pmb{x}_i^1, \bar{\pmb{x}}_i^\top]$ , where  $\pmb{x}_i^1$  is the first entry of  $\pmb{x}_i$ , and  $\bar{\pmb{x}}_i$  the remaining entries for  $0 \leq i \leq n$ . Notice that  $\frac{1}{d}\pmb{x}_i^\top\pmb{x}_j = \frac{1}{d}\pmb{z}_i\pmb{\Sigma}\pmb{z}_j$ , for any  $0 \leq i,j \leq n$ . By Bernstein inequality [Ver18, Corollary 2.8.3], we have
$$
\begin{array}{l} \mathbb {P} \left(\left| \frac {1}{d} \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {x} _ {j} \right| > t\right) \leq \mathbb {P} \left(\left| \frac {1}{d} \bar {\boldsymbol {x}} _ {i} ^ {\top} \bar {\boldsymbol {x}} _ {j} \right| > t / 2\right) + \mathbb {P} \left(\left| \frac {1}{d} \boldsymbol {x} _ {i} ^ {1} \boldsymbol {x} _ {j} ^ {1} \right| > t / 2\right) \\ \leq \exp \left(- c d \min  \{t ^ {2}, t \}\right) + \exp \left(- c t d ^ {1 - \beta}\right), \\ \end{array}
$$
for any  $i \neq j$  and  $t > 0$ . Therefore,  $\left|\frac{1}{d}\pmb{x}_i^\top \pmb{x}_j\right| = o_{d,\mathbb{P}}(d^{\beta -1}\log d)$  for  $\beta \in [1 / 2,1)$  and  $\left|\frac{1}{d}\pmb{x}_i^\top \pmb{x}_j\right| = o_{d,\mathbb{P}}(d^{-1 / 2}\log d)$  for  $\beta \in [0,1 / 2]$ . Then we have
$$
\begin{array}{l} \left\| \operatorname {o f f d i g} \left(\tilde {\boldsymbol {H}} _ {p} - \tilde {\boldsymbol {K}}\right) \right\| ^ {2} \leq \left\| \operatorname {o f f d i g} \left(\tilde {\boldsymbol {H}} _ {p} - \tilde {\boldsymbol {K}}\right) \right\| _ {F} ^ {2} \\ \leq \sum_ {i \neq j} \sum_ {k = 2 p + 1} ^ {\infty} \left| \frac {1}{d} \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {x} _ {j} \right| ^ {k} = \tilde {o} _ {d, \mathbb {P}} \left(d ^ {- \frac {1}{p}}\right). \\ \end{array}
$$
Here,  $\text{offdiag}(\cdot)$  denotes setting the diagonal entries of the matrix to be zero. In addition, following the same proof as Theorem 1 of [EK10], we can obtain a similar approximation for the diagonal part  $\text{dig}(\tilde{\pmb{H}}_p - \tilde{\pmb{K}})$ , the details of which we omit.
A.2 Hermite Expansion of Polynomial Kernel
Equipped with the above polynomial expansion of the kernel matrix, we now introduce a further approximation using orthogonal polynomials. Recall that
$$
\mathcal {R} \left(\hat {f} _ {\mathrm {k e r}}\right) = \left\| f _ {*} \right\| _ {L ^ {2}} ^ {2} - 2 \boldsymbol {y} ^ {\top} (\boldsymbol {K} + \lambda \boldsymbol {I}) ^ {- 1} \boldsymbol {E} + \boldsymbol {y} ^ {\top} (\boldsymbol {K} + \lambda \boldsymbol {I}) ^ {- 1} \boldsymbol {M} (\boldsymbol {K} + \lambda \boldsymbol {I}) ^ {- 1} \boldsymbol {y},
$$
$$
\boldsymbol {E} := \mathbb {E} _ {\boldsymbol {x}} \left[ f _ {*} (\boldsymbol {x}) k (\boldsymbol {x}, \boldsymbol {X}) \right] ^ {\top} \in \mathbb {R} ^ {n}
$$
$$
\boldsymbol {M} := \mathbb {E} _ {\boldsymbol {x}} [ k (\boldsymbol {X}, \boldsymbol {x}) k (\boldsymbol {x}, \boldsymbol {X}) ] \in \mathbb {R} ^ {n \times n}.
$$
From Lemma 6 we know that for some  $p\in \mathbb{N}$
$$
\boldsymbol {K} = \boldsymbol {H} _ {p} + \boldsymbol {\Delta} _ {K},
$$
$$
\boldsymbol {k} (\boldsymbol {x}, \boldsymbol {X}) = \boldsymbol {H} _ {p} (\boldsymbol {x}, \boldsymbol {X}) + \sigma_ {p} \boldsymbol {I} _ {n + 1} + \boldsymbol {\Delta} _ {K ^ {\prime}},
$$
for  $\| \pmb{\Delta}_K\| = o_{d,\mathbb{P}}(1)$  and  $\| \pmb{\Delta}_K^{\prime}\| = o_{d,\mathbb{P}}(1)$ , where  $\sigma_p$  is defined in Lemma 6,  $\pmb{H}_{p}:= P_{p}(\frac{1}{d}\pmb{X}\pmb{X}^{\top})$ , and  $\pmb{H}_{p}(\pmb{x},\pmb{X}):=P_{p}(\frac{1}{d}\pmb{X}\pmb{x})^{\top}$  is a row vector. Then following from Lemmas A.3, A.8 and A.13 in [BMR21], we can conclude that  $\mathcal{R}(\hat{f}_{\mathrm{ker}}) = \mathcal{R}_p + o_{d,\mathbb{P}}(1)$ , where
$$
\mathcal {R} _ {p} := \left\| f _ {*} \right\| _ {L ^ {2}} ^ {2} - 2 \boldsymbol {y} ^ {\top} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {E} _ {p} + \boldsymbol {y} ^ {\top} \left(\boldsymbol {K} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {M} _ {p} \left(\boldsymbol {K} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {y}, \tag {A.3}
$$
$$
\boldsymbol {E} _ {p} := \mathbb {E} _ {\boldsymbol {x}} \left[ f _ {*} (\boldsymbol {x}) \boldsymbol {H} _ {p} (\boldsymbol {x}, \boldsymbol {X}) \right] ^ {\top} \in \mathbb {R} ^ {n}
$$
$$
\boldsymbol {M} _ {p} := \mathbb {E} _ {\boldsymbol {x}} [ \boldsymbol {H} _ {p} (\boldsymbol {X}, \boldsymbol {x}) \boldsymbol {H} _ {p} (\boldsymbol {x}, \boldsymbol {X}) ] \in \mathbb {R} ^ {n \times n}.
$$
Now we analyze the asymptotic behavior of  $\mathcal{R}_p$  by utilizing the Hermitian polynomials to reorganize kernel  $H_{p}$ . Since  $\Sigma$  now is a diagonal matrix in (A.2), from [Rah17], we know the multivariate Hermite polynomials with respect to  $\mathcal{N}(0,\frac{1}{d}\Sigma)$  in  $\mathbb{R}^d$  are strongly orthogonal, each of which is the product of univariate Hermite polynomials on each coordinate. More precisely, for any  $k\in \mathbb{N}$  and multi-index  $\boldsymbol {I}\coloneqq (i_1,i_2,\ldots ,i_d)\in [k]^d$  with  $i_1 + i_2 + \dots +i_d = k$ , we define the  $k$ -th multivariate Hermite polynomial with respect to  $\boldsymbol{I}$  by  $h_{k,I}(\boldsymbol {x})\coloneqq \prod_{j = 1}^{d}h_{i_j}(x_j)$  where  $\boldsymbol {x} = (x_{1},\ldots ,x_{d})$  and  $h_{i_j}(\cdot)$  is the  $i_j$ -the univariate Hermite polynomials (note that the variance for the first coordinate and the remaining coordinates are different). Hence, any  $L$ -degree polynomial of  $\boldsymbol {x}\in \mathbb{R}^d$  has the form
$$
f (\boldsymbol {x}) = \sum_ {k = 0} ^ {L} \sum_ {\substack {| \boldsymbol {I} | = k \\ \boldsymbol {I} = \left\{i _ {1}, i _ {2}, \dots , i _ {d} \right\}}} c _ {k, \boldsymbol {I}} h _ {k, \boldsymbol {I}} (\boldsymbol {x}), \tag{A.4}
$$
for some coefficients  $c_{k,I} \in \mathbb{R}$ . Similar expansions using Gegenbauer polynomials for uniform unit sphere distribution have been used in [GMMM21, MMM21, GMMM20]. The decomposition we consider is specifically related to the Hermite expansions of  $\sigma_*$ :
$$
f _ {*} (\boldsymbol {x}) = \sum_ {i = k} ^ {p} \alpha_ {j} h _ {j} \left(\boldsymbol {\mu} ^ {\top} \boldsymbol {z}\right) = \sum_ {i = k} ^ {p} \alpha_ {j} h _ {j} \left(z ^ {1}\right), \tag {A.5}
$$
where  $h_j(\cdot)$  is the  $j$ -th Hermite polynomial with  $z \sim \mathcal{N}(0, I)$ . Notice that  $\mathbb{E}[h_{k,l}(\boldsymbol{x})h_{l,J}(\boldsymbol{x})] = \delta_{k,l}\delta_{I,J}$  for  $\boldsymbol{x}$  sampled from the spiked model (2.1).
By the kernel trick (e.g., see Section 3.7 of [Ver18]), a  $k$ -th monomial kernel satisfies
$$
\left\langle \boldsymbol {x} _ {i}, \boldsymbol {x} _ {j} \right\rangle^ {k} = \left\langle \boldsymbol {x} _ {i} ^ {\otimes k}, \boldsymbol {x} _ {j} ^ {\otimes k} \right\rangle ,
$$
where each entry of  $\pmb{x}_i^{\otimes k} \in \mathbb{R}^{d^k}$  is a  $k$ -th degree monomial of  $\pmb{x}_i$ . Hence we can rewrite the above inner product based on Hermite expansions of each entry of  $\pmb{x}_i^{\otimes k}$  and  $\pmb{x}_j^{\otimes k}$ . Therefore, for kernel  $H_{p}(\pmb{x}_{i},\pmb{x}_{j})$ , there exists a Hermite polynomial feature map  $\Phi :\mathbb{R}^d\to \mathbb{R}^L$  where
$$
L := \sum_ {k = 0} ^ {2 p} \binom {d + k - 1} {k} \qquad B (d, k) := \binom {d + k - 1} {k}
$$
$$
\boldsymbol {H} _ {p} \left(\boldsymbol {x} _ {i} ^ {\top} \boldsymbol {x} _ {j} / d\right) = \boldsymbol {\Phi} \left(\boldsymbol {x} _ {i} / \sqrt {d}\right) \boldsymbol {D} ^ {2} \boldsymbol {\Phi} \left(\boldsymbol {x} _ {i} / \sqrt {d}\right) ^ {\top},
$$
where each row vector  $\Phi (\cdot)$  is the concatenation of orthogonal polynomials  $h_{k,\mathbf{I}}(\cdot)$  for  $0\leq k\leq L$  and  $\pmb{D}$  is the diagonal matrix where the diagonal entries are the Hermite coefficients corresponding to  $\pmb{H}_p$  which will be specified in the sequel. We define
$$
\boldsymbol {\Phi} := \left( \begin{array}{c} \boldsymbol {\Phi} \left(\boldsymbol {x} _ {1} / \sqrt {d}\right) \\ \boldsymbol {\Phi} \left(\boldsymbol {x} _ {2} / \sqrt {d}\right) \\ \dots \\ \boldsymbol {\Phi} \left(\boldsymbol {x} _ {n} / \sqrt {d}\right) \end{array} \right) \in \mathbb {R} ^ {n \times L}, \tag {A.6}
$$
which gives  $H_{p} = \Phi \Phi^{\top}$ . Furthermore, since the variance of the first entry of  $\boldsymbol{x}_i$  is different from the other entries and the label  $y_i$  only depends on the first entry, we will separate the Hermite polynomials depending on  $x_i^1$  and the terms depending on the remaining entries  $\bar{x}_i \in \mathbb{R}^{d-1}$ . That is, we further decompose each  $\Phi(x_i / \sqrt{d}) = [\Phi_0(x_i / \sqrt{d}), \Phi(\bar{x}_i / \sqrt{d})]$ , where  $\Phi_0(x_i / \sqrt{d}) \in \mathbb{R}^{L_0}$  and
$$
L _ {0} := \sum_ {k = 0} ^ {2 p} \binom {d + k - 2} {k - 1} \qquad B _ {0} (d, k) := \binom {d + k - 2} {k - 1},
$$
where  $B_0(d, k)$  is the total number of multivariate Hermite polynomials of  $x_i$  with degree  $k$ , whose components with respect to  $x_i^1$  is not vanishing. Based on this decomposition, we also write
$$
\bar {\Phi} := \left( \begin{array}{c} \Phi (\bar {x} _ {1} / \sqrt {d}) \\ \Phi (\bar {x} _ {2} / \sqrt {d}) \\ \dots \\ \Phi (\bar {x} _ {n} / \sqrt {d}) \end{array} \right). \tag {A.7}
$$
Lemma 7. Let  $d_{\mathrm{eff}} \coloneqq d^{1 - \beta}$ . Under the assumptions of Theorem 3, we know that
where  $D^2$  is a diagonal matrix with entries
$$
\zeta (d, k, m) = \frac {g ^ {(k)} (0)}{d ^ {k - m} d _ {\text {e f f}} ^ {m}} + o _ {d} \left(d ^ {- k + m - 1} d _ {\text {e f f}} ^ {- m}\right), \tag {A.8}
$$
for  $m \in [k]$  and  $k \in [2p]$ . Furthermore, based on the definition of  $\Phi_0$  as above, we have
$$
\boldsymbol {H} _ {p} = \Phi_ {0} \boldsymbol {D} _ {0} ^ {2} \Phi_ {0} ^ {\top} + \bar {\boldsymbol {H}} _ {p},
$$
where each entry of  $\bar{H}_p(\bar{\mathbf{x}}_i,\bar{\mathbf{x}}_j)\coloneqq P_p(\frac{1}{d}\bar{\mathbf{x}}_i^\top \bar{\mathbf{x}}_j)$ , and  $D_0^2$  is a diagonal matrix defined analogously as  $D^2$  but with  $m > 0$  in (A.8). Note that  $\bar{H}_p$  is an inner-product polynomial kernel matrix of  $\bar{\mathbf{X}}\coloneqq [\bar{\mathbf{x}}_1,\dots ,\bar{\mathbf{x}}_n]\in \mathbb{R}^{(d - 1)\times n}$
Proof. This lemma follows from properties of multivariate Hermite polynomials [Chi11]. Recall
$$
\operatorname {V a r} \left(\boldsymbol {x} _ {i} ^ {1}\right) = 1 + \theta = \Theta (d ^ {\beta}).
$$
By re-normalizing the variables, we obtain
$$
P _ {p} \left(\boldsymbol {x} _ {i} ^ {\top} \boldsymbol {x} _ {j} / d\right) = \sum_ {k = 0} ^ {2 p} \sum_ {m = 0} ^ {k} \sum_ {| \boldsymbol {I} | = k, i _ {1} = m} \zeta (d, k, m) h _ {k, \boldsymbol {I}} (\boldsymbol {x} _ {i}), h _ {k, \boldsymbol {I}} (\boldsymbol {x} _ {j}),
$$
where  $\pmb{I} = (i_1, \dots, i_d)$ . As for  $\Phi_0$ , the diagonals of  $\pmb{D}_0$  correspond to  $m \neq 0$  above.
A.3 Analysis of Prediction Risk
The next lemma shows that  $\bar{H}_p$  is approximated by a linear kernel which does not aid the learning of  $f_{*}$ ; this entails that we may focus on the term involving  $\Phi_0$ . Notice that  $\bar{H}_p$  is a rotationally invariant kernel on isotropic Gaussian data  $\bar{\mathbf{X}}$  whose limiting global law has been analyzed in [CS13].
Lemma 8. Under the assumptions of Theorem 3, as  $n,d\to \infty$  proportionally, we have
$$
\left\| \bar {\boldsymbol {H}} _ {p} + \sigma_ {p} \boldsymbol {I} _ {n} - \boldsymbol {K} _ {\text {l i n}} \right\| = o _ {d, \mathbb {P}} (1),
$$
$$
\boldsymbol {K} _ {\text {l i n}} := \frac {c _ {1}}{d} \bar {\boldsymbol {X}} ^ {\top} \bar {\boldsymbol {X}} + c _ {2} \boldsymbol {I} _ {n} + \frac {c _ {3}}{d} \mathbf {1 1} ^ {\top}
$$
and  $\| \pmb{K}_{\mathrm{lin}}\| = \mathcal{O}_{d,\mathbb{P}}(1)$ . Here,  $c_{1}, c_{2}$  and  $c_{3}$  are some positive constants that only depend on  $g^{(k)}(0)$  for  $k = 0,1$  and  $2$ .
The above lemma is a consequence of Theorem 1 of [EK10], and hence we omit the detailed proof; for the formulae of constants  $c_{1}, c_{2}$  and  $c_{3}$ , we refer to Equation (57) of [BMR21]. We remark that in the following concentration analysis, the exact limiting spectral distribution of  $\pmb{K}_{\mathrm{lin}}$  does not appear, since we do not consider the exact setting of  $\beta = 1 - \frac{1}{\ell}$  for  $\ell \in \mathbb{N}$ .
Lemma 9. Under the assumptions of  $f_*$  in Theorem 3, we have that
$$
\boldsymbol {y} = \Phi_ {0} \boldsymbol {P} \alpha ,
$$
for some vector  $\alpha = [0,\dots ,0,\alpha_{k},\dots ,\alpha_{p},0,\dots ,0]^{\top}\in \mathbb{R}^{2p}$ , where  $\pmb {P}\in \mathbb{R}^{L_0\times 2p}$  is a projector onto the Hermite components with  $\pmb {I} = (i_{1},0,\dots ,0)$ .
Here  $\alpha$  represents the Hermite coefficients of the function  $f_{*}$ . Lemma 9 directly follows from (A.5) since  $\pmb{y}$  only depends on  $\pmb{x}_i^1$  for  $i\in [d]$ . Notice that here  $\pmb{P}$  selects  $h_k(\pmb{x}_i^1)$  for  $i\in [d]$  and  $k\in [2p]$  from all multivariate Hermite polynomials.
Next, we prove the following concentration statements, parallel to existing results in [GMMM21, MMM21] for spherical data. For any  $\ell \in [2p]$ , we decompose  $\Phi_0 = [\Phi_{0,\leq \ell},\Phi_{0, > \ell}]$  where  $\Phi_{0,\leq \ell}$  are composed of all Hermite polynomials  $h_{k,I}$  with  $k\leq \ell$ , and  $\Phi_{0, > \ell}$  collects all Hermite polynomials  $h_{k,I}$  with  $k > \ell$ . Similarly, we define  $D_{0,\leq \ell}$  and  $D_{0, > \ell}$  corresponding to  $\Phi_{0,\leq \ell}\in \mathbb{R}^{n\times m}$  and  $\Phi_{0, > \ell}\in \mathbb{R}^{n\times (L_0 - m)}$ , respectively. Here we denote by  $m$  the number of all Hermite polynomials  $h_{k,I}$  with  $k\leq \ell$  where  $m\gg n$ .
Lemma 10. Under the assumptions of Theorem 3, we have
$$
\Phi_ {0, \leq \ell} ^ {\top} \Phi_ {0, \leq \ell} = I _ {m} + o _ {d, \mathbb {P}} (1).
$$
This lemma shows the concentration of the Gram matrix since  $\mathbb{E}[\Phi_{0, < \ell}^{\top}\Phi_{0, \leq \ell}] = I_m$ . The proof of this lemma can be derived from the concentration result [Zhi21]; see also Lemma 11 in [GMMM21]. The following two lemmas are the crucial ingredients of the proof of Theorem 3, which have been established in Lemmas 13 and 14 in [GMMM21] for the spherical setting.
Lemma 11. Under the assumptions of Theorem 3 with  $1 - \frac{1}{\ell} < \beta < 1 - \frac{1}{\ell + 1}$  for some  $\ell \in \mathbb{N}$ , as  $n, d \to \infty$  proportionally, we have
$$
\left\| n \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I} _ {n}\right) ^ {- 1} \boldsymbol {M} _ {p} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I} _ {n}\right) ^ {- 1} - \frac {1}{n} \boldsymbol {\Phi} _ {0, \leq \ell} \boldsymbol {\Phi} _ {0, \leq \ell} ^ {\top} \right\| = o _ {d, \mathbb {P}} (1).
$$
Proof. Recall the definition of  $\Phi$  in (A.6). From Lemma 7, we can obtain that  $M_{p} = \Phi D^{4}\Phi^{\top}$  due to the orthogonal polynomial decomposition in (A.4). Hence,
$$
M = \Phi_ {0, \leq \ell} D _ {0, \leq \ell} ^ {4} \Phi_ {0, \leq \ell} ^ {\top} + \Phi_ {> \ell} D _ {> \ell} ^ {4} \Phi_ {> \ell} ^ {\top},
$$
where  $\Phi_{> \ell}$  is defined similarly as  $\Phi_{0, > \ell}$  which is formed by all orthogonal components of  $\Phi_{0, \leq \ell}$ . In general,  $\Phi_{> \ell}$  is composed by  $\Phi_{0, > \ell}$  and  $\bar{\Phi}$  defined in (A.7). Therefore,
$$
\begin{array}{l} n \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I} _ {n}\right) ^ {- 1} \boldsymbol {M} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I} _ {n}\right) ^ {- 1} \\ = n \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I} _ {n}\right) ^ {- 1} \boldsymbol {\Phi} _ {0, \leq \ell} \boldsymbol {D} _ {0, \leq \ell} ^ {4} \boldsymbol {\Phi} _ {0, \leq \ell} ^ {\top} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I} _ {n}\right) ^ {- 1} \\ + n (\boldsymbol {H} _ {p} + \lambda \boldsymbol {I} _ {n}) ^ {- 1} \Phi_ {> \ell} \boldsymbol {D} _ {> \ell} ^ {4} \Phi_ {> \ell} ^ {\top} (\boldsymbol {H} _ {p} + \lambda \boldsymbol {I} _ {n}) ^ {- 1}. \\ \end{array}
$$
Recall that  $\pmb{H}_p = \pmb{\Phi} \pmb{D}^2 \pmb{\Phi}^\top$ . By Lemma 7 we know that with high probability
$$
\left\| n (\pmb {H} _ {p} + \lambda \pmb {I} _ {n}) ^ {- 1} \pmb {\Phi} _ {> \ell} \pmb {D} _ {> \ell} ^ {4} \pmb {\Phi} _ {> \ell} ^ {\top} (\pmb {H} _ {p} + \lambda \pmb {I}) ^ {- 1} \right\| \leq C n d _ {\mathrm {e f f}} ^ {- \ell - 1} = o _ {d} (1).
$$
For the second part, we let  $\pmb{A} \coloneqq \pmb{\Phi}_{>\ell}\pmb{D}_{>\ell}^{4}\pmb{\Phi}_{>\ell}^{\top} + \lambda\pmb{I}_{n}$  and apply the Sherman-Morrison-Woodbury formula to obtain
$$
\begin{array}{l} n \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I} _ {n}\right) ^ {- 1} \boldsymbol {\Phi} _ {0, \leq \ell} \boldsymbol {D} _ {0, \leq \ell} ^ {4} \boldsymbol {\Phi} _ {0, \leq \ell} ^ {\top} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I} _ {n}\right) ^ {- 1} \\ = n \pmb {A} ^ {- 1} \pmb {\Phi} _ {0, \leq \ell} \Big (\pmb {D} _ {> \ell} ^ {- 2} + \pmb {\Phi} _ {0, \leq \ell} ^ {\top} \pmb {A} ^ {- 1} \pmb {\Phi} _ {0, \leq \ell} \Big) ^ {- 2} \pmb {\Phi} _ {0, \leq \ell} ^ {\top} \pmb {A} ^ {- 1} \\ = \boldsymbol {A} ^ {- 1} \Phi_ {0, \leq \ell} \boldsymbol {B} ^ {2} \Phi_ {0, \leq \ell} ^ {\top} \boldsymbol {A} ^ {- 1}, \\ \end{array}
$$
$$
\boldsymbol {B} := \left((n \boldsymbol {D} _ {> \ell}) ^ {- 2} + \frac {1}{n} \boldsymbol {\Phi} _ {0, \leq \ell} ^ {\top} \boldsymbol {A} ^ {- 1} \boldsymbol {\Phi} _ {0, \leq \ell}\right) ^ {- 1}.
$$
Following the proof of Lemma 13 in [GMMM21], we can show that  $\left\| \mathbf{A}^{-1} - \mathbf{I}_n\right\| = o_{d,\mathbb{P}}(1)$  and  $\| \pmb {B} - \pmb {I}_n\| = o_{d,\mathbb{P}}(1)$  due to assumption that  $1 - \frac{1}{\ell} <  \ell$  and Lemma 10.
Lemma 12. Recall the definition of  $\Phi_{0,\leq \ell}$  above. Under the assumptions of Theorem 3 with  $1 - \frac{1}{\ell} < \beta < 1 - \frac{1}{\ell + 1}$  for some  $\ell \in \mathbb{N}$ , when  $n,d\to \infty$  proportionally, we have
$$
\left\| \boldsymbol {\Phi} _ {0, \leq \ell} ^ {\top} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I} _ {n}\right) ^ {- 1} \boldsymbol {\Phi} _ {0, \leq \ell} \boldsymbol {D} _ {0, \leq \ell} - \boldsymbol {I} _ {m} \right\| = o _ {d, \mathbb {P}} (1).
$$
The proof of Lemma 12 is analogous to Lemma 11, the details of which we omit.
From (A.3), let us denote by
$$
T _ {1} := \boldsymbol {y} ^ {\top} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {E} _ {p}
$$
$$
T _ {2} := \boldsymbol {y} ^ {\top} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {M} _ {p} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {y}.
$$
Now we aim to control  $T_{1}$  and  $T_{2}$  based on Lemmas 11 and 12. We define the projection of  $\pmb{y}$  onto the Hermite components with degree less than or equals to  $\ell$  and components with degree larger than  $\ell$  as:  $\pmb{y} = \pmb{y}_{1} + \pmb{y}_{2}$ , where  $\pmb{y}_{1} = \Phi_{0, \leq \ell} \pmb{P}_{\leq \ell} \pmb{\alpha}$  and  $\pmb{y}_{2} = \Phi_{0, > \ell} \pmb{P}_{> \ell} \pmb{\alpha}$ . Here  $P_{\leq \ell}$  projects onto the Hermite components with  $I = (i_{1}, 0, \dots, 0)$  and degree no greater  $\ell$ , and  $P_{> \ell} = P - P_{\leq \ell}$ . Similarly, we decompose  $E_{p}$  as  $E_{p} = E_{p,1} + E_{p,2}$ , where we denote
$$
\boldsymbol {E} _ {p, 1} := \Phi_ {0, \leq \ell} \boldsymbol {D} _ {0, \leq \ell} \boldsymbol {P} _ {\leq \ell} \boldsymbol {\alpha},
$$
$$
\boldsymbol {E} _ {p, 2} := \Phi_ {0, > \ell} \boldsymbol {D} _ {0, > \ell} \boldsymbol {P} _ {> \ell} \boldsymbol {\alpha}.
$$
Lemma 13. Under the assumptions of Theorem 3, we have
$$
T _ {1} = \left\| P _ {\leq \ell} f _ {*} \right\| _ {L ^ {2}} ^ {2} + o _ {d, \mathbb {P}} (1), \quad T _ {2} = \left\| P _ {\leq \ell} f _ {*} \right\| _ {L ^ {2}} ^ {2} + o _ {d, \mathbb {P}} (1).
$$
Proof. For simplicity, we first control  $T_{2}$  and then apply some of these results to bound  $T_{1}$ . Let  $T_{2} = T_{21} + 2T_{22} + T_{23}$ , where
$$
T _ {2 1} := \boldsymbol {y} _ {1} ^ {\top} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {M} _ {p} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {y} _ {1},
$$
$$
T _ {2 2} := \boldsymbol {y} _ {1} ^ {\top} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {M} _ {p} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {y} _ {2},
$$
$$
T _ {2 3} := \boldsymbol {y} _ {2} ^ {\top} (\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}) ^ {- 1} \boldsymbol {M} _ {p} (\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}) ^ {- 1} \boldsymbol {y} _ {2}.
$$
Based on Lemmas 9, 11 and (A.8), we know that
$$
\begin{array}{l} T _ {2 1} = \frac {1}{n ^ {2}} \boldsymbol {\alpha} ^ {\top} \boldsymbol {P} _ {\leq \ell} ^ {\top} \boldsymbol {\Phi} _ {0, \leq \ell} ^ {\top} \boldsymbol {\Phi} _ {0, \leq \ell} \boldsymbol {\Phi} _ {0, \leq \ell} ^ {\top} \boldsymbol {\Phi} _ {0, \leq \ell} \boldsymbol {P} _ {\leq \ell} \boldsymbol {\alpha} \\ = \| P _ {\leq \ell} f _ {*} \| _ {L ^ {2}} ^ {2} + o _ {d, \mathbb {P}} (1). \\ \end{array}
$$
For  $T_{23}$ , following from (23) in [GMMM21], when  $d_{\mathrm{eff}}^{\ell + 1} < 1$ , by Markov's inequality, we have
$$
\begin{array}{l} T _ {2 3} = \frac {1}{n ^ {2}} \pmb {\alpha} ^ {\top} \pmb {P} _ {> \ell} ^ {\top} \pmb {\Phi} _ {0, > \ell} ^ {\top} \pmb {\Phi} _ {0, \leq \ell} \pmb {\Phi} _ {0, \leq \ell} ^ {\top} \pmb {\Phi} _ {0, > \ell} \pmb {P} _ {> \ell} \pmb {\alpha} + o _ {d, \mathbb {P}} (1), \\ = \frac {d _ {\text {e f f}} ^ {\ell + 1}}{n} \| P _ {> \ell} f _ {*} \| _ {L ^ {2}} ^ {2} + o _ {d, \mathbb {P}} (1) = o _ {d, \mathbb {P}} (1), \tag {A.9} \\ \end{array}
$$
since polynomials in  $\Phi_{0,\leq \ell}$  are orthogonal to polynomials in  $\Phi_{0, > \ell}$ . Notice that
$$
\left| T _ {2 2} \right| \leq 2 T _ {2 1} ^ {1 / 2} T _ {2 3} ^ {1 / 2} = o _ {d, \mathbb {P}} (1).
$$
This finishes the proof for  $T_{2}$ .
Now let us consider  $T_{1} = T_{11} + T_{12} + T_{13}$ , where
$$
T _ {1 1} := \boldsymbol {y} _ {1} ^ {\top} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {E} _ {1, p},
$$
$$
T _ {1 2} := \boldsymbol {y} _ {2} ^ {\top} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {E} _ {1, p},
$$
$$
T _ {1 3} := \boldsymbol {y} ^ {\top} \left(\boldsymbol {H} _ {p} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {E} _ {2, p}.
$$
From Lemmas 9 and 12, we directly have that
$$
\begin{array}{l} T _ {1 1} = \boldsymbol {\alpha} ^ {\top} \boldsymbol {P} _ {\leq \ell} \boldsymbol {\alpha} + o _ {d, \mathbb {P}} (1) \\ = \| P _ {\leq \ell} f _ {*} \| _ {L ^ {2}} ^ {2} + o _ {d, \mathbb {P}} (1). \\ \end{array}
$$
Applying (A.9) from above, we have  $T_{12} \leq T_{23} \| P_{\leq \ell} f_* \|_{L^2}^2 = o_{d,\mathbb{P}}(1)$ . Lastly, we know that