SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jimmy Ba $^{1,2,3}$ , Murat A. Erdogdu $^{1,2}$ , Taiji Suzuki $^{4,5}$ , Zhichao Wang $^{6}$ , Denny Wu $^{7,8}$
<sup>1</sup>University of Toronto, <sup>2</sup>Vector Institute, <sup>3</sup>xAI, <sup>4</sup>University of Tokyo, <sup>5</sup>RIKEN AIP, <sup>6</sup>University of California San Diego, <sup>7</sup>New York University, <sup>8</sup>Flatiron Institute
{jba,erdogdu}@cs.toronto.edu, taiji@mist.i.u-tokyo.ac.jp, zhw036@ucsd.edu, dennywu@nyu.edu
We consider the problem of learning a single-index target function  $f_{*}:\mathbb{R}^{d}\to \mathbb{R}$  under the spiked covariance data:
$$
f _ {*} (\boldsymbol {x}) = \sigma_ {*} \left(\frac {1}{\sqrt {1 + \theta}} \langle \boldsymbol {x}, \boldsymbol {\mu} \rangle\right), \quad \boldsymbol {x} \sim \mathcal {N} (0, \boldsymbol {I} _ {d} + \theta \boldsymbol {\mu} \boldsymbol {\mu} ^ {\top}), \quad \theta \asymp d ^ {\beta} \text {f o r} \beta \in [ 0, 1),
$$
where the link function  $\sigma_{*}:\mathbb{R}\to \mathbb{R}$  is a degree-  $p$  polynomial with information exponent  $k$  (defined as the lowest degree in the Hermite expansion of  $\sigma_{*}$ ), and it depends on the projection of input  $\pmb{x}$  onto the spike (signal) direction  $\pmb{\mu}\in \mathbb{R}^{d}$ . In the proportional asymptotic limit where the number of training examples  $n$  and the dimensionality  $d$  jointly diverge:  $n,d\rightarrow \infty, n / d\rightarrow \psi \in (0,\infty)$ , we ask the following question: how large should the spike magnitude  $\theta$  be, in order for  $(i)$  kernel methods,  $(ii)$  neural networks optimized by gradient descent, to learn  $f_{*}$ ?
We show that for kernel ridge regression,  $\beta \geq 1 - \frac{1}{p}$  is both sufficient and necessary. Whereas for two-layer neural networks trained with gradient descent,  $\beta > 1 - \frac{1}{k}$  suffices. Our results demonstrate that both kernel methods and neural networks benefit from low-dimensional structures in the data. Further, since  $k \leq p$  by definition, neural networks can adapt to such structures more effectively.
Learning under spiked covariance. Real-world data is often high-dimensional but it also exhibits certain low-dimensional structures. Indeed, the number of input features is exceedingly large in machine learning, yet most relevant information is concentrated in a low-dimensional subspace [LV07, HTFF09]. Such structures have important consequences on learning performance. In random matrix theory, high dimensionality is reflected by the proportional asymptotic limit where the numbers of training examples and input features diverge to infinity at the same rate, whereas the low-dimensional structure is often described by a spiked random matrix model [Joh01, BAP05, BS06, BGN11] in which a low-rank signal ("spike") is hidden in a large-dimensional noise ("bulk"). In this work, we restrict ourselves to the simple setting where the signal is rank-1, and consider the following data-generating process:
$$
\boldsymbol {x} \sim \mathcal {N} \left(0, \boldsymbol {I} _ {d} + \theta \boldsymbol {\mu} \boldsymbol {\mu} ^ {\top}\right), \quad f _ {*} (\boldsymbol {x}) = \sigma_ {*} \left(\frac {1}{\sqrt {1 + \theta}} \langle \boldsymbol {x}, \boldsymbol {\mu} \rangle\right), \tag {1.1}
$$
where the goal is to estimate the target function (teacher)  $f_{*}$ , which is a single-index model depending on the signal direction  $\mu \in \mathbb{R}^d$ , and  $\sigma_{*}: \mathbb{R} \to \mathbb{R}$  is an unknown link function. The signal  $\mu$  is also embedded in the spiked input data, and  $\theta > 0$  controls the spike magnitude; we scale  $\theta \asymp d^{\beta}$  for  $\beta \in [0,1)$ , where larger  $\beta$  indicates a more prominent low-dimensional structure, and hence the learning problem becomes easier. We aim to characterize the efficiency of learning  $f_{*}$  using kernel methods and neural networks (NNs) in relation to the strength of the low-dimensional signal  $\theta$ .
Prior results: isotropic data. When  $\theta = 0$ , the input features are isotropic and do not reveal any information about the target  $f_{*}$ . In this setting, the performance of kernel methods and two-layer NNs have been extensively studied in the proportional asymptotic limit.
- Kernel methods. For isotropic Gaussian or spherical input data, the performance of random features (RF) regression and kernel ridge regression (KRR) has been precisely characterized in the proportional limit [MM22, GLK+20, DL20, BMR21]. However, in this regime, RF models and KRR suffers from the "curse of dimensionality" [Bac17, YS19] and cannot achieve vanishing generalization error unless  $f_{*}$  is linear [EK10, GMMM21, HL22]. More generally, for KRR to learn a degree- $p$  polynomial target, a sample complexity of  $n = \Omega(d^{p})$  is required [GMMM21, LRZ20, DWY21, MMM21].
- Neural networks + Gradient descent. While NNs can efficiently approximate a single-index model (with polynomial link  $\sigma_{*}$ ), the optimization complexity of gradient-based learning on isotropic data is known to be governed by the information exponent, defined as the index of the first non-zero Hermite coefficient of  $\sigma_{*}$  [AGJ21]. For NNs trained with gradient descent (GD) to learn a target with information exponent  $k$ , recent works showed that a sample complexity of  $n = \tilde{\Theta}(d^k)$  suffices<sup>1</sup> [AGJ21, DLS22, BBSS22, ABAM23]. Hence, in the proportional regime ( $n \asymp d$ ), learnability has only been established for  $k = 1$  [AAM22,  $\mathrm{BES}^{+}22$ , MHPG+23, BMZ23].
These prior results delineate the limitation of kernel methods and GD-trained NNs under isotropic data in the high-dimensional proportional scaling: KRR can only learn linear functions on the input, and NNs learn targets with information exponent  $k = 1$ , which cannot cover many important problems such as phase retrieval [Fie82, CC15, TV23]. Following the intuition that low-dimensional structure in the input data can make the learning problem easier, we ask the following question.
If the input data contains low-dimensional structure given by the spiked covariance model (1.1), can kernel methods and neural networks learn a larger class of  $f_{*}$  in the proportional regime?
We answer this question in the affirmative by showing that in the proportional asymptotic limit when the spike magnitude  $\theta$  reaches a certain threshold, both KRR and GD-trained two-layer NN can learn the target function  $f_{*}$ , where the link function  $\sigma_{*}$  is a degree- $p$  polynomial with information exponent  $k$  (see Definition 1). Our findings are summarized below and illustrated in Figure 1.
- For KRR with the inner-product kernel, we give a sharp analysis of the prediction risk and show that to learn a link function  $\sigma_{*}$  with degree  $p$  in the proportional regime, a spike magnitude of  $\theta = \Omega \left(d^{1 - \frac{1}{p}}\right)$  is both necessary and sufficient.
- For a two-layer NN with ReLU activation, we upper bound the prediction risk when representation learning is performed via one gradient descent step on the first-layer parameters (analogous to  $\mathrm{[BES^{+}22}$ , DLS22]), and show that a spike magnitude of  $\theta = \omega\left(d^{1 - \frac{1}{k}}\right)$  is sufficient to learn  $\sigma_{*}$  with information exponent  $k$ .
Figure 1: Spike magnitude  $\theta \asymp d^{\beta}$  for KRR and GD-trained NN to learn the target  $f_{*}$  with degree  $p$  and information exponent  $k$  in the proportional regime. When  $k\neq p$ , KRR requires a stronger low-dimensional structure (i.e., larger spike) to learn  $f_{*}$ .
Our analysis reveals that both KRR and NNs can benefit from the low-dimensional structure (spike) in the input data, and they can learn a wider class of target functions in the proportional regime compared to the isotropic setting (i.e.,  $p > 1$  for KRR, and  $k > 1$  for NNs). Furthermore, since for a given link function  $k \leq p$  by definition, we know that NNs require a smaller spike magnitude  $\theta$  to learn the same target compared to KRR (see Figure 1). In other words, a GD-trained NN can utilize such a low-dimensional structure more effectively than KRR.
<sup>1</sup>The appearance of  $k$  in the exponent of the dimensionality is also consistent with the correlational statistical query (CSQ) lower bound [DLS22, ABAM23], which relates to the hardness of learning via gradient descent.
Related works. Related to our problem setting, [GMMM20] studied the sample complexity of KRR and NNs in a spiked covariance model, and showed that both models indeed benefit from a stronger low-dimensional structure (i.e., larger spike). However, their analysis assumed the number of spikes to be diverging and cannot cover the proportional regime we consider. More importantly, the NN analysis is limited to approximation, and gradient-based optimization guarantee is not given.
In [RGKZ21], the authors presented a non-rigorous analysis on classifying XOR mixtures using an RF model and a two-layer NN trained with online SGD and showed that the NN can learn the task under smaller spike magnitude. Such a task is parallel to the information exponent  $k = 2$  setting (see [ABAM23]); our results rigorously demonstrate that similar separation exists for a different class of target functions with potentially higher degree and information exponent.
2 Preliminaries: Problem Setting and Assumptions
Notations.  $\| \cdot \|$  denotes the  $\ell_2$  norm for vectors and the  $\ell_2\to \ell_2$  operator norm for matrices, and  $\| \cdot \| _F$  is the Frobenius norm.  $\mathcal{O}_d(\cdot)$  and  $o_d(\cdot)$  stand for the standard big-O and little-o notations, where the subscript highlights the asymptotic variable; we write  $\tilde{\mathcal{O}} (\cdot)$  when (poly-)logarithmic factors are ignored.  $\mathcal{O}_{d,\mathbb{P}}(\cdot)$  (resp.  $o_{d,\mathbb{P}}(\cdot))$  represents big-O (resp. little-o) in probability as  $d\rightarrow \infty$ $\Omega (\cdot),\Theta (\cdot),\omega (\cdot)$  are defined analogously. Denote  $\gamma$  as the standard Gaussian distribution in  $\mathbb{R}$ . Given  $f:\mathbb{R}^d\to \mathbb{R}$ , we denote its  $L^p$  -norm with respect to the data distribution as  $\| f\|_{L^p}$ .
2.1 Basic Assumptions
We consider Gaussian input data with a spiked covariance:
$$
\boldsymbol {x} \sim \mathcal {N} \left(0, \boldsymbol {I} _ {d} + \theta \boldsymbol {\mu} \boldsymbol {\mu} ^ {\top}\right), \quad \theta \asymp d ^ {\beta} \text {f o r s o m e} \beta \in [ 0, 1), \tag {2.1}
$$
where the signal direction  $\pmb{\mu} \in \mathbb{R}^d$  satisfies  $\|\pmb{\mu}\| = 1$ , and the spike magnitude  $\theta$  is allowed to grow with the input dimensionality  $d$  as specified by the exponent  $\beta$ : as  $\beta$  gets larger, the spike magnitude increases and hence the data exhibits stronger low-dimensional structure. Following the terminology in [GMMM20], we may define the effective dimensionality of the input data as  $d_{\mathrm{eff}} \asymp d^{1 - \beta}$ , which captures the intuition that a larger spike renders the features more low-dimensional. As we will see, the sample complexity of KRR and two-layer NN is decided by this effective dimensionality.
Teacher model. We consider a student-teaching setting, where the labels  $y$  are generated from a teacher model (target function)  $f_{*}:\mathbb{R}^{d}\to \mathbb{R}$ . In the spiked covariance model (2.1), it is known that input directions with large variations are often good predictors of the labels — indeed, this is the main reason that principal component analysis is used in practice [HTFF09]. We therefore consider the following single-index target function where the index features align with the spike direction  $\mu$ :
$$
y = f _ {*} (\boldsymbol {x}) = \sigma_ {*} \left(\frac {1}{\sqrt {1 + \theta}} \langle \boldsymbol {x}, \boldsymbol {\mu} \rangle\right), \tag {2.2}
$$
where the link function  $\sigma_{*} \in L^{2}(\gamma)$  is centered such that  $\mathbb{E}[\sigma_{*}(z)] = 0$  for  $z \sim \mathcal{N}(0,1)$  (this can be achieved by subtracting the mean from the training labels as in [DLS22]), and the normalization factor  $(1 + \theta)^{-1/2}$  ensures that the  $L^2$  norm of  $f_*$  remains constant. We remark that given the prior knowledge of low-dimensional structure, we may efficiently learn  $f_*$  by first performing PCA on the input features; our goal, however, is not to construct an optimal learning algorithm, but to understand the behavior of KRR and two-layer NN without such data preprocessing, similar to [GMMM20].
We also assume  $\sigma_{*}$  is a degree-  $p$  polynomial with information exponent (IE)  $k$  defined below.
Definition 1 (Information exponent). Let  $\{h_j\}_{j=0}^{\infty}$  denote the normalized Hermite polynomials. The information exponent of  $f \in L^{2}(\gamma)$ , which we denote by  $k$ , is the index of the first non-zero Hermite coefficient of  $f$ , that is, given the Hermite expansion  $f = \sum_{j=0}^{\infty} \alpha_j h_j$ ,  $k := \min \{j > 0 : \alpha_j \neq 0\}$ .
It is clear that by definition, we always have  $k \leq p$ , and equality is only achieved when  $\sigma_*$  is a "pure" degree- $k$  Hermite polynomial. Intuitively, IE measures the magnitude of information contained in the gradient, and larger  $k$  implies increased gradient descent complexity [AGJ21, BBSS22]. Such definition is also related to the recently introduced leap complexity [ABA22].
Remark. In (2.2) we do not corrupt the training labels  $y$  with i.i.d. label noise, meaning that, in light of the bias-variance decomposition, we analyze the bias term describing the extent  $f_{*}$  is learned by the student model. If label noise is introduced, the additional variance term (describing the "overfitting" due to noise) can be handled by standard concentration argument.
2.2 Learning Objective and Training Procedure
Given training examples  $\{(x_i, y_i)\}_{i=1}^n$ , where  $x_i \stackrel{\text{i.i.d.}}{\sim} \mathcal{N}(0, I_d + \theta \mu \mu^\top)$  and  $y_i = f_*(x_i)$ , we consider two student models obtained via the following training procedure in the proportional regime:
$$
\text {P r o p o r t i a l} n, d \rightarrow \infty , \quad n / d \rightarrow \psi \in (0, \infty). \tag {2.3}
$$
Student model I: Kernel ridge regression. We focus on the inner-product kernel:  $k(\boldsymbol{x}, \boldsymbol{y}) = g\left(\frac{\langle\boldsymbol{x}, \boldsymbol{y}\rangle}{d}\right)$  for some  $g \in L^{2}(\gamma)$  (similar argument can also apply to rotationally invariant kernels as in [EK10, DWY21]). Denoting the associated reproducing kernel Hilbert space (RKHS) with  $\mathcal{H}$ , the kernel ridge regression estimator is given by
$$
\hat {f} _ {\ker} = \underset {f \in \mathcal {H}} {\operatorname {a r g m i n}} \left\{\frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - f \left(\boldsymbol {x} _ {i}\right)\right) ^ {2} + \lambda \| f \| _ {\mathcal {H}} ^ {2} \right\} \Rightarrow \hat {f} _ {\ker} (\boldsymbol {x}) = k (\boldsymbol {x}, \boldsymbol {X}) ^ {\top} \left(\boldsymbol {K} + \lambda \boldsymbol {I}\right) ^ {- 1} \boldsymbol {y}, \tag {2.4}
$$
where  $\lambda > 0$  is the ridge parameter, and  $K \in \mathbb{R}^{n \times n}$  is the kernel Gram matrix with entries  $K_{i,j} = k(\pmb{x}_i, \pmb{x}_j)$ . Spectral properties of the kernel matrix and the prediction risk of KRR have been extensively studied in various high-dimensional settings [EK10, GMMM21, BMR21, XHM+22, LY22]. Since the kernel function is fixed before seeing the data, we intuitively expect that KRR cannot effectively adapt to a low-dimensional structure in the learning problem.
Student model II: Two-layer neural network. We aim to learn the following width-  $N$  network:
$$
f _ {\mathrm {N N}} (\boldsymbol {x}) = \frac {1}{\sqrt {N}} \sum_ {i = 1} ^ {N} a _ {i} \sigma \left(\left\langle \boldsymbol {x}, \boldsymbol {w} _ {i} \right\rangle + b _ {i}\right) = \left\langle \boldsymbol {a}, N ^ {- 1 / 2} \sigma \left(\boldsymbol {W} ^ {\top} \boldsymbol {x} + \boldsymbol {b}\right) \right\rangle , \tag {2.5}
$$
where  $\sigma : \mathbb{R} \to \mathbb{R}$  is a nonlinear activation, and the trainable parameters  $\pmb{W} \in \mathbb{R}^{d \times N}$ ,  $\pmb{a}, \pmb{b} \in \mathbb{R}^N$  are initialized from standard Gaussian distribution. We require the network width to be  $N = \Omega(d^{\varepsilon})$  for some small  $\varepsilon > 0$ . Such a choice entails that it is sufficient to have  $\Theta(d^{1 + \varepsilon})$  total training parameters, which is almost proportional to the sample size  $n \asymp d$ ; in contrast, for KRR we need to store the kernel Gram matrix with  $n^2$  entries, which is less computationally efficient at test time.
It is known that NNs can adapt to the learning problem via representation learning, in which the trained features encode relevant information of the target function. To realize this advantage, we update the parameters in (2.5) via the two-stage procedure introduced in  $\mathrm{[BES^{+}22}$ , DLS22,  $\mathrm{BEG^{+}22}$ , where we first learn the representation by taking one gradient descent step on the first-layer parameters  $\mathbf{W}$ , and then estimate the second-layer parameters  $\mathbf{a}$  separately (which is a convex problem). This procedure is summarized as follows (see Algorithm 1 in Appendix for more details).
1. Feature learning for the 1st layer: We optimize the representation of the NN via gradient descent, where the training objective is the empirical squared loss  $\mathcal{L}(f) = \frac{1}{n}\sum_{i=1}^{n}(f(\pmb{x}_i) - y_i)^2$ . Specifically, denote the  $i$ -th column of the initialized weight matrix  $\pmb{W}_0$  as  $\pmb{w}_i^{(0)} \in \mathbb{R}^d$ , and the initialized NN as  $f_{\mathrm{NN}}^0$ , we take one GD step with learning rate  $\eta$  as follows,
$$
\boldsymbol {w} _ {i} ^ {(1)} \leftarrow \boldsymbol {w} _ {i} ^ {(0)} - \eta \sqrt {N} \cdot \nabla_ {\boldsymbol {w} _ {i} ^ {(0)}} \mathcal {L} \big (f _ {\mathrm {N N}} ^ {(0)} \big).
$$
Due to the anisotropic input data, the pre-activation  $\langle \pmb{x},\pmb{w}_i^{(1)}\rangle$  may blow up if the trained parameter  $\pmb{w}_i^{(1)}$  aligns with the signal direction  $\pmb{\mu}$ . To circumvent this issue, we normalize each neuron to have (roughly) unit pre-activation, i.e., for  $i\in [N]$ , we perform the weight normalization:
$$
\boldsymbol {w} _ {i} ^ {(1)} \leftarrow \boldsymbol {w} _ {i} ^ {(1)} \cdot \left(\frac {1}{n} \sum_ {j = 1} ^ {n} \left\langle \boldsymbol {x} _ {j}, \boldsymbol {w} _ {i} ^ {(1)} \right\rangle^ {2}\right) ^ {- 1 / 2}.
$$
At high level, this resembles the often-used normalization layers in deep learning [IS15, BKH16].
2. Ridge regression for the 2nd layer. After obtaining the updated first-layer parameters  $\mathbf{W}_1 \in \mathbb{R}^{d \times N}$ , we optimize the second-layer  $\mathbf{a}$  by solving the ridge regression objective – this can be done by using GD to solve an  $\ell_2$ -penalized least squares. To circumvent the dependence between the training data  $\mathbf{X}$  and the learned  $\mathbf{W}_1$ , we follow [BES+22] and estimate the regression coefficients  $\hat{\mathbf{a}}$  using a new set of training data  $\{\tilde{x}_i, \tilde{y}_i\}_{i=1}^n$  with the same sample size  $n$ .
Denoting the feature matrix on the fresh training set  $\{\tilde{X},\tilde{y}\}$  as  $\Phi \coloneqq \frac{1}{\sqrt{N}}\sigma (\tilde{\boldsymbol{X}}\boldsymbol{W}_1 + \boldsymbol {b})\in \mathbb{R}^{n\times N}$ , the ridge regression estimator can be obtained by:  $\hat{\boldsymbol{a}} = \mathrm{argmin}_{\boldsymbol{a}}\left\{\frac{1}{n}\| \tilde{\boldsymbol{y}} -\Phi \boldsymbol {a}\| ^2 +\lambda \| \boldsymbol {a}\| ^2\right\}$ .
Learning in the proportional limit. Given a target function  $f_{*}$  and a learned (student) model  $\hat{f}$ , we evaluate the model performance using the prediction risk:
$$
\mathcal {R} (\hat {f}) := \mathbb {E} _ {\boldsymbol {x}} [ (\hat {f} (\boldsymbol {x}) - f _ {*} (\boldsymbol {x})) ^ {2} ] = \| \hat {f} - f _ {*} \| _ {L ^ {2}} ^ {2},
$$
where the expectation is taken over the (anisotropic) Gaussian data distribution. To benchmark the model performance in the proportional limit (2.3), we introduce the following notion of learnability.
Definition 2 (Learnability in the proportional regime). We say an algorithm learns the target function  $f_{*}$  in the proportional regime if, for any (small) constant  $\epsilon > 0$ , there exists some constant  $\psi_{*} > 0$  such that when  $n / d \to \psi \geq \psi_{*}$ , the algorithm constructs  $\hat{f}$  using  $n$  i.i.d. training examples, and achieves  $\mathcal{R}(\hat{f}) = \| \hat{f} - f_{*} \|_{L^{2}}^{2} \leq \epsilon$  with probability 1 when  $n, d \to \infty$  proportionally.
Based on this definition, in order for a student model  $\hat{f}$  (e.g., KRR or two-layer NN) to learn the teacher  $f_{*}$  in the proportional regime, it needs to achieve arbitrarily small prediction risk with high probability, and with linear sample complexity (i.e., using  $n \asymp d$  training samples).
3 Kernel Ridge Regression
For an inner-product kernel, the performance of KRR depends on the behavior of the pairwise inner-product of the training examples  $\langle x_i, x_j \rangle$ . Intuitively speaking, since the inner-products concentrate in high dimensions, the kernel matrix can be approximated in operator norm via low-degree Taylor expansion, and thus KRR can only learn  $f_*$  that is low-degree (e.g., see [DWY21, WZ23]). Introducing a low-dimensional structure to the data is one way to counteract such near-orthogonality in high dimensions. Hence we expect that KRR can achieve better performance when the spike magnitude  $\theta$  becomes larger – this intuition will be made rigorous in the following subsections.
3.1 Sharp Analysis of the Prediction Risk
In this section, we study the prediction risk of KRR defined in (2.4) with a positive semidefinite kernel given by  $k(\pmb{x}_i,\pmb{x}_j) = g(\langle \pmb{x}_i,\pmb{x}_j\rangle /d)$ , i.e.,  $g^{(k)}(0)\geq 0$  for all  $k$  (see [SS02, Section 13.1]). Recall that the strength of the low-dimensional component is determined by the spike magnitude  $\theta \asymp d^{\beta}$ ; in the following, we characterize how the exponent  $\beta \in [0,1)$  affects the performance of KRR.
We denote by  $P_{\leq \ell}: L^2 \to L^2$  the orthogonal projector onto the subspace of polynomials of degree at most  $\ell$  for any  $\ell \in \mathbb{N}$ . Similarly, we denote by  $P_{> \ell} = \operatorname{Id} - P_{\leq \ell}$  the projector to the orthogonal complement where  $\operatorname{Id}$  is the identity operator. To illustrate this notion, consider  $\ell = 1$ , then we have
$$
f _ {*} (\boldsymbol {x}) = P _ {\leq 1} f _ {*} (\boldsymbol {x}) + P _ {> 1} f _ {*} (\boldsymbol {x}) = a ^ {*} + \langle \boldsymbol {b} ^ {*}, \boldsymbol {x} \rangle + P _ {> 1} f _ {*} (\boldsymbol {x}),
$$
where  $a^*, b^* = \operatorname{argmin}_{a,b} \mathbb{E}_x[(f_*(x) - a - \langle b, x \rangle)^2]$ . Note that any polynomial  $f_*$  with degree at most  $\ell$  satisfies  $P_{>\ell} f_* = 0$ . Denoting the KRR model as  $\hat{f}_{\mathrm{ker}}$ , we have the following sharp analysis of the asymptotic prediction risk.
Theorem 3. Given any fixed  $\ell \in \mathbb{N}$ . Suppose that  $g^{(k)}(0) > 0$  for all  $k \leq \ell$ , and the spike magnitude scales as  $\theta \asymp d^{\beta}$  for  $\beta \in \left(1 - \frac{1}{\ell}, 1 - \frac{1}{\ell + 1}\right)$ . Then as  $n, d \to \infty$ ,  $n / d \to \psi \in (0, \infty)$ , the prediction risk of the KRR estimator  $\hat{f}_{\mathrm{ker}}$  with  $\lambda = \mathcal{O}_d(1)$  satisfies the following with probability  $I$ ,
$$
\mathcal {R} (\hat {f} _ {\mathrm {k e r}}) - \| P _ {> \ell} f _ {*} \| _ {L ^ {2}} ^ {2} = o _ {d} (1).
$$
As a corollary of Theorem 3, based on Definition 2, we can obtain a sharp threshold of  $\beta$  for learning a degree- $p$  polynomial link function in the proportional regime (Definition 2).