SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yicheng Li, Haobo Zhang
Center for Statistical Science, Department of Industrial Engineering
{liyc22,zhang-hb21}@mails.tsinghua.edu.cn
Qian Lin *
qianlin@tsinghua.edu.cn
The widely observed 'benign overfitting phenomenon' in the neural network literature raises the challenge to the 'bias-variance trade-off' doctrine in the statistical learning theory. Since the generalization ability of the 'lazy trained' overparametrized neural network can be well approximated by that of the neural tangent kernel regression, the curve of the excess risk (namely, the learning curve) of kernel ridge regression attracts increasing attention recently. However, most recent arguments on the learning curve are heuristic and are based on the 'Gaussian design' assumption. In this paper, under mild and more realistic assumptions, we rigorously provide a full characterization of the learning curve in the asymptotic sense under a power-law decay condition of the eigenvalues of the kernel and also the target function. The learning curve elaborates the effect and the interplay of the choice of the regularization parameter, the source condition and the noise. In particular, our results suggest that the 'benign overfitting phenomenon' exists in over-parametrized neural networks only when the noise level is small.
Kernel methods, in particular kernel ridge regression (KRR), have been one of the most popular algorithms in machine learning. Its optimality under various settings has been an active topic since Caponnetto and De Vito [2007], Andreas Christmann [2008]. The renaissance of kernel methods arising from the neural tangent kernel (NTK) theory [Jacot et al., 2018], which shows that over-parametrized neural networks can be well approximated by certain kernel regression with the corresponding NTK, has posed further challenges about the interplay of generalization, regularization and noise level. For example, it has been observed empirically that over-parametrized neural networks can fit any data perfectly but also generalize well [Zhang et al., 2017], which contradicts to our traditional belief of bias-variance trade-off [Vapnik, 1999].
The aforementioned 'benign overfitting phenomenon' that overfitted neural networks generalize well attracts lots of attention recently. Researchers provide various explanations to reconcile the contradiction between it and the bias-variance trade-off principle. For example, Belkin et al. [2019] proposed the 'double descent theory' to explain why large model can generalize well; some other works (e.g., Liang and Rakhlin [2020]) argued that kernel interpolating estimators can generalize
*Qian Lin also affiliates with Beijing Academy of Artificial Intelligence, Beijing, China
well in high dimensional settings. In contrast to the 'benign overfitting phenomenon', several other works (e.g., Rakhlin and Zhai [2018], Li et al. [2023a]) recently showed that kernel interpolation can not generalize in traditional fixed dimension setting. In order to understand the 'benign overfitting phenomenon', it would be of great interest to characterize the learning curve: the curve of the exact order of the generalization error of a certain algorithm (e.g., KRR) varying with respect to different choices of regularization parameters.
Recently, several works (e.g., Bordelon et al. [2020], Cui et al. [2021]) depicted the learning curve of KRR under the Gaussian design assumption that the eigenfunctions (see (5)) are i.i.d. Gaussian random functions. Though it is easy to figure out that the Gaussian design assumption can not be true in most scenarios, with some heuristic arguments, Cui et al. [2021] provide a description of the learning curves of KRR with respect to the regularization, source condition and noise levels. These works offered us some insights on the learning curve of KRR which strongly suggests that the learning curve should be U-shaped if the observations are noisy or monotone decreasing if the observations are noiseless.
In this paper, we consider the learning curves of KRR under the usual settings (without the Gaussian design assumption). Under mild assumptions, we rigorously prove the asymptotic rates of the excess risk, including both upper and lower bounds. These rates show the interplay of the eigenvalue decay of the kernel, the relative smoothness of the regression function, the noise and the choice of the regularization parameter. As a result, we obtain the traditional U-shaped learning curve for the noisy observation case and a monotone decreasing learning curve for the noiseless case, providing a full picture of the generalization of KRR in the asymptotic sense. Combined with the NTK theory, our results may also suggest that 'the benign overfitting phenomenon' may not exist if one trains a very wide neural network.
The main contribution of this paper is that we remove the unrealistic Gaussian design assumption in previous non-rigorous works [Bordelon et al., 2020, Cui et al., 2021] and provide mathematically solid proof of the exact asymptotic rates of KRR with matching upper and lower bounds.
To be precise, let us introduce the quantities  $\lambda$ , the regularization parameter in (1);  $\beta$ , the eigenvalue decay rate in (6), which characterizes the span of the underlying reproducing kernel Hilbert space (RKHS); and  $s$ , the smoothness index in (12), describes the relative smoothness of the regression function with respect to the RKHS. Here we note that larger  $\beta$  implies better regularity the RKHS and also larger  $s$  also implies better relative smoothness. Then, the asymptotic rates of the generalization error (excess risk)  $R(\lambda)$  in the noisy case is roughly
$$
R (\lambda) = \left\{ \begin{array}{l l} \Theta \big (\lambda^ {\min  (s, 2)} + \sigma^ {2} \lambda^ {- 1 / \beta} / n \big), & \quad \text {i f} \quad \lambda = \Omega (n ^ {- \beta}); \\ \Omega (\sigma^ {2}), & \quad \text {i f} \quad \lambda = O (n ^ {- \beta}); \end{array} \right.
$$
where  $n$  is the number of the samples and  $\sigma^2$  is the noise level. This result justifies the traditional U-shaped learning curve (see also Figure 1 on page 6) with respect to the regularization parameter.
For the technical part, we use the bias-variance decomposition and determine the exact rates of the both terms. Since the variance term was already considered in Li et al. [2023a], the main focus of this work is the bias term. Our technical contributions include:
- When the regularization parameter  $\lambda$  is not so small, that is,  $\lambda = \Omega(n^{-\beta})$ , we provide sharp estimates of the asymptotic orders (Lemma 4.1) of the bias term with both upper and lower bounds. Our result holds for both the well-specified case ( $s \geq 1$ ) and the mis-specified case ( $s \in (0,1)$ ), which improves the upper bounds given in Zhang et al. [2023a].
- We further show an upper bound (Lemma A.12) of the bias term in the nearly interpolating case, i.e.,  $\lambda = O(n^{-\beta})$ . The upper bound is tight and matches the information-theoretic lower bound provided in Proposition 4.4.
- Combining these results, we provide learning curves of KRR for both the noisy case (Theorem 3.2) and the noiseless case (Theorem 3.4). The results justify our traditional belief of the bias-variance trade-off principle.
- Our new techniques can also be generalized to other settings and might be of independent interest.
The optimality of kernel ridge regression has been studied extensively [Caponnetto and De Vito, 2007, Steinwart et al., 2009, Fischer and Steinwart, 2020, Zhang et al., 2023a]. Caponnetto and De Vito [2007] provided the classical optimality result of KRR in the well-specified case and the subsequent works further considered the mis-specified case. However, these works only provided an upper bound and the worst-case (minimax) lower bound, which are not sufficient for determining the precise learning curve. In order to answer the "benign overfitting" phenomenon [Bartlett et al., 2020, Liang and Rakhlin, 2020], several works [Rakhlin and Zhai, 2018, Buchholz, 2022, Beaglehole et al., 2022] tried to provide a lower bound for the kernel interpolation, which is a limiting case of KRR, but these works only focused on particular kernels and their techniques can hardly be generalized to provide a lower bound for KRR.
Another line of recent works considered the generalization performance of KRR under the Gaussian design assumption of the eigenfunctions [Bordelon et al., 2020, Jacot et al., 2020, Cui et al., 2021, Mallinar et al., 2022]. In particular, the learning curves of KRR was described in Bordelon et al. [2020], Cui et al. [2021], but heuristic arguments are also made in addition to the unrealistic Gaussian design assumption. Though the heuristic arguments are inspirational, a rigorous proof is indispensable if one plans to perform further investigations. In this work, we provide the first rigorous proof for most scenarios of the smoothness  $s$ , eigenvalue decay rate  $\beta$ , noise level  $\sigma^2$  and the regularization parameter  $\lambda$  based on the most common/realistic assumptions.
Recently, in order to show the so-called "saturation effect" in KRR, Li et al. [2023b] proved the exact asymptotic order of both the bias and the variance term when the regression function is very smooth and the regularization parameter  $\lambda$  is relatively large. Inspired by their analysis, Li et al. [2023a] showed the exact orders of the variance term. Our work further determines the orders of the bias term, completing the full learning curve or KRR.
KRR is also connected with Gaussian process regression [Kanagawa et al., 2018]. Jin et al. [2021] claimed to establish the learning curves for Gaussian process regression and thus for KRR. However, as pointed out in Zhang et al. [2023b], there is a gap in their argument. Moreover, their results are also more restrictive than ours, see Section 3.3 for a comparison.
Notations We write  $L^p(\mathcal{X}, \mathrm{d}\mu)$  for the Lebesgue space and sometimes abbreviate it as  $L^p$ . We use asymptotic notations  $O(\cdot)$ ,  $o(\cdot)$ ,  $\Omega(\cdot)$  and  $\Theta(\cdot)$ , and use  $\tilde{\Theta}(\cdot)$  to suppress logarithm terms. We also write  $a_n \asymp b_n$  for  $a_n = \Theta(b_n)$ . We will also use the probability versions of the asymptotic notations such as  $O_{\mathbb{P}}(\cdot)$ . Moreover, to present the results more clearly, we denote  $a_n = O^{\mathrm{poly}}(b_n)$  if  $a_n = O(n^p b_n)$  for any  $p > 0$ ,  $a_n = \Omega^{\mathrm{poly}}(b_n)$  if  $a_n = \Omega(n^{-p} b_n)$  for any  $p > 0$ ,  $a_n = \Theta^{\mathrm{poly}}(b_n)$  if  $a_n = O^{\mathrm{poly}}(b_n)$ , and  $a_n = \Omega^{\mathrm{poly}}(b_n)$ ; and we add a subscript  $\mathbb{P}$  for their probability versions.
Let  $\mathcal{X} \subset \mathbb{R}^d$  be compact and  $\rho$  be a probability measure on  $\mathcal{X} \times \mathbb{R}$ , whose marginal distribution on  $\mathcal{X}$  is denoted by  $\mu$ . Suppose that we are given  $n$  i.i.d. samples  $(x_1, y_1), \ldots, (x_n, y_n)$  from  $\rho$ . Let  $k$  be a continuous positive definite kernel  $k$  over  $\mathcal{X}$  and  $\mathcal{H}$  be the separable reproducing kernel Hilbert space (RKHS) associated with  $k$ . Then, kernel ridge regression (KRR) obtains the regressor  $\hat{f}_{\lambda}$  via the following convex optimization problem
$$
\hat {f} _ {\lambda} = \underset {f \in \mathcal {H}} {\arg \min } \left(\frac {1}{n} \sum_ {i = 1} ^ {n} \left(y _ {i} - f \left(x _ {i}\right)\right) ^ {2} + \lambda \| f \| _ {\mathcal {H}} ^ {2}\right), \tag {1}
$$
where  $\lambda > 0$  is the regularization parameter. Let us denote  $X = (x_{1},\ldots ,x_{n})$  and  $\pmb{y} = (y_{1},\dots ,y_{n})^{T}$ . A closed form of (1) can be provided by the representer theorem [Andreas Christmann, 2008]:
$$
\hat {f} _ {\lambda} (x) = \mathbb {K} (x, X) (\mathbb {K} (X, X) + n \lambda) ^ {- 1} \boldsymbol {y} \tag {2}
$$
where  $\mathbb{K}(x,X) = (k(x,x_1),\ldots ,k(x,x_n))$  and  $\mathbb{K}(X,X) = \bigl (k(x_i,x_j)\bigr)_{n\times n}$ .
In terms of the generalization performance of  $\hat{f}_{\lambda}$ , we consider the excess risk with respect to the squared loss
$$
\mathbb {E} _ {x \sim \mu} \left[ \hat {f} _ {\lambda} (x) - f _ {\rho} ^ {*} (x) \right] ^ {2} = \left\| \hat {f} _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2} (\mathcal {X}, \mathrm {d} \mu)} ^ {2}, \tag {3}
$$
where  $f_{\rho}^{*}(x) \coloneqq \mathbb{E}_{\rho}[y \mid x]$  is the conditional expectation and is also referred to as the regression function. We aim to provide asymptotic orders of (3) with respect to  $n$ .
2.1 The integral operator
We will introduce the integral operator, which is crucial for the analysis, as the previous works [Capon-netto and De Vito, 2007, Lin et al., 2018]. Denote by  $\mu$  the marginal probability measure of  $\rho$  on  $\mathcal{X}$ . Since  $k$  is continuous and  $\mathcal{X}$  is compact, let us assume  $\sup_{x\in \mathcal{X}}k(x,x)\leq \kappa^2$ . Then, it is known [Andreas Christmann, 2008, Steinwart and Scovel, 2012] that we have the natural embedding  $S_{\mu}:\mathcal{H}\rightarrow L^{2}$ , which is a Hilbert-Schmidt operator with Hilbert-Schmidt norm  $\| S_{\mu}\|_{\mathrm{HS}}\leq \kappa$ . Let  $S_{\mu}^{*}:L^{2}\to \mathcal{H}$  be the adjoint operator of  $S_{\mu}$  and  $T = S_{\mu}S_{\mu}^{*}:L^{2}\to L^{2}$ . Then, it is easy to show that  $T$  is an integral operator given by
$$
(T f) (x) = \int_ {\mathcal {X}} k (x, y) f (y) \mathrm {d} \mu (y), \tag {4}
$$
and it is self-adjoint, positive and trace-class (thus compact) with trace norm  $\| T\| _1\leq \kappa^2$  [Caponnetto and De Vito, 2007, Steinwart and Scovel, 2012]. Moreover, the spectral theorem of compact self-adjoint operators and Mercer's theorem [Steinwart and Scovel, 2012] yield the decompositions
$$
T = \sum_ {i \in N} \lambda_ {i} \left\langle \cdot , e _ {i} \right\rangle_ {L ^ {2}} e _ {i}, \quad k (x, y) = \sum_ {i \in N} \lambda_ {i} e _ {i} (x) e _ {i} (y), \tag {5}
$$
where  $N \subseteq \mathbb{N}$  is an index set,  $\{\lambda_i\}_{i \in N}$  is the set of positive eigenvalues of  $T$  in descending order, and  $e_i$  is the corresponding eigenfunction. Furthermore,  $\{e_i\}_{i \in N}$  forms an orthonormal basis of  $\overline{\operatorname{Ran} S_\mu} \subseteq L^2$  and  $\left\{\lambda_i^{1/2} e_i\right\}_{i \in N}$  forms an orthonormal basis of  $\overline{\operatorname{Ran} S_\mu^*} \subseteq \mathcal{H}$ .
The eigenvalues  $\lambda_{i}$  actually characterize the span of the RKHS and the interplay between  $\mathcal{H}$  and  $\mu$ . Since we are interested in the infinite-dimensional case, we will assume  $N = \mathbb{N}$  and assume the following polynomial eigenvalue decay as in the literature [Caponnetto and De Vito, 2007, Fischer and Steinwart, 2020, Li et al., 2023b], which is also referred to as the capacity condition or effective dimension condition. Larger  $\beta$  implies better regularity of the functions in the RKHS.
Assumption 1 (Eigenvalue decay). There is some  $\beta > 1$  and constants  $c_{\beta}, C_{\beta} > 0$  such that
$$
c _ {\beta} i ^ {- \beta} \leq \lambda_ {i} \leq C _ {\beta} i ^ {- \beta} (i = 1, 2, \dots), \tag {6}
$$
where  $\lambda_{i}$  is the eigenvalue of  $T$  defined in (5).
Such a polynomial decay is satisfied for the well-known Sobolev kernel [Fischer and Steinwart, 2020], Laplace kernel and, of most interest, neural tangent kernels for fully-connected multilayer neural networks [Bietti and Mairal, 2019, Bietti and Bach, 2020, Lai et al., 2023].
2.2 The embedding index of an RKHS
We will consider the embedding index of an RKHS to sharpen our analysis. Let us first define the fractional power  $T^s: L^2 \to L^2$  for  $s \geq 0$  by
$$
T ^ {s} (f) = \sum_ {i \in N} \lambda_ {i} ^ {s} \left\langle f, e _ {i} \right\rangle_ {L ^ {2}} e _ {i}. \tag {7}
$$
Then, the interpolation space [Steinwart and Scovel, 2012, Fischer and Steinwart, 2020, Li et al., 2023b]  $[\mathcal{H}]^s$  is defined by
$$
\left[ \mathcal {H} \right] ^ {s} = \operatorname {R a n} T ^ {s / 2} = \left\{\sum_ {i \in N} a _ {i} \lambda_ {i} ^ {s / 2} e _ {i} \mid \sum_ {i \in N} a _ {i} ^ {2} <   \infty \right\} \subseteq L ^ {2}, \tag {8}
$$
with the norm  $\left\| \sum_{i\in N}a_i\lambda_i^{s / 2}e_i\right\|_{[\mathcal{H}]^s} = \left(\sum_{i\in N}a_i^2\right)^{1 / 2}$ . One may easily verify that  $[\mathcal{H}]^s$  is also a separable Hilbert space with an orthonormal basis  $\left\{\lambda_i^{s / 2}e_i\right\}_{i\in N}$ . Moreover, it is clear that  $[\mathcal{H}]^0 = \overline{\mathrm{Ran}S_\mu}\subseteq L^2$  and  $[\mathcal{H}]^1 = \overline{\mathrm{Ran}S_\mu^*}\subseteq \mathcal{H}$ . It can also be shown that if  $s_1 > s_2\geq 0$ , the inclusions  $[\mathcal{H}]^{s_1}\hookrightarrow [\mathcal{H}]^{s_2}$  are compact [Steinwart and Scovel, 2012].
Now, we say  $\mathcal{H}$  has an embedding property of order  $\alpha \in (0,1]$  if  $[\mathcal{H}]^\alpha$  can be continuously embedded into  $L^{\infty}(\mathcal{X},\mathrm{d}\mu)$ , that is, the operator norm
$$
\| [ \mathcal {H} ] ^ {\alpha} \hookrightarrow L ^ {\infty} (\mathcal {X}, \mu) \| = M _ {\alpha} <   \infty . \tag {9}
$$
Moreover, Fischer and Steinwart [2020, Theorem 9] shows that
$$
\left\| \left[ \mathcal {H} \right] ^ {\alpha} \hookrightarrow L ^ {\infty} (\mathcal {X}, \mu) \right\| = \left\| k _ {\mu} ^ {\alpha} \right\| _ {L ^ {\infty}} := \operatorname {e s s} \sup  _ {x \in \mathcal {X}, \mu} \sum_ {i \in N} \lambda_ {i} ^ {\alpha} e _ {i} (x) ^ {2}. \tag {10}
$$
Therefore, since  $\sup_{x\in \mathcal{X}}k(x,x)\leq \kappa^2$ , we know that (9) always holds for  $\alpha = 1$ . By the inclusion relation of interpolation spaces, it is clear that if  $\mathcal{H}$  has the embedding property of order  $\alpha$ , then it has the embedding properties of order  $\alpha'$  for any  $\alpha' \geq \alpha$ . Consequently, we may introduce the following definition [Zhang et al., 2023b]:
Definition 2.1. The embedding index  $\alpha_0$  of an RKHS  $\mathcal{H}$  is defined by
$$
\alpha_ {0} = \inf  \left\{\alpha : \| [ \mathcal {H} ] ^ {\alpha} \hookrightarrow L ^ {\infty} (\mathcal {X}, \mu) \| = M _ {\alpha} <   \infty \right\}. \tag {11}
$$
It is shown in Fischer and Steinwart [2020, Lemma 10] that  $\alpha_0 \geq \beta$  and we assume the equality holds as the following assumption.
Assumption 2 (Embedding index). The embedding index  $\alpha_0 = 1 / \beta$ , where  $\beta$  is the eigenvalue decay in (6).
Lots of the usual RKHSs satisfy this embedding index condition. It is shown in Steinwart et al. [2009] that Assumption 2 holds if the eigenfunctions are uniformly bounded, namely  $\sup_{i\in N}\| e_i\|_{L^{\infty}} < \infty$ . Moreover, Assumption 2 also holds for the Sobolev RKHSs, RKHSs associated with periodic translation invariant kernels and RKHSs associated with dot-product kernels on spheres, see Zhang et al. [2023a, Section 4].
Before presenting our main results, we have to introduce a source condition on the regression function. Since we will establish both precise learning rates, we have to characterize the exact smoothness order of  $f_{\rho}^{*}$  rather than merely assume  $f_{\rho}^{*}$  belongs to some interpolation space  $[\mathcal{H}]^s$ .
Assumption 3 (Source condition). There are some  $s > 0$  and a sequence  $(a_{i})_{i\geq 1}$  such that
$$
f _ {\rho} ^ {*} = \sum_ {i = 1} ^ {\infty} a _ {i} \lambda_ {i} ^ {s / 2} i ^ {- 1 / 2} e _ {i} \tag {12}
$$
and  $0 < c \leq |a_i| \leq C$  for some constants  $c, C$ .
Remark 3.1. Assumption 3 is also considered in Cui et al. [2021, Eq. (8)] and a slightly weaker version of it is given in Jin et al. [2021, Assumption 5]. We only consider this simple form since there is no essential difference in the proof to consider the weaker version. From the definition (8) we can see that Assumption 3 implies  $f_{\rho}^{*} \in [\mathcal{H}]^{t}$  for any  $t < s$  but  $f_{\rho}^{*} \notin [\mathcal{H}]^{s}$ .
3.1 Noisy case
Let us first consider the noisy case with the following assumption:
Assumption 4 (Noise). We assume
$$
\mathbb {E} _ {(x, y) \sim \rho} \left[ (y - f _ {\rho} ^ {*} (x)) ^ {2} \mid x \right] = \sigma^ {2} > 0, \quad \mu \text {- a . e .} x \in \mathcal {X}. \tag {13}
$$
Figure 1: An illustration of the learning curves when choosing  $\lambda = n^{-\theta}$ . First row: The bias-variance plot and the error curves for the noisy and noiseless cases. Second row: Tow phase diagrams of the asymptotic rates of the excess risk with respect to parameter pairs  $(\theta, s)$  and  $(\theta, \tau)$ , where we set  $\sigma^2 = n^{-\tau}$  and  $\tilde{s} = \min(s, 2)$ . In the "underfitting" ("overfitting") region, bias (variance) is dominating. The "interpolating" region refers to the extreme cases of overfitting that the excess risk is lower bounded by a constant. For the first diagram we consider the case of constant noise. For the second diagram, the red vertical line shows the crossover of the noisy regime to the noiseless regime and an upper bound for the blank area on the upper-right corner is unknown yet.
For technical reason, we have to further assume the kernel to be Hölder-continuous, which is first in introduced in Li et al. [2023b]. This assumption is satisfied for the Laplace kernel, Sobolev kernels and neural tangent kernels.
Assumption 5. The kernel  $k$  is Hölder-continuous, that is, there exists some  $p \in (0,1]$  and  $L > 0$  such that
$$
\left| k \left(x _ {1}, x _ {2}\right) - k \left(y _ {1}, y _ {2}\right) \right| \leq L \| \left(x _ {1}, x _ {2}\right) - \left(y _ {1}, y _ {2}\right) \| _ {\mathbb {R} ^ {d \times d}} ^ {p}, \quad \forall x _ {1}, x _ {2}, y _ {1}, y _ {2} \in \mathcal {X}. \tag {14}
$$
Theorem 3.2. Under Assumptions 1-5, suppose  $\lambda \asymp n^{-\theta}$  for  $\theta >0$ . Then,
$$
\mathbb {E} \left[ \left\| \hat {f} _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} ^ {2} \mid X \right] = \left\{ \begin{array}{l l} \tilde {\Theta} _ {\mathbb {P}} \left(n ^ {- \min  (s, 2) \theta} + \sigma^ {2} n ^ {- (1 - \theta / \beta)}\right), & i f \quad \theta <   \beta \\ \Omega_ {\mathbb {P}} ^ {\text {p o l y}} (\sigma^ {2}), & i f \quad \theta \geq \beta , \end{array} \right. \tag {15}
$$
where  $\tilde{\Theta}_{\mathbb{P}}$  can be replaced with  $\Theta_{\mathbb{P}}$  for the first case if  $s \neq 2$ .
Remark 3.3. The two terms in the first case in Theorem 3.2 actually correspond to the bias and the variance term respectively. Balancing the two terms, we find the optimal regularization is  $\theta_{\mathrm{op}} = \frac{\beta}{\tilde{s}\beta + 1}$  and the optimal rate is  $\frac{\tilde{s}\beta}{\tilde{s}\beta + 1}$ , where  $\tilde{s} = \min(s, 2)$ , which recovers the classical optimal rate results [Caponnetto and De Vito, 2007]. Moreover, while we treat  $\sigma^2$  as fixed for simplicity, we can also allow  $\sigma^2$  to vary with  $n$ . Then, we can recover the results in Cui et al. [2021].
3.2 Noiseless case
Theorem 3.4. Under Assumptions 1-3, assume further that the noise is zero, i.e.,  $y = f_{\rho}^{*}(x)$ . Then, we have:
- Suppose  $\lambda \asymp n^{-\theta}$  for  $\theta \in (0, \beta)$ , we have
$$
\mathbb {E} \left[ \left\| \hat {f} _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} ^ {2} \mid X \right] = \tilde {\Theta} _ {\mathbb {P}} \left(n ^ {- \min  (s, 2) \theta}\right), \tag {16}
$$
where  $\tilde{\Theta}_{\mathbb{P}}$  can be replaced with  $\Theta_{\mathbb{P}}$  if  $s \neq 2$ .
- Suppose  $\lambda \asymp n^{-\theta}$  for  $\theta \geq \beta$  and assume further that  $s > 1$ . Then,
$$
\mathbb {E} \left[ \left\| \hat {f} _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} ^ {2} \mid X \right] = O _ {\mathbb {P}} ^ {\text {p o l y}} \left(n ^ {- \min  (s, 2) \beta}\right). \tag {17}
$$
Moreover, we have the information-theoretical lower rate:
$$
\sup  _ {\left\| f _ {\rho} ^ {*} \right\| _ {[ \mathcal {H} ] ^ {s}} \leq R} \mathbb {E} \left[ \left\| \hat {f} _ {\lambda} - f _ {\rho} ^ {*} \right\| _ {L ^ {2}} ^ {2} \mid X \right] = \Omega (n ^ {- s \beta}), \tag {18}
$$
where  $R > 0$  is a fixed constant.