Title: On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decay
Venue: NeurIPS 2023
Accepted: yes
Keywords: generalization

reproducing kernel Hilbert space

bias-variance trade-off
Abstract: The widely observed 'benign overfitting phenomenon' in the neural network literature raises the challenge to the `bias-variance trade-off' doctrine in the statistical learning theory.
Since the generalization ability of the 'lazy trained' over-parametrized neural network can be well approximated by that of the neural tangent kernel regression,
the curve of the excess risk (namely, the learning curve) of kernel ridge regression attracts increasing attention recently.
However, most recent arguments on the learning curve are heuristic and are based on the 'Gaussian design' assumption.
In this paper, under mild and more realistic assumptions, we rigorously provide a full characterization of the learning curve in the asymptotic sense
under a power-law decay condition of the eigenvalues of the kernel and also the target function.
The learning curve elaborates the effect and the interplay of the choice of the regularization parameter, the source condition and the noise.
In particular, our results suggest that the 'benign overfitting phenomenon' exists in over-parametrized neural networks only when the noise level is small.

On the Asymptotic Learning Curves of Kernel Ridge Regression under Power-law Decay

Abstract

The widely observed 'benign overfitting phenomenon' in the neural network literature raises the challenge to the 'bias-variance trade-off' doctrine in the statistical learning theory. Since the generalization ability of the 'lazy trained' overparametrized neural network can be well approximated by that of the neural tangent kernel regression, the curve of the excess risk (namely, the learning curve) of kernel ridge regression attracts increasing attention recently. However, most recent arguments on the learning curve are heuristic and are based on the 'Gaussian design' assumption. In this paper, under mild and more realistic assumptions, we rigorously provide a full characterization of the learning curve in the asymptotic sense under a power-law decay condition of the eigenvalues of the kernel and also the target function. The learning curve elaborates the effect and the interplay of the choice of the regularization parameter, the source condition and the noise. In particular, our results suggest that the 'benign overfitting phenomenon' exists in over-parametrized neural networks only when the noise level is small.

1 Introduction

Kernel methods, in particular kernel ridge regression (KRR), have been one of the most popular algorithms in machine learning. Its optimality under various settings has been an active topic since Caponnetto and De Vito [2007], Andreas Christmann [2008]. The renaissance of kernel methods arising from the neural tangent kernel (NTK) theory [Jacot et al., 2018], which shows that over-parametrized neural networks can be well approximated by certain kernel regression with the corresponding NTK, has posed further challenges about the interplay of generalization, regularization and noise level. For example, it has been observed empirically that over-parametrized neural networks can fit any data perfectly but also generalize well [Zhang et al., 2017], which contradicts to our traditional belief of bias-variance trade-off [Vapnik, 1999].

The aforementioned 'benign overfitting phenomenon' that overfitted neural networks generalize well attracts lots of attention recently. Researchers provide various explanations to reconcile the contradiction between it and the bias-variance trade-off principle. For example, Belkin et al. [2019] proposed the 'double descent theory' to explain why large model can generalize well; some other works (e.g., Liang and Rakhlin [2020]) argued that kernel interpolating estimators can generalize

well in high dimensional settings. In contrast to the 'benign overfitting phenomenon', several other works (e.g., Rakhlin and Zhai [2018], Li et al. [2023a]) recently showed that kernel interpolation can not generalize in traditional fixed dimension setting. In order to understand the 'benign overfitting phenomenon', it would be of great interest to characterize the learning curve: the curve of the exact order of the generalization error of a certain algorithm (e.g., KRR) varying with respect to different choices of regularization parameters.

Recently, several works (e.g., Bordelon et al. [2020], Cui et al. [2021]) depicted the learning curve of KRR under the Gaussian design assumption that the eigenfunctions (see (5)) are i.i.d. Gaussian random functions. Though it is easy to figure out that the Gaussian design assumption can not be true in most scenarios, with some heuristic arguments, Cui et al. [2021] provide a description of the learning curves of KRR with respect to the regularization, source condition and noise levels. These works offered us some insights on the learning curve of KRR which strongly suggests that the learning curve should be U-shaped if the observations are noisy or monotone decreasing if the observations are noiseless.

In this paper, we consider the learning curves of KRR under the usual settings (without the Gaussian design assumption). Under mild assumptions, we rigorously prove the asymptotic rates of the excess risk, including both upper and lower bounds. These rates show the interplay of the eigenvalue decay of the kernel, the relative smoothness of the regression function, the noise and the choice of the regularization parameter. As a result, we obtain the traditional U-shaped learning curve for the noisy observation case and a monotone decreasing learning curve for the noiseless case, providing a full picture of the generalization of KRR in the asymptotic sense. Combined with the NTK theory, our results may also suggest that 'the benign overfitting phenomenon' may not exist if one trains a very wide neural network.

1.1 Our contributions

The main contribution of this paper is that we remove the unrealistic Gaussian design assumption in previous non-rigorous works [Bordelon et al., 2020, Cui et al., 2021] and provide mathematically solid proof of the exact asymptotic rates of KRR with matching upper and lower bounds.

To be precise, let us introduce the quantities λ , the regularization parameter in (1); β , the eigenvalue decay rate in (6), which characterizes the span of the underlying reproducing kernel Hilbert space (RKHS); and s , the smoothness index in (12), describes the relative smoothness of the regression function with respect to the RKHS. Here we note that larger β implies better regularity the RKHS and also larger s also implies better relative smoothness. Then, the asymptotic rates of the generalization error (excess risk) R ( λ ) in the noisy case is roughly

where n is the number of the samples and σ 2 is the noise level. This result justifies the traditional U-shaped learning curve (see also Figure 1 on page 6) with respect to the regularization parameter.

For the technical part, we use the bias-variance decomposition and determine the exact rates of the both terms. Since the variance term was already considered in Li et al. [2023a], the main focus of this work is the bias term. Our technical contributions include:

When the regularization parameter λ is not so small, that is, λ = Ω( n -β ) , we provide sharp estimates of the asymptotic orders (Lemma 4.1) of the bias term with both upper and lower bounds. Our result holds for both the well-specified case ( s ≥ 1 ) and the mis-specified case ( s ∈ (0 , 1) ), which improves the upper bounds given in Zhang et al. [2023a].

We further show an upper bound (Lemma A.12) of the bias term in the nearly interpolating case, i.e., λ = O ( n -β ) . The upper bound is tight and matches the information-theoretic lower bound provided in Proposition 4.4.

Combining these results, we provide learning curves of KRR for both the noisy case (Theorem 3.2) and the noiseless case (Theorem 3.4). The results justify our traditional belief of the bias-variance trade-off principle.

Our new techniques can also be generalized to other settings and might be of independent interest.

1.2 Related works

The optimality of kernel ridge regression has been studied extensively [Caponnetto and De Vito, 2007, Steinwart et al., 2009, Fischer and Steinwart, 2020, Zhang et al., 2023a]. Caponnetto and De Vito [2007] provided the classical optimality result of KRR in the well-specified case and the subsequent works further considered the mis-specified case. However, these works only provided an upper bound and the worst-case (minimax) lower bound, which are not sufficient for determining the precise learning curve. In order to answer the 'benign overfitting' phenomenon [Bartlett et al., 2020, Liang and Rakhlin, 2020], several works [Rakhlin and Zhai, 2018, Buchholz, 2022, Beaglehole et al., 2022] tried to provide a lower bound for the kernel interpolation, which is a limiting case of KRR, but these works only focused on particular kernels and their techniques can hardly be generalized to provide a lower bound for KRR.

Another line of recent works considered the generalization performance of KRR under the Gaussian design assumption of the eigenfunctions [Bordelon et al., 2020, Jacot et al., 2020, Cui et al., 2021, Mallinar et al., 2022]. In particular, the learning curves of KRR was described in Bordelon et al. [2020], Cui et al. [2021], but heuristic arguments are also made in addition to the unrealistic Gaussian design assumption. Though the heuristic arguments are inspirational, a rigorous proof is indispensable if one plans to perform further investigations. In this work, we provide the first rigorous proof for most scenarios of the smoothness s , eigenvalue decay rate β , noise level σ 2 and the regularization parameter λ based on the most common/realistic assumptions.

Recently, in order to show the so-called 'saturation effect' in KRR, Li et al. [2023b] proved the exact asymptotic order of both the bias and the variance term when the regression function is very smooth and the regularization parameter λ is relatively large. Inspired by their analysis, Li et al. [2023a] showed the exact orders of the variance term. Our work further determines the orders of the bias term, completing the full learning curve or KRR.

KRR is also connected with Gaussian process regression [Kanagawa et al., 2018]. Jin et al. [2021] claimed to establish the learning curves for Gaussian process regression and thus for KRR. However, as pointed out in Zhang et al. [2023b], there is a gap in their argument. Moreover, their results are also more restrictive than ours, see Section 3.3 for a comparison.

Notations We write L p ( X , d µ ) for the Lebesgue space and sometimes abbreviate it as L p . We use asymptotic notations O ( · ) , o ( · ) , Ω( · ) and Θ( · ) , and use ˜ Θ( · ) to suppress logarithm terms. We also write a n ≍ b n for a n = Θ( b n ) . We will also use the probability versions of the asymptotic notations such as O P ( · ) . Moreover, to present the results more clearly, we denote a n = O poly ( b n ) if a n = O ( n p b n ) for any p > 0 , a n = Ω poly ( b n ) if a n = Ω( n -p b n ) for any p > 0 , a n = Θ poly ( b n ) if a n = O poly ( b n ) , and a n = Ω poly ( b n ) ; and we add a subscript P for their probability versions.

2 Preliminaries

Let X ⊂ R d be compact and ρ be a probability measure on X × R , whose marginal distribution on X is denoted by µ . Suppose that we are given n i.i.d. samples ( x 1 , y 1 ) , . . . , ( x n , y n ) from ρ . Let k be a continuous positive definite kernel k over X and H be the separable reproducing kernel Hilbert space (RKHS) associated with k . Then, kernel ridge regression (KRR) obtains the regressor ˆ f λ via the following convex optimization problem

where λ > 0 is the regularization parameter. Let us denote X = ( x 1 , . . . , x n ) and y = ( y 1 , . . . , y n ) T . A closed form of (1) can be provided by the representer theorem [Andreas Christmann, 2008]:

n × n

In terms of the generalization performance of ˆ f λ , we consider the excess risk with respect to the squared loss

where f ∗ ρ ( x ) := E ρ [ y | x ] is the conditional expectation and is also referred to as the regression function. We aim to provide asymptotic orders of (3) with respect to n .

2.1 The integral operator

We will introduce the integral operator, which is crucial for the analysis, as the previous works [Caponnetto and De Vito, 2007, Lin et al., 2018]. Denote by µ the marginal probability measure of ρ on X . Since k is continuous and X is compact, let us assume sup x ∈X k ( x, x ) ≤ κ 2 . Then, it is known [Andreas Christmann, 2008, Steinwart and Scovel, 2012] that we have the natural embedding S µ : H → L 2 , which is a Hilbert-Schmidt operator with Hilbert-Schmidt norm ∥ S µ ∥ HS ≤ κ . Let S ∗ µ : L 2 →H be the adjoint operator of S µ and T = S µ S ∗ µ : L 2 → L 2 . Then, it is easy to show that T is an integral operator given by

and it is self-adjoint, positive and trace-class (thus compact) with trace norm ∥ T ∥ 1 ≤ κ 2 [Caponnetto and De Vito, 2007, Steinwart and Scovel, 2012]. Moreover, the spectral theorem of compact selfadjoint operators and Mercer's theorem [Steinwart and Scovel, 2012] yield the decompositions

where N ⊆ N is an index set, { λ i } i ∈ N is the set of positive eigenvalues of T in descending order, and e i is the corresponding eigenfunction. Furthermore, { e i } i ∈ N forms an orthonormal basis of Ran S µ ⊆ L 2 and { λ 1 / 2 i e i } i ∈ N forms an orthonormal basis of Ran S ∗ µ ⊆ H .

The eigenvalues λ i actually characterize the span of the RKHS and the interplay between H and µ . Since we are interested in the infinite-dimensional case, we will assume N = N and assume the following polynomial eigenvalue decay as in the literature [Caponnetto and De Vito, 2007, Fischer and Steinwart, 2020, Li et al., 2023b], which is also referred to as the capacity condition or effective dimension condition. Larger β implies better regularity of the functions in the RKHS.

Assumption 1 (Eigenvalue decay) . There is some β > 1 and constants c β , C β > 0 such that

where λ i is the eigenvalue of T defined in (5).

Such a polynomial decay is satisfied for the well-known Sobolev kernel [Fischer and Steinwart, 2020], Laplace kernel and, of most interest, neural tangent kernels for fully-connected multilayer neural networks [Bietti and Mairal, 2019, Bietti and Bach, 2020, Lai et al., 2023].

2.2 The embedding index of an RKHS

We will consider the embedding index of an RKHS to sharpen our analysis. Let us first define the fractional power T s : L 2 → L 2 for s ≥ 0 by

Then, the interpolation space [Steinwart and Scovel, 2012, Fischer and Steinwart, 2020, Li et al., 2023b] [ H ] s is define by

with the norm ∥ ∥ ∥ ∑ i ∈ N a i λ s/ 2 i e i ∥ ∥ ∥ [ H ] s = (∑ i ∈ N a 2 i ) 1 / 2 . One may easily verify that [ H ] s is also a separable Hilbert space with an orthonormal basis { λ s/ 2 i e i } i ∈ N . Moreover, it is clear that [ H ] 0 = Ran S µ ⊆ L 2 and [ H ] 1 = Ran S ∗ µ ⊆ H . It can also be shown that if s 1 > s 2 ≥ 0 , the inclusions [ H ] s 1 ↪ → [ H ] s 2 are compact [Steinwart and Scovel, 2012].

Now, we say H has an embedding property of order α ∈ (0 , 1] if [ H ] α can be continuously embedded into L ∞ ( X , d µ ) , that is, the operator norm

Moreover, Fischer and Steinwart [2020, Theorem 9] shows that

Therefore, since sup x ∈X k ( x, x ) ≤ κ 2 , we know that (9) always holds for α = 1 . By the inclusion relation of interpolation spaces, it is clear that if H has the embedding property of order α , then it has the embedding properties of order α ′ for any α ′ ≥ α . Consequently, we may introduce the following definition [Zhang et al., 2023b]:

Definition 2.1. The embedding index α 0 of an RKHS H is defined by

It is shown in Fischer and Steinwart [2020, Lemma 10] that α 0 ≥ β and we assume the equality holds as the following assumption.

Assumption 2 (Embedding index) . The embedding index α 0 = 1 /β , where β is the eigenvalue decay in (6).

Lots of the usual RKHSs satisfy this embedding index condition. It is shown in Steinwart et al. [2009] that Assumption 2 holds if the eigenfunctions are uniformly bounded, namely sup i ∈ N ∥ e i ∥ L ∞ < ∞ . Moreover, Assumption 2 also holds for the Sobolev RKHSs, RKHSs associated with periodic translation invariant kernels and RKHSs associated with dot-product kernels on spheres, see Zhang et al. [2023a, Section 4].

3 Main Results

Before presenting our main results, we have to introduce a source condition on the regression function. Since we will establish both precise learning rates, we have to characterize the exact smoothness order of f ∗ ρ rather than merely assume f ∗ ρ belongs to some interpolation space [ H ] s .

Assumption 3 (Source condition) . There are some s > 0 and a sequence ( a i ) i ≥ 1 such that

and 0 < c ≤ | a i | ≤ C for some constants c, C .

Remark 3.1. Assumption 3 is also considered in Cui et al. [2021, Eq. (8)] and a slightly weaker version of it is given in Jin et al. [2021, Assumption 5]. We only consider this simple form since there is no essential difference in the proof to consider the weaker version. From the definition (8) we can see that Assumption 3 implies f ∗ ρ ∈ [ H ] t for any t < s but f ∗ ρ / ∈ [ H ] s .

3.1 Noisy case

Let us first consider the noisy case with the following assumption:

Assumption 4 (Noise) . We assume

0

s

+1

n

s

2

n

1

2

error rate

(

2

)

Var

0

s

+1

n

s

1

2

error rate

(

2

)

Noiseless case

Noisy case

0

2

0

Overfitting

(

n

(1

/

)

)

Underfitting

(

n

s

)

Interpolating,

(1)

Saturation

(

n

2

)

op=

s

+1

op = 2 +1

Optimal

-1

0 (constant noise)

s

s

+1

Overfitting

(

2

n

(1

/

)

)

Underfitting

(

n

s

)

Underfitting

(

n

s

)

Interpolating,

(1)

(

n

s

)

Effectively

noisy

Effectively

noiseless

op

= (

+1)

s

+1

op=

Optimal

s

For technical reason, we have to further assume the kernel to be Hölder-continuous, which is first in introduced in Li et al. [2023b]. This assumption is satisfied for the Laplace kernel, Sobolev kernels and neural tangent kernels.

Assumption 5. The kernel k is Hölder-continuous, that is, there exists some p ∈ (0 , 1] and L > 0 such that

Theorem 3.2. Under Assumptions 1-5, suppose λ ≍ n -θ for θ > 0 . Then,

̸

where ˜ Θ P can be replaced with Θ P for the first case if s = 2 .

Remark 3.3. The two terms in the first case in Theorem 3.2 actually correspond to the bias and the variance term respectively. Balancing the two terms, we find the optimal regularization is θ op = β ˜ sβ +1 and the optimal rate is ˜ sβ ˜ sβ +1 , where ˜ s = min( s, 2) , which recovers the classical optimal rate results [Caponnetto and De Vito, 2007]. Moreover, while we treat σ 2 as fixed for simplicity, we can also allow σ 2 to vary with n . Then, we can recover the results in Cui et al. [2021].

3.2 Noiseless case

Theorem 3.4. Under Assumptions 1-3, assume further that the noise is zero, i.e., y = f ∗ ρ ( x ) . Then, we have:

Suppose λ ≍ n -θ for θ ∈ (0 , β ) , we have

̸

where ˜ Θ P can be replaced with Θ P if s = 2 .

Suppose λ ≍ n -θ for θ ≥ β and assume further that s > 1 . Then,

Moreover, we have the information-theoretical lower rate:

where R > 0 is a fixed constant.

Remark 3.5. Theorem 3.4 shows that the generalization error of KRR in the noiseless case is monotone decreasing when θ increases and reaches the optimal rate n -β when θ ≥ β if s ≤ 2 . Since the case θ →∞ corresponds to kernel interpolation, our result implies that kernel interpolation is optimal when there is no noise. In contrast, as shown in Theorem 3.2 (or Li et al. [2023a]), kernel interpolation can not generalize in the noisy case. For the case s > 2 , the KRR method suffers from saturation and the resulting convergence rate is limited to n -2 β , while the possible lower rate is n -sβ .

3.3 Discussion

Our results provide a full picture of the generalization of KRR, which is in accordance with our traditional belief of the bias-variance trade-off principle: the generalization error is a U-shaped curve with respect to the regularization parameter λ in the noisy case and is monotone decreasing in the noiseless case. See Figure 1 on page 6 for an illustration.

Our rates coincide with the upper rates in the traditional KRR literature [Caponnetto and De Vito, 2007, Fischer and Steinwart, 2020]. Moreover, our results also recover the learning curves in Cui et al. [2021], but we do not need the strong assumption of Gaussian design eigenfunctions as in Cui et al. [2021], which may not be true in most cases. Our assumptions are mild and hold for a large class of kernels including the Sobolev kernels and the neural tangent kernels (NTK) on spheres.

Our results are based on the bias-variance decomposition and determining the rates for each term respectively. In the proof of Li et al. [2023b], they determined the rates of the variance term under the condition that θ < 1 2 and that of the bias term when s ≥ 2 and θ < 1 . The subsequent work Li et al. [2023a] proved the rates of the variance term when θ < β and provided a near constant lower bound for θ ≥ β . Considering the counterpart, our works further prove the rates of the bias term, which finally enables us to determine the complete learning curve of KRR.

The connection between KRR and Gaussian process regression also results in the connection between their learning curves. Jin et al. [2021] claimed to show learning curves for Gaussian process regression. However, regardless of the gap in their proof as pointed out in Zhang et al. [2023b], their results are more restrictive than ours. Considering a boundedness assumption of the eigenfunctions that ∥ e i ∥ ∞ ≤ Ci τ for some τ ≥ 0 , they could only cover the regime of θ < β/ (1 + 2 τ ) . Moreover, to approach the θ = β regime for the Ω(1) bound in the noisy case or the optimal rate in noiseless case, they have to require τ = 0 , that is, the eigenfunctions are uniformly bounded, but it is not true for some kernels such as dot-product kernels on spheres (and thus for NTK) since in general spherical harmonics are not uniformly bounded. In contrast, our embedding index assumption still holds in this case.

4 Proof sketch

We first introduce the following sample versions of the auxiliary integral operators, which are commonly used in the related literature [Caponnetto and De Vito, 2007, Fischer and Steinwart, 2020, Li et al., 2023b]. We define the sampling operator K x : R →H by K x y = yk ( x, · ) , whose adjoint K ∗ x : H → R is given by K ∗ x f = f ( x ) . The sample covariance operator T X : H → H is defined by

and the sample basis function is g Z := 1 n ∑ n i =1 K x i y i ∈ H . As shown in Caponnetto and De Vito [2007], the operator form of KRR writes

Then, the traditional bias-variance decomposition [Li et al., 2023b, Zhang et al., 2023a] yields

Let us further define and

where

4.1 The noisy case

To prove the desired result, we have to establish the asymptotic orders of both Bias 2 ( λ ) and Var ( λ ) . We first prove the asymptotic order of Bias 2 ( λ ) as one of our technical contributions. As far as we know, we are the first to provide such a lower bound in (25).

Lemma 4.1. Under Assumptions 1,2,3, suppose λ ≍ n -θ for θ ∈ (0 , β ) . Then,

̸

where ˜ Θ P can be replaced with Θ P if s = 2 .

Proof sketch of Lemma 4.1 . Denote ˜ s = min( s, 2) . We first introduce the regularized regression function f λ := T ( T + λ ) -1 f ∗ ρ and triangle inequality implies

There is no randomness in the first term and we can use the expansion (12) and (5) to show that ∥ ∥ f λ -f ∗ ρ ∥ ∥ L 2 = ˜ Θ ( n -˜ sθ ) . Then, we have to prove the error term ∥ ∥ ∥ ˜ f λ -f λ ∥ ∥ ∥ L 2 to be infinitesimal with respect to the main term, which is the main difficulty since it requires a refined analysis. Previous work only consider the case θ = β ˜ sβ +1 (corresponding to the optimal regularization) and show an O ( n -˜ sθ ) bound rather than the o ( n -˜ sθ ) bound that we require. For the proof, we (1) apply the concentration techniques in Fischer and Steinwart [2020]; (2) consider the L q -embedding property in Zhang et al. [2023a] for the mis-specified case when s is small; (3) sharpen the estimation by exploiting the embedding property α 0 = 1 /β and θ < β . For the detail, see Section 2.2 in the supplementary material.

The variance term has been analyzed in Li et al. [2023a]. We present the following proposition as a combination of Proposition 5.3 and Theorem 5.10 in Li et al. [2023a].

Proposition 4.2. Under Assumptions 1-5, suppose that λ ≍ n -θ . Then,

4.2 The noiseless case

For the noiseless case, the variance term vanishes in (23), and thus we only need to consider the bias term. Since we have already established the estimation for large λ in Lemma 4.1, we focus on the case of small λ .

Lemma 4.3. Under Assumptions 1,2,3, assume further s > 1 . Suppose λ ≍ n -θ for θ ≥ β . Then,

Proof sketch of Lemma A.12 . Intuitively, we hope to bound Bias 2 ( λ ) with Bias 2 ( ˜ λ ) for ˜ λ > λ such that concentration still works. However, we can not directly derive no monotone property of Bias ( λ ) . Nevertheless, since f ∗ ρ ∈ H when s > 1 , the bias term can be written as

Then, by operator calculus we can show that

reducing λ to ˜ λ . Now, we can replace T X with T using concentration results and derive the desired upper bound.

The following proposition shows that the upper bound in Lemma A.12 matches the informationtheoretical lower bound. The proof follows idea of the minimax principle [Micchelli and Wahba, 1979] and is deferred to the supplementary material.

Proposition 4.4. Suppose Assumption 1 holds and s ≥ 1 . For any X = ( x 1 , . . . , x n ) , we have

where we note that here Bias ( λ ) is viewed as a function depending also on f ∗ ρ and X .

5 Experiments

Lots of numerical experiments on both synthetic data and real data are done to study to learning curves of KRR [Li et al., 2023b, Cui et al., 2021]. In this section, we consider numerical experiments on a toy model to verify our theory.

Let us consider the kernel k ( x, y ) = min( x, y ) and x ∼ U [0 , 1] . Then, the corresponding RKHS is [Wainwright, 2019]

and the eigenvalue decay rate β = 2 . Moreover, the eigensystem of k is known to be λ i = ( 2 i -1 2 π ) -2 and e i ( x ) = √ 2 sin ( 2 i -1 2 πx ) , which allows us to directly compute the smoothness of certain functions. For some f ∗ , we generate data from the model y = f ∗ ( x ) + ε where ε ∼ N (0 , 0 . 05) and perform KRR with λ = cn -θ for different θ 's with some fixed constant c . Then, we numerically compute the variance, bias and excess risk by Simpson's formula with N ≫ n nodes. Repeating the experiment for n ranged in 1000 to 5000, we can estimate the convergence rate r by a logarithmic least-squares log err = r log n + b on the values (variance, bias and excess risk). The results are collected in Table 1 on page 10. It can be seen that the resulting values basically match the theoretical values and we conclude that our theory is supported by the experiments. For more experiments and more details, we refer to the supplementary material.

6 Conclusion

In this paper, we prove rigorously the learning curves of KRR, showing the interplay of the eigenvalue decay of the kernel, the relative smoothness of the regression function, the noise and the choice of the regularization parameter. The results justify our traditional bias-variance trade-off principle and provide a full picture of the generalization performance of KRR. These results will help us better understand the generalization mystery of neural networks.

As for future works, we notice that for the nearly interpolating regime when θ ≥ β , there are still some missing parts due to technical limitations. We expect that further analysis will prove the exact orders of the variance term like that given in Mallinar et al. [2022] under the Gaussian design assumption. We also hypothesize that Lemma A.12 still holds in the mis-specified case ( s < 1 ).

Acknowledgments and Disclosure of Funding

This work is supported in part by the Beijing Natural Science Foundation (Grant Z190001) and National Natural Science Foundation of China (Grant 11971257).

References

Ingo Steinwart (auth.) Andreas Christmann. Support Vector Machines . Information Science and Statistics. Springer-Verlag New York, New York, NY, 1 edition, 2008. ISBN 0-387-77242-1 0-387-77241-3 978-0-387-77241-7 978-0-387-77242-4. doi: 10.1007/978-0-387-77242-4.

Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, December 2020. ISSN 0027-8424, 1091-6490. doi: 10.1073/pnas.1907378117.

Daniel Beaglehole, Mikhail Belkin, and Parthe Pandit. Kernel ridgeless regression is inconsistent in low dimensions. (arXiv:2205.13525), June 2022. doi: 10.48550/arXiv.2205.13525.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Alberto Bietti and Francis Bach. Deep equals shallow for ReLU networks in kernel regimes. arXiv preprint arXiv:2009.14397 , 2020.

Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In Advances in Neural Information Processing Systems , volume 32, 2019.

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Proceedings of the 37th International Conference on Machine Learning , pages 1024-1034. PMLR, November 2020.

Simon Buchholz. Kernel interpolation in Sobolev spaces is not consistent in low dimensions. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 3410-3440. PMLR, July 2022.

Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 7(3):331-368, 2007. doi: 10.1007/ s10208-006-0196-8.

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems , 34:10131-10143, 2021.

Simon-Raphael Fischer and Ingo Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. Journal of Machine Learning Research , 21:205:1-205:38, 2020.

Junichi Fujii, Masatoshi Fujii, Takayuki Furuta, and Ritsuo Nakamoto. Norm inequalities equivalent to Heinz inequality. Proceedings of the American Mathematical Society , 118(3):827-830, 1993.

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Arthur Jacot, Berfin ¸ Sim¸ sek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Kernel alignment risk estimator: Risk prediction from training data. June 2020. doi: 10.48550/arXiv.2006. 09796.

Hui Jin, Pradeep Kr Banerjee, and Guido Montúfar. Learning curves for Gaussian process regression with power-law priors and targets. (arXiv:2110.12231), November 2021.

Motonobu Kanagawa, Philipp Hennig, Dino Sejdinovic, and Bharath K. Sriperumbudur. Gaussian processes and kernel methods: A review on connections and equivalences. arXiv preprint arXiv:1807.02582 , 2018.

Jianfa Lai, Manyun Xu, Rui Chen, and Qian Lin. Generalization ability of wide neural networks on R. (arXiv:2302.05933), February 2023. doi: 10.48550/arXiv.2302.05933.

Yicheng Li, Haobo Zhang, and Qian Lin. Kernel interpolation generalizes poorly. (arXiv:2303.15809), March 2023a. doi: 10.48550/arXiv.2303.15809.

Yicheng Li, Haobo Zhang, and Qian Lin. On the saturation effect of kernel ridge regression. In International Conference on Learning Representations , February 2023b.

Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel "ridgeless" regression can generalize. The Annals of Statistics , 48(3), June 2020. ISSN 0090-5364. doi: 10.1214/19-AOS1849.

Junhong Lin, Alessandro Rudi, L. Rosasco, and V. Cevher. Optimal rates for spectral algorithms with least-squares regression over Hilbert spaces. Applied and Computational Harmonic Analysis , 48: 868-890, 2018. doi: 10.1016/j.acha.2018.09.009.

Shao-Bo Lin, Xiangyu Chang, and Xingping Sun. Kernel interpolation of high dimensional scattered data. (arXiv:2009.01514), September 2021.

Neil Mallinar, James B. Simon, Amirhesam Abedsoltan, Parthe Pandit, Mikhail Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: A taxonomy of overfitting. (arXiv:2207.06569), July 2022. doi: 10.48550/arXiv.2207.06569.

Charles A. Micchelli and Grace Wahba. Design problems for optimal surface interpolation. Technical report, Wisconsin Univ-Madison Dept of Statistics, 1979.

Alexander Rakhlin and Xiyu Zhai. Consistency of interpolation with Laplace kernels is a highdimensional phenomenon. (arXiv:1812.11167), December 2018.

Barry Simon. Operator Theory . American Mathematical Society, Providence, Rhode Island, November 2015. ISBN 978-1-4704-1103-9 978-1-4704-2763-4. doi: 10.1090/simon/004.

Ingo Steinwart and C. Scovel. Mercer's theorem on general domains: On the interaction between measures, kernels, and RKHSs. Constructive Approximation , 35(3):363-417, 2012. doi: 10.1007/ S00365-012-9153-3.

Ingo Steinwart, D. Hush, and C. Scovel. Optimal rates for regularized least squares regression. In COLT , pages 79-93, 2009.

Vladimir Vapnik. The Nature of Statistical Learning Theory . Springer science & business media, 1999.

Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. doi: 10.1017/9781108627771.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. (arXiv:1611.03530), February 2017.

Haobo Zhang, Yicheng Li, and Qian Lin. On the optimality of misspecified spectral algorithms. (arXiv:2303.14942), March 2023a. doi: 10.48550/arXiv.2303.14942.

Haobo Zhang, Yicheng Li, Weihao Lu, and Qian Lin. On the optimality of misspecified kernel ridge regression. In International Conference on Machine Learning , 2023b.

A Detailed proofs

The first step of the proof is the traditional bias-variance decomposition. Let us further define

and

Recalling (20), we have

so that

Taking expectation over the noise ϵ conditioned on X , since ε | x are independent noise with mean 0 and variance σ 2 , we have

where

A.1 The variance term

Theorem A.1. Under Assumptions 1-5, suppose that λ ≍ n -θ . Then,

The computation in Li et al. [2023b] shows that

Then, Theorem A.1 directly follows from Proposition 5.3 and Theorem 5.10 in Li et al. [2023a].

A.2 The bias term

Theorem A.2. Under Assumptions 1,2,3, suppose λ ≍ n -θ for θ ∈ (0 , β ) . Then,

̸

where ˜ Θ P can be replaced with Θ P if s = 2 .

Let us define the regularized version of the regression function

Then, the triangle inequality implies that

Then, the proof of Theorem A.2 is the combination of the following Lemma A.3 (with γ = 0 ) and Lemma A.4, showing that the main term ∥ ∥ f λ -f ∗ ρ ∥ ∥ L 2 = ˜ Θ P ( n -min( s, 2) θ/ 2 ) and the error term

Lemma A.3. Under Assumptions 1 and 3, for any 0 ≤ γ < s , we have

Proof. From the definition of interpolating norms, letting p = ( s -γ ) / 2 , we have

Then result then follows by applying Proposition B.2 for the last series.

The following lemma shows the error term in (36) is infinitesimal, whose proof relies on fine-grained concentration results established in Section A.3.

Lemma A.4. Under Assumptions 1-3. Suppose λ ≍ n -θ with θ ∈ (0 , β ) , then

Proof. We begin with

From operator calculus we know ∥ ∥ ∥ T 1 2 T -1 2 λ ∥ ∥ ∥ ≤ 1 . Moreover, since θ < β and the embedding index α 0 = 1 /β , by Lemma B.5 we get ∥ ∥ ∥ T 1 2 λ T -1 Xλ T 1 2 λ ∥ ∥ ∥ ≤ 3 with high probability as long as n is sufficiently large. For the last term in (40), we have

Therefore, Lemma A.5 and Lemma A.10 show that

for both s > α 0 and s ≤ α 0 cases.

A.3 Approximation results

Let us further denote

Then, it is easy to check that

The following lemma deals with the easy case when s > α 0 .

Lemma A.5. Suppose Assumptions 1-3 hold and s > α 0 . Let λ ≍ n -θ with θ ∈ (0 , β ) and δ ∈ (0 , 1) . Then, for α > α 0 = β -1 being sufficiently close, it holds with probability at least 1 -δ that

where ˜ s = min( s, 2) . Consequently,

Before proving Lemma A.5, we have to introduce the following proposition bounding the RKHS norm of the regularized basis function, which is a part of Li et al. [2023a, Corollary 5.6].

Proposition A.6. Suppose H has embedding index α 0 . Then for any α > α 0 ,

Proof of Lemma A.5 . To use Bernstein inequality in Lemma B.4, let us bound the m -th moment of ξ ( x ) :

The first term in (45) is bounded through (44). For the second term, since s > α 0 , using the embedding condition and Lemma A.3, we have

where we notice that min( s -α, 2) = min( s, 2 + α ) -α ≥ ˜ s -α for the last inequality. Moreover, Lemma A.3 also implies

Plugging in these estimations in (45), we get

The proof is then complete by Lemma B.4.

The case of s ≤ α 0 is more difficult. We will use the truncation technique introduced in Zhang et al. [2023a]. The following lemma can be proven similarly to Lemma A.3.

Lemma A.7. Under Assumptions 1 and 3, for any 0 ≤ γ < s +2 , we have

Proof. Simply notice that

where p = ( s +2 -γ ) / 2 . Then we can apply Proposition B.2.

Then, we are able to show the following concentration result about the truncated ξ i 's, whose proof resembles that of Lemma A.5.

Lemma A.8. Suppose Assumptions 1-3 hold and s ≤ α 0 . Let λ ≍ n -θ with θ ∈ (0 , β ) and δ ∈ (0 , 1) . For any t > 0 , denote Ω t = { x ∈ X : ∣ ∣ f ∗ ρ ( x ) ∣ ∣ ≤ t } and ¯ ξ ( x ) = ξ ( x ) 1 { x ∈ Ω t } . Then, for α > α 0 = β -1 being sufficiently close, it holds with probability at least 1 -δ that

Consequently, if t ≍ n l with l < 1 -α + s 2 θ , we have

Proof. We follow the same routine of the proof of Lemma A.5 and obtain (45) with ξ replaced with ¯ ξ . The only difference is that we have to control

where we apply Lemma A.7 at the second inequality. Then, (46) changes to

and the rest follows.

To bound the extra error terms caused by truncation, we have to use the following proposition about the L q embedding of the RKHS [Zhang et al., 2023a, Theorem 5].

Proposition A.9. Under Assumption 2, for any 0 < s ≤ α 0 and α > α 0 , we have embedding

Lemma A.10. Suppose Assumptions 1-3 hold and s ≤ α 0 . Let λ ≍ n -θ with θ ∈ (0 , β ) and δ ∈ (0 , 1) . Then

Proof. We will choose t = n l for some l that will be determined later and choose some α > α 0 being sufficiently close. Using the same notations as in (49), we decompose

The first term in (49) is already bounded by (49) if l < 1 -α + s 2 θ . To bound the second term in (52), we notice that

Since Markov's inequality yields

where q = 2 α α -s , we get

So the second term vanishes with high probability as long as l > 1 /q .

For the third term in (52), using (44), we get

Consequently, if l > αθ q , then and thus

Finally, the three requirements of l are

where q = 2 α α -s . Since θ < β = α -1 0 , we can choose α sufficiently close to α 0 such that θα < 1 . Then,

showing that we can choose some l satisfying all the requirements and the proof is finish.

A.4 The noiseless case

The case when λ = n -θ for θ < β is already covered in Theorem A.2. For the case θ ≥ β , the approximation Lemma B.5 no longer holds, and we must reduce it to the former case. However, there is no direct monotone property of Bias ( λ ) . Nevertheless, we have the following monotone relation about the operator norms, whose proof utilizes the idea in Lin et al. [2021, Proposition 6.1] with modification.

Proposition A.11. Let ψ λ = λ ( T X + λ ) -1 ∈ B ( H ) . Suppose λ 1 ≤ λ 2 , then for any s, p ≥ 0 ,

Proof. Let us denote by ⪯ the partial order induced by positive operators. Since the function λ ↦→ λ z + λ is monotone decreasing with λ , we obtain ψ 2 p λ 1 ⪯ ψ 2 p λ 2 , which further implies

Then, since ∥ A ∥ 2 = ∥ AA ∗ ∥ , we have

and the equality in (54) is proven by ∥ A ∥ = ∥ A ∗ ∥ .

Lemma A.12. Under Assumption 1 ,2,3, assume further s > 1 . Suppose λ ≍ n -θ for θ ≥ β . Then,

,

Proof. Since f ∗ ρ is given in (12) and s > 1 , we have f ∗ ρ ∈ [ H ] t for 1 ≤ t < s . In particular, f ∗ ρ ∈ H , so the bias term can also be written as

Moreover, from the construction (8) of [ H ] t , we may assume f ∗ ρ = T t/ 2 g for some g ∈ L 2 with ∥ g ∥ L 2 ≤ C , and restrict further that t ≤ 2 . Let ˜ λ ≍ n -l for l ∈ (0 , β ) . Then, using the same notation in Proposition A.11, we have

where we use Lemma B.6 for the last inequality. Finally, since ˜ λ ≍ n -l for l < β , Lemma B.5 implies that with high probability we have

Therefore, we obtain

Since t < min( s, 2) and l < β can arbitrarily close, we conclude (55).

Proof of Proposition 4.4 . Let us denote F = { f : ∥ ∥ f ∗ ρ ∥ ∥ [ H ] s ≤ R } for convenience. Since f ∗ ρ ∈ H , the bias term can be given by

for a linear operator L X = T X ( T X + λ ) -1 on H . Then,

where in (a) we use the relation between the interpolation spaces and in (b) we use the fact that and ∥ A -B ∥ ≥ λ n +1 ( A ) for any operator B with rank at most n (see, for example, Simon [2015, Section 3.5]).

B Auxiliary results

Proposition B.1. Let

Then,

If α = 0 , then f ( z ) is monotone decreasing.

If α ∈ (0 , 1) , then f ( z ) is monotone increasing in [0 , αλ 1 -α ] , and decreasing in [ αλ 1 -α , + ∞ ) . Consequently, f ( z ) ≤ λ α -1 .

If α ≥ 1 , then f ( z ) monotone increasing on [0 , + ∞ ) .

Proof. We simply notice that and hence

Proposition B.2. Suppose c β i -β ≤ λ i ≤ C β i -β and p > 0 , then as λ → 0 , we have

Proof. We first consider the case when p < 1 . Since c β i -β ≤ λ i ≤ C β i -β , from Proposition B.1, letting q = p 1 -p , we have

Therefore,

For S 1 , noticing C β i -β > qλ implies i < ( qλ/C β ) -1 /β , we have

For S 2 , using Proposition B.1 again we have

where we note that the last integral is bounded above by a constant. Therefore, we conclude that ∥ ∥ f λ -f ∗ ρ ∥ ∥ 2 [ H ] γ ≤ Cλ 2 p -2 . For the lower bound, if C β i -β ≤ qλ , we have

where we note that the last integral is independent of λ . For the case p = 1 , by Proposition B.1, we have

For the lower bound, we have

For the case p > 1 , by Proposition B.1, we have

since the last series is summable. The lower bound is derived by

Proposition B.3. Under Assumption 1, for any p ≥ 1 , we have

Proof. Since c i -β ≤ λ i ≤ Ci -β , we have

for some constant C . The lower bound is similar.

The following inequality about vector-valued random variables is well-known in the literature [Caponnetto and De Vito, 2007].

Lemma B.4. Let H be a real separable Hilbert space. Let ξ, ξ 1 , . . . , ξ n be i.i.d. random variables taking values in H . Assume that

Then for fixed δ ∈ (0 , 1) , one has

Particularly, a sufficient condition for (59) is

The following concentration result has been shown in Fischer and Steinwart [2020], Zhang et al. [2023a]. We use the form in Li et al. [2023a, Proposition 5.8] for convenience, see also Zhang et al. [2023a, Lemma 12].

Lemma B.5. Suppose H has embedding index α 0 and Assumption 1 holds. Let λ = λ ( n ) → 0 satisfy λ = Ω ( n -1 /α 0 + p ) for some p > 0 and fix arbitrary α ∈ ( α 0 , α 0 + p ) . Then, for all δ ∈ (0 , 1) , when n is sufficiently large, with probability at least 1 -δ ,

where C > 0 is a constant no depending on δ, n, α , and we also have

The following operator inequality[Fujii et al., 1993] will be used in our proofs.

Lemma B.6 (Cordes' Inequality) . Let A,B be two positive semi-definite bounded linear operators on separable Hilbert space H . Then

The following lemma is a consequence of the fact that x r is operator monotone when r ∈ (0 , 1] and is Lipschitz when r > 1 , see Zhang et al. [2023a, Lemma 35] or Lin et al. [2018, Lemma 5.8].

Lemma B.7. Suppose that A and B are two positive self-adjoint operators on some Hilbert space, then

for r ∈ (0 , 1] , we have

for r ≥ 1 , denote c = max( ∥ A ∥ , ∥ B ∥ ) , we have

C Experiments

C.1 Details of experiments in the main text

Recall that in the experiments section of the main text, we considered the kernel k ( x, y ) = min( x, y ) and x ∼ U [0 , 1] . We know the eigensystem of k that λ i = ( 2 i -1 2 π ) -2 and e i ( x ) = √ 2 sin ( 2 i -1 2 πx ) . For the three target functions used in the experiments, simple calculation shows that the relative smoothness (source condition) of cos(2 πx ) , sin(2 πx ) , sin ( 3 2 πx ) are 0 . 5 , 1 . 5 , ∞ respectively.

For some f ∗ , we generate data from the model y = f ∗ ( x ) + ε where ε ∼ N (0 , 0 . 05) and perform KRR with λ = cn -θ for different θ 's with some fixed constant c . Then, we numerically compute the variance, bias and excess risk by Simpson's formula with N ≫ n nodes. Repeating the experiment for n ranged in 1000 to 5000 with an increment of 100, we can estimate the convergence rate r by a logarithmic least-squares log err = r log n + b on the resulting values (variance, bias and excess risk). Figure 2 on page 22 shows the corresponding curves of the results in Table 1 in the main text. Note that for each setting, we tried different c 's in the regularization parameter λ = cn -θ and show the curves under the best choice of c ( c = 0 . 005 ).

1000

5000

2×10 3

3×10 3

4×10 3

n

5.5

5.0

4.5

4.0

3.5

3.0

=0.2. r=-0.90

=0.4. r=-0.80

=0.5. r=-0.75

=1.0. r=-0.49

1000

5000

2×10 3

3×10 3

4×10 3

n

4.5

4.0

3.5

3.0

2.5

2.0

log bias

Bias;  f=cos2pix

=0.2. r=-0.13

=0.4. r=-0.22

=0.5. r=-0.26

=1.0. r=-0.54

=2.0. r=-1.05

=3.0. r=-1.05

1000

5000

2×10 3

3×10 3

4×10 3

n

10

8

6

4

log bias

Bias;  f=sin2pix

=0.4. r=-0.68

=0.5. r=-0.84

=1.0. r=-1.69

=2.0. r=-3.26

=3.0. r=-3.26

1000

5000

2×10 3

3×10 3

4×10 3

n

12

10

8

6

4

log bias

=0.2. r=-0.40

=0.4. r=-0.82

=1.0. r=-2.21

=2.0. r=-3.99

=3.0. r=-3.98

1000

5000

2×10 3

3×10 3

4×10 3

n

3.2

3.0

2.8

2.6

2.4

2.2

2.0

1.8

1.6

log excess risk

Excess risk;  f=cos2pix

=0.2. r=-0.13

=0.4. r=-0.22

=0.5. r=-0.26

=1.0. r=-0.52

=2.0. r=-0.09

=3.0. r=-0.09

1000

5000

2×10 3

3×10 3

4×10 3

n

4.50

4.25

4.00

3.75

3.50

3.25

3.00

2.75

log excess risk

Excess risk;  f=sin2pix

=0.4. r=-0.69

=0.5. r=-0.79

=1.0. r=-0.49

1000

5000

2×10 3

3×10 3

4×10 3

n

5.0

4.5

4.0

3.5

3.0

log excess risk

Excess risk;  f=eigen1

=0.2. r=-0.42

=0.4. r=-0.81

=0.5. r=-0.77

=1.0. r=-0.49

C.2 Learning curves with different noises

Cui et al. [2021] discussed the 'crossover from the noiseless to noisy regime' and shown the interaction between the magnitude of noise and the sample size. As discussed in Remark 3.2 in the main text, our theory can also reflect this interaction. In Figure 3 on page 23, we exhibit the learning curves with different magnitudes of noises and visualize this interaction. Note that in the following the sample size is chosen as 10 , 20 , · · · , 100 , 120 , · · · , 1000 , 1100 , · · · , 5000 , and we use the same kernel and data generation process as before. We repeat the experiments for 100 times for each sample size and present the average excess risk.

10

5000

n

3.0

2.5

2.0

1.5

1.0

log excess risk

Excess risk;

=1.0;  f=cos2pix

=0.01

=0.02

=0.05

=0.08

=0.10

=0.15

=0.20

10

5000

n

4.0

3.5

3.0

2.5

2.0

1.5

1.0

log excess risk

Excess risk;

=2.0;  f=cos2pix

=0.01

=0.02

=0.05

=0.08

=0.10

=0.15

=0.20

10

5000

n

5.5

5.0

4.5

4.0

3.5

3.0

2.5

2.0

1.5

log excess risk

Excess risk;

=1.0;  f=sin2pix

=0.01

=0.02

=0.05

=0.08

=0.10

=0.15

=0.20

10

5000

n

4.0

3.5

3.0

2.5

2.0

1.5

log excess risk

Excess risk;

=2.0;  f=sin2pix

=0.01

=0.02

=0.05

=0.08

=0.10

=0.15

=0.20

10

5000

5.0

4.5

4.0

3.5

3.0

2.5

2.0

1.5

log excess risk

Excess risk;

=0.01

=0.02

=0.05

=0.08

=0.10

=0.15

=0.20

10

5000

4.0

3.5

3.0

2.5

2.0

1.5

log excess risk

Excess risk;

=0.01

=0.02

=0.05

=0.08

=0.10

=0.15

=0.20

n

n

In the above settings, the bias decays faster than variance. Figure 3 on page 23 shows that the excess risk decays fast when n is relatively small and coincides the theoretical asymptotic rate in Theorem 3.2 when n is large. The crossover happens for smaller n when the magnitude of noise is larger. Similar phenomenon has also been reported by Cui et al. [2021, FIG.2, FIG.3]. In addition, comparing the sample size when crossover happens for three target functions, our results show that the crossover happens for smaller n when the function is smoother, which is also consistent with Theorem 3.2.

10

5000

n

4.0

3.5

3.0

2.5

2.0

1.5

log excess risk

Excess risk;

=0;  f=cos2pix

=0.01

=0.02

=0.05

=0.08

=0.10

=0.15

=0.20

10

5000

n

4.0

3.5

3.0

2.5

2.0

1.5

log excess risk

Excess risk;

=0;  f=sin2pix

=0.01

=0.02

=0.05

=0.08

=0.10

=0.15

=0.20

10

5000

n

4.0

3.5

3.0

2.5

2.0

1.5

log excess risk

Excess risk;

=0.01

=0.02

=0.05

=0.08

=0.10

=0.15

=0.20

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a rigorous analysis of the asymptotic rates of kernel ridge regression. Building on works which analyzed the Gaussian design case, this work recovers the same rates using less severe restrictions on the distribution of features. The novel contribution here beyond similar works of Li et al 2023 (a,b) is to characterize the rate of the bias term. The authors derive a complete phase diagram of the possible behaviors and provide the optimal scaling of the ridge parameter $\\lambda \\sim n^{-\\theta}$.  They provide experiments on simple sinusoidal regression problems where eigenvalues and eigenfunctions are exactly known and show that the theoretical rates match the experimental rates.\n\nSTRENGTHS:\nThis paper gives a rigorous analysis of kernel regression rates when eigenvalues fall in a power law $\\lambda_i \\sim i^{-\\beta}$. The specifically novel contribution in this work is obtaining the rate of decay of the bias term and giving an exact set of assumptions that give these rates (embedding index, source condition, Holder condition on kernel, etc). \n\nThe analysis recovers known (correct) results, which have been proven in the worst case, and calculated exactly under Gaussian design assumptions. The authors also provide empirical evidence to support their claimed rates on a toy data distribution, which is good for a theory paper.\n\nWEAKNESSES:\n**Obtained learning curves pre-exist in the literature**\n\nThe rates obtained match prior non-rigorous analyses and so do not exhibit *new* behaviors over what has been studied in the prior works. That said, the rigorous proof under relaxed set of conditions is still useful.\n\n**Title seems too broad**\n\nThe title seems to cover too much territory. I would recommend a more descriptive title like \"Rigorous analysis of bias and variance rates for KRR\" \n\n**Relevant works**\n\nWhile the paper does mostly a good job of citing relevant prior works, it misses a few relevant works on this area\n\n1. [Spigler et al 2019](https://arxiv.org/abs/1905.10843) which obtains rates in the noiseless regime using statistical physics techniques. \n2. [Canatar et al 2021](https://www.nature.com/articles/s41467-021-23103-1), which also gives the bias and variance terms for noisy kernel ridge regression under the Gaussian design with the replica method.  \n3. [Simon et al 2023](https://openreview.net/forum?id=FDbQGCAViI), which uses a different heuristic to derive the average case KRR learning curve and introduces the idea of 'learnabilities' as a way of reasoning about the KRR predictor.\n\nQUESTIONS:\n1. Can similar analysis techniques be applied to other decay rates? Suppose the eigenvalues fell not as a power law but instead exponentially fast $\\lambda_i = e^{- i \\beta}$. Could the same techniques be used to prove rates in this case ? Another setting of interest are polynomial regression problems in high dimension such as those studied by [Canatar et al 2021](https://www.nature.com/articles/s41467-021-23103-1). Is a rigorous proof of these curves possible as well in the high dimensional limit?\n\n2. The authors claim that \"that the benign overfitting phenomenon may not exist if one trains a very wide neural network.\"  Which of the results in the paper do the authors thinks support this claim?\n\nTypos\n1. line 6 \"attentions\" should be \"attention\"\n2. line 17 \"arisen\" -> \"arising\"\n3. line 32  \"attentions\" -> \"attention\"\n4. line 86 \"orders\" -> \"order\"\n5. line 291 \"mistery\" -> \"mystery\"\n\nLIMITATIONS:\nThe authors address the technical limitation of their current approach in the $\\theta \\geq \\beta$ regime and leave the solution open as a future problem.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzed generalization error rates for the Kernel Ridge Regression (KRR) problem. The authors make use of several generic assumptions, including (i) eigenvalue decay, (ii) Embedding index, (iii) source condition, and (iv) Holder continuity of the kernel. It is worth noting that assumptions (i), (ii), and (iv) are generally applicable to popular kernels. The findings of this study align with previous knowledge: in the absence of noise, interpolation achieves optimality, while in the presence of noise, a well-known Bias-Variance tradeoff emerges.\n\nSTRENGTHS:\n1. The paper is well-written, exhibiting a high level of clarity and coherence in its structure and language.\n2. The main result presented in the paper is not only insightful but also highly intriguing, contributing valuable insights to the research area.\n3. The authors have demonstrated diligent referencing and citation of relevant prior work, showcasing a thorough understanding of the existing literature in the field.\n\nWEAKNESSES:\nWhile I believe the paper is well-written, I feel that there are some areas where the authors assume background knowledge from the reader or make claims without providing adequate supporting discussions. I have identified a few specific points that I believe could benefit from clarification or additional explanation:\n\n1. The contribution section appears to be highly technical. It would be helpful to provide a high-level, less technical overview of the contributions before diving into the specifics. For example, it would be beneficial to provide a brief explanation of the significance of variables such as β and s, as these variables are not defined prior to that section. \n\n2. In line 47, it is mentioned that \"our results...MAY suggest the benign...\". The term \"may\" introduces uncertainty, but it is unclear how this conclusion was reached or what factors contribute to this interpretation/uncertainty. It seems that the authors may have intended to convey that since wide neural networks are essentially kernels, and the rates clearly demonstrate that noise impairs generalization, it is unlikely for the noise to have a benign effect. If this is indeed the case, it would be helpful to present this point in a clearer and less convoluted manner.\n\n3. The second row of Figure 1 mentions \"overfitting\" and \"underfitting,\" but there is no prior discussion or explanation of these terms in the paper. It would be beneficial to define these terms and provide some context.This would enhance the understanding of the results presented in the figure. Overall the second row figures were unclear to me.\n\n4. In line 290, the statement mentions that \"These results will help us better understand the generalization mistery of neural networks.\" \nIt is unclear how this conclusion was reached or what specific discussions led to it. It would be valuable to elaborate on the implications of the results and provide a more thorough explanation of how they contribute to a better understanding of the generalization of neural networks.(by the way \"mistery\" is a typo!)\n\nOverall, by addressing these points, the paper can become more accessible to a broader range of readers and ensure a clearer understanding of the key concepts and conclusions.\n\nQUESTIONS:\nIn addition to Weaknesses section I have Following questions,\n\nI understand assumption 2 and 3 are common for these kind of analysis. But would you please elaborate,\n\n1. What is the significance of the \"embedding index\"? What are the implications if α << 1/β or α >> 1/β?\n\n2. What are the limitations of assumption 3? From my understanding, as \"s\" increases, the function f* becomes smoother since it belongs to H^s. Could you please clarify when this assumption holds or fails?\n\n\nI am willing to increase my score if the authors address these questions in Weaknesses/Questions sections.\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors focus on the learning curve of kernel ridge regression, an important topic in theoretical machine learning, which is also used to approximate the generalization ability of 'lazy trained/NKT regime' neural networks. \n\nThe authors provide a nearly full characterization of the learning curve in the setting of Source/Capacity conditions, considering the effects and interplay of the regularization parameter, the source condition, and the noise, rigorously closing many gaps in the literature, and in particular the picture arising from recent works in Neurips.\n\nThe paper's contributions include: \n* Providing sharp estimates of the asymptotic orders of the bias term when the regularization parameter is not too small. This result holds for both well-specified and misspecified cases, improving the upper bounds given in previous research.\n* Showing an upper bound of the bias term in the nearly interpolating case. This upper bound is tight and matches the information-theoretic lower bound.\n* Providing learning curves of KRR for both noisy and noiseless cases, and the interplay between the two.\n\nSTRENGTHS:\nI believe this is a strong paper. \n\nMost recent arguments on the learning curve of kernel methods were not entirely rigorous, and are based on the ‘Gaussian design’ assumption, which is certainly very restrictive. Even within the realm of Gaussians, Bordelon et al used a non-rigorous the replica method, while Cui et al used results from CGMT, but extrapolated their domain of validity. Random matrix arguments were also used by Jin et al, but some gap existed in the argument. \n\nThis paper bridges the mathematical gap and manages to prove rigorously a nearly complete picture, removing the Gaussian assumptions. It closes a clear gap in the mathematics of Kernel ridge regression.\n\nWEAKNESSES:\nI do not believe the paper has any particular weaknesses.\n\nQUESTIONS:\n* I would be interested to know more about the limitation of the predictions. Removing the Gaussian assumption is quite a feat, but are they any restrictions on the data structure except the power law distribution/embedding dimension? Indeed, it seems there are only Eigenvalue decay assumptions in the paper! Do this means that any distribution of data in the RKHS is OK and that only the Covariance matrix matters? It would help me immensely to be explicit and to give counter examples.\n\n* With respect to Cui et al, it seems to me that the authors prove their entire phase diagram (Theorem 1 and 2), except for the left region of the blue region in Fig. 1 of Cui et al. I believe. The regime where θ ≥ β, is also very interesting, and I would be interested in understanding the reason why it is harder to control. Is it a possibility that in this region the data structure becomes relevant so that the Gaussian assumption is needed or do the authors believe it is only a technical problem? I remember a similar problem arises in Jacot et al.\n\nLIMITATIONS:\nFrom a theoretical point of view, the paper is certainly addressing its limitation and has no ethical issue that I can foresee. It would still be interesting to be given the code to reproduce the figure in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the asymptotics of kernel ridge regression learning curves when the kernel eigenvalues and the ridge parameter both obey a power law.\n\nSTRENGTHS:\n1. The paper is readable and well written. \n\n2. It derives asymptotic results for KRR for a specific choice of kernel eigenvalues and regularization. \n\n3. The introduction of technical definitions is clear, and sufficient background for previous results is provided.\n\nWEAKNESSES:\n1. The claims of the paper are confusing to me. The double-descent phenomena (in this case sample-wise in the noisy case) implies that there is a second descent after the U-shaped curve, while the authors did not seem to reproduce that result.\n\n2. The learning curves plotted in Figure 1. are with respect to the ridge parameter, while learning curves often refer to the generalization error as a function of training set size.\n\n3. Experiments are not sufficient to demonstrate the tightness of the theoretical results. \n\n4. The references (Bordelon et al. [2020], Cui et al. [2021]) cited in the paper demonstrate near perfect agreement with experimental learning curves despite the Gaussian assumption, but a discussion relating those results to the theory here is missing.\n\n5. The claim regarding wide neural networks and benign overfitting stated in line 47 is neither explained nor demonstrated by experiments.\n\n6. In general, a clear comparison to previous literature is missing, and it is hard to understand the novelty brought by this paper.\n\nQUESTIONS:\n1. In line 56, $s$ is not defined. Also, it does not match the main result (i.e. it should be $s \\geq 2$  rather than $s \\geq 1$).\n\n2. In random matrix theory, precise asymptotics can be obtained by relying on Gaussian equivalence principles and provide almost perfect learning curves for kernel regression (see Cui et.al. 2021). What insights the theory developed here brings in addition to the previous results and why do you think Gaussian assumption works so well?\n\nLIMITATIONS:\nThe limitations section is missing.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the generalization error/excess risk of the kernel ridge regression (KRR). Using the classical bias-variance decomposition, the authors gives bounds on the bias term, which is claimed to be tighter than previous literatures. Together with the variance term results from another paper  \"Yicheng Li, Haobo Zhang, and Qian Lin. Kernel interpolation generalizes poorly, March 2023a.\", the authors conclude the bounds on the excess risk asymptotic in the sample size, where the effect of the decay of the ridge is stated explicitly.\n\nSTRENGTHS:\nThe paper gives a comprehensive reviews on the excess risk of KRR. There are experiments in both main text and appendix supporting the claimed result.\n\nWEAKNESSES:\nMy biggest concern would be that this paper is merely an incremental paper from \"Yicheng Li, Haobo Zhang, and Qian Lin. Kernel interpolation generalizes poorly, March 2023a.\" \n\nAccording to the remarks 3.3 and 3.5, the main theorems 3.2 and 3.4 basically recover the results from previous works: \"Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics, 7(3):331–368, 2007. doi: 10.1007/ s10208-006-0196-8.\", \"Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 34:10131–10143, 2021.\" and \"Yicheng Li, Haobo Zhang, and Qian Lin. Kernel interpolation generalizes poorly, March 2023a.\"  Other than tightening the bounds, I hardly see more novelty or improvement from the main result.\n\nQUESTIONS:\nCould you explain more on the novelty of the paper?\n\nLIMITATIONS:\nThis is a theoretical paper, where the assumptions 1-5 are stated clearly. The authors has remarked that these assumptions hold in general settings.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}