Title: A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression
Venue: NeurIPS 2024
Accepted: yes
Keywords: kernel ridge regression

generalization
Abstract: This paper conducts a comprehensive study of the learning curves of kernel ridge regression (KRR) under minimal assumptions.
Our contributions are three-fold: 1) we analyze the role of key properties of the kernel, such as its spectral eigen-decay, the characteristics of the eigenfunctions, and the smoothness of the kernel; 2) we demonstrate the validity of the Gaussian Equivalent Property (GEP), which states that the generalization performance of KRR remains the same when the whitened features are replaced by standard Gaussian vectors, thereby shedding light on the success of previous analyzes under the Gaussian Design Assumption; 3) we derive novel bounds that improve over existing bounds across a broad range of setting such as (in)dependent feature vectors and various combinations of eigen-decay rates in the over/underparameterized regimes.

A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression

Abstract

This paper conducts a comprehensive study of the learning curves of kernel ridge regression (KRR) under minimal assumptions. Our contributions are three-fold: 1) we analyze the role of key properties of the kernel, such as its spectral eigen-decay, the characteristics of the eigenfunctions, and the smoothness of the kernel; 2) we demonstrate the validity of the Gaussian Equivalent Property (GEP), which states that the generalization performance of KRR remains the same when the whitened features are replaced by standard Gaussian vectors, thereby shedding light on the success of previous analyzes under the Gaussian Design Assumption; 3) we derive novel bounds that improve over existing bounds across a broad range of setting such as (in)dependent feature vectors and various combinations of eigen-decay rates in the over/underparameterized regimes.

1 Introduction

Kernel ridge regression (KRR) is a central tool in machine learning due to its ability to provide a flexible and efficient framework for capturing intricate patterns within data. Additionally, it stands as one of the earliest endeavors in statistical machine learning, with ongoing research into its generalization properties [12, 45]. Over the past few years, kernels have experienced a resurgence in importance in the field of deep learning theory [49, 8, 6], partly because many deep neural networks (DNNs) can be interpreted as approaching specific kernel limits as they converge [25, 1, 9].

One central topic in machine learning theory is the learning curve of the regressor in the fixed input dimensional setting as the sample size grows to infinity. Formally: let n be the sample size, λ = λ ( n ) be the ridge regularization parameter depending on n and R n be the test error/excess risk of the ridge regression. For large n , the test error R n should decay with n as R n = O n, P ( g ( n )) for some function g : R → R such that g ( n ) n →∞ - - - - → 0 . The decay of g with respect to n provides an upper bound on the learning curve of the ridge regressor and will be the main focus of this paper. To conduct our analysis, we concentrate on several crucial properties of the kernel, including its spectral eigen-decay and the characteristics of the eigenfunctions, which we will elaborate on next.

Properties of the eigenfunctions In a series of studies [10, 16, 34], the feature vectors are substituted by random Gaussian vectors following the Gaussian Design (GD) Assumption, and the

learning curve is derived using the Replica method. In a separate body of research [29, 30, 31], it is demonstrated that similar learning curves occur for Hölder continuous kernels under an assumption called the Embedding Condition (EC) (see Section A for more details). Consequently, there is a fundamental mismatch between the distribution of the feature vector and the Gaussian random vectors used in [10, 16, 34]: in the former case, each coordinate is highly dependent on the others, whereas in the Gaussian Design case, each coordinate operates independently from the others. Astonishingly, however, both settings share similar learning curves. This phenomenon, initially identified by [21, 3, 42, 20], is termed the Gaussian Equivalence Property . This prompts the question:

Q1: When and why does the Gaussian Equivalence Property exist?

Spectral eigen-decay Many recent papers [10, 35, 44, 36] have attempted to characterize the test error solely by the (kernel) eigenspectrum decay. It is for instance common to differentiate between different eigenspectrum decays: [29, 30, 31] assumes the embedding condition (EC) and Hölder continuity to kernel with polynomial eigen-decay; [33] assumes either polynomial or exponential eigen-decay (with noiseless labels) under the Maximum Degree-of-Freedom (MaxDof) Assumption; [36] assumes some concentration and the so-called hypercontractivity on eigenfunctions.

However, [14] pointed out that the characterization of generalization performance solely by the spectral eigen-decay might oversimplify the generalization performance of ridge regression. In relation to the second question, we further ask:

Q2: Under what conditions is the generalization error fully determined by the eigen-decay?

Additional assumptions and settings In addition to the two properties above, other hyperparameters or settings, including capacity of the kernels/feature vectors, the ridge regularization decay, the source condition of the target function, the noise level in the output label, and the amount of over-parameterization, play an important role in the analysis of the learning curve of ridge regression. Within the present body of research, various papers establish bounds on the test error of KRR across diverse assumptions and settings (we refer the reader to Section A for further elaboration). It is therefore of significant interest to ask:

Q3: Is there a unifying theory explaining the generalization under minimal assumptions?

Contributions We address questions Q1-3 through the following contributions:

(i) Unified theory: We provide a unifying theory of the test error of KRR across a wide variety of settings (see Subsection 2.2 and Table 1 in Section 3).

(ii) Validation and GEP: We show that the generalization performance with independent (Gaussian) features and dependent (kernel) features coincides asymptotically and it solely depends on the eigen-decay under strong ridge regularization, hence validating the Gaussian Equivalent Property (GEP) (see Subsection 3.2) .

(iii) New and sharpened bounds: We provide novel bounds of the KRR test error that improve over prior work across various settings (see Subsections 3.2).

(iv) Smoothness and generalization: We relate the spectral eigen-decay to kernel smoothness (see Appendix B.2) and hence to the kernel's generalization performance.

2 Setting

In this section, we introduce the basic notation for ridge regression, which includes high-dimensional linear regression and kernel ridge regression as special cases.

2.1 Notations

Suppose p ∈ N ∪ {∞} . Let x = ( x k ) p k =1 ∈ R p be a random (feature) vector sampled from some distribution µ on R p . Let n ∈ N be an integer and denote by x 1 , ..., x n n i.i.d. draw of x . Denote the input matrix X ∈ R n × p to be a matrix with rows x ⊤ i . By fixing an orthonormal basis, we assume that the covariance matrix is diagonal:

where the eigenvalues λ 1 ≥ λ 2 ≥ ... ≥ λ p > 0 is a decreasing sequence of positive numbers. 1

We also assume that y ∈ R n is an output vector such that

where θ ∗ ∈ R p is a deterministic vector, ϵ ∈ R n is a random vector whose entries are i.i.d. drawn from a centered random variable ϵ with variance E [ ϵ 2 ] = σ 2 < ∞ and independent to X .

Then the linear regressor

is the minimizer of the empirical mean square loss (MSE) problem:

where λ ≥ 0 is the ridge 2 . This paper focuses on bounding the test error, with which we can analyse the learning curve of the regressor. To do so, we use the following well-known bias-variance decomposition.

Definition 2.1 (Bias-variance decomposition) . Consider input-output pairs ( X , y ) of sample size n and a ridge λ ≥ 0 . Define the test error R to be the population mean squared error between the regressor and the true label averaged over noise.

Note that R is a random variable depending on the samples ( X , y ) and the ridge λ ≥ 0 . Hence, we can also view R = R n as a random variable indexed in n , where the samples ( X , y ) are n i.i.d. drawn input-output pairs and λ is chosen to depend on n .

We decompose the test error into a bias B and variance V , which is typical for most KRR literature [32, 23, 6, 46, 29, 30, 31, 13, 14]:

Remark 2.2 (Noiseless labels) . If there is no noise in the label, that is, ϵ = 0 , the test error R is simply the bias B . Hence, the analysis of the bias term B in this paper is directly applicable to the noiseless labels setting.

We now summarize the combinations of assumptions and settings made in this paper.

2.2 Assumptions and Settings

Polynomial/exponential eigen-decay We consider two types of spectral decay rates, namely, polynomial and exponential decay rates, because: 1) polynomial eigen-decay is, roughly speaking , equivalent to the case where the RKHS is comprised of at most finitely many continuous derivatives; 2) the exponential eigen-decay is, possibly up to a canonical change in the relevant function space, equivalent to the case where the RKHS consists of smooth (infinitely differentiable) functions. For the formal definition of the eigen-decay, see Assumptions (PE) and (EE). For further details and explanation on the relationship between eigen-decay and smoothness, we refer the reader to Section B.

Source condition Many previous works [6, 16, 7, 30] include the so-called source condition as assumptions on the target. If the task is proper, that is, θ ∗ ∈ H = H 1 , we have s ≥ 1 . More generally, a larger source coefficient s implies a smoother target θ ∗ in the RKHS H .

Definition 2.3 (Interpolation space) . Let s ≥ 0 be a real number. Define the interpolation space

Assumption (SC) (Source Condition) . The source coefficient of a target coefficient θ ∗ is defined as

See Subsection A.5 for more elaborations for the source coefficient s in polynomial or in exponential decay.

Strong/weak ridge We set the ridge λ = λ ( n ) ≥ 0 to depend on the sample size n . The ridge is considered strong (relative to the eigen-decay) if λ ≽ λ min { n,p } , that is, if λ/λ min { n,p } n →∞ - - - - → 0 ; otherwise, it is considered weak. Intuitively, the ridge is weak when it is negligible compared to the entries in the kernel matrix, effectively making it ridgeless.

To summarize the assumptions discussed previously, let ( λ k ) p k =1 be the eigenvalues of the kernel K , and θ ∗ def. = ( θ ∗ k ) p k =1 the coefficients of the target function being learned in the eigen-basis defined by K . Then we assume either of the following assumptions:

̸

Assumption (PE) (Polynomial Eigen-decay) . Assume that λ k = Θ k ( k -1 -a ) , | θ ∗ k | = Θ k ( k -r ) , λ = Θ n ( n -b ) for some constants a, b, r > 0 , where a + 2 = 2 r unless specified. 3 Hence, if Assumption (SC) holds, the source coefficient is s = 2 r + a 1+ a . We call the ridge λ strong if b ∈ (0 , 1+ a ] , and weak if b ∈ (1 + a, ∞ ] , under the convention that b = ∞ implies λ = 0 .

̸

Assumption (EE) (Exponential Eigen-decay) . Assume that λ k = Θ k ( e -ak ) , θ ∗ k = Θ k ( e -rk ) , λ = Θ n ( e -bn ) for some constants a, b, r > 0 , where a = 2 r unless specified. 4 Hence, if Assumption (SC) holds, the source coefficient is s = 2 r + a a = 2 r a +1 . We call the ridge λ strong if b ∈ (0 , a ] , and weak if b ∈ ( a, ∞ ] , under the convention that b = ∞ implies λ = 0 .

Generic/independent features Our analysis centers on the assumptions regarding feature vectors, with a focus on the dependencies between coordinates, particularly exploring two cases:

Generic features (GF): include the cases where the feature vectors are dependent on each other, for example, the feature vectors from the following kernels:

dot-product kernels on hyperspheres;

kernels with bounded eigenfunctions;

radial base function (RBF) and shift-invariant kernels;

kernels on hypercubes,

satisfy Assumption (GF). Most previous literature [30, 33, 36, 19] have assumptions that only a proper subset of the above kernels satisfies. Therefore, we believe that we are operating under the minimal assumptions that exist in the field.

Independent features (IF): replace the feature vector with sub-Gaussian random vector with independent coordinates. A special case is the Gaussian Design assumption (GD) used in literature [44, 35, 16].

For further explanations regarding the assumptions, we refer the reader to Section A.

3 Main result

We first present an overview of the test error bounds across various properties, assumptions, and regimes. Our main results, summarized in Table 1, describe the learning curve in the overparameterized regime, in terms of the bias B and variance V decomposition (see Equation (5)). Then, we will discuss the implications of our results in depth.

̸

3.1 Overview

Table 1 summarizes many of our results in the over-parameterized regime under various combinations of the assumptions described in subsection 2.2. The bounds are expressed in terms of the sample size n as O n ( · ) or ˜ O n ( · ) (ignoring logarithmic terms). Whenever we can also prove a matching lower bound, we replace O n ( · ) with Θ n ( · ) . We write ( · ) + def. = max {· , 0 } .

ridge regularization parameter (Assumptions (PE) and (EE)), σ = E ϵ is the noise level , and s > 0 is a technical parameter often determined by a and r (e.g. under Assumption (SC)). Here ˜ s def. = min { s, 2 } .

Results in blue indicate either previously unstudied regimes or improvements in available rates in a studied regime. See Table 6 for more comparisons and Subsection 2.2 for details on various settings.

Before delving into the detailed discussion of our comprehensive results in Subsection 3.2, let us highlight some important observations from Table 1.

Asymptotic bounds The upper bound illustrates the asymptotic relationship between the test error and sample size n as well as the following constants: a related to the eigen-decay, b related to the ridge, r related to the target function and σ 2 related to the label noise.

Independent feature (IF) versus generic features (GF) The bounds in both cases coincide under strong ridge (see the left columns of Table 1); meanwhile, under weak ridge (see the right columns of Table 1), the bounds with generic features are looser than those with independent features. In Subsection 3.2, we will explain the necessity of this difference and hence showcase the previous limitations in the literature, which has widely adopted the Gaussian Design Assumption (GD) under the weak ridge/interpolation regime.

Novel bound of bias under weak ridge A notably sophisticated bound (on the upper right corner of Table 1)

is novel to the best of our knowledge. Our improvement compared to previous literature [7] under various eigen-decay (in terms of a ) and target coefficients' decay (in terms of r ) is shown in Figure 1. By comparison, we can see that the decay of our novel bound in Equation 6 is faster than previous results. Also, we prove that the upper bound in the middle green region ,where s ∈ (1 , 2) , is sharp.

Experimental validations of the results in Table 1 are given in Section 6.

In the under-parameterized regime, the bias B and variance V terms can be bounded similarly as in over-parameterized regime. We postpone the details of these results to Section F.

3.2 Detailed discussion

In this subsection, we elaborate more on the details of our results shown in Table 1.

Independent and generic features Table 1 indicates that the test error exhibits the same upper bound with either independent or generic features under strong ridge conditions in the overparameterized regime. This similarity arises from the bounds in both cases being derived from the

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

r

(

n

(2

r

+

a

)

)

(1)

(

n

(

r

a

)

)

(

n

2(1 +

a

)

)

s<1

s<1

1<s<2

s>2

Our result on

r

=

a

2

r

=2+

a

r

= 1

2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

(1)

(

n

(

r

a

)

)

(

n

(2

a

)

)

Previous result on

r = a

2r = 2 + a

a

a

same Master inequalities that we will introduce later (see Section 4). However, under weak ridge conditions, the empirical kernel spectrum displays qualitative differences, as reported in [7, 14]. From Figure 2, we can see that V = O (1) with Laplacian kernel in the left plot and V diverges with the neural tangent kernel (with 1 hidden layer) in the right plot. Hence under weak ridge and polynomial eigen-decay, the case distinction of the bound

in Table 1 is necessary, as Assumption (GF) includes the cases of Gaussian Design (GD) (or more generally independent features (IF)) and Laplacian kernel which yields V = Θ (1) , the so-called tempered overfitting from [35]; as well as the case of neural tangent kernel (NTK) which yields V n →∞ - - - - →∞ , the so-called catastrophic overfitting. In particular, our proof shows that the Gaussian Equivalent Property (GEP) does not hold under weak ridge.

200

400

600

800

1000

n

0.40

0.41

0.42

0.43

0.44

0.45

Overfitting of Laplacian kernel

200

400

600

800

1000

n

0

500

1000

1500

2000

Overfitting of NTK with L=1

We are now prepared to address the first question posed in Section 1:

Q1: When and why does the Gaussian Equivalence Property (GEP) exist?

The Master inequalities provide the same non-asymptotic bounds for both cases under a strong ridge. However, GEP does not hold under weak ridge! 5

In particular, our work implies that previous works [10, 16, 34, 44, 35] under the Gaussian Design assumption (GD) can be applied only when the ridge is strong.

Upon finalizing the preparation for this paper, we became aware of concurrent research conducted by [19], which also concerns the Gaussian Equivalence Property (GEP) in the non-asymptotic setting. For more comparison between their assumptions and ours, we refer the reader to Section A.

Importance of ridge To address the second question posed in Section 1, it is evident that in either the under-parameterized setting with any ridge (see Section F) or the over-parameterized regime with a strong ridge (see Table 1), the bounds for both B and V remain the same, irrespective of the features:

Q2: Under what conditions is the generalization error fully determined by the eigen-decay? Either (i) in the under-parameterized setting; or (ii) with a strong ridge in the over-parameterized regime.

Several results [39, 33, 36] have suggested that the test error bound can be characterized by the covariance spectrum Σ , but they implicitly require the ridge λ > 0 to be larger than some threshold. This paper clearly demonstrates the necessity of the presence of a strong ridge in such analyses.

From Table 1, we can see that the eigen-decay also affects the test error qualitatively. As mentioned in [33], the bias term B decays polynomially (or, respectively, exponentially) when the eigen-decay is polynomial (or, respectively, exponential). However, we prove that V decays only polynomially at a rate at most O ( 1 n ) , regardless of the eigen-decay. Hence, in a noisy setting with polynomial eigen-decay, one can find an optimal ridge λ = Θ n ( n -1+ a (1+ a )˜ s +1 ) to balance both terms B and V as in [16, 31]. In contrast, in noisy settings with exponential eigen-decay, V dominates the test error.

The bound of the variance term V with exponential eigen-decay under weak ridge is omitted in Table 1 due to the so-called catastrophic overfitting phenomenon observed in [35, 14].

Improved upper bound During our research, we discovered that we can improve the upper bound of B (Equation 6) in the over-parameterized regime with polynomial decay, weak ridge, and Assumption (GF) by adapting the result from [31] and integrating it with the insights from [7] (see the upper right corner in Table 1 or 6, and Figure 1 for visualization).

Theorem 3.1 (Improved upper bound) . Suppose Assumption (GF) holds. Assume the eigen-spectrum and the target coefficient both have polynomial decay, that is, λ k = Θ k ( k -1 -a ) and | θ ∗ k | = Θ k ( k -r ) . Let s = 2 r + a 1+ a be the source coefficient defined in Definition (SC). Then the kernel ridgeless regression has the following learning rate for the bias term B :

where n is the sample size and ( · ) + def. = max( · , 0) .

For a detailed explanation of our improvements, refer to our novel Proposition D.5 and the known Proposition D.6 from [7] in Section D.

Lower bound of test error It is of theoretical interest to provide lower bounds as well. For independent features, we can prove that the upper bound is tight using the result from [46], generalizing the result from [16], which only showed upper bounds in Gaussian Design Assumption (GD). However, for generic features, we can only provide lower bounds in some but not all settings, using the result from [31]. We summarize our results in Table 2.

See Section E for details on our proof of matching lower bounds. We note that there is some KRR literature, such as [31, 36, 19], that discusses matching upper and lower bounds of test error under more assumptions, which is beyond the scope of this paper. For a comparison of assumptions in different papers, see Section A.

Finally, we summarize the above discussion by answering the third question we raised in Section 1:

Q3: Is there a unifying theory on the generalization performance under minimal assumptions? Yes, this paper considers assumptions (IF) and (GF) which cover a wide range of kernel settings under any regularization and source conditions.

4 Proof sketch

All the above results in Table 1 can be derived using the following proof strategy:

We prove a concentration result on the whitened features z under Assumptions (GF) or (IF) (see Section G).

Using the above result, we bound the condition number of the (truncated) kernel matrix (see Section G) as in [7, 14], which will be used to bound the test error in the next step.

Combining the bound of the condition number with the result from [6, 46], which we call the Master inequalities (see the paragraph below for details), we can compute the non-asymptotic test error bound for various settings (see Section D).

In the over-parameterized regime, we derive the asymptotic behavior of the learning curves by plugging in the eigen-decay and the choice of the target function and ridge in the above non-asymptotic bound (see Section D).

Using the results from [46], we are able to show that the asymptotic upper bound for independent features (IF) is tight (see section E).

For generic features (GF), we only provide tight bound results in limited settings (see section E). As shown in Figure 2 in Section 3, the generic feature Assumption (GF) includes a broad variety of features where a universal matching lower bound does not exist.

In the under-parameterized regime, the Master inequalities also give upper bounds for B and V , which might not be tight if the ridge λ is strong. However, we present another way to obtain tight bounds without the Master inequalities (see Section F for more details).

We summarize the proof techniques with a flowchart in Figure 4.

Master Inequalities We will now briefly introduce the key inequality used in our analysis (further details can be found in the Appendix). We will use the subscript > k or ≤ k to refer to the submatrix of a matrix consisting of columns with index > k or ≤ k (see definition C.3). The analysis relies on two crucial matrices: A k = X >k X ⊤ >k + nλ I n ∈ R n × n , defined for any k ∈ N , which is employed to partition the spectrum, and a whitened input matrix Z def. = XΣ -1 / 2 ∈ R n × p . Additionally, we require the following definitions.

Definition 4.1 (Concentration coefficients [7]) . Define the quantities:

where s i ( · ) denotes the i -th largest singular value of the matrix.

Definition 4.2 (Effective ranks [6]) . Let k ∈ N . Define two quantities:

The Master inequalities provide upper bounds on the bias and the (scaled) variance term of the test error in the following form:

The term appearing in the above bound can be categorized into two distinct components: the "probably constant" part (highlighted in blue) and the "decay" part (highlighted in gray). The "probably constant" part consists of terms that, with high probability, are bounded both below and above by positive constants-these bounds represent a primary contribution of this paper. On the other hand, the "decay" part can be approximated using basic calculus, once a specific constant k ∈ N , smaller than the sample size n , is selected. Together, these two parts allow us to derive the KRR learning rate for all combinations of eigen-spectra, features, and ridge parameters discussed in the preceding sections. Furthermore, [46] provides a corresponding lower bound under certain assumptions, demonstrating that the decay of the upper bound matches that of the lower bound. Establishing that Assumption (IF) satisfies these assumptions is another key contribution of this work. For the formal definitions of the terms in the above inequalities, we refer the reader to Propositions C.6 and C.7 in the appendix.

5 Related work

Webriefly discuss related previous works and compare them with our results in the over-parameterized regime: (see Table 6 for more visual illustration)

[16] considered the upper bound of B and V in polynomial eigen-decay under any ridge and under the Gaussian Design Assumption. Our result proves both the matching upper and lower bound with the same decay rate under a weaker assumption (IF). This implies that we validate the Gaussian Equivalence Property for Sub-Gaussian Design.

[30, 31] proved tight upper bounds of B and V for Hölder continuous kernels under polynomial eigen-decay, strong ridge and the so-called Embedding condition (EC). [7] recovered the upper bounds under Assumption (GF).

For polynomial decay under weak ridge, [31] provided a tight upper bound of B when the source condition s > 1 ; while [7] provided a loose bound regardless of s . Hence we modify the proof in [31] under Assumption (GF) instead and combine it with [7] to obtain a novel upper bound on the bias.

[7] also provided an upper bound of V under polynomial decay, weak ridge and Assumption (GF).

[14] showed that V is bounded above and below by positive constants under polynomial eigendecay, weak ridge and Assumption (IF), and it exhibits the so-called tempered overfitting from [35].

[33] provided tight upper bounds of B for both polynomial and exponential eigen-decay for kernels under the so-called Maximal Degree-of-Freedom (MaxDoF) Assumption. We recover their result under Assumption (GF) instead of Assumption (MaxDoF).

We apply the result from [46] and [31] to obtain a matching lower bound in some settings and strengthen our result from O n ( · ) to Θ n ( · ) .

6 Experiments

Due to page constraints, this section focuses solely on experiments validating the Gaussian Equivalent Property (GEP). For detailed experiments on other contributions, refer to Section I.

We consider a simple example: let λ k = ( 2 k -1 2 π ) -1 -a and ψ k ( · ) = √ 2 sin ( 2 k -1 2 π · ) such that ∥ ψ k ∥ L 2 µ = 1 for µ = unif[0 , 1] ; let θ ∗ k = ( 2 k -1 2 π ) -r . For p = ∞ and a = 1 , the regression coincides with kernel ridge regression with kernel k ( x, x ′ ) = min { x, x ′ } defined on the interval [0 , 1] [48]. Similar experiments have been conducted on this kernel k by [30, 33]. However, to simulate regression for independent features (IF), the feature rank p must be finite. In the following experiment, we choose p = 2000 , and the sample size n ranges from 100 to 1000, with ridge parameter λ = ( 2 n -1 2 π ) -b where b ∈ [0 , 1 + a ] .

In Figure 3, our experiment demonstrates the GEP, as the learning curves with kernel features (Sine features) and independent features (Gaussian features z ∼ N (0 , I p ) or Rademacher features z ∼ (unif {± 1 } ) p ) coincide and match the theoretical decay.

10

2

10

3

2×10

2

3×10

2

4×10

2

6×10

2

n (log scale)

10

10

10

9

10

8

10

7

Test Error (log scale)

Bias, a=1, b=2, r=1

Sine feature

Rademacher feature

Gaussian feature

Slope=-3.11

10

2

10

3

2×10

2

3×10

2

4×10

2

6×10

2

n (log scale)

10

9

10

8

10

7

10

6

Test Error (log scale)

Bias, a=0.5, b=1.5, r=1.5

Sine feature

Rademacher feature

Gaussian feature

Slope=-3.22

10

2

10

3

2×10

2

3×10

2

4×10

2

6×10

2

n (log scale)

10

4

10

3

Test Error (log scale)

Sine feature

Rademacher feature

Gaussian feature

Slope=-1.00

10

2

10

3

2×10

2

3×10

2

4×10

2

6×10

2

n (log scale)

2×10

2

3×10

2

4×10

2

Test Error (log scale)

Sine feature

Rademacher feature

Gaussian feature

Slope=-0.51

7 Conclusion

In this paper, we present a unifying theory and a comprehensive analysis of the learning curve of kernel ridge regression under various settings. We elucidate the coincidence of learning curves between the Gaussian Design (GD) setting (or more generally, independent features (IF)) and the kernel setting (or more generally, generic features (GF)), and validate the Gaussian Equivalence Property under strong ridge. In addition to recovering previous results, we also improve test error bounds under specific circumstances, thus filling a gap in the existing literature.

Future potential work Our results also raise several theoretical questions. For instance: i) The upper bound in several regions of the phase diagram in Figure 1 is not known to be sharp. It would be of interest to either improve it or find matching lower bounds in those regions; ii) Why is there a qualitative difference in overfitting with the Laplacian kernel and the neural tangent kernel, as shown in Figure 2? Can one further distinguish different cases in the generic feature Assumption (GF) to explain such differences? Additionally, how tight would the bound V = O ( σ 2 n 2 a ) in Eq. (7) be?

Addressing these questions could provide deeper insights into the behavior of kernel ridge regression under various conditions and contribute to further advancing our understanding in machine learning.

Acknowledgments

A. Kratsios acknowledges financial support from an NSERC Discovery Grant No. RGPIN-202304482 and No. DGECR-2023-00230. A. Kratsios also acknowledges that resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute 6 .

References

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning , pages 322-332. PMLR, 2019.

Kendall Atkinson and Weimin Han. Spherical harmonics and approximations on the unit sphere: an introduction , volume 2044 of Lecture Notes in Mathematics . Springer, Heidelberg, 2012.

Benjamin Aubin, Florent Krzakala, Yue Lu, and Lenka Zdeborová. Generalization error in high-dimensional perceptrons: Approaching bayes error with convex optimization. Advances in Neural Information Processing Systems , 33:12199-12210, 2020.

Francis Bach. Breaking the curse of dimensionality with convex neural networks, 2016.

Francis Bach. High-dimensional analysis of double descent for linear regression with random projections. arXiv preprint arXiv:2303.01372 , 2023.

Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Daniel Barzilai and Ohad Shamir. Generalization in kernel regression under realistic assumptions, 2023.

Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 541-549. PMLR, 10-15 Jul 2018.

Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. Advances in Neural Information Processing Systems , 32, 2019.

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning , pages 1024-1034. PMLR, 2020.

David Borthwick. Spectral theory-basic concepts and applications , volume 284 of Graduate Texts in Mathematics . Springer, Cham, [2020] ©2020.

Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 7:331-368, 2007.

Tin Sum Cheng, Aurelien Lucchi, Ivan Dokmani´ c, Anastasis Kratsios, and David Belius. A theoretical analysis of the test error of finite-rank kernel ridge regression. Annual Conference on Neural Information Processing Systems , 2023.

Tin Sum Cheng, Aurelien Lucchi, Anastasis Kratsios, and David Belius. Characterizing overfitting in kernel ridgeless regression through the eigenspectrum. In The Eleventh International Conference on Learning Representations , 2024.

Samuel N Cohen, Deqing Jiang, and Justin Sirignano. Neural q-learning for solving pdes. Journal of Machine Learning Research , 24(236), 2023.

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems , 34:10131-10143, 2021.

Harold Donnelly. Bounds for eigenfunctions of the Laplacian on compact Riemannian manifolds. J. Funct. Anal. , 187(1):247-261, 2001.

Junichi Fujii, Masatoshi Fujii, Takayuki Furuta, and Ritsuo Nakamoto. Norm inequalities equivalent to heinz inequality. Proceedings of the American Mathematical Society , 118(3):827830, 1993.

Georgios Gavrilopoulos, Guillaume Lecué, and Zong Shang. A geometrical analysis of kernel ridge regression and its applications, 2024.

Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. The gaussian equivalence of generative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning , pages 426-471. PMLR, 2022.

Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. Modeling the influence of data structure on learning in neural networks: The hidden manifold model. Phys. Rev. X , 10:041044, Dec 2020.

Moritz Haas, David Holzmüller, Ulrike Luxburg, and Ingo Steinwart. Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. Advances in Neural Information Processing Systems , 36, 2024.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. Annals of statistics , 50(2):949, 2022.

Simon Hubbert, Emilio Porcu, Chris J. Oates, and Mark Girolami. Sobolev spaces, kernels and discrepancies over hyperspheres. Transactions on Machine Learning Research , 2023.

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

Deqing Jiang, Justin Sirignano, and Samuel N Cohen. Global convergence of deep galerkin and pinns methods for solving partial differential equations. arXiv preprint arXiv:2305.06000 , 2023.

Vladimir Koltchinskii and Karim Lounici. Concentration inequalities and moment bounds for sample covariance operators. Bernoulli , 23(1):110-133, 2017.

Lars Larsson-Cohn. L p -norms of Hermite polynomials and an extremal problem on Wiener chaos. Ark. Mat. , 40(1):133-144, 2002.

Yicheng Li, Haobo Zhang, and Qian Lin. On the saturation effect of kernel ridge regression. In The Eleventh International Conference on Learning Representations , 2022.

Yicheng Li, Haobo Zhang, and Qian Lin. Kernel interpolation generalizes poorly. Biometrika , 2023.

Yicheng Li, Haobo Zhang, and Qian Lin. On the asymptotic learning curves of kernel ridge regression under power-law decay. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel 'ridgeless' regression can generalize. The Annals of Statistics , 48(3), Jun 2020.

Jihao Long, Xiaojun Peng, and Lei Wu. A duality analysis of kernel ridge regression in the noiseless regime, 2024.

Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. Advances in Neural Information Processing Systems , 34:18137-18151, 2021.

Neil Mallinar, James B. Simon, Amirhesam Abedsoltan, Parthe Pandit, Mikhail Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: A taxonomy of overfitting, 2022.

Theodor Misiakiewicz and Basil Saeed. A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator, 2024.

Étienne Pardoux. Backward stochastic differential equations and viscosity solutions of systems of semilinear parabolic and elliptic PDEs of second order. In Stochastic analysis and related topics, VI (Geilo, 1996) , volume 42 of Progr. Probab. , pages 79-127. Birkhäuser Boston, Boston, MA, 1998.

Etienne Pardoux and Shige Peng. Backward stochastic differential equations and quasilinear parabolic partial differential equations. In Stochastic Partial Differential Equations and Their Applications: Proceedings of IFIP WG 7/1 International Conference University of North Carolina at Charlotte, NC June 6-8, 1991 , pages 200-217. Springer, 2005.

Lorenzo Rosasco, Mikhail Belkin, and Ernesto De Vito. On learning with integral operators. Journal of Machine Learning Research , 11(2), 2010.

Saburou Saitoh and Yoshihiro Sawano. Theory of reproducing kernels and applications , volume 44 of Developments in Mathematics . Springer, Singapore, 2016.

Isaac J Schoenberg. Positive definite functions on spheres. Duke Mathematical Journal , 9(1):96 - 108, 1942.

Mohamed El Amine Seddik, Cosme Louart, Mohamed Tamaazousti, and Romain Couillet. Random matrix theory proves that deep learning representations of gan-data behave as gaussian mixtures. In International Conference on Machine Learning , pages 8573-8582. PMLR, 2020.

Barry Simon. Operator theory , volume 4. American Mathematical Soc., 2015.

James B Simon, Madeline Dickens, Dhruva Karkada, and Michael Deweese. The eigenlearning framework: A conservation law perspective on kernel ridge regression and wide neural networks. Transactions on Machine Learning Research , 2023.

Ingo Steinwart and Andreas Christmann. Support vector machines . Springer Science & Business Media, 2008.

Alexander Tsigler and Peter L Bartlett. Benign overfitting in ridge regression. J. Mach. Learn. Res. , 24:123-1, 2023.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices , page 210-268. Cambridge University Press, 2012.

Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization, 2017.

Haobo Zhang, Yicheng Li, and Qian Lin. On the optimality of misspecified spectral algorithms, 2023.

Nikita Zhivotovskiy. Dimension-free bounds for sums of independent matrices and simple tensors via the variational principle. Electronic Journal of Probability , 29:1-28, 2024.

Appendix

The appendix will be organized in the following way:

The detailed discussion of the assumptions used in this paper and the comparison to previous literature is presented in Section A;

an extended discussion of the eigen-decay assumptions is presented in Section B;

the main tool of this paper, the Master inequalities, is presented in Section C;

the details of the proof on the asymptotic in over-parameterized regime in Table 1 is presented in Section D;

the proof for a matching lower bound is presented in Section E;

the details of the proof on the non-asymptotic bound in under-parameterized regime is presented in Section F;

the concentration result on the whitened feature z and the conditioning of the kernel matrix are presented in Section G;

related technical results from previous literature are collected in Section H;

experimental results are shown in Section I;

extra tables of summary can be found in Section J.

The summary of our proof can be found in the flowchart in Figure 4.

Concentration (Section G)

Master inequalities (Section C)

Conditioning (Section G)

Test error upper bound (Section D)

Θ

n

(

·

)

bound

O

n

(

·

)

bound

matching lower bound

(Section E)

unknown

under-param.

(Section E)

Big-O Notation We use the standard Big-O notations O · ( · ) , o · ( · ) , Ω · ( · ) , Θ · ( · ) to represents the asymptotic behaviours of the bounds in this paper, where the subscript indicates that the constant in the bound is independent to the variables in the subscript. We use ˜ · to suppress logarithmic terms. With abuse of notation, we use the same notations for their probability versions, when the context is clear.

A Assumptions in details

The primary assumptions made in this paper concern the whitened feature vectors z ∈ R p : Assumption (GF) (Generic features [7]) . Let p ∈ N ∪ {∞} . Assume the isotropic feature vector z def. = Σ -1 / 2 x ∈ R p and the covariance matrix Σ ∈ R p × p are such that: 7

Remark A.1 ( k = 0 ) . For k = 0 , we take the convention that β 0 def. = ess sup z max { ∥ z ∥ 2 Σ Tr[ Σ ] , ∥ z ∥ 2 Σ 2 Tr[ Σ 2 ] } .

Remark A.2 (Intuition) . The intuition is that the three fractions above have expected values of 1:

concentration on z at each truncation point k ∈ N .

Remark A.3 (Weaken Assumption (GF)) . One can further replace the essential supremum and infinum by a high probability guarantee. The argument on the test error bounds would follow analogously with a weaker probability. For simplicity reasons, we only consider the assumption in the above formulation.

Assumption (GF) encompasses both cases of dependent and independent features z k , which can have qualitatively different effects on the test error.

To compare the realistic kernel setting with the Gaussian Design setting (GD) used in previous literature [10, 34, 16, 44, 35], which replaces the feature vectors with Gaussian random vectors (with independent entries), we consider a weaker version of the Gaussian Design assumption commonly required in KRR literature [6, 13, 5, 36, 19]:

Assumption (IF) (Independent sub-Gaussian features) . Suppose p ∈ N . Assume that the features z k 's in the isotropic feature vector z are independent to each other and there exists a constant G > 0 such that the sub-Gaussian norm ∥ z k ∥ ψ 2 of any feature z k is bounded by G . In over-parameterized regime, we also assume that there exists some constant η > 1 independent to n such that p > ηn .

Remark A.4 (Infinite rank) . With Assumption (GF), the feature dimension p can be taken as ∞ ; with Assumption (IF), however, if no further boundedness assumption is imposed, the norm ∥ x ∥ 2 = √∑ p k =1 λ k z 2 k can be undefined if p = ∞ . However, if the sub-Gaussian variables z k 's are all bounded in the sense of Assumption (GF), then p can be chosen to be infinity as well.

Remark A.5 (Special case) . If the features z k are additionally bounded, then Assumption (IF) becomes a special case of Assumption (GF). The former assumes independence of the features, while the latter does not.

Both assumptions (GF) and (IF) are much weaker than those in most KRR literature in the sense that: i) there is no explicit assumption on the input space X ; ii) the distribution of the feature vector need not even be continuous! It is surprising that even with such minimal assumptions, one can derive tight bounds on the test error.

For a more detailed comparison between the assumptions in this paper and those in previous literature, see Section A.

We next compare the assumptions (IF), (GF) with the assumptions in previous literature.

A.1 Among assumptions with independent features

The Gaussian Design Assumption, often employed in various literature such as [10, 16, 34, 20, 35, 44], can be considered a special case of Assumption (IF):

Assumption (GD) (Gaussian Design) . The feature vector z ∼ N (0 , I p ) is distributed as isotropic Gaussian.

Note that Assumption (IF) is much weaker than Assumption (GD) in the sense that: i) the distribution of the feature vector z is no longer rotational invariant or enjoys anti-concentration property; 8 ii) the distribution of each feature z k needs not be continuous; iii) the features z k 's needs not be identical.

Meanwhile, Assumption (IF) appear commonly in KRR literature, for example [6, 46, 23, 5, 13, 14].

A.2 Among assumptions with dependent features

Examples that satisfy Assumption (GF), which is first introduced by [7], includes:

dot-product kernels on hyperspheres;

kernels with bounded eigenfunctions;

radial base function (RBF) and shift-invariant kernels;

kernels on hypercubes.

In particular, the features from Assumption (IF) also satisfies the weakened version of Assumption (GF). See Remark A.3 for details.

Before comparing our assumptions with previous literature, we first define the ∞ -norm in the ridge regression setting analog to its kernel counterpart:

Definition A.6 ( ∞ -norm) . Fix a distribution µ on R p and let v ∈ R p . Define

Remark A.7 . Note that ∥·∥ ∞ is indeed a semi-norm and for all v ∈ R p , we have

Choose k = 0 and Assumption (GF) and Remark A.1 imply:

as we assume that Tr[ Σ ] < ∞ .

Embedding condition [29, 50, 30, 31] used the Embedding Condition to show tight bounds of learning curve with polynomial eigen-decay:

Assumption (EC) (Embedding Condition) . Given a random vector x ∈ R p with covariance Σ = E [ xx ⊤ ] , let z = Σ -1 / 2 x be the isotropic random vector. Define the embedding coefficient:

Suppose the eigen-decay is polynomial: λ k = Θ k ( k -1 -a ) for some a > 0 . Assume the embedding coefficient η 0 = 1 1+ a .

Remark A.8 (General range of η 0 ) . Rewrite the expression:

where we write x = ( x k ) p k =1 ∈ R p and z = ( z k ) p k =1 ∈ R p . In the kernel case, it is obvious that η 0 ≤ 1 : for all inputs x ∈ X with embedding x = K ( x, · ) in the feature space H , sup x k ( x, x ) =

∥ x ∥ 2 2 < ∞ . Since the eigenvalues have polynomial decay, we have η 0 ≥ 1 1+ a , or otherwise there were some η ∈ ( η 0 , 1 1+ a ) such that:

contradicting Assumption (EC). Thus one have η 0 ∈ [ 1 1+ a , 1] in general.

Hence Assumption (GF) is a much weaker assumption than Assumption (EC) in the sense that the former one only requires η 0 ≤ 1 , which holds in general; while the latter one requires η 0 = 1 1+ a , which is the smallest possible size of η 0 . Hence, many examples mentioned in [50], that satisfy Assumption (EC) also satisfy Assumption (GF). For instance, the examples include:

dot-product kernels on hyperspheres;

kernels with bounded eigenfunctions;

shift-invariant periodic kernels.

Nevertheless, [29, 50, 30, 31] also require Hölder continuity of the kernel K while Assumption (GF) requires no continuity assumption.

Maximal degree of freedom We rewrite and simplify the assumption used in [33] with our notation: Assumption (MaxDoF) (Maximal Degree-of-Freedom) . For any λ > 0 and s ≥ 1 , define:

Assume: there exists n ∈ N such that

for some constant δ ∈ (0 , 1) .

Usually, the constant s is chosen as the source coefficient in Assumption (SC). Again, Assumption (MaxDoF) requires some extra boundedness condition on the norm of x . Examples that satisfy Assumption (MaxDoF) include:

dot-product kernels on hyperspheres;

kernels with bounded eigenfunctions;

periodic kernels on hypercubes,

which also satify Assumption (GF).

Low/High-degree feature concentration [36] proved the convergence of the test error to its 'deterministic equivalent' using a list of assumptions for each fixed sample size n . To avoid introducing extra notations, we list their assumption here in a semi-rigorous way:

Assumption (LH) (Low/High-degree feature concentration) . Fix n ∈ N and λ ≥ 0 . Suppose there exists an integer k ≤ p such that:

the low-degree feature vector x ≤ k ∈ R k is highly concentrated;

with high probability, the high-degree feature vector x >k ∈ R p -k satisfies:

where C > 1 is a constant depending on n, k .

Point 3 in Assumption (LH) is closely related to Assumption (GF) since:

According to [36], examples that satisfy Assumption (LH) include:

kernel satisfying Assumption (IF);

dot-product kernel on hyperspheres.

Moment-equivalent assumption [19] use a geometric argument with Dvoretzky-Milman Theorem to bound the test error from above under the assumptions:

Assumption (ME) ( L 2+ ϵ -L 2 Moment-Equivalent Assumption) . Fix a sample ( x i ) n i =1 . Suppose there exists some ϵ, κ > 0 such that:

for any k ∈ N and any function f ∈ H >k in the high-degree feature space, we have ∥ f ∥ L 2+ ϵ ≤ κ ∥ f ∥ L 2 .

(a) if ϵ > 2 , then no extra assumption is required;

∥

(

z

i

)

>k

Tr[

i

=1

,...,n

Σ

∥

2

Σ

2

>k

2

>k

]

with high probability, max i =1 ,...,n ∥ ( z i ) ≤ k ∥ 2 2 k = O k (1) .

While points 2-4 are implied by the weakened version of Assumption (GF), point 1 is a strong geometric assumption on the RKHS H . According to [19], the addition of point 1 can drop the logarithmic terms in the upper bound of test error under weak ridge, and thus strengthening some results in Table 1 from ˜ Θ( · ) to Θ( · ) . Examples satisfying Assumption (ME) includes:

rotational invariant kernel.

A.3 Between independent and dependent features: random features

A random feature setting can be considered as a midpoint between independent and dependent features: Let W ∈ R p × d be a random matrix, typically with i.i.d. random entries, and z ∈ R d be the input vector, usually comprising independent entries. We define the random feature x ∈ R p as a one-hidden-layer neural network: x def. = φ ( Wz ) , where φ : R → R represents an activation function acting entry-wise on Wz . Thus, the coordinates of the pre-activation are independent, while those of the post-activation are not.

However, it is worth noting the Gaussian Equivalent Property highlighted in [20], which asserts that in wide neural networks, the distribution of the feature vector x is approximately Gaussian. Consequently, it reduces to the case of independent features previously discussed.

A.4 Interpolation space

Many KRR literature [29, 30, 31, 33] introduced the notation of interpolation space and the source condition on the L 2 -integrable target function f ∗ . This can be easily translated to the notion of generalized vector norm.

Definition A.9 (Interpolation space) . Let s ≥ 0 be a real number. Define the interpolation space

3.

with high probability, max

≈

1

;

Evidently, if s 1 ≤ s 2 , then ∥ θ ∥ Σ 1 -s 2 ≤ ∥ θ ∥ Σ 1 -s 1 and hence H s 1 ⊃ H s 2 . The intuition is that, in the KRR setting, the space H s interpolates between the L 2 -space L 2 µ ( X ) ⊃ H 0 and the RKHS H = H 1 .

A.5 Source condition

Given the asymptotic of the eigen-decay λ k and the target coefficient θ ∗ , we can find the largest possible source coefficient s . For instance, if λ k = Θ k ( k -1 -a ) for some a > 0 and | θ ∗ k | = Θ k ( k -r ) , then we can choose s to be 2 r + a 1+ a :

Suppose λ k = Θ k ( e -ak ) , | θ ∗ k | = Θ k ( e -rk ) for a, r > 0 . Then we can choose s to be 2 r a +1 :

A.6 Kernel ridge regression

KRR can be regarded as ridge regression on the feature space, where the positive definite symmetric (PDS) kernel K sends each input x ∈ X to a vector/function K ( x, · ) in the corresponding reproducing kernel Hilbert space (RKHS) H , and the kernel regressor is simply a linear regressor in the RKHS H . Here we briefly explain how to translate the notations above in the kernel ridge regression (KRR) setting.

With abuse of notation, let µ be a (data-generating) distribution on an input space X . Let K : X × X → R be a positive definite symmetric kernel with corresponding reproducing kernel Hilbert space (RKHS) H . By Mercer decomposition, we have for all x, x ′ ∈ supp( µ ) ⊂ X :

where λ k 's are the eigenvalues of K indexed in decreasing order and ψ k : X → R 's are eigenfunctions of K forming an orthonormal basis on L 2 µ ( X ) : E x ∼ µ [ ψ k ( x ) ψ l ( x )] = δ kl for all k, l ∈ N . Hence for

each x ∈ supp( µ ) ⊂ X , K ( x, · ) is a feature vector in the RKHS H , which has an orthonormal basis { ϕ k = 1 √ λ k ψ k } p k =1 .

Let y = f ∗ ( x ) + ϵ be the labels where f ∗ ∈ L 2 µ ( X ) and ϵ is a centered random variable with finite variance independent to x . Let ( x i , y i ) n i =1 be a set of i.i.d. drawn input-output pairs. The kernel ridge regression is in the form:

which minimum admits the form:

where K x = ( K ( x, x i )) n i =1 ∈ R n , K = [ K ( x i , x j )] n i,j =1 ∈ R n × n and y = ( y i ) n i =1 . Note that if we write ψ x = ( ψ k ( x )) p k =1 ∈ R p , Ψ = [ ψ k ( x i )] ik ∈ R n × p and Λ = diag { λ k } p k =1 ∈ R p × p , by Mercer decomposition, we have K x = ΨΛ ψ x , K = ΨΛΨ ⊤ . We are ready to translate our linear ridge regression to the KRR setting. See Table 5 for details.

B Smoothness and spectral decay rate

In this section, we will explain the reason to assume the eigen-decay to be either polynomial or exponential in the paper.

Concretely, the "polynomial decay regime" arises as the Neural Tangent Kernel (NTK) limit of MultiLayer Perceptrons (MLPs) with an activation function of the form σ s ( t ) def. = max0 , t s , where s ≥ 1 , as studied in [4]. In cases where s is an integer, σ s represents the ( s -1) -th iterated anti-derivative of the ReLU function, making it exactly s times weakly differentiable.

Suppose X is the d -dimensional sphere S d def. = { u ∈ R d : | u | = 1 } (with its usual Riemannian metric). In this scenario, every neural network function f : S d → R belongs to the Sobolev space H s ( S d ) (see Proposition B.7 in Appendix B.2), characterized by having s continuous and integrable (weak) partial derivatives on S d . Additionally, the NTK limit manifests as a "radially symmetric kernel", specifically of the form k ( x, y ) = κ ( x ⊤ y ) for some suitable function κ : [ -1 , 1] → R .

It is worth noting that the choice of the smoothness level, and hence the decay rate, naturally emerges when approximately solving Partial Differential Equations (PDEs), as demonstrated in works such as [26, 15]. In these scenarios, it is desired that the kernel regressor exhibits the correct level of smoothness matching that of the theoretical solution to the PDE, with its higher-order derivatives also converging. It is emphasized that the solution to several PDEs only possesses finitely many derivatives, as seen in (viscosity) solutions to PDEs arising in stochastic control [37, 38].

In the remainder of this section, we justify the use of Assumptions (PE) and (EE) by establishing a connection between the spectral eigen-decay and the smoothness of the Reproducing Kernel Hilbert Space (RKHS), which is independent of the rest of the paper. Readers primarily interested in the proofs presented in Table 1 may opt to skip this section and proceed to Section C during their initial reading.

B.1 Notations

Fix a positive integer d ∈ N and let X be a non-empty subset of R d . We equip X with a regular Borel probability measure µ , which we specify shortly. We denote the space of (equivalence classes of) µ -square-integrable on R d by L 2 µ ( R d ) , equipped with its usual L 2 µ -norm ∥ · ∥ L 2 µ .

Consider a kernel κ : X × X → R with its associated to the RKHS ( H , ⟨· , ·⟩ κ ) (abbreviated by H ). We denote the induced norm on H by ∥ · ∥ κ . By the Mercer decomposition theorem, see [40, Theorem 2.30], there is some M ∈ N ∪ {∞} such that we may write

where ( ψ i ) i M =0 are normalized eigenfunctions of the linear operator T κ ( f )( x ) = ∫ x ∈X κ ( x, u ) f ( u ) du on L 2 µ ( X ) and each eigenvalue ( λ i ) ∞ i =0 is non-negative; which, form an orthonormal basis of H .

We motivate our analysis, by the following connection between the spectral decay of dot-product kernels on spheres and the smoothness of functions in their RKHS.

Example B.1 (Radial Kernels and Sobolev Spaces on Spheres) . In the setting of Examples B.3 and (B.5) , suppose that there is a function k : [ -1 , 1] → R such that

for each x, y ∈ S d . As shown in [41], κ is a kernel if and only if there is a summable sequence of non-negative real numbers ( α i ) ∞ i =0 satisfying k ( t ) = ∑ ∞ i =0 α i t i for each t ∈ [ -1 , 1] . Kernels of the form (9) are called radial. The RKHS H associated to κ consists of all functions f : S d → R with

for a square-summable real sequence ( α j,i ) ∞ ,J i j,i =1 ; i.e. ∑ ∞ i =0 ∑ J i j =1 α 2 j,i < ∞ (where ( α i,j ) ∞ ,J i i,j =1 depends only on the radial function k , and thus only on the radial kernel κ ; and ( ¯ ψ j,i ) ∞ ,J i j,i =1 are the

spherical harmonics described in Example B.3. As discussed in [24, Chapter 4], for any k > 0 u ∈ H belongs to the Sobolev space H k 0 ( S d ) if and only if: for each f ∈ H the coefficients ( µ j ) ∞ j =0 in the representation (10) satisfy

As discussed in [22, Lemma B.1] in this case the norms of H and on H s ( S d ) are equivalent 9 .

B.2 Preliminaries: regular domains and Sobolev spaces

We consider the case where X is a sufficiently regular open subset of the d -dimensional Euclidean space so that we may comfortably describe the smoothness of functions on X using classical notions of tools. We therefore assume the following.

Assumption (DR) (Domain Regularity) . Either one of the following holds:

(i) Compact manifold: ( X , g ) is a d -dimensional compact Riemannian manifold, with d ≥ 2 , and µ = V g ( X ) -1 V g is the uniform law on ( X , g ) and V g is its volume measure,

(ii) Bounded Euclidean domain: X is a non-empty bounded open domain in R d with Lipschitz boundary and µ is the normalized Lebesgue measure.

In cases (i)-(ii), X can be viewed as a Riemannian manifold (possibly with boundary).

In what follows, we let ∆ denote either of the following Laplacian, depending on which of assumption (DR) we have assumed. If (i) or (ii) holds then ∆ will denote the usual Laplacian-Beltrami operator on X , note that when (iii) holds then ∆ = ∑ d ∂ 2 2 . We denote the eigenfunctions of ∆

by ψ i ) i ∈ N whose respective eigenvalues λ 1 ≤ λ 2 ≤ . . . are arranged in a non-decreasing order. If (iii) holds, then ∆ will denote the Dirichlet Laplacian (see [11, Section 6.1.2]) on X (with its usual Euclidean Riemannian metric) which acts on the spaces of functions vanishing on the boundary ∂ X of X ; namely on the homogeneous Sobolev space H 1 0 ( X ) .

i =1 ∂x i ( ¯

Example B.2. Under assumption (i), the eigenfunctions of ∆ are the Hermite polynomials on L 2 µ ( R d )

For concreteness, the size, in L 1 µ -norm, of the Hermite polynomials is computed in [28, Theorem 2.1] where it it shown that ∥ ¯ ψ i ∥ L 1 µ ≲ ( i !) 1 / 2 i 1 / 2 (1 + O (1 /i )) .

Example B.3. Suppose that X is the d -dimensional sphere S d def. = { x ∈ R d +1 : ∥ x ∥ 2 = 1 } equipped with the (usual) Riemannian metric induced by inclusion into the d + 1 -dimensional Euclidean space R d +1 . Let µ be the normalized (uniform) Riemannian (volume) measure on R d . Then, ( ¯ ψ i,j ( x )) ∞ ,J i i =1 ,j =1 are an orthogonrmal basis of L 2 µ consisting of spherical Harmonics (counting multiplicities J i ∈ N + for each eigenvalue i ∈ N + ); that is the restriction of a homogeneous polynomial f : R d +1 → R satisfying ∆ f = 0 to there sphere S d . Spherical harmonics are not very large (in uniform norm) [17, Theorem 1.6] and one can show that ∥ ¯ ψ i,j ∥ ≲ λ ( d -1) / 4 i , for each i ∈ N + . See [2, Chapter 2] for further details on spherical harmonics.

B.2.1 Sobolev spaces on Riemannian manifolds

Let ( X , g ) be a compact (smooth) Riemannian manifold, possibly with a boundary ∂ X . For any k ∈ N + , we define C 2 k ( M ) be the set of smooth functions f on X satisfying

where µ is the Riemannian volume measure on ( X , g ) , ∇ is the covariant derivative thereon. Then, the Sobolev space H k 0 ( X , g ) is the Hilbert space obtained as the completion of C p k ( X ) with respect

to the norm

Remark B.4 (The Sobolev Spaces H k 0 ( X , g ) and H k ( X , g ) ) . Observe that when X is a compact Riemannian manifold without boundary, i.e. when ∂ X = ∅ ; then the condition f ( x ) = 0 for all x in the boundary set ∂ X holds vacuously. In this case, this condition can be ignored, and the spaces H k 0 ( X , g ) coincide with the usual Sobolev spaces H k ( X , g ) for Riemannian manifolds without boundary; where H k ( X , g ) is defined much the same way as H k 0 ( X , g ) but without requiring the homogeneous boundary condition in (11).

A more convenient expression for the norm of Sobolev spaces on spheres can be arrived at by manipulating the definition of the Laplacian. We summarize this norm in the next example.

Example B.5 (Sobolev spaces on spheres) . Fix k ∈ N + . Following the discussion on [2, page 120-121], the f ∈ H k 0 ( S d , g ) if and only if the following norm

is finite; where c d def. = ( d -2) / 2 . That is, the norms ∥ · ∥ k and ∥ · ∥ H k ( X ) are equivalent.

The key takeaway of Example B.5, is that the smoothness of a function on there sphere ( S d , g ) , i.e. the largest k for which f ∈ H k 0 ( S d , g ) , can be expressed in terms of the decay rate of its projection onto the basis of spherical harmonics. Indeed, this expression (12) implies that f ∈ H k 0 ( S d , g ) only if ∣ ∣ ∫ x ∈ S d f ( x ) ¯ ψ i,j ( x ) µ ( dx ) ∣ ∣ 2 ∈ O ( ( j + c d ) -2 k -ϵ ) for all ε > 0 .

This observation was used in [24, Chapter 4] and circa [22, Lemma B.1] to characterize the smoothness of functions in the RKHS associated to certain radial kernels, e.g. certain NTK limits. We now generalize this argument, and use it to relate the spectral decay rate of a kernel to the smoothnes/regularity of the functions in its associated RKHS.

B.3 Standardization operator

We require that the L 2 µ ( X ) norm of the eigenfunctions are non-vanishing and non-exploding.

Assumption (ND) (Non-degenerate L 2 -Norm) . Suppose that:

(i) Non-vanishing: 0 < inf i ∈ [ M ] ∥ ¯ ψ i ∥ L 2 µ ( X ) ,

(ii) Non-exploding: sup i ∈ [ M ] ∥ ¯ ψ i ∥ L 2 µ ( X ) < ∞ .

Again, if ( ¯ ψ i ) ∞ i =0 is orthonormal then (i) and (ii) necessarily hold and this is a non-assumption.

To generalize the discussion in Example B.1, we first define the linear standardization operator

Example B.6 (Standardization Is Inclusion for Radial Kernels on Spheres) . In the setting of Example B.1, we have that (up to reordering) ψ i,j = ¯ ψ i,j . Since, in that case, for k large enough on has H = H k 0 ( S d , g ) then A is simply the inclusion operator of H k 0 ( S d , g ) into L 2 µ ( S d ) .

In what follows, when we assume Assumption (DR) (iii), we will use C ∞ ( X ) to denote the set of smooth functions on the closure X of X in the norm topology on R d . Under Assumption (DR) (i)-(ii), we use C ∞ ( X ) to denote C ∞ ( X ) ; that is, the set of smooth functions on the smooth manifold X .

Proposition B.7 (Identification: Spectral Decay and Standardized Regularity) . Suppose that Assumption (DR) holds and that ( ψ i ) i M =0 are orthonormal in H .

Then, the linear operator A , defined in (13) , is an isometric embedding when M ≤ ∞ (resp. isometric isomorphism when M = ∞ ). Moreover, it characterizes the 'regularity' of functions in H via:

Sub-Exponential Decay: If κ has infinite-rank ( M = ∞ ) and there exists an r > 0 such that |⟨ f, ψ i ⟩ κ | ≲ i -r and inf ε> 0 lim i →∞ |⟨ f, ψ i ⟩ κ | i -r + ε = ∞ then

Exponential Decay: If κ has infinite-rank ( M = ∞ ) and there exists some r > 0 such that |⟨ f, ψ i ⟩ κ | ≲ e -r i , then

Finite-Rank:

(ii) If Assumption (DR) (ii) holds then: κ has finite rank ( M < ∞ ) then A ( f ) is smooth.

(iii) If Assumption (DR) (iii) holds then: κ has finite rank ( M < ∞ ) then A ( f ) is realanalytic.

A direct corollary of the first statement in Proposition B.7 guarantees that A preserves convergence. Consequentially, any finite-rank truncation or any sequence of learners converging to a limiting function in the RKHS do so if and only if their standardized versions do; moreover, both sequences converge at the same speed due to A being an isometric embedding.

Corollary B.8 (Preservation of Convergence Rates) . If ( f N ) N ∈ N is a sequence in H converging to some f ∈ H then: for each N ∈ N on has ∥ f N -f ∥ κ = ∥ A ( f N ) -A ( f ) ∥ L 2 µ ( R d ) .

We now are now prepared to prove Proposition B.7.

Proof of Proposition B.7. We first show that A is a linear isometric embedding (resp. linear isomorphism). This allows us to relay the spectral decay rate of any function f in H to that of its 'standardization' A ( f ) in L 2 µ ( X ) . We then use this identification, together with results from spectral theory (see [11] for references) to obtain our conclusion.

Step 1 A is a Linear Isometric Embedding (Resp. Linear Isomorphism):

Observe that A admits a linear left-inverse given by

In particular, A is a linear isomorphism if and only if M = ∞ ; since otherwise B is not a two-sided (linear) inverse. Furthermore, A is an isometric embedding (resp. isometric isomorphism when M = ∞ ) since due to the following. For each f ∈ H we have that

where (14) and (16) held by virtue of ( ψ i ) i M =0 being an orthonormal basis of H . From (14)- (17) we conclude that for each f ∈ H we have ∥ A ( f ) ∥ L 2 µ ( X ) = ∥ f ∥ κ . Thus, A is an isometric embedding.

Finite Rank Kernel:

Suppose that M < ∞ . We consider two cases in Assumption (DR) separately.

Under Assumption (DR) (ii), [11, Corollary 9.28] guarantees that each ¯ ψ 0 , . . . , ¯ ψ M is smooth. Since any finite sum of real analytic functions is again smooth then, in this case, A ( f ) is smooth.

Under Assumption (DR) (iii), [11, Theorem 6.8] each ¯ ψ 0 , . . . , ¯ ψ M is real-analytic. Since any finite sum of real analytic functions is again real-analytic then A ( f ) is real-analytic.

Infinite-Finite Rank Kernel:

Suppose that M = ∞ and fix s > 0 . By definition of ( ψ i ) ∞ i =0 and of the Sobolev space H s ( X ) we have that: for each f ∈ L 2 µ ( X )

Since f ∈ L 2 µ ( X ) if and only if ∥ f ∥ H s ( X ) is well-defined and finite. Consequentially, the elements of the Sobolev space H s ( X ) are characterized by the summability of the right-hand side of (18). That is, for each f ∈ L 2 µ ( X ) one has

By Assumption (ND), both 0 < C def. = inf i ∈ [ M ] ∥ ¯ ψ i ∥ L 2 µ ( X ) and ¯ C def. = sup i ∈ [ M ] ∥ ¯ ψ i ∥ L 2 µ ( X ) < ∞ are well-defined, non-negative real numbers. Therefore, for every i ∈ N

From (20) we deduce that: for each f ∈ L 2 µ ( X ) and every s > 0

Consequentially, (18) implies that

Step 2 - Asymptotics of Laplacian Eigenspectrum Under Assumption (DR) (ii), Weyl's law for compact Riemannian manifolds (see e.g. [11, Corollary 9.35]) implies that there are dimensional (depending on d , X , and on g ) constants 0 < C g W :1 ≤ C g W :2 < ∞ such that: for each i ∈ N

where v d def. = V ( { x ∈ R d : ∥ x ∥ 2 ≤ 1 } ) = π d/ 2 Γ(1+ d/ 2) and where V is the d -dimensional Lebesgue measure on R d .

Under Assumption (DR) (iii), Weyl's law (see e.g. [11, Theorem 6.27]) implies that: there are dimensional (depending on d and on X ) constants 0 < ˜ C W :1 ≤ ˜ C W :2 < ∞ such that: for each i ∈ N

With some abuse of notation, for i = 1 , 2 , we write C W : i for C g W : i or ˜ C W : i depending on which of Assumptions (DR) (ii) or (iii) are being used. We therefore, wild a mild abuse of notation, we collect (22) and (23) into the following single expression:

where the constant c > 0 is defined by

Combining (21) with (24), we find that

Pulling out the common c 2 s/d factor in all summands on the right-hand side of (25) we find that: for any s > 0 any f ∈ L 2 µ ( X ) belongs to H s ( X ) if and only if

Sub-Exponential Decay: Suppose that there exists an r > 0 such that |⟨ f, ψ i ⟩ κ | ≲ i -r and for every ε > 0 we have lim i →∞ |⟨ f, ψ i ⟩ κ | i -r + ε = ∞ then (26) implies that

If Assumption (DR) (ii) or (iii) holds the Sobolev embedding theorem for Riemannian manifolds with boundary, as formulated in [11, Theorem 9.26], implies that H s r 0 ( X ) ⊆ C k r ( X ) where s r -d/ 2 > k r for some k r ≥ 0 to be determined shortly; where for us s r def. = (1 + r ) d/ 2 . Therefore, dr/ 2 > k r . Consequentially, for each f ∈ H we have that

Exponential Decay: Suppose that there exists some r > 0 such that |⟨ f, ψ i ⟩ κ | ≲ e -r i . Then, then (26) implies that A ( f ) ∈ ∩ s> 0 H s 0 ( X ) . By the Sobolev Embedding Theorem, in the bounded case ([11, Theorem 9.2.6])

C Master inequalities

In this section, we will present and proof the Master inequalities, which is crucial in our argument.

Our approach bounds each term in the bias-variance decomposition in line (5):

under minimal assumptions.

C.1 More notations

We first introduce some more notations. We define the empirical covariance matrix:

the whitened/isotropic feature vector where E µ [ zz ⊤ ] = I p ∈ R p × p . Denote the whitened input matrix by Z def. = XΣ -1 / 2 ∈ R n × p , then the rows of Z are isotropic centered i.i.d. random vectors.

We introduce the notation of a generalized vector norm, which will be frequently used in this paper: √

Definition C.1 (Generalized vector norm) . We denote ∥ v ∥ M def. = v ⊤ Mv for any positive definite matrix M and vector v .

Remark C.2 (Examples) . Let v ∈ R p and we have ∥ v ∥ 2 2 = v ⊤ I p v = ∥ v ∥ 2 I p , where I p is the identity matrix. Another example is that the euclidean norm of x equals to the Σ -norm of its whitened counterpart z :

Furthermore, if M 1 ≼ M 2 , we have ∥ v ∥ M 1 ≤ ∥ v ∥ M 2 . In particular, since Σ ≼ λ 1 I p , we have

Lastly, it is convenient to write the expected values in a generalized vector norm: fix a distribution µ , an input block X ∈ R n × p , where its rows are drawn i.i.d. by µ , and denote by ˆ µ = 1 n ∑ n i =1 δ x i the empirical distribution of µ , then for any v ∈ R p , we have

Definition C.3 (Truncated Terms) . For any vector v ∈ R M , write v = v ≤ k ⊕ v >k ∈ R k ⊕ R M -k . For any square matrix M ∈ R M × M , write M = ( M ≤ k ∗ ∗ M >k ) with M ≤ k ∈ R k × k and M >k ∈ R ( M -k ) × ( M -k ) . Analogously, for any non-square matrix M ∈ R N × M , we write M = M ≤ k ⊕ M >k ∈ R N × k ⊕ R N × ( M -k ) . Also write M -k def. = M ≤ k -1 ⊕ M >k ∈ R N × ( M -1) and M l : j = [ M i,k ] j k = l +1 ∈ R N × ( j -l ) .

Denote A def. = XX ⊤ + nλ I n ∈ R n × n . For any k ∈ N , denote

The symmetric matrix A k plays an important role in the analysis. Intuitively, when k is chosen appropriately, A k is approximately equal to a scaled identity, with spectrum bounded by some constants from below and from above.

We introduce a few more definitions that have been used in the previous literature.

Definition C.4 (Effective ranks [6]) . Let k ∈ N . Define two quantities:

and denote by

Definition C.5 (Concentration coefficients [7]) . Define the quantities: 11

where s i ( · ) denotes the i -th largest singular value of the matrix.

To enhance readability, we will omit the subscript · n,k when the context is clear. The intuition of the concentration coefficients is that for fixed k ∈ N and n large enough, ζ and ξ concentrate around some constants; when k is smaller but scales with n , ρ converges to a constant, as it essentially represents the condition number of A k . As mentioned earlier, the spectrum of A k is bounded. Therefore, selecting an appropriate k ensures that all three concentration coefficients converge to certain constants.

C.2 Statements and proofs

Now we are ready to present the Master inequalities, which are proved in [46, 7]:

Proposition C.6 (Master inequality for B , [46, 7]) . Let k ∈ N be an integer. For any δ ∈ (0 , 1) , with probability at least 1 -δ ,

Proof. We begin by bounding the terms that appear in Lemma H.8: plug in Definition C.5 of ρ, ζ, ξ , since

we have

We regroup the terms into:

By Lemma G.6, we can further simplify the expression into a high probability bound: for any δ ∈ (0 , 1) , with a probability at least 1 -δ , it holds that

Note that 1 /δ > 1 and we obtain the result.

Proposition C.7 (Master inequality for V , [46, 7]) . Let k ∈ N be an integer. Then the variance is bounded by:

where σ 2 def. = E [ ϵ 2 ] ≥ 0 is the noise level.

Proof. We start from Lemma H.9:

C.3 Sanity check

Recall that B def. = ∥ ˆ θ ( X θ ∗ ) -θ ∗ ∥ 2 Σ . Let k ∈ N be an integer. For any δ ∈ (0 , 1) , with probability at least 1 -δ , we have

We see that:

the regressor barely learns the tail of the target (the eigenfunctions with small eigenvalues), thus the first term ∥ θ ∗ >k ∥ 2 Σ >k in the upper bound is just a multiple of the Σ -norm square (or L 2 µ -norm square in KRR setting) of the target's tail;

the second term indicates that the regressor learns well if the ridge λ is small, showing the trad-off between (noiseless) interpolation and regularization;

the term ∥ ∥ θ ∗ ≤ k ∥ ∥ 2 Σ -1 ≤ k corresponds to norm square of the target's head in the interpolation space H 2 . This term will explain the so-called saturation effect reported in [29] where the bias ceases to improve once the source coefficient s in Assumption (SC) surpasses 2. Consequently, the bias term B bound in Table 1 is closely related to the constant min s, 2 .

[46, 7] also derived a non-asymptotic bound of V . Recall that V def. = E ϵ [ ∥ ˆ θ ( ϵ ) ∥ 2 Σ ] . Let k ∈ N be an integer. Then

We see that:

the upper bound is proportional to noise level σ 2 = E [ ϵ 2 ] ≥ 0 ;

we will show later that for n large enough, the second term is approximately k n , hence the decay rate of V is at most O n,k ( n -1 ) .

The derivation of Eq (30) and (31), which we refer to as the Master inequalities in this paper, is purely algebraic, and the statement holds regardless of the assumptions on the eigen-decay and the features. One can optimally choose the integer k ∈ N to achieve a tight upper bound, as we will demonstrate later in this paper. Surprisingly, the Master inequalities alone are sufficient to establish tight bounds for most cases under various settings.

D Over-parameterized regime

In this section, we provide the proof for the upper bounds presented in Table 1. For all the statements in this section, the big O notation ˜ O ( · ) can be replaced by O ( · ) if a +2 = 2 r .

A brief summary of the propositions can be found in Table 3.

D.1 Bias under strong ridge

Proposition D.1 (Asymptotic upper bound of the bias term with polynomial decay) . Suppose: λ k = Θ k ( k -1 -a ) , | θ ∗ k | = O k ( k -r ) and λ = Θ k ( k -b ) for some a, b, r > 0 . Furthermore, suppose Assumption (GF) (or resp. (IF)) holds, and δ, ξ, ζ = Θ n,k (1) with probability at least 1 -φ . Then, for any t ∈ (0 , 1] , with probability at least 1 -δ -φ (or resp. 1 -δ -φ -e -n ), the bias is bounded by:

for some constant

with with following lower bound:

Proof. For simplicity, we only prove the statement for all a, r except when a +2 = 2 r . In such case, the argument follows similarly with O ( · ) replaced by ˜ O ( · ) .

For any k, n ∈ N , pick k = ˜ Θ n,k ( n t ) for some t ∈ (0 , 1] . Suppose the event δ, ξ, ζ = Θ n,k (1) happens, then:

B

where we apply Proposition C.6, Lemma G.8 in line (34); we apply Lemma H.10: ∥ ∥ θ ∗ ≤ k ∥ ∥ 2 Σ -1 ≤ k

= Θ k ( k (2+ a -2 r ) + ) where x + def. = max { x, 0 } , as well as plug in Tr[ Σ >k ] = Θ k ( k -a ) in line (35); we plug in k = ˜ Θ n,k ( n t ) for some t ∈ (0 , 1] in line (36).

̸

Hence the bias B has a decay in form of O n ( ρ 3 n -C ) where

is a constant defined in nested minima and maxima. To bound C from below, we make two case distinctions: b < 1 + ta or b ≥ 1 + ta .

For b < 1 + ta , we have

For b ≥ 1 + ta , we have

Replacing C by its lower bound, we obtain the claimed decay.

Proposition D.2 (Asymptotic upper bound of bias under strong ridge and polynomial eigen-decay) . Suppose: λ k = Θ k ( k -1 -a ) , | θ ∗ k | = O k ( k -r ) and λ = Θ k ( k -b ) for some a, r > 0 and b ∈ (0 , 1 + a ] . Fix a constant δ ∈ (0 , 1) . If, additionally, Assumption (GF) (or resp. (IF)) holds, then, with probability at least 1 -δ -o ( n -1 ) (or resp. 1 -δ -3 e -c 1 n ), the bias is bounded by:

with s def. = 2 r + a 1+ a .

Proof. Use Proposition D.1 as the backbone, and use Propositions G.1 and G.2 to obtain the high probability guarantee for ξ, ζ = Θ n,k (1) . Then for different value of b , we plug different values

of t ∈ (0 , 1] and set k = Θ n,k ( n t ) , such that ρ = ρ n,k = ˜ O n,k (1) by Proposition G.3, in order to obtain different decay rates of bias.

Assume b ∈ [0 , a +1) . Pick k = Θ k,n ( n t ) with t = b 1+ a . As above, it holds that ρ = ˜ Θ n,k (1) with high probability. This time, we plug in t = b 1+ a in the definition of C . By the choice of b , we have b < 1 + ta and hence

Finally, assume b = a +1 . Pick k = Θ k,n ( n t ) with t = b 1+ a = 1 . But this time, since b ≥ 1 + ta , we plug in t = b 1+ a = 1 into the second lower bound of C in line (33):

The argument for exponential decay follows analogously as above. Recall Assumption (EE): λ k = Θ k ( e -ak ) for some a > 0 . Suppose | θ ∗ k | = Θ k ( e -rk ) and λ = Θ n ( 1 n e -bn ) 12 for some b, r > 0 . Then the asymptotic bounds follows similarly to the polynomial case. For sake of completeness, in this section, we list out some propositions for illustration.

Proposition D.3 (Asymptotic upper bound of bias under strong ridge and exponential eigen-decay) . Suppose λ k = Θ k ( e -ak ) , | θ ∗ k | = Θ k ( e -rk ) and λ = Θ n ( e -bn ) for some a, r > 0 and b ∈ (0 , a ) . If Assumption (GF) (or resp. (IF)) holds, then with probability at least 1 -δ -o n ( 1 n ) (or resp. 1 -δ -3 e -c 1 n ), the bias is bounded by:

̸

for some constants c 1 > 0 , where s = 2 r a +1 > 0 is the source coefficient.

Remark D.4 . Note that the factor n in the case of a = 2 r does not play a big role asymptotically: by taking a slightly smaller ˜ b < b , the decay is exponential in ˜ bn .

Proof. By Proposition C.6, with probability at least 1 -δ (or resp. 1 -δ -e -n ), the bias is bounded by:

By Proposition G.1 and G.2, there exists some c > 1 such that, if cβ k k log k ≤ n (or resp. k/c ≤ n ), then with probability at least 1 -2 exp ( -c β k n k ) (or resp. 1 -2 exp ( -c 1 n ) ), it holds that

Fix a δ ∈ (0 , 1) . With probability at least 1 -δ -2 exp ( -c β k n k ) (or resp. 1 -δ -e -n -2 e -c 1 n ), it holds that:

where the last inequality comes from Proposition G.4. For b < a , set t def. = b a < 1 .By Proposition G.4 again, we have ρ = Θ k,n (1) with probability at least 1 -o n ( 1 n ) (or resp. 1 -e -n ). Set s def. = 2 r + a a be the source coefficient and we write:

̸

D.2 Bias under weak ridge

Proposition D.5 (Asymptotic upper bound of bias under weak/effectively no ridge for s > 1 ) . Suppose Assumption (GF) (or resp. (IF)) holds, and the source coefficient s > 1 . Then with high probability, the bias has decay:

Proof. By definition, rewrite the bias into:

Denote P λ def. = X ⊤ ( XX ⊤ + nλ I n ) -1 X ∈ R p × p . By Sherman-Morrison-Woodbury formula,

is a positive definite matrix such that the matrix I p -P λ is monotonic increasing in λ , that is:

for all ˜ λ ≥ λ > 0 , since the map λ ↦→ λ λ + x is monotone. Hence, for t ∈ [1 , min { 2 , s } ) 13 ,

where we rewrite line (40) in line (42); we apply the monotonicity from line (41) in line (43); we use the fact that ∥ M 1 M 2 ∥ op = ∥ M 2 M 1 ∥ op for symmetric matrices M 1 , M 2 in line (44); we use Lemma H.17 to pull out the power ( t -1) from the matrix to its operator norm in line (45); we use the fact that ∥ ∥ ∥ M 1 / 2 1 M 1 / 2 2 ∥ ∥ ∥ 2 op = ∥ ∥ ∥ M 1 / 2 1 M 1 / 2 2 ( M 1 / 2 1 M 1 / 2 2 ) ⊤ ∥ ∥ ∥ op = ∥ ∥ ∥ M 1 / 2 1 M 2 M 1 / 2 1 ∥ ∥ ∥ op for any positive definite symmetric matrices 14 M 1 , M 2 in line (46); by the monotonicity from line (41) again and the fact that Σ 1 / 2 is positive definite, we choose ˜ λ ≥ λ ≽ λ n to be strong in the sense of polynomial/exponential eigen-decay in line (47): ˜ λ = Θ n ( n -1 -a ) for polynomial case and ˜ λ = Θ n ( e -an ) for exponential case.

Let v ∈ S p -1 ⊂ R p be a unit vector such that ∥ ∥ Σ 1 / 2 ( I p -P ˜ λ ) Σ 1 / 2 ∥ ∥ 2 op = ∥ ∥ Σ 1 / 2 ( I p -P ˜ λ ) Σ 1 / 2 v ∥ ∥ 2 2 . By definition, we have

hence the vector Σ 1 / 2 v lies on the interpolation space H 2 . Note that the expression

is just the bias term of another task Σ 1 / 2 v with source coefficient 2 and ridge ˜ λ = Ω n ( λ n ) on the same dataset X . Apply Proposition D.2 with polynomial decay and Proposition D.3 with exponential decay on that new task with strong ridge ˜ λ , with high probability, the above term has a decay rate:

Plug in line (48) into line (47) to obtain the decay rate of the original bias: with high probability:

This holds for any t ∈ [1 , min { s, 2 } ) , hence we conclude that

For polynomial eigen-decay, we can still apply the Master inequality for B to obtain another upper bound:

Proposition D.6 (Asymptotic upper bound of bias under weak/no ridge for any s > 0 ) . Suppose: λ k = Θ k ( k -1 -a ) , | θ ∗ k | = O k ( k -r ) and λ = Θ k ( k -b ) for some a, b, r > 0 . Fix a constant δ ∈ (0 , 1) . If, additionally, Assumption (GF) (or resp. (IF)) holds, then, with probability at least 1 -δ -O n ( 1 log n ) (or resp. 1 -δ -3 e -c 1 n ), the bias is bounded by:

Proof. The proof is similar to Proposition D.2. Take k = Θ k,n ( n log n ) (or resp. k = n/c for some constant c > 1 ) and control ξ, ζ by Propositions G.1 and G.2. Now, by Proposition G.3, with probability at least 1 -c 1 log n , since b ≥ 1 + a ,

(or resp. ρ = O n,k (1) .) Now by Proposition D.1, since b ∈ [1 + a, ∞ ] and we choose t = 1 , with high probability,

Plug in ρ = ˜ O n,k ( n a ) and t = 1 to obtain:

The index in the above bound can be negative, causing the upper bound vacuous. By line (41), since B = ∥ ( I p -P λ ) θ ∗ ∥ 2 Σ ≤ ∥ θ ∗ ∥ 2 Σ , there is a trivial bound that B = O n (1) . Combining both results, we have

D.3 Variance with polynomial eigen-decay

Proposition D.7 (Asymptotic upper bound of the variance term with strong ridge) . Suppose λ k = Θ k ( k -1 -a ) , | θ ∗ k | = Θ k ( k -r ) , and λ = Θ n ( n -b ) for some a, r > 0 and b ∈ (0 , a + 1) . Furthermore, suppose Assumption (GF) (or resp. (IF)) holds. With probability at least 1 -o n ( 1 n ) (or resp. 1 -3 e -c 1 n ), the variance term is bounded:

where

(or resp.

for n ∈ N large enough.

Proof. We divide the case where Assumption (GF) or (IF) holds.

Suppose Assumption (GF) holds. By Proposition G.2, there exists constants c > 1 , c 1 > 0 such that, for k ≤ n c log n , with probability at least 1 -8 e -c 1 n/k , it holds that

Also, by definition, it always holds that

By Proposition C.7,

Let c > 1 be the constant in Proposition G.3. Choose k = ⌊ b 1+ a n ⌋ ≤ n c log n for n ∈ N large enough. By Proposition G.3, with probability at least 1 -o n ( 1 n ) ,

by the choice of b ∈ (0 , 1 + a ) . Combine the result with Lemma H.12, with probability at least 1 -o n ( 1 n ) , we have

to conclude the claim.

Suppose Assumption (IF) holds. The argument follows analogously. As in Proposition G.2, choose k = ⌊ b 1+ a ⌋ < n/c for n ∈ N large enough. With probability at least 1 -2 e -c 1 n , we have

By Proposition G.3, with probability at least 1 -e -n ,

by the choice of b ∈ (0 , a +1) . By possibly choosing a new constant c 1 > 0 , with probability at least 1 -3 e -c 1 n , we have ξ, ζ, ρ bounded. The rest of the argument follows similarly.

Proposition D.8 (Asymptotic upper bound of variance term with weak/no ridge) . Suppose λ k = Θ k ( k -1 -a ) , | θ ∗ k | = Θ k ( k -r ) , and λ = Θ n ( n -b ) for some a, r > 0 and b ∈ [ a +1 , ∞ ] . Furthermore, suppose Assumption (GF) (or resp. (IF)) holds. With probability at least 1 -o n ( 1 log n ) (or resp. 1 -3 e -c 1 n ), the variance term is bounded:

Proof. We divide the case where Assumption (GF) or (IF) holds.

Suppose Assumption (GF) holds. By Proposition G.2, there exists constants c > 1 , c 1 > 0 such that, for k ≤ n c log n , with probability at least 1 -8 e -c 1 n/k , it holds that

Choose k = ⌊ n c log n ⌋ . Then by Proposition G.3, with probability at least 1 -o n ( 1 log n ) , it holds that

by the choice of b . Moreover, since k = n c log n , by Proposition C.7, we have

Suppose Assumption (IF) holds, choose k = ⌊ n/c ⌋ for some constant c > 1 as in Propositions G.3. with probability at least 1 -3 e -c 1 n , it holds that

Then by Proposition C.7, we have

D.4 Variance with exponential decay

Proposition D.9 (Asymptotic upper bound of the variance term with strong ridge) . Suppose λ k = Θ k ( e -ak ) , | θ ∗ k | = Θ k ( e -rk ) , and λ = Θ n ( 1 n e -bn ) for some a, r > 0 and b ∈ (0 , a ) . Furthermore, suppose Assumption (GF) (or resp. (IF)) holds. With probability at least 1 -o n ( 1 n ) (or resp. 1 -3 e -c 1 n ), the variance term is bounded:

for n ∈ N large enough.

Proof. The argument is similar to Proposition D.7. The difference is the bound of ρ and the second term in Proposition C.7: if ξ, ζ = Θ n,k (1) , by Lemma H.12,

Choose k = ⌊ b a ⌋ < 1 . Combine Proposition G.4 and the above inequality to obtain the result.

E Matching lower bound

In this section, we provide the proof for the matching lower bounds presented on Table 1. A quick summary of propositions can be found in Table 4.

E.1 Bias under weak ridge and source coefficient between 1 and 2

Proposition E.1 (Asymptotic lower bound of bias, Proposition 4.4 in [31], Theorem 3.4 in [33]) . Suppose Assumption (PE)/(EE) holds. Then

̸

where s is the source coefficient.

Remark E.2 . Note that for both polynomial (with 2 r = 2 + a ) and exponential decays (with 2 r = a ), by Proposition D.5, we have B = O ( λ max { s, 2 } n ) for source coefficient s ≥ 1 . Hence when s ∈ [1 , 2] , the upper and lower bound matches.

Proof. Recall the expression of the bias in line (40):

where P def. = X ⊤ ( XX ⊤ + nλ I p ) -1 X . Recall the source coefficient s satisfies: ∥ θ ∗ ∥ Σ 1 -s < ∞ . By definition of operator norm, we have:

By [43], ∥ M 1 -M 2 ∥ op ≥ s n +1 ( M 1 ) for any operator M 2 with rank at most n . Note that P λ ∈ R p × p is of rank n , hence

E.2 Independent features and prior signs

Instead of bounding the bias from below in the worst case scenario, we can also bound it in an average sense. In this case, we need to pose an assumption on the target coefficient θ ∗ .

Assumption (PS) (Prior Signs) . Assume target coefficient θ ∗ ∈ R p is drawn from a distribution ϑ where its entries θ ∗ k 's are drawn independently of each other and have the same distributions as the random variables -θ ∗ k 's.

̸

Proposition E.3 (Asymptotic lower bound of bias with polynomial decay and independent features) . Suppose Assumption (PS) holds and λ k = Θ k ( k -1 -a ) , E θ ∗ [ ( θ ∗ k ) 2 ] = O k ( k -2 r ) and λ = Θ n ( n -b ) for some a, b, r > 0 . Fix a constant δ ∈ (0 , 1) . If, additionally, Assumption (IF) holds, then, with probability at least 1 -δ -11 e -c 1 n , the bias is bounded by:

for s def. = 2 r + a 1+ a .

Proof. By Lemma H.14, it holds that:

Hence, we want to bound the term s n ( A -l ) -1 and ∥ ∥ z ( l ) ∥ ∥ 2 2 from above.

If Assumption (IF) holds, then apply Lemma G.7 on A -l instead of A , l = 1 , ..., p , with probability at least 1 -2 e -n ,

By Lemma H.7, with probability at least 1 -2 e -n , we have,

Plug in the above two inequalities, by union bound, with probability at least 1 -4 e -n ,

where r k = Tr[ Σ >k ] ∥ Σ >k ∥ op for any k . If we choose k = n/c for some constant c > 1 such that A k = Θ n ( nλ n ) , then Tr[ Σ >k ] = Θ n (Tr[ Σ >n ]) . Hence

By Lemma H.15, with probability at least 1 -8 e -n ,

where B is defined as in Theorem H.16. since r k = Tr[ Σ >k ] ∥ Σ >k ∥ op = Θ k ( kλ k λ k ) = Θ k ( k ) , hence the fraction r k n = Θ n,k (1) and r k satisfies the condition of Theorem H.16. By Theorem H.16, we have upper bound B matching the lower bound B :

By Propositions C.6, D.2 and D.6, we have, with probability at least 1 -δ -3 e c 1 n ,

All together, with probability at least 1 -δ -3 e -c 1 n -8 e -n ,

Proposition E.4 (Asymptotic lower bound of bias with exponential decay and independent features under strong ridge) . Suppose Assumption (PS) holds and λ k = Θ k ( e -ak ) , E θ ∗ [ ( θ ∗ k ) 2 ] = O k ( e -2 rk ) and λ = Θ n ( e -bn ) for some a, r > 0 and b ∈ (0 , a ) . Fix a constant δ ∈ (0 , 1) . If, additionally, Assumption (IF) holds, then, with probability at least 1 -δ -7 e -c 1 n , the bias is bounded by:

for s def. = 2 r a +1 .

Proof. The proof is similar to Proposition E.3. By Lemma H.14, it holds that:

By Lemma H.7, with probability at least 1 -2 e -n , we have,

By the choice of b ∈ (0 , a ) :

Plug in the above two inequalities, with probability at least 1 -2 e -n ,

where r k = Tr[ Σ >k ] ∥ Σ >k ∥ op = Θ k (1) . By Lemma H.15, with probability at least 1 -4 e -n ,

where B is defined as in Theorem H.16. since r k = Tr[ Σ >k ] ∥ Σ >k ∥ op = Θ k (1) , pick k such that r k > 1 . This satisfies the condition of Theorem H.16. By Theorem H.16, we have upper bound B matching the lower bound B :

By Propositions C.6, and D.3, we have, with probability at least 1 -δ -3 e c 1 n ,

All together, with probability at least 1 -δ -3 e -c 1 n -4 e -n ,

Lemma E.5 (Asymptotic lower bound of variance with polynomial eigen-decay and independent features) . Suppose λ k = Θ k ( k -1 -a ) , λ = Θ n ( n -b ) . Additionally, suppose Assumption (IF) holds. Then with probability at least 1 -ce -n/c , it holds that

Proof. This is a consequence of Propositions C.7, D.7 Lemma H.13 and Theorem H.16: with probability at least 1 -3 e -c 1 n , by choosing k = { ⌈ n b 1+ a ⌉ , λ strong n/c , λ weak , we have

where V def. = k n + n Tr[ Σ 2 >k ] ( nλ +Tr[ Σ >k ]) 2 is defined as in Theorem H.16. Since the polynomial eigen-decay λ k satisfies the condition in Theorem H.16, we have

By Lemma H.13, V = { Ω n ( n -1+ b 1+ a ) , λ strong Ω n (1) , λ weak . By taking a larger constant c > 0 , the above events hold with probability at least 1 -ce -n/c .

Lemma E.6 (Asymptotic lower bound of variance with exponential eigen-decay and independent features) . Suppose λ k = Θ k ( e -ak ) , λ = Θ n ( e -bn ) . Additionally, suppose Assumption (IF) holds. Then with probability at least 1 -ce -n/c , it holds that

Proof. The proof follows a similar pattern to that of Proposition E.5. Ensure that the exponential eigen-decay also meets the condition outlined in Theorem H.16. Consequently, V = Θ n,k ( V ) , and the remaining argument proceeds accordingly.

F Under-parameterized regime

For the under-parameterized regime, one can clearly use the Master inequalities in Section C to bound the test error from above by setting k = p < n . However, one can even prove an asymptotic convergence in the following way.

we first prove the convergence of the empirical covariance matrix.

Lemma F.1 (Convergence of ˆ Σ ) . Suppose Assumption (GF) (or resp. (IF)) holds. Fix an ϵ ∈ (0 , 1) . Then there exists a constant c > 1 such that, for p < n 1 -ϵ c log n (or resp. p < n 1 -ϵ /c ), with probability at least 1 -o n ( n -ϵ ) , it holds that

where Σ def. = E [ xx ⊤ ] ∈ R p × p and ˆ Σ def. = 1 n ∑ n i =1 x i x ⊤ i ∈ R p × p .

Proof. We argue similarly as in Proposition G.2. Fix an ϵ ∈ (0 , 1) .

Suppose Assumption (GF) holds. We have ∥ z ∥ 2 2 ≤ β p p by definition of β k . Apply Theorem H.3 on the whitened input block Z ∈ R n × p : there exists some constant c 1 > 0 such that, with probability at least 1 -2 pe -c 1 t 2 , we have:

By setting t = √ log n c 1 and the choice of p < n 1 -ϵ c log n , with probability at least 1 -2 cn ϵ log n , we have

Hence, for n large enough, with probability at least 1 -2 cn ϵ log n , it holds that:

Suppose Assumption (GF) holds. By Lemma H.7 and union bound, we have ∥ z i ∥ 2 2 ≤ 2 p for all i = 1 , ..., n with probability 1 -2 ne -c 1 n . When this event happens, apply Theorem H.3 on the whitened input block Z ∈ R n × p , the rest of the argument follows similarly.

Proposition F.2 (Convergence of the bias and variance terms) . Suppose Assumption (GF) (or resp. (IF)) holds. Then there exists an ϵ ∈ (0 , 1) and a constant c > 1 such that, for p < n 1 -ϵ c log n (or resp. p < n 1 -ϵ /c ), with probability at least 1 -o n ( n -ϵ ) , it holds that

and hence:

Proof. We begin with B . By line (40) and the formula M -1 1 -M -1 2 = M -1 1 ( M 1 -M 2 ) M -1 2 , rewrite the bias term B as:

Denote ∆ = ( ˆ Σ -Σ )( λ I p + Σ ) -1 . Note that

We apply the matrix difference formula iteratively:

we write:

Finally, write

Now we argue similarly for V . By Lemma H.19:

We argue similarly as in Proposition F.2: with probability at least 1 -o n ( n -ϵ ) , we have

Plugging in the values of n, p and other metrics a, b, r in Proposition F.2, we can obtain the

G Concentration of features

In the following, we shall give sufficient conditions to control the concentration coefficients ζ n,k , ξ n,k and ρ n,k . Note that ρ n,k , and ξ n,k depends on the whitened/isotropic features z but not on the spectrum Σ . We start from the easiest to the most difficult.

Proposition G.1 (Control on ξ n,k ) . Let k ∈ N be an integer. Recall that ξ n,k def. = s 1 ( Z ⊤ ≤ k Z ≤ k ) n . If Assumption (GF) (or resp. (IF)) holds, then with probability at least 1 -2 exp ( -1 2 β 2 k n ) (or resp. 1 -2 exp ( -c 1 kn ) ), it holds that

Proof. Since the largest singular value is larger than the average of the singular values,

By Lemma G.5, with respective probability,

Proposition G.2 (Control on ζ n,k ) . Let k ≤ n be an integer. Recall that ζ n,k def. = s 1 ( Z ⊤ ≤ k Z ≤ k ) s n ( Z ⊤ ≤ k Z ≤ k ) . If Assumption (GF) (or resp. (IF)) holds, then there exists some c > 1 such that, if cβ k k log k ≤ n (or resp. k/c ≤ n ), then with probability at least 1 -2 exp ( -c β k n k ) (or resp. 1 -2 exp ( -c 1 n ) ), it holds that

Proof. For the case where Assumption (GF) holds, see Lemma 2 in [7]. Suppose Assumption (IF) holds, then by Theorem H.2, there exists constants C 1 , C 2 > 0 , such that, with probability at least 1 -2 exp ( -C 1 t 2 ) , the spectrum of random matrix Z ≤ k ∈ R n × k is bounded:

Set t = √ n/ 4 and c = 1 16 C 2 so that if k/c ≤ n , the bound becomes:

and hence ζ n,k ≤ ( 3 2 √ n ) / ( 1 4 √ n ) = 6 with a probability at least 1 -2 exp ( -C 1 16 n ) . Set c 1 = C 1 16 and c 2 = 6 to conclude the statement.

Proposition G.3 (Control on ρ n,k ) . Let k ≤ n be an integer. Recall that ρ n,k def. = n ∥ Σ >k ∥ op + s 1 ( A k ) s n ( A k ) . If Assumption (PE) holds and Assumption (GF) (or resp. (IF)) holds, then there exists some c > 1 such that, if k = n c log n (or resp. k = n/c ), then with probability at least 1 -O n ( 1 log n ) (or resp. 1 -3 e -n ), it holds that

If λ = Θ n ( n -b ) with b ∈ (0 , 1 + a ] and Assumption (GF) holds. If k = ⌈ n b 1+ a ⌉ , with probability at least 1 -O n ( 1 n ) , it holds that

Proof. The statement where Assumption (GF) holds is proved in Theorems 4 and 5 in [7]. The statement where Assumption (IF) holds can be proved proved via Lemma G.7: since:

with a probability of at least 1 -3 e -n .

Proposition G.4 (Control on ρ n,k under exponential eigen-decay) . Let k ≤ n be an integer. If λ k = Θ k ( e -ak ) , λ = Θ n ( e -bn ) for some a > 0 and b ∈ (0 , a ) , and Assumption (GF) (or resp. (IF)) holds, choose k = ⌈ b a n ⌉ , then with probability at least 1 -o n ( 1 n ) (or resp. 1 -exp( -n ) ), it holds that

Proof. For the case where Assumption (GF) holds, by Corollary 1 in [7], with probability at least 1 -o n ( 1 n ) , it holds that

For the case where Assumption (IF) holds, the upper bound of s 1 ( A k ) is proven in Lemma G.7. In both cases, the derivation of bounding ρ is the same:

Lemma G.5 (Lemma 1 in [7]) . Let k be an integer. Suppose Assumption (GF) (or resp. (IF)) holds. Then with probability at least 1 -2 exp ( -1 2 β k n ) (or. resp. 1 -2 exp ( -c 1 kn ) ), it holds that

Proof. If Assumption (GF) holds, then

Set M = β k k and by Hoeffding's inequality, the above trace concentrates:

Set t = nk/ 2 to conclude the statement.

Analogously, if Assumption (IF) holds, for i = 1 , ..., n and l = 1 , ..., k , ( z ( l ) i ) 2 -1 is centered sub-exponential variable with sub-exponential norm ∥ ∥ ∥ ( z ( l ) i ) 2 -1 ∥ ∥ ∥ ψ 1 ≲ G 2 . By Lemma H.7, with probability at least 1 -2 exp ( -c 1 kn ) ,

Lemma G.6 (Lemma 3 in [7]) . For any k ≤ n and δ ∈ (0 , 1) , with probability at least 1 -δ , it holds that

Proof. Since E [ 1 n ∥ ∥ X >k θ ∗ >k ∥ ∥ 2 2 ] = ∥ ∥ θ ∗ >k ∥ ∥ 2 Σ >k , we use Markov's inequality to obtain the result.

Lemma G.7. Suppose Assumption (IF) hold. Then for any integer k ≤ n ,

If Assumption (PE)/(EE) holds, then with probability at least 1 -e -n , it holds that s 1 ( A k ) = O n,k ( nλ k + nλ ) .

If Assumption (PE) holds, then with probability at least 1 -2 e -n , it holds that s n ( A k ) = Ω n,k ( nλ n + k + nλ ) ,

for p large enough.

Proof. First we prove statement 1. For any k ∈ N ,

Hence, set t = n in Theorem H.5, with probability at least 1 -e -n , it holds that

By triangle inequality, ∥ ∥ X ⊤ >k X >k ∥ ∥ op ≤ n ∥ ∥ 1 n X ⊤ >k X >k -Σ >k ∥ ∥ op + n ∥ Σ >k ∥ op = O n,k ( nλ k ) . Hence, with the same probability,

Now we prove statement 2. Note that the smallest singular value of a matrix does not increase after discarding a column: for some constant η > 1 to be determined, it holds that

By Assumption (IF), the columns of Z k : ηn are independent to each other. By Theorem H.2, with probability at least 1 -2 e -c 1 t 2 , it holds that

Choose t = √ n/c 1 and η > ( 2 + 1 c 1 + c 2 ) 2 (given that p > ηn large enough), then with probability at least 1 -2 e -n , it holds that

Hence with the same probability, by Assumption (PE),

LemmaG.8 (Theorem 2 in [7]) . Suppose Assumption (GF) (or resp. (IF)) holds, then with probability at least 1 (or resp. 1 -2 pe -c 1 n ), it holds that

Suppose Assumptions (IF) holds. Furthermore, if Assumptions (PE) or (EE) holds, then one obtains a probability bound which allows arbitrary large p : with probability at least 1 -e -n , it holds that

Proof. Since the trace of a matrix is the sum of its eigenvalues, we obtain:

By Assumption (GF), since ess sup x ∥ x >k ∥ 2 2 Tr[ Σ >k ] ≤ β k , we have

If Assumption (IF) holds, then for each l > k , the random variable ( z ( l ) ) 2 -1 is centered subexponential, hence with probability at least 1 -2 exp ( -c 1 n ) , it holds that

By union bound, with probability at least 1 -2( p -k ) exp ( -c 1 n ) , it holds that

If Assumptions (IF) and (PE)/(EE) hold, then by Lemma G.7: with probability at least 1 -e -n , it holds that

for any k ∈ N . Hence with the same probability,

H Technical lemmata

Theorem H.1 (Theorem 1 and Example 1 in [51]) . Assume that M 1 , M 2 , .., M n are independent copies of a d × d positive semi-definite symmetric random matrix M with E [ M ] = Σ , which satisfies:

for some constant κ ≥ 1 and for any v ∈ R d . Then, for any t > 0 , with probability at least 1 -e -t , the inequality holds:

whenever n ≥ 4 r + t , and r def. = Tr[ Σ ] ∥ Σ ∥ op is the effective rank.

Theorem H.2 (Theorem 5.39 and Remark 5.40 in [47]) . Let A be an N × n matrix with independent rows A i of sub-Gaussian random vector with covariance Σ def. = E [ A i A i ⊤ ] ∈ R n × n . Then there exists constants C 5 , C 6 > 0 15 (depending only on the sub-Gaussian norm of entries of A ), such that for any t ≥ 0 , with probability at least 1 -2 e -C 5 t 2 , we have

where δ = C 6 √ n N + t N . In particular, if Σ = I n , we have

Theorem H.3 (Theorem 5.41 in [47]) . Let A be an N × n matrix whose rows A i are independent isotropic random vectors in R n . Let m> 0 be a number such that ∥ A i ∥ 2 ≤ √ m a.s. for all i . Then for every t ≥ 0 , it holds that:

Theorem H.4 (Theorem 5.58 in [47]) . Let A be an N × n matrix ( N ≥ n ) with independent columns A i ∈ R N of sub-Gaussian isotropic random vector with with ∥ A i ∥ 2 = √ N almost surely. Then there exists constants C 8 , C 9 > 0 (depending only on the sub-Gaussian norm of entries of A ), such that for any t ≥ 0 , with probability at least 1 -2 e -C 8 t 2 , we have

Theorem H.5 (Theorem 9 in [27]) . Let A be an N × n matrix with i.i.d. columns C i ∈ R N of sub-Gaussian random vector with covariance Σ def. = E [ C i C ⊤ i ] ∈ R N × N . Then there exists a constant c 1 > 0 , such that for any t ≥ 0 , with probability at least 1 -e -t , we have

where r = Tr[ Σ ] ∥ Σ ∥ op is the effective rank of the covariance Σ .

Remark H.6 . Theorem H.5 is different from Theorem H.2 in a few ways: first, the upper bound in the former one contains the term n N , which requires n < N in order to obtain a good concentration, while that in the latter one contains the term r n = ∑ N k =1 Σ kk n , which can still be bounded if N > n and the decay of Σ kk is fast enough; second, former one requires i.i.d. columns while the latter one requires only independent rows.

Lemma H.7 (Sub-Exponential Deviation, see Corollary 5.17 in [47]) . Let N ∈ N . Let X 1 , ..., X N be independent centered random variables with sub-exponential norms bounded by B . Then for any δ > 0 ,

where C 7 > 0 is an absolute constant.

In particular, if X ∼ χ ( N ) is the Chi-square distribution, then P { | X N -1 | > t } ≤ 2 e -Nt 2 / 8 , ∀ t ∈ (0 , 1) .

Lemma H.8 (Lemma 28 in [46]; Lemma 14 in [7]) . Suppose that for some k < N , the matrix A k is positive definite, then the following inequality holds:

Lemma H.9 (Lemma 27 in [46]; Lemma 13 in [7]) . For any k ∈ N , we have

, where σ 2 def. = E [ ϵ 2 ] ≥ 0 .

LemmaH.10 (Lemma 18 in [7]) . Assume λ k = Θ k ( k -1 -a ) and | θ ∗ k | = O k ( k -r ) for some a, r > 0 . Then the square norms ∥ ∥ θ ∗ >k ∥ ∥ 2 Σ >k and ∥ ∥ θ ∗ ≤ k ∥ ∥ 2 Σ -1 ≤ k have bounds:

where ( x ) + def. = max { x, 0 } .

Lemma H.11. Assume λ k = Θ k ( e -ak ) and | θ ∗ k | = O k ( e -rk ) for some a, r > 0 . Then the square norms ∥ ∥ θ ∗ >k ∥ ∥ 2 Σ >k and ∥ ∥ θ ∗ ≤ k ∥ ∥ 2 Σ -1 ≤ k have bounds:

Lemma H.12. Recall that

If λ k = Θ k ( k -1 -a ) for some a > 0 , then

If λ k = Θ k ( e -ak ) for some a > 0 , then

Proof. By simple calculus, ∑ p l = k +1 k -1 -a = Θ k (∫ ∞ k t -1 -a dt ) = Θ k ( t -a ) , similarly, ∑ p l = k +1 k -2 -2 a = Θ k (∫ ∞ k t -2 -2 a dt ) = Θ k ( t -1 -2 a ) . If λ k = Θ k ( k -1 -a ) for some a > 0 , then

Lemma H.13 (Lemma 7 in [46]) . Suppose Assumption (IF) holds. Then there exists some constant c > 0 such that, for any k < n/c , with probability at least 1 -ce -n/c , its holds that

Lemma H.14 (Lemma 8 in [46]) . Suppose Assumption (PS) holds. Furthermore, if Assumption (GF) (or resp. (IF)) holds, then with probability 1 -1 , the following inequality holds:

̸

where the expectation is taken as described in Assumption (PS), and A -l def. = ∑ l ′ = l λ l ′ z ( l ′ ) ( z ( l ′ ) ) ⊤ + nλ I n ∈ R n × n where z ( l ) ∈ R n denotes the l -th column of the whitened feature block Z def. = XΣ -1 / 2 .

Proof. By Assumption (PS), the expected value over θ ∗ on the bias admits the following expression as in line (40):

where P λ def. = X ⊤ ( XX ⊤ + nλ I p ) -1 X . Denote z ( l ) ∈ R n be the l -th column of the whitened feature block Z def. = XΣ -1 / 2 ∈ R n × p . Then the l -th diagonal element of the matrix ( I p -P λ ) Σ ( I p -P λ ) can be written as

̸

where A def. = XX ⊤ + nλ I p . If we write A -l def. = A -λ l z ( l ) ( z ( l ) ) ⊤ , by Sherman-Morrison-Woodbury formula, we have

and hence the averaged bias is bounded by:

Lemma H.15 (Lemma 9 in [6]) . Suppose that { X k } p k =1 is a sequence of non-negative random variables, and that { t k } p k =1 is sequence of non-negative real numbers (with at least one of which strictly positive), such that for some δ ∈ (0 , 1) , with a probability at least 1 -δ , η k > t k for all k = 1 , ..., p . Theb with probability at least 1 -2 δ ,

Lemma H.16 (Theorem 10 in [46]) . Let k ∈ N be an integer. Denote

Fix constants a > 0 and b > 1 n . There exists a constant c > 0 that only depends on a, b such that: if either r k n ∈ ( a, b ) or k = min { κ : r κ > bn } , then

Lemma H.17 (Corde's inequality, [18]) . For any positive definite symmetric matrices 16 M 1 , M 2 and positive number m ∈ [0 , 1] , it holds that ∥ M m 1 M m 2 ∥ op ≤ ∥ M 1 M 2 ∥ m op .

Proposition H.18. Let λ > 0 . The bias term B def. = E x [ ( x ⊤ ˆ θ ( X θ ∗ ) -x ⊤ θ ∗ ) 2 ] has the following expression:

where ˆ Σ def. = 1 n X ⊤ X .

Proof. By definition, rewrite the bias into:

Denote P λ def. = X ⊤ ( XX ⊤ + nλ I n ) -1 X ∈ R p × p . By Sherman-Morrison-Woodbury formula,

Hence B = λ 2 ∥ ∥ ∥ ( λ I p + ˆ Σ ) -1 θ ∗ ∥ ∥ ∥ 2 Σ

Proposition H.19 (Variance expression) . The variance term V def. = E x , ϵ [ ( x ⊤ ˆ θ ∗ ( ϵ ) ) 2 ] has the following expression:

Proof. By definition,

I Experiments in details

Some of the results in Table 1 have already been validated by experiments in previous literature. Hence, this section will focus on the novel result of this paper:

Same decay rate for independent (IF)/generic (GF) features under strong ridge;

Decay of B for s < 1 under weak ridge;

Decay of V under weak ridge (tempered vs catastrophic overfitting).

All experiments were conducted on a computer with a 2.3 GHz Quad-Core Intel Core i7 processor. The code for the experiments is available in the supplementary materials.

I.1 Bias under strong ridge

we first consider a simple example: let λ k = ( 2 k -1 2 π ) -1 -a , ψ k ( · ) = √ 2 sin ( 2 k -1 2 π · ) such that ∥ ψ k ∥ L 2 µ = 1 for µ = unif[0 , 1] ; let θ ∗ k = ( 2 k -1 2 π ) -r . For p = ∞ and a = 1 , the regression coincides with the kernel ridge regression with kernel k ( x, x ′ ) = min { x, x ′ } defined on the interval [0 , 1] by [48]. [30, 33] have conducted similar experiments on this kernel k . However, to simulate the regression for independent features (IF), the feature rank p has to be finite. In the following experiment, we choose p = 2000 , the sample size n ranges from 100 to 1000, ridge λ = λ n = ( 2 n -1 2 π ) -1 -a .

The first thing to check is whether both independent features ((IF)) and generic features ((GF)) satisfy

under strong ridge, where ˜ s def. = min s, 2 and s = 2 a + r 1+ a . To accurately obtain the bias term B , we compute the exact formula:

as shown in Proposition H.18, rather than computing the squared difference ( ˆ f ( x ) -f ∗ ( x ) ) 2 by evaluating on test points as done in [31], or by using an integral function as in [33]. To demonstrate the Gaussian Equivalent Property (GEP), we also compute B after replacing the Sine feature vector ψ = ψ k ( x ) p k =1 with a random Gaussian vector z ∼ N (0 , 1) or a random Rademacher vector z ∼ (unif {± 1 } ) p . It is worth noting that the random Rademacher vector z ∼ (unif {± 1 } ) p satisfies Assumption (IF). This shows that our statement holds more generally than under the Gaussian Design Assumption ((GD)). From Figure 5, we observe that for different choices of a and r (and hence s ), the bias decays at its theoretical rate for all three different features.

10

2

10

3

2×10

2

3×10

2

4×10

2

6×10

2

n (log scale)

10

10

10

9

10

8

10

7

Test Error (log scale)

Bias, a=1, b=2, r=1

Sine feature

Rademacher feature

Gaussian feature

Slope=-3.11

10

2

10

3

2×10

2

3×10

2

4×10

2

6×10

2

n (log scale)

10

9

10

8

10

7

10

6

Test Error (log scale)

Bias, a=0.5, b=1.5, r=1.5

Sine feature

Rademacher feature

Gaussian feature

Slope=-3.22

I.2 Bias under weak ridge

For weak ridge, we find that the decay rate is better than this theoretical rate (see Figure 6). One explanation for this is the estimation error of replacing the kernel K ( x, x ′ ) = min { x, x ′ } by its finite rank approximation; another reason is that the decay flattens for large sample sizes n ≫ 1000 . However, the learning curve of independent features (Gaussian or Rademacher) behaves similarly to the dependent one (Sine). The left plot in Figure 6 fits our theoretical result, as s > 1 , while the right plot shows that our theoretical bound is too pessimistic. However, we suspect this is due to the fact that the eigenfunctions on the one-dimensional input space are simply sines, which are uniformly bounded.

3

10

2

10

3

2×10

2

3×10

2

4×10

2

6×10

2

n (log scale)

10

10

10

9

10

8

10

7

10

6

Test Error (log scale)

Bias, a=1, b=4, r=1

Sine feature

Rademacher feature

Gaussian feature

Slope=-3.25

10

2

10

2×10

2

3×10

2

4×10

2

6×10

2

n (log scale)

10

5

10

4

10

3

Test Error (log scale)

Bias, a=0.25, b=2.5, r=0.5

Sine feature

Rademacher feature

Gaussian feature

Slope=-1.54

I.3 Variance under strong ridge

Analogous to B , we want to check whether both independent features ((IF)) and generic features ((GF)) satisfy

under strong ridge, where σ 2 = E [ ϵ 2 ] is the noise level. To ease the computation, instead of computing the expression of V directly from Proposition H.19:

where ˆ Σ = UDU ⊤ is the singular value decomposition of ˆ Σ . Figure 7 confirms the Gaussian Equivalent Property (GEP) under strong ridge.

All dotted lines in Figures 5, 6 and 7 are regression of the learning curve with Gaussian features.

I.4 Variance under weak ridge

As reported in [7, 14], by setting λ = 0 , ˆ f is indeed the norm minimum interpolant, which may demonstrates tempered or catastrophic overfitting as n →∞ . For this example, we focus on another setting where we take samples uniformly from a unit 2-disk and approximate V by evaluating the regressor of the zero function on the test point. We set λ = 0 and compute the kernel ridgeless

10

2

10

3

2×10

2

3×10

2

4×10

2

6×10

2

n (log scale)

10

4

10

3

Test Error (log scale)

Sine feature

Rademacher feature

Gaussian feature

Slope=-1.00

10

2

10

3

2×10

2

3×10

2

4×10

2

6×10

2

n (log scale)

2×10

2

3×10

2

4×10

2

Test Error (log scale)

Sine feature

Rademacher feature

Gaussian feature

Slope=-0.51

200

400

600

800

1000

n

0.40

0.41

0.42

0.43

0.44

0.45

Overfitting of Laplacian kernel

200

400

600

800

1000

n

0

500

1000

1500

2000

Overfitting of NTK with L=1

regression with the Laplacian kernel K ( x, z ) = e -∥ x -z ∥ 2 and the neural tangent kernel K ( x, z ) = x ⊤ zκ 0 ( x ⊤ z )+ κ 1 ( x ⊤ z ) where κ 0 ( t ) def. = 1 -1 π arccos( t ) , κ 1 ( t ) def. = 1 π ( t ( π -arccos( t )) + √ 1 -t 2 ) .

In Figure 8, we can see that although having polynomial eigen-decay, the Laplacian kernel exhibits tempered overfitting, as termed by [35], while the NTK catastrophic overfitting.

J Tables

Linear

Kernel

x

K

(

x,

·

)

z

ψ

x

Σ

Λ

Z

Ψ

XX

⊤

K

∥·∥

2

=

∥·∥

I

p

∥·∥

H

∥·∥

Σ

1

-

s

∥·∥

H

s

∥·∥

Σ

∥·∥

L

2

µ

(

X

)

∥·∥

∞

∥·∥

∞

on

supp(

µ

)

θ

∗

f

∗

strong

weak

Feature

(IF)

(GF)

(IF)

(GF)

Poly (PE)

B

[16]

[31, 7]

[16]

[31, 7]

V

[16]

[30, 7]

[16, 14]

[7]

Exp (EE)

B

this paper

[33]

this paper

[33]

V

this paper

this paper

-

-

strong

weak

Feature

(IF)

(GF)

(IF)

(GF)

Poly (PE)

B

-

O

(

n

-

b

˜

s

)

-

O

(

n

-

(min

{

2(

r

-

a

)

,

2

-

a

}

)

+

)

V

-

˜

O

(

σ

2

n

-

1+

b

a

+1

)

Θ

(

σ

2

)

˜

O

(

σ

2

n

2

a

)

Exp (EE)

B

-

-

-

-

V

-

-

catastrophic overfitting

strong

weak

Feature

(IF)

(GF)

(IF)

(GF)

Poly (PE)

B

O

(

n

-

b

˜

s

)

-

O

(

n

-

(1+

a

s

)

O

(

n

-

(1+

a

s

)

, s >

1

V

O

(

σ

2

n

-

1+

b

a

+1

)

-

-

-

Exp (EE)

B

-

O

(

e

-

b

˜

sn

)

-

O

(

e

-

a

˜

sn

)

, s >

1

V

-

-

catastrophic overfitting

strong

weak

Feature

(IF)

(GF)

(IF)

(GF)

Poly (PE)

B

Θ

(

n

-

b

˜

s

)

Θ

(

n

-

b

˜

s

)

Θ

(

n

-

(1+

a

s

)

{

O

(

n

-

(1+

a

s

)

,

s >

1

˜

O

(

n

-

(min

{

2(

r

-

a

)

,

2

-

a

}

)

+

)

,

s

1

V

Θ

(

σ

2

n

-

1+

b

a

+1

)

Θ

(

σ

2

n

-

1+

b

a

+1

)

Θ

(

σ

2

)

˜

O

(

σ

2

n

2

a

)

,

Ω

(

σ

2

)

Exp (EE)

B

Θ

(

e

-

b

˜

sn

)

O

(

e

-

b

˜

sn

)

O

(

e

-

a

˜

sn

)

, s >

1

O

(

e

-

a

˜

sn

)

, s >

1

V

Θ

(

σ

2

n

-

1+

b

a

)

O

(

σ

2

n

-

1+

b

a

)

catastrophic overfitting

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims asserted in Abstract and the questions raised in the introduction section 1 are directly addressed in the main section of our paper; namely in section 3 and the additional material in the appendix section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: This is a theoretical paper where assumptions are clearly stated. We admit that verifying the conditions of the assumptions (e.g., eigen-decay rate or eigen-functions of kernels in realistic datasets) can be challenging, and we limit our experiments to relatively simple examples. Further investigation into the generalization performance of kernel ridge regression on realistic datasets is left for future work.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All mathematical objects are clearly stated, and further explain, in section §2.2. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experiment is done on some synthetic settings, which details are clearly stated in I.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code for the experiments is uploaded as supplementary materials.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Error bars are displayed in the figures in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Section I.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments were conducted on a computer with a 2.3 GHz Quad-Core Intel Core i7 processor.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Since our paper is purely theoretical, it does not involve human subjects or participants, data-related concerns, nor are there any negative societal implications. Rather, by elucidating the theoretical foundations of some kernel ridge regression models, we help verify the reliability of commonly used machine learning models.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper focuses on enhancing the theoretical underpinnings of machine learning, leading to predominantly positive societal impacts. By emphasizing the importance of principled machine learning and furthering our understanding of AI capabilities, it contributes to the advancement of technology in a constructive manner.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: As our contributions are theoretical justifications of commonly used AI models, this does not apply here.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our paper does not use new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper conducts a comprehensive study of the learning curves of kernel ridge regression (KRR) under minimal assumptions

STRENGTHS:
The authors claimed that they provide a comprehensive analysis on the learning curves in kernel ridge regression.

WEAKNESSES:
The learning curves of kernel ridge regression have been extensively studied in recent literature. e.g.,  
1. "Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural  Information Processing Systems, 34:10131–10143, 2021.", 
2. "Yicheng Li, Haobo Zhang, and Qian Lin. On the asymptotic learning curves of kernel ridge  regression under power-law decay.", and 
3.  "Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard,  and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a  teacher-student model. Advances in Neural Information Processing Systems, 34:18137–18151,382 2021."

It is really hard for me to understand the difference between the current paper with the above papers.

QUESTIONS:
The writing style of this paper is quite challenging for me to engage with. I find the presentation could be improved in terms of clarity and readability. 

For instance, could the author make their own results and assumptions in a more clear way? There is not even a single explicit theorem  provided for me to verify. It is quite unusual for a theoretical paper to be presented in this manner. I am uncertain if this document has been generated by an artificial intelligence.

LIMITATIONS:
NONE
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA recent line of work has derived excess error rates for kernel ridge regression in a source and capacity setting under the assumption of Gaussian universality of the kernel features. This work investigates the validity of this assumption in this context.\n\nThe main result is that while the rates derived under Gaussian design are correct in the strong regularization regime, the excess error rate can be faster in the weak regularization regime.\n\nSTRENGTHS:\nThis is a solid work. The manuscript is clearly written: the context is well explained and the reading is smooth. The results fit in an established literature studying excess error rates for kernel ridge, which recently has seen a revival of interest in the context of deep learning (NTK) and neural scaling laws. Therefore, I believe it is of significant interest to the theoretical community in NeurIPS.\n\nWEAKNESSES:\nA minor weakness of this work is that it mostly combines existing technical results. But I think this is minor since in the end the conclusions are novel and interesting\n\nQUESTIONS:\n- In the introduction, the authors say their work addresses three questions. While Q1 and Q2 are precisely addressed by the results, I find that the answer to Q3 falls short. First, the assumption (GF) is surely weaker, but it is still constraining. Second, some of the results in Table 1 are only upper bounds. I understand that most of the excess rate results in the classical kernel literature are also upper bounds, and the authors explicitly discuss that under (GF) it is not possible to derive a matching lower bound, but there is nothing telling us that the picture is not richer in these regimes. Perhaps my problem is with the phrasing of Q3, which differently from Q1 and Q2 is vague.\n\n- I miss a discussion on the intuition behind the (GF) assumption. For instance, in the Gaussian design approximation, one possible intuition is the identification of \"orthogonality\" (of the features) with \"independence\". Do you have an intuitive understanding of the two conditions in L440? Why does strong regularization justifies independence?\n\n- Related to the question above, how (GF) differs from the concentration assumption in previous work, e.g. (a1, a2, b1) in [36]. Note that the formulas derived under similar concentration conditions from [36] allow to recover exactly the Gaussian design rates from [16], see the contemporary recent work [DLM]. This suggests (GF) is strictly weaker?\n\n- Since the main result in this work dialogues with previous literature, I suggest commenting and comparing the important notation (source, capacity, regularization decay, etc) in this work with the ones employed in the relevant Gaussian design literature, e.g. [10, 16, 35, 44]. For example, a table like Table 1 in [44] or Table 2 in [16].\n\n- Can you please elaborate on Remark A.5?\n\n[DLM] https://arxiv.org/abs/2405.15699\n\n**Minor points**:\n\n- The authors discuss the \"over-parametrized\" and \"under-parametrized\" regime in the text, but never define what they mean. While this can be inferred from the text, it would be good to precisely define it, since this terminology is used in different ways in the ML theory literature.\n\n- For the sake of completeness, it would be better to add the definition of the source in (L106-L108) to the main text in a final version.\n\n- Unpublished pre-prints in the bibliography are missing the arXiv identifiers.\n\n- L110, define $\\succeq$ in the notation section.\n\n- L597, maybe $\\psi_{k}$ and $\\phi_{k}$ are switched?\n\n- L634, \"*Consider a kernel $\\kappa:\\mathcal{X}\\times\\mathcal{X}\\to\\mathbb{R}$ be a kernel with [...]*\"\n\n- Eq. below L651, missing right bracket.\n\n- Assumption \"Domain Regularity (DR)\" in Appendix B.2 there are two items (i), (ii) but (iii) is mentioned in the paragraph below (L662-L669) twice.\n\n- L679, precise what \"$\\lambda$\" in $||\\bar{\\psi}_{i,j}||\\lesssim \\lambda^{(d-1)/4}$ is.\n\nLIMITATIONS:\nLimitations are discussed in the \"Future potential work\" section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies learning curves of kernel ridge regression (KRR) under minimal assumptions. The authors analyze the role of key properties of the kernel, such as its spectral eigen-decay, the characteristics of the eigenfunctions, and smoothness of the kernel. They also demonstrate the validity of the Gaussian Equivalent Property (GEP), which states that the generalization performance of KRR remains the same when the whitened features are replaced by standard Gaussian vectors. Additionally, they derive new improved bounds across several settings.\n\nSTRENGTHS:\n- The authors study the learning curves for various settings, including weak ridge vs strong ridge, independent features vs generic features, and polynomial vs exponential kernel eigenvalue decay. This comprehensive study provides a deeper understanding of the behavior of KRR in different scenarios.\n- An improved bound is presented for the bias under the weak ridge assumption. In particular, the authors show that the generalization performance with independent (Gaussian) features and dependent (kernel) features coincides asymptotically and it solely depends on the eigen-decay under strong ridge regularization, hence validating the Gaussian Equivalent Property (GEP).\n- The paper provides an answer to the key question \"Under what condition the generalization error fully determined by the eigen-decay?\"--- 1) in under-parameterized setting; or 2) with strong ridge in over-parameterized regime.\n\nWEAKNESSES:\n- The presented results seem to be based on a set of different assumptions while comparing with the current bounds. However, it is unclear how the discrepancy in the assumptions impacts the bounds. For example, the paper compares the presented bounds with related work using Hölder continuous kernels or the Embedding Condition, but it is not clear how the differences in assumptions affect the comparison.\n- The numerical studies demonstrate the validity of the bounds for a couple of constructed kernels. However, it would be beneficial to consider more general/practical kernels to assess the practical impact of this work.\n\nQUESTIONS:\nDoes the result hold under the assumptions used in the related work, e.g., Embedding Condition?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the learning curve of kernel ridge regression under both eigendecay assumption and the source condition assumption. Assuming the fixed input dimension setting, this paper derives the finite sample bound for the bias and variance where features can either be generic or independent. Depending on the assumptions used, matching lower bound are also provided. After this, the authors provide a unifying theory of the KRR test error and demonstrates the settings where Gaussian equivalence property holds. They also provide an answer for when the test error is fully determinded by the eigen-decay of the kernel. Finally, they provide simulation experiments to validate the bound on the test error.\n\nSTRENGTHS:\nBy providing the matching upper and lower bounds for the test error under IF and GF where strong ridge is used, the paper shows that the Gaussian equivalence property only holds when strong ridge is used. In dosing so, the paper also provides a novel master inequality for both bias and variance.\n\nIn many cases, for example the expoenential eigendecay under strong ridge, the paper provides the sharpest learning rates so far.\n\nThe paper aslo addresses the question of when the learning rate of KRR is fully determined by the kernel eigenspectrum.\n\nWEAKNESSES:\nIn demosntrating the GEP, the paper seems only provide the upper bound for bias and variance under generic features while a matching lower bound is missing. Given this, it is not completely convincing that the GEP holds although the author did show that the upper bound under GF matches with IF. So I was wondering if the author could explain this a bit or can detail the challenges in obtaining the lower bound.\n\nRecently, there is a growing interest in studying the KRR when the output is infinite-dimensional, see e.g. \n\"Towards Optimal Sobolev Norm Rates for the Vector-Valued Regularized Least-Squares Algorithm.\" (2024)\n&\"Optimal Rates for Vector-Valued Spectral Regularization Learning Algorithms.\" (2024)\n\nI am curious whether the results hold in the setting where the output is infinite dimensional. Maybe the author could draw some link between their results and this setting.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}