Title: A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: Kernel

regression

bias-variance

generalization
Abstract: Existing statistical learning guarantees for general kernel regressors often yield loose bounds when used with finite-rank kernels. Yet, finite-rank kernels naturally appear in a number of machine learning problems, e.g. when fine-tuning a pre-trained deep neural network's last layer to adapt it to a novel task when performing transfer learning.  We address this gap for finite-rank kernel ridge regression (KRR) by deriving sharp non-asymptotic upper and lower bounds for the KRR test error of any finite-rank KRR. Our bounds are tighter than previously derived bounds on finite-rank KRR and, unlike comparable results, they also remain valid for any regularization parameters.

A Theoretical Analysis of the Test Error of Finite-Rank Kernel Ridge Regression

Abstract

Existing statistical learning guarantees for general kernel regressors often yield loose bounds when used with finite-rank kernels. Yet, finite-rank kernels naturally appear in several machine learning problems, e.g. when fine-tuning a pre-trained deep neural network's last layer to adapt it to a novel task when performing transfer learning. We address this gap for finite-rank kernel ridge regression (KRR) by deriving sharp non-asymptotic upper and lower bounds for the KRR test error of any finite-rank KRR. Our bounds are tighter than previously derived bounds on finite-rank KRR, and unlike comparable results, they also remain valid for any regularization parameters.

1 Introduction

Generalization is a central theme in statistical learning theory. The recent renewed interest in kernel methods, especially in Kernel Ridge Regression (KRR), is largely due to the fact that deep neural network (DNN) training can be approximated using kernels under appropriate conditions [19, 3, 10], in which the test error is more tractable analytically and thus enjoys stronger theoretical guarantees. However, many prior results have been derived under conditions incompatible with practical settings. For instance [24, 26, 27, 29] give asymptotic bounds on the KRR test error, which requires the input dimension d to tend to infinity. In reality, the input dimension of the data set and the target function is typically finite. A technical difficulty to provide a sharp non-asymptotic bound on the KRR test error comes from the infinite dimensionality of the kernel 1 . While this curse of dimensionality may be unavoidable for infinite-rank kernels (at least without additional restrictions), one may likely derive tighter bounds in a finite-rank setting. Therefore, other works [6, 2] focused on a setting where the kernel is of finite-rank, where non-asymptotic bounds and exact formula of test error can be derived.

Since different generalization behaviours are observed depending on whether the rank of the kernel rank M is smaller than the sample size N , one typically differentiates between the underparameterized ( M < N ) and over-parameterized regime ( M > N ). We focus on the former due to its relevance to several applications in the field of machine learning, including random feature

models [32, 35, 25, 14, 17] such as reservoir computers [14, 15, 16, 11] where all the hidden layers of a deep learning model are randomly generated, and only the final layer is trainable, or when fine-tuning the final layers of pre-trained deep neural networks for transfer learning [4] or in few-shot learning [38]. The practice of only re-training pre-trained deep neural networks final layer [40, 23] is justified by the fact that earlier layers encode general features which are common to similar tasks thus fine-tuning can support's a network's ability to generalize to new tasks [21]. In this case, the network's frozen hidden layers define a feature map into a finite-dimensional RKHS, which induces a finite-rank kernel; however, similar finite-rank kernels are often considered [13] in which features are extracted directly from several hidden layers in a deep pre-trained neural network which is then fed into a trainable linear regressor.

Contribution Our main objective is to provide sharp non-asymptotic upper and lower bounds for the finite-rank KRR test error in the under-parameterized regime. We make the following contributions:

(i) Non-vacuous bound in ridgeless case: In contrast to prior work, our bounds exhibit better accuracy when the ridge parameter λ → 0 , matching the intuition that a smaller ridge yields a smaller test error;

(ii) Sharp non-asymptotic lower bound: We provide a sharp lower bound of test error, which matches our derived upper bound as the sample size N increases. In this sense, our bounds are tight;

(iii) Empirical validation: We experimentally validate our results and show our improvement in bounds over [6].

As detailed in Section D, Table 1 compares our results to the available risk-bounds for finite-rank KRR.

Organization of Paper Our paper is organized as follows: Section 2 motivates the importance of the under-parameterized finite-rank KRR. Section 3 introduces the notation and necessary background material required in our results' formulation. Section 4 summarizes our main findings and illustrates their implications via a numerical study. All notation is tabulated in Appendix A for the reader's convenience. All proofs can be found in Appendices B and C, and numerical validation of our theoretical findings in Appendix D.

2 Applications

In this section, we motivate the importance of the under-parameterized finite-rank KRR in practical settings. For readers more interested in our main results, please start from Section 3.

Application: Fine-tuning Pre-Trained Deep Neural Networks For Transfer Learning Consider the transfer-learning problem of fine-tuning the final layer of a pre-trained deep neural network model

f : R d → R D so that it can be adapted to a task that is similar to what it was initially trained for. This procedure defines a finite-rank kernel regressor because, for any x ∈ R d , f can be factored as

where A is a D × d L -matrix, σ · denotes element-wise application of a univariate non-linear function, for l = 1 , . . . , L , L is a positive integer, and W ( l ) : R d l -1 → R d l , d = d 0 . In pre-training, all parameters defining the affine maps W ( l ) in the hidden layers σ · W ( L ) , . . . , σ · W (1) are simultaneously optimized, while in fine-tuning, only the parameters in the final A are trained, and the others are frozen. This reduces f to a finite-rank kernel regressor with finite-rank kernel given for x, ˜ x ∈ R d by

Stably optimizing f to the new task requires strict convexity of the KRR problem

where the hyperparameter λ > 0 ensures strong convexity of the KRR's loss function and where ∥ A ∥ F denotes the Frobenius norm of A. The unique solution ˆ A to (1), determines the optimally trained KRR model ˆ f ( x ) def. = ˆ Aϕ ( x ) corresponding to the finite-rank kernel K .

Application: Random Feature Models Popularized by [32] to enable more efficient kernel computations, random feature model has recently seen a substantial spike in popularity and has been the topic of many theoretical works [20, 27]. Note that random feature models are finite-rank kernel regressors once their features are randomly sampled [32, 35, 25, 14, 11].

Application: General Use We emphasize that, though fine-tuning and random feature models provide simple typical examples of when finite-rank KRR arise in machine learning, there are several other instances where our results apply, e.g. when deriving generalization bounds for infinite-rank kernels by truncation, thereby replacing them by finite-rank kernels; e.g. [27].

3 Preliminary

We now formally define all concepts and notation used throughout this paper. A complete glossary is found in Appendix A.

3.1 The Training and Testing Data

We fix a (non-empty) input space X ⊂ R d and a target function ˜ f : X → R which is to be learned by the KRR from a finite number of i.i.d. (possibly noisy) samples Z def. = ( X , y ) = ( ( x i ) N i =1 , ( y i ) N i =1 ) ∈ R d × N × R N . The inputs x i are drawn from a sampling distribution ρ on X and outputs are modelled as y i = ˜ f ( x i ) + ϵ i ∈ R N × 1 for some i.i.d. independent random variable ϵ i which is also independent of the X , satisfying E [ ϵ ] = 0 and E [ ϵ 2 ] def. = σ 2 ≥ 0 . Our analysis is set in the space of all square-integrable 'function' with respect to the sampling distribution L 2 ( ρ ) def. = { f : X → R : E x ∼ ρ [ f ( x ) 2 ] < ∞} .

3.2 The Assumption: The Finite-Rank Kernel Ridge Regressor

As in [2, 6], we fix a rank M ∈ N + for the kernel K .

Definition 3.1 (Finite Rank Kernel) . Let M be a positive integer. A (finite) rankM kernel K is a map K : X × X → R defined for any x, x ′ ∈ X by

where the positive numbers { λ k } M k =1 are called the eigenvalues, and orthonormal functions ψ k ∈ L 2 ρ are called the eigenfunctions. 2

Remark 3.2 (Eigenvalues are Ordered) . Without loss of generality, we will assume that the kernel K 's eigenvalues are ordered λ 1 ≥ λ 2 ≥ · · · ≥ λ M > 0 .

We denote by H def. = span { λ -1 / 2 1 ψ 1 , . . . , λ -1 / 2 M ψ M } the reproducing kernel Hilbert space (RKHS) associated to K . See Appendix B for additional details on kernels and RKHSs.

Together, the kernel K and a ridge λ > 0 define an optimal regressor for the training data Z .

Definition 3.3 (Kernel Ridge Regressor (KRR)) . Fix a ridge λ > 0 , the regressor f Z ,λ of the finite-rank kernel K is the (unique) minimizer of

3.3 The Target Function

The only regularity assumed of the target function ˜ f is that it is square-integrable with respect to the sampling distribution ρ ; i.e. ˜ f ∈ L 2 ρ . We typically assume that ˜ f contains strictly more features than can be expressed by the finite-rank kernel K . Thus, ˜ f can be arbitrarily complicated even if the kernel K is not. Since { ψ k } M k =1 is an orthonormal set of L 2 ρ , we decompose the target function ˜ f as

for some real numbers ˜ γ k 's and ˜ γ >M and for some normal function ψ >M orthogonal to { ψ k } M k =1 . We call ψ >M the orthonormal complement and ˜ γ >M the complementary coefficient. The component ˜ f ≤ M def. = ∑ M k =1 ˜ γ k ψ k of ˜ f is in H . For the case ˜ γ >M = 0 , we call it a consistent case, as the target function ˜ f lies in the hypothesis set H , else we call it an inconsistent case.

Alternatively, the orthonormal complement ψ >M can be understood as some input-dependent noise. Assume we have chosen a suitable finite rank kernel K with corresponding RKHS H such that the target function lies in H . For this purpose, we can write the target function as ˜ f ≤ M = ∑ M k =1 ˜ γ k ψ k for some real numbers ˜ γ k 's. Suppose that we sample in a noisy environment; then for each sample input x i , the sample output y i can be written as

3.4 Test Error

Next, we introduce the subject of interest of this paper in more detail. Our statistical analysis quantifies the deviation of the learned function from the ground truth of the test error .

Definition 3.4 (KRR Test Error) . Fix a sample Z = ( X , ˜ f ( X ) + ϵ ) . The finite-rank KRR (3) 's test error is

The analysis of this paper also follows the classical bias-variance decomposition, thus we write

where bias measures the error when there is no noise in the label, that is,

and variance is defined to be the difference between test error and bias: variance def. = R Z ,λ -bias.

4 Main Result

We now present the informal version of our main result, which gives high-probability upper and lower bounds on the test error. This result is obtained by bounding both the bias and variance, and the probability that the bounds hold is quantified with respect to the sampling distribution ρ . Here we emphasize the condition that N > M and hence our statement is valid only in under-parametrized case.

We can assume the data-generating distribution ρ and eigenfunctions ψ k are well-behaved in the sense that:

Assumption 4.1 (Sub-Gaussian-ness) . We assume that the probability distribution of the random variable ψ k ( x ) , where x ∈ ρ , has sub-Gaussian norm bounded by a positive constant G > 0 , for all k ∈ { 1 , ..., M } ∪ { > M } 3 .

In particular, if the random variable ψ k ( x ) is bounded, the assumption 4.1 is fulfilled.

Theorem 4.2 (High Probability Bounds on Bias and Variance) . Suppose Assumption 4.1 holds, for N > M sufficient large, there exists some constants C 1 , C 2 independent to N and λ such that, with a probability of at least 1 -2 /N w.r.t. random sampling, we have the following results simultaneously:

(i) Upper-Bound on Bias: The bias is upper bounded by:

where we denote ˜ f ≤ M def. = ∑ M k =1 ˜ γ k ψ k = ˜ f -˜ γ >M ψ >M ;

(ii) Lower-Bound on Bias: The bias is lower bounded by an analogous result:

(iii) Upper-Bound on Variance: The variance is upper bounded by:

(iv) Lower-Bound on Variance: The variance is lower bounded by an analogous result:

For λ → 0 , we have a simpler bound on the bias: with a probability of at least 1 -2 /N , we have

If ˜ γ 2 >M = 0 , then we are in the consistent case, meaning that ˜ f belongs to H . In this case, we have a simpler bound on the bias: with a probability of at least 1 -2 /N , we have

Proof Sketch: The main technical tools are 1) more careful algebraic manipulations when dealing with terms involving the regularizer λ and 2) the use of a concentration result for a sub-Gaussian random covariance matrix in [37] followed by the Neumann series expansion of a matrix inverse. Hence, unlike previous work, our result holds for any λ , which can be chosen independence to N . The complete proof of this result, together with the explicit form of the lower bound, is presented in Theorems C.19 and C.20 in the Appendix C. □

Remark 4.3. Note that one can also easily derive bounds on the expected value of the test error to allow comparisons with prior works, such as [6, 36, 33].

Remark 4.4. The main technical difference from prior works is that we consider the basis { ψ k } M k =1 in the function space L 2 ρ instead of { λ -1 / 2 k ψ k } M k =1 in the RKHS H . This way, we exploit decouple the effect of spectrum from the sampling randomness to obtain a sharper bound. For further details, see Remark C.12 in Appendix C.

Combining Theorem 4.2 and the bias-variance decomposition in Definition 3.4, we have both the upper and lower bounds of the test error on KRR.

Corollary 4.4.1. Under mild conditions on the kernel K , for N sufficiently large, there exist some constants C 1 , C 2 independent to N and λ such that, with a probability of at least 1 -2 /N w.r.t. random sampling, we have the bounds on the test error R Z ,λ :

In particular, we have:

The corresponding lower bounds are given analogously:

See Section 5 for more details and validations of Theorem 4.2.

5 Discussion

In this section, we first elaborate on the result from Theorem 4.2, which we then compare in detail to prior works, showcasing the improvements this paper makes. Finally, we discuss several future research directions.

5.1 Elaboration on Main Result

Bias From Eq. (8), we can draw the following observations: 1) The term ˜ γ 2 >M in the upper bound is the finite rank error due to the inconsistency between RKHS and the orthogonal complement of the target function, which cannot be improved no matter what sample size we have. We can also view this term as the sample-dependent noise variance (see Eq. (5)). Hence, unlike the sample-independent noise variance σ in Eq. (10), the sample-dependent noise variance does not vanish when N →∞ .

2) Note that the third term 1 4 ∥ ˜ f ∥ 2 L 2 ρ √ log N N is a residue term proportional to ∥ ˜ f ∥ 2 L 2 ρ and vanishes

when N → ∞ , which means we have better control of the sample-dependent noise around its expected value for large N . Also, note that the factor ∥ ˜ f ∥ 2 L 2 ρ = ∑ M k =1 ˜ γ 2 k + ˜ γ 2 >M depends solely on the target function ˜ f but not on the kernel K or ridge parameter λ . 3) The second plus the fourth terms ( 1 + 2 √ log N N ) λ ∥ ˜ f ≤ M ∥ 2 H depends strongly on the kernel training: the sum is proportional to the ridge λ , and the RKHS norm square ∥ ˜ f ≤ M ∥ 2 H = ∑ M k =1 ˜ γ 2 k λ k measures how well-aligned the target functions with the chosen kernel K is. Again, a larger N favors the control as the residue √ log N N → 0 as N → ∞ . 4) The fifth term C 1 log N N is a small residue term with fast decay rate, which the other terms will overshadow as N →∞ .

Ridge parameter The bounds in Eq. (12) demonstrate that in the ridgeless case, the bias can be controlled solely by the finite rank error ˜ γ 2 k with a confidence level depending on N ; also the upper and lower bounds coincide as N →∞ .

Variance For the variance bounds in Eq. (10), we have similar results: the variance can be controlled solely by the (sample-independent) noise variance σ 2 with a confidence level depending on N , also the upper and lower bounds coincides as N →∞ .

5.2 Comparison with Prior Works

We discuss how our results add to, and improve on, what is known about finite-rank KRRs following the presentation in Table 1.

Classical tools to study generalization The test error measures the average difference between the trained model and the target function. It is one of the most common measures to study the generalization performance of a machine learning model. Nevertheless, generally applicable statistical learning-theoretic tools from VC-theory [8], (local) Rademacher complexities [22, 7, 30], PAC-Bayes methods [1, 28], or smooth optimal transport theory [18] all yield pessimistic bounds on the KRR test error. For example, Mohri et al. [30] bound the generalization error using Rademacher Complexity: there exists some constant C > 0 independent to N and λ such that, with a probability of at least 1 -τ :

If we set τ to 2 /N , the decay in the generalization gap is O (√ log N N ) , which is too slow compared to other kernel-specific analyses.

Truncated KRR Amini et al. [2] suggests an interesting type of finite-rank kernels: for any (infinite-rank) kernel K ( ∞ ) , fix a sample Z = ( X , y ) of size N and define a rankN kernel K by the eigendecomposition of the kernel matrix K ( ∞ ) . Note that different random sample results in a completely different rankN kernel K . Assume that the target function ˜ f lies in the N -dimensional RKHS corresponding to K , then one can obtain an exact formula for the expected value of the test error of the kernel K (but not only the original K ( ∞ ) ). However, the formula obtained in [2] only takes into account the expectation over the noise ϵ , but not over the samples x . Hence, our paper yields a more general result for the test error.

Upper bound Comparison To the best of our knowledge, the following result is the closest and most related to ours:

Theorem 5.1. (Proposition 7.4 in [6]) With notation as before, assume, in addition, that: 1) ˜ γ >M = 0 , that is, ˜ f ∈ H ; 2) for all k = 1 , ..., M , we have √ ∑ M k =1 λ k ≤ R ; 3) and N ≥ ( 4 3 + R 2 8 λ log 14 R 2 λτ ) .

Then we have, with a probability of at least 1 -τ ,

In comparison to [6], Theorem 4.2 makes the following significant improvements:

(i) In Theorem 5.1, as the ridge parameter λ → 0 , the minimum requirement of N and the upper bound of the variance explode; while in our case, both the minimum requirement of N and the bounds on the variance are independent of λ .

̸

(ii) While [6] also mentioned the case where ˜ γ >M = 0 , however their bound for the inconsistent case is implicit; 4 while our work clearly states the impact of ˜ γ >M on the test error.

(iii) Our upper bounds are sharper than [6]:

(a) For the bias, assume ˜ γ >M = 0 for comparison purposes. Then by line (13) our upper bound would become:

which improves [6]'s upper bound bias ≤ 4 λ ∥ ˜ f ∥ 2 H significantly. First, we observe a minor improvement in the constant (by a factor of 1 4 ) when N →∞ . Second and more importantly, for finite N , we observe a non-asymptotic decay on the upper bound, while [6]'s upper bound is insensitive to N .

(b) For the variance, if we replace τ by 2 /N in [6]'s upper bound: with probability of at least 1 -2 /N ,

which has a decay rate of log N N , our work shows a much faster decay of 1 N . Moreover, we show that the "unscaled" variance, that is N · ( variance ) , is actually bounded by O ( 1 + √ log N N ) , while [6]'s upper bound on it would explode.

(iv) We also provide a lower bound for the test error that matches with the upper bound when N →∞ , while [6] does not derive any lower bound.

While our work offers overall better bounds, there are some technical differences between the bounds of [6] and ours:

(i) We require some explicit mild condition on input distribution ρ and on kernel K , while [6] rely on the assumption √ ∑ M k =1 λ k ≤ R which gives an alternative implicit requirement;

(ii) Our work eventually assumes under-parametrization M < N , while [6] does not requires this. Essentially, we exploit the assumption on the under-parameterized regime to obtain a sharper bound than [6]. In short, we select the L 2 ρ basis ψ k for analysis while [6] selects the basis in RKHS, which eventually affects the assumptions and conclusions. For more details, see Appendix C.

Nyström Subsampling and Random Feature Model Nyström Subsampling and Random feature models are two setting closely related to our work. [34, 35] bound the test error from above where the regularizer λ = λ ( N ) decays as N →∞ . In comparison, one main contribution of our work is that we provide both tighter upper bound and tighter lower bound than these prior works (they derive the same convergence rate on the upper bound up to constants but they do not derive a lower bound). Another major difference is that our bounds work for any regularization independent to N .

Effective Dimension In [34, 35] and other related works on kernel, the quantity N ( λ ) = ∑ M k =1 λ k λ k + λ = Tr [ ¯ P ] is called the effective dimension and it appeared as a factor in the upper bound. In our work, we can define a related quantity N 2 ( λ ) = ∑ M k =1 λ 2 k ( λ k + λ ) 2 = Tr [ ¯ P 2 ] , which appears in our bound (See Proposition C.14 for details.). Note that N 2 ( λ ) ≤ N ( λ ) ≤ M . Indeed, we can sharpen the factor M N in line (10) to N 2 ( λ ) N .

5.3 Experiments

We examine the test error and showcase the validity of the bounds we derived for two different finite-rank kernels: the truncated neural tangent kernel (tNTK) and the Legendre kernel (LK) (see details of their definitions in Appendix D). Figure 1 shows the true test error as a function of the sample size N . Figure 2 plots our upper bound compared to [6], clearly demonstrating significant improvements. We for instance note that the bound is tighter for any value of λ . Finally, Figure 9 shows both the upper and lower bound closely enclose the true test error. For more details on the experiment setup, including evaluating the lower bound, please see Appendix D.

3

2

1

0

1

2

3

x

1.0

0.5

0.0

0.5

1.0

y

tNTK training

Target Function

Regressor

Sample

25

50

75

100

125

150

175

200

N

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

Test Error

Median

Quartiles

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

x

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

y

Regressor

Target Function

Sample

25

50

75

100

125

150

175

200

N

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Test Error

Median

Quartiles

5.4 Future Research Direction

An interesting extension would be to combine our results with [36, 10] in the over-parameterized regime to give an accurate transition and observe double descent. However, since we are interested in the sharpest possible bounds for the under-parameterized regime, we treat these cases separately. We will do the same for the over-parameterized regime in future work. A special case is where the parameterization is at the threshold M = N and λ = 0 . The blow-up of variance as M → N is well-known, where [5, 9] gives empirical and theoretical reports on the blow-up for kernels. We expect that we can formally prove this result for any finite-rank kernels using our exact formula in Proposition C.7 and some anti-concentration inequalities on random matrices.

25

50

75

100

125

150

175

200

N

4

2

0

2

4

log Scale

Varying Sample Size on tNTK

Bach's Upper Bound

0

200

400

600

800

1000

1/

4

3

2

1

0

1

2

3

Varying Ridge on tNTK

Bach's Upper Bound

25

50

75

100

125

150

175

200

N

4

3

2

1

0

1

2

3

4

log Scale

Bach's Upper Bound

0

200

400

600

800

1000

1/

3

2

1

0

1

2

3

Bach's Upper Bound

25

50

75

100

125

150

175

200

7

6

5

4

3

2

log scale

Test Error on tNTK

Median

Quartiles

25

50

75

100

125

150

175

200

Test Error on LK

Median

Quartiles

N

N

References

Pierre Alquier, James Ridgway, and Nicolas Chopin. On the properties of variational approximations of Gibbs posteriors. J. Mach. Learn. Res., 17:Paper No. 239, 41, 2016.

Arash Amini, Richard Baumgartner, and Dai Feng. Target alignment in truncated kernel ridge regression. Advances in Neural Information Processing Systems, 35:21948-21960, 2022.

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pages 322-332. PMLR, 2019.

Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. Factors of transferability for a generic convnet representation. IEEE transactions on pattern analysis and machine intelligence, 38(9):1790-1802, 2015.

Francis Bach. High-dimensional analysis of double descent for linear regression with random projections. arXiv preprint arXiv:2303.01372, 2023.

Francis Bach. Learning theory from first principles. Online version, 2023.

Peter L. Bartlett, Olivier Bousquet, and Shahar Mendelson. Local Rademacher complexities. Ann. Statist., 33(4):1497-1537, 2005.

Peter L. Bartlett and Shahar Mendelson. Rademacher and Gaussian complexities: risk bounds and structural results. J. Mach. Learn. Res., 3(Spec. Issue Comput. Learn. Theory):463-482, 2002.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machinelearning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849-15854, 2019.

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pages 1024-1034. PMLR, 2020.

Enea Monzio Compagnoni, Anna Scampicchio, Luca Biggio, Antonio Orvieto, Thomas Hofmann, and Josef Teichmann. On the effectiveness of randomized signatures as reservoir for learning rough dynamics. IEEE IJCNN 2023, 2023.

Eernesto De Vito, Lorenzo Rosasco, and Andrea Caponnetto. Discretization error analysis for tikonov regularization. Analysis and Applications, 04(01):81-99, 2006.

Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 647-655, Bejing, China, 22-24 Jun 2014. PMLR.

L. Gonon, L. Grigoryeva, and J.P Ortega. Approximation bounds for random neural networks and reservoir systems. Annals of Applied Probability, 2022.

Lukas Gonon, Lyudmila Grigoryeva, and Juan-Pablo Ortega. Risk bounds for reservoir computing. Journal of Machine Learning Research (JMLR), 21, 2020.

Lukas Gonon, Lyudmila Grigoryeva, and Juan-Pablo Ortega. Infinite-dimensional reservoir computing. arXiv preprint arXiv:2304.00490, 2023.

Calypso Herrera, Florian Krach, Pierre Ruyssen, and Josef Teichmann. Optimal stopping via randomized neural networks. arXiv preprint arXiv:2104.13669, 2021.

Songyan Hou, Parnian Kassraie, Anastasis Kratsios, Jonas Rothfuss, and Andreas Krause. Instance-dependent generalization bounds via optimal transport. arXiv preprint arXiv:2211.01258, 2022.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.

Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Implicit regularization of random feature models. In International Conference on Machine Learning, pages 4631-4640. PMLR, 2020.

Haotian Ju, Dongyue Li, and Hongyang R Zhang. Robust fine-tuning of deep neural networks with hessian-based generalization guarantees. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 10431-10461. PMLR, 17-23 Jul 2022.

Sham M Kakade, Karthik Sridharan, and Ambuj Tewari. On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. Advances in neural information processing systems, 21, 2008.

Xuhong LI, Yves Grandvalet, and Franck Davoine. Explicit inductive bias for transfer learning with convolutional networks. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2825-2834. PMLR, 10-15 Jul 2018.

Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel 'ridgeless' regression can generalize. The Annals of Statistics, 48(3), Jun 2020.

Fanghui Liu, Xiaolin Huang, Yudong Chen, and Johan AK Suykens. Random features for kernel approximation: A survey on algorithms, theory, and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):7128-7148, 2021.

Fanghui Liu, Zhenyu Liao, and Johan Suykens. Kernel regression in high dimensions: Refined analysis beyond double descent. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pages 649-657. PMLR, 13-15 Apr 2021.

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Generalization error of random feature and kernel methods: hypercontractivity and kernel matrix concentration. Applied and Computational Harmonic Analysis, 2021.

Zakaria Mhammedi, Peter Grünwald, and Benjamin Guedj. Pac-bayes un-expected bernstein inequality. Advances in Neural Information Processing Systems, 32, 2019.

Theodor Misiakiewicz. Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression, 2022.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.

K. B. Petersen and M. S. Pedersen. The matrix cookbook, nov 2012. Version 20121115.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.

Basri Ronen, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. Advances in Neural Information Processing Systems, 32, 2019.

Alessandro Rudi, Raffaello Camoriano, and Lorenzo Rosasco. Less is more: Nyström computational regularization. Advances in Neural Information Processing Systems, 28, 2015.

Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. Advances in neural information processing systems, 30, 2017.

Alexander Tsigler and Peter L. Bartlett. Benign overfitting in ridge regression. Journal of Machine Learning Research, 24(123):1-76, 2023.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027, 2010.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.

Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.

Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Advances in neural information processing systems, 27, 2014.

Appendix

A Glossary

B Classical KRR Theory

In an effort to keep our manuscript as self-contained as possible, we recall the Mercer decomposition, representer theorem for kernel ridge regression as well as the form of the bias-variance tradeoff in the KRR context.

B.1 Mercer Decomposition

We begin with a general kernel K ( ∞ ) : X × X → R .

Proposition B.1. [12] Fix a sample distribution ρ . Let K ( ∞ ) : X ×X → R be a reproducing kernel with corresponding RKHS H ( ∞ ) . There exists a decreasing sequence of real numbers λ 1 ≥ λ 2 ≥ ... , called the eigenvalues of the kernel K ( ∞ ) ; and a sequence of pairwise-orthonormal functions { ψ k } ∞ k =1 ⊂ L 2 ρ , called the eigenfunctions of K ( ∞ ) , such that for all x, x ′ ∈ X , we have

In particular, we assume λ k = 0 , ∀ k > M . In this case, we say the kernel K ( x, x ′ ) = ∑ M k =1 λ k ψ k ( x ) ψ k ( x ′ ) is of finite rank M with corresponding (finite-dimensional) RKHS H , recovering equation (2).

The first of these results, allows us to explicitly express the finite-rank kernel ridge regressor f Z ,λ .

Proposition B.2 (Representer Theorem - [39, Chapter 12]) . Let R def. = ( K + λN I N ) -1 ∈ R N × N be the resolvent matrix and recall the kernel ridge regressor f Z ,λ given by equation (3):

Then, for every x ∈ X , we have the expression

where K x def. = [ K ( x i , x )] N i =1 ∈ R N × 1 .

B.2 Compact Matrix Expression

First, let Ψ def. = ( ψ k ( x i )) M,N k =1 ,i =1 be the random M × N matrix defined by evaluating the M eigenfunctions on all input training instances X def. = ( x i ) N i =1 , Λ def. = diag[ λ k ] ∈ R M × M , and ψ ( x ) def. = [ ψ k ( x )] M k =1 ∈ R M × 1 . The advantage of this notation is that we can rewrite the equations in a more compact form. For equation (17):

Last but not least, we define some important quantities for later analysis.

Definition B.3 (Fluctuation matrix) . The fluctuation matrix is the random M × M -matrix given by ∆ def. = 1 N ΨΨ ⊤ -I M . Our analysis will often involve the operator norm of ∆ , which we denote by δ def. = ∥ ∆ ∥ op .

The fluctuation matrix ∆ measures the first source of randomness in the KRR's test error. Namely it encodes the degree of non-orthonormality between the vectors obtained by evaluating of the M eigenfunctions ψ 1 , . . . , ψ M on the input X .

The second source of randomness in the KRR's test error comes from the empirical evaluation of the dot product of the eigenfunction ψ k 's and the orthogonal complement ψ >M :

Definition B.4 (Error Vector) . E def. = 1 N Ψ ψ >M ( X ) is called the error vector.

The random matrix ∆ and the random vector E are centered; i.e. E X [ ∆ ] = 0 and E X [ E ] = 0 .

for equation (4):

B.3 Bias-Variance Decomposition

The derivation of several contemporary KRR generalization bounds [6, 26, 27] involves the classical Bias-Variance Trade-off:

Proposition B.5 (Bias-Variance Trade-off) . Fix a sample Z . Recall the definition 3.4 of test error R Z ,λ , bias, and variance:

Then, we can write variancetest = E x,ε ( K ⊤ x R ε ) 2 and hence the test error R Z ,λ admits a decomposition:

C Proofs

In this section, we will derive the essential lemmata and propositions for proving the main theorems.

C.1 Formula Derivation

We begin with writing the test error in convenient forms.

C.1.1 Bias

We first derive, from the definition of the bias, a convenient expression to proceed:

Proposition C.1 (Bias Expression) . Let Ψ >M def. = [ ψ >M ( x i )] N i =1 as an 1 × N - row vector, ( Ψ Ψ >M ) as an ( M +1) × N matrix. Denote P def. = ( P ≤ M P >M ) = ΛΨR ( Ψ ⊤ Ψ ⊤ >M ) ∈ R M × ( M +1) , P ≤ M ∈ R M × M and P >M ∈ R M × 1 . Then the bias admits the following expression:

Proof. Recall that, by equations (18) and (19), we can write

Hence

in line (20), we use Parseval's identity.

We proceed by reformulating the projection matrix P , first with the left matrix P ≤ M :

Lemma C.2. Recall the following notations

Define the symmetric random matrix B def. = ( I M + ∆ + λ Λ -1 ) -1 . It holds that P ≤ M = I M -λ BΛ -1 .

Proof. We first observe that

From lines (21)- (22) and the definition of the fluctuation matrix ∆ we deduce

Rearranging (23) and applying the definition of B we find that

Arguing analogously for the right matrix P >M , we draw the subsequent similar conclusion.

Lemma C.3. Recall the following notations

We have that P >M = B E .

Proof. Similarly to (21)- (22) we note that

Analogously to the computations in (24)-(25)

Lemma C.4 (Fitting Error) . Recall the notation

We have fitting error = ∥ ∥ B ( λ Λ -1 ˜ γ -E ˜ γ >M )∥ ∥ 2 2 .

Proof. By lemmata C.2 and C.3,

Hence we come up with a new expression of the bias:

Proposition C.5 (Bias) . Recall that B def. = ( I M + ∆ + λ Λ -1 ) -1 . The bias E x ( f λ X ( x ) -˜ f ( x ) ) 2 has the following expression:

Proof. We apply Proposition C.1 and Lemma C.4 to obtain the result.

C.1.2 Variance

If we consider noise in the label, we have to compute the variance part of the test error.

We can further simplify the variance part:

Theorem C.7 (Variance) . Recall that B def. = ( I M + ∆ + λ Λ -1 ) -1 . The variance part, variance, can be expressed as:

Proof. We argue similarly as in lemma C.2. Since

therefore, we deduce that

By leveraging the identity M = Ψ ⊤ Λ 2 Ψ and elementary properties of the trace map, the computations in (26)-(27) imply that

in more detail: in line (28), we use the definition of M ; in line (29), we use the fact that both Λ and R are symmetric; in line (30), we use line (27); in line (31), we use the cyclicity of the trace; in line (32), we use the definition of ∆ ; in line (33), we use the symmetry of B . We obtain the result upon applying Lemma C.6.

C.1.3 Test Error

The Bias-Variance trade-off (see Proposition B.5) decomposed the KRR's test error into two terms, the bias and variance. Since Propositions C.5 and C.7 give us exact expressions for the bias and variance, respectively, we deduce the following exact expression for the KRR's test error.

Theorem C.8 (Exact Formula for KRR's Test Error) . The test error R Z ,λ of KRR equals

where B def. = ( I M + ∆ + λ Λ -1 ) -1 .

Proof. We begin with the bias/variance decomposition:

For the validation of the Theorem C.8, please see Appendix D for details.

The matrix B plays an important role in the expression since it encodes most information of the KRR. Therefore, the following subsection will discuss the approximation of the matrix B .

C.2 Matrix Approximation

Recall that the matrix B def. = ( I M + ∆ + λ Λ -1 ) -1 is the inverse of a random matrix. The following lemma helps to approximate B . Informally, it says that: given that δ def. = ∥ ∆ ∥ op < 1 . We have

in operator norm ∥ · ∥ op for an M × M matrix ¯ P depending only on the M eigenvalues { λ k } M k =1 and on the ridge λ > 0 . More precisely we have the following.

Lemma C.9 ( B -Expansion) . Given that δ def. = ∥ ∆ ∥ op < 1 . It holds that

where ¯ P def. = diag [ λ k λ k + λ ] k = Λ ( Λ + λ I M ) -1 ∈ R M × M .

Proof. Set A = I M + λ Λ -1 and repeatedly use the formula ( A + ∆ ) -1 = A -1 -A -1 ∆ ( A + ∆ ) -1 from [31], we have

Note that A -1 = ( I M + λ Λ -1 ) -1 = Λ ( Λ + λ I M ) -1 = ¯ P with operator norm λ 1 λ 1 + λ < 1 , hence we have ( A -1 ∆ ) n +1 = ( -¯ P∆ ) n +1 → 0 in operator norm as n →∞ . Hence

in operator norm.

Due to the convergence result in lemma C.9, it is natural to define:

Definition C.10. For any n ∈ N ∪ {∞} , write B ( n ) = ∑ n s =0 ( -¯ P∆ ) s ¯ P . For example, We have

Although lemma C.9 is valid when δ < 1 , we need a slightly stronger condition that δ is upper bounded by an arbitrary constant strictly small than 1. For simplicity, we assume this constant to be 1 2 in the following lemma:

Lemma C.11 ( B -Approximation) . Assume that δ def. = ∥ ∆ ∥ op < 1 2 . Let B ( n ) = ∑ n s =0 ( -¯ P∆ ) s ¯ P be the n th-order approximation of the matrix B as in definition C.10. Then we have

Proof. We first bound the operator norm of the matrix B : since the minimum singular value of the matrix ¯ P -1 + ∆ is at least

Hence ∥ ∥ B -B ( n ) ∥ ∥ op ≤ ∥ ∥ ¯ P∆ ∥ ∥ n +1 op ∥ B ∥ op < ∥ ∆ ∥ n +1 op · 2 = 2 δ n +1 , since we have ∥ ∥ ¯ P ∥ ∥ op = λ 1 λ 1 + λ < 1 .

Note that the upper bound 1 2 of δ can be replaced by any constant strictly small than 1 to get a similar conclusion.

Remark C.12. Using the concentration result from random matrix theory, for M < N , one can show with high probability that the operator norm δ of the fluctuation matrix ∆ is less than 1. 5

See subsection C.3 for details. Then we can use the the above lemmata C.9 and C.11 to approximate the test error of KRR:

Proposition C.13 (Bias Approximation) . Fix a sample Z of ρ such that δ def. = ∥ ∆ ∥ op < 1 2 . Then the biastest term is bounded above and below by

where ¯ P def. = Λ ( Λ + λ I M ) -1 , w = λ Λ -1 ˜ γ -˜ γ >M E , and p ( δ ) def. = 5 + 4 δ + 4 δ 2 . By writing E = ( η k ) M k =1 , the bounds simplify to

and hence

Also, we have

Proof. Let w = λ Λ -1 ˜ γ -˜ γ >M E . We apply lemma C.4 followed by the 1st-order approximation B (1) of the matrix B in lemma C.11:

Hence we have the upper bound:

We argue similarly for the lower bound using: ∥ A ∥ op ∥ v ∥ 2 2 ≥ v ⊤ A v ≥ -∥ A ∥ op ∥ v ∥ 2 2 for any A ∈ R M × M , v ∈ R M × 1 .

We argue similarly for variance.

Proposition C.14 (Variance Approximation) . Fix a sampling Z such that δ def. = ∥ ∆ ∥ op < 1 2 . Then we have

where p ( δ ) def. = 5 + 4 δ +4 δ 2 , and σ 2 def. = E [ ϵ 2 ] is the noise variance.

Proof. Note that Tr A ≤ M ∥ A ∥ op for any matrix A ∈ R M × M . Since B 2 ( I M + ∆ ) = ( B (1) ) 2 ( I M + ∆ ) + 2 B (1) ( B -B (1) ) ( I M + ∆ ) + ( B -B (1) ) 2 ( I M + ∆ ) , we can bound the residue term by δ :

For the main terms, we have

We apply Theorem C.7 to yield a bound on variance:

Note that the above propositions C.13 and C.14 give absolute (non-probabilistic) bounds on the test error, once δ is controlled.

C.3 Concentration Results

In this subsection, we focus on bounding the operator norm δ of the fluctuation matrix ∆ .

First, we establish some concentration results.

Lemma C.15 (Theorem 3.59 in [37]) . Let A be an n × N matrix with independent isotropic subGaussian columns in R n which sub-gaussian norm is bounded by a positive constant G > 0 . Then for all t ≥ 0 , with probability at least 1 -2 exp( -1 3 t 2 ) , we have

where a def. = C √ n N + t √ N , for all constant C ≥ 12 G 2 .

Proof. Let a def. = C √ n N + t √ N with C > 0 to be determined, and ϵ def. = max { a, a 2 } . The first step to show that :

for some 1 4 -net N on the sphere S n -1 ⊂ R n . Choose such a net N with |N| < ( 1 + 2 1 / 4 ) n = 9 n .

Let A i be the i th column of the matrix A and let Z i def. = A ⊤ i x be a random variable. By definition of A , Z i is centered with unit variance with sub-Gaussian norm upper bounded by G . Note that G ≥ 1 √ 2 E [ Z 2 i ] 1 / 2 = 1 √ 2 , and the random variable Z 2 i -1 is centered and has sub-exponential norm upper bounded by 4 G 2 . Hence by an exponential deviation inquality 6 , we have, for any x ∈ S n -1 :

Then by union bound, we have

for C ≥ √ 2 e log 9 G 2 . Since 12 > √ 2 e log 9 , for simplicity, we assume C > 12 G 2 . Moreover, since G ≥ 1 √ 2 , we have 1 2 e -1 G -4 ≤ 1 3 , we have

Then by the 1 4 -net argument, with probability at least 1 -2 exp ( -1 3 t 2 ) , we have

Lemma C.16. Assume Assumption 4.1 holds and that N > exp(4(12 G 2 ) 2 ( M +1)) . Then with a probability of at least 1 -2 /N , we have

Proof. Set n = M +1 , A = ( Ψ ≤ M ψ >M ( X ) ⊤ ) ∈ R ( M +1) × N . Then

where η >M def. = 1 N ∑ N i =1 ψ >M ( x i ) 2 -1 . On one hand, the operator norm of the above matrix bounds δ and ∥ E M ∥ 2 from above:

and

On the other hand, set t = 1 2 log N, C = 12 G 2 , since N > exp(4 C 2 ( M +1)) , we have

By Lemma C.16, then with probability of at least 1 -2 exp( -1 3 t 2 ) = 1 -2 exp( -1 12 ) /N > 1 -2 /N , we have

Combine the both results and we conclude the upper bounds.

In particular, as N → ∞ , δ vanishes almost surely. In empirical calculation, if the requirement N > exp(4(12 G 2 ) 2 ( M +1)) exponential in M is too demanding for a large integer M , we can take t = N s for any positive number s ∈ ( 0 , 1 2 ) instead of t = 1 2 log N . In this way, we decrease the requirement to N polynomial in M in sacrificing the decay from O (√ logN N ) to O ( N s -1 / 2 ) . For simplicity purpose, we do not list out the result with this decay in this paper.

C.4 Refined Test Error Analysis

We can apply the above concentration results to refine the following bounds on the finite-rank KRR test error. First of all, we realize the decay of target function coefficient comparable to the spectral decay:

Definition C.17 (Comparable Decay) . Denote r def. = min k {| ˜ γ k /λ k |} and r def. = max k {| ˜ γ k /λ k |} .

C.4.1 Refined Bounds on Bias

Recall that Proposition C.13 bounding the bias in terms of δ and η k . For the former one, we can choose: for N > max { exp(4(12 G 2 ) 2 ( M +1)) , 9 } , by Lemma C.16, with probability of at least 1 -2 /N , we have δ ≤ √ log N N < √ log 9 9 < 1 2 . For the latter one, we have to control the vector w :

Lemma C.18. Let w = λ Λ -1 ˜ γ -˜ γ >M E . We have

Proof. Since λ 2 r 2 M ≤ ∥ ∥ λ Λ -1 ˜ γ ∥ ∥ 2 2 ≤ λ 2 r 2 M and ∥ ˜ γ >M E ∥ 2 2 = ˜ γ 2 >M ∥ E ∥ 2 2 , we have

Similarly, we can bound ∥ ∥ ¯ P w ∥ ∥ 2 . Observe that:

Since 1 ≥ λ λ k + λ ≥ λ λ M + λ , we have the upper bound:

where ˜ f ≤ M def. = ∑ M k =1 ˜ γ k ψ k = ˜ f -˜ γ >M ψ >M . For the lower bound, we have:

Similarly, since 4 λλ k ≤ ( λ k + λ ) 2 ,

And

Combining the above result, we state the following theorem:

Theorem C.19. For N > max { exp(4(12 G 2 ) 2 ( M +1)) , 9 } and for any constant C 1 > 8 ( λr √ M + 1 2 | ˜ γ >M | ) 2 + 5 2 ∥ ˜ f ∥ 2 L 2 ρ (independent to N ), with a probability of at least 1 -2 /N , we have the upper and lower bounds of bias:

For λ → 0 , we have a simpler bound: with a probability of at least 1 -2 /N , we have

For ˜ γ 2 >M = 0 , that is ˜ f ∈ H , we have a simpler upper bound on bias: with a probability of at least 1 -2 /N , we have

Proof. By Proposition C.13 and Lemma C.18,

where in line (39), we use Proposition C.13; in line (40), we use Lemma C.18; in line (40), we use the fact that 2 ab ≤ a 2 + b 2 where a = | ˜ γ >M | , b = ∥ ˜ f ≤ M ∥ L 2 ρ .

Now we apply the concentration result in Lemma C.16: with a probability of at least 1 -2 /N :

where we choose C 1 > 0 to be such that:

Hence we have an upper bound for the bias. We argue similarly for the lower bound:

For λ → 0 , note that w →-˜ γ >M E . This yields

Hence, by plugging in δ < 1 2 , with probability of at least 1 -2 /N ,

For lower bound, it follows similarly:

and we obtain line (37). For the case where ˜ γ >M = 0 , recalculate and simplify line (40) to obtain line (38).

C.4.2 Refined Bounds on Variance

Similarly, we can refine Theorem C.14 to get a bound on the variance:

Theorem C.20. For N > max { (12 G ) 4 ( M +1) 2 , 9 } , and set C 2 = 12 (independent to N ), with a probability of at least 1 -2 /N , we have the upper and lower bounds of variance:

Proof. We argue analogously as in Theorem C.19: by Proposition C.14 and Lemma C.16, we have

Hence we can choose C 2 = 12 . For the lower bound, since λ 2 k ( λ k + λ ) 2 > λ 2 M ( λ M + λ ) 2 , we have

Note that in both Theorems C.19 and C.20, the constants C 1 , C 2 > 0 is not optimized.

D Numerical Validation

In this section, we illustrate our result for KRR with two different finite rank kernels.

D.1 Truncated NTK

First, we need to define a finite-rank kernel K : X × X → R . We set X = S 1 ⊂ R 2 . By reparametrization, we write S 1 ∼ = [0 , 2 π ] / 0 ∼ 2 π . We assume the data are drawn uniformly on the circle, that is ρ X = unif[ S 1 ] . We can use the Fourier functions cos( k · ) , sin( k · ) as the orthogonal eigenfunctions of the kernel. We define the NTK

for all θ, θ ′ ∈ [0 , 2 π ] . 3) We choose a rankM truncation K ( θ, θ ′ ) = ∑ M k =1 λ k ψ k ( θ ) ψ k ( θ ′ ) for all θ, θ ′ ∈ [0 , 2 π ] . For the first few eigenvalues of the kernel, please see Table 3 for example. Before

proceeding to test error computation, we present a training example, Figure 4, to give readers an intuition on the truncated NTK (tNTK).

3

2

1

0

1

2

3

x

1.0

0.5

0.0

0.5

1.0

y

Target Function

Regressor

Sample

3

2

1

0

1

2

3

x

tNTK Training

Target Function

Regressor

Sample

D.2 Test Error Computations

In the following tNTK training, we set the hyperparameters as follows:

Target function We choose a simple target function ˜ f ( x ) = cos x = 1 √ 2 ψ 2 ( x ) . Throughout the experiment, we set the noise variance σ 2 = 0 . 05 .

Ridge We choose λ = σ 2 N . In Figure 5 (left), we set N = 50 , λ = 0 . 05 / 50 for tNTK training; (right) we set set λ = 0 . 05 / 50 for varying N from 10 to 200.

Error bars In Figure 7 (right), for each value of N , we run over 10 iterations of random samples and compute the test error. The error bars are shown as the difference between the upper and the lower quartiles.

3

2

1

0

1

2

3

x

1.0

0.5

0.0

0.5

1.0

y

tNTK training

Target Function

Regressor

Sample

25

50

75

100

125

150

175

200

N

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

Test Error

Varying Sample Size

Median

Quartiles

Lower bound See the subsection below.

D.3 Bound Comparison

We continue with the experiment on the tNTK this time with varying N and compare our upper bound with [6].

Upper bounds In Figure 6, the expression of Bach's and our upper bounds are directly computed:

where the constants ∥ ˜ f ∥ 2 H and R 2 can be computed directed from the choice of kernel and target function. For simplicity reason, we drop the residue term C 1 log N N since it is overshadowed by the other terms and the constant C 1 is not optimized.

D.4 Legendre Kernel

To illustrate the bounds with another finite-rank, we choose a simple legendre kernel (LK):

where P k is the Legendre polynomial of degree k , and λ k > 0 are the eigenvalues.

25

50

75

100

125

150

175

200

N

4

2

0

2

4

log Scale

Bach's Upper Bound

0

200

400

600

800

1000

1/

4

3

2

1

0

1

2

3

Bach's Upper Bound

Eigenvalues To better compare the Legendre kernel K with the NTK, we choose λ k = C · ( k +1) -2 of quadratic decay such that the spectral sums are the same: ∑ ∞ k =0 λ k = 0 . 5 . Hence we choose C = 0 . 5 / ∑ ∞ k =1 k -2 = 3 π 2 .

Target function We choose a simple target function ˜ f ( x ) = x 2 = 1 3 P 0 ( x ) + 2 3 P 2 ( x ) . Throughout the experiment, we set the noise variance σ 2 = 0 . 05 .

D.5 Test Error Computation

Ridge As before, our bound suggests that, to balance the bias and the variance with a fixed N , we can choose λ = σ 2 N . In Figure 7 (left), we set N = 50 , λ = 0 . 05 / 50 for KRR training; (right) we set set λ = 0 . 05 / 50 for varying N from 10 to 200.

Error bars In Figure 7 (right), for each value of N , we run over 10 iterations of random samples and compute the test error. The error bars are shown as the different between the upper and the lower quartiles. The median is taken as average.

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

x

0.4

0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

y

Legendre Kernel Training

Regressor

Target Function

Sample

25

50

75

100

125

150

175

200

N

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Test Error

Varying Sample Size

Median

Quartiles

Upper bounds In Figure 8, the expression of Bach's and our upper bounds are directly computed:

where the constants ∥ ˜ f ∥ 2 H and R 2 can be computed directed from the choice of kernel and target function.

25

50

75

100

125

150

175

200

N

4

3

2

1

0

1

2

3

4

log Scale

Varying Sample Size on LK

Bach's Upper Bound

0

200

400

600

800

1000

1/

3

2

1

0

1

2

3

Bach's Upper Bound

Lower Bound Last but not least, we need to show our lower bound is valid. To see this clearly, we need to write the bound in exact sums instead of in HKRS norm square ∥ ˜ f ∥ 2 H : namely, we compute I

instead of using the inequality (42) in Lemma C.18; and

instead of using the inequality (42) in Theorem C.20. Then we can compute our bounds as:

and we drop the residue terms C 1 log N N and C 2 σ 2 N M log N N by the same reason as before. From Figure 3, we can see that our bounds precisely describe the decay of the test error. Our bounds are not 'bounding' the test errors in smaller instances due to the absence of the residue terms, which increases the interval of confidence of our approximation. But for larger instances, say N > 100 , all upper and lower bounds, and the averaged test error converge to the same limit.

25

50

75

100

125

150

175

200

N

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

Test Error

Test Error on tNTK and its Bounds

Median

Quartiles

25

50

75

100

125

150

175

200

N

7

6

5

4

3

2

log scale

log Scale

Median

Quartiles

25

50

75

100

125

150

175

200

N

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Test Error

Test Error on LK and its Bounds

Median

Quartiles

25

50

75

100

125

150

175

200

N

7

6

5

4

3

log Scale

Median

Quartiles

Peer reviews:
comments: SUMMARY:
The authors address the problem of kernel ridge regression with a finite rank kernel, in the under-paramatrized regime. They prove sharper upper bounds on the test error, compared to previous existing works.

STRENGTHS:
The discussion is very well-written and easy to follow, with a number of illustrations being provided to set up the problem. Careful and detailed comparison to previous work is given, making it easy to understand the novelty of the present manuscript. Overall, the addressed problem is an important one, and I believe the derived bound will be of interest to the community. I have not checked the proof, and therefore give a low confidence score.

WEAKNESSES:
I do not have any major concern. A minor remark is that while all experimental details are provided in Appendix D, it would be good to add at least a short description of the target functions and regularization used in Fig.1 and 2 in the main text or in the corresponding captions.

QUESTIONS:
- (minor) While the comparison with the bound of [6] in Fig. 2 is compelling, is there a reason why the bound of [30] is not also plotted? Its addition, or a brief sentence justifying its non-inclusion, would be welcome.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-14

comments: SUMMARY:
The paper highlights the inadequacy of existing statistical learning guarantees for general kernel regressors when applied to finite-rank kernels. The authors have addressed this issue by developing non-asymptotic upper and lower bounds for the test error of finite-rank kernel ridge regression. These new bounds are more precise than the previously established ones and are applicable for any regularization parameters. This research provides a more dependable framework for utilizing finite-rank kernels in machine learning applications.

STRENGTHS:
1. The paper addresses an important gap in the current understanding of machine learning algorithms by developing more accurate and reliable bounds for finite-rank kernel ridge regression, which is frequently used in practical applications.

2. The research provides a more precise and dependable framework for using finite-rank kernels in machine learning problems, which could result in better performance and more efficient algorithms.

WEAKNESSES:
1. The paper only considers under-parameterized regime.

2. Low bound is a main argument of this paper, but all details are given in the Appendix.

QUESTIONS:
1. It looks a little bit weird that you don't need to take $\lambda=O(N^\theta)$ for some $\theta<0$ to balance the bias and variance. Any insights for this?

2. In Theorem 4.1 and Corollary 4.3.1, what are the mild conditions on the kernel K? I cannot find the mild condition on input distribution either.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the test error of ridge regression with a finite rank kernel. The finite rank kernel appears in several practical settings such as transfer learning and random neural networks. The authors provide a new analysis in this setting using tools from random matrix theory. A detailed comparison to other generalization bounds is presented.\n\nSTRENGTHS:\n1. New generalization results in a practical and challenging setting.\n2. New analysis techniques.\n\nWEAKNESSES:\n1. The comparison to standard generalization results is not clear (Eq. (12)). Both bounds scale as $\\sqrt{\\frac{\\log(n)}{n}}$. It is not clear which one is tighter.\n2. The technical details of the proof are not given and the novelty of the analysis is not explained in detail. Instead, most of the paper is devoted to a discussion and experiments.\n3. There are some unclear technical issues (see questions below).\n\nQUESTIONS:\n1. The bound in Theorem 4.1 and the bound in Eq. (12) both scale as $\\sqrt{\\frac{\\log(n)}{n}}$. Which one is tighter and why? What is C in Eq. (12)?\n2. The test error definition is not standard: Shouldn’t the label be $y$ in the MSE and not $\\tilde{f}$? Is the definition of the test error the same in other works? Also, the bias-variance decomposition does not seem to be standard. Usually the learned predictor is the same in the bias expression (e.g., see the Wikipedia definition of bias-variance decomposition), whereas here the predictor is different: it changes from $f_{Z,\\lambda}$ to $f_{(X,f(X)),\\lambda)}$. Is this standard for analyzing kernel methods?\n3. Where in the analysis is the under-parameterized assumption ($M <N$) needed?\n4. In Eq. (10) if $\\tilde{\\gamma}_{>M} = 0$, the bias is exactly zero for all $N$? Is this a mistake?\n\nLIMITATIONS:\nLimitations are not discussed in detail. Some of the limitations are mentioned throughout the paper. Maybe it will be helpful to add a limitations section and summarize all limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper studies the kernel ridge regression under the non-asymptotic setting. The authors give the upper and lower bounds for bias and variance term, respectively. The authors argue the results improve upon those in Bach 2023.

STRENGTHS:
The paper is well-structured. The authors give rigorous proofs, following by careful experiments.

WEAKNESSES:
1. The paper requires further improvement and polishing in writing. To name a few, line 57-58; line 107-110.

2. The paper would benefit from a more consistent and standardized use of symbols and notations. For example, in line 90, it would be better to use L_2(\rho) instead of L_\rho^2. In lien 92, the notation of \tilde{f}(\textbf{X}) is not proper. In Definition 3.1, it would be better to include the decreasing order of eigenvalues in the statement rather than adding an additional remark 3.2. In line 200-207, notation K^{(\infty)}

3. As mentioned in Bach 2023, more refined bounds can be found in Rudi et al. 2015, Rudi and Rosasco 2017. However, the authors failed to mention them and other related results in the comparison. In the absence of such comparisons, it is hard to tell the novelty and improvements of the current submission.

4. The dependency of \lambda seems to be incorrect for the variance term.

5. Given Corollary 4.3.1 in the submission, I cannot see significant improvements against those in Bach 2023. Also, the authors did not give a proper explanation for considering \lambda goes to 0.

QUESTIONS:
Could the authors mention the lower bound for the problem? 

With optimal choice of \lambda, we can derive the optimal upper bound for the test error. However, the current result does not show a bias-variance tradeoff with respect to \lambda. Could the authors explain?

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-08

comments: SUMMARY:
This paper gives a refined analysis of the test error of kernel ridge regression with a finite kernel in the underparametrized regime, where the rank M of the kernel is smaller than the number of data points N. The analysis improves upon previous ones by providing tighter non-asymptotic upper and lower bounds, which remain tight even as the ridge parameter goes to zero. When the target function lies in the RKHS, the bounds show that estimation is consistent as N goes to infinity.

STRENGTHS:
The paper is well written and clear. The authors state the contributions clearly and compare carefully with previous work. Establishing tight bounds for this basic learning setting is an important problem.

WEAKNESSES:
No major weaknesses, just some minor typos. I would also suggest that the authors include the explicit lower bounds in the main paper. Given that this a major focus of the paper it should not be required for the reader to go to the appendix. It would also be nice for the appendix to give a bit clearer account for the technical differences in the proof techniques of their bounds versus previous ones.

QUESTIONS:
- Can the authors elaborate the technical difficulties in extending beyond the finite rank setting, or even when M ~ N? Can analogous results be shown when the kernel is approximately low rank?

- It is unclear that realistic NTK or random features will fall in this underparametrized regime. Can the authors give elaborate if this should be the case? Is it possible to experiment in more realistic NTK settings to truncate the kernel and apply the bounds?

LIMITATIONS:
No limitations. Authors give some directions or future work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-27