Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime

1 Introduction . Kernel methods are among the most popular models in machine learning . Despite their relative simplicity , they define a powerful framework in which non-linear features can be exploited without leaving the realm of convex optimisation . Kernel methods in machine learning have a long and rich literature dating back to the 60s [ 1 , 2 ] , but have recently made it back to the spotlight as a proxy for studying neural networks in different regimes , e.g . the infinite width limit [ 3–6 ] and the lazy regime of training [ 7 ] . Despite being defined in terms of a non-parametric optimisation problem , kernel methods can be mathematically understood as a standard parametric linear problem in a ( possibly infinite ) Hilbert space spanned by the kernel eigenvectors ( a.k.a features ) . This dual picture fully characterizes the asymptotic performance of kernels in terms of a trade-off between two key quantities : the relative decay of the eigenvalues of the kernel ( a.k.a . its capacity ) and the coefficients of the target function when expressed in feature space ( a.k.a . the source ) . Indeed , a sizeable body of work has been devoted to understanding the decay rates of the excess error as a function of these two relative decays , and investigated whether these rates are attained by algorithms such as stochastic gradient descent [ 8 , 9 ] . Rigorous optimal rates for the excess generalization error in kernel ridge regression and are wellknown since the seminal works of [ 10 , 11 ] . However , recent interesting works [ 12 , 13 ] surprisingly reported very different - and actually better - rates supported by numerical evidences . These papers appeared to either not comment on this discrepancy [ 13 ] , or to attribute this apparent contradiction to a difference between typical and worse-case analysis [ 12 ] . As we shall see , the key difference between these works stems instead from the fact that most of classical works considered noisy data and fine-tuned regularization , while [ 12 , 13 ] focused on noiseless data sets . This observation raises a number of questions : is there a connection between both sets of exponents ? Are Gaussian design 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . exponents actually different from worst-case ones ? What about intermediary setups ( for instance noisy labels with generic regularization , noiseless labels with varying regularization ) and regimes ( intermediary sample complexities ) ? How does infinitesimal noise differ from no noise at all ? Main contributions — In this manuscript , we answer all the above questions , and redeem the apparent contradiction by reconsidering the Gaussian design analysis . We provide a unifying picture of the decay rates for the excess generalization error , along a more exhaustive characterization of the regimes in which each is observed , evidencing the interplay of the role of regularization , noise and sample complexity . We show in particular that typical-case analysis with a Gaussian design is actually in perfect agreement with the statistical learning worst-case data-agnostic approach . We also show how the optimal excess error decay can transition from the recently reported noiseless value to its well known noisy value as the number of samples is increased . We illustrate this crossover from the noiseless regime to the noisy regime also in a variety of KRR experiments on real data . Related work — The analysis of decay rates for kernel methods and ridge regression is a classical topic in statistical learning theory [ 10 , 11 , 14 , 15 ] . In this classical setting , decay exponents for optimally regularized noisy linear regression on features with power-law co-variance spectrum have been provided . Interestingly , it has been shown that such optimal rates can be obtained in practice by stochastic gradient descent , without explicit regularization , with single-pass [ 16 , 17 ] or multi-pass [ 8 ] algorithms , as well as by randomized algorithms [ 18 ] . Closed-form bounds for the prediction error have been provided in a number of worst-case analyses [ 18 , 19 ] . We show how the decay rates given in the present paper can also be alternatively deduced therefrom in Appendix E. The recent line of work on the noiseless setting includes contributions from statistical learning theory [ 9 , 18 ] and statistical physics [ 12 , 13 ] . This much more recent second line of work proved decay rates for a given , constant regularization . An example of noise-induced crossover is furthermore mentioned in [ 9 ] . The interplay between noisy and noiseless regimes has also been investigated in the related Gaussian Process literature [ 20 ] . The study of ridge regression with Gaussian design is also a classical topic . Ref . [ 21 ] considered a model in which the covariates are isotropic Gaussian in Rp , and computed the exact asymptotic generalization error in the high-dimensional asymptotic regime p , n→∞ with dimension-to-samplecomplexity ratio p/n fixed . This result was generalised to arbitrary co-variances [ 22 , 23 ] using fundamental results from random matrix theory [ 24 ] . Non-asymptotic rates of convergence for a related problems were given in [ 25 ] . Previous results also existed in the statistical physics literature , e.g . [ 26–29 ] . Gaussian models for regression have seen a surge of popularity recently , and have been used in particular to study over-parametrization and the double-descent phenomenon , e.g . in [ 30–43 ] . 2 Setting . Consider a data set D = { xµ , yµ } nµ=1 with n independent samples from a probability measure ρ on X × Y , where X ⊂ Rd is the input and Y ⊂ R the response space . Let K be a kernel andH denote its associated reproducing kernel Hilbert space ( RKHS ) . Kernel ridge regression ( KRR ) corresponds to the following non-parametric minimisation problem : min f∈H 1 n n∑ µ=1 ( f ( xµ ) − yµ ) 2 + λ||f ||2H . ( 1 ) where || · ||H is the norm associated with the scalar product inH , and λ ≥ 0 is the regularisation . The convenience of KRR is that it admits a dual representation in terms of a standard parametric problem . Indeed , the kernel K can be diagonalized in an orthonormal basis { φk } ∞k=1 of L2 ( X ) : ∫ X ρx ( dx ′ ) K ( x , x′ ) φk ( x ′ ) = ηkφk ( x ) ( 2 ) where { ηk } ∞k=1 are the corresponding ( non-negative ) kernel eigenvalues and ρx is the marginal distribution over X . It is convenient to define the re-scaled basis of kernel features ψk ( x ) =√ ηkφk ( x ) and to work in matrix notation in feature space : define φ ( x ) ≡ { φk ( x ) } pk=1 ( with p possibly infinite ) ψ ( x ) = Σ 1 2φ ( x ) Ex∼ρx [ φ ( x ) φ ( x ) > ] = 1p , Ex′∼ρx [ K ( x , x′ ) φ ( x′ ) ] = Σφ ( x ) , ( 3 ) where Σ ≡ Ex∼ρx [ ψ ( x ) ψ ( x ) > ] = diag ( η1 , η2 , ... , ηp ) is the features co-variance ( a diagonal operator in feature space ) . With this notation , we can rewrite eq . ( 4 ) in feature space as a standard parametric problem for the following empirical risk : R̂n ( w ) = 1 n n∑ µ=1 ( w > ψ ( xµ ) − yµ ) 2 + λ w > w . ( 4 ) Our main results concern the typical averaged performance of the KRR estimator , as measured by the typical prediction ( out-of-sample ) error g = EDE ( x , y ) ∼ρ ( f̂ ( x ) − y ) 2 , ( 5 ) where the first average is over the data D = { xµ , yµ } and the second over a fresh sample ( x , y ) ∼ ρ . In what follows we assume the labels yµ ∈ Y were generated , up to an independent additive Gaussian noise with variance σ2 , by a target function f ? ( not necessarily belonging toH ) : yµ d = f ? ( xµ ) + σN ( 0 , 1 ) , ( 6 ) and we denote by θ ? the coefficients of the target function in the features basis f ? ( x ) = ψ ( x ) > θ ? . As we will characterize below , whether the target function f ? belongs or not toH depends on the relative decay coefficients θ ? with respect to the eigenvalues of the kernel . We often refer to θ ? as the teacher . While the present results and discussion are provided for additive gaussian noise for simplicity , our method are not restricted to this particular form of noise , and a more complete extension of the results for other noise settings is left for future work . We are then interested in the evolution of the excess error g − σ2 as the number of samples n is increased . Capacity and source coefficients — Motivated by the discussion above , we focus on ridge regression in an infinite dimensional ( p → ∞ ) space H with Gaussian design uµ def= ψ ( xµ ) d= N ( 0 , Σ ) with ( without loss of generality ) diagonal co-variance Σ = diag ( η1 , η2 , ... ) . We expect however the results of this manuscript to be universal for a large class of distributions beyond the Gaussian one . In particular , we anticipate that the gaussianity assumption should be amenable to being relaxed to sub-gaussians [ 44 ] or even any concentrated distribution [ 45 , 46 ] . Following the statistical learning terminology , we introduce two parameters α > 1 , r ≥ 0 , herefrom referred to as the capacity and source conditions [ 14 ] , to parametrize the difficulty of the target function and the learning capacity of the kernel tr Σ 1 α < ∞ , ||Σ 12−rθ ? ||H < ∞ . ( 7 ) As in [ 9 , 12 , 13 , 23 ] , we consider the particular case where both the spectrum of Σ and the teacher components θ ? k have exactly a power-law form satisfying some source/capacity conditions : ηk = k −α , θ ? k = k − 1+α ( 2r−1 ) 2 . ( 8 ) While this follows the standard naming convention , it is useful to translate these notations to the ones used in others works , and we give a dictionary in Appendix B , Table 2 . For instance , α is b and r is ( a − 1 ) /2b in [ 13 ] , while the change of variable α = αS/d and r = ( αT /αS − d ) /2 allows to recover the notation from [ 12 ] . The power law ansatz ( 8 ) is empirically observed to be a rather good approximation for some real simple datasets and kernels , see Fig . 7 in Appendix C. The parameters α , r introduced in ( 8 ) control the complexity of the data and of the teacher respectively . A large α can be seen as characterizing an effectively low dimensional ( and therefore easy to fit ) data distribution . By the same token , a large r signals a good alignment of the teacher with the important directions of the data co-variance , and therefore an a priori simple learning task . The regularization λ is allowed to vary with n according to a power-law λ = n− ` . This very general form allows us to encompass both the zero regularization case ( corresponding to ` =∞ ) and the case where λ = λ ? is optimized , with some optimal decay rate ` ? . Note also that this power law form implies that λ is assumed positive . While this is indeed the assumption of [ 10 , 14 ] with which we intend to make contact , [ 37 ] have shown that the optimal λ may in some settings may be negative . Some numerical experiments suggest that removing the positivity constraint on λ while optimizing does not affect the results presented in this manuscript . A more detailed investigation is left to future work .

Summary: This submission studies the asymptotic generalization error of kernel ridge regression (KRR) under Gaussian design with specific eigen-decay conditions. Major contributions include (i) specifying the decay rate of the generalization error based on the noise level and source / capacity conditions of the kernel / target function, (ii) identifying different regimes of learning, and (iii) deciding the optimal ridge regularization. Theoretical results are verified in various experiments.