Title: Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionality
Venue: NeurIPS 2024
Accepted: yes
Keywords: Kernel ridge regression

benign overfitting

tempered overfitting

Gaussian kernel
Abstract: We consider the overfitting behavior of minimum norm interpolating solutions of Gaussian kernel ridge regression (i.e. kernel ridgeless regression), when the bandwidth or input dimension varies with the sample size. For fixed dimensions, we show that even with varying or tuned bandwidth, the ridgeless solution is never consistent and, at least with large enough noise, always worse than the null predictor. For increasing dimension, we give a generic characterization of the overfitting behavior for any scaling of the dimension with sample size. We use this to provide the first example of benign overfitting using the Gaussian kernel with sub-polynomial scaling dimension.  All our results are under the Gaussian universality ansatz and the (non-rigorous) risk predictions in terms of the kernel eigenstructure.

Overfitting Behaviour of Gaussian Kernel Ridgeless Regression: Varying Bandwidth or Dimensionality

Abstract

We consider the overfitting behavior of minimum norm interpolating solutions of Gaussian kernel ridge regression (i.e. kernel ridgeless regression), when the bandwidth or input dimension varies with the sample size. For fixed dimensions, we show that even with varying or tuned bandwidth, the ridgeless solution is never consistent and, at least with large enough noise, always worse than the null predictor. For increasing dimension, we give a generic characterization of the overfitting behavior for any scaling of the dimension with sample size. We use this to provide the first example of benign overfitting using the Gaussian kernel with sub-polynomial scaling dimension. All our results are under the Gaussian universality ansatz and the (non-rigorous) risk predictions in terms of the kernel eigenstructure.

1 Introduction

A central question in learning theory is how learning algorithms can generalize well even when returning models that perfectly fit (i.e. interpolate) noisy training data. This phenomenon was observed empirically by Zhang et al. [63], and does not align with the traditional belief from statistical learning theory that overfitting to noise leads to poor generalization. Consequently, it attracted significant interest in recent years, and there has been much effort to understand the overfitting behavior of linear models, kernel methods, and neural networks.

In this paper, we study the overfitting behavior of Kernel Ridge Regression (KRR) with Gaussian kernel, namely, the behavior of the limiting test error when training on noisy data as the number of samples tends to infinity by insisting on interpolation (achieving zero training error). When the input dimension and bandwidth are fixed, the overfitting behavior is known to be 'catastrophic' [37], i.e. for any nonzero noise, the test risk tends to infinity as the sample size increases. However, this is not how Gaussian Kernel Ridge Regression is typically used in practice. In fixed dimension, the bandwidth is tuned, that is decreased, when the sample size increases [51, 19]. Additionally, it makes sense to study the behaviour when the input dimension increases with sample size (as in, e.g. linear models with proportional scaling [42]). This could be because when more data is available, more input features are used, even with a kernel; because as more resources are available we scale up both the input dimension and amount of data used; or to capture the fact that very large scale problems typically involve both more samples and higher input dimension. But unlike with linear models, where the dimension must scale linearly with the number of samples in order to allow for interpolation, when a kernel is used we can study the behaviour even when the input dimensionality increases much slower, and ask how slowly it could increase without catastrophic overfitting. Previous studies on kernel ridgeless regression considered polynomial increasing dimension (i.e. dimension ∝ sample-size a , for 0 < a ≤ 1 ) [4, 64, 24, 38, 40], but not subpolynomial scaling.

We aim to provide a more comprehensive picture of overfitting with Gaussian KRR by studying the overfitting behavior with varying bandwidth or with arbitrarily varying dimension, including

sub-polynomially. In particular, we show that for fixed dimension, even with varying bandwidth, the interpolation learning is never consistent and generally not better than the null predictor (either the test error tends to infinity or is finite but it is almost always not better than the null predictor). For increasing dimension, we give an upper and lower bound on the test risk for any scaling of the dimension with sample size, which indicates in many cases whether the overfitting is catastrophic (test error tends to infinity), tempered (test error tends to a constant), or benign (consistent). Our result agrees with the polynomial scaling of the dimension with sample size, showing tempered overfitting for an exponent that is a reciprocal of an integer and benign overfitting for any other exponent [4, 64]. Moreover, our result goes further, as we show the first example of sub-polynomially scaling dimension that achieves benign overfitting for the Gaussian kernel. Additionally, we show that a class of dot-product kernels on the sphere is inconsistent when the dimension scales logarithmically with sample size. All our results are under the Gaussian universality ansatz and the non-rigorous but well-established risk predictions in terms of the kernel eigenstructure [50, 66, 8, 57].

Related work

The test performance of overfitting models has been extensively studied for linear regression [27, 6, 3, 43, 45, 16, 32, 58, 53, 65, 30, 54, 13, 2, 49, 25], linear classification [12, 55, 10, 44, 42, 49, 36, 52, 56, 18], neural networks [20, 21, 11, 34, 60, 61, 39, 33, 23, 31, 29], and kernel methods. Below we focus on overfitting in kernel ridge regression.

Overfitting in fixed dimension with fixed kernel. In Mallinar et al. [37], the authors show that the minimum norm interpolating solution for Gaussian kernel with fixed bandwidth overfits catastrophically. In [15, 4, 64], the authors derive bounds on the test risk of minimum norm interpolant with a fixed kernel under various assumptions. Our result for fixed dimension will only apply to the Gaussian kernel, but it allows for any varying or adaptively chosen bandwidth.

Inconsistency of Kernel Ridge Regression. The case of varying bandwidth has been considered in Beaglehole et al. [5], Rakhlin and Zhai [47], Haas et al. [26]. In Beaglehole et al. [5], the authors show that there exists a specific data distribution for which the minimum norm interpolanting solution for a particular set of translation invariant kernels is not consistent. In Rakhlin and Zhai [47], the authors show that for input distributions on the unit ball, the Laplace kernel is inconsistent, even with varying bandwidth. In Haas et al. [26], the authors show that under different assumptions on the data distribution, for a general class of (potentially varying) kernels in fixed dimension, any differentiable function that overfits the data and is not much different from the minimum norm interpolant is inconsistent. All of these works only consider whether we can achieve consistency. None of these results apply in the case of data distributions that we are considering, and even if the predictor is not consistent, we ask how bad is it by comparing it to the null predictor and whether it might be tempered.

Overfitting in increasing dimensions. Many papers have studied the setup where the dimension increases with sample size [4, 64, 66, 38, 40, 28, 59], in particular when the dimension is a function of sample size (or vice versa), but they all consider only the case of a polynomial scaling of dimension and sample size. It was shown that in this case the minimum norm interpolating solution of dot product kernels on the sphere can be benign, depending on if the exponent is not an integer. We generalize these results to any scaling of the dimension and sample size. Our results recover the existing results in the case of polynomially scaling dimension and show benign overfitting in a certain sub-polynomial scaling. Having sub-polynomimal scaling of the dimension allows us to expand the set of possible target functions from only polynomials of a bounded degree, as in the case of polynomial scaling of dimension [4, 64, 66, 38, 40], to, in our case of sub-polynomially increasing dimension, polynomials of any degree and even non-polynomial functions.

2 Problem formulation and assumptions

Kernel ridge(less) regression and the Gaussian kernel. Let D be an unknown distribution over X × Y ⊆ R d × R and let { ( x i , y i ) } i m =1 ∼ D m be a dataset consisting of m samples. For simplicity, we will assume that the distribution of the target is given by a target function f ∗ of the input x ∈ X

with zero mean independent noise ξ with variance σ 2 , that is y ∼ f ∗ ( x ) + ξ . We note that our results can be extended to a distribution agnostic setting, as analyzed by Zhou et al. [66].

Let K : X × X → R be a positive semi-definite kernel function. Let ∥ f ∥ K be the norm of f in the RKHS H K corresponding to K . For a predictor f , let R ( f ) and ˆ R ( f ) be the test and training risk of f ,

Two important risks to consider are the risk of the null predictor, f ≡ 0 , which we will denote by R (0) = E D [( y ) 2 ] , and the Bayes (or irreducible) risk, which we will denote σ 2 or R ( f ∗ ) . Using this notation, the risk of the null predictor is R (0) = σ 2 + E D [( y ) 2 ] . Bayes risk represents the minimum possible risk that can be achieved by any predictor. For a regularization parameter δ , the regularized ridge solution ˆ f δ is given by

We are interested in the minimum norm interpolating (ridgeless) solution ˆ f 0 = lim δ → 0+ ˆ f δ , namely

We will focus on the Gaussian kernel, which is given by K ( x 1 , x 2 ) = exp ( -∥ x 1 -x 2 ∥ 2 2 τ 2 m ) , where m is the sample size and τ m is a predetermined bandwidth parameter that can vary with sample size. The Gaussian kernel is widely used and achieves good error rates for a variety of learning tasks [51, 19]. Gaussian KRR achieves optimal convergence to the best possible (Bayes) error for learning any function in a Besov space of high enough order (essentially bounded and twice differentiable in the weak sense) under very mild assumptions on the distribution of the input and target X × Y [19]. For ridge regression with the Gaussian kernel and under a standard data distribution assumption, the minimum distance between samples decreases with sample size so it makes sense to also decrease τ m . Additionally, decreasing τ m with sample size helps to achieve good convergence rates theoretically [19].

Main question. We will consider the problem of learning using the minimum norm interpolating solution ˆ f 0 of KRR. We want to understand the limiting behavior of test risk R ( ˆ f 0 ) as the sample size increases m → ∞ , that is lim m →∞ R ( ˆ f 0 ) . It suffices to understand lim sup m →∞ R ( ˆ f 0 ) and lim inf m →∞ R ( ˆ f 0 ) and this way we do not assume the existence of the limit. In this work, we use the taxonomy of benign, tempered, and catastrophic overfitting from Mallinar et al. [37], which indicates whether lim m →∞ R ( ˆ f 0 ) is the Bayes (optimal) error, a non-optimal but constant error, or infinity. Note that in this taxonomy, the null predictor can be classified as tempered. Therefore, we will compare the limiting risk to the risk of the null predictor R (0) in order to understand whether the performance of the interpolating solution is non-trivial.

Main tool: Eigenframework and a closed form of the test risk. Our main tool will be the closed form of the test risk predicted by the eigenframwork [50]. Under the eigenframework, we can write down the closed form of the test risk using Mercer's theorem decomposition of a kernel function K .

Given a positive semi-definite kernel function K : X × X → R , we can decompose it as

where λ k and ϕ k are the eigenvalues and eigenfunctions of the integral operator associated to K . The eigenfunctions { ϕ k } are an orthonormal basis of L 2 D X ( X ) , where D X is the marginal distribution of X . We denote the Bayes optimal target function by f ∗ , and expand it in the kernel basis { ϕ i } ∞ i =1 as f ∗ ( x ) = ∑ ∞ i =1 β i ϕ i ( x ) . To state the close form of the test risk we will introduce a few quantities. Let effective regularization , κ δ , be the solution to ∑ ∞ i =1 λ i λ i + κ δ + δ κ δ = m. Furthermore, let L i,δ = λ i λ i + κ δ

and E δ = m m -∑ ∞ i =1 L 2 i,δ . Then, the predicted risk , i.e. the predicted closed form of the test risk of ˆ f δ , is given by

where σ 2 is the Bayes error of D . Equation (2) was initially heuristically derived using the replica method or continuous approximations to the learning curves inspired by the Gaussian process literature [7]. In [50], it is dervied using a conservation law. Note that [66] shows that the predicted closed form of the test risk from [50] extends to general target distributions. There is strong evidence that the predicted risk is a good estimate of the true test risk, namely R ( ˆ f δ ) ≈ ˜ R ( ˆ f δ ) . Indeed, a number of works use the predicted risk closed form to estimate the test risk of KRR [58, 8, 66]. The following assumption on Gaussian design ansatz is used by all of these works.

Assumption 1 (Gaussian design ansatz, cf. Zhou et al. [66]) . When sampling ( x, · ) ∼ D , we have that the Gaussian universality holds for the eigenfunctions in the sense that the expected risk is unchanged if we replace ϕ with ˜ ϕ , where ˜ ϕ is Gaussian with appropriate parameters, i.e. ˜ ϕ ∼ N (0 , diag { λ i } ) .

This assumption appears to hold for real datasets as well, namely, the predictions computed for Gaussian design agree well with the experiments on kernel regression using real data [8, 50, 57]. As discussed in Zhou et al. [66], under this assumption, the equivalence R ( ˆ f δ ) ≈ ˜ R ( ˆ f δ ) holds in a few ways. First, in an appropriate asymptotic limit in which the sample size m and the number of eigenmodes in a given eigenvalue grow proportionally, the equivalence holds [27, 1]. Second, if the eigenstructure of the task is fixed, the error between the two can be bounded by a decaying function of m [14]. Finally, various numerical experiments show that the error between the two is small even for a small sample size m [8, 50]. Specifically, Canatar et al. [8] (see Figure 5 there) gives empirical evidence that the predicted risk closely approximates the true risk for the Gaussian kernel with data uniform on a sphere, which is the setting that we consider in some of our results.

There has been some recent progress in bounding the error between R ( ˆ f δ ) and ˜ R ( ˆ f δ ) unconditionally. Misiakiewicz and Saeed [41] shows that the error will tend to zero if the dimension d grows fast enough with the sample size. Additionally, they provide strong empirical evidence that the predicted risk is close to the test risk for a real dataset (MNIST) and Gaussian kernel, see Figure 1 in [41].

Formally, we will prove results about the predicted risk ˜ R ( ˆ f δ ) , but as previously presented evidence suggests, treating R ( ˆ f δ ) ≈ ˜ R ( ˆ f δ ) as equivalence is sufficient for understanding the behavior of KRR.

We note that using the eigenframework might introduce restrictions for which kernels some of our results apply, as concurrent work showed that the eigenframework prediction might not hold for the NTK in fixed dimension [4, 15]. Our two main results concern the Gaussian kernel, for which we described ample empirical evidence that the eigenlearning predictions hold [8, 41]. Understanding the limitations of the eigenframework is an important future research direction.

3 Fixed dimension: Gaussian kernel with varying bandwidth

We will assume that the source distribution is uniform on a d dimensional sphere, that the target function is square integrable, and that the target distribution is given by the target function with an independent noise.

Assumption 2 (Target function and data distribution) . Let D be the distribution over X × Y = S d -1 × R , such that the X marginal, denoted by D X , is Unif ( S d -1 ) . We will assume that for a target function f ∗ ∈ L 2 D X ( S d -1 ) , the marginal Y distribution is given by y ∼ f ∗ ( x ) + ξ , where ξ has mean zero and variance σ 2 > 0 . We write f ∗ = ∑ i β i ϕ i , where { ϕ i } is the L 2 D X ( S d -1 ) eigenbasis corresponding to the kernel K (Equation (1)). If we write β = ( β 1 , β 2 , . . . ) , then we have that ∥ β ∥ 2 2 = E D X ( ( f ∗ ( x )) 2 ) . We will use the notation ∥ f ∗ ∥ 2 = ∥ β ∥ 2 2 .

The assumption on the distribution on X is common in the literature on KRR [38, 4, 64, 22]. The assumption that x ∼ Unif ( S d -1 ) can be relaxed to a more general setting where x is uniformly distributed on other manifolds that are diffeomorphic to the sphere using the results of Li et al. [35], although that will not be the focus of this paper.

Note that here we vary the bandwidth τ m so both ˆ f 0 and R ( ˆ f 0 ) will depend on the bandwidth τ m as well as m . We will identify three different regimes of bandwidth scaling. We will show that the minimum norm interpolating solution exhibits either tempered or catastrophic overfitting, and we will argue that it is almost always worse than the risk of the null predictor.

Theorem 3 (Overfitting behavior of Gaussian kernel in fixed dimension) . Under Assumption 2, the following bounds hold for the predicted risk ˜ R ( ˆ f 0 ) of the minimum norm interpolating solution of Gaussian KRR:

If τ m = o ( m -1 d -1 ) , then ˜ R (0) ≤ lim inf m →∞ ˜ R ( ˆ f 0 ) ≤ lim sup m →∞ ˜ R ( ˆ f 0 ) < ∞ . More precisely, if τ m ≤ m -1 d -1 t ( m ) , where t ( m ) → 0 as m →∞ , then there is a scalar c d that depends only on the dimension and m 0 that depends on t ( m ) such that for all m>m 0 we have ˜ R ( ˆ f 0 ) > σ 2 +(1 -c d t ( m ) d -1 2 ) ∥ f ∗ ∥ 2 .

If τ m = ω ( m -1 d -1 ) , then lim m →∞ ˜ R ( ˆ f 0 ) = ∞ . Hence, for large enough m we have ˜ R ( ˆ f 0 ) > ˜ R (0) .

If τ m = Θ( m -1 d -1 ) , then lim sup m →∞ R ( ˆ f 0 ) < ∞ . Moreover, Suppose that C 1 m -1 d -1 ≤ τ m ≤ C 2 m -1 d -1 for some constants C 1 and C 2 , then there exist η, µ > 0 that depend only on d , C 1 , and C 2 , such that for all m we have ˜ R ( ˆ f 0 ) >µ ∥ f ∗ ∥ 2 +(1+ η ) σ 2 . Consequently, ˜ R ( ˆ f 0 ) > ˜ R (0) as long as σ 2 > 1 -µ η ∥ f ∗ ∥ 2 .

Theorem 3 shows that the minimum norm interpolating solution of Gaussian KRR cannot be consistent when data is distributed uniformly on the sphere, even with varying or adaptively chosen bandwidth. Additionally, in the first two modes of bandwidth change, the minimum norm interpolating solution is never better than the null predictor. In the third case, the interpolating solution is worse than null for noise that is not too small. This shows that even though the minimum norm interpolating predictor is classified as tempered in the first and third cases of scaling of the bandwidth, it is still worse than the trivial null predictor. Note that our analysis does not exclude the possibility that for τ m = Θ( m 1 d -1 ) there exists small enough σ 2 for which the interpolating solution is better than the null predictor. We leave this as an open question. In Appendix A, we provide further empirical justification for Theorem 3.

4 Increasing dimension

For the case of increasing dimension, we consider the problem of learning a sequence of distributions D ( d ) over X × Y = R d × R given by y ∼ f ∗ d ( x ) + ξ d using a sequence of kernels K ( d ) . Here, ξ d is independent noise with mean 0 and variance σ 2 > 0 . Formally, the kernel and the target function can change with the dimension d , but we will think of it as the same kernel and target with higher dimensional input. Furthermore, d will increase with sample size m , i.e. d = d ( m ) (or analogously m will increase with d ). A common assumption, which we also adopt, is that the projections of the target function f ∗ d onto the eigenfunctions ϕ ( d ) k of the kernels K ( d ) are uniformly bounded [4, 64].

Assumption 4 (Target function and distribution in increasing dimension) . Consider learning a sequence of target functions f ∗ d with a sequence of kernels K ( d ) . Let the target function f ∗ d have only S d nonzero coefficients (where S d can change with d ), so f ∗ d = ∑ S d i =1 β ( d ) i ϕ ( d ) i where ϕ ( d ) i are from Equation (1) and | β ( d ) i | ≤ B , i.e. ∥ β ∥ ∞ ≤ B , for B that is independent of d .

The functions that can be represented in this form depend on the number of nonzero coefficients, S d , and the kernel that we are using. In particular, for dot-product kernels on the sphere it includes all polynomials of degree k ≤ k d where k d is such that the multiplicities of first k d eigenfunctions are at most S d . If S d grows with d , then this set will include much more general functions. See Remark 14 for further discussion.

First, we will consider a general kernel K since Theorem 7 and Theorem 9 will apply more generally. Then, we will apply these results to the cases of dot-product and Gaussian kernels.

The multiplicities of eigenvalues will play an important role in bounding the test risk, both from above and below, so we will introduce the following related notation.

Definition 5 (Lower and upper index) . Let ˜ λ k be the k -th non-repeating eigenvalue of a kernel K and let N ( k ) be its multiplicity. Let N k = N (1) + · · · + N ( k ) . Let m be the sample size. Let k m be defined as the maximal k such that there is less than m eigenvalues with index k , i.e. k m = max { k ∈ N | N (1) + · · · + N ( k ) < m } . Define the lower index L m and the upper index U m as follows L m = N (1) + · · · + N ( k m ) and U m = N (1) + · · · + N ( k m +1) . When the dimension changes with sample size, we sometimes denote N ( k ) by N ( d, k ) .

We will first state a generic bound on the test risk for any data distribution, kernel with a bounded sum of eigenvalues, sample size, and dimension. This bound will be informative when we scale the dimension d with sample size m , but it holds for any kernel that satisfies the following assumption.

Assumption 6 (Bounded sum of eigenvalues) . Assume that the kernel K has a bounded sum of eigenvalues, i.e. there is a constant A such that ∑ ∞ i =1 N ( i ) ˜ λ i ≤ A . For a sequence of kernels K ( d ) , assume that all such A ( d ) are bounded by some constant A . 1

This is a reasonable assumption for most dot-product kernels, as we show in Appendix C.4. It also implicitly sets the scale of the kernel.

Theorem 7 (Test risk upper bound for kernel ridgeless regression) . Let d and m be any dimension and sample size. Define L m , U m , k m , N ( i ) , N l , and ˜ λ k as in Definition 5. Consider KRR with a kernel K satisfying Assumption 6 for some A . Assume that for some integer l , the target function f ∗ satisfies Assumption 4 with at most N l nonzero coefficients. Then, the predicted risk of the minimum norm interpolating solution is bounded by the following:

Alternatively, we can bound the risk using ( ∑ l i =1 N ( i ) 1 ˜ λ i ) instead of ( ∑ l i =1 N ( i ) 1 ˜ λ 2 i ) (see Theorem 20 in the appendix).

We also establish a generic inconsistency result for any data distribution, kernel with a bounded sum of eigenvalues, sample size, and dimension based on the upper and lower indices from Definition 5. We will further need to assume that the eigenvalues are bounded away from zero. Similarly, this will be useful when scaling dimension d with sample size, but it holds generally.

Assumption 8 (Lower bound on eigenvalues) . Assumme that the kernel K has eigenvalues that are not too small, i.e. there is a constant b such that max i ≤ k m ( 1 ˜ λ i ) < m -L m b . For a sequence of kernels K ( d ) , assume that for the corresponding m = m ( d ) (since d = d ( m ) , we can also "invert" the dependence) all such b ( d ) are bounded below by some b .

This assumption will hold for most dot-product kernels and we will show it for Gaussian kernel in Appendix C.4.

Theorem 9 (Test risk lower bound for any kernel ridgeless regression) . Let k m and L m be as in Definition 5. Consider learning a target function f ∗ , with some sample size m . Let K be a kernel satisfying Assumption 6 and Assumption 8 for some A and b . Consider the minimum norm interpolating solution of KRR (with any data distribution) with kernel K . Then, for the predicted risk of minimum norm interpolating solution, the following lower bound holds:

To apply Theorem 7 for varying dimension d , we would additionally require that A is uniformly bounded for all d and kernel K ( d ) and also that l = l ( d ) changes with d such that Assumption 4 holds with S d = N l ( d ) . For dot-product kernels K ( d ) on the sphere, if we let K ( d ) ( x, y ) = h ( d ) ( ∥ x -y ∥ ) ,

we will have A = sup d h ( d ) (0) , so if h ( d ) does not change with d we can take A = h (0) (see Appendix C.4 for more details). Specifically, this holds for the Gaussian kernel on the sphere with A = 1 .

To apply Theorem 9 to the case of increasing dimension, we would require that the bounds ∑ i N ( d, i ) ˜ λ i ≤ A and max i ≤ k m ( 1 ˜ λ i ) < m -L m b hold for all d and kernels K ( d ) . Usually, the condition max i ≤ k m ( 1 ˜ λ i ) < m -L m b will be satisfied for b = 1 . We will show it for the two cases of sub-polynomially scaling dimensions with a Gaussian kernel. For the polynomial scaling dimension, it is reasonable to assume it for general dot-product kernels, as discussed in Appendix C.4.

Now, we will show that using Theorem 7 and Theorem 9 we can recover the behavior of the minimum norm interpolating solution for polynomially increasing dimension [4, 64], i.e. tempered overfitting for integer exponent and benign for non-integer exponent. Here, we will need to additionally assume that the eigenvalue decay is not too fast.

Assumption 10 (Eigenvalue decay) . The eigenvalues do not decrease too quickly, i.e. for k m as in Definition 5, we have that there is a constant c such that max i ≤ k m ( 1 ˜ λ i ) ≤ cN ( k m ) . For increasing dimension, we require that max i ≤ k m ( 1 ˜ λ i ) ≤ cN ( d, k m ) for all m (i.e. all d , as d and m both increase).

This assumption is stronger than Assumption 8, but as we show in Appendix C.4, it is reasonable for dot-product kernels on the sphere and even the NTK.

Corollary 11 (Dot-product kernels with polynomially increasing dimension, recovering the results of [24, 38, 40, 4, 64]) . Consider the problem of learning a sequence of target functions f ∗ d satisfying Assumption 4 with S d ≤ Θ( d ⌊ α ⌋ ) with a dot-product kernel K ( x, y ) = h ( ∥ x -y ∥ ) with h (0) = 1 on the sphere S d -1 (where h does not depend on d , i.e. A = 1 from Assumption 6)) that further satisfies Assumption 10. Let d α m = Θ(1) for α ∈ (0 , ∞ ) . Then the overfitting behavior of the minimum norm interpolating solution is benign if α is not an integer and tempered if α is an integer.

Additionally, we will show that for d = log m , we cannot get benign overfitting, i.e. consistency with a class of dot-product kernels on the sphere. Similarly, as in the previous corollary, this will hold for any sequence where d = log m even only asymptotically.

Corollary 12 (Inconsistency with dot-product kernels in logarithmically scaling dimension) . Let K ( d ) be a sequence of dot-product kernels on S d -1 that satisfy Assumption 8. Let the dimension d grows with sample size as d = log 2 m (i.e. m = 2 d ). Then, the minimum norm interpolant cannot exhibit benign overfitting for any such sequence K ( d ) , i.e. there exists an absolute constant η > 0 such that for all d, m, ˜ R ( ˆ f 0 ) > (1 + η ) σ 2 .

On the other hand, using Theorem 7, we will establish the first case of sub-polynomial scaling dimension with benign overfitting using the Gaussian kernel and data on the sphere. We will use d = exp( √ log m ) .

Corollary 13 (Benign overfitting with Gaussian kernel and sub-polynomial dimension) . Let K be the Gaussian kernel on the sphere S d -1 with a fixed bandwidth, and take a sequence of dimensions d and sample sizes m that scale as d = exp ( √ log m ) (in particular, we take l ∈ N such that d = 2 2 l and m = 2 2 2 l with l = 1 , 2 , 3 . . . ). Consider learning a sequence of target functions f ∗ d as in Assumption 4 with S d ≤ m 1 4 . Then, we have that the minimum norm interpolating solution achieves the Bayes error in the limit ( m,d ) →∞ . In particular, for d ≥ 4 and m ≥ 16 we have

Remark 14 (Allowed target functions) . The set of allowed target functions f ∗ d in Corollary 13, i.e. with sub-polynomial scaling dimension, is strictly larger than the set of allowed target functions for polynomially scaling dimension, as in Corollary 11 and [38, 4, 64]. In particular, for polynomially scaling dimension d α m = Θ(1) , the result holds only if the target function is a polynomial of degree at most ⌊ α ⌋ . On the other hand, Corollary 13 shows that sub-polynomially scaling dimension allows for the target function to be a polynomial of arbitrary degree, as well as non-polynomial functions. In particular, in dimension d , we can represent polynomials of degree up to Θ(log 2 d ) .

5 Proofs outline

In this section, we discuss the main proof ideas. All formal proofs are provided in the appendix.

By Equation (2), to understand how the test risk of the minimum norm interpolating solution behaves, it suffices to understand how the eigenvalues corresponding to the kernel K and thus the quantities E 0 , L i, 0 behave. In Zhou et al. [66], the authors show that E 0 is bounded both above and below in terms of the effective rank of the systems of eigenvalue { λ } ∞ i =1 , defined by r k := ∑ ∞ i = k +1 λ i λ k +1 . We will use this, along with directly bounding L i, 0 .

If K is a dot-product kernel on the sphere (such as the Gaussian kernel), we can take the eigenfunctions ϕ i to be the spherical harmonics Y ks , where k ≥ 0 and s ∈ [1 , N ( d, k )] . Here N ( d, k ) = (2 k + d -2)( k + d -3)! k !( d -2)! is the multiplicity of the k -th spherical harmonic. All Y ks for the same k will have the same eigenvalue, which we will denote ˜ λ k . In this case, we can write a closed-form expression for the eigenvalues of Gaussian kernel, ˜ λ k , in terms of the bandwidth τ m and modified Bessel functions of the first kind I v ( x ) [46] (see Appendix D). Using the closed form of ˜ λ i and the multiplicities of eigenvalues, we can understand how the test risk of the minimum norm interpolating solution R ( ˆ f 0 ) behaves as m →∞ , which tells us its overfitting behavior.

Additionally, E 0 appearing in Equation (2), will be informative. For example, if lim m →∞ E 0 > 1 then the overfitting cannot be benign. The following bound using the effective rank r k holds [66]: For k < m such that r k + k > m we have

Proof sketch of Theorem 3. We will focus on the lower bounds in this case, as the result is negative. The key elements to understanding the effective rank r k and the test risk Equation (2) is to understand how the ratios of eigenvalues ˜ λ k +1 ˜ λ k and the multiplicities N ( d, k ) behave. Using the closed form of the eigenvalues of Gaussian kernel and the properties of modified Bessel functions [46, 48], with some computation (see Theorem 28 in the appendix for the computations), we can derive the following bounds on ratios of eigenvalues

From these bounds, we can derive tight bounds on ˜ λ k + j ˜ λ k for indices k and j using simple but long calculations (see Theorem 28 in the appendix). If k = o ( 1 τ m ) and j = o ( 1 τ m ) , then ˜ λ k + j ˜ λ k ≈ 1 . If k ≤ Θ ( 1 τ m ) and j = Θ ( 1 τ m ) , then ˜ λ k + j ˜ λ k = Θ(1) . If k = ω ( 1 τ m ) , then ˜ λ k + j ˜ λ k = o ( 1 j n ) for any integer n ∈ N (i.e. it deceases super-polynomialy). For N ( d, l ) it holds that N ( d, l ) = Θ( l d -2 ) and N l := N ( d, 1) + · · · + N ( d, l ) = Θ( l d -1 ) . Therefore if ˜ l is the index such that ˜ λ ˜ l = λ l , we have that ˜ l = Θ( l 1 d -1 ) .

For τ m ≥ ω ( m -1 d -1 ) , we will take l = (1 + 1 √ m ) m and show that r l = o ( m ) . Then, the bound in Equation (6) will imply that E 0 ≥ 1 + √ m , so from Equation (2), ˜ R ( ˆ f 0 ) > E 0 σ 2 = √ mσ 2 . Note that for l = (1 + 1 √ m ) m , we have that ˜ l = Θ( m 1 d -1 ) = ω ( 1 τ m ) . Note that for r l -1 , we have that

For k ≥ m it holds that

since N ( d, ˜ l + i ) < N ( d, j ) i d -2 and ˜ λ ˜ l + i ˜ λ ˜ l < 1 i d . So indeed r l -1 = o ( m ) which implies r l = o ( m ) .

For τ m = o ( m -1 d -1 ) and τ m = Θ( m -1 d -1 ) , we will directly analyze Equation (2). For k = Θ( 1 τ m ) , we have that ˜ λ i ˜ λ 1 > 1 2 for all i ≤ k . Let L i = ˜ λ i ˜ λ i + κ 0 . Note that L i > 1 2 L 1 for i ≤ k . Note that

So, we have that for all i that L i < L 1 < m Θ ( ( 1 τm ) d -1 ) . From Equation (2), we have that

For τ m = o ( m 1 d -1 ) this is sufficient. Additionally, by a similar computation as above, in this case, r 1 = ω ( m ) , so E 0 is bounded by Equation (5). For τ m = Θ( m 1 d -1 ) , using Equation (5) and Equation (6), by showing that for l = Θ( m ) , r l = Θ( m ) , we have Θ(1) > E 0 > 1 + Θ(1) .

Proof sketch of Theorem 7. Let L i = ˜ λ i ˜ λ i + κ 0 . Note that (1 -L i ) 2 = κ 2 0 ( κ 0 +˜ λ i ) 2 . Then, Equation (2) can be rewritten and bounded as (with an abuse of notation for β i )

Note that κ 2 0 ( κ 0 +˜ λ i ) 2 < κ 2 0 ˜ λ 2 i and also L i ≤ ˜ λ i κ 0 . Therefore, we have that ˜ R ( ˆ f 0 ) ≤ E 0 B 2 κ 2 0 ( ∑ l i =1 N ( d, i ) 1 ˜ λ 2 i ) + E 0 σ 2 . Note that m = ∑ i N ( d, i ) L i < ∑ i N ( d, i ) ˜ λ i κ 0 < A κ 0 , so κ 0 < A m . Finally, to bound E 0 , note that in Equation (5) we can choose k = L m < m , then r k + k > U m > m , so E 0 ≤ ( 1 -L m m ) -1 ( 1 -m U m ) -1 . Combining these with κ 0 < A m gives the claim of Theorem 7. The proof of the alternative bound is harder and will be delayed to the appendix.

Proof sketch of Theorem 9. By Theorem 3 from Zhou et al. [66], if k is the first k < m such that k + br k ≥ m , then E 0 ≥ ( 1 -( b b +1 ) 2 k m ) -1 . Since max i ≤ k m ( 1 ˜ λ i ) < m -L m b , for l < N ( d, 1)+ · · · + N ( d, k m ) , we have that br l + l ≤ N ( d, 1)+ · · · + N ( d, k m ) -1+ b max i ≤ k m ( 1 ˜ λ i ) < L m + m -L m ≤ m , so the first l for which r l + l > m is l = L m = N ( d, 1) + · · · + N ( d, k m ) . Plugging in k = L m we get that E 0 ≥ ( 1 -( b b +1 ) L m m ) -1 , so from Equation (6), we have ˜ R ( ˆ f 0 ) ≥ ( 1 -( b b +1 ) 2 L m m ) -1 σ 2 .

Proof sketch of Corollary 11. Note that an analogous proof holds for any d α m → c , for a constant c , but we take equality for simplicity. If k is a constant, i.e. it does not change with the dimension, we have that N ( d, k ) = Θ( d k ) . Therefore, k m = ⌊ α ⌋ if α is a non-integer and α -1 if α is an integer. If α is a non-integer, then L m = N ( d, k m ) + · · · + N ( d, 1) = Θ( d ⌊ α ⌋ ) and U m = N ( d, k m +1) + · · · + N ( d, 1) = Θ( d ⌊ α ⌋ +1 ) . So we have that L m m = d ⌊ α ⌋-α , m L m = d α -⌊ α ⌋-1 , N ( d, ⌊ α ⌋ ) = d ⌊ α ⌋ . Note that k m = ⌊ α ⌋ . We have from Assumption 10 that max i ≤ k m 1 ˜ λ i = O ( N ( d, k m )) . Note that Theorem 7 holds with ( ∑ l i =1 N ( d, i ) 1 ˜ λ i ) instead of ( ∑ l i =1 N ( d, i ) 1 ˜ λ 2 i ) . Therefore, we have

Therefore, lim m →∞ ˜ R ( ˆ f 0 ) = σ 2 . So, if α is not an integer, we get benign overfitting. If α is an integer, note that N ( d, 1) + · · · + N ( d, α -1) = o ( d α ) , so k m = α . Then there are c u , c l ∈ (0 , 1) such that c l < L m m ≤ c u < 1 . So, Theorem 9 holds for b = 1 2 , so ˜ R ( ˆ f 0 ) > 1 1 -c 1 9 σ 2 . This shows that lim inf m →∞ ˜ R ( ˆ f 0 ) ≥ 1 1 -c 1 9 σ 2 > σ 2 . The upper bound follows as above. Since we assumed that max i ≤ k m 1 ˜ λ i = O ( N ( d, k m )) , and max i ≤ k m N ( d, i ) = N ( d, k m ) = Θ( d α ) , we have that ∑ k m i =1 N ( d, i ) 1 ˜ λ i = Θ( d 2 α ) , so

So, we conclude that for integer α we have tempered overfitting and for non-integer α we have benign overfitting. This recovers results of Ghorbani et al. [24], Mei et al. [38], Misiakiewicz [40], Barzilai and Shamir [4], Zhang et al. [64].

Proof sketch of Corollary 12. Note that this holds for any scaling of d and m where d = Θ(log m ) , but we take this particular one for concreteness. As we show in the appendix (Theorem 21), it is not hard to see that L m ≈ α l m for some constant α l < 1 and L m ≈ α u m for some constant α u > 1 . Theorem 9 implies that we cannot get benign overfitting in this case, i.e. for all m , ˜ R ( ˆ f 0 ) > 1 1 -( b b +1 ) 2 α l σ 2 . This shows that lim inf m →∞ ˜ R ( ˆ f 0 ) ≥ 1 1 -( b b +1 ) 2 α l σ 2 > σ 2 .

Proof sketch of Corollary 13. We have that for the Gaussian kernel on the sphere, Theorem 7 holds with A = 1 (see Appendix C.4 for further details). First, we will compute k m and hence L m and U m . After some tedious calculation (Theorem 23 in the appendix), we see that for k ≤ 2 l + l -1 we have N ( d, 1) + · · · + N ( d, k ) = o ( m ) , but for k = 2 l + l we have N ( d, 1) + · · · + N ( d, k ) > m . This shows that k m = 2 l + l -1 and so again after long calculations L m = Θ( m log m ) and U m > Θ( md 0 . 89 ) . To estimate ∑ k i =1 N ( d, i ) ˜ λ i , note that eigenvalues are decreasing and iN ( d, i ) = iN ( d, i +1) 2 i + d -2 2 i + d i +1 i + d -2 = o ( N ( d, i +1)) (take k ≪ d ), so it suffices to estimate N ( d, k ) 1 ˜ λ k . Now, from Equation (7) and an estimate on the size of the first eigenvalue (Corollary 31) ˜ λ 1 = Θ( 1 d α ) , for fixed α > 0 , we can estimate the size ˜ λ k . We have that

for arbitrarily small ε . For k = 7 k m 24 , from additional calculation (Theorem 23 in the appendix), we have that m 1 4 ≤ N ( d, k ) ≤ m 1 3 and d k < m 7 24 . So ∑ k m i =1 N ( d, i ) 1 ˜ λ 2 i < m . Plugging these back into Theorem 7 gives the desired result.

6 Summary

In this paper, we considered the minimum norm interpolating solution of kernel ridge regression in fixed dimension with Gaussian kernel and varying or adaptively chosen bandwidth, and in increasing dimension with various kernels. In fixed dimension, we showed that if the source distribution is uniform on the sphere, then the minimum norm interpolating solution is inconsistent for any choice of bandwidth, and usually worse than the null predictor, except possibly in one particular scaling of bandwidth and with small noise. Furthermore, we showed a general upper and lower bound on the test risk, which we applied in the case of increasing dimension to recover the currently known results about polynomially increasing dimension and show two new results: We showed that no dot-product kernel on the sphere can achieve consistency for logarithmic scaling of the dimension, and obtained the first case of sub-polynomially scaling dimension where the minimum norm interpolating solution exhibits benign overfitting, namely with Gaussian kernel and dimension scaling as d = exp( √ log m ) .

Acknowledgments and Disclosure of Funding

We would like to thank Theodor Misiakiewicz and Sam Buchanan for useful discussions.

This work was done as part of the NSF-Simons funded collaboration on the Theoretical Foundations of Deep Learning (https://deepfoundations.ai), and primarily while GV was at TTIC. GV is supported by research grants from the Center for New Scientists at the Weizmann Institute of Science, and the Shimon and Golde Picker - Weizmann Annual Grant.

References

Francis Bach. High-dimensional analysis of double descent for linear regression with random projections. arXiv preprint arXiv:2303.01372 , 2023.

Peter L. Bartlett and Philip M. Long. Failures of model-dependent generalization bounds for least-norm interpolation. The Journal of Machine Learning Research , 22(1):9297-9311, 2021.

Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Daniel Barzilai and Ohad Shamir. Generalization in kernel regression under realistic assumptions. arXiv preprint arXiv:2312.15995 , 2023.

Daniel Beaglehole, Mikhail Belkin, and Parthe Pandit. On the inconsistency of kernel ridgeless regression in fixed dimensions. arXiv preprint arXiv:2205.13525 , 2023.

Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science , 2(4):1167-1180, 2020.

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learn- ing curves in kernel regression and wide neural networks. International Conference on Machine Learning, PMLR , pages 1024-1034, 2020.

Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Communications , 12(1): 1-12, 2021.

Yuan Cao, Zhiying Fang, Yue Wu, Ding-Xuan Zhou, and Quanquan Gu. Towards understanding the spectral bias of deep learning. arXiv preprint arXiv:1912.01198 , 2019.

Yuan Cao, Quanquan Gu, and Mikhail Belkin. Risk bounds for over-parameterized maximum margin classification on sub-gaussian mixtures. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Yuan Cao, Zixiang Chen, Mikhail Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. arXiv preprint arXiv:2202.06526 , 2022.

Niladri S. Chatterji and Philip M. Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. Journal of Machine Learning Research , 22(129):1-30, 2021.

Niladri S. Chatterji, Philip M. Long, and Peter L Bartlett. The interplay between implicit bias and benign overfitting in two-layer linear networks. arXiv preprint arXiv:2108.11489 , 2021.

Chen Cheng and Andrea Montanari. Dimension free ridge regression. arXiv preprint arXiv:2210.08571 , 2022.

Tin Sum Cheng, Aurelien Lucchi, Anastasis Kratsios, and David Belius. Characterizing overfitting in kernel ridgeless regression through the eigenspectrum. arXiv preprint arXiv:2402.01297 , 2024.

Geoffrey Chinot and Matthieu Lerasle. On the robustness of the minimum ℓ 2 interpolator. arXiv preprint arXiv:2003.05838 , 2020.

W. J. Conover. Practical Nonparametric Statistics . Wiley Series in Probability and Statistics, 1999.

Konstantin Donhauser, Nicolo Ruggeri, Stefan Stojanovic, and Fanny Yang. Fast rates for noisy interpolation require rethinking the effect of inductive bias. In International Conference on Machine Learning (ICML) , 2022.

Mona Eberts and Ingo Steinwart. Optimal regression rates for svms using gaussian kernels. Electronic Journal of Statistics , page Vol. 7 1-42, 2013.

Spencer Frei, Niladri S. Chatterji, and Peter L. Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. In Conference on Learning Theory (COLT) , 2022.

Spencer Frei, Gal Vardi, Peter L Bartlett, and Nathan Srebro. Benign overfitting in linear classifiers and leaky relu networks from kkt conditions for margin maximization. arXiv preprint arXiv:2303.01462 , 2023.

Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Ronen Basri. On the similarity between the laplace and neural tangent kernels. arXiv preprint arXiv:2007.01580 , 2020.

Erin George, Michael Murray, William Swartworth, and Deanna Needell. Training shallow relu networks on noisy data using hinge loss: when do we overfit and is it benign? Advances in Neural Information Processing Systems , 36, 2024.

Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. The Annals of Statistics , 49(2):1029-1054, 2021.

Nikhil Ghosh and Mikhail Belkin. A universal trade-off between the model size, test loss, and training loss of linear predictors. arXiv preprint arXiv:2207.11621 , 2022.

Moritz Haas, David Holzmüller, Ulrike von Luxburg, and Ingo Steinwart. Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. 37th Conference on Neural Information Processing Systems (NeurIPS 2023) , 2023.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. arXiv peprint arXiv:1903.08560 , 2020.

Hong Hu, Yue M. Lu, and Theodor Misiakiewicz. Asymptotics of random feature regression beyond the linear scaling regime. arXiv preprint arXiv:2403.08160v1 , 2023.

Nirmit Joshi, Gal Vardi, and Nathan Srebro. Noisy interpolation learning with shallow univariate relu networks. arXiv preprint arXiv:2307.15396 , 2023.

Peizhong Ju, Xiaojun Lin, and Jia Liu. Overfitting can be harmless for basis pursuit, but only to a degree. Advances in Neural Information Processing Systems , 33:7956-7967, 2020.

Kedar Karhadkar, Erin George, Michael Murray, Guido Montúfar, and Deanna Needell. Benign overfitting in leaky relu networks with moderate input dimension. arXiv preprint arXiv:2403.06903 , 2024.

Frederic Koehler, Lijia Zhou, Danica J Sutherland, and Nathan Srebro. Uniform convergence of interpolators: Gaussian width, norm bounds, and benign overfitting. arXiv preprint arXiv:2106.09276 , 2021.

Guy Kornowski, Gilad Yehudai, and Ohad Shamir. From tempered to benign overfitting in relu neural networks. Advances in Neural Information Processing Systems , 36, 2024.

Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting for two-layer relu networks. arXiv preprint arXiv:2303.04145 , 2023.

Yicheng Li, Zixiong Yu, Guhan Chen, and Qian Lin. Statistical optimality of deep wide neural networks. arXiv preprint arXiv:2305.02657 , 2023.

Tengyuan Liang and Benjamin Recht. Interpolating classifiers make few mistakes. arXiv preprint arXiv:2101.11815 , 2021.

Neil Mallinar, James B. Simon, Amirhesam Abedsoltan, Parthe Pandit, Mikhail Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: A taxonomy of overfitting. arXiv preprint arXiv:2207.06569v2 , 2022.

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Generalization error of random feature and kernel methods: hypercontractivity and kernel matrix concentration. Applied and Computational Harmonic Analysis , 59:3-84, 2022.

Xuran Meng, Difan Zou, and Yuan Cao. Benign overfitting in two-layer relu convolutional neural networks for xor data. arXiv preprint arXiv:2310.01975 , 2023.

Theodor Misiakiewicz. Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression. arXiv:2204.10425 , 2022.

Theodor Misiakiewicz and Basil Saeed. A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator. arXiv preprint arXiv:2403.08938 , 2024.

Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime. Preprint arXiv:1911.01544v3 , 2023.

Vidya Muthukumar, Kailas Vodrahalli, Vignesh Subramanian, and Anant Sahai. Harmless interpolation of noisy data in regression. IEEE Journal on Selected Areas in Information Theory , 2020.

Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai. Classification vs regression in overparameterized regimes: Does the loss function matter? Journal of Machine Learning Research , 22(222):1-69, 2021.

Jeffrey Negrea, Gintare Karolina Dziugaite, and Daniel Roy. In defense of uniform convergence: Generalization via derandomization with an application to interpolating predictors. In International Conference on Machine Learning , pages 7263-7272, 2020.

Minh Ha Quang and Yuan Yao. Mercer's theorem, feature maps, and smoothing. Conference: Learning Theory, 19th Annual Conference on Learning Theory, COLT , 2006.

Alexander Rakhlin and Xiyu Zhai. Consistency of interpolation with laplace kernels is a high-dimensional phenomenon. arXiv preprint arXiv:1812.11167 , 2018.

Javier Segura. Simple bounds with best possible accuracy for ratios of modified bessel functions. arXiv preprint arXiv:2207.02713v3 , 2023.

Ohad Shamir. The implicit bias of benign overfitting. In Conference on Learning Theory , pages 448-478. PMLR, 2022.

James B. Simon, Madeline Dickens, Dhruva Karkada, and Michael R. DeWeese. The eigenlearning framework: A conservation law perspective on kernel regression and wide neural networks. arXiv preprint arXiv:2110.03922 , 2021.

Ingo Steinwart and Andreas Christmann. Support vector machines . Springer Science & Business Media, 2008.

Christos Thrampoulidis, Samet Oymak, and Mahdi Soltanolkotabi. Theoretical insights into multiclass classification: A high-dimensional asymptotic view. Advances in Neural Information Processing Systems , 33:8907-8920, 2020.

A. Tsigler and P. L. Bartlett. Benign overfitting in ridge regression. arXiv peprint arXiv:2009.14286 , 2020.

Guillaume Wang, Konstantin Donhauser, and Fanny Yang. Tight bounds for minimum l1-norm interpolation of noisy data. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2022.

Ke Wang and Christos Thrampoulidis. Binary classification of gaussian mixtures: Abundance of support vectors, benign overfitting and regularization. arXiv peprint arXiv:2011.09148 , 2021.

Ke Wang, Vidya Muthukumar, and Christos Thrampoulidis. Benign overfitting in multiclass classification: All roads lead to interpolation. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how realworld neural representations generalize. In International Conference on Machine Learning , Proceedings of Machine Learning Research, 2022.

Denny Wu and Ji Xu. On the optimal weighted ℓ 2 regularization in overparameterized linear regression. Advances in Neural Information Processing Systems , 33:10112-10123, 2020.

Lechao Xiao, Hong Hu, Theodor Misiakiewicz, Yue M. Lu, and Jeffrey Pennington. Precise learning curves and higher-order scaling limits for dot product kernel regression. arXiv preprint: arXiv:2205.14846v3 , 2023.

Xingyu Xu and Yuantao Gu. Benign overfitting of non-smooth neural networks beyond lazy training. In International Conference on Artificial Intelligence and Statistics , pages 11094-11117. PMLR, 2023.

Zhiwei Xu, Yutong Wang, Spencer Frei, Gal Vardi, and Wei Hu. Benign overfitting and grokking in relu networks for xor cluster data. arXiv preprint arXiv:2310.02541 , 2023.

Zhen-Hang Yang and Yu-Ming Chu. On approximating the modified bessel function of the first kind and toader-qi mean. Journal of Inequalities and Applications , 2016.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations , 2017.

Haobo Zhang, Weihao Lu, and Qian Lin. The phase diagram of kernel interpolation in large dimensions. arXiv preprint arXiv:2404.12597v1 , 2024.

Lijia Zhou, Frederic Koehler, Pragya Sur, Danica J Sutherland, and Nathan Srebro. A non-asymptotic moreau envelope theory for high-dimensional generalized linear models. arXiv preprint arXiv:2210.12082 , 2022.

Lijia Zhou, James B Simon, Gal Vardi, and Nathan Srebro. An agnostic view on the cost of overfitting in (kernel) ridge regression. arXiv preprint arXiv:2306.13185 , 2023.

A Empirical Justification

100

80

60

mean test error

noise level

null predictor error

40

20

Q

0

2000

4000

6000

8000

10000

m

(a) τ m = o ( m -1 d -1 ) with σ 2 = 1 and d = 6 . We see that the test error tends to something that is equal to or larger than the test risk of the null predictor.

mean test error

20000

noise level

null predictor error

15000

10000

5000

200

400

600

800

1000

m

mean test error

18000

noise level

null predictor error

16000

14000

12000

10000

8000

0

2000

4000

6000

8000

10000

m

(c) τ m = Θ( m -1 d -1 ) with σ 2 = 10000 and d = 6 . We see that the test error is well above the risk of the null predictor.

Figure 1: We plot the dependence of the test error of the Gaussian kernel ridgeless predictor for y = f ∗ ( x ) + ξ , with ξ ∼ N (0 , σ 2 ) , f ∗ = 10 , x ∼ Unif ( S d -1 ) , and σ 2 and d as above, and sample size m . Light blue lines represent the test error for 100 different runs of the experiment and dark blue is the mean of all those runs. The red dotted line is the noise level σ 2 and the yellow dotted line is the error the null predictor achieves in this setting. We see that the test error concentrates around its mean and that the behavior is as predicted by Theorem 3.

In this section, we provide empirical justification for the predictions of Theorem 3, which deals with the case of fixed dimension. Figure 1 shows the dependence of the test error on the sample size for the Gaussian Kernel Ridgeless prediction with varying bandwidth, as in the setup of Theorem 3. Specifically, we consider y = f ∗ ( x ) + ξ where ξ ∼ N (0 , σ 2 ) , f ∗ = 10 , σ 2 is the noise level, and

x ∼ Unif ( S d -1 ) . We vary the values of d and σ 2 and the bandwidth scaling τ m as follows: for τ m = o ( m -1 d -1 ) we take σ 2 = 1 and d = 6 , for τ m = ω ( m -1 d -1 ) we take σ 2 = 10 and d = 4 , and for τ m = Θ( m -1 d -1 ) we take σ 2 = 10000 and d = 6 . We compare the test error (light blue) and the mean of the test error for 100 different runs of the experiment (dark blue) with the noise level (the Bayes risk; red) and the risk of the null predictor (yellow). We see that for all three cases, our predictions agree with the experiments, that is, in the case τ m = o ( m -1 d -1 ) the test error is eventually above the null predictor error but finite, in the case τ m = ω ( m -1 d -1 ) the test error increases with sample size, and in the case τ m = Θ( m -1 d -1 ) the test error is above null predictor error. The code to reproduce these experiments can be found at https://github.com/marko-medvedev/overfitting-behavior-of-gaussian-kernel-ridgeless-regression .

We ran the experiments on one A6000 GPU.

Statistical significance of experimental results: In Figure 2 and Figure 3, we present evidence for the statistical significance of our experimental results. We examine whether a significant mass of the distribution of the mean of the test loss, L , exhibits the same behavior as L . We use 50 of the 100 experiments to estimate an interval [ L low , L high ] and the other 50 to test whether this interval contains p = 0 . 8 of the total mass of the distribution of L at α = 0 . 05 significance level. In Figure 2, we plot L low and L high, the dark blue lines. We perform one lower-tailed and one upper-tailed test [17]. The null hypotheses are: 1 -1 -p 2 population quantile is no greater than L high and 1 -p 2 population quantile is at least as great as L low. Indeed, both L low and L high have the desired behavior. Furthermore, in Figure 3, we report the relevant test statistics, T 1 which is the number of draws of L no greater than L high and T 2 which is the number of draws of L less than L low . t 1 and t 2 are the minimum numbers such that P ( Y ≤ t 1 ) = α and P ( Y ≤ t 2 ) = 1 -α , where Y is a binomially distributed random variables with parameters p and n = 50 . The leftmost bar plot compares T 1 and t 1 , the middle bar plot compares T 2 and t 2 , and the rightmost bar plot shows the p -value and the significance α . To accept the null hypothesis we need T 1 > t 1 , T 2 ≤ t 2 , and high p -value. We report an aggregated p-value. We accept the null hypothesis in both cases - the test statistics T 1 and T 2 satisfy the required conditions and the p -values are high.

Overall, we can conclude that the prediction of Theorem 3 agrees with the empirical observations that are statistically significant.

(a)

τ

m

=

o

(

m

-

1

d

-

1

)

with

σ

2

= 1

and

d

= 6

.

(b)

τ

m

=

ω

(

m

-

1

d

-

1

)

with

σ

2

= 10

and

d

= 4

.

τ

m

= Θ(

m

-

1

d

-

1

)

with

σ

2

= 10000

and

d

= 6

100

80

Max loss estimate

error

60

Min loss estimate

noise level

null predictor error

40

20

0

0

2000

4000

6000

8000

10000

m

8000

Max loss estimate

7000

Min loss estimate

noise level

6000

null predictor error

5000

4000

3000

2000-

1000

200

400

600

800

1000

m

16000

Max loss estimate

Min loss estimate

noise level

14000

null predictor error

12000

error

10000

8000

0

2000

4000

6000

8000

10000

m

(c)

.

(a)

τ

m

=

o

(

m

-

1

d

-

1

)

with

σ

2

= 1

and

d

= 6

.

(b)

τ

m

=

ω

(

m

-

1

d

-

1

)

with

σ

2

= 10

and

d

= 4

.

(c)

τ

m

= Θ(

m

-

1

d

-

1

)

with

σ

2

= 10000

and

d

= 6

.

Comparison of T_1 and t_1

Comparison of T_2 and t_2

Comparison of p-values and significance level

50

1.0

p-value

alpha

40

0.8

OE

0.6 -

20

0.4 -

10

0.2

3500 4000 4500 5000 6000 7000 8000 9000 10000 11000

3500 4000 4500 5000 6000 7000 8000 9000 10000 11000

0.0

3500 4000 4500 5000 6000 7000 8000 9000 10000 11000

Sample size m

Sample size m

Sample size m

Comparison of T_1 and t_1

Comparison of T_2 and t_2

Comparison of p-values and significance level

50

T_1

p-value

alpha

40

0.8

OE

0.6 -

20

0.4

10

0.2

1002003004005006007008009001000

1002003004005006007008009001000

0.0

1002003004005006007008009001000

Sample size m

Sample size m

Sample size m

Comparison of T_1 and t_1

Comparison of T_2 and t_2

Comparison of p-values and significance level

O

p-value

40

0.8

30

0.6

20

0.4

10

0.2

35004000 4500 5000 6000 700080009000 10000 11000

3500 4000 45005000 6000 7000 8000 9000 1000011000

3500 4000 4500 5000 6000 7000 8000 9000 10000 11000

Sample size m

Sample size m

Sample size m

B The case of fixed dimension

Cost of overfitting The cost of overfitting, introduced in [66], is defined as the ratio of the test risk of an interpolating solution and the test risk of the optimally regularized solution. It measures how worse off are we interpolating rather than regularizing.

Definition 15 (Cost of overfitting) . Let ˆ f δ ∗ be the optimally regularized solution, i.e.

For a distribution D over X × R , we will define the cost of overfitting C ( D , m ) as

Zhou et al. [66] show that agnostically to the distribution D , the behavior of the cost of overfitting is tightly controlled by the effective ranks of systems of eigenvalue { λ i } ∞ i =1 of a given kernel (coming from Mercer's theorem)

We will assume additionally that the eigenvalues are nonzero, so r k and R k are well defined on N . Since we chose that the eigenvalues are sorted and since they square summable, we have that

The following two results of [66] summarize how we will use effective ranks to control the cost of overfitting. Proposition 16 gives a necessary and sufficient condition for the overfitting to benign in terms of the cost of overfitting, i.e. that C ( D , m ) → 1 , in terms of effective ranks.

Proposition 16 (Necessary and sufficient condition for benign overfitting [66]) . Let k n be the smallest integer k ∈ N ∪ { 0 } for which n < k + r k . Then E 0 → 1 if and only if

Proposition 17 gives a sufficient condition for ˜ R ( ˆ f 0 ) →∞ in terms of effective ranks.

Proposition 17 (Sufficient condition for catastrophic overfitting [66]) . Let ε > 0 and let k = (1+ ε ) m . If lim k →∞ r k k = 0 then E 0 →∞ , i.e. the overfitting is catastrophic.

Using sharp bounds on eigenvalues { λ i } ∞ i =1 , derived in Theorem 28, we can derive bounds on effective ranks so that we can apply the results of [66] and Equation (2) to bound the test risk of minimum norm interpolant.

When the input is distributed uniformly on the sphere, the eigenfunctions { ϕ k } in the Mercer decomposition of the kernel function can be taken to be the spherical harmonics Y ks . Using the FunkHecke formula we can find the closed form of eigenvalues for the Gaussian kernel [46], which are given in terms of bandwidth τ m and modified Bessel function of the first kind I v ( x ) . In Appendix D, we will derive bounds on the sizes of the eigenvalues using the properties of Bessel functions and their multiplicities.

We need to take a particular input distribution, in order to able to compute the eigenvalues explicitly and to know their multiplicities. The result of Cao et al. [9] offers one possible way of generalizing our result to more general different distributions

We split the proof into two parts. First, we will bound the cost of overfitting. Then, we will provide lower bounds on the test risk of Gaussian KRR.

Bounds on cost of overfitting The bounds on cost of overfitting will be used to show whether the test risk is bounded above or away from Bayes risk.

Lemma 18 (Cost of overfitting for Gaussian KRR) . Let X ∼ Unif ( S d -1 ) . Then the following bounds hold for the minimum norm interpolating solution of KRR and its cost of overfitting C ( D , m ) and E 0 :

If τ m ≤ m -1 d -1 t ( m ) , where t ( m ) → 0 as m →∞ then there is a constant C I that depends on d and C , such that

If τ m ≥ m -1 d -1 t ( m ) , where t ( m ) →∞ as m →∞ then there is a integer m 0 depending on d and t ( m ) such that for all m ≥ m 0

If C 1 m -1 d -1 ≤ τ m ≤ C 2 m -1 d -1 then there exist constants C I > 0 and C II depending on C 1 , C 2 and d such that

Proof. We will break down the proof into the proof of the three parts.

Part 1: We will use Proposition 16 to prove this that E 0 → 1 . We can see the upper bound on E 0 from Theorem 2 [66]. We have that for k < m

Now we will show that for k = 1 , we have that k + r k > m and lim m →∞ m R 1 = 0 . This implies that conditions of Proposition 16 hold so E 0 → 1 . We will rewrite r k = ∑ i>k λ k λ k +1 using ˜ λ i . We have that

where N ( d, 1) = d , so ˜ 1 = 1 and ˜ N ( d, ˜ 1) = d -1 . Therefore, we have

Let T = 2 τ 2 m . Taking √ Tg ( m ) 2 ≤ j ≤ √ Tg ( m ) , k = 1 , we can apply the bound Theorem 28 on ˜ λ 1+ j ˜ λ 1 . Therefore, we have that

In particular, we have that for m large enough, for all such j it holds that ˜ λ 1+ j ˜ λ 1 > 1 2 . Going back to r 1 , this means that

If √ Tg ( m ) is not an integer, we can take its integer part and add a small constant since √ Tg ( m ) will be large. Note that τ m ≤ m -1 d -1 t ( m ) , so

Therefore

so as long as g ( m ) t ( m ) → ∞ , r 1 > m for m large enough. We can take g ( m ) = √ t ( m ) . Then, in Proposition 16, we can select k m = 1 , since 1+ r 1 > m for large enough m . Note also that R 1 ≥ r 1 , so we also have that R 1 ≥ r 1 > m 1 ( g ( m ) t ( m ) ) d -1 , so

This finishes the proof of the first claim.

Part 2: Let τ m ≥ m -1 d -1 t ( m ) , where t ( m ) →∞ as m →∞ . We will use Proposition 17 to prove this claim. Using Theorem 5 from [66] we know that for any ε > 0 if r k = o ( m ) for k = (1 + ε ) m then the following bound holds

We will show that there is a constant U > 0 such that for any k ≥ m , we have that

By Theorem 28, we have that for j ≥ √ Tg ( m ) where g ( m ) →∞ as m →∞ that

Let k = (1 + ε ) m . We have that

Now, if we take j ≥ j 0 = √ Tg ( m ) , it holds that

Therefore, we have that

Note that by Theorem 28, for j ≥ l √ Tg ( m ) we have that

Note also that this bounds holds for all l simultaneously. Therefore, we can write

Second to last inequality is true because g ( m ) →∞ as m →∞ . The last inequality follows from the fact that ∑ ∞ l =1 ( l +1) d -1 exp( -l 2 4 ) is bounded by a constant. Note that the bound is independent of k , so we can vary ε with m as well. If we choose ε = 1 √ m , the desired result follows.

Part 3: Let C 1 m -1 d -1 ≤ τ m ≤ C 2 m -1 d -1 . Note that the following two bounds hold for E 0 [66]:

For k < m such that r k + k > m

Note that in this case c ′ 1 m 1 d -1 ≤ √ T = √ 2 τ 2 m ≤ c ′ 2 m 1 d -1 , where c ′ 2 = √ 2 C 2 1 (and similarly for indices 1 and 2 swapped with inequalities reversed). √

Theorem 28 shows that for i is such that i ≤ ( k T ) , we have that for ˜ λ i it holds that

Take k = (1 + ε ) m . We want to show that there is a constant B > 0 that depends on the dimension d and the constant C such that

Let ˜ k = ((1 + ε ) m ) 1 d -1 . Note that by an analogous proof to Theorem 28, we have that for the size of eigenvalues it holds

For k ≥ m it holds that

For l ˜ k ≤ j ≤ ( l +1) ˜ k , we have that

Wehave that ˜ k = 1 c ′ (1+ ε ) 1 d -1 c ′ m 1 d -1 = 1 c ′ (1+ ε ) 1 d -1 √ T = α √ T for α = 1 c ′ (1+ ε ) 1 d -1 . Therefore, we have that

as m →∞ i.e. √ T →∞ . Therefore, for m large enough, for all l we have that if l ˜ k ≤ j ≤ ( l +1) ˜ k then

The last inequality is true because we can take m large enough so that α √ T and √ T α are both greater than 400 d . Then we have that for all l

The last inequality holds because of the following

Therefore, we can bound r k as follows

Therefore, applying claim 1 , we have that k -m + r k < ( C (1 + ε ) + ε ) m , so we have that k -m k -m + r k > ε ( C (1+ ε )+ ε ) , so it follows that E 0 > (1 -(1+ ε ) ε ( C (1+ ε )+ ε ) ) -1 > 1 .

Note that the above proof actually shows that for any k 1 ≤ k , we have that r k 1 ≤ Cm . Now we want to show that r k is lower bounded and that we can take k < m such that k + r k > m . Again apply Theorem 28, we can generalize

to other ratios as well. Let j ≤ k √ T and let k √ T ≤ i ≤ ( k +1) √ T . Then we have that

Note that c ′ 1 m 1 d -1 ≤ √ T = √ 2 τ 2 m ≤ c ′ 2 m 1 d -1 , where c ′ 2 = √ 2 C 2 1 (and similarly for indices 1 and 2 swapped with inequalities reversed). Therefore, we have that the following bound holds on r k for any k < m :

Note that since √ T ≥ c ′ 1 m 1 d -1 then N √ T ≥ n l ( c ′ 1 ) d -1 m , so ˜ m ≤ 1 c ′ 1 1 n 1 d -1 l . Therefore, if we take j = ˜ k , l = 1 c ′ 1 1 n 1 d -1 l , and any l √ T ≤ i ≤ ( l + 1) √ T , we have that (since exp( -2 l ( l + 1)) < exp( -l 2 ) )

Note that there is at least N ( l +1) √ T -N l √ T ≥ n l ( ( l +1) d -1 -l d -1 ) m such indices i , we have that

where β = n l ( ( l +1) d -1 -l d -1 ) exp( -2 l ( l +1)) . Note that β is independent of k , so in particular for k = (1 -β/ 2) m , we will have r k + k > m . Therefore we can use the upper bound on E 0 to see that

Since β only depends on d and C 1 and C 2 , this finishes the proof.

In particular, Lemma 18 shows that when τ m = o ( m -1 d -1 ) or τ m = ω ( m -1 d -1 ) , then lim sup ˜ R ( ˆ f 0 ) ≤ ∞ .

Lower bound for the test risk of Gaussian KRR Now we will show lower bounds for the test risk of Gaussian KRR.

Lemma 19 (Lower bound for the test risk) . Under Assumption 2, the following bounds hold for the risk ˜ R ( ˆ f 0 ) of the minimum norm interpolating solution of Gaussian KRR:

If τ m = o ( m -1 d -1 ) , then ˜ R (0) ≤ lim inf ˜ R ( ˆ f 0 ) ≤ lim sup m →∞ ˜ R ( ˆ f 0 ) < ∞ . More precisely, if τ m ≤ m -1 d -1 t ( m ) , where t ( m ) → 0 as m →∞ , then there is a scalar c d that depends only on the dimension so that for all such t ( m ) there is m 0 that depends on t ( m ) such that for all m>m 0 we have ˜ R ( ˆ f 0 ) > σ 2 +(1 -c d t ( m ) d -1 2 ) ∥ f ∗ ∥ 2 .

If τ m = ω ( m -1 d -1 ) , then lim m →∞ ˜ R ( ˆ f 0 ) = ∞ . More precisely, if τ m ≥ m -1 d -1 t ( m ) , where t ( m ) →∞ as m →∞ then there is an integer m 0 that depends on the dimension and t ( m ) such that for m>m 0 , we have that tR ( ˆ f 0 ) → √ m ( σ 2 ) (note that ˜ R (0) is bounded). Hence, for large enough m we have ˜ R ( ˆ f 0 ) > ˜ R (0) .

If τ m = Θ( m -1 d -1 ) , then lim sup m →∞ R ( ˆ f 0 ) < ∞ . Moreover, suppose that C 1 m -1 d -1 ≤ τ m ≤ C 2 m -1 d -1 for some constants C 1 and C 2 , then there exist η, µ > 0 that depend only on d , C 1 , and C 2 , such that for all m we have ˜ R ( ˆ f 0 ) >µ ∥ f ∗ ∥ 2 +(1+ η ) σ 2 . Consequently, ˜ R ( ˆ f 0 ) > ˜ R (0) as long as σ 2 > 1 -µ η ∥ f ∗ ∥ 2 .

Proof. All three proofs will similarly follow by directly analyzing Equation (2).

Part 1: Note that for any i , we have that ˜ λ i + κ 0 ≤ ˜ λ 1 + κ 0 , so we have that L i ≥ ˜ λ i ˜ λ 1 L 1 . From Theorem 28 we know that for j ≤ √ Tt ( m ) 1 2 , we have that

so there is m 0 that depends on t ( m ) such that for all m>m 0

for all j ≤ √ Tt ( m ) 1 2 . Therefore we have that

Note that ( N ( d, 1) + . . . N ( d, √ Tt ( m )) ) > c l ( √ Tt ( m ) 1 2 ) d -1 = c ′ l m ( t ( m )) -d -1 2 , where c ′ l = √ 2 c l . Therefore, we have that L 1 < 2 c ′ l ( t ( m )) d -1 2 . Therefore, we have that

From Equation (2), we have that then

This shows the first claim.

For the second part, note the following property of L i,δ = λ i λ i + κ δ : if for j > i , we have that L i,δ > L j,δ . Let λ j λ i > c for some fixed c > 0 and j > i , then we have that

By Theorem 28, we have that for j ≤ √ Tg ( m ) with g ( m ) → 0 as m →∞

So there is m>m 0 such that for m>m 0

Note that

For this it suffices to have since L i,δ are decreasing. Note that there is at least N ( d, 1) + · · · + N ( d, √ Tg ( m )) > n l ( √ Tg ( m )) d -1 > c u m ( g ( m ) t ( m ) ) d -1 of indices i such that

Therefore, if we select g ( m ) = √ t ( m ) , then we have that

This implies that for all j simultaneously

This finishes the proof of the first part.

Part 2: Note that we showed that for m large enough, E 0 > √ m . This shows that

So, there is m 0 that depends only on t ( m ) such that for m>m 0 we have that ˜ R ( ˆ f 0 ) > √ mσ 2 .

Part 3: Note first that L i ≥ L j for i ≤ j . Let T = 2 τ 2 m . From Lemma 26 it follows that

Take α such that N ( d, 1) + · · · + N ( d, α T ) > 2 m . Then, we have that

Note that (1 -L 1 ) 2 < (1 -L i ) 2 and

If κ 0 > λ 1 , then 2 κ 0 > λ 1 + κ 0 , so 1 2 κ 0 < 1 λ 1 + κ 0 . This shows that κ 2 0 ( λ 1 + κ 0 ) 2 > 1 4 . Otherwise κ 0 ≤ λ 1 , so κ 0 + λ 1 ≤ 2 λ 1 . This implies that

From Theorem 28 it follows that λ α √ T 4 λ 2 1 > exp( -α 2 ) . Since ( 1 n u 2 C 2 1 ) 1 d -1 < α < ( 1 n l 2 C 2 2 ) 1 d -1 , we know that exp ( -α 2 ) > exp ( -( 1 n l 2 C 2 2 ) 2 d -1 ) , so µ = exp ( -( 1 n l 2 C 2 2 ) 2 d -1 ) . Combining Lemma 18 and Lemma 19, we get Theorem 3.

This translates to

C The case of increasing dimension

In Appendix C.1 we will prove Theorem 7 and Theorem 9. In Appendix C.2, we will prove related claims about the eigenvalue multiplicities for relevant scalings of the dimension Corollary 12 and Corollary 13. In Appendix C.3, we will give a detailed proof of Corollary 13. In Appendix C.4, we will discuss when the conditions posed on kernels are satisfied.

C.1 Proofs of Theorem 7 and Theorem 9

Theorem 20 (Error scaling for any kernel ridge regression) . Let d and m be any dimension and sample size. Define L m , U m , k m , N ( i ) , N l , and ˜ λ k as in Definition 5. Consider KRR with a kernel K satisfying Assumption 6 for some A . Assume that for some integer l , the target function f ∗ satisfies Assumption 4 with at most N l nonzero coefficients. Then, the risk of the minimum norm interpolating solution is bounded by the following:

Additionally, we can get an alternative bound if we swap ( ∑ l i =1 N ( d, i ) 1 ˜ λ 2 i ) with ( ∑ l i =1 N ( d, i ) 1 ˜ λ i ) , i.e.

Proofs of Theorem 7 and Theorem 20: Note first that Equation (2) implies that

Let L i = ˜ λ i ˜ λ i + κ 0 . Then, we can rewrite it as

with a slight of abuse of notation for β i . From Equation (5), we have that for k < m

Take k = L m . Note that by Definition 5, L m < m Then λ k +1 = ˜ λ k m +1 and it repeats N ( d, k m +1) times. Therefore, we have that

Therefore, r k + k > N ( d, k m +1) + L m = L m > m . Therefore, we can apply Equation (5). We get that

Note now that

Note that ˜ λ i + κ 0 > κ 0 so we have that L i < ˜ λ i κ 0 . Therefore, we have that

Therefore, we have that κ 0 < A m . Since (1 -L i ) 2 = κ 2 0 ( κ 0 +˜ λ ) 2 < κ 2 0 ˜ λ 2 . Since there are only l d nonzero β i , we have that

Going back to Equation (2), we have

This shows the first bound. For the second bound, note that

So if we take β to have only l nonzero terms, we get

From this, we have that

Therefore, we have that

Therefore, we have the improved inequality from

This gives the second bound on the test risk, as we can bound

[Lower bound for test risk in increasing dimension] Let k m and L m be as in Definition 5. Consider learning a target function f ∗ , with some sample size m . Let K be a kernel satisfying Assumption 8 for some A and b . Consider the minimum norm interpolating solution of KRR (with any data distribution) with kernel K . Then, for the risk of minimum norm interpolating solution, the following lower bound holds:

Proof of Theorem 9: By Theorem 3 from Zhou et al. [66], if k is the first k < m such that k + br k ≥ m , then

Therefore, it suffices to show that the first such k is actually L m = N ( d, 1) + · · · + N ( d, k m ) . Note that

Since max i ≤ k m ( 1 ˜ λ i ) < m -L m b , for l < N ( d, 1) + · · · + N ( d, k m ) , we have that br l + l ≤ N ( d, 1) + · · · + N ( d, k m ) -1 + b max i ≤ k m ( 1 ˜ λ i ) < L m + m -L m ≤ m , so the first l for which r l + l > m is l = L m = N ( d, 1) + · · · + N ( d, k m ) . Plugging in k = L m we get that E 0 ≥ ( 1 -( b b +1 ) L m m ) -1 , so from Equation (6), we have ˜ R ( ˆ f 0 ) ≥ ( 1 -( b b +1 ) 2 L m m ) -1 σ 2 .

C.2 Dot-product kernels on the sphere

First we will prove results about the eigenvalue multiplicites Corollary 12 and Corollary 13.

Theorem 21 (Log-scaling multiplicity) . Let d = log 2 m , i.e. m = 2 d , and let k d be an index for which the following holds:

Then the following hold:

dN ( d, d 5 ) 2 d is decreasing and has lim d →∞ dN ( d, d 5 ) 2 d → 0 . This also holds for dN ( d,k ) 2 d with k = f ( d ) ≤ d 5 for all d .

N ( d, d 2 ) 2 d is increasing and has lim d →∞ N ( d, d 2 ) 2 d → ∞ .This also holds for N ( d,k ) 2 d with k = f ( d ) ≥ d 2 for all d .

There is an absolutet constant d 0 such that for d > d 0 , we have that d 5 ≤ k m ≤ d 2 .

There are absolute constants c l -1 = 1 54 , c l = 1 9 , and c l +1 = 2 3 such that N ( d, k m + i ) > c l + i m , for all i ∈ {± 1 , 0 } . Additionally, there are constants c u -1 = 1 3 , c u = 1 , c u +1 = 6 , such that N ( d, k m + i ) < c u + i m for all i ∈ {± 1 , 0 } .

Furthermore, if d log m = Θ(1) , then we will also have L m = Θ( m ) and L m = Θ( m ) .

Proof. We will split the proof into three parts. Note first that N ( d, k ) increases as k increases. This can be seen from the ratio of consecutive multiplicities

Part 1: We will use Stirling's approximation to estimate N ( d, k ) . It states that n ! ≈ ( n e ) n √ 2 πn . Therefore, we have that

Since N ( d, k ) is increasing in k and k ≤ d 5 , so we can just plug in k = d 5 in the expression. Therefore

Note that ( 5 1 / 5 ( 6 5 ) 6 / 5 ) < 1 . 8 , so ( 5 1 / 5 ( 6 5 ) 6 / 5 ) d √ d < 1 . 8 d √ d . In particular, we have that

Part 2: The same calculation in this case gives

but note that ( 2 1 2 ( 3 2 ) 3 2 ) > 2 . 5 , so N ( d, d 2 ) 2 d > ( 2 . 5 2 ) d 1 √ d →∞ .

Part 3: Note that if k m ≤ d 5 then for d large enough

Similarly, note that if k m > d 2 , then N ( d, k m ) > d for d large enough. Both of these imply hat d 5 ≤ k m ≤ d 2 when d > d 0 ( d 0 is determined by when the inequalities above start holding).

Part 4: Note that when d 5 ≤ k ≤ d 2 , we have that 6 > N ( d,k +1) N ( d,k ) = 2 k + d 2 k + d -2 k + d -2 k +1 > 3 for d large enough, since the firs ratio is close to 1 and the second is 1 + d -2 k +1 . This implies that N ( d,k ) N ( d,k +1) < 1 3 whenever d is large enough (larger than an absolute constant) and k ≤ d 2 . Note now that

This now implies that N ( d, k m ) > 1 6 N ( d, k m +1) > 1 9 m , N ( d, k m -1) > 1 54 m . Similarly we have that N ( d, k m ) < m , so N ( d, k m +1) < 6 m and N ( d, k m -1) < 1 3 m .

Note that the same proof works even if d is not exactly equal to log 2 m . This shows that whenever d log m = Θ(1) , we have that L m = Θ( m ) and L m = Θ( m ) .

Remark 22. Note that this does not say that we will always have tempered overfitting for d log m = Θ(1) as sometimes it can happen that L m → m , so we cannot apply Equation (5) and Equation (6). In this case we expect the overfitting to be catastrophic.

Theorem 23 (Sub-polynomial scaling multiplicity) . Let K be any dot product kernel on the sphere S d -1 (with nonzero eigenvalues). Let l ∈ N and let m = 2 2 2 l and let d = 2 2 l . This corresponds to the case d = exp ( √ log m ) , which is sub-polynomial. Then, for the upper and lower index, L m and L m , we have the following

Additionally, for k = αk m , where α is a constant, we have that N ( d, k ) ≈ m α , where ≈ means up to sub-polynomial factors.

Proof. Let k m be the maximum k for which

We want to show that N ( d, k m ) = o ( m ) and N ( d, k m + 1) = Ω( m ) . Then we can take k = N ( d, 1) + · · · + N ( d, k m ) and so k m → 0 and

Note that we just need to estimate N ( d, k m ) precisely. Let k = 2 l + l . We want to show that N ( d, k ) > m . Note that by Stirling's approximation n ! ≈ ( n e ) n √ 2 πn

Only the term d k k k √ k ( 1 + k d -3 d ) k + d -3 is asymptotic here, as the rest tends to a constant as d →∞ . Note that since for all l > 5 (i.e. d )

we have that

Note that we have that then

Consider now the expression k log d -k log k -1 2 log k + k 2 d + k -3 k d . For k = 2 l + l -1 , we have that

In particular, this implies that for k ≤ 2 l +( l -1) , we have that

Now repeating the same computation for k = 2 l + l , we have that

This shows that for k ≥ 2 l + l ,

To imply now that k m = k = 2 l + l -1 , it suffices to show that N ( d, 1) + · · · + N ( d, k ) = o ( m ) . But note that we have k ≪ d , so

Therefore, we have computed with proof that k m = 2 l + l -1 . So we can now compute L m and L m with L m = N ( d, 1) + · · · + N ( d, k m ) . Note that

Since k << d , note that N ( d,k +1) N ( d,k ) > 2 , so we have that N ( d,k ) N ( d,k +1) < 1 2 , which implies that

Therefore, we have that L m m = N ( d, 1)+ N ( d, 2)+ ··· + N ( d,k m ) m < 3 2 N ( d,k m ) m ≤ 3 2 log m . From what we computed previously, we have that

Note finally that for k = α 2 l , the leading term in the exponent of N ( d, k ) as derived above is α 2 2 l , all other terms are at most l 2 l , which is sub-polynomial in m , so this shows that N ( d, αk ) ≈ m α .

Remark 24. Note that for the polynomially increasing d , there are sequences ( m,d ) for which we do not get benign overfitting, i.e. when d = m k , for integer k . Similarly in this case, there are sequences of ( m,d ) for which we do not get benign overfitting, but is much harder to identify them.

C.3 Proof of Corollary 13

Corollary 3 (Benign overfitting with Gaussian kernel and sub-polynomial dimension) . Let K be the Gaussian kernel on the sphere S d -1 with a fixed bandwidth, and suppose that for l ∈ N the dimension and sample size scale as d = 2 2 l , m = 2 2 2 l , i.e. d = exp ( √ log m ) . Consider learning a sequence of target functions f ∗ d as in Assumption 4 with S d ≤ m 1 4 . Then, we have that the minimum norm interpolating solution achieves the Bayes error in the limit ( m,d ) →∞ . In particular, for d ≥ 4 and m ≥ 16 we have

If we take S d = poly ( d ) , then we can improve the 1 m error dependence to 1 m 2 -ε . Furthermore, we can improve the bound on S d by reducing the rate of convergence. Let s d be an integer such that S d ≤ N s d . Then error dependence is 1 m 2 s d N ( d, s d ) d (1+ ε ) s d < 1 m 2 d 2(1+ ε ) s d .

Proof: We have that from Theorem 23 that in this case k m = 2 l + l -1 = log d +loglog d -1 , L m = Θ( m log m ) and L m ≥ Θ( md 0 . 89 ) . Therefore, we have that

To finish the proof, we need to estimate N ( d, 1) 1 ˜ λ 2 1 + . . . N ( d, k m ) 1 ˜ λ 2 km . Note that the eigenvalues are sorted and N ( d, k -1) = o ( N ( d, k +1)) since k m = o ( d ) . This implies that it suffices to estimate N ( d, k m ) 1 ˜ λ 2 km . Note that for the first eigenvalue, we have from Corollary 31 for α = 1 + 2 τ 2 m .

So, for ˜ λ k , it holds that

Therefore, we have that

Note also that if k = βk m for some constant β then

in the sense of Theorem 23. So, N ( d, k m ) 1 ˜ λ 2 km < m β d 2 k (1+ ε ) -2 α . We want to take k as large as possible, so we will take it k = βk m . Then d 2 k ≈ m 2 β . Therefore, as long as 3 β < 2 , we will have a polynomially scaling error. So we can take β = 2 3 . Note that with Theorem 20, we actually get even better dependence of the error, like m β d k (1+ ε ) -α . Plugging the first estimate into Theorem 7, for 3 α = 1 , we have that

If we want to achieve S d = poly ( d ) , note that we just need to take k to not scale wit d . That is, if we want deg S d = n , then we can take k = n . The eigenvalue will be

The multiplicity will be N ( d, n ) = Θ( d n ) . Note that then error scales as d 5 n m 2 , i.e. since d is sub poly m , we can take any such n .

C.4 Kernel conditions

We have three main assumptions on kernels. First, the sum of its eigenvalues is bounded.

Assumption 6 (Bounded sum of eigenvalues) . Assume that the kernel K has a bounded sum of eigenvalues, i.e. there is a constant A such that ∑ ∞ i =1 N ( i ) ˜ λ i ≤ A . For a sequence of kernels K ( d ) , assume that all such A ( d ) are bounded by some constant A . 2

Second, there is a lower bound on the eigenvalues.

Assumption 8 (Lower bound on eigenvalues) . Assumme that the kernel K has eigenvalues that are not too small, i.e. there is a constant b such that max i ≤ k m ( 1 ˜ λ i ) < m -L m b . For a sequence of kernels K ( d ) , assume that for the corresponding m = m ( d ) (since d = d ( m ) , we can also "invert" the dependence) all such b ( d ) are bounded below by some b .

And third, the eigenvalues don't decrease too fast.

Assumption 10 (Eigenvalue decay) . The eigenvalues don't decrease too quickly, i.e. for k m as in Definition 5, we have that there is a constant c such that max i ≤ k m ( 1 ˜ λ i ) ≤ cN ( k m ) . For increasing dimension, we require that max i ≤ k m ( 1 ˜ λ i ) ≤ cN ( d, k m ) .

Note that Assumption 6 will hold quite broadly, in particular at least for dot product kernels on the sphere (that are effectively the same kernel just in increasing dimensions). Note also that the Assumption 10 is almost always stronger than Assumption 8. This follows from the definition of L m and k m , i.e. since L m = N (1) + · · · + N ( k m ) , we have that N ( k m ) < m and as long as L m < Θ( m ) , we can take b large. This is usually the case (we show previously in Appendix C.2 for dot-product kernels on the sphere). For d = ω (log m ) , we even have L m = o ( m ) . So in particular N ( d, k m ) < Θ( m -L m ) and for d = ω (log m ) , we have N ( d, k m ) = o ( m -L m ) , so indeed Assumption 10 is stronger than Assumption 8.

Assumption 6 and Assumption 10 are also taken to be true in the literature on the polynomially scaling dimension [4, 64].

First assumption: For dot-product kernels on the sphere, we know that the eigenfunction ϕ i are actually the spherical harmonics Y ks . Let K ( x, y ) = h ( ∥ x -y ∥ ) . Note that for x ∈ S d -1 , it holds that

So the assumption in Theorem 7 holds with A = h (0) . For the Gaussian kernel, this is A = 1 . Similarly, A = 1 for Laplace kernel.

Therefore, since

Second Assumption: For the eigenvalue assumption, max i ≤ k m ( 1 ˜ λ i ) < m -L m Ab , note the following. We will usually have L m = o ( m ) . In particular, whenever d = ω (log m ) for a dot-product kernel on the sphere this will hold.

The eigegnvalue assumption will hold for the Gaussian kernel in the non-integer polynomial regime, d α = m . In Appendix C.3, we showed that 1 ˜ λ k < d k , so since k m = ⌊ α ⌋ , we will have 1 ˜ λ k < d α -Θ( d ⌊ α ⌋ ) . Similarly, we can show this for the Gaussian kernel for d = log m and d = exp( √ log m ) regime. Note that in Corollary 31, we showed that ˜ λ 1 > 1 d α and ˜ λ k > 1 d α + k , so then 1 ˜ λ k Additionally, in the d = exp( √ log m ) regime, we showed that L m = o ( m ) and that 1 ˜ λ k = Θ( L m ) for any k = Θ( k m ) < k m . The same proof shows that this assumption holds for any sub-polynomially scaling dimension and Gaussian kernel.

Furthermore, this assumption can be weakened to the following: there exists b > 0 such for all k ≤ L m , b ∑ i>k λ i < λ k +1 ( m -k ) . Additionally, since the assumption is specific to our approach, it seems to be possible to weaken it even further.

Third assumption: This assumption holds for dot-product kernels on the sphere. It was shown in Zhang et al. [64] (Lemma 5.2.1) under another assumption similar to (1) that it holds for polynomially scaling dimension. It is also assumed in Barzilai and Shamir [4] (page 11) for polynomially scaling dimension. Furthermore, Cao et al. [9] (Theorem 4.3) shows that this holds for Neural Tangent Kernel even more broadly, i.e. for all i and not only i ≤ k m .

D Sharp bounds on the eigenvalues of the Gaussian kernel

In this section, we will summarize and prove the results about eigenvalues of Gaussian and Laplace kernels when X ∼ Unif ( S d -1 ) .

Given a positive semi-definite kernel function K : X × X → R , we can decompose it as

where λ k and ϕ k are the eigenvalues and eigenfunctions of the operator associated to K , L K : L 2 D X ( S d -1 ) → L 2 D X ( S d -1 )

L K is a Hilbert-Schmidt operator and has a countable system of non-negative eigenvalues λ k satisfying ∑ ∞ k =1 λ 2 k < ∞ . The corresponding L 2 D X ( S d -1 ) -normalized eigenfunctions { ϕ k ( x ) } ∞ k =1 form an orthonormal basis of L 2 D X ( S d -1 ) . The eigenfunctions in this case are given by spherical harmonics, Y i,s . The eigenvalues corresponding to all Y i,s for fixed i have the same eigenvalue, ˜ λ i . Eigenvalues ˜ λ i are reach repeated N ( d, i ) = (2 i + d -2)( i + d -3)! i !( d -2)! times. So the sequence { λ i } ∞ i =1 = ˜ λ 1 , . . . , ˜ λ 1 , ˜ λ 2 , . . . , ˜ λ 2 , . . . . Using Funk-Hecke formula, we can compute ˜ λ k explicitly, both for the case of Gaussian [46] kernel. For Gaussian kernel, we have the following theorem about the eigenvalues ˜ λ k .

Theorem 25 (Eigenvalues of the Gaussian kernel [46]) . Let X ∼ Unif ( S d -1 ) , with d ∈ N and d ≥ 2 . For K ( x, t ) = exp ( -∥ x -t ∥ 2 τ 2 m ) , τ m > 0 , and k ∈ N 0 we have that

Each ˜ λ k occurs with multiplicity N ( d, k ) = (2 k + d -2)( k + d -3)! k !( d -2)! (we use ˜ λ notation to indicate that it has multiplicity) and its corresponding eigenfunction are the spherical harmonics of order k on S d -1 . Here, I v ( z ) , v, z ∈ C is the modified Bessel function of the first kind

It will be useful to know the size of N ( d, i ) . Also, the size of the sum of the first k multiplicities will be useful. Let N k = ∑ k i =1 N ( d, i ) .

Lemma 26 (Size of multiplicity) . For N ( d, i ) = (2 k + d -2)( k + d -3)! k !( d -2)! , it holds that

Additionally, there exist constants n l , n u > 0 depending on the dimension d such that N k is bounded below and above by the following

Proof. Since (2 k + j ) ≤ (2 k ) j for j ≥ 2 and ( k +1) ≤ 2 k , we have that

and

Note that by Bernoulli's formula

Therefore, we have that

Lemma 27 (Inverting the index) . If the index of an eigenvalue ˆ λ j is such that N k -1 ≤ j ≤ N k -1 , then ˆ λ j = λ k . We will denote such j with ˜ k , i.e. ˜ k is an index such that ˆ λ ˜ k = λ k .

Proof. Immediate from the fact that { λ i } ∞ i =1 is a sequence with ˜ λ i repeating N ( d, i ) times, in that order.

An interesting property of eigenvalues of the Gaussian kernel is that they are sorted because the Modified Bessel functions are [48]. In particular, I v +1 ( x ) < I v ( x ) for all v, x > 0 , so ˜ λ i +1 < ˜ λ i . This is not the case for the Laplace kernel.

Theorem 28 (Size of Ratios of Eigenvalues for Gaussian Kernel) . Let T = ( 2 τ 2 m ) . For j ≤ √ Tt ( m ) , k ≤ √ Tt ( m ) , where t ( m ) → 0 as m →∞ , we have that

For j ≥ √ Tt ( m ) , where t ( m ) →∞ as m →∞ we have that for any k

Proof. First of all, note that from [48], we have that for v ≥ 1 2

In particular, this means that for the eigenvalues ˜ λ k , we have

This can be bounded with a simpler expression as follows

We will use these bounds to derive tight bounds for the ratios ˜ λ k + j ˜ λ k . Note the following

Note now that

Note that since ( x + j -i )( x + i ) = x 2 + ix +( j -i ) i , we have that ( x + j -i )( x + i ) ≥ ( x + j ) x , therefore

We use j +1 and j -1 to account for the fact that j might be odd when we split into ( j -1) / 2 pairs. When j is even, we split into j 2 pairs and use the fact that the term with exponent j +1 2 is larger.

Let T = ( 2 τ 2 m ) . We can bound the ratio ˜ λ k + j ˜ λ k tightly now.

When j ≤ √ Tm -δ , k ≤ √ Tm -δ , we have that

When j ≥ √ Tm δ , we have that for any k

To see why this is true, note first that

as m →∞ . For the second inequality, note that

We also have that

as m →∞ . Note that we can turn the limits a m → t ( m ) into inequalities of the form (1 -ε ) t ( m ) < a m < (1 + ε ) t ( m ) for some ε and all m since the convergence is uniform.

The following is a simple corollary.

Corollary 29. Let T = ( 2 τ 2 m ) . If T > 1 and the index of the eigenvalue i is such that i ≤ ( k √ T ) d -1 , we have that for λ i it holds that

Proof. Note that ( 1 1+ k √ T ) k √ T is increasing in √ T and note that √ T increases as m increases. Note that for √ T = 1 it suffices to show e ( k ) > 1 + k which is true for all k as long as √ T > 1 .

The following simple bound also holds for eigenvalues of Gaussian kernel.

Proposition 30 (Ratio of eigenvalues bounded above [46]) . For the eigenvalues associated to the Gaussian Kernel ˜ λ k of bandwidth τ m we have that

It is straightforward to convert the bounds on ratios of eigenvalues Theorem 25 to bounds on the sizes of the actual eigenvalues.

Corollary 31 (Sizes of the eigenvalues of Gaussian kernel) . The following bounds hold for the largest eigenvalue of the Gaussian kernel

Therefore, for ˜ λ k , we have that and

Proof. Note that from [62] we have

We also have from [48]

Then

So then

So then

Furthermore, for τ m fixed we have that

By repeating the same argument, the claim about ˜ λ k follows. Note that

Therefore, we have that

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All the contributions are clearly stated in the introduction, particularly the third paragraph of the introduction. The assumptions and limitations are also clearly stated in that paragraph.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: All assumptions are clearly and concisely stated. We also provide detailed reasoning for each one of our assumptions. The assumptions are further discussed in the appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are stated clearly before each result is presented. We give an outline of proofs in the main body and complete proofs in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The description of the experiments that are provided along with the figures is sufficient to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide code and instructions required to reproduce our experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the necessary details are included in the descriptions of the figures.

Guidelines:

. The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report appropriate information about the statistical significance of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The compute resources are stated along the explanation of the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes, the paper conforms to all aspects of the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper concerns the overfitting behaviour of Gaussian kernel ridge regression (KRR) with varying bandwidth or dimensionality. The contribution is two-fold: \n\n1. In fixed-dimension, the ridgeless solution of Gaussian KRR is not consistent with any varying or tuned bandwidth. \n\n2. In high dimension with input dimension $d=2^{2^l}$, sample size $m=2^{2^{2^l}}$ for any arbitrary integer $l$, the overfitting of Gaussian KRR is benign.\n\nAll the results are under the Gaussian universality Ansatz and the (non-rigorous) risk predictions in terms of the kernel eigenstructure introduced in [Simon2023]. \n\nReference:\n\n- James B. Simon, Madeline Dickens, Dhruva Karkada, and Michael R. DeWeese. The eigenlearning framework: A conservation law perspective on kernel regression and wide neural networks. arXiv preprint arXiv:2110.03922, 2021.\n\nSTRENGTHS:\nThe two contributions are novel.\n\nWEAKNESSES:\nHowever, I have serious concerns about the assumptions made in this paper. Firstly, the eigenframework under the Gaussian Universality Ansatz (GUA) is non-rigorous and actually fails to explain the catastrophic overfitting of NTK in fixed dimensions, as reported in \\cite{Barzilai2023} and \\cite{Cheng2024}. I am uncertain whether benign overfitting would still hold in high-dimensional settings. This paper does not offer experimental validation either, so I cannot ensure the correctness of the claim for Gaussian kernels (not its proxy under GUA).\n\nSecondly, the catastrophic overfitting of Gaussian kernels with fixed bandwidth is well-known. It is not difficult to imagine that Gaussian KRR with varying bandwidth remains inconsistent. While the first contribution of this paper is novel, I question whether it is significant enough for paper acceptance, especially given my concerns regarding the second contribution.\n\nReference:\n- Barzilai, Daniel, and Ohad Shamir. \"Generalization in kernel regression under realistic assumptions.\" arXiv preprint arXiv:2312.15995 (2023).\n- Cheng, Tin Sum, et al. \"Characterizing overfitting in kernel ridgeless regression through the eigenspectrum.\" arXiv preprint arXiv:2402.01297 (2024).\n\nQUESTIONS:\nThe most important question would be if the authors could provide some experimental validations on these two claims.\n\nLIMITATIONS:\nAll assumptions are stated in the claims. However, as mentioned before, GUA and Eigenframework assumptions are too unrealistic and might provide wrong results.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-04"}

comments: SUMMARY:
This paper discusses the generalization ability of Gaussian kernel interpolation, an interesting topic. However, the authors sidestep the most challenging part by assuming the 'Gaussian universality ansatz,' which significantly simplifies the problem. With this assumption, their work essentially consists of elementary computations. Furthermore, their argument heavily relies on the earlier work by Lijia Zhou, James B. Simon, Gal Vardi, and Nathan Srebro, titled 'An agnostic view on the cost of overfitting in 473 (kernel) ridge regression,' an arXiv preprint with the identifier arXiv:2306.13185, published in 2023."

STRENGTHS:
A detailed calculation of the eigenvalue structure of the Gaussian kernel with various bandwidths has been performed. This could be valuable for future studies.

WEAKNESSES:
No significant contribution was made to the kernel interpolation problem itself. The major workhorse was developed in earlier work by Lijia Zhou, James B. Simon, Gal Vardi, and Nathan Srebro, as detailed in their paper titled 'An agnostic view on the cost of overfitting in 473 (kernel) ridge regression,' published as an arXiv preprint with the identifier arXiv:2306.13185 in 2023.

QUESTIONS:
Perhaps due to my lack of knowledge, could the author elaborate on their contributions beyond the detailed computation of the Gaussian kernel with varying bandwidths?

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper looks at the problem of kernel (ridgeless) regression with Gaussian kernel and data on a sphere. The paper studies two cases, first we fix the dimension $d$ and are allowed to send the number of data points $m \\to \\infty$ and the width of the kernel $\\tau_m \\to 0$, second, we are allowed to scale $d$. For both problems they study how a quantity known as the predicted risk scales. *It is important to note that the paper assumes that this is a good estimate of the true excess risk, which is not immediately evident and I think requires more justification (see weaknesses).*\n\nIn the first case they show that if $\\tau_m$ goes to zero too quickly, we have tempered overfitting for the *preditcted risk* and that it has a worse risk than the null estimator. If it goes to 0 too slowly, then we have catastrophic overfitting for the *predicted risk*, and that if it goes to zero at the appropriate rate, we have tempered overfitting again and that we may outperform the null estimator. The paper also presents some results about the case when $d$ scales as well.\n\nSTRENGTHS:\nThe paper uses the predicted risk as an estimate of the true risk. The prdicted risk is given by \n\n$$ \\tilde{R}(\\tilde{f}\\_{\\delta})  = \\mathcal{E}\\_\\delta \\left(\\sum\\_{i=1}^\\infty  (1-\\mathcal{L}\\_{i,\\delta})^2 \\beta_i^2 + \\sigma^2 \\right) $$\n\nWhere $\\beta_i$ are the coefficients of the target function in the eigenbasis of the kernel function and $\\mathcal{E}, \\mathcal{L}$ are coefficients based on eigenvalues and $\\sigma^2$ is the noise variance. *If we assume that this is a good estimate of the excess risk*. Then to understand the risk, we have reduced the problem to understanding how $\\mathcal{E}, \\mathcal{L}$ **and** $\\beta_i$ scale. \n\n**This paper studies how $\\mathcal{E}_0$ scales by providing some bounds on the eigenvalues of the Gaussian kernel function. This I think is the primary contribution of the paper.**\n\nHowever, there are some issues, see weaknesses.\n\nWEAKNESSES:\nThere are a few weaknesses. The current reject score is due to my concerns with the proofs. \n\n1) **Theorem 1**\n\nI have some concerns with the results in Theorem 1. Theorem 1 states that unless we meet some very specific conditions, the null predictor (i.e. the zero function) outperforms kernel ridge regression. This seems quite surprising to me. \n\nTaking a closer look at the proof, we see that proof is broken down into two Lemmas. One proves the upper bounds, the other lower bounds. However, Lemma 1, only deals with $\\mathcal{E}_0$. However, I think something has to be said about $\\sum\\_{i=1}^\\infty  (1-\\mathcal{L}\\_{i,\\delta})^2 \\beta_i^2$. \n\nIf I understand correctly, when we change $m$, we change the width, which changes the kernel function. However, since $\\lambda_i$ and $\\phi_i$ are the dependent on $K$. Hence $\\beta_i$ which are the coefficients of $f$ in this eigenbasis are dependent on $m$. Hence $\\sum\\_{i=1}^\\infty  (1-\\mathcal{L}\\_{i,\\delta})^2 \\beta_i^2$ has a subtle dependence on $m$ which I think needs to be accounted for. Because apriori it is not clear to me that this is bounded in $m$. That is, $\\sum_i \\beta_i^2(m) = S(m)$ could be unbounded in $m$. \n\nI did not go through the lower bound proofs as carefully, but I do think the the dependence of $\\beta_i$ on $m$ is addresses. Please correct me if I am wrong. \n\nHence I think either another assumption needs to be added (and if it is added a discussion of its reasonableness would be appreciated) or that terms needs to be explicity dealt with. \n\n2) Eigenlearding Framework and Predicted Risk\n\nIn general, I would prefer results showing that the predicted risk is a good estimate of the true risk. The paper on lines 116-131 attempt to this. However, the primary papers that have been cited [1,12,25] are all about **linear** regression and not **kernel** regression. Specifically, they all assume that $y_i = \\beta^T x_i + \\epsilon_i$ and that we are fitting a linear function. This is not quite the setting of this paper, as we have an infinite dimensional feature vector space that we are working in. Hence I do not think those papers are sufficient theoretical justification. \n\nMy other concern with this, is that the discussion from lines 98-111 **do not assume that** $y = f(x) + \\epsilon_i$ which I think is needed in [47]. The paper presents for general distributiuons on $\\mathcal{X}$ and $\\mathcal{Y}$. I do not think the eigenlearning framework captures the following situation $x_i  = z_i + \\epsilon_j$ and $y_i = f(z_i)$, but the paper here, as written, would claim this. See [A,B] for papers that address such a situation, [B] even has results on benign overfitting. \n\n\n[A] Sonthalia, Rishi and Raj Rao Nadakuditi. “Training Data Size Induced Double Descent For Denoising Feedforward Neural Networks and the Role of Training Noise.” Trans. Mach. Learn. Res. 2023 (2023): n. pag. \n\n[B] Kausik, C., Srivastava, K., & Sonthalia, R. (2023). Double Descent and Overfitting under Noisy Inputs and Distribution Shift for Linear Denoisers.TMLR 2024\n\n3) References \n\nI think the long list of references on lines 45-48 need to be changed. The paper should either further subcategorize them, or cite fewer papers. In the case that the authors decide to further subvategorize them. I think they are missing some references [A,B,C,D]. Additionally, some papers are cited twice, see 25, 26, and 23, 24. \n\n[C] Xiao, L., Hu, H., Misiakiewicz, T., Lu, Y., & Pennington, J. (2022). Precise Learning Curves and Higher-Order Scalings for Dot-product Kernel Regression. Neural Information Processing Systems.\n\n[D] Hu, H., Lu, Y.M., & Misiakiewicz, T. (2024). Asymptotics of Random Feature Regression Beyond the Linear Scaling Regime. ArXiv, abs/2403.08160.\n\nQUESTIONS:\nPlease see weaknesses\n\nLIMITATIONS:\nThe limitations are well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the overfitting behavior of Gaussian Kernel Ridge Regression (KRR) in high-dimensional settings, focusing on how the model performance is influenced by the choice of the bandwidth parameter and sample size.\n\nThey aimed to provide a more comprehensive picture of overfitting with Gaussian KRR by studying the overfitting behavior with varying bandwidth or with arbitrarily varying dimension, including sub-polynomially. In particular, they showed that for fixed dimension, even with varying bandwidth, the interpolation learning is never consistent and generally not better than the null predictor (either the test error tends to infinity or is finite but it is almost always not better than the null predictor). For increasing dimension, they gave an upper and lower bound on the test risk for any scaling of the dimension with sample size, which indicates in many cases whether the overfitting is catastrophic, tempered or benign. They showed the first example of sub-polynomially scaling dimension that achieves benign overfitting for the Gaussian kernel. Additionally, they showed that a class of dot-product kernels on the sphere is inconsistent when the dimension scales logarithmically with sample size.\n\nSTRENGTHS:\nThe presentation of the paper looks really in good order with adequate clarifications and rich literature reviews. The motivation is clear to me. The proof provided also looks correct in general.,\n\nThey clearly provided an upper and lower bound on the test risk for any scaling of the dimension with sample size, which provides a clear and comprehensive understanding on the issue of overfitting with Gaussian KRR, which is not restricted on particular regimes, while it also discovers new result for sub-polynomially varying dimension. I found the proof technique quite interesting and insightful.\n\nAlthough the contribution is not extremely outstanding or groundbreaking, the results completely reflect the effect of varying bandwidth or dimensionality in a very comprehensive way and it certainly facilitates the understanding of this issue and also provides a very insightful theoretical guidance of picking these crucial parameters.\n\nWEAKNESSES:\nThe first and a clear weakness would be lack of experiments. Since the results in the paper focus on the overfitting behavior with varying bandwidth or with arbitrarily varying dimension, including sub-polynomially, it should benefit a lot and is very instructive to study the real effects guided by the theories on the simulated or real-word datasets. \n\nThe results should be sensitivity to tuning parameters. The paper highlights the complexity of balancing these parameters but does not provide robust, practical guidelines for optimal tuning.\n\nDifferent data distributions and kernel eigenstructures result in non-uniform impacts of the tuning parameters, complicating the tuning process further. It is better the paper could consider more general date distributions in Assumption 2 or at least can provide some insights on how the results may vary given different date distributions.\n\nQUESTIONS:\nCan you explain the theoretical basis for the catastrophic overfitting observed with certain bandwidth scalings and the choice of kernels intuitively?\n\nWhat are the practical guidelines for choosing the bandwidth parameter in real-world applications since there is no experiment?\n\nHow does the proposed method perform across different types of data distributions besides Assumption 2?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors derive estimates of the population risk of the kernel (ridgeless) regression estimator for spherical data when either the kernel bandwidth or the dimension of the data depends\non the number of training points $m$. To do so, authors rely on the eigenframework and prior work, which provides (under assumptions on the distributions of the random variable $\\phi(X)$ where $\\phi$ is the kernel feature map and $X$ is the input random variable) formulas for the population risk of the kernel ridge regression estimator as a function of the eigenvalues of the kernel integral operator. Using these formulas, authors then derive nonasymptotic estimates of these quantities in order to establish the limit value of the risk, as well as its dependency w.r.t $m$ and $d$. They obtain that\n\n- for a fixed dimension, overfitting is always harmfull (tempered or catasrophic) unless the bandwidth scales in precisely in $m^{-1/d-1}$ in which case overfitting is shown to be tempered for large enough noise in the data (Theorem 1)\n- for a fixed bandwidth, but varying dimensionthey derivelower and upper bound on the population risk. This allows them to\n\t- recover known results stating that when the dimension size increases polynomially with sample size, overfitting is benign or tempered\n\t- show that when the dependency is logarithmic, overfitting is not benign (under additional assumptions verified for the gaussian kernel)\n\t- show that when the dependency is subpolynomial $\\exp(\\sqrt{m)$ scaling, overfitting can be benign\n\nSTRENGTHS:\nThe paper is clear, with a good effort spent on litterature review and comparison with prior work.\n\nThe main technical contributions seem to be bounds on the spectra of dot-product kernels on the sphere, which if novel (I am not an expert of this domain) could be of independent interest and are welcomed.\n\nWEAKNESSES:\n- This paper investigates the behavior of **ridgeless** (e.g. no RKHS norm penalty) kernel estimators, something which should be made clearer.  For example, kernel ridge regression is part of the paper title, whereas the paper does not study kernel ridge regression.\n- The eigenframework which they use rely on a number of assumptions to be valid.\n\nQUESTIONS:\nIn the analysis, the data is assumed to be on the sphere. Can the authors comment on how they expect the results to change for data\n  distributed on the whole of $\\mathbb  R^d$?\n\n\nTypos:\n\nl 294\nl 298\n\nLIMITATIONS:\nMentioned in weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}