Title: Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension
Venue: NeurIPS 2023
Accepted: yes
Keywords: benign overfitting

kernels

neural tangent kernel

consistency

learning theory
Abstract: The success of over-parameterized neural networks trained to near-zero training error has caused great interest in the phenomenon of benign overfitting, where estimators are statistically consistent even though they interpolate noisy training data. While benign overfitting in fixed dimension has been established for some learning methods, current literature suggests that for regression with typical kernel methods and wide neural networks, benign overfitting requires a high-dimensional setting, where the dimension grows with the sample size. In this paper, we show that the smoothness of the estimators, and not the dimension, is the key: benign overfitting is possible if and only if the estimator's derivatives are large enough. We generalize existing inconsistency results to non-interpolating models and more kernels to show that benign overfitting with moderate derivatives is impossible in fixed dimension. Conversely, we show that benign overfitting is possible for regression with a sequence of spiky-smooth kernels with large derivatives. Using neural tangent kernels, we translate our results to wide neural networks. We prove that while infinite-width networks do not overfit benignly with the ReLU activation, this can be fixed by adding small high-frequency fluctuations to the activation function. Our experiments verify that such neural networks, while overfitting, can indeed generalize well even on low-dimensional data sets.

Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension

Abstract

The success of over-parameterized neural networks trained to near-zero training error has caused great interest in the phenomenon of benign overfitting, where estimators are statistically consistent even though they interpolate noisy training data. While benign overfitting in fixed dimension has been established for some learning methods, current literature suggests that for regression with typical kernel methods and wide neural networks, benign overfitting requires a high-dimensional setting where the dimension grows with the sample size. In this paper, we show that the smoothness of the estimators, and not the dimension, is the key: benign overfitting is possible if and only if the estimator's derivatives are large enough. We generalize existing inconsistency results to non-interpolating models and more kernels to show that benign overfitting with moderate derivatives is impossible in fixed dimension. Conversely, we show that rate-optimal benign overfitting is possible for regression with a sequence of spiky-smooth kernels with large derivatives. Using neural tangent kernels, we translate our results to wide neural networks. We prove that while infinite-width networks do not overfit benignly with the ReLU activation, this can be fixed by adding small high-frequency fluctuations to the activation function. Our experiments verify that such neural networks, while overfitting, can indeed generalize well even on low-dimensional data sets.

1 Introduction

While neural networks have shown great practical success, our theoretical understanding of their generalization properties is still limited. A promising line of work considers the phenomenon of benign overfitting, where researchers try to understand when and how models that interpolate noisy training data can generalize (Zhang et al., 2021, Belkin et al., 2018, 2019). In the high-dimensional regime, where the dimension grows with the number of sample points, consistency of minimum-norm interpolants has been established for linear models and kernel regression (Hastie et al., 2022, Bartlett et al., 2020, Liang and Rakhlin, 2020, Bartlett et al., 2021). In fixed dimension, minimum-norm interpolation with standard kernels is inconsistent (Rakhlin and Zhai, 2019, Buchholz, 2022).

In this paper, we shed a differentiated light on benign overfitting with kernels and neural networks. We argue that the dimension-dependent perspective does not capture the full picture of benign overfitting. In particular, we show that harmless interpolation with kernel methods and neural networks is possible, even in small fixed dimension, with adequately designed kernels and activation functions. The key is to properly design estimators of the form 'signal+spike'. While minimum-norm criteria have widely been considered a useful inductive bias, we demonstrate that designing unusual norms can resolve the shortcomings of standard norms. For wide neural networks, harmless interpolation can be

realized by adding tiny fluctuations to the activation function. Such networks do not require explicit regularization and can simply be trained to overfit (Figure 1).

-

π

-

π

2

0

π

2

π

Angle (radians)

-

2

-

1

0

1

Prediction

a.

Laplace

Spiky-smooth

-

π

-

π

2

0

π

2

π

Angle (radians)

-

2

-

1

0

1

Prediction

b.

ReLU

Spiky-smooth

Signal network

0

10

0

10

1

10

2

10

3

Epochs

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

RMSE

c.

ReLU

Spiky-smooth

On a technical level, we additionally prove that overfitting in kernel regression can only be consistent if the estimators have large derivatives. Using neural tangent kernels or neural network Gaussian process kernels, we can translate our results from kernel regression to the world of neural networks (Neal, 1996, Jacot et al., 2018). In particular, our results enable the design of activation functions that induce benign overfitting in fixed dimension: the spikes in kernels can be translated into infinitesimal fluctuations that can be added to an activation function to achieve harmless interpolation with neural networks. Such small high frequency oscillations can fit noisy observations without affecting the smooth component too much. Training finite neural networks with gradient descent shows that spiky-smooth activation functions can indeed achieve good generalization even when interpolating small, low-dimensional data sets (Figure 1 b,c).

Thanks to new technical contributions, our inconsistency results significantly extend existing ones. We use a novel noise concentration argument (Lemma D.6) to generalize existing inconsistency results on minimum-norm interpolants to the much more realistic regime of overfitting estimators with comparable Sobolev norm scaling, which includes training via gradient flow and gradient descent with 'late stopping' as well as low levels of ridge regularization. Moreover, a novel connection to eigenvalue concentration results for kernel matrices (Proposition 5) allows us to relax the smoothness assumption and to treat heteroscedastic noise in Theorem 6. Lastly, our Lemma E.1 translates inconsistency results from bounded open subsets of R d to the sphere S d , which leads to results for the neural tangent kernel and neural network Gaussian processes.

2 Setup and prerequisites

General approach. We consider a general regression problem on R d with an arbitrary, fixed dimension d and analyze kernel-based approaches to solve this problem: kernel ridge regression, kernel gradient flow and gradient descent, minimum-norm interpolation, and more generally, overfitting norm-bounded estimators. We then translate our results to neural networks via the neural network Gaussian process and the neural tangent kernel. Let us now introduce the formal framework.

Notation. We denote scalars by lowercase letters x , vectors by bold lowercase letters x and matrices by bold uppercase letters X . We denote the eigenvalues of A as λ 1 ( A ) ≥ . . . ≥ λ n ( A ) and the Moore-Penrose pseudo-inverse by A + . We say that a probability distribution P has lower and upper bounded density if its density p satisfies 0 < c < p ( x ) < C for suitable constants c, C and all x on a given domain.

Regression setup. We consider a data set D = (( x 1 , y 1 ) , . . . , ( x n , y n )) ∈ ( R d × R ) n with i.i.d. samples ( x i , y i ) ∼ P , written as D ∼ P n , where P is a probability distribution on R d × R . We define X := ( x 1 , . . . , x n ) and y := ( y 1 , . . . , y n ) ⊤ ∈ R n . Random variables ( x , y ) ∼ P denote

test points independent of D , and P X denotes the probability distribution of x . The (least squares) empirical risk R D and population risk R P of a function f : R d → R are defined as

We assume Var( y | x ) < ∞ for all x . Then, R P is minimized by the target function f ∗ P ( x ) = E [ y | x ] , and the excess risk of a function f is given by

We call a data-dependent estimator f D consistent for P if its excess risk converges to 0 in probability, that is, for all ε > 0 , lim n →∞ P n ( D ∈ ( R d × R ) n | R P ( f D ) -R P ( f ∗ P ) ≥ ε ) = 0 . We call f D consistent in expectation for P if lim n →∞ E D R P ( f D ) -R P ( f ∗ P ) = 0 . We call f D universally consistent if is it consistent for all Borel probability measures P on R d × R .

Solutions by kernel regression. Recall that a kernel k induces a reproducing kernel Hilbert space H k , abbreviated RKHS (more details in Appendix B). For f ∈ H k , we consider the objective with regularization parameter ρ ≥ 0 . Denote by f t,ρ the solution to this problem that is obtained by optimizing on L ρ in H k with gradient flow until time t ∈ [0 , ∞ ] , using fixed a regularization constant ρ > 0 , and initializing at f = 0 ∈ H k . We show in Appendix C.1 that it is given as

where k ( x , X ) denotes the row vector ( k ( x , x i )) i ∈ [ n ] and k ( X , X ) = ( k ( x i , x j )) i,j ∈ [ n ] the kernel matrix. f t,ρ elegantly subsumes several popular kernel regression estimators as special cases: (i) classical kernel ridge regression for t → ∞ , (ii) gradient flow on the unregularized objective for ρ ↘ 0 , and (iii) kernel 'ridgeless' regression f ∞ , 0 ( x ) = k ( x , X ) k ( X , X ) + y in the joint limit of ρ → 0 and t → ∞ . If k ( X , X ) is invertible, f ∞ , 0 is the interpolating function f ∈ H k with the smallest H k -norm.

From kernels to neural networks: the neural tangent kernel (NTK) and the neural network Gaussian process (NNGP) . Denote the output of a NN with parameters θ on input x by f θ ( x ) . It is known that for suitable random initializations θ 0 , in the infinite-width limit the random initial function f θ 0 converges in distribution to a Gaussian Process with the so-called Neural Network Gaussian Process (NNGP) kernel (Neal, 1996, Lee et al., 2018, Matthews et al., 2018). In Bayesian inference, the posterior mean function is then of the form f ∞ ,ρ . With minor modifications (Arora et al., 2019, Zhang et al., 2020), training infinitely wide NNs with gradient flow corresponds to learning the function f t, 0 with the neural tangent kernel (NTK) (Jacot et al., 2018, Lee et al., 2019). If only the last layer is trained, the NNGP kernel should be used instead (Daniely et al., 2016). For ReLU activation functions, the RKHS of the infinite-width NNGP and NTK on the sphere S d is typically a Sobolev space (Bietti and Bach, 2021, Chen and Xu, 2021), see Appendix B.4. Using other parametrizations induces feature learning infinite-width limits for neural networks (Yang and Hu, 2021); an analysis of such neural network algorithms is left for future work.

3 Related work

We here provide a short summary of related work. A more detailed account is provided in Appendix A.

Kernel regression. With appropriate regularization, kernel ridge regularization with typical universal kernels like the Gauss, Matérn, and Laplace kernels is universally consistent (Steinwart and Christmann, 2008, Chapter 9). Optimal rates in Sobolev RKHS can also be achieved using cross-validation of the regularization ρ (Steinwart et al., 2009) or early stopping rules (Yao et al., 2007, Raskutti et al., 2014, Wei et al., 2017). The above kernels as well as NTKs and NNGPs of standard fully-connected neural networks are rotationally invariant. In the high-dimensional regime, the class of functions that is learnable with rotation-invariant kernels is quite limited (Donhauser et al., 2021, Ghorbani et al., 2021, Liang et al., 2020).

Inconsistency results. Besides Rakhlin and Zhai (2019) and Buchholz (2022), Beaglehole et al. (2023) derive inconsistency results for ridgeless kernel regression given assumptions on the spectral tail in the Fourier basis, and contemporaneously propose a special case of our spiky-smooth kernel sequence to mimic kernel ridge regression without providing any quantitative statements. Li et al. (2023) show that polynomial convergence is impossible for common kernels including ReLU NTKs. Mallinar et al. (2022) conjecture inconsistency for interpolation with ReLU NTKs based on their semirigorous result , which essentially assumes that the eigenfunctions can be replaced by structureless Gaussian random variables. Lai et al. (2023) show an inconsistency-type result for overfitting twolayer ReLU NNs with d = 1 , but for fixed inputs X . They also note that an earlier inconsistency result by Hu et al. (2021) relies on an unproven result. Mücke and Steinwart (2019) show that global minima of NNs can overfit both benignly and harmfully, but their result does not apply to gradient descent training. Overfitting with typical linear models around the interpolation peak is inconsistent (Ghosh and Belkin, 2022, Holzmüller, 2021).

Classification. For binary classification, benign overfitting is a more generic phenomenon than for regression (Muthukumar et al., 2021, Shamir, 2022), and consistency has been shown under linear separability assumptions (Montanari et al., 2019, Chatterji and Long, 2021, Frei et al., 2022), through complexity bounds for reference classes (Cao and Gu, 2019, Chen et al., 2021) or as long as the total variation distance of the class conditionals is sufficiently large and f ∗ ( x ) = E [ y | x ] lies in the RKHS with bounded norm (Liang and Recht, 2023). Chapter 8 of Steinwart and Christmann (2008) discusses how the overlap of the two classes may influence learning rates under positive regularization.

4 Inconsistency of overfitting with common kernel estimators

We consider a regression problem on R d in arbitrary, fixed dimension d that is solved by kernel regression. In this section, we derive several new results, stating that overfitting estimators with moderate Sobolev norm are inconsistent, in a variety of settings. In the next section, we establish the other direction: overfitting estimators can be consistent when we adapt the norm that is minimized.

4.1 Beyond minimum-norm interpolants: general overfitting estimators with bounded norm

Existing generalization bounds often consider the perfect minimum norm interpolant. This is a rather theoretical construction; estimators obtained by training with gradient descent algorithms merely overfit and, in the best case, approximate interpolants with small norm. In this section, we extend existing bounds to arbitrary overfitting estimators whose norm does not grow faster than the minimum norm that would be required to interpolate the training data. Before we can state the theorem, we need to establish some technical assumptions.

Assumptions on the data generating process. The following assumptions (as in Buchholz (2022)) allow for quite general domains and distributions. They are standard in nonparametric statistics.

(D1) Let P X be a distribution on a bounded open Lipschitz domain Ω ⊆ R d with lower and upper bounded Lebesgue density. Consider data sets D = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } , where x i ∼ P X i.i.d. and y i = f ∗ ( x i )+ ε i , where ε i is i.i.d. Gaussian noise with positive variance σ 2 > 0 and f ∗ ∈ C ∞ c (Ω) \ { 0 } denotes a smooth function with compact support.

Assumptions on the kernel. Our assumption on the kernel is that its RKHS is equivalent to a Sobolev space. For integers s ∈ N , the norm of a Sobolev space H s (Ω) can be defined as

where D α denotes partial derivatives in multi-index notation for α . It measures the magnitude of derivatives up to some order s . For general s > 0 , H s (Ω) is (equivalent to) an RKHS if and only if s > d/ 2 . For example, Laplace and Matérn kernels (Kanagawa et al., 2018, Example 2.6) have Sobolev RKHSs. The RKHS of the Gaussian kernel H Gauss is contained in every Sobolev space, H Gauss ⊊ H s for all s ≥ 0 (Steinwart and Christmann, 2008, Corollary 4.36). Due to its smoothness, the Gaussian kernel is potentially even more prone to harmful overfitting than Sobolev kernels (Mallinar et al., 2022). We make the following assumption on the kernel:

(K) Let k be a positive definite kernel function whose RKHS H k is equivalent to the Sobolev space H s for some s ∈ ( d 2 , 3 d 4 ] .

Now we are ready to state the main result of this section:

Theorem 1 ( Overfitting estimators with small norms are inconsistent ) . Let assumptions (D1) and (K) hold. Let c fit ∈ (0 , 1] and C norm > 0 . Then, there exist c > 0 and n 0 ∈ N such that the following holds for all n ≥ n 0 with probability 1 -O (1 /n ) over the draw of the data set D with n samples: Every function f ∈ H k that satisfies the follwing two conditions

(N) ∥ f ∥ H k ≤ C norm ∥ f ∞ , 0 ∥ H k (norm comparable to minimum-norm interpolant (1) ),

In words: In fixed dimension d , every differentiable function f that overfits the training data and is not much 'spikier' than the minimum RKHS-norm interpolant is inconsistent!

Proof idea. Our proof follows a similar approach as Rakhlin and Zhai (2019), Buchholz (2022), and also holds for kernels with adaptive bandwidths. For small bandwidths, ∥ f ∞ , 0 ∥ L 2 ( P X ) ≪∥ f ∗ ∥ L 2 ( P X ) because f ∞ , 0 decays to 0 between the training points, which shows that purely 'spiky' estimators are inconsistent. In this case, the lower bound is independent of σ 2 . For all other bandwidths, interpolating Θ( n ) many noisy labels y i incurs Θ(1) error in an area of volume Ω(1 /n ) around Θ( n ) data points with high probability, which accumulates to a total error Ω(1) . Our observation is that the same logic holds when overfitting by a constant fraction. Formally, we show that f ∗ and f must then be separated by a constant on a constant fraction of training points, with high probability, by using the fact that a constant fraction of the total noise cannot concentrate on less than Θ( n ) noise variables, with high probability (Lemma D.6). The full proof can be found in Appendix D.

Assumption (O) is necessary in Theorem 1, because optimally regularized kernel ridge regression fulfills all other assumptions of Theorem 1 while achieving consistency with minimax optimal convergence rates (see Section 3). The necessity of Assumption (N) is demonstrated by Section 5.

The following proposition establishes that Theorem 1 covers the entire overfitting regime of the popular (regularized) gradient flow estimators f t,ρ for all times t ∈ [0 , ∞ ] and any regularization ρ ≥ 0 . The proof in Appendix C.2 also covers gradient descent.

Remark 3 ( Dimension dependency ) . Some works argue that for specific sequences of kernels ( k d ) d ∈ N , the constant c in Theorem 1 decreases with increasing dimension d (Liang et al., 2020, Liang and Rakhlin, 2020, Mallinar et al., 2022). In Theorem 1, if the equivalence constants in Assumption (K) are uniformly bounded in d , the behavior in d might still depend on the definition of the Sobolev norms. Overall, similar to Rakhlin and Zhai (2019) and Buchholz (2022), our proof techniques do not allow to easily obtain a dependence on d . ◀

Proposition 2 ( Popular estimators fulfill the norm bound (N) ) . For arbitrary t ∈ [0 , ∞ ] and ρ ∈ [0 , ∞ ) , f t,ρ as defined in (1) fulfills Assumption (N) with C norm = 1 .

4.2 Inconsistency of overfitting with neural kernels

We would now like to apply the above results to neural kernels, which would allow us to translate our inconsistency results from the kernel domain to neural networks. However, to achieve this, we need to take one more technical hurdle: the equivalence results for NTKs and NNGPs only hold for probability distributions on the sphere S d (detailed summary in Appendix B.4). Lemma E.1 provides the missing technical link: It establishes a smooth correspondence between the respective kernels, Sobolev spaces, and probability distributions. The inconsistency of overfitting with (deep) ReLU NTKs and NNGP kernels then immediately follows from adapting Theorem 1 via Lemma E.1.

Theorem 4 ( Overfitting with neural network kernels in fixed dimension is inconsistent ) . Let c ∈ (0 , 1) , and let P be a probability distribution with lower and upper bounded Lebesgue density on an arbitrary spherical cap T := { x ∈ S d | x d +1 < v } ⊆ S d , v ∈ ( -1 , 1) . Let k either be

(ii) the fully-connected ReLU NNGP kernel without biases of any fixed depth L ≥ 3 , and d ≥ 6 .

(i) the fully-connected ReLU NTK with 0 -initialized biases of any fixed depth L ≥ 2 , and d ≥ 2 , or

(O) 1 n ∑ n i =1 ( f ( x i ) -y i ) 2 ≤ (1 -c fit ) · σ 2 (training error of f is below Bayes risk)

has an excess risk that satisfies

Then, if f t,ρ fulfills Assumption ( O ) with probability at least c over the draw of the data set D , f t,ρ is inconsistent for P .

Theorem 4 also holds for more general estimators as in Theorem 1, cf. the proof in Appendix E.

Mallinar et al. (2022) already observed empirically that overfitting common network architectures yields suboptimal generalization performance on large data sets in fixed dimension. Theorem 4 now provides a rigorous proof for this phenomenon since sufficiently wide trained neural networks and the corresponding NTKs have a similar generalization behavior (e.g. (Arora et al., 2019, Theorem 3.2)).

4.3 Relaxing smoothness and noise assumptions via spectral concentration bounds

In this section, we consider a different approach to derive lower bounds for the generalization error of overfitting kernel regression: through concentration results for the eigenvalues of kernel matrices. On a high level, we obtain similar results as in the last section. The novelty of this section is on the technical side, and we suggest that non-technical readers skip this section in their first reading.

We define the convolution kernel of a given kernel k as k ∗ ( x , x ′ ) := ∫ k ( x , x ′′ ) k ( x ′′ , x ′ ) d P X ( x ′′ ) , which is possible whenever k ( x , · ) ∈ L 2 ( P X ) for all x . The latter condition is satisfied for bounded kernels. Our starting point is the following new lower bound:

Proposition 5 ( Spectral lower bound ) . Assume that the kernel matrix k ( X , X ) is almost surely positive definite, and that Var( y | x ) ≥ σ 2 for P X -almost all x . Then, the expected excess risk satisfies

Using concentration inequalities for kernel matrices and the relation between the integral operators of k and k ∗ , it can be seen that for t = ∞ and ρ = 0 , every term in the sum in Eq. (3) should converge to 1 as n →∞ . However, since the number of terms in the sum increases with n and the convergence may not be uniform, this is not sufficient to show inconsistency in expectation. Instead, relative concentration bounds that are even stronger than the ones by Valdivia (2018) would be required to show inconsistency in expectation. However, by combining multiple weaker bounds and further arguments on kernel equivalences, we can still show inconsistency in expectation for a class of dot-product kernels on the sphere, including certain NTK and NNGP kernels (Appendix B.4):

Theorem 6 ( Inconsistency for Sobolev dot-product kernels on the sphere ) . Let k be a dot-product kernel on S d , i.e., a kernel of the form k ( x , x ′ ) = κ ( ⟨ x , x ′ ⟩ ) , such that its RKHS H k is equivalent to a Sobolev space H s ( S d ) , s > d/ 2 . Moreover, let P be a distribution on S d × R such that P X has a lower and upper bounded density w.r.t. the uniform distribution U ( S d ) , and such that Var( y | x ) ≥ σ 2 > 0 for P X -almost all x ∈ S d . Then, for every C > 0 , there exists c > 0 independent of σ 2 such that for all n ≥ 1 , t ∈ ( C -1 n 2 s/d , ∞ ] , and ρ ∈ [0 , Cn -2 s/d ) , the expected excess risk satisfies

The assumptions of Theorem 6 and Theorem 4 differ in several ways. Theorem 6 applies to arbitrarily high smoothness s and therefore to ReLU NTKs and NNGPs in arbitrary dimension d . Moreover, it applies to distributions on the whole sphere and allows more general noise distributions. On the flip side, it only shows inconsistency in expectation, which we believe could be extended to inconsistency for Gaussian noise. Moreover, it only applies to functions of the form f t,ρ but provides an explicit bound on t and ρ to get inconsistency. For t = ∞ , the bound ρ = O ( n -2 s/d ) appears to be tight, as larger ρ yield consistency for comparable Sobolev kernels on R d (Steinwart et al., 2009, Corollary 3).

We only prove Theorem 6 for dot-product kernels on the sphere since we can show for these kernels that H k ∗ is equivalent to a Sobolev space (Lemma F.13), while this is not true for open domains Ω (Schaback, 2018). However, an improved understanding of H k ∗ for such Ω could potentially allow to extend our proof to the non-spherical case.

The spectral lower bounds in Theorem F.2 show that our approach can directly benefit from developing better kernel matrix concentration inequalities. Conversely, the investigation of consistent kernel interpolation might provide information about where such concentration inequalities do not hold.

5 Consistency via spiky-smooth estimators - even in fixed dimension

In Section 4, we have seen that when common kernel estimators overfit, they are inconsistent for many kernels and a wide variety of distributions. We now design consistent interpolating kernel estimators. The key is to violate Assumption (N) for every fixed Sobolev RKHS norm ∥ · ∥ H k and introduce an inductive bias towards learning spiky-smooth functions.

5.1 Almost universal consistency of spiky-smooth ridgeless kernel regression

In high dimensional regimes (where the dimension d is supposed to grow with the number of data points), benign overfitting of linear and kernel regression has been understood by an additive decomposition of the minimum-norm interpolant into a smooth regularized component that is responsible for good generalization, and a spiky component that interpolates the noisy data points while not harming generalization (Bartlett et al., 2021). This inspires us to enforce such a decomposition in arbitrary fixed dimension by adding a sharp kernel spike ρ ˇ k γ n to a common kernel ˜ k . In this way, we can still generate any Sobolev RKHS (see Appendix G.2).

Definition 7 ( Spiky-smooth kernel ) . Let ˜ k denote any universal kernel function on R d . Wecall it the smooth component. Consider a second, translation invariant kernel ˇ k γ of the form k γ ( x , y ) = q ( x -y γ ) , for some function q : R d → R . We call it the spiky component. Then we define the ρ -regularized spiky-smooth kernel with spike bandwidth γ as

We now show that the minimum-norm interpolant of the spikysmooth kernel sequence with properly chosen ρ n , γ n → 0 is consistent for a large class of distributions, on a space with fixed (possibly small) dimension d . We establish our result under the following assumption (as in Mücke and Steinwart (2019)), which is weaker than our previous Assumption (D1).

-

0

.

1

0

.

0

0

.

1

0

.

9

1

.

0

1

.

1

{

ρ

{

γ

Laplace

Spiky-smooth

(D2) There exists a constant β X > 0 and a continuous function ϕ : [0 , ∞ ) → [0 , 1] with ϕ (0) = 0 such that the data generating probability distribution satisfies P X ( B t ( x )) ≤ ϕ ( t ) = O ( t β X ) for all x ∈ Ω and all t ≥ 0 (here B t ( x ) denotes the Euclidean ball of radius t around x ).

Theorem 8 ( Consistency of spiky-smooth ridgeless kernel regression ) . Assume that the training set D consists of n i.i.d. pairs ( x , y ) ∼ P such that the marginal P X fulfills (D2) and E y 2 < ∞ . Let the kernel components satisfy:

˜ k is a universal kernel, and ρ n → 0 and nρ 4 n →∞ .

Then the minimum-norm interpolant of the ρ n -regularized spiky-smooth kernel sequence k n := k ρ n ,γ n is consistent for P .

ˇ k γ n denotes the Laplace kernel with a sequence of positive bandwidths ( γ n ) fulfilling γ n ≤ n -2+ α d ( ( 9 4 + α 2 ) ln n ) -1 , where α > 0 arbitrary.

Remark 9 ( Benign overfitting with optimal convergence rates ) . Suppose that we have a Sobolev target function f ∗ ∈ H s ∗ (Ω) \ { 0 } , that the noise satisfies a moment condition and that P X has an upper- and lower-bounded density on a Lipschitz domain or the sphere. Then, we show in Theorem G.5 that, by using smooth components ˜ k whose RKHS is equivalent to a Sobolev space H s s > max( s , d/ 2) ˇ k

, ∗ , and choosing the spike components γ n as in Theorem 8, the minimum-norm interpolant of k n := k ρ n ,γ n achieves the convergence rate n -s ∗ ( s ∗ + d/ 2) log 2 ( n ) when choosing the quasi-regularization ρ n properly. Moreover, for s ∗ > d/ 2 , this rate is known to be optimal up to the factor log 2 ( n ) (Remark G.6). Since optimal rates can be achieved both with optimal regularization and with interpolation, our results show that in Sobolev RKHSs, overfitting is neither intrinsically helpful nor harmful for generalization with the right choice of kernel function. ◀

Proof idea. With sharp spikes γ → 0 , it holds that ˇ k γ ( X , X ) ≈ I n , with high probability. Hence, ridgeless kernel regression with the spiky-smooth kernel interpolates the training set while approximating kernel ridge regression with the smooth component ˜ k and regularization ρ .

The theorem even holds under much weaker assumptions on the decay behavior of the spike component ˇ k γ n , including Gaussian and Matérn kernels. The full version of the theorem and its proof can be found in Appendix G. It also applies to kernels and distributions on the sphere S d .

Remark 10 ( Interplay between smoothness and dimensionality ) . Irrespective of the dimension d , we achieve benign overfitting with estimators in RKHS of arbitrary degrees of smoothness. With increasing d , for the Laplace kernel the spike bandwidth is allowed to be chosen as γ n = Ω( n -(2+ α ) /d ) , α > 0 , for covariate distributions with upper bounded Lebesgue density (see Remark G.2). Hence the magnitude of derivatives of the spikes is allowed to scale less aggressively with increasing dimension. ◀

5.2 From spiky-smooth kernels to spiky-smooth activation functions

So far, our discussion revolved around the properties of kernels and whether they lead to estimators that are consistent. We now turn our attention to the neural network side. The big question is whether it is possible to specifically design activation functions that enable benign overfitting in fixed, possibly small dimension. We will see that the answer is yes: similarly to adding sharp spikes to a kernel, we add tiny fluctuations to the activation function. Concretely, we exploit (Simon et al., 2022, Theorem 3.1). It states that any dot-product kernel on the sphere that is a dot-product kernel in every dimension d can be written as an NNGP kernel or an NTK of two-layer fully-connected networks with a specifically chosen activation function. Further details can be found in Appendix H.

Theorem 11 ( Connecting kernels and activation functions (Simon et al., 2022)) . Let κ : [ -1 , 1] → R be a function such that k d : S d × S d → R , k d ( x , x ′ ) = κ ( ⟨ x , x ′ ⟩ ) is a kernel for every d ≥ 1 . Then, there exist b i ≥ 0 with ∑ ∞ i =0 b i < ∞ such that κ ( t ) = ∑ ∞ i =0 b i t i , and for any choice of signs ( s i ) i ∈ N 0 ⊆ {-1 , +1 } , the kernel k d can be realized as the NNGP kernel or NTK of a two-layer fully-connected network without biases and with activation function

Here, h i denotes the i -th Probabilist's Hermite polynomial normalized such that ∥ h i ∥ L 2 ( N (0 , 1)) = 1 .

The following proposition justifies the approach of adding spikes ρ 1 / 2 ϕ ˇ k γ to an activation function to enable harmless interpolation with wide neural networks. Here we state the result for the case of the NTK; an analogous result holds for induced NNGP activation functions.

Proposition 12 ( Additive decomposition of spiky-smooth activation functions ) . Fix ˜ γ, ρ > 0 arbitrary. Let k = ˜ k + ρ ˇ k γ denote the spiky-smooth kernel where ˜ k and ˇ k γ are Gaussian kernels of bandwidth ˜ γ and γ , respectively. Assume that we choose signs { s i } i ∈ N and then the activation functions ϕ k NTK , ϕ ˜ k NTK and ϕ ˇ k γ NTK as in Theorem 11. Then, for γ > 0 small enough, it holds that

Proof idea. When the spikes are sharp enough ( γ small enough), the smooth and the spiky component of the activation function are approximately orthogonal in L 2 ( N (0 , 1)) (Figure 3c), so that the spikysmooth activation function can be approximately additively decomposed into the smooth activation component ϕ ˜ k and the spike component ϕ ˇ k responsible for interpolation.

To motivate why the added spike functions ρ 1 / 2 ϕ ˇ k γ should have small amplitudes, observe that Gaussian activation components ϕ ˇ k γ satisfy

Hence, the average amplitude of NNGP spike activation components ρ 1 / 2 ϕ ˇ k γ does not depend on γ , while the average amplitude of NTK spike components decays to 0 with γ → 0 . Since consistency requires the quasi-regularization ρ → 0 , the spiky component of induced NTK as well as NNGP activation functions should vanish for large data sets n →∞ to achieve consistency.

-

2

-

1

0

1

2

-

0

.

4

-

0

.

2

0

.

0

0

.

2

0

.

4

a.

γ

= 0.2

γ

= 0.1

γ

= 0.05

-

3

-

2

-

1

0

1

2

3

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

b.

0

25

50

75

100

125

150

175

Index

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

0

.

10

Coefficient value

c.

γ

= 0.2

γ

= 0.07

γ

= 0.04

γ

= 0.02

6 Experiments

Now we explore how appropriate spiky-smooth activation functions might look like and whether they indeed enable harmless interpolation for trained networks of finite width on finite data sets. Further experimental results are reported in Appendix I.

6.1 What do common activation functions lack in order to achieve harmless interpolation?

To understand which properties we have to introduce into activation functions to enable harmless interpolation, we plot NTK spike components ϕ ˇ k γ induced by the Gaussian kernel (Figure 3a,b) as well as their Hermite series coefficients (Figure 3c). Remarkably, the spike components ϕ ˇ k γ approximately correspond to a shifted, high-frequency sin -curve, when choosing the signs s i in (5) to alternate every second i , that is s i = +1 iff ⌊ i/ 2 ⌋ even (Figure 3a). Proposition H.1 shows that the NNGP activation functions correspond to the fluctuation function

where the last equation follows from the trigonometric addition theorem. For small bandwidths γ , the NTK activation functions are increasingly well approximated (Appendix I.6) by

With decreasing bandwidth γ → 0 the frequency increases, while the amplitude decreases for the NTK and remains constant for the NNGP (see Eq. (6)). Plotting equivalent spike components ϕ ˇ k γ with different choices of the signs s i (Figure 3b and Appendix I.5) suggests that harmless interpolation requires activation functions that contain small high-frequency oscillations or that explode at large | x | , which only affects few neurons. The Hermite series expansion of suitable activation functions should contain non-negligible weight spread across high-order coefficients (Figure 3c). While Simon et al. (2022) already truncate the Hermite series of induced activation functions at order 5 , Figure 3c shows that an accurate approximation of spiky-smooth activation functions requires the truncation index to be larger than 2 /γ . Only a careful implementation allows us to capture the high-order fluctuations in the Hermite series of the spiky activation functions. Our implementation can be found at https://github.com/moritzhaas/mind-the-spikes .

6.2 Training neural networks to achieve harmless interpolation in low dimension

In Figure 1, we plot the results of (a) ridgeless kernel regression and (b) trained 2-layer neural networks with standard choices of kernels and activation functions (blue) as well as our spiky-smooth alternatives (orange). We trained on 15 points sampled i.i.d. from x = ( x 1 , x 2 ) ∼ U ( S 1 ) and y = x 1 + ε with ε ∼ N (0 , 0 . 25) . The figure shows that both the Laplace kernel and standard ReLU networks interpolate the training data too smoothly in low dimension, and do not generalize well. However, our spiky-smooth kernel and neural networks with spiky-smooth activation functions achieve close to optimal generalization while interpolating the training data with sharp spikes.

We achieve this by using the adjusted activation function with high-frequency oscillations x ↦→ ReLU( x ) + ω NTK ( x ; 1 5000 ) as defined in Eq. (8). With this choice, we avoid activation functions with exploding behavior, which would induce exploding gradients. Other choices of amplitude and frequency in Eq. (8) perform worse. To bring our neural networks close to the kernel regime, we use the neural tangent parameterization (Jacot et al., 2018) and make the networks very wide (20000 hidden neurons). To ensure that the initial function is identically zero, we use the antisymmetric initialization trick (Zhang et al., 2020). Over the course of training (Figure 1c), the standard ReLU network exhibits harmful overfitting, whereas the NN with a spiky-smooth activation function quickly interpolates the training set with nearly optimal generalization. Training details and hyperparameter choices can be found in Appendix I.1. Although the high-frequency oscillations perturb the gradients, the NN with spiky smooth activation has a stable training trajectory using gradient descent with a large learning rate of 0 . 4 or stochastic gradient descent with a learning rate of 0 . 04 . Since our activation function is the sum of two terms, we can additively decompose the network into its ReLU-component and its ω NTK -component. Figure 1b and Appendix I.2 demonstrate that our interpretation of the ω NTK -component as 'spiky' is accurate: The oscillations in the hidden neurons induced by ω NTK interfere constructively to interpolate the noise in the training points and regress to 0 between training points. This entails immediate access to the signal component of the trained neural network in form of its ReLU-component.

7 Conclusion

Conceptually, our work shows that inconsistency of overfitting is quite a generic phenomenon for regression in fixed dimension. However, particular spiky-smooth estimators enable benign overfitting, even in fixed dimension. We translate the spikes that lead to benign overfitting in kernel regression into infinitesimal fluctuations that can be added to activation functions to consistently interpolate with wide neural networks. Our experiments verify that neural networks with spiky-smooth activation functions can exhibit benign overfitting even on small, low-dimensional data sets.

Technically, our inconsistency results cover many distributions, Sobolev spaces of arbitrary order, and arbitrary RKHS-norm-bounded overfitting estimators. Lemma E.1 serves as a generic tool to extend generalization bounds to the sphere S d , allowing us to cover (deep) ReLU NTKs and ReLU NNGPs.

Future work. While our experiments serve as a promising proof of concept, it remains unclear how to design activation functions that enable harmless interpolation of more complex neural network architectures and data sets. As another interesting insight, our consistent kernel sequence shows that although kernels may have equivalent RKHS (see Appendix G.2), their generalization error can differ arbitrarily much; the constants of the equivalence matter and the narrative that depth does not matter in the NTK regime as in Bietti and Bach (2021) is too simplified. More promisingly, analyses that extend our analysis in the infinite-width limit to a joint scaling of width and depth could help us to understand the influence of depth (Fort et al., 2020, Li et al., 2021, Seleznova and Kutyniok, 2022). Finite-sample analyses of moderate-width neural networks with feature learning parametrizations (Yang and Hu, 2021) and other initializations could enable to understand how to induce a spiky-smooth inductive bias in feature learning neural architectures.

Acknowledgments and Disclosure of Funding

Funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC 2075 - 390740016 and EXC 2064/1 - Project 390727645, as well as the DFG Priority Program 2298/1, project STE 1074/5-1. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Moritz Haas and David Holzmüller. We want to thank Tizian Wenzel and Václav Voráˇ cek for interesting discussions. We also thank Nadine Große, Jens Wirth, and Daniel Winkle for helpful comments on Sobolev spaces, and Nilotpal Sinha for pointing us to Laurent series.

References

Robert A. Adams and John J.F. Fournier. Sobolev Spaces . Elsevier Science, 2003.

Michael Aerni, Marco Milanta, Konstantin Donhauser, and Fanny Yang. Strong inductive biases provably prevent harmless interpolation. In International Conference on Learning Representations (ICLR) , 2023.

Mikhail S Agranovich. Sobolev spaces, their generalizations and elliptic problems in smooth and Lipschitz domains . Springer, 2015.

Ludovic Arnould, Claire Boyer, and Erwan Scornet. Is interpolation benign for random forest regression? In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2023.

Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Russ R. Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Yaroslav Averyanov and Alain Celisse. Early stopping and polynomial smoothing in regression with reproducing kernels. arXiv:2007.06827 , 2020.

Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Peter L. Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta Numerica , 30:87-201, 2021.

Daniel Beaglehole, Mikhail Belkin, and Parthe Pandit. On the inconsistency of kernel ridgeless regression in fixed dimensions. SIAM Journal on Mathematics of Data Science , 5(4):854-872, 2023.

Mikhail Belkin, Siyuan Ma, and Soumik Mandal. To understand deep learning we need to understand kernel learning. In International Conference on Machine Learning (ICML) , 2018.

Mikhail Belkin, Alexander Rakhlin, and Alexandre B. Tsybakov. Does data interpolation contradict statistical optimality? In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2019.

Jordan Bell. The singular value decomposition of compact operators on Hilbert spaces, 2014.

Alberto Bietti and Francis Bach. Deep equals shallow for ReLU networks in kernel regimes. In International Conference on Learning Representations (ICLR) , 2021.

Simon Buchholz. Kernel interpolation in sobolev spaces is not consistent in low dimensions. In Conference on Learning Theory (COLT) , 2022.

Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Bernd Carl and Irmtraud Stephani. Entropy, Compactness and the Approximation of Operators . Cambridge University Press, Cambridge, 1990.

Niladri S. Chatterji and Philip M. Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. Journal of Machine Learning Research (JMLR) , 2021.

Lin Chen and Sheng Xu. Deep neural tangent kernel and laplace kernel have the same rkhs. In International Conference on Learning Representations (ICLR) , 2021.

Zixiang Chen, Yuan Cao, Difan Zou, and Quanquan Gu. How much over-parameterization is sufficient to learn deep ReLU networks? In International Conference on Learning Representations (ICLR) , 2021.

Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Fan Chung and Linyuan Lu. Concentration inequalities and martingale inequalities: a survey. Internet Mathematics , 3(1):79 - 127, 2006.

Amit Daniely, Roy Frostig, and Yoram Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In Advances in Neural Information Processing Systems (NeurIPS) , 2016.

Ernesto De Vito, Nicole Mücke, and Lorenzo Rosasco. Reproducing kernel hilbert spaces on manifolds: Sobolev and diffusion spaces. Analysis and Applications , 19(03):363-396, 2021.

Ronald A DeVore and Robert C Sharpley. Besov spaces on domains in R d . Transactions of the American Mathematical Society , 335(2):843-864, 1993.

Luc Devroye, Laszlo Györfi, and Adam Krzy˙ zak. The Hilbert kernel regression estimate. Journal of Multivariate Analysis , 65(2):209-227, 1998.

Eleonora Di Nezza, Giampiero Palatucci, and Enrico Valdinoci. Hitchhiker's guide to the fractional Sobolev spaces. Bulletin des Sciences Mathematiques , 136(5):521-573, 2012.

Konstantin Donhauser, Mingqi Wu, and Fanny Yang. How rotational invariance of common kernels prevents generalization in high dimensions. In International Conference on Machine Learning (ICML) , 2021.

Konstantin Donhauser, Nicolò Ruggeri, Stefan Stojanovic, and Fanny Yang. Fast rates for noisy interpolation require rethinking the effect of inductive bias. In International Conference on Machine Learning (ICML) , 2022.

Lutz Duembgen. Bounding standard gaussian tail probabilities. arXiv:1012.2063 , 2010.

David E. Edmunds and Hans Triebel. Function Spaces, Entropy Numbers, Differential Operators . Cambridge University Press, Cambridge, 1996.

Simon Fischer and Ingo Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. Journal of Machine Learning Research (JMLR) , 2020.

Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Spencer Frei, Niladri S Chatterji, and Peter Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. In Conference on Learning Theory (COLT) , 2022.

Semjon Gerschgorin. Über die Abgrenzung der Eigenwerte einer Matrix. Izv. Akad. Nauk. USSR Otd. Fiz.-Mat. Nauk , 1931.

Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. The Annals of Statistics , 49(2):1029 - 1054, 2021.

Nikhil Ghosh and Mikhail Belkin. A universal trade-off between the model size, test loss, and training loss of linear predictors. arXiv:2207.11621 , 2022.

Tilmann Gneiting. Strictly and non-strictly positive definite functions on spheres. Bernoulli , 19(4): 1327-1349, 2013.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949 - 986, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In IEEE international conference on computer vision (ICCV) , 2015.

David Holzmüller. On the universality of the double descent peak in ridgeless regression. In International Conference on Learning Representations (ICLR) , 2021.

David Holzmüller and Ingo Steinwart. Training two-layer relu networks with gradient descent is inconsistent. Journal of Machine Learning Research (JMLR) , 23(181):1-82, 2022.

Roger A. Horn and Charles R. Johnson. Matrix Analysis . Cambridge University Press, 2013.

Tianyang Hu, Wenjia Wang, Cong Lin, and Guang Cheng. Regularization matters: A nonparametric perspective on overparametrized neural network. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2021.

Simon Hubbert, Quoc Le Gia, and Tanya Morton. Spherical Radial Basis Functions, Theory and Applications . Springer International Publishing, 2015.

Simon Hubbert, Emilio Porcu, Chris J. Oates, and Mark Girolami. Sobolev spaces, kernels and discrepancies over hyperspheres. Transactions on Machine Learning Research (TMLR) , 2023.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems (NeurIPS) , 2018.

Ziwei Ji, Justin D. Li, and Matus Telgarsky. Early-stopped neural networks are consistent. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Fredrik Johansson et al. mpmath: a Python library for arbitrary-precision floating-point arithmetic (version 1.3.0) , 2023. https://mpmath.org/ .

Motonobu Kanagawa, Philipp Hennig, Dino Sejdinovic, and Bharath K. Sriperumbudur. Gaussian processes and kernel methods: A review on connections and equivalences. arXiv:1805.08845v1 , 2018.

Jianfa Lai, Manyun Xu, Rui Chen, and Qian Lin. Generalization ability of wide neural networks on R . arXiv:2302.05933 , 2023.

Béatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. The Annals of Statistics , 28(5):1302 - 1338, 2000.

Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S. Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as Gaussian processes. In International Conference on Learning Representations (ICLR) , 2018.

Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha SohlDickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Mufan Li, Mihai Nica, and Dan Roy. The future is log-gaussian: Resnets and their infinite-depth-andwidth limit at initialization. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Yicheng Li, Haobo Zhang, and Qian Lin. Kernel interpolation generalizes poorly. Biometrika , 2023.

Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel 'ridgeless' regression can generalize. Annals of Statistics , 48(3):1329-1347, 2020.

Tengyuan Liang and Benjamin Recht. Interpolating classifiers make few mistakes. Journal of Machine Learning Research (JMLR) , 24(20):1-27, 2023.

Tengyuan Liang, Alexander Rakhlin, and Xiyu Zhai. On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels. In Conference on Learning Theory (COLT) , 2020.

Jacques Louis Lions and Enrico Magenes. Non-homogeneous boundary value problems and applications: Vol. 1 . Springer Science & Business Media, 2012.

Neil Rohit Mallinar, James B Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward a refined taxonomy of overfitting. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations (ICLR) , 2018.

Andrew D. Mcrae, Santhosh Karnik, Mark Davenport, and Vidya K. Muthukumar. Harmless interpolation in regression and classification with structured features. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2022.

Robert E. Megginson. An Introduction to Banach Space Theory . Springer-Verlag, New York, 1998.

Leon Mirsky. On the trace of matrix products. Mathematische Nachrichten , 20(3-6):171-174, 1959.

Andrea Montanari, Feng Ruan, Youngtak Sohn, and Jun Yan. The generalization error of max-margin linear classifiers: High-dimensional asymptotics in the overparametrized regime. arXiv:1911.01544 , 2019.

Vidya Muthukumar, Adhyyan Narang, Vignesh Subramanian, Mikhail Belkin, Daniel Hsu, and Anant Sahai. Classification vs regression in overparameterized regimes: Does the loss function matter? JMLR , 22(1):10104-10172, 2021.

Nicole Mücke and Ingo Steinwart. Global minima of DNNs: The plenty pantry. arXiv:1905.10686 , 2019.

Radford M. Neal. Priors for Infinite Networks . Springer New York, 1996.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS) , 2019.

Albrecht Pietsch. Eigenvalues and s-Numbers . Geest & Portig K.-G., Leipzig, 1987.

Iosif Pinelis. Exact lower and upper bounds on the incomplete gamma function. Mathematical Inequalities & Applications , 23(4):1261-1278, 2020.

Alexander Rakhlin and Xiyu Zhai. Consistency of interpolation with Laplace kernels is a highdimensional phenomenon. In Conference on Learning Theory (COLT) , 2019.

Akshay Rangamani, Lorenzo Rosasco, and Tomaso Poggio. For interpolating kernel machines, minimizing the norm of the erm solution maximizes stability. Analysis and Applications , 21:193 215, 2023.

Garvesh Raskutti, Martin J. Wainwright, and Bin Yu. Early stopping and non-parametric regression: An optimal data-dependent stopping rule. Journal of Machine Learning Research (JMLR) , 15(11): 335-366, 2014.

Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning . MIT Press, 2005.

Hans Richter. Zur Abschätzung von Matrizennormen. Mathematische Nachrichten , 18(1-6):178-187, 1958.

Robert Schaback. Superconvergence of kernel-based interpolation. Journal of Approximation Theory , 235:1-19, 2018.

Cornelia Schneider and Nadine Große. Sobolev spaces on Riemannian manifolds with bounded geometry: General coordinates traces. Mathematische Nachrichten , 286(16), 2013.

Mariia Seleznova and Gitta Kutyniok. Neural tangent kernel beyond the infinite-width limit: Effects of depth and initialization. In International Conference on Machine Learning (ICML) , 2022.

Ohad Shamir. The implicit bias of benign overfitting. In Conference on Learning Theory (COLT) , 2022.

James Benjamin Simon, Sajant Anand, and Mike Deweese. Reverse engineering the neural tangent kernel. In International Conference on Machine Learning (ICML) , 2022.

Ingo Steinwart. Consistency of support vector machines and other regularized kernel machines. IEEE Transactions on Information Theory , 2001.

Ingo Steinwart. A short note on the comparison of interpolation widths, entropy numbers, and Kolmogorov widths. J. Approx. Theory , 215:13-27, 2017.

Ingo Steinwart and Andreas Christmann. Support Vector Machines . Springer New York, 2008.

Ingo Steinwart and Clint Scovel. Mercer's theorem on general domains: on the interaction between measures, kernels, and RKHSs. Constr. Approx. , 35:363-417, 2012.

Ingo Steinwart, Don R. Hush, and Clint Scovel. Optimal rates for regularized least squares regression. In Conference on Learning Theory (COLT) , 2009.

Daniel W. Stroock et al. Essentials of integration theory for analysis . Springer, 2011.

Sattar Vakili, Michael Bromberg, Jezabel Garcia, Da-shan Shiu, and Alberto Bernacchia. Uniform generalization bounds for overparameterized neural networks. arXiv:2109.06099 , 2021.

Ernesto Araya Valdivia. Relative concentration bounds for the spectrum of kernel matrices. arXiv:1812.02108 , 2018.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science . Cambridge University Press, 2018.

Guorong Wang, Yimin Wei, Sanzheng Qiao, Peng Lin, and Yuzhuo Chen. Generalized Inverses: Theory and Computations . Springer, 2018.

Yutong Wang and Clayton Scott. Consistent interpolating ensembles via the manifold-Hilbert kernel. In Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Yuting Wei, Fanny Yang, and Martin J Wainwright. Early stopping for kernel boosting algorithms: A general analysis with localized complexities. In Advances in Neural Information Processing Systems (NeurIPS) , 2017.

Holger Wendland. Scattered Data Approximation . Cambridge University Press, 2005.

Xingyu Xu and Yuantao Gu. Benign overfitting of non-smooth neural networks beyond lazy training. In International Conference on Artificial Intelligence and Statistics (AISTATS) , 2023.

Greg Yang and Edward J. Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning (ICML) , 2021.

Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. On early stopping in gradient descent learning. Constructive Approximation , 26:289-315, 2007.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

Yaoyu Zhang, Zhi-Qin John Xu, Tao Luo, and Zheng Ma. A type of generalization error induced by initialization in deep neural networks. In Proceedings of The First Mathematical and Scientific Machine Learning Conference , 2020.

Appendices

Appendix Contents.

A Detailed related work

Motivated by Zhang et al. (2021) and Belkin et al. (2018), an abundance of papers have tried to grasp when and how benign overfitting occurs in different settings. Rigorous understanding is mainly restricted to linear (Bartlett et al., 2020), feature (Hastie et al., 2022) and kernel regression (Liang and Rakhlin, 2020) under restrictive distributional assumptions. In the well-specified linear setting under additional assumptions, the minimum-norm interpolant is consistent if and only if k ≪ n ≪ d , the topk eigendirections of the covariate covariance matrix align with the signal, followed by sufficiently many 'quasi-isotropic' directions with eigenvalues of similar magnitude (Bartlett et al., 2020).

Kernel methods. The analysis of kernel methods is more nuanced and depends on the interplay between the chosen kernel, the choice of regularization and the data distribution. L 2 -generalization error bounds can be derived in the eigenbasis of the kernel's integral operator (Mcrae et al., 2022), where upper bounds of the form √ y ⊤ k ( X , X ) -1 y /n promise good generalization when the regression function f ∗ is aligned with the dominant eigendirections of the kernel, or in other words, when ∥ f ∗ ∥ H is small. Most recent work focuses on high-dimensional limits, where the data dimensionality d →∞ . For d →∞ , the Hilbert space and its norm change, so that consistency results that demand bounded Hilbert norm of rotation-invariant kernels do not even include simple functions like sparse products (Donhauser et al., 2021, Lemma 2.1). In the regime d l + δ ≤ n ≤ d l +1 -δ , rotation-invariant (neural) kernel methods (Ghorbani et al., 2021, Donhauser et al., 2021) can in fact only learn the polynomial parts up to order l of the regression function f ∗ , and fully-connected NTKs do so. Liang et al. (2020) uncover a related multiple descent phenomenon in kernel regression, where the risk vanishes for most n →∞ , but peaks at n = d i for all i ∈ N . The slower d grows, the slower the optimal rate n -1 2 i +1 between the peaks. Note, however, that these bounds are only upper bounds, and whether they are optimal remains an open question to the best of our knowledge. Another recent line of work analyzes how different inductive biases, measured in ∥ · ∥ p -norm minimization, p ∈ [1 , 2] , (Donhauser et al., 2022) or in the filter size of convolutional kernels (Aerni et al., 2023), affects the generalization properties of minimum-norm interpolants. While the risk on noiseless training samples (bias) decreases with decreasing p or small filter size, the sensitivity to noise in the training data (variance) increases. Hence only 'weak inductive biases', that is large p or large filter sizes, enable harmless interpolation. Our results suggest that to achieve harmless interpolation in fixed dimension one has to construct and minimize more unusual norms than ∥ · ∥ p -norms.

Regularised kernel regression achieves optimal rates. With appropriate regularization, kernel ridge regularization with typical universal kernels like the Gauss, Matérn, and Laplace kernels is universally consistent (Steinwart and Christmann, 2008, Chapter 9). Steinwart et al. (2009, Corollary 6) even implies minimax optimal nonparametric rates for clipped kernel ridge regression with Sobolev kernels and f ∗ ∈ H β where d/ 2 < β ≤ s for the choice ρ n = n -2 s/ (2 β + d ) . Although f ∗ is not necessarily in the RKHS, KRR is adaptive and can still achieve optimal learning rates. Lower smoothness β of f ∗ as well as higher smoothness of the kernel should be met with faster decay of ρ n . Optimal rates in Sobolev RKHS can also be achieved using cross-validation of the regularization ρ (Steinwart et al., 2009), early stopping rules based on empirical localized Rademacher (Raskutti et al., 2014) or Gaussian complexity (Wei et al., 2017) or smoothing of the empirical risk via kernel matrix eigenvalues (Averyanov and Celisse, 2020).

Lower bounds for kernel regression. Besides Rakhlin and Zhai (2019) and Buchholz (2022), Beaglehole et al. (2023) derive inconsistency results for kernel ridgeless regression given assumptions on the spectral tail in the Fourier basis. Mallinar et al. (2022) provide a characterization of kernel ridge regression into benign, tempered and catastrophic overfitting using a heuristic approximation of the risk via the kernel's eigenspectrum, essentially assuming that the eigenfunctions can be replaced by structureless Gaussian random variables. A general lower bound for ridgeless linear regression Holzmüller (2021) predicts bad generalization near the 'interpolation threshold', where the dimension of the feature space is close to n , also known as the double descent phenomenon. In this regime, Ghosh and Belkin (2022) also consider overfitting by a fraction beyond the noise level and derive a lower bound for linear models.

Benign overfitting in fixed dimension. Only few works have established consistency results for interpolating models in fixed dimension. The first statistical guarantees for Nadaraya-Watson kernel

smoothing with singular kernels were given by Devroye et al. (1998). Optimal non-asymptotic results have only been established more recently. Belkin et al. (2019) show that Nadaraya-Watson kernel smoothing achieves minimax optimal convergence rates for a ∈ (0 , d/ 2) under smoothness assumptions on f ∗ , when using singular kernels such as truncated Hilbert kernels K ( u ) = ∥ u ∥ a 2 1 ∥ u ∥≤ 1 , which do not induce RKHS that only contain weakly differentiable functions (as our results do). By thresholding the kernel they can adjust the amount of overfitting without affecting the generalization bound. To the best of our knowledge, rigorously proving or disproving analogous bounds for kernel ridge regression remains an open question. Arnould et al. (2023) show that median random forests are able to interpolate consistently in fixed dimension because of an averaging effect introduced through feature randomization. They conjecture consistent interpolation for Breiman random forests based on numerical experiments.

Classification. For binary classification tasks, benign overfitting is a more generic phenomenon than for regression tasks (Muthukumar et al., 2021, Shamir, 2022). Consistency has been shown under linear separability assumptions (Montanari et al., 2019, Chatterji and Long, 2021, Frei et al., 2022) and through complexity bounds with respect to reference classes like the 'Neural Tangent Random Feature' model (Cao and Gu, 2019, Chen et al., 2021). Most recently, Liang and Recht (2023) have shown that the 0-1-generalization error of minimum RKHS-norm interpolants ˆ f 0 is upper bounded by ∥ ˆ f 0 ∥ 2 H n and analogously that kernel ridge regression ˆ f ρ generalizes as y ⊤ ( k ( X , X )+ ρ I ) -1 y n , where the numerator upper bounds ∥ ˆ f ρ ∥ 2 H . Their bounds imply consistency as long as the total variation distance between the class conditionals is sufficiently large and the regression function has bounded RKHS-norm, and their Lemma 7 shows that the upper bound is rate optimal. Under a noise condition on the regression function f ∗ ( x ) = E [ y | x ] for binary classification and bounded ∥ f ∗ ∥ H , our results together with Liang and Recht (2023) reiterate the distinction between benign overfitting for binary classification and inconsistent overfitting for least squares regression for a large class of distributions in kernel regression over Sobolev RKHS. Chapter 8 of Steinwart and Christmann (2008) discusses how the overlap of the two classes may influence learning rates under positive regularization. Using Nadaraya-Watson kernel smoothing, Wang and Scott (2022) offer the first consistency result for a simple interpolating ensemble method with data-independent base classifiers.

Connection to neural networks. It is known that neural networks can behave like kernel methods in certain infinite-width limits. For example, the function represented by a randomly initialized NN behaves like a Gaussian process with the NN Gaussian process (NNGP) kernel, which depends on details such as the activation function and depth of the NN (Neal, 1996, Lee et al., 2018, Matthews et al., 2018). Hence, Bayesian inference in infinitely wide NNs is GP regression, whose posterior predictive mean function is of the form f ∞ ,ρ , where ρ depends on the assumed noise variance. Moreover, gradient flow training of certain infinitely wide NNs is similar to gradient flow training with the so-called neural tangent kernel (NTK) (Jacot et al., 2018, Lee et al., 2019, Arora et al., 2019), and the correspondence can be made exact using small modifications to the NN to remove the stochastic effect of the random initial function (Arora et al., 2019, Zhang et al., 2020). In other words, certain infinitely wide NNs trained with gradient flow learn functions of the form f t, 0 .

When considering the sphere Ω = S d , the NTK and NNGP kernels of fully-connected NNs are dot-product kernels, i.e., k ( x , x ′ ) = κ ( ⟨ x , x ′ ⟩ ) for some function κ : [ -1 , 1] → R . Moreover, from Bietti and Bach (2021) and Chen and Xu (2021) it follows that the RKHSs of typical NTK and NNGP kernels for the ReLU activation function are equivalent to the Sobolev spaces H ( d +1) / 2 ( S d ) and H ( d +3) / 2 ( S d ) , respectively, cf Appendix B.4.

Regarding consistency, Ji et al. (2021) use the NTK correspondence to show that early-stopped wide NNs for classification are universally consistent under some assumptions. On the other hand, Holzmüller and Steinwart (2022) show that zero-initialized biases can prevent certain two-layer ReLU NNs from being universally consistent. Lai et al. (2023) show an inconsistency-type result for overfitting two-layer ReLU NNs with d = 1 , but for fixed inputs X . They also note that an earlier inconsistency result by Hu et al. (2021) relies on an unproven result. Li et al. (2023) show that consistency with polynomial convergence rates is impossible for minimum-norm interpolants of common kernels including ReLU NTKs. Mallinar et al. (2022) conjecture tempered overfitting and therefore inconsistency for interpolation with ReLU NTKs based on their semi-rigorous result and the results of Bietti and Bach (2021) and Chen and Xu (2021). Xu and Gu (2023) establish consistency of overfitting wide 2-layer neural networks beyond the NTK regime for binary classification in very

high dimension d = Ω( n 2 ) and for a quite restricted class of distributions (the mean difference µ of the class conditionals needs to fulfill µ = Ω(( d/n ) 1 / 4 log 1 / 4 ( md/n )) and µ = O (( d/n ) 1 / 2 ) ).

B Kernels and Sobolev spaces on the sphere

B.1 Background on Sobolev spaces

We say that two Hilbert spaces H 1 , H 2 are equivalent, written as H 1 ∼ = H 2 , if they are equal as sets and the corresponding norms ∥ · ∥ H 1 and ∥ · ∥ H 2 are equivalent.

Let Ω be an open set with C ∞ boundary. In this paper, we will mainly consider ℓ 2 -balls for Ω . There are multiple equivalent ways to define a (fractional) Sobolev space H s (Ω) , s ∈ R ≥ 0 , these are equivalent in the sense that the resulting Hilbert spaces will be equivalent. For example, H s (Ω) can be defined through restrictions of functions from H s ( R d ) , through interpolation spaces, or through Sobolev-Slobodetski norms (see e.g. Chapter 5 and 14 in Agranovich, 2015 and Chapters 7-10 in Lions and Magenes, 2012). Some requirements on Ω can be relaxed, for example to Lipschitz domains, by using more general extension operators (e.g. DeVore and Sharpley, 1993). Since our results are based on equivalent norms and not specific norms, we do not care which of these definitions is used. Further background on Sobolev spaces can be found in Adams and Fournier (2003), Wendland (2005) and Di Nezza et al. (2012).

B.2 General kernel theory and notation

There is a one-to-one correspondence between kernel functions k and the corresponding reproducing kernel Hilbert spaces (RKHS) H k . Mercer's theorem (Steinwart and Christmann, 2008, Theorem 4.49) states that for compact Ω , continuous k and a Borel probability measure P X on Ω whose support is Ω , the integral operator T k,P X : L 2 ( P X ) → L 2 ( P X ) given by

can be decomposed into an orthonormal basis ( e i ) i ∈ I of L 2 ( P X ) and corresponding eigenvalues ( λ i ) i ∈ I ≥ 0 , λ l ↘ 0 , such that

We write λ i ( T k,P X ) := λ i . Moreover, k ( x , x ′ ) = ∑ i ∈ I λ i e i ( x ) e i ( x ′ ) converges absolutely and uniformly, and the RKHS is given by

The corresponding inner product between f = ∑ i ∈ I a i √ λ i e i ∈ H and g = ∑ i ∈ I b i √ λ i e i ∈ H can then be written as

We use asymptotic notation O, Ω , Θ for integers n in the following way: We write

Above, we require that the inequality f ( n ) ≤ Cg ( n ) holds for all n and not only for n ≥ n 0 . This implies that if f ( n ) = Ω( g ( n )) , then f must be nonzero whenever g is nonzero. This is an important detail when arguing about equivalence of RKHSs, since it allows the following statement: If we have two kernels k, ˜ k with Mercer representations

with identical eigenfunctions e i and eigenvalues satisfying λ i = Θ( ˜ λ i ) , then the associated RKHSs are equivalent by (B.1) and (B.2).

B.3 Dot-product kernels on the sphere

A kernel of the form k ( x , x ′ ) = κ ( ⟨ x , x ′ ⟩ ) for some function κ is called dot-product kernel . Dotproduct kernels are rotationally invariant. Especially, NTKs and NNGPs of fully-connected NNs restricted to the sphere S d are dot-product kernels. Moreover, kernels like the Laplace, Matérn, and Gaussian kernels that only depend on the distance between their inputs are also dot-product kernels when restricted to the sphere S d . Therefore, in this section, we will assume that k : S d × S d → R is a dot-product kernel.

We can then leverage some convenient results from the theory of dot-product kernels on the sphere, which are summarized in more detail by Hubbert et al. (2023). Let { Y l, 1 , . . . , Y l,N l,d } be a real orthonormal basis for the space of spherical harmonics of degree l within L 2 ( S d ) . Moreover, let ω d be the surface area of S d , then the ˜ Y l,i := √ ω d Y l,i form a corresponding orthonormal basis w.r.t. the uniform distribution U ( S d ) . Then, a Mercer representation of k is given by

with ˜ µ l = µ l /ω d . Especially, the integral operator T k, U ( S d ) for the uniform distribution U ( S d ) has eigenvalues ˜ µ l with multiplicity N l,d and eigenfunctions ˜ Y l,i . The RKHS of k is then given by

Since the index l can be zero, we will denote decay asymptotics for l in the form Θ(( l +1) -q ) and not Θ( l -q ) , cf. our definition of Θ notation in Appendix B.2.

Lemma B.1 (Sobolev dot-product kernels on the sphere) . For a dot-product kernel k on S d as above, the RKHS H k is equivalent to the Sobolev space H s ( S d ) , s > d/ 2 , if and only if µ l = Θ(( l +1) -2 s ) . In this case, we have

̸

Step 1: Ordering the eigenvalues. Consider a permutation π : N 0 → N 0 such that

Proof. Step 0: Equivalence. If µ l = Θ(( l +1) -2 s ) , it is stated in Section 3 in Hubbert et al. (2023) that H k ∼ = H s ( S d ) . Onthe other hand, if µ l = Θ(( l +1) -2 s ) , it is easy to see that H k is not equivalent to the RKHS of a kernel with µ l = Θ(( l +1) -2 s ) . It remains to show λ i ( T k, U ( S d ) ) = Θ( i -2 s/d ) .

We can then define the partial sums

For S l -1 < i ≤ S l , we then have λ i ( T k, U ( S d ) ) = ˜ µ π ( l ) .

Step 2: Show π ( i ) = Θ( i ) . Let c, C > 0 such that c ( l +1) -2 s ≤ ˜ µ l ≤ C ( l +1) -2 s for all l ∈ N 0 . For indices i, j ∈ N 0 , we have the implications

Therefore, for i ≥ 1 and j ≥ 0 ,

We can thus conclude that π ( i ) + 1 = Θ( i +1) .

Step 3: Individual Eigenvalue decay. As explained in Section 2.1 in Hubbert et al. (2023), we have N l,d = Θ(( l +1) d -1 ) . Therefore,

Now, let i ≥ 1 and let l ∈ N 0 such that S l -1 < i ≤ S l . We have i ≥ Ω( l d ) , and i ≤ O (( l +1) d ) , which implies i = Θ(( l +1) d ) since i ≥ 1 . Therefore,

B.4 Neural kernels

Several NTK and NNGP kernels have RKHSs that are equivalent to Sobolev spaces on S d . In the following cases, we can deduct this from known results:

Consider fully-connected NNs with L ≥ 3 layers without biases and the activation function φ ( x ) = max { 0 , x } m , m ∈ N 0 . Especially, the case m = 1 corresponds to the ReLU activation. Vakili et al. (2021) generalize the result by Bietti and Bach (2021) from m = 1 to m ≥ 1 , showing that the NTK-RKHS is equivalent to H s ( S d ) for s = ( d +2 m -1) / 2 and the NNGP-RKHS is equivalent to H s ( S d ) for s = ( d + 2 m + 1) / 2 . For m = 0 , Bietti and Bach (2021) essentially show that the NNGP-RKHS is equivalent to H s ( S d ) for s = ( d +2 2 -L ) / 2 . However, all of the aforementioned result have the problem that the main theorem by Bietti and Bach (2021) allows for the possibility that finitely many µ l are zero, which can change the RKHS. Using our Lemma B.2 below, it follows that all µ l are in fact nonzero for NNGPs and NTKs since they are kernels in every dimension d using the same function κ independent of the dimension. Hence, the equivalences to Sobolev spaces stated before are correct.

Chen and Xu (2021) prove that the RKHS of the NTK corresponding to fully-connected ReLU NNs with zero-initialized biases and L ≥ 2 (as opposed to no biases and L ≥ 3 above) layers is equivalent to the RKHS of the Laplace kernel on the sphere. Since the Laplace kernel is a Matérn kernel of order ν = 1 / 2 (see e.g. Section 4.2 in Rasmussen and Williams (2005)), we can use Proposition 5.2 of Hubbert et al. (2023) to obtain equivalence to H s ( S d ) with s = ( d +1) / 2 . Alternatively, we can obtain the RKHS of the Laplace kernel from Bietti and Bach (2021) combined with Lemma B.2.

Bietti and Bach (2021) also show that under an integrability condition on the derivatives, C ∞ activations induce NTK and NNGP kernels whose RKHSs are smaller than every Sobolev space.

Lemma B.2 (Guaranteeing non-zero eigenvalues) . Let κ : [ -1 , 1] → R , let d ≥ 1 , and let

Suppose that k d +2 is a kernel. Then, k d is a kernel. Moreover, if the corresponding eigenvalues µ l of k d satisfy µ l > 0 for infinitely many l , then they satisfy µ l > 0 for all l ∈ N 0 .

Proof. The fact that k d is a kernel follows directly from the inclusion Φ d +2 ⊆ Φ d mentioned in Gneiting (2013). For D ∈ { d, d +2 } , let µ l,d be the sequence of eigenvalues µ l associated with k D . Then, as mentioned for example by Hubbert et al. (2023), the Schoenberg coefficients b l,d satisfy

Especially, the Schoenberg coefficients b l,d have the same sign as the eigenvalues µ l,d . We use

where the inequality follows from the fact that k d +2 is a kernel and the equality is the statement of Corollary 3 by Gneiting (2013). In any of the three cases, b l +2 ,d > 0 implies b l,d > 0 . Hence, if b l,d > 0 for infinitely many l , then b l,d > 0 for all l , which implies µ l,d > 0 for all l .

C Gradient flow and gradient descent with kernels

C.1 Derivation of gradient flow and gradient descent

Here, we derive expressions for gradient flow and gradient descent in the RKHS for the regularized loss

Note that we will take derivatives in the RKHS with respect to f , which is different from taking derivatives w.r.t. the coefficients c in a model f ( x ) = c ⊤ k ( X , x ) .

In the RKHS-Norm, the Fréchet derivative of L is

which is represented in H k by

Now assume that f = ∑ n i =1 a i k ( x i , · ) = a ⊤ k ( X , · ) . Then,

Especially, under gradient flow of f , the coefficients a follow the dynamics

which is solved for a (0) = 0 by

which is the closed form expression (1) of f t,ρ .

For gradient descent, assuming that f GD t,ρ = c ⊤ t,ρ k ( X , · ) , we have

If f GD 0 ,ρ ≡ 0 , the coefficients evolve as c 0 = 0 and

For an analysis of gradient descent for kernel regression with ρ = 0 , we refer to, e.g., Yao et al. (2007).

C.2 Gradient flow and gradient descent initialized at 0 have monotonically growing H -norm

In the following proposition we show that under gradient flow and gradient descent with sufficiently small learning rates initialized at 0 , the RKHS norm grows monotonically with time t . This immediately implies that Assumption (N) with C norm = 1 holds for all estimators f t,ρ from (1).

Proposition C.1.

(i) For any t ∈ [0 , ∞ ] and any ρ ≥ 0 , f t,ρ from (1) fulfills Assumption (N) with C norm = 1 .

(ii) For any t ∈ N 0 ∪ {∞} and any ρ ≥ 0 , with sufficiently small fixed learning rate 0 ≤ η ≤ 1 2( ρ + λ max ( k ( X , X )) /n ) , f GD t,ρ fulfills Assumption (N) with C norm = 1 .

Proof. Proof of (i):

We write f t,ρ ( x ) = k ( x , X ) c t,ρ , where c t,ρ := A t,ρ ( X ) y . We now show that the RKHS-norm of f t,ρ grows monotonically in t , by using the eigendecomposition k ( X , X ) = U Λ U ⊤ , where Λ = diag ( λ 1 , . . . , λ n ) ∈ R n × n is diagonal and U ∈ R n × n is orthonormal, and writing ˜ y := U ⊤ y . Then it holds that

Proof of (ii):

Expanding the iteration in the definition of c t,ρ yields

We again use the eigendecomposition k ( X , X ) = U Λ U ⊤ , where Λ = diag ( λ 1 , . . . , λ n ) ∈ R n × n is diagonal and U ∈ R n × n is orthonormal, and write ˜ y := U ⊤ y . Then, using sufficiently small learning rates 0 ≤ η t ≤ 1 2( ρ + λ max ( k ( X , X )) /n ) in all time steps t ∈ N , it holds that

The last display shows that ∥ f GD t,ρ ∥ 2 H grows monotonically in t , strictly monotonically if η t ∈ (0 , 1 2( ρ + λ max ( k ( X , X )) /n ) ) holds for all t . It also shows that if ρ ′ ≥ ρ then ∥ f GD t,ρ ′ ∥ H ≤ ∥ f GD t,ρ ∥ H for any t ∈ N ∪ {∞} . To see that ∥ f GD t,ρ ∥ 2 H ≤ ∥ f ∞ , 0 ∥ 2 H for all t ∈ N ∪ {∞} and all ρ ≥ 0 , observe that with fixed learning rates η t = η ∈ (0 , 1 2( ρ + λ max ( k ( X , X )) /n ) ) ⊆ (0 , 1 2 λ max ( k ( X , X )) /n ) , for all t ∈ N ∪ {∞} it holds that

Since it suffices to consider the case ρ → 0 , using the above derivation in (C.1) yields ∥ f GD t,ρ ∥ 2 H ≤ ∥ f ∞ , 0 ∥ 2 H for all t ∈ N , which concludes the proof.

D Proof of Theorem 1

Our goal in this section is to prove Theorem D.1, which can be seen as a generalization of Theorem 1 to varying bandwidths. To be able to speak of bandwidths, we need to consider translation-invariant kernels. Although Theorem 1 is formulated for general kernels with Sobolev RKHS, it follows from Theorem D.1 since we can always find, for a fixed bandwidth, a translation-invariant kernel with equivalent RKHS, such that only the constant C norm changes in the theorem statement.

To generate the RKHS H s , Buchholz (2022) uses the translation-invariant kernel k B ( x , y ) = u B ( x -y ) defined via its Fourier transform ˆ u B ( ξ ) = (1 + | ξ | 2 ) -s . Adapting the bandwidth, the kernel is then normalized in the usual L 1 -sense,

If k γ denotes a L 1 -normalized translation-invariant kernel with bandwidth γ > 0 , i.e. there exists a q : R d → R such that k γ ( x, y ) = γ -d q ( x -y γ ) , then inequality (D.2) holds with c independent of the sequence of bandwidths ( γ n ) n ∈ N ⊆ (0 , 1) , as long as f D fulfills (N) for the sequence ( H γ n ) n ∈ N with constant C norm > 0 .

Proof. By assumption, the RKHS norm ∥ · ∥ H k induced by the kernel k (or k γ if we allow bandwidth adaptation) is equivalent to the RKHS norm ∥ · ∥ H γ induced by a kernel of the form (D.1) with an arbitrary but fixed choice of bandwidth γ ∈ (0 , 1) , which means that there exists a constant C γ > 0 such that 1 C γ ∥ f ∥ H γ ≤ ∥ f ∥ H k ≤ C γ ∥ f ∥ H γ for all f ∈ H k . Hence the minimum-norm interpolant g D,γ in H γ satisfies

where ∥ g D ∥ H k ≤ ∥ g D,γ ∥ H k because, g D is the minimum-norm interpolant in H k .

Now consider the RKHS norm ∥ · ∥ ˜ H γ of a translation-invariant kernel k γ . Then the functions { h p ( x ) = e ip · x } p ∈ R d are eigenfunctions of the kernel's integral operator, so that the RKHS norm can be written as (Rakhlin and Zhai, 2019)

where ˆ f denotes the Fourier transform of f .

By assumption we know that there exists a C γ 0 > 0 such that 1 C γ 0 ∥ f ∥ H γ 0 ≤ ∥ f ∥ ˜ H γ 0 ≤ C γ 0 ∥ f ∥ H γ 0 holds for some fixed bandwidth γ 0 > 0 , then substituting by ˜ ω = γ γ 0 ω yields

Theorem D.1 ( Inconsistency of overfitting estimators ) . Let assumptions (D1) and (K) hold. Let c fit ∈ (0 , 1] and C norm > 0 . Then, there exist c > 0 and n 0 ∈ N such that the following holds for all n ≥ n 0 with probability 1 -O (1 /n ) over the draw of the data set D with n samples: For every function f ∈ H k with the excess risk satisfies

In the same way we get 1 C γ 0 ∥ f ∥ H γ ≤ ∥ f ∥ ˜ H γ for arbitrary γ ∈ (0 , 1) . This shows that the constant C γ 0 , that quantifies the equivalence between ∥ · ∥ H γ and ∥ · ∥ ˜ H γ does not depend on the bandwidth γ . Finally Proposition D.4, Proposition D.2 and Remark D.3 together yield the result.

The following proposition generalizes the inconsistency result for large bandwidths, Proposition 4 in Buchholz (2022), beyond interpolating estimators to estimators that overfit at least an arbitrary constant fraction beyond the Bayes risk and whose RKHS norm is at most a constant factor larger than the RKHS norm of the minimum-norm interpolant. Compared to Rakhlin and Zhai (2019), Buchholz gets a statement in probability over the draw of a training set D and less restrictive assumptions on the domain Ω and dimension d .

Proposition D.2 ( Inconsistency for large bandwidths ) . Let c fit ∈ (0 , 1] and C norm > 0 . Let the data set D = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } be drawn i.i.d. from a distribution P that fulfills Assumption (D1), let g D,γ be the minimum-norm interpolant in H := H γ with respect to the kernel (D.1) for a bandwidth γ > 0 . Then, for every A > 0 , there exist c > 0 and n 0 ∈ N such that the following holds for all n ≥ n 0 with probability 1 -O (1 /n ) over the draw of the data set D with n samples:

For every function f ∈ H that fulfills Assumption (O) with c fit and Assumption (N) with C norm the excess risk satisfies

where c depends neither on n nor on 1 > γ > An -1 /d > 0 .

Remark D.3. Proposition D.2 holds for any kernel that fulfills Assumption (K). The reason is that any kernel k that fulfills assumption (K) and the kernel defined in (D.1) have the same RKHS and equivalent norms. Therefore every function f ∈ H k = H γ (equality as sets) that fulfills Assumptions (O) and (N) for the kernel k also fulfills Assumptions (O) and (N) with an adapted constant C norm for the kernel (D.1). ◀

Proof. Step 1: Generalizing the procedure in Buchholz (2022).

We write [ n ] = { 1 , . . . , n } and follow the proof of Proposition 4 in Buchholz (2022). Define u ( x ) = f ( x ) -f ∗ ( x ) . We need to show that with probability at least 1 -O ( n -1 ) over the draw of D it holds that ∥ u ∥ L 2 ( P X ) ≥ c > 0 , where c depends neither on n nor on γ .

For this purpose we show that with probability at least 1 -3 n -1 over the draw of D there exist a constants c ′′ , κ ′′ > 0 depending only on c fit and a subset P ′′ ⊆ [ n ] with |P ′′ | ≥ ⌊ κ ′′ · n ⌋ such that

̸

Then via Lemma 7 in Buchholz (2022) as well as Lemma D.7 we can choose a large subset P ′′′ ⊆ [ n ] of the training point indices with |P ′′′ | ≥ n -|P ′′ | / 2 , such that the x i for i ∈ P ′′′ are well-separated in the sense that min { i,j ∈P ′′′ , i = j } ∥ x i -x j ∥ ≥ d min with d min := c ′′′ n -1 /d , where c ′′′ depends on c fit , d , the upper bound on the Lebesgue density C u and on the smoothness of the RKHS s . Then the intersection P ′′ ∩P ′′′ contains at least |P ′′ | 2 points. Now we can replace P ′ in the proof of Proposition 4 for s ∈ N in Buchholz (2022) by the intersection P ′′ ∩ P ′′′ . The rest of the proof applies without modification, where (42) holds by our assumption ∥ f ∥ H ≤ C H ∥ g D ∥ H . Our modifications do not affect Buchholz' arguments for the extension to s / ∈ N .

Given a choice of κ ′′ , c ′′ > 0 , consider the event (over the draw of D )

Step 2: The existence of P ′′ .

With the proper choices of c ′′ and κ ′′ independent of n and f , we will show P ( E ) ≤ 3 n -1 . We will find a small c ′′ > 0 such that if f ∗ and f are closer than c ′′ on too many training points ˜ P

and f overfits by at least the fraction c fit , the noise variables ε i on the complement ˜ P c would have to be unreasonably large, contradicting the event E 6 i defined below, and implying (D.3) with high probability. We will use the notation ∥ f ∥ 2 P := ∑ i ∈P f ( x i ) 2 and ∥ y ∥ 2 P := ∑ i ∈P y 2 i .

Step 2b: Noise bounds.

Lemma D.6 ( i ) states that there exists a κ ′′ > 0 small enough such that the event (over the draw of D )

fulfills, for n large enough, P ( E 6 i ) ≥ 1 -n -1 .

Lemma D.6 ( ii ) implies that there exists a c lower > 0 such that the event (over the draw of D )

fulfills, for n large enough, P ( E 6 ii ) ≥ 1 -n -1 .

Lemma D.5 states that the total amount of noise ∥ ε ∥ 2 [ n ] concentrates around its mean nσ 2 . More precisely, we will use that for any c ε ∈ (0 , 1) the event (over the draw of D )

fulfills P ( E 5 ) ≥ 1 -exp ( -n · ( 1 -c ε 2 ) 2 ) .

Step 2c: Lower bounding ∥ ε ∥ 2 ˜ P c .

In step 2c, assume we choose ˜ c fit > 0 such that ˜ c fit ∥ f ∗ -y ∥ 2 ˜ P ≤ ∥ f -y ∥ 2 ˜ P . Then by the overfitting Assumption (O) it holds that

Given some function f ∈ H , assume in steps 2c and 2d that event E holds and that ˜ P ⊆ [ n ] denotes a subset of the training set that fulfills | ˜ P| ≥ ⌈ (1 -κ ′′ ) n ⌉ and | f ∗ ( x i ) -f ( x i ) | < c ′′ ∀ i ∈ ˜ P .

If we restrict ourselves to event E 5 , dropping the term ∥ f -y ∥ 2 ˜ P c in (D.4), then dividing by ˜ c fit and subtracting the result from the inequality in the definition of event E 5 yields

Step 2d: Choosing the constants.

If we choose c ε := 1 -c fit 4 and ˜ c fit := 2 -2 c fit 2 -c fit ∈ (0 , 1) , then (D.5) becomes

Now it is left to show that the condition ˜ c fit ∥ f ∗ -y ∥ 2 ˜ P ≤ ∥ f -y ∥ 2 ˜ P , that is required for Step 2c, holds with high probability with our choice of ˜ c fit .

With some arbitrary but fixed ε lower ∈ (0 , √ c lower ) , choose c ′′ := (1 - √ ˜ c fit )( √ c lower 1 -κ ′′ -ε lower √ 1 -κ ′′ ) σ . Then on event E 6 ii , for n large enough, it holds that

By definition of ˜ P , it holds that

so that

Now, using the triangle inequality, (D.7) and (D.6) yields the condition required for Step 2c,

Step 2e: Upper bounding the probability of E .

To conclude, we have seen in steps 2c and 2d that on E ∩ E 6 ii ∩ E 5 , it holds that

On E 6 i , it holds that

Hence E 6 i ∩ E ∩ E 6 ii ∩ E 5 = ∅ . This implies E ⊆ ( E 5 ∩ E 6 i ∩ E 6 ii ) c , where the right hand side is independent of f ∈ H and just depends on the training data D . Since P ( E 6 i ) ≥ 1 -n -1 and P ( E 6 ii ∩ E 5 ) ≥ 1 -n -1 -exp ( -n · ( 1 -c ε 2 ) 2 ) , it must hold that, for n large enough,

The following proposition generalizes the inconsistency result for small bandwidths, Proposition 5 in Buchholz (2022), beyond interpolating estimators to estimators whose RKHS norm is at most a constant factor larger than the RKHS norm of the minimum-norm interpolant. The intuition is that if the bandwidth is too small, then the minimum-norm interpolant g D,γ returns to 0 between the training points. Then ∥ g D,γ ∥ L 2 ( ρ ) is smaller and bounded away from ∥ f ∗ ∥ L 2 ( ρ ) . We can replace g D,γ by any other function f ∈ H that fulfills Assumption (N).

Proposition D.4 ( Inconsistency for small bandwidths ) . Under the assumptions of Proposition D.2, there exist constants B,c > 0 such that, with probability 1 -O ( n -1 ) over the draw of D : For any function f ∈ H that fulfills Assumption (N) but not necessarily Assumption (O), the excess risk satisfies

where c depends neither on n nor on γ < Bn -1 /d .

Proof. Denote the upper bound on the Lebesgue density of P X by C u . The triangle inequality implies

where ∥ f ∥ 2 ≤ ∥ f ∥ H follows from the fact that the Fourier transform ˆ k of the kernel satisfies ˆ k ( ξ ) ≤ 1 . Now in the proof of Lemma 17 in Buchholz (2022) a > 0 can be chosen smaller to generalize the statement to where c 9 depends on c u , f ∗ , d, s and C norm . Finally we can choose B small enough such that Eq. (32) in Buchholz (2022) can be replaced by C H √ C u ∥ g D,γ ∥ H ≤ 2 3 ∥ f ∗ ∥ L 2 ( P X ) so that we get

D.1 Auxiliary results for the proof of Theorem 1

Lemma D.5 (Concentration of χ 2 n variables) . Let U be a chi-squared distributed random variable with n degrees of freedom. Then, for any c ∈ (0 , 1) it holds that

Proof. Lemma 1 in Laurent and Massart (2000) implies for any x > 0 ,

Solving c = 1 2 √ x for x yields x = n ( 1 -c ) .

Lemma D.6. Let ε 1 , . . . , ε n be i.i.d. N (0 , σ 2 ) random variables, σ 2 > 0 . Let ( ε 2 ) ( i ) denote the i -th largest of ε 2 1 , . . . , ε 2 n .

(i) Aconstant fraction of noise cannot concentrate on less than Θ( n ) points: For all constants α, c > 0 there exists a constant C ∈ (0 , 1) such that with probability at least 1 -n -α , for n large enough,

(ii) Θ( n ) points amount to a constant fraction of noise: For all constants α > 0 and κ ∈ (0 , 1) there exists a constant c > 0 such that with probability at least 1 -n -α , for n large enough,

Proof. Without loss of generality, we can assume σ 2 = 1 .

(i) For a constant C ∈ (0 , 1) yet to be chosen, consider the sum

For T > 0 yet to be chosen, we consider the random set I T := { i ∈ [ n ] | ε 2 i > T } and denote its size by K := |I T | . To bound K , we note that K = ξ 1 + . . . + ξ n , where ξ i = 1 ε 2 i >T . We first want to bound p T := E ξ i = P ( ε 2 i > T ) .

The random variables ε 2 i follow a χ 2 1 -distribution, whose CDF we denote by F ( t ) and whose PDF is

Step 1: Tail bounds. Following Duembgen (2010), we have for x > 0 :

for some absolute constant C 1 . Moreover, we use Φ and ϕ to denote the CDF and PDF of N (0 , 1) , respectively.

Hence, for t > 0 , we have

By choosing T := -2 log( C √ π/ 32) > 0 , we obtain

Step 2: Bounding K . The random variables ξ i from above satisfy ξ i ∈ [0 , 1] . By Hoeffding's inequality (Steinwart and Christmann, 2008, Theorem 6.10), we have for τ > 0

We choose τ := C 2 n/ 2 , such that with probability ≥ 1 -exp( -C 2 n/ 2) , we have

Suppose that this holds. Then, K ≤ np T + Cn/ 2 < Cn and, since K is an integer, K ≤ ⌊ Cn ⌋ . This implies

We now want to bound ∑ K i =1 ( ε 2 ) ( i ) . To this end, we note that conditioned on K = k for some k ∈ [ n ] , the k random variables ( ε i ) i ∈I T are i.i.d. drawn from the distribution of ε 2 given ε 2 > T , for ε ∼ N (0 , 1) . By X,X 1 , X 2 , . . . , we denote i.i.d. random variables drawn from the distribution of ε 2 -T | ε 2 > T . This means that conditioned on K = k ,

Step 3: Conditional expectation. The density of X is given by

where we have used that for t > 0 , since T = -2 log( C √ π/ 32) ≥ -2 log( √ π/ 32) ≈ 1 . 008 . We can now bound

Step 4: Conditional subgaussian norm. For t ≥ 0 ,

Since the denominator 2 in 2 exp( -t/ 2) is constant, by Proposition 2.7.1 and Definition 2.7.5 in Vershynin (2018), the subexponential norm ∥ X ∥ ψ 1 is therefore bounded by an absolute constant C 3 . Moreover, by Excercise 2.7.10 in Vershynin (2018), we have ∥ X -E X ∥ ψ 1 ≤ C 4 ∥ X ∥ ψ 1 ≤ C 5 for absolute constants C 4 , C 5 .

Step 5: Conditional Concentration. Now, Bernstein's inequality for subexponential random variables (Vershynin, 2018, Corollary 2.8.1) yields for t ≥ 0 and some absolute constant C 6 > 0 :

We choose t = C 5 Cn and obtain for k ≤ Cn

Step 6: Final bound. From Step 2, we know that K ≤ ⌊ Cn ⌋ with probability ≥ 1 -exp( -C 2 n/ 2) . Moreover, in this case, Step 5 yields

with probability ≥ 1 -exp( -C 6 Cn ) . By Eq. (D.9), we therefore have

Since lim C ↘ 0 -C log( C ) = 0 , we can choose C ∈ (0 , 1) such that -4 C log( C √ π/ 32) + C 7 C < c for the given constant c > 0 from the theorem statement, and obtain the desired bound with high probability in n .

(ii) Since the ε 2 i are non-negative and their distribution has a density, there must exist T > 0 with P ( ε 2 i < T ) ≤ (1 -κ ) / 4 . Similar to the proof of (i), we then want to bound K := |{ i ∈ [ n ] | ε 2 i < T }| = ξ 1 + . . . + ξ i with ξ i = 1 ε 2 i <T . The ξ i ∈ [0 , 1] are independent with E ξ i = P ( ε 2 i < T ) ≤ (1 -κ ) / 4 . As in Step 2 of (i), Hoeffding's inequality then yields for τ > 0 :

We set τ := (1 κ ) 2 n/ 2 , such that with probability 1 exp((1 κ ) 2 n/ 2) , we have

In this case, we have

where the right-hand side is lower bounded by c := (1 -κ ) T/ 8 for n large enough.

The next lemma is a generalization of Lemma 9 in Buchholz (2022) to arbitrary fractions κ of the training points. Therefore, for any κ ∈ (0 , 1) define

Lemma D.7 (Generalization of Lemma 9 in Buchholz (2022)) . Let κ, ν ∈ (0 , 1) , and let c Ω > 0 be a constant that satisfies P X (dist( x , ∂ Ω) < c Ω ) ≤ κ . Let P = { x 1 , . . . , x n } be i.i.d. points distributed according to the measure P X , which has lower and upper bounded density on its entire

bounded open Lipschitz domain Ω ⊆ R d , C l ≤ p X ( x ) ≤ C u . Then there exists a constant Θ > 0 depending on d, C u , ν such that with probability at least 1 -exp ( -3 κn 7 ) there exists a good subset P ′ ⊆ P , |P ′ | ≥ (1 -7 κ ) n , with the following properties: For x ∈ P ′ we have dist( x , ∂ Ω) ≥ c Ω , | x -y | > δ min ( κ ) for x = y ∈ P ′ , and for all x ∈ P ′ we have

̸

Proof. First by the definition of δ min , it holds that

Also for all y ∈ Ω

for some Θ > 0 depending only on C u , d and ν . We conclude that for each j

Also P (dist ( x j , ∂ Ω) < c Ω ) < κ . The union bound implies that

We use a martingale construction similar to the one in Lemma 7 of Buchholz (2022) by defining

Now define S n := ∑ n i =1 1 E i . Using the filtration F i = σ ( x 1 , . . . , x i ) , S n can be decomposed into S n = M n + A n , where M n is a martingale and A n is predictable with respect to F n . We then get A n ≤ ∑ n i =1 P ( E i |F i -1 ) ≤ 3 κn as well as Var( M i |F i -1 ) ≤ 3 κ . Hence Freedman's inequality Theorem D.8 yields

̸

This implies that with probability at least 1 -exp ( -3 κn 7 ) we can find a subset P s = { z 1 , . . . , z m } with |P s | ≥ (1 -6 κ ) n on which it holds that min i = j | z i -z j | ≥ δ min , dist ( z j , ∂ Ω) ≥ c Ω and

̸

Using Markov's inequality we see that there are at most κn points in P s such that

̸

Removing those points we find a subset P ′ ⊂ P s such that |P ′ | ≥ (1 -7 κ ) n and for each z ∈ P ′

̸

Theorem D.8 (Freedman's inequality, Theorem 6 . 1 in Chung and Lu (2006)) . Let M i be a discrete martingale adapted to the filtration F i with M 0 = 0 that satisfies for all i ≥ 0

Then

E Translating between R d and S d

Since the RKHS of the ReLU NTK and NNGP kernels mentioned in Theorem 4 are equivalent to the Sobolev spaces H ( d +1) / 2 ( S d ) and H ( d +3) / 2 ( S d ) , respectively (Chen and Xu, 2021, Bietti and Bach, 2021) (detailed summary in Appendix B.4). Inconsistency of functions in these RKHS that fulfill Assumptions (O) and (N), as in Theorem 1, follows immediately by adapting Theorem 1 via Lemma E.1. In particular, inconsistency holds for the gradient flow and gradient descent estimators f t,ρ and f GD t,ρ as soon as they overfit with lower bounded probability.

For arbitrary open sphere caps T := { x ∈ S d | x d +1 < v } , v ∈ ( -1 , 1) , and the open unit ball B 1 (0) := { y ∈ R d | ∥ y ∥ 2 < 1 } , define the scaled stereographic projection ϕ : T → B 1 (0) ⊆ R d as where the normalization constant c v = √ 1 -v 1+ v ensures surjectivity.

Straightforward calculations show that ϕ defines a diffeomorphism. Its inverse ϕ -1 : B 1 (0) → T is given by

We can translate kernel learning with the kernel k on S d and the probability distribution P , where P X is supported on T , to kernel learning with a transformed kernel ˜ k and ˜ P using a sufficiently smooth diffeomorphism like ϕ : T → B 1 (0) ⊆ R d . If the RKHS of k is equivalent to H s ( S d ) then the RKHS of ˜ k is equivalent to H s ( B 1 (0)) . We formalize this argument in the following lemma. As a consequence it suffices to prove all inconsistency results for Sobolev kernels on B 1 (0) .

Lemma E.1 ( Transfer to sphere caps ) . Let k be a kernel on S d whose RKHS is equivalent to a Sobolev space H s ( S d ) . For fixed v ∈ ( -1 , 1) , consider an 'open sphere cap' T := { x ∈ S d | x d +1 < v } . Furthermore, consider a distribution P such that P X is supported on T and has lower and upper bounded density p X on T , i.e. 0 < C l ≤ p X ( x ) ≤ C u < ∞ for all x ∈ T . Then

˜ P := P ◦ ψ -1 with ψ ( x , y ) := ( ϕ ( x ) , y ) defines a probability distribution such that ˜ P ˜ X has lower and upper bounded density on B 1 (0) ⊆ R d ,

˜ k ( x , x ′ ) := k ( ϕ -1 ( x ) , ϕ -1 ( x ′ )) defines a positive definite kernel on B 1 (0) ⊆ R d whose RKHS is equivalent to the Sobolev space H s ( B 1 (0)) ,

and kernel learning with ( k, P ) or with ( ˜ k, ˜ P ) is equivalent in the following sense:

For every function f ∈ H ( k | T ) the transformed function ˜ f = f ◦ ϕ -1 ∈ H ( ˜ k ) has the same RKHS norm, i.e. ∥ f ∥ H ( k | T ) = ∥ ˜ f ∥ H ( ˜ k ) . Furthermore, the excess risks of f over P and of ˜ f over ˜ P coincide, i.e.

where ˜ f ∗ ˜ P (˜ x ) = E ( ˜ X, ˜ Y ) ∼ ˜ P ( ˜ Y | ˜ X = ˜ x ) denotes the Bayes optimal predictor under ˜ P .

Remark E.2. Many kernel regression estimators can be explicitly written as f k D ( x ) = ˆ f n ( k ( x , X ) , k ( X , X ) , y ) where ˆ f n : R n × R n × n × R n → R denotes a measurable function for all n ∈ N . Then the explicit form is preserved under the transformation, i.e. f ◦ ϕ -1 = f ˜ k ˜ D with the transformed data set ˜ D = { ( ϕ ( x i ) , y i ) } i ∈ [ n ] . ◀

Proof of Lemma E.1. Step 1: Bounded density. For i ∈ [ d ] , j ∈ [ d +1] , the partial derivatives of ϕ are given by

Given an arbitrary multi-index α , the partial derivatives ∂ α ϕ i ∈ L 2 ( T ) , ∂ α ϕ -1 j ∈ L 2 ( B 1 (0)) are bounded for all i ∈ [ d ] , j ∈ [ d +1] , using x d +1 ≤ v < 1 and the inverse function theorem.

Now define ˜ k ( x , x ′ ) := k ( ϕ -1 ( x ) , ϕ -1 ( x ′ )) , ψ ( x , y ) := ( ϕ ( x ) , y ) and ˜ P := P ◦ ψ -1 . Then using integration by substitution (Stroock et al., 2011, Theorem 5.2.16), the Lebesgue density of ˜ P X is given by where

Jϕ and Jϕ -1 can be continuously extended to ¯ T and ¯ B 1 (0) , respectively. Then, since Jϕ -1 is continuous on a compact set and because ϕ with the extended domain remains a diffeomorphism so that Jϕ -1 cannot attain the value 0 , there exists a constant C ϕ > 0 such that 1 C ϕ ≤ Jϕ -1 (˜ x ) ≤ C ϕ for all ˜ x ∈ B 1 (0) . Hence, p ˜ X is lower and upper bounded.

Step 2: Excess risks coincide. If ( ˜ X, ˜ Y ) ∼ ˜ P , the Bayes predictor of ˜ Y given ˜ X is given by ˜ f ∗ (˜ x ) = E ( ˜ Y | ˜ X = ˜ x ) = f ∗ ( ϕ -1 (˜ x )) .

Let π 1 ( x , y ) = x be the projection onto the first component. Then, ϕ ( π 1 ( x , y )) = ϕ ( x ) = π 1 ( ϕ ( x ) , y ) = π 1 ( ψ ( x , y )) and hence

Step 3: Transformed RKHS. We want to show that H ( k | T ) → H ( ˜ k ) , f ↦→ f ◦ ϕ -1 defines an isometric isomorphism, which especially shows the statement ∥ f ∥ H ( k | T ) = ∥ ˜ f ∥ H ( ˜ k ) from the proposition. For this, we use the following theorem characterizing RKHSs:

Theorem E.3 (Theorem 4.21 in Steinwart and Christmann (2008)) . Let k : X × X → R be a positive definite kernel function with feature space H 0 and feature map Φ 0 : X → H 0 . Then

is the only RKHS for which k is a reproducing kernel.

A feature map for k | T is given by Φ : T →H ( k | T ) , Φ( x ) = k ( x , · ) . Hence a feature map for ˜ k is given by Φ ◦ ϕ -1 : B 1 (0) →H ( k | T ) . Theorem E.3 states that

as well as

As ϕ -1 is bijective, this characterization induces an isometric isomorphism between H ( k | T ) and H ( ˜ k ) by mapping f = ⟨ w, Φ( · ) ⟩ H ( k | T ) ∈ H ( k | T ) to ˜ f = f ◦ ϕ -1 = ⟨ w, Φ ◦ ϕ -1 ( · ) ⟩ H ( k | T ) ∈ H ( ˜ k ) . This shows ∥ f ∥ H ( k | T ) = ∥ ˜ f ∥ H ( ˜ k ) .

Step 4: RKHS of ˜ k . We now show that the RKHS of ˜ k , denoted as H ( ˜ k ) , is equivalent to H s ( B 1 (0)) . To this end, denoting A◦ ϕ := { f ◦ ϕ | f ∈ A} and A| T := { f | T | f ∈ A} , we show the following equality of sets (ignoring the norms):

Since ϕ is bijective, this implies H ( ˜ k ) = H s ( B 1 (0)) as sets, and the norm equivalence then follows from Lemma F.7.

Equality (I) follows from Step 3. Equality (II) follows from Theorem E.3 by observing that if Φ is a feature map for k , then Φ | T is a feature map for k | T . Equality (III) holds by assumption. To show (IV), we need a characterization of H s ( S d ) that allows to work with charts like ϕ .

Step 4.1: Chart-based characterization of H s ( S d ) . A trivialization of a Riemannian manifold ( M,g ) with bounded geometry of dimension d consists of a locally finite open covering { U α } α ∈ I of M , smooth diffeomorphisms κ α : V α ⊂ R d → U α , also called charts, and a partition of unity { h α } α ∈ I of M that fulfills supp ( h α ) ⊆ U α , 0 ≤ h α ≤ 1 and ∑ α ∈ I h α = 1 . An admissible trivialization of ( M,g ) is a uniformly locally finite trivialization of M that is compatible with geodesic coordinates, for details see (Schneider and Große, 2013, Definition 12).

In our case, define an open neighborhood of T by U 1 := { x ∈ S d | x d +1 < v + ε } with some ε ∈ (0 , 1 -v ) arbitrary but fixed, and U 2 := { x ∈ S d | x d +1 > v + ε/ 2 } . It holds that U 1 ∪ U 2 = S d . Moreover, there exists an appropriate partition of unity consisting of C ∞ functions h 1 , h 2 : S d → [0 , 1] . Especially, we have h 1 ( T ) ⊆ h 1 ( U c 2 ) = { 1 } . Let ϕ 1 : U 1 → B r 1 (0) denote the stereographic projection with respect to x 0 = (0 , . . . , 0 , 1) as above, scaled such that ϕ 1 | T = ϕ and hence ϕ 1 ( T ) = B 1 (0) . Similarly, let ϕ 2 : U 2 → B r 2 (0) denote an arbitrarily scaled stereographic projection with respect to x 0 = (0 , . . . , 0 , -1) . Then ( { U 1 , U 2 } , { ϕ -1 1 , ϕ -1 1 } , { h 1 , h 2 } ) yields an admissible trivialization of S d consisting of only two charts. A detailed derivation can be found in (Hubbert et al., 2015, Section 1.7). Therefore (Schneider and Große, 2013, Theorem 14) lets us define the Sobolev norm on S d (up to equivalence) as 2

for any distribution g ∈ D ′ ( S d ) (i.e. any continuous linear functional on C ∞ c ( S d ) ). Then g ∈ H s ( S d ) if and only if ∥ g ∥ H s ( S d ) < ∞ .

Step 4.2: Showing (IV). First, let g ∈ H s ( S d ) . Then, as we saw in Step 4.1, we must have ∥ ( h 1 g ) ◦ ϕ -1 1 ∥ H s ( R d ) < ∞ and thus ( h 1 g ) ◦ ϕ -1 1 ∈ H s ( R d ) . By our discussion in Appendix B.1, we then have

Now, let f ∈ H s ( B 1 (0)) . Then, again following our discussion in Appendix B.1, there exists an extension ¯ f ∈ H s ( R d ) with ¯ f | B 1 (0) = f . The set B := ϕ 1 ( U 1 \ U 2 ) is a closed ball B r (0) of radius which shows g | T ∈ H s ( B 1 (0)) ◦ ϕ .

1 < r < r 1 . Hence, we can find φ ∈ C ∞ ( R d ) with φ ( B 1 (0)) = { 1 } and φ (( B r (0)) c ) = { 0 } . Since φ is smooth with compact support, we have φ · ¯ f ∈ H s ( R d ) . Define

By construction, we have f S d ( x ) = 0 for all x ∈ U 2 . Hence, the equivalent Sobolev norm from Step 4.1 is

which shows f S d ∈ H s ( S d ) . But then, f ◦ ϕ = f S d | T ∈ H s ( S d ) | T . In total, we obtain H s ( S d ) | T = H s ( B 1 (0)) ◦ ϕ , which shows (IV).

F Spectral lower bound

F.1 General lower bounds

A common first step to analyze the expected excess risk caused by label noise is to perform a biasvariance decomposition and integrate over y first (see e.g. Liang and Rakhlin, 2020, Hastie et al., 2022, Holzmüller, 2021), which is also used in the following lemma.

Lemma F.1. Consider an estimator of the form f X , y ( x ) = ( v X , x ) ⊤ y . If Var P ( y | x ) ≥ σ 2 for P X -almost all x , then the expected excess risk satisfies

Proof. A standard bias-variance decomposition lets us lower-bound the expected excess risk by the estimator variance due to the label noise, which can then be further simplified:

Here, the conditional covariance matrix can be lower bounded in terms of the Loewner order (which is defined as A ⪰ B ⇔ B -A positive semi-definite):

since the labels y i are conditionally independent given X . We therefore obtain

Proposition 5 ( Spectral lower bound ) . Assume that the kernel matrix k ( X , X ) is almost surely positive definite, and that Var( y | x ) ≥ σ 2 for P X -almost all x . Then, the expected excess risk satisfies

Proof. Recall from Eq. (1) that

By setting ( v X , x ) ⊤ := k ( x , X ) A t,ρ ( X ) , we can write f X , y ,t,ρ ( x ) := f t,ρ ( x ) = ( v X , x ) ⊤ y . Using Lemma F.1, we then obtain

Since

we conclude

Richter (1958) showed (see also Mirsky, 1959) that for two symmetric matrices B , C , we have tr( BC ) ≥ ∑ n i =1 λ i ( B ) λ n +1 -i ( C ) . We can therefore conclude

As A t,ρ ( X ) A t,ρ ( X ) ⊤ is built only out of the matrices k ( X , X ) and I n , it is not hard to see that A t,ρ ( X ) A t,ρ ( X ) ⊤ has the same eigenbasis as k ( X , X ) with eigenvalues

It remains to order these eigenvalues correctly. To this end, we observe that for λ > 0 , the function g ( λ ) := 1 -e -2 tλ λ satisfies

Therefore, g is nonincreasing, hence the sequence ( ˜ λ i ) is nondecreasing and thus

from which the claim follows.

Theorem F.2. Let k be a kernel on a compact set Ω and let P X be supported on Ω . Suppose that k ( X , X ) is almost surely positive definite and that Var( y | x ) ≥ σ 2 for P X -almost all x . Fix constants c > 0 and q, C ≥ 1 . Suppose that λ i := λ i ( T k,P X ) ≥ ci -q . Let I ( n ) be the set of all i ∈ [ n ] for which

both hold at the same time with probability ≥ 1 / 2 . Moreover, let I ( n ) := max { m ∈ [ n ] | [ m ] ⊆ I ( n ) } . Then, there exists a constant c ′ > 0 depending only on c, C such that for all ρ ∈ [0 , ∞ ) and t ∈ (0 , ∞ ] , the following two bounds hold:

Remark F.3. Theorem F.2 provides two lower bounds, one for general 'concentration sets' I ( n ) and one that applies if concentration holds for a sequence of 'head eigenvalues' { 1 , . . . , I ( n ) } ⊆ I ( n ) . If I ( n ) ≈ |I ( n ) | , the latter bound is stronger for larger regularization levels, and this bound would be particularly suitable for typical forms of relative concentration inequalities for kernel matrices. However, in this paper, we obtain concentration only for 'middle eigenvalues' I ( n ) = { i | εn ≤ i ≤ (1 -ε ) n } , and therefore we only use the first bound in the proof of Theorem 6. ◀

Proof of Theorem F.2. Step 1: Miscellaneous inequalities. For x > 0 ,

Moreover, since (1 + a ) 2 ≤ (1 + a ) 2 +(1 -a ) 2 = 2 + 2 a 2 , we have for a = -1 :

Step 2: Applying the eigenvalue bound. Define

By Proposition 5, we have

Since S i ( X ) is almost surely positive, we can focus on the case where (F.1) and (F.2) hold, which is true with probability ≥ 1 / 2 by assumption for i ∈ I ( n ) . Hence,

We can upper-bound the denominator, using C ≥ 1 , as which yields

Step 3: Analyzing the sum. We want to analyze the behavior of the sum

for β := ρ + t -1 c > 0 . We first obtain the trivial bound

̸

Moreover, we can bound and distinguish three cases:

(a) If β ≥ 1 , we bound

(b) If β ∈ ( I ( n ) -q , 1) , we observe that

and therefore

(c) If β ∈ (0 , I ( n ) -q ] , we similarly find that

Moreover, there is an absolute constant c 1 > 0 such that for any β > 0 , because

(a) β -2 = min { β -2 , β -1 /q , I ( n ) } for β ≥ 1 ,

(c) I ( n ) = min { β -2 , β -1 /q , I ( n ) } for β ∈ (0 , I ( n ) -q ] .

Step 4: Putting it together. Combining the trivial bound in Step 3 with Eq. (F.4) and Eq. (F.5), we obtain

for a suitable constant c ′ > 0 depending only on c and C .

Moreover, from Eq. (F.6), we obtain

for a suitable constant ˜ c ′′ > 0 depending only on c . Again, (F.4) and (F.5) yield

F.2 Equivalences of norms and eigenvalues

Later, we will use concentration inequalities for kernel matrix eigenvalues proved for specific kernels, which we then want to transfer to other kernels with equivalent RKHSs. In this subsection, we show that this is possible.

Definition F.4 ( C -equivalence of matrices and norms) . Let n ≥ 1 and let K , ˜ K ∈ R n × n be symmetric. For C ≥ 1 , we say that K and ˜ K are C -equivalent if their ordered eigenvalues satisfy for all i ∈ [ n ] . Moreover, we say that two norms ∥· ∥ A , ∥· ∥ B on a vector space V are C -equivalent if

for all v ∈ V .

◀

Lemma F.5. Let n ≥ 1 and let K , ˜ K ∈ R n × n be symmetric. Then, K and ˜ K are C -equivalent iff the Moore-Penrose pseudoinverses K + and ˜ K + are C -equivalent.

Proof. This follows from the fact that if K has eigenvalues λ 1 , . . . , λ n , then K + has eigenvalues 1 /λ 1 , . . . , 1 /λ n , where we define 1 / 0 := 0 . (A detailed proof would be a bit technical due to the sorting of eigenvalues.)

Lemma F.6. Let k : X × X → R be a kernel on a set X . Then, for any y ∈ R n , where H k is the RKHS associated with k and f ∗ k, y is the minimum-norm regression solution

Proof. It is well-known that f ∗ k, y ( x ) = ∑ n i =1 α i k ( x , x i ) , where α := K + y (see e.g. Rangamani et al., 2023). We then have

where the last step follows from a standard identity for the Moore-Penrose pseudoinverse (see e.g. Section 1.1.1 in Wang et al., 2018).

Lemma F.7. Let H 1 and H 2 be two RKHSs with H 1 ⊂ H 2 . Then there exists a constant C > 0 such that ∥ f ∥ H 2 ≤ C ∥ f ∥ H 1 .

Proof. Let I : H 1 →H 2 be the inclusion map, i.e. I h := h for all h ∈ H 1 . Obviously, I is linear and we need to show that I is bounded. To this end, let ( h n ) n ≥ 1 ⊂ H 1 be a sequence such that there exist h ∈ H 1 and g ∈ H 2 with h n → h in H 1 and Ih n → g in H 2 . This implies h n → h pointwise and h n = Ih n → g pointwise, which in turn gives h = g . The closed graph theorem, see e.g. (Megginson, 1998, Theorem 1.6.11), then shows that I is bounded.

Applying Lemma F.7 twice shows that RKHSs H 1 and H 2 with H 1 = H 2 automatically have C -equivalent norms for a suitable constant C ≥ 1 . The following result investigates the corresponding kernels.

Proposition F.8 (Equivalent kernels have equivalent kernel matrices) . Let k, ˜ k : X × X → R be kernels such that their RKHSs are equal as sets and the corresponding RKHS-norms are C -equivalent as defined in Definition F .4. Then, for any n ≥ 1 and any x 1 , . . . , x n ∈ X , the corresponding kernel matrices k ( X , X ) , ˜ k ( X , X ) are C 2 -equivalent.

Proof. Let i ∈ [ n ] . For y ∈ R n we have, using the notation of Lemma F.6:

Now, by the Courant-Fischer-Weyl theorem,

By switching the roles of k and ˜ k , we obtain that k ( X , X ) + and ˜ k ( X , X ) + are C 2 -equivalent. By Lemma F.5 k ( X , X ) and ˜ k ( X , X ) are then also C 2 -equivalent.

To prove Theorem 6 for arbitrary input distributions P X with lower and upper bounded densities, we need the following theorem investigating the corresponding eigenvalues of the integral operator.

LemmaF.9 (Integral operators for equivalent densities have equivalent eigenvalues) . Let k : X×X → R be a kernel and let µ, ν be finite measures on X whose support is X such that ν has an lower and upper bounded density w.r.t. µ . Then, λ i ( T k,ν ) = Θ( λ i ( T k,µ )) .

Proof. Let p be such an upper bounded density, that is, d ν = p d µ and there exist c, C > 0 such that c ≤ p ( x ) ≤ C for all x ∈ X . For f ∈ L 2 ( ν ) , we have

Hence, the linear operator

is well-defined and continuous. It is also easily verified that A is bijective. Moreover, we have

and

Since T k,µ and T k,ν are compact, self-adjoint, and positive, we can use the Courant-Fischer minmax principle for operators (see e.g. Bell, 2014) to obtain

Here, we have used that since A is bijective, the subspaces AV for dim( V ) = i are exactly the i -dimensional subspaces of L 2 ( µ ) . Our calculation above shows that λ i ( T k,µ ) ≤ O ( λ i ( T k,ν )) . Since d µ = 1 p d ν with the lower and upper bounded density 1 /p , we can reverse the roles of ν and µ to also obtain λ i ( T k,ν ) ≤ O ( λ i ( T k,µ )) , which proves the claim.

Lemma F.10 (Integral operators of equivalent kernels have equivalent eigenvalues) . Let k, ˜ k : X × X → R be bounded kernels with RKHSs H and ˜ H satisfying H = ˜ H as sets. Moreover, let C ≥ 1 be a constant such that the corresponding RKHS-norms are C -equivalent and let ν be a finite measure on X . If there exist constants q > 0 and c > 0 with then we also have

where K q > 0 is a constant only depending on q .

Proof. Wefollow the ideas outlined in (Steinwart, 2017, Section 3). To this end, let I k,ν : H → L 2 ( ν ) be the embedding h ↦→ [ h ] ∼ , which is defined and compact since k is bounded and ν is finite, see e.g. (Steinwart and Scovel, 2012, Lemma 2.3). We write S k,ν := I ∗ k,ν for its adjoint, which in turn gives I k,ν = S ∗ k,ν . Then (Steinwart and Scovel, 2012, Lemma 2.2) shows T k,ν = S ∗ k,ν ◦ S k,ν . We denote the i -th (dyadic) entropy number of I k,ν by ε i ( I k,ν ) , see e.g. Carl and Stephani (1990), (Edmunds and Triebel, 1996, Chapter 1.3.1), or (Steinwart and Christmann, 2008, Chapter 6) for

a definition 3 . Moreover, we denote the i -approximation and singular numbers of I k,ν by a i ( I k,ν ) , respectively s i ( I k,ν ) . Since I k,ν compactly acts between Hilbert spaces, we then have a i ( I k,ν ) = s i ( I k,ν ) , see e.g. (Pietsch, 1987, Chapter 2.11). This implies

for all i ≥ 1 by the very definition of singular numbers. Finally, analogous definitions and considerations are made for the kernel ˜ k . From C -1 ∥ · ∥ H ≤ ∥ · ∥ ˜ H ≤ C ∥ · ∥ H we then conclude that

by the multiplicativity of entropy numbers, see e.g. (Edmunds and Triebel, 1996, Chapter 1.3.1). Now, (F.8) and our eigenvalue assumption yield

and Carl's inequality, see e.g. (Carl and Stephani, 1990, Theorem 3.1.1) then gives

where ˜ K q is a constant only depending on q . By (Carl and Stephani, 1990, Inequality (3.0.9)) and (F.9) we then obtain

Another application of (F.8) then yields the assertion for K q := 4 ˜ K 2 q .

F.3 Kernel matrix eigenvalue bounds

For upper bounds on the eigenvalues of kernel matrices, we use the following result:

Proposition F.11 (Kernel matrix eigenvalue upper bound in expectation) . For m ≥ 1 , we have

Proof. Theorem 7.29 in Steinwart and Christmann (2008) shows that

where T k,µ : L 2 ( µ ) → L 2 ( µ ) , f ↦→ ∫ k ( x, · ) f ( x ) d µ ( x ) is the integral operator corresponding to the measure µ and T k,D is the corresponding discrete version thereof. We set µ := P X and need to show that k ( X , X ) /n has the same eigenvalues as T k,D if D and X contain the same data points x 1 , . . . , x n . Consider a fixed D . Then, we can write T k,D ( f ) = n -1 ABf , where

Then, k ( X , X ) /n is the matrix representation of n -1 BA with respect to the standard basis of R n . But AB and BA have the same non-zero eigenvalues, which means that

from which the claim follows.

To obtain a lower bound, we want to leverage the lower bound by Buchholz (2022) for a certain radial basis function kernel with data generated from an open subset of R d . However, we want to consider different kernels and distributions on the whole sphere. The following theorem bridges the gap by going to subsets of the data on a sphere cap, projecting them to R d , and using the kernel equivalence results from Appendix F.2:

Theorem F.12 (Kernel matrix eigenvalue lower bound for Sobolev kernels on the sphere) . Let k be a kernel on S d such that its RKHS H k is equivalent to a Sobolev space H s ( S d ) with smoothness s > d/ 2 . Moreover, let P X be a probability distribution on S d with lower and upper bounded density. Let the rows of X ∈ R n × d are drawn independently from P X . Then, for ε ∈ (0 , 1 / 20) , there exists a constant c > 0 and n 0 ∈ N such that for all n ≥ n 0 ,

holds with probability ≥ 4 / 5 for all m ∈ N with 1 ≤ m ≤ (1 -11 ε ) n .

Proof. We can choose a suitably large sphere cap T such that P X ( T ) ≥ 1 -ε . Define the conditional distribution P T ( · ) := P X ( ·| T ) . Out of the points X = ( x 1 , . . . , x n ) , we can consider the submatrix X T = ( x i 1 , . . . , x i N ) ⊤ of the points lying in T . Conditioned on N , these points are i.i.d. samples from P T . Moreover, by applying Markov's inequality to a Bernoulli distribution, we obtain N ≥ (1 -10 ε ) n with probability ≥ 9 / 10 . We fix a value of N ≥ (1 -10 ε ) n in the following and condition on it.

We denote the centered unit ball in R d by B 1 ( R d ) . Using a construction as in Lemma E.1, we can transport k and P T from T to the unit ball B 1 ( R d ) using a rescaled stereographic projection feature map ϕ , such that we obtain a kernel k ϕ and a distribution P ϕ = ( P T ) ϕ on B 1 ( R d ) that generate the same distribution of kernel matrices as k with P T , and such that H k ϕ ∼ = H s ( B 1 ( R d )) . The rows of X ϕ := ϕ ( X T ) are i.i.d. samples from P ϕ . Moreover, we know that P ϕ has an lower and upper bounded density w.r.t. the Lebesgue measure on B 1 ( R d ) .

In order to apply the results from Buchholz (2022), we define a translation-invariant reference kernel on R d through the Fourier transform

see Eq. (3) in Buchholz (2022). The RKHS of k ref on R d is equivalent to the Sobolev space H s ( R d ) . Therefore, the RKHS of k ref | B 1 ( R d ) ,B 1 ( R d ) is H s ( B 1 ( R d )) , cf. the remarks in Appendix B.1 and Lemma F.7.

Now, let 1 ≤ m ≤ (1 -11 ε ) n , which implies

We apply Theorem 12 by Buchholz (2022) with bandwidth γ = 1 and α = 2 s to λ m and obtain with probability at least 1 -2 /N :

as long as N is large enough such that (1 -ε ) N < N -32 ln( N ) , which is the case if n is large enough. Here, the constant c 3 from Buchholz (2022) does not depend on N or m , but only on α , d , and the upper and lower bounds on the density, which in our case depend on ε through the choice of T . Since α = 2 s > d , we have n α/d -1 > 1 and therefore

Now, we want to translate this to the kernel k . Since the RKHSs of k ϕ and k ref on B 1 ( R d ) are both equivalent to H s ( B 1 ( R d )) , the kernels themselves are C -equivalent for some constant C ≥ 1 as defined in Definition F.4. Therefore, Proposition F.8 shows that the corresponding kernel matrices are C 2 -equivalent, which implies

By Cauchy's interlacing theorem, we therefore have

Denoting the event where λ m ( k ∗ ( X , X ) /n ) ≥ c 5 C -2 n -2 s/d by A , we thus have where the last step holds for sufficiently large n .

F.4 Spectral lower bound for dot-product kernels on the sphere

An application of the spectral generalization bound in Proposition 5 requires a lower bound on eigenvalues of the kernel matrix k ∗ ( X , X ) . To achieve this, we need to understand the properties of the convolution kernel k ∗ . Since the eigenvalues of T k ∗ ,P X are the squared eigenvalues of T k,P X , one might hope that if H k is equivalent to a Sobolev space H s , then H k ∗ is equivalent to a Sobolev space H 2 s . Unfortunately, this is not the case in general, as H k ∗ might be a smaller space that involves additional boundary conditions (Schaback, 2018). However, perhaps since the sphere is a manifold without boundary, the desired characterization of H k ∗ holds for dot-product kernels on the sphere:

Lemma F.13 (RKHS of convolution kernels) . Let k be a dot-product kernel on S d such that its RKHS H k is equivalent to a Sobolev space H s ( S d ) with smoothness s > d/ 2 , and let P X be a distribution on S d with lower and upper bounded density. Then, the RKHS H k ∗ of the kernel

is equivalent to the Sobolev space H 2 s ( S d ) .

Proof. Define

For the corresponding integral operator, we have

This means that the corresponding eigenvalues are the squares of the eigenvalues of the corresponding integral operator of k . Especially, we obtain the Mercer representations

where Lemma B.1 yields µ l = Θ(( l + 1) -2 s ) , hence µ 2 l = Θ(( l + 1) -4 s ) and hence H k ∗ , unif ∼ = H 2 s ( S d ) .

Next, we show the equality of the ranges of the integral operators:

Let p X be a density of P X w.r.t. the uniform distribution U ( S d ) . If f ∈ R ( T k, U ( S d ) ) , there exists g ∈ L 2 ( U ( S d )) with f = T k, U ( S d ) g . But then, since p X is lower bounded, we have g/p X ∈ L 2 ( P X ) and therefore

The equality of the ranges yields for the RKHSs (as sets)

An analogous argument shows that R ( T k,P X ) ⊆ R ( T k, U ( S d ) ) since p X is upper bounded.

Applying Lemma F.7 twice then shows H k ∗ ∼ = H 2 s ( S d ) .

Theorem 6 ( Inconsistency for Sobolev dot-product kernels on the sphere ) . Let k be a dot-product kernel on S d , i.e., a kernel of the form k ( x , x ′ ) = κ ( ⟨ x , x ′ ⟩ ) , such that its RKHS H k is equivalent to a Sobolev space H s ( S d ) , s > d/ 2 . Moreover, let P be a distribution on S d × R such that P X has a lower and upper bounded density w.r.t. the uniform distribution U ( S d ) , and such that Var( y | x ) ≥ σ 2 > 0 for P X -almost all x ∈ S d . Then, for every C > 0 , there exists c > 0 independent of σ 2 such that for all n ≥ 1 , t ∈ ( C -1 n 2 s/d , ∞ ] , and ρ ∈ [0 , Cn -2 s/d ) , the expected excess risk satisfies

Proof. Step 0: Preparation. Since the Sobelev space H 2 s ( S d ) is dense in the space of continuous functions S d → R , the kernel k is universal. Applying (Steinwart and Christmann, 2008, Corollary 5.29 and Corollary 5.34) for the least squares loss thus shows that k is strictly positive definite. If we have mutually distinct x 1 , . . . , x n , the corresponding Gram matrix k (( x i , x j )) n i,j =1 is therefore invertible. Now, our assumptions on P guarantee that X consists almost surely of mutually distinct observations, and therefore k ( X , X ) is almost surely invertible.

By Proposition 5, we know that

for a suitable constant c n > 0 depending on n but not on σ 2 , t, ρ , since the kernel matrix eigenvalues are nonzero almost surely. It is therefore sufficient to show the desired statement (with c independent of n, σ 2 , t, ρ ) for sufficiently large n .

In the following, we assume n ≥ 40 and set ε := 1 / 100 .

Step 1: Eigenvalue decay for the integral operator. From Lemma B.1, we know that

Therefore, by Lemma F.9, we know that

Step 2: Eigenvalue upper bound. Next, we want to upper-bound suitable eigenvalues of the form λ i ( k ( X , X ) /n ) using Proposition F.11. Using Step 1, we derive

with constants independent of m ≥ 1 . For sufficiently large n , we can choose m ∈ N ≥ 1 such that εn ≤ m ≤ 2 εn . Then, Proposition F.11 yields

Since E X λ i ( k ( X , X ) /n ) is decreasing with i , we have for i ≥ 4 εn ≥ 2 m :

Step 3: Eigenvalue lower bounds. From Lemma F.13, we know that H k ∗ ∼ = H 2 s ( S d ) . Therefore, we can apply Lemma F.13 to both k and k ∗ and obtain for sufficiently large n and suitable constants c 1 , c 2 > 0 that

Step 4: Final result. Now, using the value of m from Step 2, consider an index i with 2 m ≤ i ≤ (1 -11 ε ) n . Since 2 m ≤ 4 εn and ε = 1 / 100 , there are at least n/ 2 such indices. By combining Step 3 and Step 1, we have individually hold with probability ≥ 4 / 5 for all i ∈ N with 1 ≤ i ≤ (1 -11 ε ) n . By the union bound, both bounds hold at the same time with probability ≥ 3 / 5 .

with probability ≥ 3 / 5 . By applying Markov's inequality to Step 2, we obtain with probability ≥ 9 / 10 . Therefore, by the union bound, all three inequalities hold simultaneously with probability ≥ 1 / 2 . Moreover, for q = 2 s/d , we have λ i ( T k,P X ) ≥ c 5 i -q by Step 1. We can thus apply the first lower bound from Theorem F.2 to obtain

G Proof of Theorem 8

Here we denote the solution of kernel ridge regression on D with the kernel function k and regularization parameter ρ > 0 as

and write ˆ f k 0 ( x ) = k ( x , X ) k ( X , X ) + y for the minimum-norm interpolant in the RKHS of k .

While Theorem 1 states that overfitting kernel ridge regression using Sobolev kernels is always inconsistent as long as the derivatives remain bounded by the derivatives of the minimum-norm interpolant of the fixed kernel (Assumption (N)), here we show that consistency over a large class of distributions is achievable by designing a kernel sequence, which can have Sobolev RKHS, that consists of a smooth component for generalization and a spiky component for interpolation.

Recall that ˜ k denotes any universal kernel function for the smooth component, and ˇ k γ denotes the kernel function of the spiky component with bandwidth γ . Then we define the ρ -regularized spiky-smooth kernel with spike bandwidth γ as

Let B t ( x ) := { y ∈ R d | | x -y | ≤ t } denote the Euclidean ball of radius t ≥ 0 around x ∈ R d .

(D2) There exists a constant β X > 0 and a continuous function ϕ : [0 , ∞ ) → [0 , 1] with ϕ (0) = 0 such that P X ( B t ( x )) ≤ ϕ ( t ) = O ( t β X ) for all x ∈ Ω and all t ≥ 0 .

The kernel ˇ k γ of the spiky component should fulfill the following weak assumption on its decay behaviour. For example, Laplace, Matérn, and Gaussian kernels all fulfill Assumption (SK).

(SK) There exists a function ε : (0 , ∞ ) × [0 , ∞ ) → [0 , 1] such that for any bandwidth γ > 0 and any δ > 0 it holds that

(ii) ε ( γ, δ ) is monotonically increasing in γ ,

(i) ε ( γ, 0) = 1 ,

(iii) For all x , y ∈ Ω , if | x -y | ≥ δ then | ˇ k γ ( x , y ) | ≤ ε ( γ, δ ) ,

(iv) For any rates β X , β k > 0 there exists a rate β γ > 0 such that, if δ n = Ω( n -β X ) and γ n = O ( n -β γ ) , then ε ( γ n , δ n ) = O ( n -β k ) .

Theorem G.1 ( Consistency of spiky-smooth ridgeless kernel regression ) . Assume that the training set D consists of n i.i.d. pairs ( x , y ) ∼ P such that the marginal P X fulfills (D2) and E y 2 < ∞ . Let the kernel components satisfy:

ˇ k γ n denotes a kernel function that fulfills Assumption (SK) with a sequence of positive bandwidths ( γ n ) fulfilling γ n = O (exp( -βn )) for some arbitrary β > 0 .

˜ k denotes an arbitrary universal kernel, and ρ n → 0 and nρ 4 n →∞ .

Then the minimum-norm interpolant of the ρ n -regularized spiky-smooth kernel sequence k n := k ρ n ,γ n is consistent for P .

Remark G.2 ( Spike bandwidth scaling ) . Under stronger assumptions on ϕ and ε in assumptions (D2) and (SK), the spike bandwidths γ n can be chosen to converge to 0 at a much slower rate. For example, if we choose ˇ k γ to be the Laplace kernel, choosing bandwidths 0 < γ n ≤ δ β ln n yields, for separated points | x -y | ≥ δ ,

For probability measures with upper bounded Lebesgue density, we can choose δ n = n -2+ α d and β = 9 4 + α 2 for consistency or β = 11 4 + α 2 for optimal convergence rates, for any fixed α > 0 , in the proof of Theorem 8. Hence the Laplace kernel only requires a slow bandwidth decay rate of

γ n = Ω ( n -2+ α d α ln( n ) ) , where α > 0 arbitrary. For the Gaussian kernel an analogous argument yields

γ n = Ω ( n -4+2 α d α ln( n ) ) . The larger the dimension d , the slower the required bandwidth decay.

◀

Remark G.3 ( Generalizations ) . If one does not care about continuous kernels, one could simply take a Dirac kernel as the spike and then obtain consistency for all atom-free P X . However, we need a continuous kernel to be able to translate it to an activation function for the NTK. Beyond kernel regression, the spike component ˇ k γ does not even need to be a kernel, it just needs to fulfill Assumption (SK) or a similar decay criterion. Then one could still use the 'quasi minimum-norm estimator' x ↦→ ( ˜ k + ρ n ˇ k γ n )( x , X ) · ( ˜ K + ρ n ˇ K γ n ) + y . ◀

Remark G.4 ( Consistency with a single kernel function ) . Without resorting to kernel sequences as we do, there seems to be no rigorous proof showing that ridgeless kernel regression can be consistent in fixed dimension. In future work, can an analytical expression of such a kernel be found? According to the semi-rigorous results in Mallinar et al. (2022) a spectral decay like λ k = Θ( k -1 · log α ( k )) , α > 1 could lead to such a kernel. ◀

Proof of Theorem G.1. Given any universal kernel, (Steinwart, 2001, Theorem 3.11 or Example 4.6) implies universal consistency of kernel ridge regression if ρ n → 0 and nρ 4 n →∞ . Hence, for any ε > 0 it holds that

Due to the triangle inequality in L 2 ( P X ) , we know

̸

̸

This in turn implies

Using ∥ ( ˜ K + ρ n I n ) -1 ∥ ≤ 1 λ min ( ˜ K )+ ρ n ≤ ρ -1 n and ∥ ˇ K γ n -I n ∥ ≤ ( n -1) ε n , Lemma G.8 implies

It is left to show that k n fulfills

For this purpose we decompose the above difference into the difference of ˇ K γ n := ˇ k γ n ( X , X ) and I n and a remainder term depending on ˇ k γ n . We denote the 2 -operator norm by ∥·∥ and the Euclidean norm in R n by | · | . For any x ∈ R d it holds that

Consequently we get

We now bound the individual terms in Eq. (G.1) and (G.2). To this end, fix any α > 0 . Bounding Eq. (G.1) :

Since we assumed y i i.i.d. and E y 2 1 < ∞ , the Markov inequality implies, with b n = E y 2 1 · n α ,

Stated differently, with probability at least 1 -n -α it holds that | y | 2 ≤ E y 2 1 · n 1+ α .

In order to bound the spectrum of ˇ K γ n , Lemma G.7 implies that there exists a positive sequence δ α ( n ) = n -2+ α β X such that with probability at least 1 -O ( n -α ) it holds that

̸

Since ( γ n ) fulfills γ n = O ( n -β γ ) for any β γ > 0 , by Assumption (SK) there exists a sequence ε n = o ( ρ n n -2 -α 2 ) such that ε ( γ n , δ α ( n )) ≤ ε n . Assumption (SK) further implies that whenever min i,j ∈ [ n ]: i = j | x i -x j | ≥ δ α ( n ) it holds that ( ˇ K γ n ) ii = 1 and 0 ≤ ( ˇ K γ n ) ij ≤ ε ( γ n , δ α ( n )) ≤ ε n for i = j . Then Gershgorin's theorem (Gerschgorin, 1931) implies that for all eigenvalues of ˇ K γ n

Using | ˜ k ( x , X i ) | ≤ 1 for all i ∈ [ n ] yields the naive bound ∥ ˜ k ( x , X ) ∥ 2 ≤ n .

Bounding Eq. (G.2) :

Combining all terms in Eq. (G.1) yields its convergence to 0 as the product satifies the rate O ( n 4+ α ρ -2 n ε 2 n ) = o (1) with probability at least 1 -2 n -α .

The analysis below is restricted to the event of probability at least 1 -2 n -α , on which the bound on Eq. (G.1) holds.

Since ( n -1) ε n → 0 , for any C > 1 it holds for n large enough,

Finally we show sup x ′ ∈ R d E x ˇ k γ n ( x , x ′ ) 2 ≤ 2 n -(2+ α ) for n large enough.

Fix an arbitrary x ′ ∈ R d . Then by construction of δ α ( n ) and ε n it holds that

Since ε 2 n = o ( ρ 2 n n -4 -α ) , we get E x ˇ k γ n ( x , x ′ ) 2 ≤ 2 n -(2+ α ) for n large enough.

Combining all terms in Eq. (G.2) yields its convergence to 0 with the rate O ( n -(2+ α ) · 1 · n 1+ α ) = O ( n -1 ) with probability at least 1 -2 n -α , which concludes the proof.

The following theorem shows that the minimum-norm interpolants of the spiky-smooth kernel sequence can achieve optimal convergence rates for Sobolev target functions, as long as ρ n is properly chosen. We therefore introduce Assumption (D3), which resembles Assumption (D1) but allows more general target functions f ∗ ∈ H s ∗ (Ω) \ { 0 } , s ∗ > 0 , that may lie outside of the RKHS.

(D3) Let Ω = S d or let Ω ⊆ R d be a bounded open Lipschitz domain. Let P X be a distribution on Ω with lower- and upper-bounded Lebesgue density. Consider i.i.d. data sets D = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } ⊆ Ω × R , where x i ∼ P X , f ∗ ( x ) = E [ y | x ] ∈ H s ∗ (Ω) \ { 0 } , s ∗ > 0 , with ∥ f ∗ ∥ L ∞ ( P X ) < B ∞ for some constant B ∞ > 0 , E y 2 < ∞ and there are constants σ, L > 0 such that

for P X -almost all x ∈ Ω and all m ≥ 2 .

The above moment condition holds for additive Gaussian noise with variance σ 2 > 0 . Hence Assumption (D1) is strictly stronger than Assumption (D3). The spike components ˇ k γ n can also be chosen as in Theorem G.1.

Theorem G.5. Assume Assumption (D3) holds and that the kernel components satisfy:

the RKHS ˜ H of ˜ k satisfies ˜ H = H s as sets with s > max( s ∗ , d/ 2) ,

Then there exists a constant C > 0 independent of n and there is a sequence ( ρ n ) n ∈ N of order n -s/ ( s ∗ + d/ 2) such that the minimum-norm interpolant f ρ n ,γ n of the ρ n -regularized spiky-smooth kernel sequence k n := k ρ n ,γ n fulfills, with probability at least 1 -6 n -(1 ∧ α ) , for n large enough,

ˇ k γ n denotes the Laplace kernel with a sequence of positive bandwidths ( γ n ) fulfilling γ n ≤ n -2+ α d ( ( 11 4 + α 2 ) ln n ) -1 , where α > 0 is arbitrary.

Proof. Step 1: Kernel ridge regression ˆ f ˜ k ρ n with optimal regularization achieves the desired convergence rate, with high probability.

We slightly modify the proof of Theorem 8. Instead of using (Steinwart, 2001, Theorem 3.11 or Example 4.6), we use results of Fischer and Steinwart (2020). Here we first note that in the case

H = H s (Ω) , Ω ⊆ R d as RKHSs, we could directly use (Fischer and Steinwart, 2020, Corollary 5). Since we only have equivalent norms and also want to consider Ω = S d , see Lemma F.7, we need to resort to the underlying more general result (Fischer and Steinwart, 2020, Theorem 1). To this end, we first need to verify its Assumptions (EMB), (EVD), (SRC), and (MOM).

Step 1.1: Verifying (MOM). The moment condition (MOM) on the noise distributions holds since we assumed it in Assumption (D3).

Step 1.2: Simpler equivalent spaces. We verify the remaining conditions by analyzing them for a nicer equivalent RKHS H and with uniform distribution ν on Ω . For the non-spherical case Ω ⊆ R d , we choose H := H s (Ω) . For the case Ω = S d , we choose H as an RKHS associated to a dot-product kernel k with µ l = Θ(( l +1) -2 s ) , such that H ∼ = H s ∼ = ˜ H by Lemma B.1. In each case, H = ˜ H with equivalent norms, and L 2 ( ν ) = L 2 ( P X ) with equivalent norms since we assumed in (D3) that P X has an upper- and lower-bounded density.

Step 1.3: Verifying (EVD+) . It suffices to verify the eigenvalue decay condition (EVD+) for H and ν , since Lemma F.10 and Lemma F.9 then allow to transfer it to ˜ H and P X .

For Ω = S d , our definition of H together with Lemma B.1 directly yields (EVD+) for H and ν with p := d 2 s .

For Ω ⊆ R d , as pointed out in front of (Fischer and Steinwart, 2020, Corollary 5), it is well-known that H satisfies the polynomial eigenvalue decay assumption (EVD+) for p := d 2 s .

Step 1.4: Verifying (EMB) and (SRC). The remaining two conditions of (Fischer and Steinwart, 2020, Theorem 1) are stated in terms of so-called power spaces, which in turn can be described by interpolation spaces of the real method. We therefore quickly recall these spaces. To this end, let us assume that we have two Banach spaces E and F such that F ⊂ E and the corresponding inclusion map is continuous. Then the so-called K -functional of an x ∈ E is defined by

For q ∈ (0 , 1) and x ∈ E we then define

and [ E,F ] q, 2 := { x ∈ E : ∥ x ∥ q, 2 < ∞} . Let us now consider the cases ( E,F ) = ( L 2 ( ν ) , H ) and ( E,F ) = ( L 2 ( P X ) , ˜ H ) . Now, for a suitable constant C , H and ˜ H are C -equivalent, and L 2 ( ν ) and L 2 ( P x ) are also C -equivalent. We then find

for all t > 0 and f ∈ L 2 (Ω) , and consequently we have for all q ∈ (0 , 1) with C -equivalent norms. Now, (Steinwart and Scovel, 2012, Theorem 4.6) shows that the power space [ H ] q ν defined in (Steinwart and Scovel, 2012, Equation (36)) satisfies

with equivalent norms, and an analogous result is true for ˜ H and P X . Moreover, H is dense in L 2 ( ν ) , and therefore (Steinwart and Scovel, 2012, Equations (36) and (18)) together with (Steinwart and Scovel, 2012, Lemma 2.2) show [ H ] 0 ν = L 2 ( ν ) as spaces. Again, we analogously find [ ˜ H ] 0 P X = L 2 ( P X ) Consequently, for all 0 ≤ q < 1 we have with equivalent norms. From this we easily deduce that the Assumptions (EMB) and (SRC) are satisfied for ( ˜ H , P X ) if and only if they are satisfied for ( H , ν ) .

Step 1.4.1: Non-spherical case. Now, let Ω ⊆ R d . Then, (EMB) and (SRC) are satisfied for ( H , ν ) for β := s ∗ /s and an arbitrary but fixed α ∈ ( p, min { 1 , p + β } ) as outlined in front of (Fischer and Steinwart, 2020, Corollary 5). Applying Part ii) of (Fischer and Steinwart, 2020, Theorem 1) for

γ = 0 then shows that there exists a sequence ( ρ n ) n ∈ N of order n -s/ ( s ∗ + d/ 2) and a constant K 1 > 0 independent of n such that, for n large enough,

Step 1.4.2: Spherical case. Suppose Ω = S d . Using the differently normalized spherical harmonics Y l,i and ˜ Y l,i from Appendix B.3, we obtain for the power spaces:

where the first equation follows from (Steinwart and Scovel, 2012, Eq. (36)), the second one from our definition of the µ l in Step 1.2, and the last one from (Hubbert et al., 2023, Section 3). Again, we can choose an arbitrary but fixed α ∈ ( p, min { 1 , p + β } ) . Then, αs > ps = d/ 2 , which means that [ H ] α ν = H αs is an RKHS with bounded kernel (De Vito et al., 2021, Theorem 8), hence the embedding condition (EMB) holds. Similarly, (SRC) holds for β := s ∗ /s and the result follows as above.

Step 2: ˆ f k n 0 and ˆ f ˜ k ρ n are close in L 2 ( P X ) , with high probability.

Since s ∗ ( s ∗ + d/ 2) < 1 , it suffices to show that k n fulfills, for some constant K 2 > 0 ,

Since γ n ≤ n -2+ α d ( ( 11 4 + α 2 ) ln n ) -1 , it holds that | ˇ k γ n ( x , y ) | ≤ ε n := ρ n n -5+ α 2 (cf. Remark G.2). Then the product of all terms in Eq. (G.1) satisfies O ( n 4+ α ρ -2 n ε 2 n ) = o ( n -1 ) with probability at least 1 -2 n -α . On the same event, the bound on Eq. (G.2) remains of order O ( n -1 ) , which shows Eq. (G.4). Combining (G.3) and (G.4) with the triangle inequality in L 2 ( P X ) concludes the proof.

Remark G.6 (Optimality of the rates) . In the setting of Theorem G.5, we can apply (Fischer and Steinwart, 2020, Theorem 2) in order to obtain lower bounds on the achievable rates. We have already verified the conditions (MOM), (EVD+), (EMB), and (SRC) in the proof of Theorem G.5. In the case s ∗ > d/ 2 , we have β = s ∗ /s > d/ (2 s ) = p and can therefore choose α ∈ ( p, β ) such that β > α . Then, (Fischer and Steinwart, 2020, Theorem 2) yields a lower bound on the rate of the form (with constant probability)

which matches the rates in Theorem G.5 up to log terms.

G.1 Auxiliary results for the proof of Theorem 8

The distributional Assumption (D2) immediately implies that the training points are separated with high probability.

Lemma G.7. Assume (D2) is fulfilled with β X > 0 . Then with probability at least 1 -O ( n -α ) ,

̸

Proof. For any i ∈ [ n ] , the union bound implies

̸

̸

◀

Another union bound yields

̸

Choosing δ α ( n ) = n -2+ α β X yields ϕ ( δ α ( n )) = O ( 1 n 2+ α ) , which concludes the proof.

The following lemma bounds ∥ A -1 -B -1 ∥ via ∥ A -1 ∥ and ∥ A -B ∥ . Similar results can for example be found in (Horn and Johnson, 2013, Section 5.8).

Lemma G.8. Let A , B ∈ R n × n be invertible matrices and let ∥ · ∥ be a submultiplicative matrix norm with ∥ I n ∥ = 1 . If A and B fulfill ∥ A -1 ∥∥ A -B ∥ < 1 , then it holds that

Proof. Because of ∥ A -1 ( A -B ) ∥ ≤ ∥ A -1 ∥∥ A -B ∥ < 1 we get

I

∥

-

A

-

(

-

A

-

)

-

-∥

∥∥

-

1

-

A

B

yields

A

A

1

))

1

B

-

-

B

I

A

A

1

1

= (

(

1

-

-

-

B

A

I

A

1

-

Writing

=

(

(

Now write B -1 -A -1 = A -1 ( A -B ) B -1 to get

B

A

A

-

-

∥ ≤ ∥

∥∥

B

-

A

1

1

Combining the last two inequalities concludes the proof.

∥

G.2 RKHS norm bounds

Here we show that if ˜ k and ˇ k γ have RKHS equivalent to some Sobolev space H s , s > d/ 2 , then the RKHS of the spiky-smooth kernel k ρ,γ is also equivalent to H s , for any fixed ρ, γ > 0 . Hence all members of the spiky-smooth kernel sequence may have RKHS equivalent to a Sobolev space H s and are individually inconsistent due to Theorem 1; yet the sequence is consistent. This shows that when arguing about generalization properties based on RKHS equivalence, the constants matter and the narrative that depth does not matter in the NTK regime as in Bietti and Bach (2021) is too simplified.

The following proposition states that the sum of kernels with equivalent RKHS yields an RKHS that is equivalent to the RKHS of the summands. For example, the spiky-smooth kernel with Laplace components possesses an RKHS equivalent to the RKHS of the Laplace kernel.

Proposition G.9. Let H 1 and H 2 denote the RKHS of k 1 and k 2 respectively. If H 1 = H 2 then the RKHS H of k = k 1 + k 2 fulfills H = H 1 . Moreover, if C ≥ 1 is a constant with 1 C ∥ f ∥ H 2 ≤ ∥ f ∥ H 1 ≤ C ∥ f ∥ H 2 , then we have 1 √ 2 C ∥ f ∥ H 1 ≤ ∥ f ∥ H ≤ ∥ f ∥ H 1 .

Proof. The RKHS of k = k 1 + k 2 is given by H = H 1 + H 2 with norm

To see this we consider the map Φ : X →H 1 ×H 2 defined by Φ( x ) := (Φ 1 ( x , · ) , Φ 2 ( x , · )) for all x ∈ X , where X is the set, the spaces H i live on and Φ i ( x ) := k i ( x , · ) . The reproducing property of k 1 and k 2 immediately ensures that Φ is a feature map of k 1 + k 2 and Theorem E.3 then shows

as well as the formula for the norm on H . Now let f ∈ H . Considering the decomposition f = f 1 +0 then gives ∥ f ∥ H ≤ ∥ f ∥ H 1 . Moreover, for f = f 1 + f 2 with f i ∈ H i we have

Taking the infimum over all decomposition then yields the estimate ∥ f ∥ H 1 ≤ √ 2 C ∥ f ∥ H .

-

-

B

∥∥

∥

.

B

B

))

1

∥

.

-

1

A

-

1

which implies

H Spiky-smooth activation functions induced by Gaussian components

Here we explore the properties of the NNGP and NTK activation functions induced by spiky-smooth kernels with Gaussian components.

To offer some more background, it is well-known that NNGPs and NTKs on the sphere S d are dot-product kernels, i.e., kernels of the form k d ( x , x ′ ) = κ ( ⟨ x , x ′ ⟩ ) , where the function κ has a series representation κ ( t ) = ∑ ∞ i =0 b i t i with b i ≥ 0 and ∑ ∞ i =0 b i < ∞ . The function κ is independent of the dimension d of the sphere. Conversely, all such kernels can be realized as NNGPs or NTKs (Simon et al., 2022, Theorem 3.1).

As dot-product kernel k ( x , y ) = κ ( ⟨ x , y ⟩ ) on the sphere, the Gaussian kernel has the simple analytic expression,

For spiky-smooth kernels k = ˜ k + ρ ˇ k γ with Gaussian components ˜ k and ˇ k γ of width ˜ γ and γ respectively, we get Taylor series coefficients

Now Theorem 11 states that as soon as κ induces a dot-product kernel for every input dimension d , then the dot-product kernels can be written as the NNGP kernel of a 2-layer fully-connected network without biases and with the induced activation function

or as the NTK of a 2-layer fully-connected network without biases and with the induced activation function

where h i denotes the i -th Probabilist's Hermite polynomial normalized such that ∥ h i ∥ L 2 ( N (0 , 1)) = 1 and s i ∈ {-1 , +1 } are arbitrarily chosen for all i ∈ N 0 .

Now we can study the induced activation functions if we know the kernel's Taylor coefficients ( b i ) i ∈ N 0 . If infinitely many b i > 0 , then infinitely many activation functions induce the same dotproduct kernel, with different choices of the signs s i . For alternating signs s i = ( -1) i , the symmetry property h i ( -x ) = ( -1) i h i ( x ) of the Hermite polynomials implies

To form an orthonormal basis of L 2 ( N (0 , 1)) the unnormalized Probabilist's Hermite polynomials He i have to be normalized by h i ( x ) = 1 √ i ! He i ( x ) . We can use the identity exp( xt -t 2 2 ) = ∑ ∞ i =0 He i ( x ) t i i ! with t = √ 2 /γ to analytically express the NNGP activation of the Gaussian kernel with all s i = +1 as the exponential function

Remarkably, the Gaussian kernel can not only be induced by an exponential activation function, but also by a single shifted sine activation function. This is shown in the following proposition.

with Taylor expansion

Proposition H.1 ( Trigonometric Gaussian NNGP activation functions ) . For any γ > 0 and the bi-alternating choice of signs { ( -1) ⌊ i/ 2 ⌋ } i =0 , 1 , 2 ,... , the Gaussian kernel of bandwidth γ can be realized as the NNGP kernel of a two-layer fully-connected network without biases and with activation function

Proof. We write c = 2 /γ . We need to show that

We will use the fact that

with the choices z 1 = i √ c/ 2 and z 2 = -i √ c/ 2 . Now, using e iax + b = e b (cos( ax ) + i sin( ax )) , observe that

An analogous calculation yields

Finally, using H i ( x/ √ 2) = 2 i/ 2 He i ( x ) , we get

For ϕ NNGP ( x ) = ∑ ∞ i =0 s i √ b i h i ( x ) , we get ∥ ϕ ∥ 2 L 2 ( N (0 , 1)) = ∑ ∞ i =0 b i invariant to the choice { s i } i ∈ N . For Gaussian NNGP activation components with bandwidth γ > 0 this yields

because { h i } i ∈ N 0 is an ONB of L 2 ( N (0 , 1)) . Analogously, for Gaussian NTK activation components, we get

This implies that the average amplitude of NNGP activation functions does not depend on γ , while the average amplitude of NTK activation functions decays with γ → 0 .

By the fact h ′ n ( x ) = √ nh n -1 ( x ) , we know that any activation function ϕ ( x ) = ∑ ∞ i =0 s i a i h i ( x ) has the derivative ϕ ′ ( x ) = ∑ ∞ i =0 s i a i +1 √ i +1 · h i ( x ) as long as ∑ ∞ i =0 | a i +1 √ i +1 | < ∞ .

The following proposition formalizes the additive approximation ϕ k ≈ ϕ ˜ k + ρ 1 / 2 ϕ ˇ k γ , and quantifies the necessary scaling of γ for any demanded precision of the approximation.

Proposition H.2. Fix ˜ γ, ρ > 0 arbitrary. Let k = ˜ k + ρ ˇ k γ denote the spiky-smooth kernel where ˜ k and ˇ k γ are Gaussian kernels of bandwidth ˜ γ and γ , respectively. Assume that we choose the activation functions ϕ k NTK , ϕ ˜ k NTK and ϕ ˇ k γ NTK as in Theorem 11 with same signs { s i } i ∈ N . Then, for γ > 0 small enough, it holds that

Proof. Let b i,γ = 2 i γ i i ! exp( -2 /γ ) denote the Taylor coefficients of the Gaussian kernel. All considered infinite series converge absolutely.

for any I ∈ N . To bound ( I ) observe

An analogous calculation for ( III ) yields

So overall we get the bound

Now, defining c := 2 /γ ,

where Γ( k +1 , c ) denotes the upper incomplete Gamma function. Choosing I = ⌊ c 2 π ⌋ , (Pinelis, 2020, Theorem 1.1) yields, for c ≥ 121 ,

where we used ( I +1)! 1 /I ≤ I for I ≥ 3 in the first line, Stirling's approximation in the second line, and ( c 2 π +1) ( ln(2 π +1) + 1 ) ≤ c/ 2 for c ≥ 121 in the last line. It is obvious that

To quantify the rate of convergence, we use the bound Γ( I + 1 , c 0 ) ≥ e -c 0 I !(1 + c 0 / ( I + 1)) I , which follows from applying Jensen's inequality to Γ( I + 1 , c 0 ) = e -c 0 I ! E (1 + c 0 /G ) I , where G ∼ Γ( I +1 , 1) and E G = I +1 . Defining c 0 = 2 / ˜ γ , it holds that

Taking the first two terms of the Laurent series expansion of n ↦→ ( 1 + c 0 n ) n about n = ∞ yields ( 1 + c 0 I +1 ) I +1 > e c 0 (1 -c 2 0 2( I +1) ) for I large enough (where we demand γ ∈ o (˜ γ 2 ) ), thus

Plugging (H.6) and (H.7) into (H.5) yields, for γ ≤ 1 / 61 ,

For the NTK we get

We can proceed exactly as for the NNGP, but choose I = ⌊ c 2 π ⌋ -1 to get

and replace (H.7) with

I Additional experimental results

The code to reproduce all our experiments is provided in the supplementary material and under

Our implementations rely on PyTorch (Paszke et al., 2019) for neural networks and mpmath (Johansson et al., 2023) for high-precision calculations.

I.1 Experimental details of Figure 1

For the kernel experiment (Figure 1a), we used the Laplace kernel with bandwidth 0 . 4 and the spiky-smooth kernel (4) with Laplace components with ρ = 1 , ˜ γ = 1 , γ = 0 . 01 .

For the neural network experiment (Figure 1b,c) we initialize 2-layer networks with NTK parametrization (Jacot et al., 2018) and He initialization (He et al., 2015). Using the antisymmetric initialization trick from Zhang et al. (2020) doubles the network width from 10000 to 20000 and helps to prevent errors induced by the random initialization function. It might also be helpful to increase the initialization variance (Chizat et al., 2019). We train the network with stochastic gradient descent of batch size 1 over the 15 training samples with learning rate 0 . 04 for 2500 epochs. Training with gradient descent and learning rate 0 . 4 produces similar results. We use the spiky-smooth activation function given by x ↦→ ReLU ( x ) + 0 . 01 · (sin(100 x ) + cos(100 x )) , which corresponds to x ↦→ ReLU ( x ) + ω NTK ( x, 1 5000 ) , including both even and uneven Hermite coefficients.

I.2 Disentangling signal from noise in neural networks with spiky-smooth activation functions

Since our spiky-smooth activation function has the additive form σ spsm ( x ) = ReLU ( x ) + ω NTK ( x ; 1 5000 ) , we can dissect the learned neural network

into its ReLU -component

If the analogy to the spiky-smooth kernel holds and f spikes fits the noise in the labels while having a small L 2 -norm, then f ReLU would have learned the signal in the data. Indeed Figure I.1 demonstrates that this simple decomposition is useful to disentangle the learned signal from the spike component in our setting. The figure also suggests that the oscillations in the activations of the hidden layer constructively interfere to interpolate the training points, while the differing frequencies and phases approximately destructively interfere on most of the remaining covariate support. Figure I.2 shows some of the functions generated by the hidden layer neurons of the spike component f spikes . Both the phases and frequencies vary. Destructive interference in sums of many oszillations occurs, for example, under a uniform phase distribution.

An exciting direction of future work will be to understand when and why the neural networks with spiky-smooth activation functions learn the target function well, and when the decomposition into ReLU - and spike component succeeds to disentangle the noise from the signal. Particular challenges will be to design architectures and learning algorithms that provably work on complex data sets and to determine their statistical convergence rates. A different line of work could evaluate whether there exist useful spike components for deep and narrow networks beyond the pure infinite-width limit. Maybe for deep architectures is suffices to apply spiky-smooth activation functions only between the penultimate and the last layer.

and its spike component

-

π

-

π

2

0

π

2

π

Angle (radians)

-

2

-

1

0

1

Prediction

a.

Spiky-smooth

Signal network

-

π

-

π

2

0

π

2

π

Angle (radians)

-

1

.

0

-

0

.

5

0

.

0

0

.

5

b.

-

π

-

π

2

0

π

2

π

-

0

.

015

-

0

.

010

-

0

.

005

0

.

000

0

.

005

0

.

010

0

.

015

-

π

-

π

2

0

π

2

π

-

0

.

02

-

0

.

01

0

.

00

0

.

01

0

.

02

-

π

-

π

2

0

π

2

π

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

-

π

-

π

2

0

π

2

π

-

0

.

02

-

0

.

01

0

.

00

0

.

01

0

.

02

-

π

-

π

2

0

π

2

π

-

0

.

02

-

0

.

01

0

.

00

0

.

01

0

.

02

-

π

-

π

2

0

π

2

π

-

0

.

004

-

0

.

002

0

.

000

0

.

002

0

.

004

-

π

-

π

2

0

π

2

π

-

0

.

015

-

0

.

010

-

0

.

005

0

.

000

0

.

005

0

.

010

0

.

015

-

π

-

π

2

0

π

2

π

-

0

.

015

-

0

.

010

-

0

.

005

0

.

000

0

.

005

0

.

010

0

.

015

-

π

-

π

2

0

π

2

π

Angle (radians)

-

0

.

003

-

0

.

002

-

0

.

001

0

.

000

0

.

001

0

.

002

0

.

003

-

π

-

π

2

0

π

2

π

Angle (radians)

-

0

.

02

-

0

.

01

0

.

00

0

.

01

0

.

02

-

π

-

π

2

0

π

2

π

Angle (radians)

-

0

.

015

-

0

.

010

-

0

.

005

0

.

000

0

.

005

0

.

010

0

.

015

-

π

-

π

2

0

π

2

π

Angle (radians)

-

0

.

006

-

0

.

004

-

0

.

002

0

.

000

0

.

002

0

.

004

0

.

006

Of course an analogous additive decomposition exists for the minimum-norm interpolant ˆ f k 0 of the spiky-smooth kernel,

where

We plot the results in Figure I.3. Observe that the spikes f spikes regress to 0 more reliably than in the neural network.

Although spiky-smooth estimators can be consistent, any method that interpolates noise cannot be adversarially robust. The signal component f signal may be a simple correction towards robust estimators. Figure I.4 suggests that the signal components of spiky-smooth estimators behave more robustly than ReLU networks or minimum-norm interpolants of Laplace kernels in terms of finite-sample variance.

-

π

-

π

2

0

π

2

π

Angle (radians)

-

2

-

1

0

1

Prediction

a.

Spiky-smooth

Spsm signal

-

π

-

π

2

0

π

2

π

Angle (radians)

-

1

.

0

-

0

.

5

0

.

0

0

.

5

b.

I.3 Repeating the finite-sample experiments

We repeat the experiment from Figure 1 100 times, both randomizing with respect to the training set and with respect to neural network initialization.

For the kernels (Figure I.4a), observe that all minimum-norm kernel interpolants are biased towards 0 . While the Laplace kernel and the signal component (I.2) of the spiky-smooth kernel have similar averages, the spiky-smooth kernel has a slightly larger bias. However, both the spiky-smooth kernel as well as its signal component produces lower variance estimates than the Laplace kernel.

Considering the trained neural networks (Figure I.4b), the ReLU networks are approximately unbiased, but have large variance. The neural networks with spiky-smooth activation function as well as the extracted signal network (I.1) are similar on average: They are slighly biased towards 0 , but have much smaller variance than the ReLU networks.

-

π

-

π

2

0

π

2

π

Angle (radians)

-

2

-

1

0

1

Prediction

a.

Laplace

Spiky-smooth

Signal

-

π

-

π

2

0

π

2

π

Angle (radians)

-

2

-

1

0

1

2

Prediction

b.

ReLU

Spiky-smooth

Signal

0

10

0

10

1

10

2

10

3

Epochs

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

RMSE

c.

ReLU

Spiky-smooth

The training curves (Figure I.4c) offer similar conclusions as Figure 1: While the ReLU networks harmfully overfit over the course of training, the neural networks with spiky-smooth activation function quickly overfit to 0 training error with monotonically decreasing test error, which on average is almost optimal, already with only 15 training points. The spiky-smooth networks have smaller confidence bands, indicating increased robustness compared to the ReLU networks. If the ReLU networks would be early-stopped with perfect timing, they would generalize similarly well as the networks with spiky-smooth activation function.

I.4 Spiky-smooth activation functions

In Figures I.5 and I.6 we plot the 2-layer NTK activation functions induced by spiky-smooth kernels with Gaussian components, where ˜ k has bandwidth 1 , and in the first figure ρ = 1 while in the second figure ρ = 0 . 1 .

+ + - -...

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

+++...

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

-

3

-

2

-

1

0

1

2

3

-

2

.

5

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

3

-

2

-

1

0

1

2

3

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

3

-

2

-

1

0

1

2

3

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

3

-

2

-

1

0

1

2

3

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

3

-

2

-

1

0

1

2

3

γ

=

0.1

-

6

-

4

-

2

0

2

4

-

3

-

2

-

1

0

1

2

3

γ

=

0.05

-

6

-

4

-

2

0

2

-

3

-

2

-

1

0

1

2

3

γ

=

0.01

-

4

-

2

0

2

4

6

-

3

-

2

-

1

0

1

2

3

γ

=

0.005

-

2

-

1

0

1

2

3

4

5

+ + - -...

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

+++...

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

-

3

-

2

-

1

0

1

2

3

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

3

-

2

-

1

0

1

2

3

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

3

-

2

-

1

0

1

2

3

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

3

-

2

-

1

0

1

2

3

-

2

.

0

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

3

-

2

-

1

0

1

2

3

γ

=

0.1

-

6

-

4

-

2

0

2

4

-

3

-

2

-

1

0

1

2

3

γ

=

0.05

-

6

-

4

-

2

0

2

-

3

-

2

-

1

0

1

2

3

γ

=

0.01

-

4

-

2

0

2

4

6

-

3

-

2

-

1

0

1

2

3

γ

=

0.005

-

2

-

1

0

1

2

3

4

5

In Figure I.7 we plot the corresponding 2-layer NNGP activation functions with ρ = 1 . In contrast to the NTK activation functions the amplitudes of the fluctuations only depends on ρ and not on γ . Our intuition is the following: Since the first layer weights are not learned in case of the NNGP, the first layer cannot learn constructive interference, so that the oscillations in the activation function need to be larger.

The additive approximation ϕ k ≈ ϕ ˜ k + ρ 1 / 2 ϕ ˇ k γ remains accurate in all considered cases (Appendix I.6).

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

8

10

+++...

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

8

10

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

8

10

-

3

-

2

-

1

0

1

2

3

-

2

0

2

4

6

8

10

-

3

-

2

-

1

0

1

2

3

-

4

-

3

-

2

-

1

0

1

2

+ + - -...

-

3

-

2

-

1

0

1

2

3

-

4

-

3

-

2

-

1

0

1

2

3

-

3

-

2

-

1

0

1

2

3

-

4

-

3

-

2

-

1

0

1

2

3

-

3

-

2

-

1

0

1

2

3

-

4

-

3

-

2

-

1

0

1

2

3

-

3

-

2

-

1

0

1

2

3

γ

=

0.1

-

12

-

10

-

8

-

6

-

4

-

2

0

2

-

3

-

2

-

1

0

1

2

3

γ

=

0.05

-

2

0

2

4

6

8

10

-

3

-

2

-

1

0

1

2

3

γ

=

0.01

-

10

-

8

-

6

-

4

-

2

0

2

4

-

3

-

2

-

1

0

1

2

3

γ

=

0.005

-

6

-

4

-

2

0

2

4

6

8

I.5 Isolated spike activation functions

By plotting the NTK activation components corresponding to Gaussian spikes ϕ ˇ k γ with varying choices of the signs s i in Figure I.9, we observe the following properties:

All s i = +1 leads to exponentially exploding activation functions, cf. Eq. (H.2).

If s i is chosen uniformly at random, with high probability, ϕ ˇ k γ both oscillates at a high frequency around 0 and explodes for | x |→∞ .

If the signs s i alternate every second i , i.e. s i = +1 iff ⌊ i 2 ⌋ even, ϕ ˇ k γ is approximately a single shifted sin -curve with increasing frequency and decreasing/constant amplitude for NTK/NNGP activation functions, cf. Eq. (6).

-

2

-

1

0

1

2

-

1

.

5

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

1

.

5

a.

γ

= 0.2

γ

= 0.1

γ

= 0.05

-

3

-

2

-

1

0

1

2

3

-

4

-

2

0

2

4

b.

0

25

50

75

100

125

150

175

Index

0

.

0

0

.

1

0

.

2

0

.

3

Coefficient value

c.

γ

= 0.2

γ

= 0.07

γ

= 0.04

γ

= 0.02

-

2

0

2

0

.

0

0

.

2

0

.

4

+++...

-

2

0

2

0

.

000

0

.

001

0

.

002

-

2

0

2

0

2

4

6

×

10

-

5

-

2

0

2

0

2

4

6

×

10

-

8

-

2

0

2

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

+ + - -...

-

2

0

2

-

0

.

2

0

.

0

0

.

2

-

2

0

2

-

0

.

2

0

.

0

0

.

2

-

2

0

2

-

0

.

2

0

.

0

0

.

2

-

2

0

2

γ

=

0.2

-

2

0

2

-

2

0

2

γ

=

0.1

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

-

2

0

2

γ

=

0.075

-

0

.

5

0

.

0

0

.

5

1

.

0

-

2

0

2

γ

=

0.05

-

0

.

5

0

.

0

0

.

5

Figure I.10 visualizes NNGP activation functions induced by Gaussian spikes with varying bandwidth γ . Observe similar behaviour as for the NTK but amplitudes invariant to γ as predicted by Eq. (6). For smaller γ the explosion of (all+) activation functions starts at larger x , but appears sharper as can be seen in the analytic expression (H.2).

Figure I.11 resembles Figure I.9 but plotted on a larger range to visualize the exploding behaviour for | x |→∞ .

-

-

4

-

2

0

2

4

0

5

10

15

+++...

-

4

-

2

0

2

4

0

.

00

0

.

05

0

.

10

-

4

-

2

0

2

4

0

.

000

0

.

001

0

.

002

-

4

-

2

0

2

4

0

2

4

×

10

-

4

-

2

0

2

4

-

1

0

1

+ + - -...

-

4

-

2

0

2

4

-

1

0

1

-

4

-

2

0

2

4

-

1

0

1

-

4

-

2

0

2

4

-

1

0

1

-

4

-

2

0

2

4

γ

=

0.2

-

60

-

40

-

20

0

20

-

4

-

2

0

2

4

γ

=

0.1

-

40

-

20

0

-

4

-

2

0

2

4

γ

=

0.075

-

20

0

20

-

4

-

2

0

2

4

γ

=

0.05

-

20

-

10

0

10

7

+ + - -...

-

4

-

2

0

2

4

0

2

4

6

+++...

-

4

-

2

0

2

4

0

.

000

0

.

025

0

.

050

0

.

075

-

4

-

2

0

2

4

0

.

000

0

.

001

0

.

002

0

.

003

-

4

-

2

0

2

4

0

1

2

×

10

-

4

-

2

0

2

4

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

-

4

-

2

0

2

4

-

0

.

2

0

.

0

0

.

2

-

4

-

2

0

2

4

-

0

.

2

0

.

0

0

.

2

-

4

-

2

0

2

4

-

0

.

2

0

.

0

0

.

2

-

4

-

2

0

2

4

γ

=

0.2

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

-

4

-

2

0

2

4

γ

=

0.1

-

7

.

5

-

5

.

0

-

2

.

5

0

.

0

2

.

5

-

4

-

2

0

2

4

γ

=

0.075

-

4

-

2

0

2

4

-

4

-

2

0

2

4

γ

=

0.05

-

5

.

0

-

2

.

5

0

.

0

2

.

5

5

.

0

I.6 Additive decomposition and sin -fit

Here we quantify the error of the sin -approximation (8) of Gaussian NTK activation components. The additive decomposition ϕ k ≈ ϕ ˜ k + ρ 1 / 2 ϕ ˇ k γ quickly becomes accurate in the limit γ → 0 (Figures I.12 and I.13), the sin -approximation seems to converge pointwise at rate Θ( | x | γ ) , where a good approximation can be expected when | x |≪ 1 /γ . The error at large | x | arises because the spike component decays for | x |→∞ . For O (1) inputs, we conjecture that this inaccuracy does not dramatically affect the test error of neural networks when γ is chosen to be small.

-

4

-

3

-

2

-

1

0

1

2

3

4

γ

=0.05

-

0

.

2

-

0

.

1

0

.

0

0

.

1

0

.

2

-

4

-

3

-

2

-

1

0

1

2

3

4

γ

=0.04

-

0

.

2

-

0

.

1

0

.

0

0

.

1

0

.

2

-

4

-

3

-

2

-

1

0

1

2

3

4

γ

=0.02

-

0

.

15

-

0

.

10

-

0

.

05

0

.

00

0

.

05

0

.

10

0

.

15

-

4

-

3

-

2

-

1

0

1

2

3

4

γ

=0.01

-

0

.

10

-

0

.

05

0

.

00

0

.

05

0

.

10

spsm-sm

spike

-

4

-

3

-

2

-

1

0

1

2

3

4

γ

=0.05

-

0

.

06

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

0

.

06

-

4

-

3

-

2

-

1

0

1

2

3

4

γ

=0.04

-

0

.

04

-

0

.

02

0

.

00

0

.

02

0

.

04

-

4

-

3

-

2

-

1

0

1

2

3

4

γ

=0.02

-

0

.

02

-

0

.

01

0

.

00

0

.

01

0

.

02

-

4

-

3

-

2

-

1

0

1

2

3

4

γ

=0.01

-

0

.

015

-

0

.

010

-

0

.

005

0

.

000

0

.

005

0

.

010

spsm-(sm+sp)

spsm-(sm+sin)

Now we evaluate the numerical approximation quality of the sin -fits (7) and (8) to the isolated spike activation components ϕ ˇ k γ . As expected by Proposition H.1, the NNGP oscillating activation function ϕ ˇ k γ of a Gaussian spike component corresponds to Eq. (7) up to numerical errors. Both for the NNGP and for the NTK, the approximations become increasingly accurate with smaller bandwidths γ → 0 (Figure I.14). Again the approximation quality suffers for | x | → ∞ , since ϕ ˇ k γ NTK slowly decay to 0 for | x |→∞ .

-

6

-

2

-

1

0

1

2

10

-

15

10

-

12

10

-

9

10

-

6

Absolute error

a.

γ

= 0.08

γ

= 0.05

γ

= 0.025

-

2

-

1

0

1

2

10

-

4

10

-

2

b.

γ

= 0.075

γ

= 0.04

γ

= 0.01

NNGP

NTK

I.7 Spiky-smooth kernel hyper-parameter selection

To understand the empirical performance of spiky-smooth kernels on finite data sets, we generate i.i.d. data where x ∼ U ( S d ) and

with ε ∼ N (0 , σ 2 ) independent of x and evaluate the least squares excess risk of the minimum-norm interpolant. Figure I.15 shows that

the smaller the spike bandwidth γ , the better. At some point, the improvement saturates,

The spiky-smooth kernel with Gaussian components exhibits catastrophic overfitting, when γ is too large (cf. Mallinar et al. (2022)), the Laplace kernel is more robust with respect to γ .

ρ should be carefully tuned, it has large impact. As with γ → 0 ridgeless regression with the spiky-smooth kernel approximates ridge regression with ˜ k and regularization ρ , simply choose the optimal regularization ρ opt of ridge regression.

With sufficiently thin spikes and properly tuned ρ , spiky-smooth kernels with Gaussian components outperform the Laplace counterparts.

We repeat the experiment in Figure I.16 with a slightly more complex generating function and come to the same conclusions.

laplace

100

10

1

0.5

0.1

0.01

Quasi-regularisation

ρ

1e-05

0.001

0.005

0.01

0.05

Spike bandwidth

γ

2

0.25

0.023

0.028

0.039

0.088

0.17

0.25

0.023

0.028

0.039

0.088

0.17

0.25

0.024

0.029

0.041

0.09

0.17

0.24

0.029

0.035

0.047

0.096

0.18

0.17

0.1

0.11

0.12

0.16

0.19

100

10

1

0.5

0.1

0.01

Quasi-regularisation

ρ

1e-05

0.001

0.005

0.01

0.05

Spike bandwidth

γ

2

0.17

0.017

0.018

0.02

0.024

0.029

0.21

0.088

0.087

0.089

0.094

0.099

1

0.94

0.95

0.95

0.96

0.97

4.4

4.4

4.4

4.4

4.4

4.5

5.1e+02 5.1e+02 5.1e+02 5.1e+02 5.1e+02 5.1e+02

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

gaussian

100

10

1

0.5

0.1

0.01

Quasi-regularisation

ρ

1e-05

0.001

0.005

0.01

0.05

Spike bandwidth

γ

0.52

0.27

0.085

0.075

0.11

0.19

0.52

0.27

0.085

0.074

0.11

0.19

0.51

0.27

0.086

0.076

0.11

0.19

0.48

0.26

0.09

0.081

0.11

0.2

0.23

0.19

0.15

0.15

0.18

0.21

laplace

100

10

1

0.5

0.1

0.01

Quasi-regularisation

ρ

1e-05

0.001

0.005

0.01

0.05

Spike bandwidth

γ

0.52

0.37

0.22

0.16

0.084

0.055

0.45

0.34

0.25

0.21

0.15

0.13

1.1

1.1

1.1

1.1

1.1

1.1

5.8

5.8

5.8

5.8

5.9

5.9

2.8e+03 2.8e+03 2.8e+03 2.8e+03 2.8e+03 2.8e+03

gaussian

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Peer reviews:
comments: SUMMARY:
This paper generalize existing inconsistency results to non-interpolating models and more kernels to show that benign overfitting with moderate derivatives is impossible in fixed dimension.
Moreover, this paper proves that interpolation with spiky-smooth kernels can be consistent and such kernels can be induced by certain activation functions.
There are also experiments supporting the authors' claims.

STRENGTHS:
The paper is well-written and easy to follow.
It also provides us with novel understandings of the benign overfitting phenomenon: the smoothness of the estimators, and not the dimension, matters.
In particular, by considering spiky-smooth kernels, the authors find that interpolation with kernels can be consistent in fixed dimension.
This new result is very interesting.

On the technical side, the authors improve previous inconsistency results for RKHS equivalent to Sobolev RKHS of smoothness $s > d/2$ on the sphere $\mathbb{S}^d$. The techniques here are solid.

WEAKNESSES:
N/A

QUESTIONS:
1. Can Theorem 5 be extended to $H^s(\Omega)$, $\Omega \subset \mathbb{R}^d$ ?
If not, could the authors explain the difficulties here?

2. Is it possible to derive optimal rates for interpolation with spiky-smooth kernels?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the generalization behavior of overfitting methods in terms of the smoothness of the estimator, showing that only non-smooth estimators can interpolate benignly. They give a discussion of this result in the context of NTKs and their corresponding infinite-width architectures.\n\nSTRENGTHS:\nOriginality and quality: this work seems to shed some new light on a well-studied problem. I'm not very familiar with background results on asymptotic risk w.r.t. function smoothness, so it's difficult for me to assess the first several theorem statements. I'm quite familiar with the latter results on NTKs, and to my knowledge this observation about spikiness in activation functions is new, and I find it well-explained.\n\nClarity: the paper seems pretty clear and well-written.\n\nSignificance: the math here is nice, and it seems like this might lead to some more clarity regarding exotic activation functions.\n\nWEAKNESSES:\nOn significance: it's sort of unclear to me how this matters, or where we go from here. For example: the results of Mallinar et al. suggest that early stopping in nets (or including a ridge parameter in KRR, as shown by prior work too) is enough to make fitting consistent. These questions like \"how do we design our model so it overfits benignly?\" seem like they're actually missing some motivation -- why do we need to do that? Can't we just use a ridge parameter or optimal stopping?\n\nThe results around spike-inducing kernels seem quite hacky. The fact that the spike has to get smaller with dataset size makes it clear that this is just effectively adding a ridge. (The authors acknowledge this, but it's still not a surprising conclusion in my view.) It's also not useful -- the authors do not try to train their Hermite-polynomial nets, and (based on my own attempts to do such things) I suspect this may be because they're unstable, even at moderately large width!\n\nAs stated above, I'm unfamiliar with prior art around function smoothness and overfitting behavior, so there's a fair bit of uncertainty in my assessment here.\n\nQUESTIONS:\nThe authors argue that the key object to consider is estimator smoothness, not input dimension. Given that there exists a body of results that do find that input dimension is important, is it clear how to reconcile the author's new results with these old ones? For example, do the smoothness requirements get more relaxed as dimension increases?\n\nNote: I think an activation fn corresponding to the Gaussian NNGP kernel can be written down in closed form. (I do not know if this is true for the NTK, and would be curious to know if it can.)\n\nNote: a high-frequency sinusoidal activation function can also give you a spiky kernel!\n\nNote for future research: this smooth + spiky kernel thing seems like a hack. However, the results of Mallinar et al. suggest that if you could write down any kernel with an eigendecay like $\\lambda_i \\sim i^{-1} \\log^{-\\alpha} i$ for $\\alpha > 1$, you'd get a consistent estimator without the need to scale down the kernel. It seems plausible to me that by taking a sort of exponent-equals-zero limit of Bietti and Bach's analysis, you could construct a kernel that actually gives you this decay (even on the unit circle), and it'd be consistent while being (a) a continuous function and (b) not needing to be scaled with dataset size. Could be an interesting thing to look into.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper extends previous results on the inconsistency of ridgeless kernel regression in fixed dimension by showing that non-interpolating estimators whose norm grows comparably to the minimum-norm interpolator are also inconsistent. On the other hand, it is shown that so-called spiky-smooth kernels whose derivatives grow with the number of samples are able to interpolate training data consistently. These results are further specialized to Neural Tangent Kernels and Neural Network Gaussian Processes, and experimental results support the theory.\n\nSTRENGTHS:\n* The paper is generally well-written with understandable and insightful theorem statements and proof ideas, even for non-experts.\n* Modifying the kernel to achieve consistency in fixed dimension via inspirations from high-dimensional benign overfitting seems novel and interesting.\n* The assumptions seem to be transparent and clearly stated in the main text.\n\nWEAKNESSES:\n* Having explicit rate estimates in addition to asymptotic consistency guarantees could provide further insight into how the spiky-smooth kernels/activations must be designed to achieve optimal performance.\n* While mathematically exciting and valuable, the results might have limited applicability in modern settings with typically high-dimensional datasets.\n* The results concerning the NTK can be applied to networks in the kernel regime (a.k.a. \"lazy training\"), which can only provide a partial picture as there is increasing evidence that in certain settings, neural networks can generalize better in the \"feature learning\" regime, see e.g. [1].\n\n[1] Malach et al. \"Quantifying the benefit of using differentiable learning over tangent kernels.\" ICML 2021.\n\nQUESTIONS:\n* I believe it can be helpful to add a discussion on the possibility of initializing the weights of a neural network such that training remains in the NTK regime, while the initial predictor is 0, which is an assumption throughout the work.\n\n* In Equation (2) of Theorem 1, is there a hidden dependence on $\\sigma$ in $c$? In particular, is this intuition incorrect that in the noiseless setting of $\\sigma = 0$, consistent estimation ($c = 0$) must be possible?\n\n* In line 232, it is mentioned that Section 5 achieves consistency by violating Assumption (N). To my understanding, it seems like Section 5 takes advantage of having the norm (of any estimator) depend on the number of samples. On the other hand, the specific statement of Assumption (N) seems to still hold in Theorem 7, as it is stated for the minimum-norm interpolant (hence $C_\\text{norm} = 1$).\n\nMinor questions/comments:\n\n* In line 88, it is mentioned that training infinitely wide NNs with gradient flow corresponds to learning with the NTK. Perhaps it is worth mentioning that this is only true under a certain scaling that leads to the \"lazy training\" regime [2], and other smaller initialization scalings that lead to the \"mean field/feature learning\" regime can have better generalization performance [1].\n\n* In line 95, rotation-invariant kernels are suddenly discussed without prior context. It could be useful to mention their importance given that under standard initializations the NTK is rotation-invariant.\n\n[1] Malach et al. \"Quantifying the benefit of using differentiable learning over tangent kernels.\" ICML 2021.\n\n[2] Chizat et al. \"On Lazy Training in Differentiable Programming.\" NeurIPS 2019.\n\nLIMITATIONS:\nPlease see \"Weaknesses\".", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
In this paper, the authors studied the problem of benign overfitting for kernels and wide neural networks (in kernel regime) in fixed dimension. The authors showed that benign overfitting is possible if and only if the learner model has large derivatives. This implies that benign overfitting is not possible for those models with small derivatives (including ReLU NTK). On the other hand, they showed that for certain spiky-smooth activations/kernels, benign overfitting is achievable. Experiments are provided in the paper to verify the results.

STRENGTHS:
1.	The paper is clearly written and easy-to-follow. The proof sketch is given to help the readers to understand the proof easier.
2.	Understanding benign overfitting phenomenon is an important problem for deep learning. The current paper focuses on the kernel regression setting and connects to neural networks via the neural tangent kernel (NTK) theory
3.	The idea of introducing spiky terms in NTK seem to be interesting. The observation in the proof that such spiky term in NTK behaves like regularization term so that the whole solution is now approximating kernel ridge regression also seems to be interesting.

WEAKNESSES:
1.	The current paper focuses on kernel regression and neural networks in the kernel regime. It would be more interesting to go beyond the kernel regime to see if similar results would also hold (e.g., in feature learning regime).

QUESTIONS:
1.	I was wondering what the dependency on dimension $d$ would be in the results such as Theorem 1.

LIMITATIONS:
The limitation is discussed in the paper. This is a theoretical work and therefore does not seem to have negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explains that benign overfitting when the dimensionality of data is fixed is possible if one looks for estimators differently from conventional minimal norm. If one allows the estimator to be spiky, benign overfitting can still be possible. These results are extended via NTK to two-layer infinite-width neural networks. The authors showed that a nearly imperceptible sinusoid added to a RELU nonlinearity allows the trained neural network to overfit noisy data without compromising generalization.\n\nSTRENGTHS:\n(Please note that I could not have proofread the submission as well as I would have liked, due to various external obligations.) This paper extends the results on consistency / inconsistency for infinite-dimensional data to the finite dimensional data case. Although the main points - allowing the estimator to go out of its \"smooth\" way to fit noisy data - could have been argued to be expected, actually seeing this established is still a contribution of remark. The other main point - replacing commonly used non-linearity with one that contains tiny level of sinusoidal perturbation - is not expected by me, that's a very nice way of understanding the non-linearity for neural networks.\n\nWEAKNESSES:\n(Please note that I could not have proofread the submission as well as I would have liked, due to various external obligations.) This paper promotes benign overfitting, but it still leaves open whether not allowing overfitting is better or worse than going into benign overfitting, and within the space of benign overfitting how to reach good results, for a given (n,d). \n\nIn Section 6.2 I felt that the noise standard deviation of 0.25 and the neural tangent kernel of gamma = 1/5000, these two quantities are somehow linked. It could turn out that claiming that such a network can be trained to overfit without regularization might be a dangerous overstatement.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}