Title: On the Saturation Effects of Spectral Algorithms in Large Dimensions
Venue: NeurIPS 2024
Accepted: yes
Keywords: reproducing kernel Hilbert space

spectral algorithm

high-dimensional statistics

minimax rates
Abstract: The saturation effects, which originally refer to the fact that kernel ridge regression (KRR) fails to achieve the information-theoretical lower bound when the regression function is over-smooth, have been observed for almost 20 years and were rigorously proved recently for kernel ridge regression and some other spectral algorithms over a fixed dimensional domain. The main focus of this paper is to explore the saturation effects for a large class of spectral algorithms (including the KRR, gradient descent, etc.) in large dimensional settings where $n \asymp d^{\gamma}$. More precisely, we first propose an improved minimax lower bound for the kernel regression problem in large dimensional settings and show that the gradient flow with early stopping strategy will result in an estimator achieving this lower bound (up to a logarithmic factor). Similar to the results in KRR, we can further determine the exact convergence rates (both upper and lower bounds) of a large class of (optimal tuned) spectral algorithms with different qualification $\tau$'s. In particular, we find that these exact rate curves (varying along $\gamma$) exhibit the periodic plateau behavior and the polynomial approximation barrier. Consequently, we can fully depict the saturation effects of the spectral algorithms and reveal a new phenomenon in large dimensional settings (i.e.,  the saturation effect occurs in large dimensional setting as long as the source condition $s>\tau$ while it occurs in fixed dimensional setting as long as $s>2\tau$).

On the Saturation Effects of Spectral Algorithms in Large Dimensions

Abstract

The saturation effects, which originally refer to the fact that kernel ridge regression (KRR) fails to achieve the information-theoretical lower bound when the regression function is over-smooth, have been observed for almost 20 years and were rigorously proved recently for kernel ridge regression and some other spectral algorithms over a fixed dimensional domain. The main focus of this paper is to explore the saturation effects for a large class of spectral algorithms (including the KRR, gradient descent, etc.) in large dimensional settings where n ≍ d γ . More precisely, we first propose an improved minimax lower bound for the kernel regression problem in large dimensional settings and show that the gradient flow with early stopping strategy will result in an estimator achieving this lower bound (up to a logarithmic factor). Similar to the results in KRR, we can further determine the exact convergence rates (both upper and lower bounds) of a large class of (optimal tuned) spectral algorithms with different qualification τ 's. In particular, we find that these exact rate curves (varying along γ ) exhibit the periodic plateau behavior and the polynomial approximation barrier. Consequently, we can fully depict the saturation effects of the spectral algorithms and reveal a new phenomenon in large dimensional settings (i.e., the saturation effect occurs in large dimensional setting as long as the source condition s > τ while it occurs in fixed dimensional setting as long as s > 2 τ ).

1 Introduction

Let's assume we have n i.i.d. samples ( x i , y i ) from a joint distribution supported on R d × R . The regression problem, one of the most fundamental problems in statistics, aims to find a function ˆ f based on these samples such that the excess risk , ∥ ˆ f -f ⋆ ∥ 2 L 2 = E x [( f ⋆ ( x ) -ˆ f ( x )) 2 ] , is small, where f ⋆ ( x ) = E [ Y | x ] is the regression function . Many non-parametric regression methods are proposed to solve the regression problem by assuming that f ⋆ falls into certain function classes, including polynomial splines Stone (1994), local polynomials Cleveland (1979); Stone (1977), the spectral algorithms Caponnetto (2006); Caponnetto and De Vito (2007); Caponnetto and Yao (2010), etc.

Spectral algorithms, as a classical topic, have been studied since the 1990s. Early works treated certain types of spectral algorithms in their theoretical analysis (Caponnetto (2006); Caponnetto and De Vito (2007); Raskutti et al. (2014); Lin et al. (2020)). These works often consider d as a fixed constant and impose the polynomial eigenvalue decay assumption under a kernel (i.e., there exist constants 0 < c ≤ C < ∞ , such that the eigenvalues of the kernel satisfy c j -β ≤ λ j ≤ C j -β , j ≥ 1 for certain β > 1 depending on the fixed d ). They further assume that f ⋆ belongs to the reproducing kernel Hilbert space (RKHS) H associated with the kernel. Under the above assumptions, they then showed that the minimax rate of the excess risk of regression over the corresponding RKHS is lower bounded by n -β/ ( β +1) and that some (regularized) spectral algorithms, e.g., the kernel ridge regression (KRR) and the kernel gradient flow, can produce estimators achieving this minimax optimal rate.

However, subsequent studies have revealed that when higher regularity (or smoothness) of f ⋆ is assumed, KRR fails to achieve the information-theoretical lower bound on the excess risk, while kernel gradient flow can do so. Specifically, let's assume that f ⋆ belongs to the interpolation space [ H ] s of the RKHS H with s > 0 (see, e.g., Steinwart et al. (2009); Dieuleveut et al. (2017); Dicker et al. (2017); Pillaud-Vivien et al. (2018); Lin et al. (2020); Fischer and Steinwart (2020); Celisse and Wahl (2021)). It is then shown that the information-theoretical lower bound on the excess risk is n -sβ/ ( sβ +1) . When 0 < s ≤ 2 , Caponnetto and De Vito (2007); Yao et al. (2007); Lin et al. (2020); Zhang et al. (2023) have already shown that the upper bound of the excess risks of both KRR and the kernel gradient flow is n -sβ/ ( sβ +1) , and hence they are minimax optimal. On the contrary, when s > 2 , Yao et al. (2007); Lin et al. (2020) showed that the upper bound of the excess risks of kernel gradient flow is n -sβ/ ( sβ +1) while the best upper bound of the excess risks of KRR is n -2 β/ (2 β +1) (Caponnetto and De Vito (2007)). Bauer et al. (2007); Gerfo et al. (2008); Dicker et al. (2017) conjectured that the convergence rate of KRR is bounded below by n -2 β/ (2 β +1) and Li et al. (2022) rigorously proved it. The above phenomenon is often referred to as the saturation effect of KRR:

KRR is inferior to certain spectral algorithms, such as kernel gradient flow, when s > 2 .

In recent years, neural network methods have gained tremendous success in many large-dimensional problems, such as computer vision He et al. (2016); Krizhevsky et al. (2017) and natural language processing Devlin (2018). Several groups of researchers tried to explain the superior performance of neural networks on large-dimensional data from the aspects of "lazy regime" (Arora et al. (2019); Du et al. (2019, 2018); Li and Liang (2018)). They noticed that, when the width of a neural network is sufficiently large, its parameters/weights stay in a small neighborhood of their initial position during the training process. Later, Jacot et al. (2018); Arora et al. (2019); Hu et al. (2021); Suh et al. (2021); Lai et al. (2023); Li et al. (2024) proved that the time-varying neural network kernel (NNK) converges (uniformly) to a time-invariant neural tangent kernel (NTK) as the width of the neural network goes to infinity, and thus the excess risk of kernel gradient flow with NTK converges (uniformly) to the excess risk of neural networks in the 'lazy regime'.

Inspired by the concepts of the "lazy regime" and the uniform convergence of excess risk, the machine learning community has experienced a renewed surge of interest in large-dimensional spectral algorithms. The earliest works focused on the consistency of two specific types of spectral algorithms: KRR and kernel interpolation (Liang and Rakhlin (2020); Liang et al. (2020); Ghorbani et al. (2020, 2021); Mei et al. (2021, 2022); Misiakiewicz and Mei (2022); Aerni et al. (2023); Barzilai and Shamir (2023)). In comparison, results on large-dimensional kernel gradient flow were somewhat scarce, and these results largely mirrored those associated with KRR (e.g., Ghosh et al. (2021)). Recently, Lu et al. (2023) proved that large-dimensional kernel gradient flow is minimax optimal when s = 1 . Then, Zhang et al. (2024) provided upper and lower bounds on the convergence rate on the excess risk of KRR for any s > 0 . Surprisingly, they discovered that for s > 1 , the convergence rate of KRR did not match the lower bound on the minimax rate. Unfortunately, they didn't prove that certain spectral algorithms can reach the lower bound on the minimax rate they provided, and hence they didn't rigorously prove that the saturation effect of KRR occurs in large dimensions. Instead, Zhang et al. (2024) only conjectured that certain spectral algorithms (e.g., kernel gradient flow) can provide minimax optimal estimators after their main results.

If Zhang et al. (2024)'s conjecture is true, then we can safely conclude that: when the regression function f ⋆ is smooth enough, KRR is inferior to kernel gradient flow in large dimensions as well. Consequently, previous results on large-dimensional KRR may not be directly extendable to large-

dimensional neural networks, even if the neural networks are in the 'lazy regime'. The main focus of this paper is to prove this conjecture by showing that kernel gradient flow is minimax optimal in large dimensions.

1.1 Related work

Saturation effects of fixed-dimensional spectral algorithms. When the dimension d of the data is fixed, the saturation effect of KRR has been conjectured for decades and is rigorously proved in the recent work Li et al. (2022). Suppose f ⋆ ∈ [ H ] s with s > 2 . It is shown that: (i) the minimax optimal rate is n -sβ/ ( sβ +1) (Rastogi and Sampath (2017); Yao et al. (2007); Lin et al. (2020)); and (ii) the convergence rate on the excess risk of KRR is n -2 β/ (2 β +1) (Li et al. (2022)). More recently, Li et al. (2024) determined the exact generalization error curves of a class of analytic spectral algorithms, which allowed them to further show the saturation effect of spectral algorithms with finite qualification τ (see, e.g., Appendix C): suppose f ⋆ ∈ [ H ] s with s > 2 τ , then the convergence rate on the excess risk of the above spectral algorithms is n -2 τβ/ (2 τβ +1) .

New phenomena in large-dimensional spectral algorithms. In the large-dimensional setting where n ≍ d γ with γ > 0 , new phenomena exhibited in spectral algorithms are popular topics in recent machine-learning research. A line of work focused on the polynomial approximation barrier phenomenon (e.g., Ghorbani et al. (2021); Donhauser et al. (2021); Mei et al. (2022); Xiao et al. (2023); Misiakiewicz (2022); Hu and Lu (2022)). They found that, for the square-integrable regression function, KRR and kernel gradient flow are consistent if and only if the regression function is a polynomial with a low degree. Another line of work considered the benign overfitting of kernel interpolation (i.e., kernel interpolation can generalize) (e.g., Liang and Rakhlin (2020); Liang et al. (2020); Aerni et al. (2023); Barzilai and Shamir (2023); Zhang et al. (2024)). Moreover, two recent work (Lu et al. (2023); Zhang et al. (2024)) discussed two new phenomena exhibited in large-dimensional KRR and kernel gradient flow: the multiple descent behavior and the periodic plateau behavior. The multiple descent behavior refers to the phenomenon that the curve of the convergence rate ( with respect to n ) of the optimal excess risk is non-monotone and has several isolated peaks and valleys; while the periodic plateau behavior refers to the phenomenon that the curve of the convergence rate ( with respect to d ) of the optimal excess risk has constant values when γ is within certain intervals. Finally, Zhang et al. (2024) conjectured that the saturation effect of KRR occurs in large dimensions. The above works imply that these phenomena occur in many spectral algorithms in large dimensions, hence encouraging us to provide a unified explanation of these new phenomena.

1.2 Our contributions

In this paper, we focus on the large-dimensional spectral algorithms with inner product kernels, and we assume that the regression function falls into an interpolation space [ H ] s with s > 0 . We state our main results as follows:

Theorem 1.1 (Restate Theorem 4.1 and 4.2, non-rigorous) . Let s > 0 , τ ≥ 1 , and γ > 0 be fixed real numbers. Denote p as the integer satisfying γ ∈ [ p ( s +1) , ( p +1)( s +1)) . Then under certain conditions, the excess risk of large-dimensional spectral algorithm with qualification τ satisfies

where ˜ s = min { s, 2 τ } .

More specifically, we list the main contributions of this paper as follows:

In Theorem 3.1, we show that the convergence rate on the excess risk of (optimally-tuned) kernel gradient flow in large dimensions is Θ P ( d -min { γ -p,s ( p +1) } ) · poly (ln( d )) , which matches the lower bound on the minimax rate given in Theorem 3.3 (up to a logarithmic factor). We find that kernel gradient flow is minimax optimal for any s > 0 and any γ > 0 , and KRR is not minimax optimal for s > 1 and for certain ranges of γ (We provide a visual illustration in Figure 2). Consequently, we rigorously prove that the saturation effect of KRR occurs in large dimensions.

In Theorem 3.3, we enhanced the previous minimax lower bound results given in Lu et al. (2023) and Zhang et al. (2024). Specifically, we show that the minimax lower bound is Ω( d -min { γ -p,s ( p +1) } ) / poly (ln( d )) . In comparison, the previous minimax lower bound is Ω( d -min { γ -p,s ( p +1) } ) /d ε for any ε > 0 , and the additional term d ε changes the desired convergence rate.

In Section 4, we determine the convergence rate on the excess risk of large-dimensional spectral algorithms. From our results, we find several new phenomena exhibited in spectral algorithms in large-dimensional settings. We provide a visual illustration of the above phenomena in Figure 1: i) The first phenomenon is the polynomial approximation barrier, and as shown in Figure 1(a), when s is close to zero, the curve of the convergence rate of spectral algorithm drops when γ ≈ p for any integer p and will stay invariant for most of the other γ ; ii) The second one is the periodic plateau behavior, and as shown in Figure 1(b) and Figure 1(c), when 0 < s < 2 τ and γ ∈ [ p ( s +1)+ s +(max { s, τ } -τ ) /τ, ( p +1)( s +1)) for an integer p ≥ 0 , the convergence rate does not change when γ varies; iii) The final one is the saturation effect, and as shown in Figure 1(c) and Figure 1(d), when s > τ , the convergence rate of spectral algorithm can not achieve the minimax lower bound for certain ranges of γ . A detailed discussion about the above three phenomena can be found in Section 4.

(a)

(b)

(c)

(d)

2 Preliminaries

Suppose that we have observed n i.i.d. samples ( x i , y i ) , i ∈ [ n ] from the model:

where x i 's are sampled from ρ X , ρ X is the marginal distribution on X ⊂ R d +1 , y ∈ Y ⊂ R , f ⋆ is some function defined on a compact set X , and

for some fixed constant σ > 0 , where ρ is the joint distribution of ( x, y ) on X ×Y . Denote the n × 1 data vector of y i 's and the n × d data matrix of x i 's by Y and X respectively.

2.1 Kernel ridge regression and kernel gradient flow

In this subsection, we introduce two specific spectral algorithms, kernel ridge regression and kernel gradient flow, which produce estimators of the regression function f ⋆ . Afurther discussion on general spectral algorithms will be provided in Section 4.

Throughout the paper, we denote H as a separable RKHS on X with respect to a continuous and positive definite kernel function K ( · , · ) : X × X → R and there exists a constant κ satisfying

Kernel ridge regression Kernel ridge regression (KRR) constructs an estimator ˆ f KRR λ by solving the penalized least square problem

where λ > 0 is referred to as the regularization parameter. The representer theorem (see, e.g., Steinwart and Christmann (2008)) gives an explicit expression of the KRR estimator, i.e.,

Kernel gradient flow The gradient flow of the loss function L = 1 2 n ∑ i ( y i -f ( x i )) 2 induced a gradient flow in H which is given by

If we further assume that ˆ f GF 0 ( x ) = 0 , then we can also give an explicit expression of the kernel gradient flow estimator

2.2 The interpolation space

Define the integral operator T K as T K ( f )( x ) = ∫ K ( x, x ′ ) f ( x ′ ) d ρ X ( x ′ ) . It is well known that T K is a positive, self-adjoint, trace-class, and hence a compact operator (Steinwart and Scovel (2012)). The celebrated Mercer's theorem further assures that

where the eigenvalues { λ j , j = 1 , 2 , ... } is a non-increasing sequence, and the corresponding eigenfunctions { ϕ j ( · ) , j = 1 , 2 , ... } are orthonormal in L 2 ( X , ρ X ) function space.

The interpolation space [ H ] s with source condition s is defined as

with the inner product deduced from

It is easy to show that [ H ] s is also a separable Hilbert space with orthonormal basis { λ s/ 2 j ϕ j } j . Generally speaking, functions in [ H ] s become smoother as s increases (see, e.g., the example of Sobolev RKHS in Edmunds and Triebel (1996); Zhang et al. (2023).

2.3 Assumptions

In this subsection, we list the assumptions that we need for our main results.

To avoid potential confusion, we specify the following large-dimensional scenario for kernel regression where we perform our analysis: suppose that there exist three positive constants c 1 , c 2 and γ , such that

and we often assume that d is sufficiently large.

In this paper, we only consider the inner product kernels defined on the sphere. An inner product kernel is a kernel function K defined on S d such that there exists a function Φ : [ -1 , 1] → R independent of d satisfying that for any x, x ′ ∈ S d , we have K ( x, x ′ ) = Φ( ⟨ x, x ′ ⟩ ) . If we further

assume that the marginal distribution ρ X is the uniform distribution on X = S d , then the Mercer's decomposition for K can be rewritten as

where Y k,j for j = 1 , · · · , N ( d, k ) are spherical harmonic polynomials of degree k and µ k 's are the eigenvalues of K with multiplicity N ( d, 0) = 1 ; N ( d, k ) = 2 k + d -1 k · ( k + d -2)! ( d -1)!( k -1)! , k = 1 , 2 , · · · . For more details of the inner product kernels, readers can refer to Gallier (2009).

Remark 2.1 . We consider the inner product kernels on the sphere mainly because the harmonic analysis is clear on the sphere ( e.g., properties of spherical harmonic polynomials are more concise than the orthogonal series on general domains). This makes Mercer's decomposition of the inner product more explicit rather than several abstract assumptions ( e.g., Mei and Montanari (2022)). We also notice that very few results are available for Mercer's decomposition of a kernel defined on the general domain, especially when the dimension of the domain is taking into consideration. e.g., even the eigen-decay rate of the neural tangent kernels is only determined for the spheres. Restricted by this technical reason, most works analyzing the spectral algorithm in large-dimensional settings focus on the inner product kernels on spheres (Liang et al., 2020; Ghorbani et al., 2021; Misiakiewicz, 2022; Xiao et al., 2023; Lu et al., 2023, etc.). Though there might be several works that tried to relax the spherical assumption (e.g., Liang et al. (2020); Aerni et al. (2023); Barzilai and Shamir (2023), we can find that most of them (i) adopted a near-spherical assumption; (ii) adopted strong assumptions on the regression function, e.g., f ⋆ ( x ) = x [1] x [2] · · · x [ L ] for an integer L > 0 , where x [ i ] denotes the i -th component of x ; or (iii) can not determine the convergence rate on the excess risk of the spectral algorithm.

To avoid unnecessary notation, let us make the following assumption on the inner product kernel K . Assumption 1 . Φ( t ) ∈ C ∞ ([ -1 , 1]) is a fixed function independent of d and there exists a nonnegative sequence of absolute constants { a j ≥ 0 } j ≥ 0 , such that we have

where a j > 0 for any j ≤ ⌊ γ ⌋ +3 .

The purpose of Assumption 1 is to keep the main results and proofs clean. Notice that, by Theorem 1.b in Gneiting (2013), the inner product kernel K on the sphere is semi-positive definite for all dimensions if and only if all coefficients { a j , j = 0 , 1 , 2 , ... } are non-negative. One can easily extend our results in this paper when certain coefficients a k 's are zero (e.g., one can consider the two-layer NTK defined as in Section 5 of Lu et al. (2023), with a i = 0 for any i = 3 , 5 , 7 , · · · ).

In the next assumption, we formally introduce the source condition, which characterizes the relative smoothness of f ⋆ with respect to H .

Assumption 2 (Source condition) . Suppose that f ⋆ ( x ) = ∑ ∞ i =1 f i ϕ i ( x ) .

(a) f ⋆ ∈ [ H ] s for some s > 0 , and there exists a constant R γ only depending on γ , such that

̸

(b) Denote q as the smallest integer such that q > γ and µ q = 0 . Define I d,k as the index set satisfying λ i ≡ µ k , i ∈ I d,k . Further suppose that there exists an absolute constant c 0 > 0 such that for any d and k ∈ { 0 , 1 , · · · , q } with µ k = 0 , we have

̸

Assumption 2 is a common assumption when one is interested in the tight bounds on the excess risk of spectral algorithms (e.g., Caponnetto and De Vito (2007); Fischer and Steinwart (2020), Eq.(8) in Cui et al. (2021), Assumption 3 in Li et al. (2024), and Assumption 5 in Zhang et al. (2024)). Assumption 2 implies that the regression function exactly falls into the interpolation space [ H ] s , that is, f ⋆ ∈ [ H ] s and f ⋆ / ∈ [ H ] t for any t > s . For example, from the proof part I of Lemma D.14, one can check that f ⋆ with ∑ i ∈I d,p µ -s p f 2 i = ∑ i ∈I d,p +1 µ -s p +1 f 2 i = 0 can have a faster convergence rate on the excess risk.

Notations. Let's denote the norm in L 2 ( X , ρ X ) as ∥ · ∥ L 2 . For a vector x , we use x [ i ] to denote its i -th component. We use asymptotic notations O ( · ) , o ( · ) , Ω( · ) and Θ( · ) . For instance, we say two (deterministic) quantities U ( d ) , V ( d ) satisfy U ( d ) = o ( V ( d )) if and only if for any ε > 0 , there exists a constant D ε that only depends on ε and the absolute positive constants σ, κ, s, γ, c 0 , c 1 , c 2 , C 1 , · · · , C 8 > 0 , such that for any d > D ε , we have U ( d ) < εV ( d ) . We also write a n = poly ( b n ) if there exist a constant θ ≥ 0 , such that a n = Θ( b θ n ) . We use the probability versions of the asymptotic notations such as O P ( · ) , o P ( · ) , Ω P ( · ) , Θ P ( · ) . For instance, we say the random variables X n , Y n satisfying X n = O P ( Y n ) if and only if for any ε > 0 , there exist constants C ε and N ε such that P ( | X n | ≥ C ε | Y n | ) ≤ ε, ∀ n > N ε .

2.4 Review of the previous results

The following two results are restatements of Theorem 2 and Theorem 5 in Zhang et al. (2024).

Proposition 2.2. Let s ≥ 1 and γ > 0 be fixed real numbers. Denote p as the integer satisfying γ ∈ [ p ( s +1) , ( p +1)( s +1)) . Suppose that Assumption 1 and Assumption 2 hold for s and γ . Let ˆ f KRR λ be the function defined in (2) . Define ˜ s = min { s, 2 } , then there exists λ ⋆ > 0 , such that we have

where Θ P only involves constants depending on s, σ, γ, c 0 , κ, c 1 and c 2 . In addition, the convergence rates of the generalization error can not be faster than above for any choice of regularization parameter λ = λ ( d, n ) → 0 .

Proposition 2.3 (Lower bound on the minimax rate) . Let s > 0 and γ > 0 be fixed real numbers. Denote p as the integer satisfying γ ∈ [ p ( s +1) , ( p +1)( s +1)) . Let P consist of all the distributions ρ on X ×Y such that Assumption 1 and Assumption 2 hold for s and γ . Then for any ε > 0 , we have:

where Ω only involves constants depending on s, σ, γ, c 0 , κ, c 1 , c 2 and ε .

From the above two propositions, we can find that when s > 1 , the convergence rate on the excess risk of KRR does not always match the lower bound on the minimax optimal rate. Zhang et al. (2024) further conjectured that the lower bound on the minimax optimal rate provided in Proposition 2.3 is tight (ignoring the additional term d -ε ). Hence, they believed that the saturation effect exists for large-dimensional KRR.

3 Main results

In this section, we determine the convergence rate on the excess risk of kernel gradient flow as d -min { γ -p,s ( p +1) } poly (ln( d )) , which differs from the lower bound on the minimax rate provided in Proposition 2.3 by d ε for any ε > 0 . We then tighten the lower bound on the minimax rate to d -min { γ -p,s ( p +1) } / poly (ln( d )) . Based on the above results, we find that KRR is not minimax optimal for s > 1 and for certain ranges of γ . Therefore, we show that the saturation effect of KRR occurs in large dimensions.

3.1 Exact convergence rate on the excess risk of kernel gradient flow

We first state our main results in this paper.

Theorem 3.1 (Kernel gradient flow) . Let s > 0 and γ > 0 be fixed real numbers. Denote p as the integer satisfying γ ∈ [ p ( s +1) , ( p +1)( s +1)) . Suppose that Assumption 1 and Assumption 2 hold for s and γ . Let ˆ f GF t be the function defined in (4) . Then there exists t ⋆ > 0 , such that we have

where Θ P only involves constants depending on s, σ, γ, c 0 , κ, c 1 and c 2 .

Theorem 3.1 is a direct corollary of Theorem 4.1 and Example 2. Combining with the previous results in Proposition 2.3, or our modified minimax rate given in Theorem 3.3, we can conclude that large-dimensional kernel gradient flow is minimax optimal for any s > 0 and any γ > 0 . More importantly, the convergence rate of kernel gradient flow is faster than that of KRR given in Proposition 2.2 when (i) 1 < s ≤ 2 and γ ∈ ( p ( s +1) + 1 , p ( s +1) + 2 s -1) for some p ∈ N , or (ii) s > 2 and γ ∈ ( p ( s +1) + 1 , ( p +1)( s +1)) for some p ∈ N . Therefore, we have proved the saturation effect of KRR in large dimensions.

Remark 3.2 . When p ≥ 1 , the logarithm term poly (ln( d )) in (12) can be removed. When p = 0 , we have poly (ln( d )) = (ln( d )) 2 in (12). See Appendix D.4 for details.

3.2 Improved minimax lower bound

Recall that Proposition 2.3 gave a lower bound on the minimax rate as d -min { γ -p,s ( p +1) } · d -ε . The following theorem replaces the additional term d -ε (which has changed the convergence rate) into a logarithm term poly -1 (ln( d )) (which does not change the desired convergence rate).

Theorem 3.3 (Improved minimax lower bound) . Let s > 0 and γ > 0 be fixed real numbers. Denote p as the integer satisfying γ ∈ [ p ( s +1) , ( p +1)( s +1)) . Let P consist of all the distributions ρ on X × Y such that Assumption 1 and Assumption 2 hold for s and γ . Then we have:

where Ω only involves constants depending on s, σ, γ, c 0 , κ, c 1 , and c 2 .

4 Exact convergence rate on the excess risk of spectral algorithms

In this section, we will give tight bounds on the excess risks of certain types of spectral algorithms, such as kernel ridge regression, iterated ridge regression, kernel gradient flow, and kernel gradient descent.

Given an analytic filter function φ λ ( · ) with qualification τ ≥ 1 (refer to Appendix C for the definitions of analytic filter function and its qualification), we can define a spectral algorithm in the following way (see, e.g., Bauer et al. (2007)). For any y ∈ R , let K x : R →H be given by K x ( y ) = y · K ( x, · ) , whose adjoint K ∗ x : H → R is given by K ∗ x ( f ) = ⟨ K ( x, · ) , f ⟩ H = f ( x ) . Moreover, we denote by T x = K x K ∗ x and T X = 1 n ∑ n i =1 T x i . We also define the sample basis function

Now, the estimator of the spectral algorithm is defined by

Many commonly used spectral algorithms can be constructed by certain analytic filter functions. We provide two examples (kernel ridge regression and kernel gradient flow) as follows, and put two more examples (iterated ridge regression and kernel gradient descent) in Appendix C. We provide rigorous proof for these examples in Lemma C.3.

Example 1 (Kernel ridge regression) . The filter function of kernel ridge regression (KRR) is wellknown to be

Example 2 (Kernel gradient flow) . The filter function is

For any analytic filter function φ λ with qualification τ ≥ 1 and the corresponding estimator of the spectral algorithm defined in (15), the following two theorems provide exact convergence rates on the excess risk when (i) the regression function is less-smooth, i.e., we have s ≤ τ , and (ii) s > τ , where s is the source condition coefficient of the regression function given in Assumption 2.

Theorem 4.1. Let 0 < s ≤ τ and γ > 0 be fixed real numbers. Denote p as the integer satisfying γ ∈ [ p ( s + 1) , ( p + 1)( s + 1)) . Suppose that Assumption 1 and Assumption 2 hold for s and γ . Let φ λ ( z ) be an analytic filter function and ˆ f λ be the function defined in (15) . Suppose one of the following conditions holds:

then there exists λ ⋆ > 0 , such that we have

where Θ P only involves constants depending on s, σ, γ, c 0 , κ, c 1 and c 2 .

Theorem 4.2. Let s > τ and γ > 0 be fixed real numbers. Denote p as the integer satisfying γ ∈ [ p ( s +1) , ( p +1)( s +1)) . Suppose that Assumption 1 and Assumption 2 hold for s and γ . Let φ λ ( z ) be an analytic filter function and ˆ f λ be the function defined in (15) . Define ˜ s = min { s, 2 τ } , then there exists λ ⋆ > 0 , such that we have

where Θ P only involves constants depending on s, σ, γ, c 0 , κ, c 1 and c 2 . In addition, the convergence rates of the generalization error can not be faster than above for any choice of regularization parameter λ = λ ( d, n ) → 0 .

Remark 4.3 . These theorems substantially generalize the results on exact generalization error bounds of analytic spectral algorithms under the fixed-dimensional setting given in Li et al. (2024). Although the 'analytic functional argument' introduced in their proof is still vital for us to deal with the general spectral algorithms, their proof has to rely on the polynomial eigendecay assumption that λ j ≍ j -β (Assumption 1), which does not hold in large dimensions since the hidden constant factors in the assumption vary with d 11 (Lu et al. (2023)). Hence, their proof is not easy to generalize to large-dimensional spectral algorithms.

We provide some graphical illustrations of Theorem 4.1 and Theorem 4.2 in Figure 1 (with τ = 2 ) and in Appendix A (with τ = 1 , τ = 2 , τ = 4 , and τ = ∞ , corresponding to KRR, iterated ridge regression in Example 3 and kernel gradient flow).

As a direct consequence of Theorem 3.3, Theorem 4.1, and Theorem 4.2, we find that for the spectral algorithm with estimator defined in (15), it is minimax optimal if s ≤ τ and the conditions in Theorem 4.1 hold. Moreover, these results show several phenomena for large-dimensional spectral algorithms.

Saturation effect of large-dimensional spectral algorithms with finite qualification. In the large-dimensional setting and for the inner product kernel on the sphere, our results show that the saturation effect of spectral algorithms occurs when s > τ . As shown in Figure 1(c) and Figure 1(d), when s > τ , no matter how carefully one tunes the regularization parameter λ , the convergence rate can not be faster than d -min { γ -p, τ ( γ -p +1)+ p ˜ s τ +1 , ˜ s ( p +1) } , thus can not achieve the minimax lower bound d -min { γ -p,s ( p +1) } .

Periodic plateau behavior of spectral algorithms when s ≤ 2 τ . When 0 < s ≤ 2 τ and γ ∈ [ p ( s +1) + s +max { s, τ } /τ -1 , ( p +1)( s +1)) for an integer p ≥ 0 , from Theorem 4.1 and Theorem 4.2, the convergence rate on the excess risk of spectral algorithm d -s ( p +1) . The above rate does not change when γ varies, which can also be found in Figure 1(b) and Figure 1(c). BIn other words, if we fix a large dimension d and increase γ (or equivalently, increase the sample size n ), the optimal rate of excess risk of a spectral algorithm stays invariant in certain ranges. Therefore, in order to improve the rate of excess risk, one has to increase the sample size above a certain threshold.

Polynomial approximation barrier of spectral algorithms when s → 0 . From Theorem 4.1, when s is close to zero, the convergence rate d -min { γ -p,s ( p +1) } is unchanged in the range γ ∈ [ p ( s +1)+ s, ( p +1)( s +1)) , and increases in the short range γ ∈ [ p ( s +1) , p ( s +1)+ s ) . In other words, the excess risk of spectral algorithms will drop when γ exceeds p ( s +1) ≈ p for any integer p and will stay invariant for most of the other γ . We term the above phenomenon as the polynomial approximation barrier of spectral algorithms (borrowed from Ghorbani et al. (2021)), and it can be illustrated by Figure 1(a) with s = 0 . 01 .

Remark 4.4 . Ghorbani et al. (2021) discovered the polynomial approximation barrier of KRR. As shown by Figure 5 and Theorem 4 in Ghorbani et al. (2021), if s = 0 and the true function falls into L 2 = [ H ] 0 , then with high probability we have

where p is the integer satisfying γ ∈ [ p, p + 1) , λ ⋆ is defined as in Theorem 4 in Ghorbani et al. (2021), P >ℓ means the projection onto polynomials with degree > ℓ , and ε is any positive real number. Notice that (18) implies that the excess risk of KRR will drop when γ exceeds any integer and will stay invariant for other γ , and is consistent with our results for spectral algorithms.

5 Conclusion

In this paper, we rigorously prove the saturation effect of KRR in large dimensions. Let s > 0 and γ > 0 be fixed real numbers, denote p as the integer satisfying γ ∈ [ p ( s +1) , ( p +1)( s +1)) . Given that the kernel is an inner product kernel defined on the sphere and that f ⋆ falls into the interpolation space [ H ] s , we first show that the convergence rate on the excess risk of large-dimensional kernel gradient flow is Θ P ( d -min { γ -p,s ( p +1) } ) · poly (ln( d )) (Theorem 3.1), which is faster than that of KRR given in Zhang et al. (2024). We then determine the improved minimax lower bound as Ω ( d -min { γ -p,s ( p +1) } ) / poly (ln( d )) (Theorem 3.3). Combining these results, we know that kernel gradient flow is minimax optimal in large dimensions, and KRR is inferior to kernel gradient flow in large dimensions. Our results suggest that previous results on large-dimensional KRR may not be directly extendable to large-dimensional neural networks if the regression function is over-smooth.

In Section 4, we generalize our results to certain spectral algorithms. We determine the convergence rate on the excess risk of large-dimensional spectral algorithms (Theorem 4.1 and Theorem 4.2). From these results, we find several new phenomena exhibited in large-dimensional spectral algorithms, including the saturation effect, the periodic plateau behavior, and the polynomial approximation barrier.

In this paper, we only consider the convergence rate on the excess risk of optimal-tuned largedimensional spectral algorithms with uniform input distribution on a hypersphere. We believe that several results in fixed-dimensional settings with input distribution on more general domains (e.g., Haas et al. (2024); Li et al. (2024)) can indeed be extended to large-dimensional settings, although we must carefully consider the constants that depend on d . Furthermore, we believe that by considering the learning curve of large-dimensional spectral algorithms (i.e., the convergence rate on the excess risk of spectral algorithms with any regularization parameter λ > 0 ) or the convergence rate on the excess risk of large-dimensional kernel interpolation (i.e., KRR with λ = 0 ), further research can find a wealth of new phenomena compared with the fixed-dimensional setting.

Acknowledgments and Disclosure of Funding

Lin's research was supported in part by the National Natural Science Foundation of China (Grant 92370122, Grant 11971257). The authors are grateful to the reviewers for their constructive comments that greatly improved the quality and presentation of this paper.

References

Aerni, M., M. Milanta, K. Donhauser, and F. Yang (2023). Strong inductive biases provably prevent harmless interpolation. arXiv preprint arXiv:2301.07605 .

Arora, S., S. Du, W. Hu, Z. Li, and R. Wang (2019). Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning , pp. 322-332. PMLR.

Arora, S., S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang (2019). On exact computation with an infinitely wide neural net. Advances in Neural Information Processing Systems 32 .

Barzilai, D. and O. Shamir (2023). Generalization in kernel regression under realistic assumptions. arXiv preprint arXiv:2312.15995 .

Bauer, F., S. Pereverzev, and L. Rosasco (2007). On regularization algorithms in learning theory. Journal of Complexity 23 (1), 52-72.

Caponnetto, A. (2006, September). Optimal rates for regularization operators in learning theory. Technical Report CBCL Paper #264/AI Technical Report #062, Massachusetts Institute of Technology.

Caponnetto, A. and E. De Vito (2007). Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics 7 (3), 331-368.

Caponnetto, A. and Y. Yao (2010). Cross-validation based adaptation for regularization operators in learning theory. Analysis and Applications 8 (02), 161-183.

Celisse, A. and M. Wahl (2021). Analyzing the discrepancy principle for kernelized spectral filter learning algorithms. Journal of Machine Learning Research 22 (76), 1-59.

Cleveland, W. S. (1979). Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association 74 (368), 829-836.

Cui, H., B. Loureiro, F. Krzakala, and L. Zdeborová (2021). Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems 34 , 10131-10143.

Devlin, J. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 .

Dicker, L. H., D. P. Foster, and D. Hsu (2017). Kernel ridge vs. principal component regression: Minimax bounds and the qualification of regularization operators. Electronic Journal of Statistics 11 (1), 1022 - 1047.

Dieuleveut, A., N. Flammarion, and F. Bach (2017). Harder, better, faster, stronger convergence rates for least-squares regression. Journal of Machine Learning Research 18 (101), 1-51.

Donhauser, K., M. Wu, and F. Yang (2021). How rotational invariance of common kernels prevents generalization in high dimensions. In International Conference on Machine Learning , pp. 28042814. PMLR.

Du, S., J. Lee, H. Li, L. Wang, and X. Zhai (2019). Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning , pp. 1675-1685. PMLR.

Du, S. S., X. Zhai, B. Poczos, and A. Singh (2018). Gradient descent provably optimizes overparameterized neural networks. arXiv preprint arXiv:1810.02054 .

Edmunds, D. E. and H. Triebel (1996). Function Spaces, Entropy Numbers, Differential Operators . Cambridge: Cambridge University Press.

Fischer, S. and I. Steinwart (2020). Sobolev norm learning rates for regularized least-squares algorithms. Journal of Machine Learning Research 21 (205), 1-38.

Gallier, J. (2009). Notes on spherical harmonics and linear representations of lie groups. preprint .

Gerfo, L. L., L. Rosasco, F. Odone, E. D. Vito, and A. Verri (2008, 07). Spectral Algorithms for Supervised Learning. Neural Computation 20 (7), 1873-1897.

Ghorbani, B., S. Mei, T. Misiakiewicz, and A. Montanari (2020). When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems 33 , 1482014830.

Ghorbani, B., S. Mei, T. Misiakiewicz, and A. Montanari (2021). Linearized two-layers neural networks in high dimension. The Annals of Statistics 49 (2), 1029 - 1054.

Ghosh, N., S. Mei, and B. Yu (2021). The three stages of learning dynamics in high-dimensional kernel methods. arXiv preprint arXiv:2111.07167 .

Gneiting, T. (2013). Strictly and non-strictly positive definite functions on spheres. Bernoulli 19 (4), 1327 - 1349.

Haas, M., D. Holzmüller, U. Luxburg, and I. Steinwart (2024). Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. Advances in Neural Information Processing Systems 36 .

He, K., X. Zhang, S. Ren, and J. Sun (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 770-778.

Hu, H. and Y. M. Lu (2022). Sharp asymptotics of kernel ridge regression beyond the linear regime. arXiv preprint arXiv:2205.06798 .

Hu, T., W. Wang, C. Lin, and G. Cheng (2021). Regularization matters: A nonparametric perspective on overparametrized neural network. In International Conference on Artificial Intelligence and Statistics , pp. 829-837. PMLR.

Jacot, A., F. Gabriel, and C. Hongler (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems 31 .

Krizhevsky, A., I. Sutskever, and G. E. Hinton (2017). Imagenet classification with deep convolutional neural networks. Communications of the ACM 60 (6), 84-90.

Lai, J., M. Xu, R. Chen, and Q. Lin (2023). Generalization ability of wide neural networks on R . arXiv preprint arXiv:2302.05933 .

Li, Y., W. Gan, Z. Shi, and Q. Lin (2024). Generalization error curves for analytic spectral algorithms under power-law decay. arXiv preprint arXiv:2401.01599 .

Li, Y. and Y. Liang (2018). Learning overparameterized neural networks via stochastic gradient descent on structured data. Advances in Neural Information Processing Systems 31 .

Li, Y., Z. Yu, G. Chen, and Q. Lin (2024). On the eigenvalue decay rates of a class of neural-network related kernel functions defined on general domains. Journal of Machine Learning Research 25 (82), 1-47.

Li, Y., H. Zhang, and Q. Lin (2022). On the saturation effect of kernel ridge regression. In The Eleventh International Conference on Learning Representations .

Li, Y., H. Zhang, and Q. Lin (2024). On the asymptotic learning curves of kernel ridge regression under power-law decay. Advances in Neural Information Processing Systems 36 .

Liang, T. and A. Rakhlin (2020). Just interpolate: Kernel 'Ridgeless' regression can generalize. The Annals of Statistics 48 (3), 1329 - 1347.

Liang, T., A. Rakhlin, and X. Zhai (2020). On the multiple descent of minimum-norm interpolants and restricted lower isometry of kernels. In Conference on Learning Theory , pp. 2683-2711. PMLR.

Lin, J., A. Rudi, L. Rosasco, and V. Cevher (2020). Optimal rates for spectral algorithms with least-squares regression over hilbert spaces. Applied and Computational Harmonic Analysis 48 (3), 868-890.

Lu, W., H. Zhang, Y. Li, M. Xu, and Q. Lin (2023). Optimal rate of kernel regression in large dimensions. arXiv preprint arXiv:2309.04268 .

Mei, S., T. Misiakiewicz, and A. Montanari (2021). Learning with invariances in random features and kernel models. In Conference on Learning Theory , pp. 3351-3418. PMLR.

Mei, S., T. Misiakiewicz, and A. Montanari (2022). Generalization error of random feature and kernel methods: Hypercontractivity and kernel matrix concentration. Applied and Computational Harmonic Analysis 59 , 3-84.

Mei, S. and A. Montanari (2022). The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics 75 (4), 667-766.

Misiakiewicz, T. (2022). Spectrum of inner-product kernel matrices in the polynomial regime and multiple descent phenomenon in kernel ridge regression. arXiv preprint arXiv:2204.10425 .

Misiakiewicz, T. and S. Mei (2022). Learning with convolution and pooling operations in kernel methods. Advances in Neural Information Processing Systems 35 , 29014-29025.

Pillaud-Vivien, L., A. Rudi, and F. Bach (2018). Statistical optimality of stochastic gradient descent on hard learning problems through multiple passes. Advances in Neural Information Processing Systems 31 .

Raskutti, G., M. J. Wainwright, and B. Yu (2014). Early stopping and non-parametric regression: An optimal data-dependent stopping rule. Journal of Machine Learning Research 15 (11), 335-366.

Rastogi, A. and S. Sampath (2017). Optimal rates for the regularized learning algorithms under general source condition. Frontiers in Applied Mathematics and Statistics 3 , 3.

Steinwart, I. and A. Christmann (2008). Support vector machines . Springer Science & Business Media.

Steinwart, I., D. Hush, and C. Scovel (2009). Optimal rates for regularized least squares regression. In Conference on Learning Theory , pp. 79-93. PMLR.

Steinwart, I. and C. Scovel (2012). Mercer's theorem on general domains: On the interaction between measures, kernels, and rkhss. Constructive Approximation 35 , 363-417.

Stone, C. J. (1977). Consistent Nonparametric Regression. The Annals of Statistics 5 (4), 595 - 620.

Stone, C. J. (1994). The Use of Polynomial Splines and Their Tensor Products in Multivariate Function Estimation. The Annals of Statistics 22 (1), 118 - 171.

Suh, N., H. Ko, and X. Huo (2021). A non-parametric regression viewpoint: Generalization of overparametrized deep relu network under noisy observations. In International Conference on Learning Representations .

Xiao, L., H. Hu, T. Misiakiewicz, Y. M. Lu, and J. Pennington (2023). Precise learning curves and higher-order scaling limits for dot product kernel regression. Journal of Statistical Mechanics: Theory and Experiment 2023 (11), 114005.

Yang, Y. and A. Barron (1999). Information-theoretic determination of minimax rates of convergence. The Annals of Statistics 27 (5), 1564 - 1599.

Yao, Y., L. Rosasco, and A. Caponnetto (2007). On early stopping in gradient descent learning. Constructive Approximation 26 , 289-315.

Zhang, H., Y. Li, W. Lu, and Q. Lin (2023). On the optimality of misspecified kernel ridge regression. In International Conference on Machine Learning , pp. 41331-41353. PMLR.

Zhang, H., Y. Li, W. Lu, and Q. Lin (2024). Optimal rates of kernel ridge regression under source condition in large dimensions. arXiv preprint arXiv:2401.01270 .

Zhang, H., W. Lu, and Q. Lin (2024). The phase diagram of kernel interpolation in large dimensions. arXiv preprint arXiv:2404.12597 .

A Graphical illustration and numerical experiments of main results

A.1 Graphical illustration of Theorem 3.1, Theorem 4.1, and Theorem 4.2

Recall that Theorem 3.1, Theorem 4.1, and Theorem 4.2 determined the convergence rate on the excess risk of: (i) large-dimensional kernel gradient flow with s > 0 ; (ii) large-dimensional spectral algorithm with τ ≥ 1 and s ≤ τ ; and (iii) large-dimensional spectral algorithm with τ ≥ 1 and s > τ .

In Figure 1, we have provided a visual illustration of Theorem 4.1 and Theorem 4.2 when τ = 2 . Now, in Figure 2, we provide more visual illustrations of the results of spectral algorithms with τ = 1 , τ = 2 , τ = 4 , and τ = ∞ , which correspond to kernel ridge regression (KRR), iterated ridge regression in Example 3, and kernel gradient flow.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

A.2 Numerical experiments

We conducted two experiments using two specific kernels: the RBF kernel and the NTK kernel. Experiment 1 was designed to confirm the optimal rate of kernel gradient flow and KRR when s = 1 . Experiment 2 was designed to illustrate the saturation effect of KRR when s > 1 .

Experiment 1: We consider the following two inner product kernels:

(i) RBF kernel with a fixed bandwidth:

(ii) Neural Tangent Kernel (NTK) of a two-layer ReLU neural network:

The RBF kernel satisfies Assumption 1. For the NTK, the coefficients of Φ( · ) , { a j } ∞ j =0 , satisfy a j > 0 , j ∈ { 0 , 1 }∪{ 2 , 4 , 6 , . . . } and a j = 0 , j ∈ { 3 , 5 , 7 , . . . } (see, e.g., Lu et al. (2023)). As noted after Assumption 1, our results can be extended to inner product kernels with certain zero coefficients a j . Specifically, for any γ > 0 , as long as a j > 0 for j = ⌊ γ ⌋ , ⌊ γ ⌋ +1 , the proof and convergence rate remain the same. Therefore, for γ < 2 in our experiments, the convergence rates for NTK will be the same as for the RBF kernel.

We used the following data generation procedure:

where each x i is i.i.d. sampled from the uniform distribution on S d , and ϵ i i.i.d. ∼ N (0 , 1) .

We selected the training sample sizes n with corresponding dimensions d such that n = d γ , γ = 0 . 5 , 1 . 0 , 1 . 5 , 1 . 8 . For each kernel and dimension d , we consider the following regression function f ∗ :

This function is in the RKHS H , and it is easy to prove that, for any u 0 ∈ S d , Assumption 2 (b) holds for K ( u 0 , · ) with s = 1 . Therefore, Assumption 2 holds for s = 1 . We used logarithmic least squares to fit the excess risk with respect to the sample size, resulting in the convergence rate r . As shown in Figure 3 and Figure 4, the experimental results align well with our theoretical findings.

Experiment 2: We use most of the settings from Experiment 1, except that the regression function is changed to f ∗ ( x ) = √ µ s 2 N ( d, 2) P 2 ( < ξ,x > ) with s = 1 . 9 , P 2 ( t ) := ( dt 2 -1) / ( d -1) the Gegenbauer polynomial, and ξ ∈ S d . Notice that the addition formula P 2 ( < ξ,x > ) = 1 N ( d, 2) ∑ N ( d, 2) j =1 Y 2 ,j ( ξ ) Y 2 ,j ( x ) implies that

hence f ∗ ∈ [ H ] s and satisfies Assumption 2.

Our experiment settings are similar to those on page 30 of Li et al. (2022). We choose the regularization parameter for KRR and kernel gradient flow as λ = 0 . 05 · d -θ . For KRR, since Corollary D.16 suggests that the optimal regularization parameter is λ ≍ d -0 . 7 , we set θ = 0 . 7 . Similarly, based on Corollary D.16, we set θ = 0 . 5 for kernel gradient flow. Additionally, we set γ = 1 . 8 . The results indicate that the best convergence rate of KRR is slower than that of kernel gradient flow, implying that KRR is inferior to kernel gradient flow when the regression function is sufficiently smooth.

B Proof of Theorem 3.3

We first restate Theorem 3.3.

2.00

2.05

2.10

2.15

2.20

2.25

2.30

log10

n

2.10

2.05

2.00

1.95

1.90

1.85

1.80

1.75

1.70

log10

NTK,

=0.5,  theretical rate=-1

KRR, parameters chosen by CV

log10

1.03log10

n

+ 0.37

Kernel regression

log10

1.09log10

n

+ 0.42

(a)

2.70

2.75

2.80

2.85

2.90

2.95

3.00

log10

n

2.1

2.0

1.9

1.8

1.7

log10

NTK,

=0.8,  theretical rate=-1

KRR, parameters chosen by CV

log10

0.84log10

n

+ 0.38

Kernel regression

log10

0.84log10

n

+ 0.59

(b)

3.0

3.1

3.2

3.3

3.4

3.5

3.6

3.7

log10

n

2.2

2.1

2.0

1.9

1.8

1.7

log10

NTK,

=1.5,  theretical rate=-2/3

KRR, parameters chosen by CV

log10

0.63log10

n

+ 0.23

Kernel regression

log10

0.66log10

n

+ 0.21

(c)

3.0

3.1

3.2

3.3

3.4

3.5

3.6

3.7

log10

n

2.0

1.9

1.8

1.7

1.6

log10

NTK,

=1.8,  theretical rate=-5/9

KRR, parameters chosen by CV

log10

0.54log10

n

+ 0.03

Kernel regression

log10

0.56log10

n

+ 0.04

(d)

(i) When γ ∈ ( p ( s +1) , p + ps + s ] , we have

(ii) When γ ∈ ( p + ps + s, ( p +1)( s +1)] , we have

where Ω only involves constants depending on s, σ, γ, c 0 , κ, c 1 , and c 2 .

Proof of Theorem B.1. The item (ii) is a direct corollary of Theorem 5 in Zhang et al. (2024). Now we begin to proof the item (i). We need the following lemma.

2.00

2.05

2.10

2.15

2.20

2.25

2.30

log10

n

2.3

2.2

2.1

2.0

1.9

1.8

log10

RBF,

=0.5,  theretical rate=-1

KRR, parameters chosen by CV

log10

1.23log10

n

+ 0.65

Kernel regression

log10

1.31log10

n

+ 0.73

(a)

2.70

2.75

2.80

2.85

2.90

2.95

3.00

log10

n

2.40

2.35

2.30

2.25

2.20

2.15

2.10

2.05

log10

RBF,

=0.8,  theretical rate=-1

KRR, parameters chosen by CV

log10

1.06log10

n

+ 0.76

Kernel regression

log10

0.90log10

n

+ 0.32

(b)

3.0

3.1

3.2

3.3

3.4

3.5

3.6

3.7

log10

n

2.8

2.7

2.6

2.5

2.4

2.3

2.2

2.1

2.0

log10

RBF,

=1.5,  theretical rate=-2/3

KRR, parameters chosen by CV

log10

0.65log10

n

+ -0.09

Kernel regression

log10

0.72log10

n

+ -0.09

(c)

3.0

3.1

3.2

3.3

3.4

3.5

3.6

3.7

log10

n

2.5

2.4

2.3

2.2

2.1

2.0

1.9

log10

RBF,

=1.8,  theretical rate=-5/9

KRR, parameters chosen by CV

log10

0.59log10

n

+ -0.13

Kernel regression

log10

0.61log10

n

+ -0.22

(d)

Lemma B.2 (Restate Lemma 4.1 in Lu et al. (2023)) . For any δ ∈ (0 , 1) and any 0 < ˜ ε 1 , ˜ ε 2 < ∞ only depending on n , d , { λ j } , c 1 , c 2 , and γ and satisfying

where ρ f ⋆ is the joint-p.d.f. of x, y given by (1) with f = f ⋆ , B := { f ∈ H , ∥ f ∥ [ H ] s ≤ R γ

}

and V 2 , V K are the ε -covering entropies ( as defined in Yang and Barron (1999); Lu et al. (2023)) of ( B , d 2 = ∥ · ∥ 2 L 2 ) and ( D , d 2 = KL divergence ) .

Suppose γ ∈ ( p ( s +1) , p + ps + s ] . Let C ( p ) = C 12 / 10 be a constant only depending on γ , where C 12 are given in Lemma D.13. Then we introduce

we have

3.0

3.1

3.2

3.3

3.4

log10

n

4.5

4.0

3.5

3.0

2.5

2.0

1.5

log10

NTK,

=1.8

Kernel gradient flow,

=0.5

log10

0.88log10

n

+ -1.37

KRR,

=0.7

log10

0.52log10

n

+ 0.09

Let us further assume that d ≥ C , where C is a sufficiently large constant only depending on γ , s , and c 1 . By Lemma D.11 and Lemma D.13 we have

Therefore, for any d ≥ C , where C is a sufficiently large constant only depending on s , γ , and c 1 , we have

On the other hand, from Lemma D.11, Lemma D.13, and Lemma D.12, one can check the following claim:

Claim 1. Suppose γ ∈ ( p ( s +1) , p + ps + s ] . For any d ≥ C , where C is a sufficiently large constant only depending on s , γ , c 1 , and c 2 , we have

Therefore, for any d ≥ C , where C is a sufficiently large constant only depending on s , γ , c 1 , and c 2 , we have

Combining (23), (24), and (25), we finally have:

and from Lemma B.2, we get

finishing the proof.

C Definition of analytic filter functions

We first introduce the following definition of analytic filter functions (Bauer et al. (2007); Li et al. (2024)).

Definition C.1 (Analytic filter functions) . Let { φ λ : [0 , κ 2 ] → R ≥ 0 | λ ∈ (0 , 1) } be a family of functions indexed with regularization parameter λ and define the remainder function

We say that { φ λ | λ ∈ (0 , 1) } (or simply φ λ ( z ) ) is an analytic filter function if:

zφ λ ( z ) ∈ [0 , 1] is non-decreasing with respect to z and non-increasing with respect to λ .

The qualification of this filter function is τ ∈ [1 , ∞ ] such that ∀ 0 ≤ τ ′ ≤ τ (and also τ ′ < ∞ ), there exist positive constants C i only depending on τ ′ , i = 1 , 2 , 3 , 4 , 5 , such that we have

If τ < ∞ , then there exists a positive constant C 6 only depending on τ and λ 1 , such that we have

where λ 1 is the largest eigenvalue of K defined in (5); and there exist positive constants C 7 and C 8 only depending on τ , such that we have

(4) Let

Then φ λ ( z ) can be extended to be an analytic function on some domain containing D λ and the following conditions holds for all λ ∈ (0 , 1) :

■

where ˜ E, ˜ F are positive constants.

Remark C.2 . We remark that some of the above properties are not essential for the definition of filter functions in the literature (Bauer et al., 2007; Gerfo et al., 2008), but we introduce them to avoid some unnecessary technicalities in the proof. The requirements of analytic filter functions are first considered in Li et al. (2024) and used for their 'analytic functional argument', which will also be vital in our proof.

The following examples show many commonly used analytic filter functions and their proofs can be found in Lemma C.3, see also Li et al. (2024).

Example 3 (Iterated ridge regression) . Let q ≥ 1 be fixed. We define

Example 4 (Kernel gradient descent) . The gradient descent method is the discrete version of gradient flow. Let η > 0 be a fixed step size. Then, iterating gradient descent with respect to the empirical loss t steps yields the filter function

Moreover, when η is small enough, say η < 1 / (2 κ 2 ) , we have Re(1 -ηz ) > 0 for z ∈ D λ , so we can take the single-valued branch of (1 -ηz ) t even when t is not an integer. Therefore, we can extend the definition of the filter function so that λ can be arbitrary and t = ( ηλ ) -1 .

Lemma C.3. φ KRR λ , φ IT ,q λ , φ GF λ , and φ GD λ are analytic filter functions.

Proof. Notice that (i) z ≤ z + λ ≤ 2 z when z > λ ; and that (ii) λ ≤ z + λ ≤ 2 λ when z ≤ λ . Hence, the constants C 1 , C 2 , C 3 , C 4 , and C 6 are given in Li et al. (2024).

For C 5 , when z ≤ λ , we can take C 5 = min { 1 / 2 , 2 -q , e -1 , e -1 } > 0 .

For C 7 , when z > λ , we have

For C 8 , when z ≤ λ , we have

D Proof of Theorem 4.1 and Theorem 4.2

D.1 Bias-variance decomposition

We first apply a standard bias-variance decomposition on the excess risk of spectral algorithms, and readers can also refer to Zhang et al. (2023, 2024) for more details.

Recall the definition of ˆ g Z and ˆ f λ in (14) and (15). Let's define their conditional expectations as

and

Let's also define their expectations as

Then we have the decomposition

Taking expectation over the noise ϵ conditioned on X and noticing that ϵ | X are independent noise with mean 0 and variance σ 2 , we obtain the bias-variance decomposition:

where

Given the decomposition (40), we next derive the upper and lower bounds of Bias 2 ( λ ) and Var ( λ ) in the following two subsections.

Before we close this subsection, let's introduce some quantities and an assumption that will be used frequently in our proof later. Denote the true function as f ⋆ = ∞ ∑ i =1 f i ϕ i ( x ) , let's define the following quantities:

moreover, when φ λ = φ KRR λ , we denote N k ( λ ) = N k,φ KRR ( λ ) and M k ( λ ) = M k,φ KRR ( λ ) for simplicity, where k = 1 , 2 .

Assumption 3 . Suppose that

and

and and

where

Proof. Define a function

For simplicity of notations, we denote h x ( · ) = K ( x, · ) , x ∈ X in the rest of the proof. Moreover, we denote T λ := ( T + λ ) -1 and T Xλ := ( T X + λ ) -1 .

D.2 Variance term

The following proposition rewrites the variance term using the empirical semi-norm.

Proposition D.1 (Restate Lemma 9 in Zhang et al. (2024)) . The variance term in (41) satisfies that

The operator form (46) allows us to apply concentration inequalities and establish the following two-step approximation.

Approximation B The following lemma characterizes the magnitude of Approximation B in high probability. Recall the definitions of N 1 ,φ ( λ ) and N 2 ,φ ( λ ) in (42).

Lemma D.2 (Approximation B) . Suppose that (43) in Assumption 3 holds. Then, for any fixed δ ∈ (0 , 1) , with probability at least 1 -δ , we have

Since (43) in Assumption 3 holds, we have

Applying Proposition 34 in Zhang et al. (2024) for √ f and noticing that ∥ √ f ∥ L ∞ = √ ∥ f ∥ L ∞ = N 2 ,φ ( λ ) 1 2 , we have

with probability at least 1 -δ .

On the one hand, we have

On the other hand, we have

Therefore, (53) implies the desired results.

Approximation A

Lemma D.3. Suppose that (43) and (45) in Assumption 3 hold. Suppose that there exists a constant ϵ only depending on s and γ , such that λ = λ ( n, d ) satisfies n ϵ -1 N 1 ( λ ) → 0 . Then there exists an absolute constant C 1 , such that for any fixed δ ∈ (0 , 1) , when n is sufficiently large, with probability at least 1 -δ , we have

where v = N 1 ( λ ) n ln n .

Remark D.4 . The proof of Lemma D.3 is mainly based on Lemma 4.18 in Li et al. (2024). Notice that we replace the Assumption 2 in Li et al. (2024) by (45) in Assumption 3 (borrowed from Zhang et al. (2024)), since both of them can deduce same results given by Lemma 4.2 in Li et al. (2024) or Lemma 37 in Zhang et al. (2024).

Proof. We start with

Using operator calculus, we get

■

Therefore, taking the norms yields

where in the second estimation, we use ( I ) operator calculus, ( II and IV ) Proposition E.8, ( III ) Lemma E.7, and ( V ) Lemma 37 in Zhang et al. (2024) for each term respectively. Finally, from (63) in Li et al. (2024), we get

and thus there exists an absolute constant C 1 , such that we have

On the other hand, combining (52) and (43) in Assumption 3, we have ∥ φ λ ( T ) h x ∥ 2 L 2 ≤ N 2 ,φ ( λ ) , and hence

Finally, and hence

then we have

■

Final proof of the variance term Now we are ready to state the theorem about the variance term. Theorem D.5. Suppose that (43) and (45) in Assumption 3 hold. Suppose there exists a constant ϵ > 0 only depending on s and γ , such that λ = λ ( n, d ) satisfies

Proof. Recall that Var ( λ ) = σ 2 n ∫ X ∥ φ λ ( T X ) h x ∥ 2 L 2 ,n d ρ X ( x ) . Hence, when n is large enough, with probability at least 1 -δ we have

When n ≥ C , a sufficiently large constant only depending on γ and C 1 , we have

Furthermore, when N 2 1 ( λ ) n N 2 ,φ ( λ ) · n ϵ → 0 , we have I · C 1 √ ln( n ) ln λ -1 / N 2 ,φ ( λ ) → 0 and II · C 2 1 ln( n )(ln λ -1 ) 2 / N 2 ,φ ( λ ) → 0 .

Finally, from (52) we have

D.3 Bias term

In this subsection, our goal is to determine the upper and lower bounds of bias under some approximation conditions.

The triangle inequality implies that

The following lemma characterizes the dominant term of Bias ( λ ) .

Lemma D.6. For any λ > 0 , we have

and thus the deterministic term writes

Proof. We have

■

The following lemma bounds the remainder term of Bias ( λ ) when s ≥ 1 .

Lemma D.7. Suppose that (45) in Assumption 3 holds. Suppose that there exist constants ϵ and C only depending on s and γ , such that λ = λ ( n, d ) satisfies

then we have

Proof. Do the decomposition,

Bound on I : For the first term in (66), we have

Denote ξ i = ξ ( x i ) = T -1 2 λ ( K x i f ⋆ ( x i ) -T x i f λ ) . To use Bernstein inequality, we need to bound the m -th moment of ξ ( x ) :

Note that Lemma 37 in Zhang et al. (2024) shows that

By definition of M 1 ,φ ( λ ) , we also have

In addition, we have proved in Lemma D.6 that

So we get the upper bound of (67), i.e.,

Using Lemma 36 in Zhang et al. (2024) with therein notations: L = N 1 ( λ ) 1 2 M 1 ,φ ( λ ) and σ = N 1 ( λ ) 1 2 M 2 ,φ ( λ ) 1 2 , for any fixed δ ∈ (0 , 1) , with probability at least 1 -δ , we have

Bound on II : For the second term in (66), we have

For the first term in (70), we still employ the analytic functional argument:

Therefore,

where v = N 1 ( λ ) n ln n .

For the second term in (70), we have

Hence, similar to (71), we have

Definition of analytic filter functions

Combining (66), (69), (70), (71), and (72), there exists a constant C 1 only depending on δ and ˜ F , such that we have

When s < 1 , we can use the following lemma to bound the remainder term of Bias ( λ ) . This lemma is a modification of Lemma D.7, and its proof is partly based on Lemma 26 in Zhang.

Lemma D.8. Suppose that (45) in Assumption 3 holds. Suppose that there exist constants ϵ and C only depending on s and γ , such that λ = λ ( n, d ) satisfies

then we have

Proof. Similar to the proof in Lemma D.7, we have the decomposition ˜ f λ -f λ = I + II , with

Denote ξ i = ξ ( x i ) = T -1 2 λ ( K x i f ⋆ ( x i ) -T x i f λ ) . Further consider the subset Ω 1 = { x ∈ X : | f ⋆ ( x ) | ≤ t } and Ω 2 = X\ Ω 1 , where t will be chosen appropriately later. Decompose ξ i as ξ i I x i ∈ Ω 1 + ξ i I x i ∈ Ω 2 and we have the following decomposition:

Next we choose t = n 1 -s 2 + ϵ t , q = 2 1 -s -ϵ q such that

Then we can bound the three terms in (78) as follows:

( i ) For the first term in (78), denoted as I, notice that

Imitating (67) in the proof of Lemma D.7, we have

( ii ) For the second term in (78), denoted as II. Since q = 2 1 -s -ϵ q < 2 1 -s , Lemma 42 in Zhang et al. (2024) shows that,

with embedding norm less than a constant C s,κ . Then Assumption 2 (a) implies that there exists 0 < C q < ∞ only depending on γ, s and κ such that ∥ f ⋆ ∥ L q ( X ,µ ) ≤ C q . Using the Markov inequality, we have

Further, since (80) guarantees t q ≫ n , we have

( iii ) For the third term in (78), denoted as III. Since Lemma 37 in Zhang et al. (2024) implies that ∥ T -1 2 λ k ( x, · ) ∥ H ≤ N 1 ( λ ) 1 2 , µ -a.e. x ∈ X , so

where we use Cauchy-Schwarz inequality for the third inequality and Lemma D.6 for the fourth inequality. Recalling that the choices of t, q satisfy t -q ≪ n -1 and we have assumed n ϵ -1 N 1 ( λ ) → 0 , we have

Plugging (82), (84) and (87) into (78), we finish the proof.

■

Final proof of the bias term Now we are ready to state the theorem about the bias term.

Theorem D.9 ( s ≥ 1 ) . Suppose that (45) in Assumption 3 holds. Suppose that there exist constants ϵ and C only depending on s and γ , such that λ = λ ( n, d ) satisfies

Theorem D.10 ( s < 1 ) . Suppose that (45) in Assumption 3 holds. Suppose that there exist constants ϵ and C only depending on s and γ , such that λ = λ ( n, d ) satisfies then we have

D.4 Quantity calculations and conditions verification for the inner product kernels

In the previous two sections, we have successfully bounded the bias and the variance terms by the quantities M 2 ,φ ( λ ) and N 2 ,φ ( λ ) . In this subsection, we will focus on the inner product kernels on the sphere. We will (i) determine the rates for the above quantities, and (ii) verify all the conditions in Theorem D.5, Theorem D.9 and Theorem D.10.

Recall that µ k and N ( d, k ) , defined in (9), are the eigenvalues of the inner product kernel K defined on the sphere and the corresponding multiplicity. The following three lemmas (mainly cited from Lu et al. (2023)) give concise characterizations of µ k and N ( d, k ) , which is sufficient for the analysis in this paper.

Lemma D.11. For any fixed integer p ≥ 0 , there exist constants C , C 9 and C 10 only depending on p and { a j } j ≤ p +1 , such that for any d ≥ C , we have

Lemma D.12. For any fixed integer p ≥ 0 , there exist constants C only depending on p and { a j } j ≤ p +1 , such that for any d ≥ C , we have

where C 9 and C 10 are constants given in Lemma D.11.

Lemma D.13. For any fixed integer p ≥ 0 , there exist constants C 11 , C 12 and C only depending on p , such that for any d ≥ C , we have

With these lemmas, we can begin to bound the quantities M 2 ,φ ( λ ) and N 2 ,φ ( λ ) .

Lemma D.14. Suppose that Assumption 1 and Assumption 2 hold for s and an integer p . Suppose ℓ ≤ p , t = λ -1 ∈ ( d ℓ , d ℓ +1 ] . Then we have the following bound.

and thus Assumption 3 holds. Moreover, when s ≥ 1 , We have

Proof. I. We begin with M 2 ,φ ( λ ) . If s ≤ 2 τ and τ < ∞ , then we have

and when τ = ∞ , a similar argument ( taking τ ′ < τ and let τ ′ →∞ , then we have ( td -ℓ ) -2 τ ′ → 0 ) shows that M 2 ,φ ( λ ) = O ( d -s ( ℓ +1) ) . Similarly, if s ≤ 2 τ , then we have

If 2 τ < s , then

Similarly, if 2 τ < s , then we have

Now let's bound the second term N 2 ,φ ( λ ) /n . We have

Similarly, we have

For the third term, we have

Now we show that Assumption 3 holds. Notice that (45) has been verified in Lemma 20 of Zhang et al. (2024). Similarly, one can prove (43) and (44) hold using a similar proof as that for Lemma 20 of Zhang et al. (2024).

For the final term, when s ≥ 1 , we have

For Q 1 ,φ ( λ ) , when τ ≥ s/ 2 and τ < ∞ , we have

Similarly, when τ = ∞ , we can show that Q 1 ,φ ( λ ) = O ( λd -( ℓ +1)( s -1) ) . And when τ < s/ 2 , we have

For N 1 ,φ ( λ ) , we have

Therefore, when s ≥ 1 , we have

■

From Lemma D.14, we have the following three corollaries.

Corollary D.15. Let 1 ≤ s ≤ τ and γ > 0 be fixed real numbers. Denote p as the integer satisfying γ ∈ [ p ( s +1) , ( p +1)( s +1)) . Suppose one of the following cases holds for λ ⋆ = d -ℓ or λ ⋆ = d -ℓ · poly (ln( d )) :

Then we have or

Corollary D.16. Let τ < s ≤ 2 τ and γ > 0 be fixed real numbers. Denote p as the integer satisfying γ ∈ [ p ( s +1) , ( p +1)( s +1)) . Denote ∆ = γ -p ( s +1) . Suppose one of the following cases holds for λ ⋆ = d -ℓ or λ ⋆ = d -ℓ · poly (ln( d )) :

Then we have

or

Proof. Denote I = -2 ℓτ +2 pτ -ps , II = -sp -s , III = p -γ , and IV = 2 ℓ -γ -p -1 . From Lemma D.14 we have

We can verify that:

When 0 ≤ ∆ ≤ τ and ℓ = p +∆ / (2 τ ) , we have

When τ ≤ ∆ ≤ s + s/τ -1 and ℓ = p +(∆+1) / (2 τ +2) , we have

When ∆ ≥ s + s/τ -1 and ℓ = p +(∆+1 -s ) / 2 , we have

When γ < 1 and ℓ = γ/ 2 , we have

Corollary D.17. Let s < 1 and γ > 0 be fixed real numbers. Denote p as the integer satisfying γ ∈ [ p ( s + 1) , ( p + 1)( s + 1)) . Suppose one of the following cases holds for λ ⋆ = d -ℓ or λ ⋆ = d -ℓ · poly (ln( d )) :

Then we have or

D.4.1 Verification of variance conditions

Lemma D.18 (Verification of variance conditions for inner-product kernels) . Suppose n ≍ d γ and s ≥ 1 , for γ ∈ [ p ( s +1) , ( p +1)( s +1)) . For any given ℓ ≥ 0 , if

then there exists a constant ϵ > 0 only depending on s and γ , such that λ = λ ( n, d ) satisfies

Proof. From Lemma 21 in Zhang et al. (2024), we have N 1 ( λ ) ≍ λ -1 . When p = 0 , we have γ -ℓ > 0 . When p ≥ 1 , we have γ -p -1 / 2 ≥ ps -1 / 2 > 0 . Therefore, there exists a constant ϵ > 0 only depending on s and γ , such that we have

Denote q := ⌊ ℓ ⌋ . From Lemma D.14, we further have N 2 ,φ ( λ ) = Ω ( d q + λ -2 d -q -1 ) . Hence, we have

Denote ∆ := (ln( d )) 3 nλ 2 d q , ∆ ′ := (ln( d )) 3 d γ -q -1 , then when ∆ = o (1) or ∆ ′ = o (1) , we have:

Now we show that ∆ = o (1) :

When p ≥ 3 and p = 2 , s > 1 , since γ -2 ℓ + q ≥ ( γ -ℓ -1) + ( q +1 -ℓ ) > 0 , we have ∆ = o (1) .

When p = 2 , s = 1 , since 2 ℓ -q < ℓ +1 < 4 ≤ γ , we have ∆ = o (1) .

When p = 2 , s = 1 , since 2 ℓ -q < ℓ +1 < 4 ≤ γ , we have ∆ = o (1) .

When p = 1 , γ > 2 s +1 , since ℓ < 2 and hence 2 ℓ -q < 3 ≤ γ , we have ∆ = o (1) .

When p = 1 , s > 1 , γ ≤ 2 s +1 , or p = 1 , s = 1 , γ > 2 , since 2 ℓ -q ≤ 2 < γ , we have ∆ = o (1) .

When p = 1 , s = 1 , γ = 2 , since 2 ℓ -q ≤ 2 ≤ γ , we have ∆ = O ((ln( d )) -1 ) .

When p = 0 , since γ -2 ℓ ≥ 0 , we have ∆ = O ((ln( d )) -1 ) .

■

LemmaD.19 (Verification of variance conditions for inner-product kernels: saturation case) . Suppose τ < s ≤ 2 τ . Suppose n ≍ d γ , for γ ∈ [ p ( s +1)+ τ, p ( s +1)+ s + s/τ -1] . For any given ℓ ≥ 0 , if

then there exists a constant ϵ > 0 only depending on s and γ , such that λ = λ ( n, d ) satisfies

Proof. From Lemma 21 in Zhang et al. (2024), we have N 1 ( λ ) ≍ λ -1 . Notice that we have

Therefore, there exists a constant ϵ > 0 only depending on τ , s , and γ , such that we have

Denote q := ⌊ ℓ ⌋ . From Lemma D.14, we further have N 2 ,φ ( λ ) = Ω ( d q + λ -2 d -q -1 ) . Hence, we have

Denote ∆ := (ln( d )) 3 nλ 2 d q , ∆ ′ := (ln( d )) 3 d γ -q -1 . We have:

When p ≥ 1 , since

we have ∆ = o (1) .

When p = 0 , since γ > 1 , we have ∆ ′ = o (1) .

■

Lemma D.20 (Verification of variance conditions for inner-product kernels: misspecified case) . Suppose n ≍ d γ and 0 < s < 1 , for γ ∈ [ p ( s +1) , ( p +1)( s +1)) . For any given ℓ ≥ 0 , if

then there exists a constant ϵ > 0 only depending on s and γ , such that λ = λ ( n, d ) satisfies

Proof. When p ≥ 1 , it is a direct result of step 2 (the verification of the second condition in (146) of Zhang et al. (2024)) in the proof of Theorem 3 in Zhang et al. (2024) and the fact that N 2 ,φ ( λ ) ≍ N 2 ( λ ) .

When p = 0 , a similar argument as the proof for Lemma D.18 give the desired results.

D.4.2 Verification of bias conditions

Lemma D.21 (Verification of bias conditions) . Suppose 1 ≤ s ≤ τ . Suppose n ≍ d γ , for γ ∈ [ p ( s +1) , ( p +1)( s +1)) . For any given ℓ ≥ 0 , if

then there exists a constant ϵ > 0 only depending on s and γ , such that λ = λ ( n, d ) satisfies

Proof. When 1 ≤ s ≤ τ , from Lemma D.14, we have

Denote I = λ 2( s -1) d -γ + qs , II = λ -1 d -γ -( q +1)( s -1) , III = λd max { q (2 -s ) , 0 } (ln( d )) 3 , and IV = λ -1 d -s ( q +1) (ln( d )) 3 .

For any p ≥ 0 and any s ≥ 1 :

From Lemma D.18, we have IV ≪ d γ -s ( q +1) .

When γ ≥ 1 , we have γ ≥ p +1 , and hence II ≪ IV ≪ d γ -s ( q +1) ; when γ < 1 , we have II ≪ d q with q = 0 .

When p ≥ 1 or γ ∈ ( s, s + 1) , since -ℓs + qs ≤ 0 , we have I /d γ -s ( q +1) = O ( d -2( γ -ℓ -s/ 2) ) ≪ 1 ; when γ ∈ (0 , s ] , we have I = O ( d -2 sℓ +2 ℓ -γ ) = O ( d -2 sℓ ) ≪ d q with q = 0 .

■

When s ≥ 2 , we have III ≪ d q ; when s < 2 and p = 0 , we have III ≪ d q ; when s < 2 and p ≥ 1 and q ≥ 1 , since γ -ℓ -s > min { ( s + 1) q -ℓ, ps -1 / 2 } > 0 , we have III /d γ -s ( q +1) = d -( γ -ℓ -s ) -2( ℓ -q ) ≪ 1 or III /d q ≪ 1 ; when s < 2 and p ≥ 1 and q = 0 , we have III ≪ d q .

Combining all these, we get the desired results.

■

Lemma D.22. [Verification of bias conditions: saturation case] Suppose τ < s ≤ 2 τ . Suppose n ≍ d γ , for γ ∈ [ p ( s +1) , ( p +1)( s +1)) . For any given ℓ ≥ 0 , if

where τ , ∆ , ℓ 1 , ℓ 2 , and ℓ 3 are given in Lemma D.16; then there exists a constant ϵ > 0 only depending on s and γ , such that λ = λ ( n, d ) satisfies

Proof. When τ < s ≤ 2 τ , from Lemma D.14, we have

Denote I ′ = λ 2( τ -1) d -γ + q (2 τ -s ) , II = λ -1 d -γ -( q +1)( s -1) , III = λd max { q (2 -s ) , 0 } (ln( d )) 3 , and IV = λ -1 d -s ( q +1) (ln( d )) 3 .

For any p ≥ 0 and any 1 ≤ τ < s ≤ 2 τ :

From Lemma D.18 and Lemma D.19, since N 1 ( λ ) · n ϵ -1 → 0 , we have IV ≪ d γ -s ( q +1) .

When γ ≥ 1 , we have γ ≥ p +1 , and hence II ≪ IV ≪ d γ -s ( q +1) ; when γ < 1 , we have II ≪ d q with q = 0 .

When p ≥ 1 , since -ℓτ + qτ ≤ 0 and

we have I ′ /d γ -s ( q +1) ≪ 1 ; when p = 0 , we have I ′ = O ( d -2 τℓ +2 ℓ -γ ) ≪ d q with q = 0 .

When γ -p -ps ∈ [0 , τ ] ∪ [ s + s/τ -1 , s +1] , we have ℓ ≤ max { ℓ 1 , ℓ 3 } . Similar to the proof in Lemma D.21, we can show that III ≪ d γ -s ( q +1) + d q .

Finally, consider the case γ -p -ps ∈ [ τ, s + s/τ -1] . When s ≥ 2 , we have III ≪ d q ; when s < 2 , since s > 1 , we have III /d q = λd -q ( s -1) ≪ 0 .

Combining all these, we get the desired results.

■

Lemma D.23 (Verification of bias conditions: misspecified case) . Suppose 0 < s < 1 . Suppose n ≍ d γ , for γ ∈ [ p ( s +1) , ( p +1)( s +1)) . Suppose one of the following holds:

τ = ∞

s > 1 / (2 τ )

Suppose one of the following cases holds for λ = d -ℓ or λ = d -ℓ (ln( d )) 2 :

then there exists a constant ϵ > 0 only depending on s and γ , such that λ = λ ( n, d ) satisfies

Proof. When 0 < s < 1 , from Lemma D.14, we have

and the convergence rate of ∥ f λ ∥ L ∞ can be attained similar to Lemma 25 in Zhang et al. (2024). Since τ ≥ 1 , similar to the proof of Theorem 3 of Zhang et al. (2024), when 1 / 2 < s < 1 , we have

Denote I = λ -1 d -γs , II = λd p (2 -s ) (ln( d )) 3 , and III = λ -1 d -s ( p +1) (ln( d )) 3 .

For any p ≥ 0 and any 0 < s < 1 :

From Lemma D.20, we have III ≪ d γ -s ( p +1) ,

When γ ≤ ps + p + s , we can show I ≪ d p when: (1) p ≥ 1 , or (2) p = 0 and s > 1 / (2 τ ) > 0 , or (3) τ = ∞ ,

When γ > ps + p + s , we can show I ≪ d γ -s ( p +1) holds if and only if τ = ∞ or

and the above inequality holds when (1) p > 0 or (2) p = 0 , s > 1 / (2 τ ) > 0 , or (3) p = 0 , γ > ((2 τ +1) s ) / (2 τ (1 + s )) ;

When γ ≤ ps + p + s , since ℓ ≥ p > p -ps , we have II ≪ d p ;

When γ > ps + p + s , since ℓ ≥ p > p -ps , we have II ≪ d γ -s ( p +1) .

Combining all these, we get the desired results.

D.5 Final proof of Theorem 4.1 and Theorem 4.2

For each case, the proof can be done in the following steps:

(i) When λ ≥ λ ⋆ and s ≤ 2 τ , where the definition of the balanced parameter λ ⋆ can be found in Corollary D.15 and Corollary D.16, we have

where d -β ⋆ is the desired convergence rate given in Theorem 4.1 or Theorem 4.2 and β ≤ β ∗ . Similarly, when s > 2 τ , by taking s = 2 τ in Corollary D.16, we also have

(ii) When λ ≥ λ ⋆ , from Lemma D.14, Lemma D.18, Lemma D.19, Lemma D.20, Lemma D.21, Lemma D.22, and Lemma D.23, we know that conditions in Theorem D.5, Theorem D.9, and Theorem D.10 are satisfied. Therefore, we have

(iii) Finally, when s > τ , we can further show that: the convergence rates of the generalization error can not be faster than above for any choice of regularization parameter λ = λ ( d, n ) → 0 . Notice that, when s ≥ 1 , for any λ < λ ⋆ , from the monotonicity of Var ( λ ) (see, e.g., Li et al. (2024); Zhang et al. (2024)), we have

E Auxiliary lemmas

Proposition E.1. For any analytic filter function φ λ , we have ( z + λ ) φ λ ( z ) ≤ 4 and ( z + λ ) ψ λ ( z ) ≤ 4 λ .

Proof. From (28), we have ( z + λ ) φ λ ( z ) ≤ 2 max { z, λ } φ λ ( z ) ≤ 2 max { 1 , C 4 } ≤ 4 . From (27), we have ( z + λ ) ψ λ ( z ) ≤ 2 max { z, λ } ψ λ ( z ) ≤ 2 max { C 2 , 1 } λ ≤ 4 λ . ■

Lemma E.2. Let φ λ be an analytic filter function defined in Definition C.1. Then, for any s ∈ [0 , 1] , we have

■

Proof. For any z ∈ [0 , κ 2 ] , from Proposition E.1, we have ( z + λ ) φ λ ( z ) ≤ 4 . Therefore, from Proposition B.3 in Li et al. (2024), we have

Lemma E.3. Let ψ λ be defined in Definition C.1. Then, for any s > 2 τ , we have

Proof. For any z , we have

hence

E.1 Analytic functional calculus

The 'analytic functional argument' introduced in Li et al. (2024) is vital in our proof for Theorem 4.1. For readers' convenience, we collect some of the main ingredients here, see Li et al. (2024) for details.

Definition E.4 . Let A be a linear operator on a Banach space X . The resolvent set ρ ( A ) is given by

and we denote R A ( λ ) := ( A -λ ) -1 . The spectrum of A is defined by

A simple but key ingredient in the analytic functional calculus is the following resolvent identity :

The resolvent allows us to define the value of f ( A ) in analog to the form of Cauchy integral formula, where A is an operator and f is an analytic function. The following two propositions are well-known results on operator calculus.

Proposition E.5 (analytic functional calculus) . Let A be an operator on a Hilbert space H and f be an analytic function defined on D f ⊂ C . Let Γ be a contour contained in D f surrounding σ ( A ) . Then,

and it is independent of the choice of Γ .

Now, let Γ be a contour contained in D f surrounding both σ ( A ) and σ ( B ) . Using (108), we get

Proposition E.6 (Spectral mapping theorem) . Let A be a bounded self-adjoint operator and f be a continuous function on σ ( A ) . Then

Consequently, ∥ f ( A ) ∥ = sup λ ∈ σ ( A ) | f ( λ ) | ≤ ∥ f ∥

Let us define the contour Γ λ considered in Li et al. (2024) by

where η = λ/ 2 . Then, since T and T X are positive self-adjoint operators with ∥ T ∥ , ∥ T X ∥ ≤ κ 2 , we have σ ( T ) , σ ( T X ) ⊂ [0 , κ 2 ] . Therefore, Γ λ is indeed a contour satisfying the requirement in Proposition E.5.

Proposition E.7. Suppose that (45) in Assumption 3 holds. Suppose that λ = λ ( n, d ) satisfies v := N 1 ( λ ) n ln n = o (1) . Then for any fixed δ ∈ (0 , 1) , when n is sufficiently large, with probability at least 1 -δ , we have

Proof. These inequalities are direct results of (56), (58), and (59) in Zhang et al. (2024).

■

Proposition E.8 (Restate Proposition 4.13 in Li et al. (2024) with only the constant modified) . When (113) holds, there is an absolute constant that for any z ∈ Γ λ ,

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We first propose an improved minimax lower bound for the kernel regression problem in large dimensional settings in Theorem 3.3 and show that the gradient flow with early stopping strategy will result in an estimator achieving this lower bound (up to a logarithmic factor) in Theorem 3.1. We further determine the exact convergence rates of a large class of (optimal tuned) spectral algorithms with different qualification τ 's, and provide a discussion on new phenomena we find in Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We explain the reason for considering spherical data in Remark 2.1. We point out in the Conclusion section that our work only considers the optimal-tuned spectral algorithms.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We list all assumptions we need in the statement of our main theorems. We provide a complete (and correct) proof in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper concerns the convergence rate of spectral methods, particularly kernel ridge regression (KRR) and kernel gradient flow (KGF), in large-dimensional settings where the sample size $n$ is of the same magnitude as a power $\\gamma$ of the input dimension $ d $, i.e., $ n \\asymp d^\\gamma $. It reveals a new phenomenon of the saturation effect in large dimensions, which is different from its fixed-dimensional counterpart. Specifically, it shows that in large dimensions, KRR still suffers from the saturation effect while KGF does not. The key techniques involve the use of analytic filter functions to characterize the regressors from different spectral algorithms, followed by standard concentration results on the bias-variance decomposition of the excess risk.\n\nSTRENGTHS:\nThis paper offers solid theoretical results which improve over previous literature or reveal novel phenomenon. It offers important insights on spectral algorithms in large input dimensional setting, for example the interpolation (with different qualification $\\tau$) of the learning rate between KRR and KGF, the exact description of the phenomena *periodic plateau behaviour* and *polynomial approximation barrier*.\nThis paper also offers numerical validations on their claim.\n\nWEAKNESSES:\nThere is no major weakness spotted in this paper.\n\nQUESTIONS:\nI appreciate the result of this paper and hence look for any possible extension from the current result.\n\nThis paper focuses on dot-product kernels $K=\\Phi(\\langle \\cdot, \\cdot \\rangle)$ with inputs distributed uniformly on a hypersphere and with polynomial spectral decay. \n\n1. By [Belkin2018] and [Haas2024], it simply seems the function $\\Phi$ cannot be smooth but at least 1-differentiable, say $\\Phi$ is induced by the ReLU-NTK. If $\\Phi$ is smooth, for example $K$ is the Gaussian kernel, then the spectral decay is exponential. Can the result in this paper extend to this case? If yes, where is the adaptation? If no or not obvious, what would be the main technical difficulties?\n\n2. In realistic setting, uniform input distribution on a hypersphere is too restrictive. Could one relax the condition to the distributions which have support on the whole sphere instead? I recall Lemma Lemma F.9 in [Haas2024] stating the spectral decay is still polynomial in this case. Could one extend the analysis in this case?\n\nAlso, I have some technical questions concerning the appendix.\n\n3. In Eq (62) in Lemma D.7, the left hand side (LHS) should be independent to the noise, but why is there a term with $\\sigma^2$ on the right hand side (RHS)?\n\n4. Less like a question but more like a comment: I think that there is a typo in Eq (42): it should be $... \\phi\\_j^2(x)\\leq ...$ instead of $... \\phi\\_i^2(x)\\leq ...$. Also, is Eq (44) actually redundant as a special case of Eq (43) given the notation mentioned in line 602 - 603?\n\nReference:\n- Haas, Moritz, et al. \"Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension.\" Advances in Neural Information Processing Systems 36 (2024).\n- Belkin, Mikhail. \"Approximation beats concentration? An approximation view on inference with smooth radial kernels.\" Conference On Learning Theory. PMLR, 2018.\n\nLIMITATIONS:\nAll assumptions and conditions are stated clearly in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn a large-dimension setting, i.e., the dimension $d$ of the input grows polynomially with respect to the sample size $n$, this manuscript rigorously proves upper and lower bounds for spectral algorithms and shows the dependence on the qualification and the interpolation index. Consequently, the manuscript proves the saturation effect in spectral algorithms for large-dimensional data.\n\nSTRENGTHS:\n1. Identify several phenomena in large-dimensional spectral algorithms based on the derived rates. These phenomena are also illustrated by figures, making the explanations easy to follow.\n\n2. Discovered the thresholding for igniting saturation effect is different for large-dimensional and fixed-dimensional settings. Specifically, in a large-dimensional setting, the saturation effect occurs when the interpolation index exceeds the qualification, whereas in a fixed-dimensional setting, it must be more than twice the qualification.\n\nWEAKNESSES:\n1. By checking previous work [1,2] and the proofs in these works, it looks like Theorem 3.1 has been established in Section 4 of [1], while Theorem 4.1 and Theorem 4.2 are the direct extensions of partial results in Theorem 2 and Theorem 3 from [2]. For instance, the proofs of Theorems 4.1 and 4.2 are obtained by replacing the Tikhonov regularized filter function in the variance and bias decomposition of [2] with a general filter function satisfying specific conditions such as C1 and C2. Such a proof trick has been used in previous extend from KRR (Tikhonov regularization) to general spectral algorithms, i.e., [3] to [4]. \n   \n   However, unlike the extension from [3] to [4], the current manuscript seems to be a partial extension of [2] with a similar proof trick as I mentioned before. Therefore, I have concerns about the technical contribution and novelty of this manuscript as a submission to a conference. This work seems more like an extension to a journal like JMLR, etc.\n   \n   I am just not sure whether such a partial extension of a previous article with almost the same proof technique is suitable for conference publication or whether it would be better evaluated in a journal. I defer this justification to the AC. Please disregard this comment if the AC deems the current context appropriate for conference publication.\n\n2. I noticed there are some simulation experiments to confirm the saturation effect in fixed dimension KRR; see [5]. Is it possible to confirm the results in this manuscript? I understand given the rates are asymptotic, it might be hard to have thorough investigations due to the extremely large $d$. But I'm still curious if any preliminary experiments can be done. \n\n\n[1] Lu, Weihao, et al. \"Optimal rate of kernel regression in large dimensions.\" _arXiv preprint arXiv:2309.04268_ (2023).\n\n[2] Zhang, Haobo, et al. \"Optimal Rates of Kernel Ridge Regression under Source Condition in Large Dimensions.\" _arXiv preprint arXiv:2401.01270_ (2024).\n\n[3] Zhang, Haobo, et al. \"On the optimality of misspecified kernel ridge regression.\" _International Conference on Machine Learning_. PMLR, 2023.\n\n[4] Zhang, Haobo, Yicheng Li, and Qian Lin. \"On the optimality of misspecified spectral algorithms.\" _Journal of Machine Learning Research_ 25.188 (2024): 1-50.\n\n[5] Li, Yicheng, Haobo Zhang, and Qian Lin. \"On the Saturation Effect of Kernel Ridge Regression.\" International Conference on Learning Representations. (2024)\n\nQUESTIONS:\n1. I'm curious to know if it is possible to conduct a similar analysis under ultra-high-dimension settings like the dimension grows exponentially fast as the sample size $d = \\exp\\{n^{\\gamma}\\}$. Do we need additional techniques to conduct these analyses?\n2. Based on the figure, it looks like even when $s> 2\\tau$, as long as $d$ grows with $n$, the saturation effect will not happen, which is different from the fixed dimension setting. While this may be the consequence of the derived rate, can authors provide some intuition behind this?\n3. Is there a particular reason that the authors concern $\\gamma \\in p(s+1),(p+1)(s+1))$ with $p$ as integer to derive the rates? Why is this ratio an integer?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
### Summary:

 

The authors study  the saturation of spectral algorithms (KRR & GF) in high-dimensions where $n,d$ are both large,  meaning that when KRR can't achieve information theoretic lower bounds with over smooth the regression functions while kernel Gradient Flow (GF) can. 
Theorem 3.1 states the optimal convergence rate of kernel GF which matches the provided minimax lower bound in Theorem 3.3. Moreover, they find that KRR is unableto achieve this lower bound (being suboptimal) for interpolation spaces with $s >1$.

STRENGTHS:
### Pros:

- very well-written
- tightening previous results on the minimax rate of kernel GF in high dimensions
- proving the saturation of KRR in high dimensions

WEAKNESSES:
### Cons:
 
 - the main results of this paper are stated on page 7, the presentation of the results is  slow

QUESTIONS:
This is an interesting paper about the saturation of KRR in high dimensions. The authors provide several new results and the paper is well written and organized.

- line 200 -- what does $f_\star$ mean? It is only defined in the next page.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-24