Title: How many samples are needed to leverage smoothness?
Venue: NeurIPS 2023
Accepted: yes
Keywords: Statistical learning

breaking the curse of dimensionality

smoothness priors

kernel methods
Abstract: A core principle in statistical learning is that smoothness of target functions allows to break the curse of dimensionality. However, learning a smooth function seems to require enough samples close to one another to get meaningful estimate of high-order derivatives, which would be hard in machine learning problems where the ratio between number of data and input dimension is relatively small. By deriving new lower bounds on the generalization error, this paper formalizes such an intuition, before investigating the role of constants and transitory regimes which are usually not depicted beyond classical learning theory statements while they play a dominant role in practice.

How many samples are needed to leverage smoothness?

Abstract

A core principle in statistical learning is that smoothness of target functions allows to break the curse of dimensionality. However, learning a smooth function seems to require enough samples close to one another to get meaningful estimate of high-order derivatives, which would be hard in machine learning problems where the ratio between number of data and input dimension is relatively small. By deriving new lower bounds on the generalization error, this paper formalizes such an intuition, before investigating the role of constants and transitory regimes which are usually not depicted beyond classical learning theory statements while they play a dominant role in practice.

1 Introduction

The current practice of machine learning consists in feeding a machine with many samples for it to infer a useful rule. In supervised learning, the samples are input/output (I/O) pairs, and the rule is a relationship to predict outputs from inputs. Once learned, the I/O mapping can be deployed in the wild to infer useful information from new inputs. Because it was not engineered by hand, one can question the existence of unwanted behaviors. Classical guarantees regarding the mapping's correctness are offered by statistics: assuming that the training samples are independent and identically distributed according to the future use cases, it is possible to derive theorems akin to the central limit theorem.

While many statistical learning principles offer practical insights, theoretical results often appear somewhat obscure, and forming intuition about them is often challenging, which limits their impact. In this paper, we focus on one simple principle: 'smoothness allows us to break the curse of dimensionality'. The curse of dimensionality is a generic term referring to a set of high-dimensional phenomena with significant practical consequences. In supervised learning, it manifests as follows: without a good prior on the I/O mapping to be learned, one can only get good estimates of the mapping close to the observed examples; as a consequence, to obtain a good global estimate, one needs to collect enough data points to finely cover the input space, which implies that the number of data points should scale exponentially with the dimension of the input space. Yet, when provided with the information that the mapping has some structure, one might need significantly less examples to learn from. This is notably the case when the mapping is assumed to be smooth. The goal of this paper is to better understand how and when we can expect to get much better convergence guarantees when the target function is known to be smooth.

Related work. Nonparametric local estimators were introduced as soon as the field of learning began to form in the 50's [11], and their consistency was studied extensively in the second half of the twentieth century (see Stone [29], Devroye et al. [8] and references therein). Introduced for scatter plots [7], local polynomials were the first estimators to leverage smoothness to improve regression [12]. They were later replaced by kernel methods, which are a powerful way to adapt to smoothness without much fine-tuning, and were widely regarded as state-of-the-art before the deep learning era [25]. Convergence results for kernel methods can be understood through the size of their associated functional spaces [13, 32], how those sizes relate to generalization guarantees [31], and how those spaces adhere to L 2 [22, 26]. For least-squares regression, relations between the size of those spaces

Bandwidth

σ

0

.

1

0

.

3

1

Regularizer

λ

10

-

1

10

-

3

10

-

5

Samples

n

30

300

Bandwidth

σ

0

.

1

0

.

3

1

Regularizer

λ

10

-

1

10

-

3

10

-

5

Samples

n

30

300

3

30

300

Number of samples

1

0

.

1

Sigmas

Lambda: 3.04e-06

10

-

1

10

0

and generalization guarantees are usually derived through operator concentration [27, 6]. More recently, transitory regime behaviors were described in Mei et al. [16], Mei and Montanari [15], and high-dimensional phenomena that lift the need to have more data than dimensions were studied in [23, 14], with Bach [3] relating the latter analyses with the former ones.

Contributions. This work focuses on the role of smoothness in breaking the curse of dimensionality for typical supervised learning problems. At a high-level, our main contribution is to showcase the importance of transitory regimes where one does not have enough samples to leverage high-order smoothness. In such transitory regimes, the behavior of the excess risk can be quite different than its asymptotic 'stationary' behavior. Usually not well described by theory, they might be the dominating regimes in applied machine learning, where the number of samples is often relatively small compared to the input dimension (see Figure 1 for an illustration). This arguably explains the poor performance of kernel methods without strong kernel engineering in the deep learning era: they try to leverage smoothness, but usually do not access enough samples to meaningfully estimate high-order derivatives. More precisely, our contributions are twofold:

We provide two generic 'minimax' lower bounds that illustrate how the curse of dimensionality can not be fully beaten under smoothness assumptions alone.

We delve more specifically into guarantees offered by algorithms that are built to leverage smoothness assumptions, and get a fine-grained picture of some of the transitory regimes where learning takes place in practice.

All results are illustrated by numerical experiments, some of them to be found in Appendix C.

2 The Significance of Constants

This section reviews classical results in learning theory, before providing generic lower bounds when relying solely on smoothness assumptions.

2.1 Established Upper Bounds

Supervised learning is concerned with learning a function from an input space X to an output space Y from a dataset D n = ( X i , Y i ) i ∈ [ n ] of n ∈ N examples. 1 For simplicity, we will assume Y = R and X = R d , or X = T d being the torus. A learning rule, or learning algorithm, is a mapping A : D n ↦→ f n that builds a function f n : X → Y based on the dataset D n , with the goal of capturing

the underlying I/O relation. To discuss generalization to unseen examples, it is standard to model both the already collected and the future examples as independent realizations of a random pair ( X,Y ) .

Assumption 1. There exists a distribution ρ ∈ ∆ X×Y that has generated n independent training samples D n = ( X i , Y i ) i ∈ [ n ] ∼ ρ ⊗ n , and will generate future test samples independently.

Under Assumption 1, the quality of a mapping f : X → Y is measured through the excess risk

where ρ X is the marginal distribution of ρ on X and, assuming that ( Y | X = x ) has a second order moment for every x ∈ X ,

The risk R ( f ) represents the average error of guessing f ( X ) in place of Y when the error is measured through the least-squares loss. From a statistical viewpoint, it is useful to model f n = A ( D n ) as a random function (inheriting its randomness from the samples), so to study the expectation or the upper tail of the excess risk E ( f n ) . Provided that f ∗ is measurable, it is possible to find methods such that E ( f n ) converges to zero in probability. Without additional assumptions on f ∗ , it is not possible to give any guarantee on the speed of this convergence [12, Theorem 3.1]. However, when f ∗ is assumed to be smooth, the picture improves consequently.

Theorem 1 (Breaking the curse of dimensionality [12, 4]) . Under Assumption 1, when f ∗ is α -smooth for α > 0 in the sense that it admits ⌊ α ⌋ derivatives that are regular, more precisely if f ∗ ∈ C α (i.e. f is α -Hölder regular), or f ∗ ∈ H α = W α, 2 (i.e. f is α -Sobolev regular), there exists a learning rule f n = A ( D n ) that guarantees

where c is a constant independent of n . Moreover, the bound (3) is minimax optimal, in the sense that for any rule f n = A ( D n ) there exists a distribution ρ such that f ∗ ∈ C α , or f ∗ ∈ H α , and the upper bound (3) holds as a lower bound with a different constant c .

Whydoconstants matter? At first glance, when two algorithms A 1 and A 2 guarantee two different upper bounds O ( n -γ 1 ) and O ( n -γ 2 ) on the expected excess risk, A 1 will be deemed superior to A 2 if γ 1 ≥ γ 2 , since after a certain number of samples, we will have that E D n [ A 1 ( D n )] ≤ E D n [ A 2 ( D n )] . However, the constants hidden in the front of the big O s might lead to a different picture when given a small number of samples: A 1 might actually be a so-called 'galactic algorithm', similarly to Strassen's algorithm for matrix multiplication, that might not be worth using without an indecently large number of samples.

2.2 Minimax Lower Bounds

The following lower bounds show how classical algorithms that reach the minimax optimal convergence rates based on smoothness assumptions (3) necessarily present constants that are growing fast with respect to the input dimension. Our analysis holds in noisy settings. 2

Assumption 2 (Homoscedasticity) . The noise in the label ( Y | X = x ) is assumed to be independent of x ∈ X with var ( Y | X = x ) = ε 2 .

Theorem 2. Under Assumptions 2, for any algorithm A , there exists a target function f ∗ such that f ∗ ( α +1) = 0 , and

Theorem 3. Under Assumption 2, for any algorithm A , there exists a target function f ∗ such that its Fourier transform is compactly supported on the ℓ ∞ -disk of radius ω , i.e., ̂ f ∗ ( ω ′ ) = 0 for all ∥ ω ′ ∥ ∞ > ω , and

10

1

10

2

10

3

Number of samples

1

5

9

Number of dimension

10

1

10

2

10

3

Number of samples

1

5

9

Number of dimension

10

-

2

10

-

4

10

-

6

1

2

3

4

Smoothness

α

10

2

10

4

10

6

10

8

Effective dimension

Proof Sketch (detailed in Appendix). The proofs of those two new theorems consist in retaking the standard lower bound in ε 2 D/n when performing linear regression with D orthogonal features. In Theorem 2, D corresponds to the number of polynomials of degree less than α with d variables; in Theorem 3, D is the number of integer vector m ∈ Z d whose norm is smaller than ω .

Theorems 2 and 3 illustrate how the usage of strong smoothness assumptions on f ∗ can not guarantee an excess risk lower than ε 2 without accessing an indecently large number of samples with respect to the input dimension (e.g., n ≥ ( d + α d ) ≥ (1+ α/d ) d or n ≥ 2 d ω d respectively). Empirical validations are offered by Figure 2. In their inner-workings, those theorems capture how the rates derived through Theorem 1 are deceptive when one does not have enough samples compared to the size of the hypothesis space that f ∗ is assumed to belong to; and that the size of smooth functions spaces grows quite fast with respect to the dimension of the input space. Similar lower bound theorems can be proven with rates in n -2 α/ (2 α + d ) under more detailed assumptions, as illustrated with Theorem 7 in Appendix. Efficient learning beyond the limits imposed by those theorems can only take place when leveraging other priors: for example, sparsity priors would reduce the minimax rates from ε 2 D/n to ε 2 s log( D ) /n where s is the sparsity index of f ∗ [9].

3 Crisp Picture in RKHS Settings

To provide a fine-grained analysis of the phenomena at stake, this section presents stylized settings where constants and transitory regimes can be studied precisely, allowing to get a better picture of convergence rates in practical machine learning setups.

3.1 Backbone Analysis

In the following, we shall consider a feature map φ : X → H , with H a Hilbert space and φ ∈ L 2 ( ρ X ) . The map φ linearly parameterizes the space of functions

For example, H could be R k , φ seen as defining k features φ i ( x ) on inputs x ∈ X . This model can be used to estimate f ∗ through the empirical risk minimizer

In order to ensure that F can learn any function f ∗ , the features can be enriched by concatenating an infinite countable number of features together. In this setting, it is more convenient to describe the geometry induced by F through the (reproducing) kernel k : X × X → R defined as k ( x, x ′ ) =

⟨ φ ( x ) , φ ( x ′ ) ⟩ . When F can fit too many functions, the estimate (7) needs to be refined to avoid overfitting. This paper will focus on Tikhonov (also known as ridge) regularization 3

where the norm is defined from (6) as ∥ f ∥ F = inf {∥ θ ∥ | f = f θ } , but can also be expressed with the sole usage of k through the integral operator K : L 2 ( ρ X ) → L 2 ( ρ X ) ,

as ∥ f ∥ F = ∥ ∥ K -1 / 2 f ∥ ∥ L 2 ( ρ X ) , with the convention K -1 (ker K ) = { + ∞} .

The statistical quality of f n (8) depends on two central quantities, defined as

where a = 2 . The first term, known as the effective dimension, quantifies the size of the space F in which f ∗ is searched for. It relates to the variance of the estimator as a function of the dataset D n . It will capture the estimation error, the error due to the finite number of accessed samples, related to the risk of overfitting. The second term quantifies the adherence of f ∗ to F . It can be understood as the proximal distance between f ∗ on F since ( K +1) -1 = I -K ( K +1) -1 is a proximal projector. It will capture the approximation error, the error due to the fact that our model does not exactly fit the target function, related to the risk of underfitting.

Theorem 4 (High-sample regime learning behavior) . Under Assumptions 1 and 2, as well as two mild technical Assumptions 3 and 4, when f ∗ is in the closure of F in L 2 ( ρ X ) , there exists a constant c such that the estimate (8) verifies

where a n = N + ( K ) /n , and N + ( K ) = ess sup x ∼ ρ X Tr( K x ( K + 1) -1 ) with K x the rank-one operator on L 2 ( ρ X ) that maps f to the constant function equal to E [ f ] k ( x, x ) . The different notions of search space size are always related by N 2 ≤ N 1 ≤ N + . 4 Moreover, under the interpolation property K p ( L 2 ( ρ X )) ↪ → L ∞ ( ρ X ) , i.e. ∥ K p f ∥ ∞ ≤ ∥ f ∥ L 2 ( ρ X ) , it holds that N + ( λ -1 K ) = O ( λ -2 p ) ; while under the source condition f ∗ ∈ K r ( L 2 ( ρ X )) , it holds that S ( λ -1 K ) = O ( λ 2 r ) .

While the excess risk upper bound deriving from Theorem 4 is somewhat standard, the lower bound is new . This theorem states that the generalization error E D n [ E ( f n )] behaves as A ( n, K ) := ε 2 N 2 ( K ) /n + S ( K ) up to higher order terms specified in the right-hand side. Theorem 4 also holds for ridge-less regression (7) with N 1 ( K, 0) = N 2 ( K, 0) = dim F , N + ( K, 0) = ∥ K -1 ∥ -1 ∥ φ ∥ ∞ , and S ( K, 0) = ∥ f ∗ -π F f ∗ ∥ 2 , where π F is the L 2 ( ρ X ) -orthogonal projection onto F . In this setting, E D n [ E ( f n, 0 )] = A ( n, K, 0)(1 + O ( n -1 )) . More in general, we conjecture the right-hand side of Theorem 4 to be improvable with the removal of N 1 ( K ) in front of the rates (which is due to our usage of concentration inequalities on operators rather than on scalar values), the change of the second N 1 into N 2 , and the substitution of a 1 / 2 n by a n . This would show that E D n [ E ( f n )] behaves as A ( n, K )(1 + O ( a n )) . In the following, we will call very high-sample regimes situations where a n ≤ 1 , and high-sample regimes situations where N 2 ( K ) ≤ n .

3.2 Approach Generality

Despite their apparent simplicity, reproducing kernels k describe rich spaces of functions F , known as reproducing kernel Hilbert space (RKHS), namely any Hilbert space of functions with continuous pointwise evaluations [2]. Classical examples are provided by subspaces of analytical functions C ω

10

1

10

3

Samples

n

10

-

2

10

-

1

10

0

Excess risk

O

(

n

-

1

/

2

)

O

(

n

-

3

/

2

)

Stationary behavior

Theoretical bounds

Real profile

Phase transition

through the Gaussian kernels k ( x, x ′ ) = exp( -∥ x -x ′ ∥ 2 /σ 2 ) , and by the Sobolev space H ( d +1) / 2 through the exponential kernel k ( x, x ′ ) = exp( -∥ x -x ′ ∥ /σ ) . Reproducing kernels encompass several approaches and algorithms that have been suggested to leverage smoothness of the target function f ∗ .

The first approach consists in estimating local Taylor expansion through local polynomials, defined through

for P a partition on X , β ∈ N a degree, λ > 0 a regularization parameter. Intuitively in highdimension problems, where d = dim( X ) is big, leveraging local properties may not be very reasonable, since the covering of X with local neighborhoods grows exponentially with the dimension d (when X has unit volume, and neighborhoods have a fixed radius), meaning that if one wants to have enough samples per neighborhood, n should scale exponentially with d .

Rather than local properties, the second approach consists in leveraging global smoothness properties, through the estimation of Fourier coefficients. Such estimators can be built implicitly from translationinvariant kernels, defined as

for q : R d → R a basic function, σ a bandwidth parameter, and λ a regularization parameter. However, the number of frequencies smaller than a cut-off frequency, i.e., the number of trigonometric monomials x ↦→ e im ⊤ x with m ∈ Z d such that ∥ m ∥ ≤ ω , grows exponentially with the dimension, and this approach will not escape from the curse of dimensionality.

Other approaches, such as windowed Fourier estimation, or wavelets expansion estimation (aiming to reconstruct both fine local details together with coarse large-scale behaviors), could be thought of and described through the lens of RKHS. Yet, whatsoever the definition of smoothness considered (i.e., Hölder, Sobolev, Besov), all those methods will hit an inherent limit: the number of 'smooth' functions increases really fast as the dimension increases. Indeed, since their proofs simply consist in finding D linearly independent function, lower bound theorems akin to Theorems 2 and 3 could be derived without difficulties for other notions of smoothness.

All the previously described methods are usually endowed with a few hyperparameters that modify the integral operator K and the norm ∥·∥ F defining the estimator (8). Geometrically, a change of the front regularization parameter λ leads to an isotropic rescaling of the ball ∥·∥ -1 F { 1 } inside L 2 ( ρ X ) , while a change of other hyperparameters could favor certain directions, or even remove some functions in F . In practice, fitting hyperparameters through cross-validation can be understood as implicitly searching to balance and minimize N ( K ) and S ( K ) . This fitting allows all those methods to reach the performance of Theorem 1 in O ( n -2 α/ (2 α + d ) ) as we explain in Appendix.

3.3 Exploration of Transitory Regimes

Given n samples, Theorem 4 suggests to tune K so as to minimize ε 2 N ( K ) /n + S ( K ) . Interestingly, while theory tends to focus on deriving convergence rates in O ( n -2 α/ (2 α + d ) ) that maximize the coefficient α , 5 Theorem 4 can also be leveraged to characterize tightly the expected decay of the generalization error when accessing a small number of samples. To ground the discussion, we will focus on a stylized setting where N and S can be studied in detail as a function of hyperparameters.

Proposition 1 (Capacity and bias bounds) . When ρ X is uniform on the torus T d = R d / Z d , and k is a translation-invariant kernel k ( x, y ) = λ -1 q (( x -y ) /σ ) , the capacity of the space defined through k with regularization λ and bandwidth σ verifies, for a ∈ { 1 , 2 } , where # is the counting measure on Z d ⊂ R d , and ̂ q is the (discrete) Fourier transform of q . Similarly, the biases quantifying the adherence of f ∗ in F verify

Moreover on X = R d , if ρ X has a density bounded above by ρ ∞ < + ∞ , then (14) and (15) become upper bounds for µ the Lebesgue measure and ̂ f the continuous Fourier transform, at the cost of extra constants in front of their right-hand sides (respectively ρ ∞ and max( ρ ∞ , 1) for N a and S ).

Proof Sketch. This relatively standard fact follows from the assumption on ρ X which implies that K is diagonal in the Fourier domain.

Proposition 1 unlocks a precise sense of the effective dimension for the Gaussian kernel, defined with q ( x ) = exp( -∥ x ∥ 2 ) , the exponential kernel, with q ( x ) = exp( -∥ x ∥ ) , and the Sobolev kernel, with ̂ q ( ω ) = (1 + ∥ ω ∥ 2 ) -β , as well as the bias term S when approximating a function f ∈ H α with those kernels. This is reported in Table 1 and proved in Appendix B.

High-sample regimes in harmonics settings. Proposition 1 is useful to describe formally different convergence rates profiles that one may expect in practice. In particular, the linearity of the bias characterization (15) is theoretically useful to decorrelate the estimation of different power laws appearing in the Fourier transform of f . More precisely, if

with c γ being the inverse of the constant in front of the characterization of S ( σ, λ ; H γ ) in Table 1, and µ some measure with a profile that ensures the good definition of f ∗ ∈ H α , we have, taking for example k as the Gaussian kernel,

10

1

10

3

10

-

3

10

-

2

10

-

1

10

0

10

1

10

3

10

-

4

10

-

3

10

-

2

fixed (

σ, λ

)

best param

In particular, with σ 2 log( λ -1 σ d ) -1 = n -r , we get the following convergence rate profile, with c N the constant in front of the characterization of N ( σ, λ ) in Table 1,

This characterization enables us to easily create target functions exhibiting different convergence profiles, as long as we stay in the high-sample regimes where our bounds are meaningful (i.e. when the factor in 1 + O ( a n ) is relatively constant).

Fast then slow profile. The first type of profile is built from µ that charges most of its mass on fast decays, but also puts some small mass on slow decays. It corresponds to target functions that are roughly well approximated by highly smooth functions, but whose exact reconstruction needs to incorporate less regular functions. The smooth part of the function will be learned quickly, yet the non-smooth part will be learned slowly. Typical examples of such a profile are provided by non-smooth functions that can be turned into infinitely differentiable ones after introducing infinitesimal perturbations, such as f ∗ ( x ) = exp( -min( | x | 2 , M )) , which is only C 0 , but where one can expect to learn fast before stalling to estimate the C 1 -singularity. Another example is given by a function made of a sum of one low-frequency cosine with large amplitude easy to learn together with one high-frequency cosine with small amplitude much harder to learn. We illustrate these profiles on Figure 4.

Slow then fast profile. The second type of profile that can be created is for functions that are supported on a few eigenfunctions of K associated with small eigenvalues. A typical example of this profile in one dimension would be f ∗ ( x ) = cos( ωx ) for a high frequency ω . For this target function, no meaningful learning can be done when the search space is too small, because small search spaces do not contain high-frequency functions. On the other hand, when the search space is big enough, the bias quickly goes to zero, allowing for fast learning as long as one controls the estimation error. When provided with few samples, one would prefer a small search space to avoid blowing up of the estimation error, and learning will stall until enough samples are collected to explore bigger search spaces, where f ∗ could be learned quickly. We illustrate this profile on Figure 4.

Those examples illustrate how, given a target function and a range on the number of available samples, convergence behaviors might fall in regimes that do not correspond to the steady convergence rate in O ( n -2 α/ 2 α + d ) predicted by Theorem 1. The intuition beyond those examples is not specific to translation-invariant kernels in harmonics settings, but holds more generically for abstract RKHS.

Empirical study of low-sample regimes. In this work, we have focused on 'under-parameterized' situations where the parameters were set to have more samples than the effective dimension of the

resulting functional space F t ( n ) . In a deep learning world, where many phenomena are understood as taking place in the 'over-parameterized' regime, it is of interest to compare our perspective with the double descent phenomenon. Figure 1 shows the excess risk as a function of two of the three parameters ( n, σ, λ ) , as well as the graph defined by { ( n, σ, λ ) | N 1 ( λ, σ ) = n } . It illustrates a double descent phenomena with 'phase' transition governed by the passage from the low-sample to the high-sample regime. To delve into this, we investigate into regression weight learned by kernel ridge regression. When given access to the knowledge of the full distribution ρ , the estimator in (8) can be rewritten as

As such, kernel ridge regression can be seen as learning in an unsupervised fashion the weights α : X → L 2 ( ρ ) , which then indicate how to fold the input space to use information provided by the labels. At a high level, one can think of a scheme, given some input points, to perform finite differences and leverage the result to build an estimate of the target function from Taylor expansions, whatsoever would be the label observations. Figure 5 shows how, when λ is not too big, the reconstruction f ∞ ,λ ( x 0 ) ( x 0 being the same point at the bluest center on the different pictures on this Figure) depends on observations made far away from x 0 according to some periodic pattern, implicitly assuming that the target function should be regular when looked at in the Fourier domain. Similarly, one can look at the weights ̂ α X satisfying E D n [ f n ( x )] = E ( X,Y ) [ ̂ α X ( x ) Y ] , and whose closed form is given in Appendix C.4. Those weights are shown on Figures 6. They present weird behaviors when the number of data n is closed to the search space size N 2 ( K ) . Note that this double descent phenomenon actually takes place in the regularized setup, and not in the interpolation regime, contrarily to prior works on the matter [e.g. 28, 21].

4 Conclusion

In this paper, we have shown how subtle is the saying that smoothness allows to break the curse of dimensionality. In essence, without implicit bias and in presence of noise, one needs to be in the high-sample regime where the size of the search space F is smaller than the number of samples to avoid overfitting. As the input dimension grows, many more smooth functions can be defined. This constrains the diversity of functions within F , which will typically be devoid of fine-grained details (linked with high-order, eventually trigonometric, polynomials), hence unable to harness high-order smoothness without accessing a large number of samples n .

Future work. Since we have shown that smoothness alone is not a strong enough prior to build efficient learning algorithms in high-dimensions, other priors could be investigated. As such, sparsity assumptions, multi-index models, feature learning or multi-scale behaviors might offer more realistic models to break the curse of dimensionality. How deep learning models exploit such priors has been an active line of research, although linking theoretical results with 'interpretable' observations in neural networks remains challenging, and theory has not yet provided that many meaningful insights for practitioners.

-

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 0.60

-

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 1.00

-

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 2.00

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 4.00

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 8.00

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 16.00

-

-

-

Furthermore, going beyond the sole selection of a few hyperparameters, it would be interesting to understand more aggressive model selection. In particular, given some observations ( X i , Y i ) and some hypothesis classes ( F t ) t , it seems natural to trade a term that fits the data as per (7), together with a regularization term min t ∥ f ∥ F t that selects F t so that f n has a small F t norm. We understand this as a lex parsimoniae , where each F t encodes different notions of simplicity (e.g. different priors) while f n only needs to satisfy one of them.

Finally, while this work heavily relies on the least-square loss, practitioners tend to favor other losses such as the cross-entropy. How losses deform and modify the size of the search space F and its adherence properties to some target functions f ∗ is an open-question -not to mention its adherence properties when the final predictor is built as a decoding y ( x ) = arg max y f ( y | x ) in order to learn a discrete y from a score f ( y | x ) that relates to P ( X,Y ) ∼ ρ ( Y = x | X = x ) .

Experiments reproduction. All the code to run figures is available at https://github.com/ facebookresearch/rates .

Acknowledgements. VC would like to thank Alberto Bietti, Jaouad Mourtada and Francis Bach for useful discussions. SV was partially supported by the MUR Excellence Department Project MatMod@TOV awarded to the Department of Mathematics, University of Rome Tor Vergata.

References

Robert Adams and John Fournier. Sobolev Spaces . Academic Press, 1975.

Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American Mathematical Society , 1950.

Francis Bach. High-dimensional analysis of double descent for linear regression with random projections. arXiv preprint arXiv:2303.01372 , 2023.

Francis Bach. Learning Theory from First Principles . MIT press (announced), 2023.

Vivien Cabannes, Alessandro Rudi, and Francis Bach. Fast rates in structured prediction. In Conference on Learning Theory , 2021.

Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 2006.

William Cleveland. Robust locally weighted regression and smoothing scatterplots. Journal of the American Statistical Association , 1979.

Luc Devroye, Làslzó Györfi, and Gábor Lugosi. A probabilistic theory of pattern recognition . Springer, 2013.

David L. Donoho and Iain M. Johnstone. Minimax estimation via wavelet shrinkage. The Annals of Statistics , 26(3):879 - 921, 1998.

Simon Fischer and Ingo Steinwart. Sobolev norm learning rates for regularized least-squares algorithms. Journal of Machine Learning Research , 2020.

Evelyn Fix and Joseph Hodges. Discriminatory analysis. Nonparametric discrimination: Consistency properties. Technical report, School of Aviation Medicine, Randolph Field, Texas, 1951.

László Györfi, Michael Kohler, Adam Krzyzak, and Harro Walk. A Distribution-Free Theory of Nonparametric Regression . Springer, 2002.

Andrey Kolmogorov and Vladimir Tikhomirov. ε -entropy and ε -capacity of sets in functional spaces. Uspekhi Matematicheskikh Nauk , 1979.

Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel 'ridgeless' regression can generalize. The Annals of Statistics , 2020.

Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 2022.

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Generalization error of random feature and kernel methods: Hypercontractivity and kernel matrix concentration. Applied and Computational Harmonic Analysis , 59, 2022.

Jaouad Mourtada. Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices. The Annals of Statistics , 50(4), aug 2022.

Jaouad Mourtada and Lorenzo Rosasco. An elementary analysis of ridge regression with random design. Comptes Rendus. Mathématique , 2022.

Jaouad Mourtada, Tomas Vaškeviˇ cius, and Nikita Zhivotovskiy. Distribution-free robust linear regression. Mathematical Statistics and Learning , 2022.

Jaouad Mourtada, Tomas Vaškeviˇ cius, and Nikita Zhivotovskiy. Local risk bounds in statistical aggregation. Preprint , 2023.

Nicolò Pagliana, Alessandro Rudi, Ernesto De Vito, and Lorenzo Rosasco. Interpolation and learning with scale dependent kernels. In ArXiv , 2020.

Jaak Peetre. New thoughts on Besov spaces. Duke University Mathematics Series , 1976.

Alexander Rakhlin and Xiyu Zhai. Consistency of interpolation with laplace kernels is a high-dimensional phenomenon. In Conference on Learning Theory , 2019.

Carl Rasmussen and Christopher Williams. Gaussian Processes for Machine Learning . The MIT Press, 2005.

Bernhard Scholkopf and Alexander Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond . MIT press, 2001.

Steve Smale and Ding-Xuan Zhou. Estimating the approximation error in learning theory. Analysis and Applications , 2003.

Steve Smale and Ding-Xuan Zhou. Learning theory estimates via integral operators and their approximations. Constructive Approximation , 2007.

Stefano Spigler, Mario Geiger, Stéphane d'Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under- to over-parametrization affects loss landscape and generalization. Journal of Physics A: Mathematical and Theoretical , 2019.

Charles Stone. Consistent nonparametric regression. The Annals of Statistics , 1977.

Hans Triebel. Interpolation Theory, Function Spaces, Differential Operators . North-Holland Publishing Co., 1978.

Vladimir Vapnik. The Nature of Statistical Learning Theory . Springer, 1995.

Ding-Xuan Zhou. The covering number in learning theory. Journal of Complexity , 2002.

3

30

300

Number of samples

10

-

1

10

-

4

10

-

7

Sigma: 0.46

10

-

1

10

0

1

0

.

1

Sigmas

10

-

1

10

-

4

10

-

7

With 30 samples

10

-

1

10

0

A Generic Proofs and Discussions

A.1 What do we mean by Transitory Regimes?

In essence, by transitory regimes we mean any finite-time behavior that does not match an expected long-time horizon 'stationary' behavior. More precisely, let Γ = { ( n, E D n [ E ( f n )]) | n ∈ N } be the graph of the expected excess risk. Theorem 4 provides a lower-upper bound of the form Γ ⊂ { ( n, cn -γ (1 + ah ( n ))) | n ∈ N , a ∈ [ -1 , 1] } with c, γ two constants and h a function that goes to zero when its argument goes to infinity. This shows that, as n grows large, E D n [ E ( f n )] will behave as cn -γ . However, this stationary behavior in cn -γ might take time to kick in, and when only accessing a small number of samples n , our bound does not lead to strong constraints on E D n [ E ( f n )] , which might arguably exhibit a very different profile. We illustrate this idea on Figure 3.

A.2

Generic Lower Bounds

Let us recall a relatively standard lower bound for linear regression.

Theorem 5 (Linear regression minimax rates, [17]) . Let ρ θ ∈ ∆ X×Y with X = R d and Y = R be defined with its marginal over X being ρ X , and its conditionals being ( Y | X ) = θ ⊤ X + ε 2 ∼ N (0 , 1) with N the Gaussian distribution. Let D n ∼ ρ ⊗ n θ be a dataset of n samples ( X i , Y i ) , and A be an algorithm that map D n to a guess for θ . Under the assumption that n > d and that ∑ X i X ⊤ i is almost always invertible, we have

Replacing X by φ ( X ) , we get the following corollary.

Corollary 6. For any learning algorithm A targeting a function f ∗ ( X ) = φ ( X ) ⊤ θ with φ : X → R D and θ ∈ R D , there exists a θ and a distribution ρ X on X such that under Assumption 2,

The proof of Theorem 2 (resp. Theorem 3) follows by considering φ to be the concatenation of all the ( d + α α ) polynomials of degree at most α with d variables (resp. all the trigonometric polynomials x ↦→ exp( -im ⊤ x ) with ∥ m ∥ ∞ ≤ ω ). To be totally rigorous, one should incorporate the assumptions of Theorem 5, yet if we remove the extra assumption, one can actually show that the worse excess risk could be infinite [17, Proposition 1], so this does not cast shadow on our results, but would only make them stronger. We choose to present weaker results that are easier to understand and parse for the reader.

Those theorems could be proven from scratch by considering a well-thought Bayesian prior on θ that has generated the dataset D n , before lower bounding

and computing the last term explicitly. We conjecture that it should be equally possible to compute with this technique a precise 'exponentially bad' lower bound on the optimal minimax constants appearing in front of n -2 α/ (2 α + d ) ) in Theorem 1.

It is equally possible to retake the proof techniques of Theorems 2 and 3 to prove lower bound that are targeted to match the upper bound in O ( n -2 α/ (2 α + d ) ) .

Theorem 7. For any RKHS F and distribution ρ X such that the spectrum of the integral operator verifies spec K = { n -2 α/d ∣ ∣ n ∈ N } , for any algorithm A and value D > 0 , there exists a function satisfying ∥ f ∗ ∥ F ≤ D such that under Assumption 2,

Proof. The proof consists in finding a subproblem that reduces to linear regression. Under the assumptions of Theorem 5 and the additional assumption that ∥ θ ∗ ∥ 2 ≤ D , it is possible to show the minimax lower bound

for some Bayesian prior on θ [see 4, Chapter on 'lower bound', section on Bayesian analysis].

Let us now consider the case where ∥ θ ∥ ≤ D , and θ is supported on the topk eigenvectors of K , we got ∥ f ∗ ∥ = ∥ ∥ Σ -1 / 2 θ ∥ ∥ ≤ Dλ -1 / 2 k where λ k is the k -th eigenvalue of T . Hence, considering an RKHS F , under Assumption 2, for any algorithm A , there exists a function such that ∥ f ∗ ∥ F ≤ D , and

Considering λ = k 2 α/d and k = ( nD 2 /ε 2 ) 2 α/ (2 α + d )

k -leads to the result.

In terms of implications of Theorem 7, in harmonics settings, it is possible to build an integral operator such that the eigenfunctions of K are known to be regular smooth functions, e.g., the trigonometric polynomials f m : x → exp( -im ⊤ x ) for m ∈ Z d ordered according to ∥ m ∥ = ω . As such, for m ∈ Z d with ∥ m ∥ = ω , f m is expected to be ordered around the ω d -th eigenfunctions of K , typically associated with the eigenvalue λ ω d ≈ ( ω d ) -2 α/d = ω -2 α if F corresponds to H α . Using that when f ∗ is the k -th eigenfunctions of K , we have ∥ f ∗ ∥ = λ -1 k , Theorem 3 would become that for any algorithm A , under Assumption 2, there exists a function f ∗ such that ∥ f ∗ ∥ F ≤ ∥ f m ∥ F with ∥ m ∥ = ω and,

When α scales linearly with d (so to beat the curse of dimensionality), this lower bound presents a constant that is exponential with d .

A.3 Precise Excess Risk Bound

This subsection is devoted to the proof of the bound in Theorem 4.

For ease of notation, we will use the finite-dimensional notation u ⊤ w to denote the inner product ⟨ u, v ⟩ in (infinite-dimensional) Hilbert spaces. Moreover, we will simply write ∥·∥ for both ∥·∥ H

and the operator norm on H (depending on context), L 2 for L 2 ( ρ X ) , and ∥·∥ 2 for both ∥·∥ L 2 and the operator norm on L 2 .

For the sake of clarity, we have expressed all our statements in terms of operators on L 2 . For the proofs, it is more convenient to work in H . Let us introduce the embedding

From S , one can take its adjoint S ∗ and check that K = SS ∗ . K is isometric to the (non-centered) covariance operator

Note that

which implies that K is a continuous operator as soon as φ ∈ L 2 . The kernel ridge regression estimator (8) is characterized as where

Endowing R n with the scalar product ⟨ a, b ⟩ = 1 n ∑ i ∈ [ n ] a i b i , we have

It is useful to define ε i as the difference between Y i and f ∗ ( X i ) = E [ Y i | X = X i ] , which can be seen as the labeling noise and average to zero. For simplicity, we will first assume that our model is well-specified, i.e. f ∗ = Sθ ∗ for some θ ∗ . We have, with E = ( ε i ) i ∈ [ n ] ∈ R n ,

As a consequence,

Let X = ( X i ) i ∈ [ n ] . we have

where in the third equality we used I -(Σ n + 1) -1 Σ n = (Σ n + 1) -1 , and in the last one that E D n [ E | X ] = 0 . Assuming for simplicity that the noise is homoscedastic so that E [ EE ⊤ ] = ε 2 I for ε > 0 , we obtain

where the 1 /n factor arises from the fact that E ∗ = E ⊤ /n in the geometry we have considered on R n . Finally we have retrieved the following standard bias-variance decomposition result.

Lemma 2 (Bias-Variance decomposition) . When our model is well-specified so that f ∗ = Sθ ∗ with θ ∗ ∈ H , and when the noise in the label is homoscedastic with variance ε 2 , the estimator (8) verifies

We would like to get the limit when n goes to infinity in equation (18). We expect the first term to concentrate towards ∥ ∥ S (Σ + 1) -1 θ ∗ ∥ ∥ 2 2 , and the second term to Tr(Σ 2 (Σ + 1) -2 ) .

A.3.1 Bounding the bias term

Let us begin by working out the term B n = ∥ ∥ S (Σ n +1) -1 θ ∗ ∥ ∥ 2 2 . We first introduce some notation to make derivations shorter. Let E n = Σ n -Σ , Σ 1 = Σ + 1 and F n = -Σ -1 / 2 1 E n Σ -1 / 2 1 . As long as ∥ F n ∥ < 1 , we have

Let us assume for a moment that

̸

If equation (19) holds, then

̸

̸

̸

This inequality is useful as long as ∥ F n ∥ is small enough, which is not always true. When ∥ F n ∥ is large, we can instead proceed with the simpler bound

̸

̸

Therefore, rewriting the limit as

we split the bias error as

Taking the expectation and using the convexity of the absolute value, we obtain

We now proceed with an exponential concentration inequality on ∥ F n ∥ . We will use the one of Cabannes et al. [5], Eq. (25). As long as 1 ≤ ∥ K ∥ , we have

This inequality shows the restrictive notion of effective dimension, which is useful to ensure the good conditioning of the linear system implicitly encoded in (8), and, in essence, bound all moments of φ ( X ) . As long as N + ( K ) ≤ 3 n/ 2 , we can compute the integral as

We recall that the bounds above were derived under condition (19), which is rather strong. However, the attentive reader would remark that a much laxer assumption is sufficient, which we introduce thereafter.

Assumption 3. There exists a constant c such that, for all i, j ∈ N ,

Assumption 3 notably holds when F is finite dimensional with c 2 = ∥ ∥ K -1 ∥ ∥ -1 2 ( ∥ K ∥ 2 +1) . As such all our lower-bound results can be cast with finite-dimensional approximation of infinite-dimensional RKHS. Under Assumption 3, we get

̸

̸

which allows us to proceed with the precedent derivations without assuming that (19) holds.

While the previous results were achieved for f ∗ ∈ F , they can be extended by density to any f ∗ in the closure of F in L 2 ( ρ X ) , i.e. f ∈ (ker K ) ⊥ , leading to the following result.

Proposition 3. When f ∗ ∈ (ker K ) ⊥ and 1 ≤ ∥ Σ ∥ , under the technical Assumption 3, the bias term can be bounded from above and below by

A.3.2 Discussion on the bias bound

More direct upper bound. The precedent derivations can be made more direct with the following series of implications, with A ⪯ B meaning that x ⊤ Ax ≤ x ⊤ Bx for every x :

Let us assume that

This leads to the simple upper bound

Improvement directions. In essence, we expect the bias upper-lower bound to behave as -2 .

Getting this linear dependency in F n explicitly would allow to improve the bound since

Moreover, going back to the definition of F n ,

which suggests possible improvements of the bound in S ( K ) n -1 .

A.3.3 Bounding the variance term

Let us now work on the term V n = TrΣ(Σ n +1) -2 Σ n . It works similarly to the bias term, concentrating towards N 2 ( K ) . With Σ n, 1 = Σ n +1 , we have

Using that, for any A positive semi-definite and any B , Tr( AB ) ≤ ∥ B ∥ Tr( A ) , it follows that

Let us focus on the first term. Using a -1 -b -1 = a -1 ( b -a ) b -1 , we get

For the second term,

Similarly as for (19), if we assume that

then we have, as long as ∥ F n ∥ ≤ 1 / 2 c ,

For the case when ∥ F n ∥ is large, we can again proceed with a simple bound:

Splitting the full expectation as we did for the bias we thus obtain

We are left with the computation of two integrals. As long as 1 ≤ ∥ Σ ∥ , with a the coefficient appearing in the exponential

Once again, the condition (23) can be relaxed with the following assumption.

Assumption 4. There exists a constant c such that, for all i ∈ N ,

As for Assumption 3, Assumption 4 holds when F is finite-dimensional with c 2 = ∥ ∥ K -1 ∥ ∥ ( ∥ K ∥ +1) . It holds in general with c 2 = ∥ Σ ∥ +1 , although this would deteriorate the bound by a factor of λ when considering Σ to be λ -1 Σ .

We are finally ready to collect the different pieces.

Proposition 4. When f ∗ ∈ (ker K ) ⊥ and 1 ≤ ∥ Σ ∥ , under the technical Assumption 4, the variance term can be bounded from above and below by

A.3.4 Discussion to the variance bound

Once again, the bound presented here is somewhat unsatisfying, as it will not necessarily decrease faster than N ( K ) /n . If one considers target functions that are far away from F and requires a large search space (i.e. S ( λ -1 K ) decreases slowly with λ , and given a fixed number of samples n the

optimal λ n is found for N ( λ n K ) quite large compared to n ), so that N + ( λ -1 K ) N ( λ -1 K ) /n does not go to zero. Several directions can be taken to improve the bound. For example, when Y is bounded by M , the noise ε 2 can be replaced by M 2 , and it is possible to get an upper bound of the form

for a truncated version f (thres . ) n of the estimator (8) as proved by Mourtada et al. [19] for ridge-less regression and extended to ridge regression in Mourtada et al. [20]. Moreover, retaking the analysis of Mourtada and Rosasco [18], one can get a lower bound of the form

which might lead to some lower bound with

We also note that it should not be too hard to replace F n by Σ 1 / 2 Σ -1 1 E n Σ 1 / 2 Σ -1 1 , which would lead to ε 2 N 2 ( K ) /n instead of ε 2 N 1 ( K ) /n in the right-hand side of (11).

A.3.5 Full theorem

Collecting the precedent results leads to the following theorem.

Theorem 8. Under the technical Assumptions 3 and 4, as long as 1 ≤ ∥ Σ ∥ , when f ∗ belongs to the closure of F in L 2 ( ρ X ) , the estimator (8) verifies

where a n = N + ( K ) /n and some constant c .

As long as its right-hand side decreases faster then E D n [ E ( f n )] , Theorem 8 states that E D n [ E ( f n )] behaves like

When optimizing for λ when K is actually λ -1 K , assuming that N 1 ≃ N 2 , the right-hand side of Theorem 8 decreases faster than E D n [ E ( f n )] if and only if N 1 ( K ) a -1 / 2 n goes to zero with n . This implies N ( K ) 3 ≤ n , which is a much stronger condition than the high-sample regime condition N ( K ) ≤ n .

A.4 Interpolation spaces, capacity and source conditions

This section discusses the values of N ( K ) and S ( K ) , and prove the last statements of Theorem 4.

A.4.1 Variances

We begin with simple facts about the variance term.

Proposition 5 (Relation between variances) . For any kernel k , we have the relation

From

sigma

3

1

.

3

.

1

lambda

10

-

8

10

-

6

10

-

4

10

-

2

10

0

effective dimension

1

10

10

2

10

3

sigma

3

1

.

3

.

1

lambda

10

-

8

10

-

6

10

-

4

10

-

2

10

0

effective dimension

1

10

10

2

10

3

sigma

3

1

.

3

.

1

lambda

10

-

8

10

-

6

10

-

4

10

-

2

10

0

effective dimension

1

10

10

2

10

3

sigma

3

1

.

3

.

1

lambda

10

-

8

10

-

6

10

-

4

10

-

2

10

0

effective dimension

1

10

10

2

10

3

Proof. Once again, the precise study of the variance is easier in H with the operator Σ rather than in L 2 . First of all, notice that K = SS ∗ has the same spectrum as Σ = S ∗ S , so that, for a ∈ [1 , 2] ,

This shows the first part of the inequality (27):

For the second part of the inequality, we need to reformulate N + ( K ) , which is the object of the next Lemma. In view of Lemma 6, the last inequality in (27) follows from

The ends the proof of the proposition.

Lemma 6. N + ( K ) can be expressed in H as

Proof. Observe that, for x ∈ X ,

Let us introduce the operator

we get S ∗ x g = E [ g ( X ) φ ( x )] . Similarly, one can check that

K x ( g )( x ′ ) = ( S x S ∗ x g )( x ′ ) = ( S x ( E X [ g ( X )] φ ( x )))( x ′ ) = φ ( x ) ⊤ φ ( x ) E X [ g ( X )] = E [ g ] k ( x, x ) , and that

from which we deduce that there exists ε ∈ {-1 , 1 } such that

Necessarily ε = 1 since the right term is positive. Taking the essential supremum ends the proof.

The following is a reinterpretation of Proposition 29 of Cabannes et al. [5].

Proposition 7 (Capacity condition under interpolation inequalities) . When K p ( L 2 ( ρ X )) is continuously embedded in L ∞ ( ρ X ) with p ≤ 1 / 2 , there exists a constant c such that

When the function x → k ( x, x ) is bounded, the RKHS associated with k verifies

Proof. The continuous embedding means that there exists a constant c such that, for any λ ≥ 0 ,

Stated in H , we get

for every θ ∈ H . In other terms,

Let us denote by ( λ i , θ i ) the eigenvalue decomposition of Σ . Then

When considering θ = θ i , this leads to

Therefore,

where the last equality follows from basic calculus.

The second inequality follows from the fact that K 1 / 2 ( L 2 ) ↪ → L ∞ as soon as φ is bounded since, for any f = φ ( · ) ⊤ θ ∈ F = S H = K 1 / 2 ( L 2 ) ,

The previous characterization of N + leads to the claim.

A.4.2 Bias

We now focus our attention on the bias term.

Proposition 8 (Source condition) . When f ∗ ∈ K q ( L 2 ( ρ X ) with q ≤ 1 , there exists a constant c such that, for any λ ≥ 0 ,

Moreover,

Proof. The proof is straight-forward. If f ∗ = K q g with g ∈ L 2 , then

Squaring this term and multiplying it by λ 2 leads to the result.

The second equation is due to the decomposition

The previous derivations explain the final result.

A.5 Application of Theorem 4 to Taylor expansions

A natural idea to estimate a target function f ∗ ∈ C α , with C α the space of α -Hölder functions, is to estimate its Taylor expansion. This is done with local polynomials which consists in concatenating into φ ( x ) a finite number of features of the form x → 1 x ∈ A x k for k ∈ [ β ] for some β ∈ N and A in a partition of X ; together with the ridgeless estimator of (7). In this setting N 2 ( K, 0) is equal to the number of polynomials of degree at most β with d variables times the size of the partition considered. The number of such polynomials is h β (1 d ) , where h ⌊ β ⌋ is the complete homogeneous symmetric polynomial of degree β in d variables and 1 d is the vector of all ones. Thus,

where the last inequalities is due to the fact that ( n k ) ∈ [ ( n/k ) k , ( ne/k ) k ] .

When X = [0 , 1) with uniform distribution, and f = f ∗ is assumed to be ( α, L α ) -Hölder, i.e.,

by fitting Taylor expansions on intervals [( i -1) /m,i/m ) for i ∈ [ m ] and m ∈ Z + with polynomials

one can ensure that [see 12, Lemma 11.1]

where Π F denotes the orthogonal projection from L 2 onto F . The same type of result also holds for X = [0 , 1] d when using m d multivariate polynomials of degree less then ⌊ α ⌋ . Balancing the bias and the variance term, we get an excess risk that behaves as (up to higher order term)

for some constant c β that depends on β , d and grows with L β , σ 2 , the infimum being found for m ∝ ( L 2 β n ) 1 / ( d +2 β ) . This argument of the minimizer illustrates how the size of the window should depend on how smooth f ∗ is expected to be among the functions in C α . Interestingly, it equally shows that the partition size does not deteriorate with the dimension of the input space. Nor will deteriorate the percentage of the total volume contained in each region of the partition. However, the radius of those regions will scale as r = v 1 /d for v the volume of those regions, meaning that when this volume will shrink to zero, the radius will shrink slower as the dimension grows, which will lead to a slower minimization of the approximation error. Ultimately as n grows, the last upper bound will end up behaving according to the 'stationary behavior' in c α n -2 α/ (2 α + d ) .

A.6 Application of Theorem 4 to Fourier expansions

Rather than local properties, one could leverage global smoothness properties, such as fast decays of Fourier coefficients, e.g. f ∗ ∈ H α with H α the Sobolev space of functions that are α -times differentiable with derivatives in L 2 ( ρ X ) .

Fourier coefficient estimators can be built implicitly built from translation-invariant kernels, i.e. k ( x, x ′ ) = λ -1 q (( x -x ′ ) /σ ) for q : R d → R , σ a bandwidth parameter and λ > 0 a regularization parameter. Examples are provided by the Matérn kernels, defined from ˆ q ( ω ) = (1 + ∥ ω ∥ 2 ) -β , the exponential kernel, which corresponds to a Matérn kernel of low smoothness β = ( d +1) / 2 , and the Gaussian kernel, which can be seen as the limit of a Matérn kernel to infinite smoothness, defined as q ( x ) = exp( -∥ x ∥ 2 ) . The corresponding estimators 8 can be proven to guarantee the convergence rates as Theorem 1, namely

where c β notably relates to the norm of f ∗ in H β .

Proof. For the Matérn kernels, the generalization error reads, with τ = 2 β -d , up to constants and higher order terms, according to Table 1,

leading to minimax convergence rates in O ( n -2 α/ (2 α + d ) ) .

For the Gaussian kernel, we get

which is optimized for

This is optimized for

leading to the same minimax convergence rate. In particular, when σ is fixed, this leads to

Based on Theorem 4, this is true as long as N ( λ ) N 1 / 2 ∞ ( λ ) /n 1 / 2 goes to zero with n , which imposes some constraints on α when assuming f ∗ ∈ H α . However, considering a generalization of Mourtada et al. [20] from linear regression to RKHS, the upper bound is actually true without this constraint, which allows to prove the convergence rates in O ( n 2 α/ (2 α + d ) ) for any α .

A.7 Application of Theorem 4 to Sobolev spaces

We now turn ourselves to an informal more generic reformulation of the previous facts on Fourier expansions estimation based on well-known facts in approximation theory [30, 10].

Proposition 9 (Informal source condition) . When F = H β and f ∗ ∈ H α , it holds

Proof. In essence, as explained in Appendix B, K takes a function in L 2 ( ρ X ) and multiply its Fourier transform by ˆ q ( ω ) -1 = (1 + ∥ ω ∥ 2 ) β , with q defining the Matérn kernel, making it 2 β -smooth in the Sobolev sense. In harmonic settings where the Fourier functions diagonalize K and ˆ q ( ω ) parameterizes the spectrum of K , the fractional operator K p can be seen as multiplying the Fourier transform of f by ˆ q ( ω ) -p , making it 2 pβ -smooth. This fact can be extended beyond those harmonic settings, notably with interpolation inequalities. On the opposite direction, any α -smooth function can be multiplied by q ( ω ) α/ 2 β in Fourier while staying in L 2 ( ρ X ) , so that, if f ∗ is α -smooth, it belongs to K α/ 2 β .

Proposition 10 (Informal interpolation inequality) . When F = H β is the space of α -Sobolev functions,

Proof. Note that F = S H = K 1 / 2 ( L 2 ) . We have seen informally in the proof of the previous lemma how K p ( L 2 ) ⊂ H 2 pβ . Now, let us recall the Sobolev embedding theorems [1]. Under mild assumptions on ρ X , for k, r, l, s > 0

We want to use it with k = 2 pβ , r = 2 , l = 0 and s = + ∞ , which leads to p = 4 β/d .

These results could be used to explain the scaling with respect to λ in Table 1, which we will derive formally in Appendix B. They can also be used to show convergence rates in O ( n -2 α/ (2 α + d ) ) for any target functions f ∗ ∈ H α when utilizing a kernel such that F = H β (in terms of sets equally, F being eventually endowed with an other norm than Sobolev norms).

B Translation-invariant kernels and Fourier analysis

This section recalls basic facts about kernel methods and Fourier analysis, before proving Table 1.

B.1 Stylized analysis on the torus

When k is a translation-invariant kernel, i.e. k ( x, x ′ ) = q ( x -x ′ ) , the integral operator K is a convolution against q . Let us expand on the friendly case provided by the torus X = T d := R d / Z d = [0 , 1] d / ∼ , where ∼ is the relation identifying opposite faces of the hypercube, and ρ X the uniform distribution. On the torus, a translation invariant kernel is defined through q being a one-periodic function on R d . Let d x denote the Lebesgue measure on R d . The integral operator K : L 2 ( X , d x ) → L 2 ( X , d x ) is the convolution

For m ∈ Z d , define the Fourier function f m : x ↦→ exp(2 iπ ⟨ m,x ⟩ ) . One can check that the f m 's form an orthonormal family that diagonalizes K with 6

Hence, using Pythagoras theorem, we can define the norm on F through its action on Fourier coefficients as

where ̂ f m = ⟨ f, f m ⟩ L 2 ( ρ X ) , and # is the counting measure on Z d ⊂ R d .

B.1.1 First part of the proof of Proposition 1

Since K is diagonalized in Fourier, we compute the size of F for a ∈ { 1 , 2 } with

For kernels whose scales are explicitly defined through q σ = q ( x/σ ) , we have ̂ q σ ( ω ) = σ d ̂ q ( σω ) , which leads to (14).

Similarly, the bound on the bias term follows from Fourier analysis by

which provides (15).

B.1.2 Second part of the proof of Proposition 1

When ρ is a distribution that is absolutely continuous with respect to the Lebesgue measure and whose density is bounded from above, we get

where K d x is the integral operator associated to the kernel k on L 2 (d x ) . Using the fact the effective dimension is an increasing function of the eigenvalues (since x ↦→ x/ ( x +1) is increasing), and that eigenvalues are increasing with the Loewner order, this leads to

Note that those derivations are written informally (since K and K d x do not act on the same space), but could be made formal with the isomorphic covariance operators on H , plus some technicalities to make sure Σ d x is well defined (assuming φ ( X ) has a fourth-order moment against Lebesgue, or approaching K d x within its action on compact subspaces of X where it is bounded, before taking the limit of N d x ( K ) ).

For the bias term, using the fact that L 2 ( ρ X ) is continuously embedded in L 2 (d x ) and the isometry between the spatial and the Fourier domain, we get

Finally, it should be noted that the norm associated with F does not depend on the density of X , hence the formula can be written independently of ρ X . Indeed, under definition assumption, i.e. if q ∈ L 1 (d x ) , this formula can even be written with a measure of infinite mass. For example, when X = R d , one can consider the Fourier transform associated with L 2 (d x ) , and get

although some care is needed to deal with the continuous version of the spectral theorem (the set of eigenvalues being non-countable). From there the same derivations as for Proposition 1 lead to the desired result.

B.2 Sobolev spaces

Recall the action of differentiation on the Fourier transform: for m ∈ N d , | m | := ∥ m ∥ 1 , and f ∈ L 2 (d x ) ,

This characterizes the pseudo-norm

This pseudo-norm is associated with the translation-invariant kernel such that ̂ q ( ω ) = ∏ i ∈ [ d ] ω -2 m i i as per (31). Note that q is well defined when ̂ q belongs to L 1 (d x ) (by Bochner's theorem), that is | m | > d . Those observations are usual to deduce that the Matérn kernels, which are defined from ̂ q ( ω ) ∝ (1 + ∥ ω ∥ 2 2 ) -β , correspond to the Sobolev spaces H β (d x ) endowed with the norm

It follows from Bochner's theorem that H β is a reproducing kernel Hilbert space if and only if 2 β > d . Remarkably, the exponential kernel corresponds to the Matérn kernel with β = ( d +1) / 2 [24]. For the Gaussian kernel, ̂ q ( ω ) = π -d/ 2 exp( -π 2 ∥ ω ∥ 2 ) , and the associated function class F is analytic (by the Paley-Wiener theorem).

B.2.1 Functional sizes

Let us now express the capacity and bias bound within Sobolev spaces.

Proposition 11 (Sobolev capacity) . When ̂ q ( ω ) = λ -1 (1 + ∥ ω ∥ 2 ) -β for β > d , λσ -d is bounded and ρ has a bounded density, we have

Moreover, when X = T d and ρ X is uniform, we get

Proof. In this setting, Proposition 1 leads to

where we used the fact that

which is true as long as β > d/ 2 to ensure proper convergence of the last integral.

For the part on the torus, in order to get a sharp learning limit, we need to be slightly more precise. In particular, we want to relate the discrete Fourier transform integral of Proposition 1 with the continuous one through series-integral comparison, and get a lower bound on the last integral. We

will fix σ = 1 for simplicity. A simple cut of R d into unit cubes, together with the fact that our integrand is decreasing, leads to

We simplify it as

We now compute the integral with the same techniques as before:

It should be noted that this last bound is somewhat too lax, as it tends to zero when the dimension increases. Since ∑ m ∈ Z d a ( ∥ m ∥ ) for a > 0 is strictly increasing with d , we deduce that this lower bound holds for any k ≤ d .

Proposition 12 (Gaussian capacity) . When ̂ q ( ω ) = λ -1 exp( -∥ ω ∥ 2 ) and ρ has a bounded density, we have where L is defined by Eq. (32) . In particular, L ( x ) ≤ x when x < 1 , and L ( x ) ≲ log( x ) d/ 2 when x gets large. Moreover when X = T d and ρ X is uniform, we get

Proof. With the Gaussian kernel, Proposition 1 leads to

where Li d/ 2 is the polylogarithm function, hence the definition of L as

We recognize an alternating sequence, whose term amplitudes are decreasing as a function of k ∈ N when x ≤ 1 , which explains that L ( x ) is smaller than the first term in this case. The expansion of the polylogarithm function at infinity leads to the upper bound when x goes to infinity.

When it comes to a lower bound, we can proceed as the precedent lemma with

which corresponds to the integral computed for the upper bound. Once again, this also holds when d is replaced by any l ∈ [ d ] .

B.2.2 Adherence

Let us now turn our attention to the bias, i.e. the adherence of functions in those spaces. For proof readability, we will assume that ρ X has compact support. In this setting, W α, 2 is continuously embedded in C α + d/ 2 = W α + d/ 2 , ∞ , which can be defined as

Proposition 13 (Adherence of H α in H β ) . When ̂ q ( ω ) = λ -1 (1 + ∥ ω ∥ 2 ) -β , hence F = H β , if α > 2 β , for any f ∗ ∈ H α ( ρ X ) , and λ small enough, we have

If α < β and ρ X has a bounded density, then for any function f ∗ ∈ C α + d/ 2

Proof. The first part results from previous considerations on the source condition since we have the inclusion f ∗ ∈ H α ⊂ H 2 β = K ( L 2 ( ρ X )) . The second part follows from an L 1 -L ∞ Hölder inequality:

where a was set to λ 1 /β σ -d/β , and the last integral can be bounded by

Recalling the volume of the sphere completes the proof.

Proposition 14 (Adherence of H α in the Gaussian RKHS) . When F is associated with the Gaussian kernel and ρ X has a bounded density, for any f ∗ ∈ C α + d/ 2 we have

Proof. We follow the same path as for the adherence of H α in H β :

where the integral can be bounded by

The same type of derivations can be made for the bias in the lower bound.

Note that the proofs also work when the L 1 -L ∞ Hölder inequality is replaced with L ∞ -L 1 , showcasing the norm of f in H α instead of in C α + d/ 2 . We refer to Bach [4] for details.

Remark 15 (Blessing of dimensionality) . It should be noted that all our integral calculations show a constant 2 πρ ∞ vol( S d ) which will be present in front of the excess risk. As d increases, this constant goes to zero faster than any exponential profile. To see that, note how the volume of the d -sphere is always smaller than twice the one of its inscribed hypercube, whose volume is d -d/ 2 . We do not have clear intuition to understand this behavior at the time of writing.

The previous derivations could be used to derive lower bound under highly specific priors on the structure of f ∗ . However, this proposition is somewhat deceptive compared to Theorems 2 and 3 that are more generic, easier to parse, and show more clearly the curse of dimensionality.

Proposition 16 (Lower-bound application example) . For the Matérn kernel corresponding to F = H β , on the torus T d = R d / Z d with uniform measure ρ X , when f ∗ : x ↦→ cos(2 πm ⊤ x ) is a function with a single frequency m ∈ Z d ,

where γ = 4 β/ (4 β + l ) goes to one as β goes to infinity.

Proof. Using the bias and variance lower bounds decomposition and Proposition 11, we get, when f ∗ = f m is a single frequency function ˆ f ( ω ) = δ m ( ω ) with m ∈ Z d ,

From the fact that

10

1

10

2

10

3

Number of samples

1

5

9

Number of dimension

10

-

2

10

-

6

10

-

10

10

1

10

2

10

3

Number of samples

1

5

9

Number of dimension

10

-

4

10

-

2

10

0

we have that, whatsoever λ is (if it is fixed for all n independently of the realization D n ).

Once again, this lower bound is also true when d is replaced by any l ∈ [ d ] .

C Experimental details

C.1 Online solving of problems of increasing size

In order to solve a big number of least-squares problems with increasing numbers of samples, one can use recursive matrix inversion. When A ∈ R n × n , x ∈ R n and b ∈ R , one can check that where

This allows efficient computation of matrix inversion online as the number of samples increases.

C.2 Example of convergence rates for 'atomic' functions

Polynomial estimation. Consider the target function f ∗ ( x ) = 63 x 5 1 -70 x 3 1 +15 x 1 , learned with the polynomial kernel k ( x, y ) = (1 + x ⊤ y ) p with p = 5 on X = [0 , 1] d and ρ X uniform, in the interpolation regime λ = 0 . In this setting, F is exactly the space of polynomials of degree no larger than 5 , which follows from the fact that k can be rewritten through a vector φ that enumerates monomials:

As a consequence, f ∗ belongs to F , and we expect the bias term to be zero. Yet, we expect the variance, hence the generalization error, to behave as ε 2 dim F /n , where ε corresponds to some notion of variability between labels. The dimension of the class of polynomials in dimension d of degree no larger than p is ( p + d d ) . In particular, in dimension d = 100 , a polynomial of degree at

most p = 5 can have up to one hundred million coefficients, so that one needs about one hundred million observations to enter the high-sample regime and expect an excess risk E ( f n ) of order ε 2 as per Theorem 4. While one could fit a polynomial of lower maximum degree, e.g. 4 instead of 5 , since the f ∗ considered here is actually orthogonal to all polynomials of lower degree, there is no hope to obtain better rates. On Figure 2, the target function is f ∗ ( x ) = x 5 1 , and the polynomial kernel is normalized as

to avoid computational issues. The noise level ε is set to 10 -2 , and the lower bound in ε 2 dim F /n is plotted on Figure 2. In practice, this lower bound describes well the learning dynamic when the number of samples is high compared to the effective dimension of the space of functions considered.

Same examples in Fourier. To transpose the previous example in the Fourier domain, one can consider f ∗ ( x ) = cos( ω 0 x 1 ) together with some translation-invariant kernel. We illustrate this case on Figure 9. The deterioration of the rates with respect to dimension can be understood precisely. In harmonic settings, such as on the torus with uniform measure, one can consider f ∗ as an eigenfunction of the integral operator K associated with the eigenvalue λ ω 0 . The lower bound on the bias is given by

When k is translation-invariant, k ( x, y ) = q ( x -y ) , we get that

When q (0) does not depend on d , this quantity is constant. On the other hand, we expect λ ω to decrease with ∥ ω ∥ . But since the number of frequencies below ∥ ω 0 ∥ grows exponentially with the dimension, in order to keep this sum constant λ ω 0 has to decrease exponentially fast with the dimension, hence the bias will increase exponentially fast with the dimension.

Example of 'wrongfully' arbitrarily fast convergence rates. To further emphasize the importance of constants and transitory regimes, let us discuss an even simpler example. Assume that one wants to learn a polynomial of a unknown degree s ∈ N in a noiseless setting; or equivalently, learn an analytical function such that f ( s +1) = 0 for an unknown s ∈ N . This polynomial can be learned exactly when provided with as many points as the unknown coefficients in the polynomial, meaning that the generalization error will almost surely goes to zero when provided enough points. As a consequence,

where f n is defined in (7) with F the space of polynomials of any degree. In other terms, we are able to prove arbitrarily fast convergence rates . Yet, such convergence rates hide constants that are the real quantities governing convergence behaviors of any learning procedure. Figure 9 shows how the number of coefficients in a Taylor expansion of order s is once again the right quantity to look at.

C.3 Different convergence rates profiles

Figure 4 was computed with the Gaussian kernel on either the one-dimensional torus or R . One hundred runs were launched and averaged to get a meaningful estimate of the expected excess risk. Convergence rates were computed for different hyperparameters, and the best set of hyperparameters (changing with respect to the number of samples but constant over run) was taken to show the best achievable convergence rate.

Fast then slow profile. Let us focus on the example provided by f ∗ : x ↦→ exp( -max( x 2 , M )) -exp( -M ) (note that we substract exp( -M ) so that the function goes to zero at infinity, which remove the burden of learning a constant offset with the Gaussian kernel). Note that, for q σ = exp( -x 2 /σ ) , the convolution q σ ∗ f for a large σ will not modify f much, while making it analytical. This follows from Fourier analysis: if f is integrable, its Fourier transform is bounded; since a convolution corresponds to a product in Fourier, and since the Fourier transform of q σ decays exponentially fast, so does f ∗ q σ , which implies its analytical property. As a consequence, all functions are close to

Bandwidth

σ

0

.

8

0

.

5

0

.

2

Regularizer

10

-

1

10

-

2

10

-

3

Samples

n

30

300

10

-

4

10

-

3

3

30

300

Number of samples

10

-

1

10

-

4

10

-

7

Sigma: 0.34

10

-

4

10

-

2

λ

2

0

2

0

.

00

0

.

25

0

.

50

1

.

50

1

.

25

1

.

00

0

.

75

0

.

0

0

.

2

Ground truth

10

4

samples

10

2

samples

-

-

-

-

-

analytical functions, whose approximation should exhibit convergence rates in O ( n -1 ) . In particular, for f ∗ defined as before for a some M large enough, without enough observations one will not be able to distinguish between f ∗ and q σ ∗ f ∗ , and as the number of sample first increases, one will learn quite fast a smooth version of f ∗ . After a certain number of samples, the learning will stall until enough points are provided to distinguish between f ∗ and its smoothing, and learn the C 1 -singularity of the former. Figures 10 and 11 illustrate this observation. Note that similar reasoning could be made for any RKHS that is dense in L 2 ( ρ X ) .

Slow then fast profile. The slow then fast profile was computed with X = S 1 , ρ X being uniform and f ∗ : x ↦→ cos(2 πωx ) with ω = 20 . One hundred runs were launched and averaged to get an estimate of the excess risk of the estimator in (8) for different values of σ and λ . Again, the best results for different sample sizes were reported to get an estimate of convergence rates on Figure 4. A log-log-log-log plot of the results is provided on Figure 12.

C.4 Exploring the low-sample regime

Looking at the empirical weights. While the previous paragraph discusses the weights α X ( x ) when given access to the full distribution, similar derivations can be made when accessing a finite number of samples. Indeed, kernel ridge regression reads as

where

Bandwidth

σ

0

.

01

0

.

003

0

.

001

Regularizer

10

-

1

10

-

3

10

-

5

Samples

n

30

300

3

30

300

Number of samples

.

003

0

.

03

Sigmas

Lambda: 4.28e-04

10

-

4

10

-

2

10

0

.

003

0

.

03

Sigmas

10

-

1

10

-

4

10

-

7

With 300 samples

10

-

4

10

-

2

10

0

λ

-

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 0.32

-

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 0.53

-

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 1.06

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 2.11

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 4.23

0

.

5

0

.

0

0

.

5

0

High-sample ratio: 8.45

-

-

-

Note that ̂ α i ( x ) = ̂ α X i | X ( x ) where X = ( X 1 , · · · , X n ) is the input dataset. As a consequence,

In other terms, f n is a bias estimator whose average is defined as

These are the weights plotted on Figures 6 and 13. In order to compute those weights efficiently, one can use the block matrix inversion. Using the sliced indices matrix notations, we have where

Denoting we get

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first states and proves Theorem 2, which bounds the generalisation error from above and below. The authors then use this result in Section 3 to find lower bounds on this generalisation error, quantifying the curse of dimensionality with Taylor expansion in the case of uniform measure on the unit cube and with Fourier expansion in the case of uniform measure on the torus. The final section of the paper investigates the transitory regime, theoretically in the high-sample regime and empirically in the low-sample regime.\n\nSTRENGTHS:\n1. This paper considers a problem which I find very important and interesting, i.e. the sample complexity required for smoothness of target functions to start overcoming the curse of dimensionality. \n\n2. The mathematics seem mostly solid, albeit some notational inconsistencies and presentational issues (see Weaknesses and Limitations), and the authors clearly put in a lot of effort to make the proofs solid (although I didn't have time to go through all of the proofs). \n\n3. For a theory paper, the authors valued and put in a lot of effort into experimental findings where theory could not extend to, e.g. in the low-sample transitory regime. This I find very valuable.\n\nWEAKNESSES:\n1. There are a few places in the paper that leave seemingly important results as conjectures. For example, on L147, the authors state \"We expect the right-hand side to be improvable with the removal of N_1(lambda) in front of the rates (which is due to our usage of concentration inequalities on operators rather than on scalar values)\". If the improvement is needed for the subsequent statement within the paragraph and the authors have an idea of how to go about it, I wonder why they didn't do it? This comes up again on L418. \n\n2. The mathematical presentation could be more solid. For example, there are a few instances of notations being used without being introduced, and the same notation being used for different things - see Minor Comments below. Moreover, Theorem 2, which is the main result of the paper, is presented with some issues with regards to clarity - the assumptions under which the result holds can only be found scattered among the proofs in the Appendix. These assumptions do not seem as mild as the authors claim. For example, on L436, it is stated that for Assumption 2 to hold, F has to be finite-dimensional, which is very restricting, or the bound stated in the theorem has to be loosened. \n\n3. Despite the fact that the paper is in general written in a way to suggest very wide generality, the set-up is restricted in Section 2.1 to RKHSs, and the results do not seem to extend to other hypothesis spaces. Of course, RKHSs can be and indeed are large classes of functions, but do exclude e.g. neural networks and other interesting learning situations, and perhaps this restriction should be stated more clearly throughout the paper. \n\nMinor comments\n\nL16: by hands -> by hand\n\nL53: four-folds -> fourfold\n\nL70: Possibly worth defining the notation [n]={1,2,...,n}. \n\nL86: Possibly worth defining the notation for the floor function. \n\nL125-126, L159-160, L201: The notation q is used for different purposes. Perhaps worth changing one of them for consistency?\n\nL223: The same goes for the notation x used here, as opposed to the use of x in other parts of the paper. \n\nL238: In the denominator, 2alpha+d is missing the +. \n\nL368, displayed equation right below: to be consistent with (6), f_n should be f_{n,lambda}. Same is true for every f_n that comes afterwards.\n\nQUESTIONS:\nL372, displayed equation: I don't recall theta_* being introduced? I assume it's the theta in H such that <phi(x),theta_*>=f*(x)? But then this assumes that f* is in F? In fact this assumption only gets introduced on L374, so is it correct to write the displayed equation after L372?\n\nLIMITATIONS:\nThe authors adequately discussed the limitations in the conclusion section, as future work. These include the need to investigate other priors than smoothness, how neural networks exploit such priors, and other loss functions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies transitory, non-asymptotic regimes in the generalization error of kernel methods.\nIt focuses on the use of smoothness to break the curse of dimensionality, and the sample complexity needed to achieve the fast asymptotic rates.\nIn several settings, the authors show that the classical analysis overlooks constants that can be exponential in the dimension.\n\nEDIT: I have read the author's rebuttal, which partially addressed my concerns (this depends on the changes made by the authors to the writing).\n\nSTRENGTHS:\nThe paper makes a very important point that is often overlooked.\nMy lack of expertise in the area, as well as the technicality of the writing, prevent me from judging of the novelty or significance of the analysis.\n\nWEAKNESSES:\nThe paper is difficult to follow. Part of it is due to the fact that I am not an expert in this area at all, and this paper seems to be written for people with a significant deal of intuition about statistical learning analyses. I think it would be very useful for the authors to explain this intuition to the reader, rather than assuming they have it. Part of it is also due to the setting of the paper, which seems to constantly switch: in some cases the authors consider kernel ridge regression with a fixed kernel, in other cases an abstract family $\\mathcal F_t$, in yet other cases polynomials of fixed degree. In the end, I have no overview of the precise results of the paper, and it reads to me like a series of disconnected illustrations.\n\nMinor remark: I would suggest to change the notation $\\mathcal N_\\infty(\\lambda)$, as one would expect it to be equal to $\\Vert K (K + \\lambda)^{-1} \\Vert_\\infty$.\n\nTypos:\n- line 16: hands -> hand\n- line 53: four-folds -> four-fold\n- line 97: a -> an\n- line 113: space -> spaces\n- line : $2\\alpha d$ -> $2\\alpha + d$\n\nQUESTIONS:\nWhat do the authors mean by \"so to be able to consider function spaces $\\mathcal F_t$ containing interesting-enough functions with still having more samples than the effective dimension of $\\mathcal F_t$\"? (lines 173-174)?\n\nWhat exactly is the relationship between the $\\mathcal N_a$ quantities? In the same paragraph (lines 150-160), the authors state that they all behave similarly, while making the distinction between the regimes when $\\mathcal N_\\infty(\\lambda) \\leq n$ and $\\mathcal N_2(\\lambda) \\leq n$.\n\nWhich kernel is being used in Proposition 1 to define the quantities $\\mathcal N_2$ and $\\mathcal B$?\n\nCould the authors summarize their results? I do not see the link from the propositions to the claims of the paper.\n\nLIMITATIONS:\nThe authors acknowledge several limitations, among which being limited to the square loss.\n\nFrom my own (limited) understanding,the main limitation of the paper is that does not really answer the question in the title. The authors make various disconnected cases that quantities that are exponential in the dimension might appear for some learning algorithms in some settings.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to understand the transitory regimes where the data is not enough to enter the asymptotic regime. When the smoothness is leveraged through Taylor expansions, the number of samples needed to enter the aysmptotic regime is the number of polynomial in d-dimension.  When smoothness is leveraged through Fourier expansions, the picture will depend on how the Fourier coefficinet decays.\n\nSTRENGTHS:\nThe paper investigated an important problem of how the constant looked like before the polynomial rate. This makes a contribution to understand what's the prior needed to break the curse of dimensionality.\n\nWEAKNESSES:\nThis paper is conceptually very interesting. However the reviewer is very suspicious about the exact theorem proved.  The reviewer is open to raise the score if the author can hlep me to understand the theorems and their technical contribution better.\n\nFrom my understanding theorem 2 is standard.\n\nIf we characterize the smoothness using taylor expansion. The main conclusion is the sample complexity depends on the number of polynomials. I'm wonder is this conclusion not surprising? and my second question is the sample complexity not depend on the volume in high dimension?\n\nFor the kernel characterization,  from prop3 till (14) looks standard to me. Am I missing anything? From my viewpoint it may have a better characterization of the fast than slow/slow that fast regime using the representing kernel Sobolev norms. Different spectrums have the same kernel norm will have different Sobolev norm. Using this it's easy to have this characterization. (takeoff between norm and convergence rate respect to the number of samples)\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nI'm suspicious about the technical contribution made by the paper and how the theorem reveals the main information of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of learning a smooth function from iid data under different formal definitions of smoothness (e.g. learning an $\\alpha$-Holder function, learning a function in Sobolev space). The paper makes a distinction between two different regimes of sample complexities: the \"very high sample complexity\" regime and the \"high sample complexity\" regime, noting that prior work on learning functions generally proves that the excess risk decays at some rate $O(n^{-\\gamma})$ where $\\gamma$ depends on your particular smoothness assumption.\n\nSuch results have to hold for arbitrarily large $n$, meaning that these rates really describe the decay of excess risk in the \"very high sample complexity\" regime, which they argue is often only descriptive for an unreasonably large sample complexity which isn't possible in practice (i.e. the constant in the big-Oh is huge). They further argue that realistic sample complexities lie in the merely \"high sample complexity\" regime, which exhibits decay at some other rate, say $O(n^{-\\gamma'})$ for some $\\gamma' \\neq \\gamma$.\n\n---\n\nIt's worth noting the proof technique at a high level. The theory of the paper is mainly split into two key flavors of results:\n1. A tight characterization of the excess risk $\\mathcal{E}$ of a function-learning algorithm, showing that the excess risk $\\mathcal{E} \\approx A(n,\\lambda)$, where $A(n,\\lambda)$ is the sum of a variance term depending on a technical notion of \"$\\lambda$-statistical dimension\" and bias term induced by regularization parameter $\\lambda$.\n1. A series of lower bounds roughly showing that for learning $\\alpha$-Holder functions, Sobolev functions, and wavelets, the function $A(n,\\lambda)$ grows exponentially in the dimension of the data.\n\nThese problems are already known to admit excess risk decay rates of $\\mathcal{E} \\leq O(n^{-\\gamma})$ in theory, where the big-Oh hides the dependence on dimension. The above summarized two results together imply that these big-Oh rates in the prior work hide constants that are exponentially large in the data dimension.\nMore broadly, this implies that the intuition that \"assuming a function is smooth lets us avoid a sample complexity that's exponential in the data dimension\" is an insufficient intuition to carry around in data science.\nIt's more accurate to note that the sample complexity is not $O(n^d)$, but nevertheless still likely exponential in the dimension $d$.\n\nSome experiments compliment this theory.\n\nSTRENGTHS:\nThe paper has an interesting narrative, trying to add nuance to the discussion of how assuming that a function is smooth should significantly reduce the sample complexity of learning that function in $\\mathbb{R}^d$ from a $O(n^{O(d)})$ sample complexity down to a $O(n^{\\gamma})$ sample complexity, where $\\gamma$ depends on the smoothness assumption.\nIt's a nice story, showing that the big-Oh really hides a different exponential dependence on data dimension.\n\nThe presentation is usually pretty good. There were points where I did have to read sentences or paragraphs a couple times to fully understand the idea, but the relevant information is all there, and nice intuitions for the expressions are given. This needing to re-read lines could _in some but not all parts_ also be chalked up to me being not super used to reading papers in this area.\n\nThe quality of the results is also pretty strong.\nThe first core result [Theorem 2, Line 139] is a very tight characterization of how the excess risk grows, being well approximated by this $A(n,\\lambda)$ function.\nIt's nicely interpreted and its lower bounds for specific problems are compelling shows of how the excess risk rate of $O(n^{-\\gamma})$ can really hide huge exponential terms.\nI really like this proof setup and style -- though it's not perfect and lacks some connective, which I'll elaborate on in the _weaknesses_ section.\nThis proof approach also strikes me a pretty novel, and I could imagine similar analysis for other statistical problems following a similar proof strategy as this paper, with the two pronged analysis I mentioned in my summary.\n\nOverall, I like this paper.\nUnfortunately, it has some clarity of writing errors and a lack of connective tissue in some parts, which makes me hesitant to recommend publishing the paper. But I do like it quite a bit!\n\nWEAKNESSES:\nThe paper has some clarity issues and some obvious error that really harm legibility.\n\nThe most odd weakness is that many of the references to figures are incorrect. Like [Line 50] mentions Figure 4 when it should mention Figure 1. [Line 217] mentions Figure 5, when Figure 5 is clearly irrelevant to that discussion, but I'm not clear which figure is supposed to be mentioned here.\nI think Figure 8 on [Line 300] has the same issue, and I completely do not understand the message of that paragraph.\nIt's an odd (and presumably very easy to fix) error, but it really did hurt my comprehension at some vital points of the paper.\nBecause it hurt my comprehension, it acts as a very real mark against the paper.\n\n---\n\nThe paper also makes some leaps in logic that can be hard to follow.\nFor instance, Figure 2 [above Line 196] is supposed to visually convey that, when learning an $\\alpha$-Holder function, the excess risk has an exponential dependence on dimension.\nTo my understanding, if there was no such dependence on the dimension $d$ (as suggested by the convergence rate $O(n^{-\\frac{2\\alpha}{2\\alpha+d}})$) then the heatmaps in Figure 2 would not show any relationship between excess risk (the color of the heatmap) and the dimension.\nBut since the heatmaps do show such a relationship, we can empirically see that the big-Oh bound is hiding a painful dependence on $d$.\n\nThis story is not well conveyed by Figure 2's description.\nThere a basic description of what experiment generated the plots, but there's no connective tissue explaining how the plots relate back to the problem of this dimension dependence.\nI also have exactly no understanding of why the third plot in Figure 2 is present. No clue what message it's trying to evoke, nor why the x-axis of the plot is nonlinear in such an odd way.\nThis sort of lack-of-connective tissue appears in a few places, and really harms my understanding of how various puzzle pieces fit together in this paper.\n\nThis writing issue is especially painful in Section 4, where a theoretically tighter but harder to interpret bound on excess risk is presented in equation (14) [Line 265]. This harder-to-interpret rate is not really explained and instead the writing shifts to discussing a very intuitive explanation of two synthetic experiments where the excess rate decays fast-then-slow and slow-then-fast. The tie between the equation (14) and these experiments is thoroughly unclear.\n\nThese issues of clarity are really my big qualms with this paper.\nHaving not fully understood some of this technical material, or rather how this technical material relates to the experiments and intuitions, I really hesitate to accept the paper, hence the borderline judgement.\n\nQUESTIONS:\nDoes Prop 1 [Line 194] really show an exponential dependence on dimension, or is the dependence polynomial in dimension and exponential in the degree $\\alpha$?\nSpecifically, I'm thinking of the approximation $(\\frac{n}{k})^k \\leq \\binom{n}{k} \\leq (\\frac{ne}{k})^k$, which implies that\n$$\\textstyle{(1+\\frac d\\alpha)^\\alpha = (\\frac{d+\\alpha}{\\alpha})^\\alpha \\leq \\binom{d+\\alpha}{\\alpha} \\leq (\\frac{e(d+\\alpha)}{\\alpha})^\\alpha = e^\\alpha (1+\\frac{d}{\\alpha})^\\alpha}$$\nand therefore overall that the term in proposition 1 grows polynomially in $d$ for fixed alpha, and grows exponentially in $\\alpha$.\nAm I missing something here?\n\n---\n\nHere's a list of typos / recommended edits. Feel free to use whatever edits you want to use, these are all recommendations.\n\n1. [Line 10] \"they\" instead of \"that\"\n1. [Line 17] \"mapping's\" instead of \"mapping\"\n1. [Line 17] \"can we\" instead of \"to\"\n1. [Line 20] end sentence with a question mark\n1. [Line 23] add \"us\" after \"allows\"\n1. [Line 35] \"was\" instead of \"were\"\n1. [Figure 1] Define $\\mathcal{N}_1(\\sigma,\\lambda)$ or don't use the symbol.\n1. [Line 64] Sometime before this line (i.e. before Section 2), define the notion of a transitory regime. Perhaps use the sentence from Figure 1, but outside of a figure caption.\n> Those transitory regimes, where the behavior of the excess risk can be quite different than its asymptotic “stationary” behavior, are usually not well described by theory, while they might be the dominating regimes in applied machine learning when the number of samples is relatively small compared to the input dimension\n1. [Line 50] Wrong figure reference\n1. [Line 51] \"era\" instead of \"area\"\n1. [Line 135] Remove \"$\\varepsilon^2/n$\"\n1. [Line 173-174] I think the grammar of this sentence is completely borked? Or maybe there's just a word or two that's throwing me way off? I dunno what you're trying to say.\n1. [Line 213] Digest the rate of Prop 2 for us. For example, lower bound the maximum by the choice of $\\ell=d$, and show the rate of the two terms, so that we can easily tell that the second term dominates the first term when $\\|\\|\\vec m\\|\\|_2$ is large enough. (If I did my math correctly, it's roughly when $\\|\\|\\vec m\\|\\| \\geq d$?). Take this as a moment to return to the broader story about the scale of the constants in the big-Oh notation and really pull everything back together.\n1. [Line 224] Remove \"a\"\n1. [Line 254] You mention inequalities, but it's not clear if they're $\\geq$ or $\\leq$\n1. [Figure 3] Include labels for the axes and titles on the plots.\n1. [Line 262] add \"being\" after \"$c_\\gamma$\"\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nClassical results in learning theory have shown that smoothness of the function class can be leveraged to \"break\" the curse of dimensionality in empirical risk minimization. However, this paper shows that in the specific case where a Taylor expansion or a Fourier expansion is used to leverage smoothness, the constant in the sample complexity can grow exponentially in $d$, which is the dimension of the function.\n\nSTRENGTHS:\nI think the theoretical results in this paper are interesting since it tackles a very fundamental question in learning theory: \"is smoothness of the target function enough to guarantee good generalization?\" Classical results in learning theory (e.g. Theorem 1) show that the excess risk for estimating $\\alpha$-smooth functions are upper bounded as $c n^{-2 \\alpha /(2 \\alpha+d)}$, where $n$ is the sample size, $d$ is the dimension of the input, and $c$ is a constant. Thus, for a fixed $d$, as the sample size $n$ grows, the excess risk decays rapidly to zero.\n\n In this paper, the authors show that in specific cases, such as when a Taylor expansion or Fourier expansion is used to estimate $f^\\star$, the constant $c$ can actually be exponential in $d$. Therefore, at least in these specific cases, smoothness alone is not enough to guarantee efficient learning algorithm in terms of the sample complexity. I think these negative results in terms of sample complexity are of interesting because it shows that smoothness might not be a strong enough prior to guarantee good generalization.\n\nWEAKNESSES:\nI think the writing of this paper could be significantly improved. In its current state, it takes a long time to see what are the \"main results\" and main contributions of this paper. It is also hard to which parts of the paper are restating standard results or standard proofs, and which parts are technically new. On line 132 the authors state that \"the usual proof of Theorem 1 [which is a known result] relies on theorems such as Theorem 2 below\". This is very confusing to me. Is theorem 2 a standard result that the authors restate? Or is it a new result proved in this paper. How it is different from previous theorems used to prove Theorem 1? There also parts of the paper that are just difficult to understand because of the English. For instance, the sentence on line 172 makes no grammatical sense and is hard to understand. \n\nOverall I think both the organization and the writing of this paper needs to be significantly improved, otherwise it would be hard for readers to spot the main contributions of this paper. The organization and the writing is the main issue preventing me from giving this paper a higher score, as I think this paper could benefit greatly from a major revision. \n\n**Update** -- After the rebuttal I have decided to increase the score by 1, because the revised version outlined in the rebuttal pdf is much clearer than the current version. I hope the authors can implement all these changes, as promised.\n\nQUESTIONS:\nI also have trouble understanding the role of the meta-algorithm described on line 161. How is it related to the main results regarding Taylor and Fourier expansions?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}