Title: Dimension-free deterministic equivalents and scaling laws for random feature regression
Venue: NeurIPS 2024
Accepted: yes
Keywords: random features

deterministic equivalents

error rates

random matrix theory

scaling laws
Abstract: In this work we investigate the generalization performance of random feature ridge regression (RFRR). Our main contribution is a general deterministic equivalent for the test error of RFRR. Specifically, under a certain concentration property, we show that the test error is well approximated by a closed-form expression that only depends on the feature map eigenvalues. Notably, our approximation guarantee is non-asymptotic, multiplicative, and independent of the feature map dimension---allowing for infinite-dimensional features. We expect this deterministic equivalent to hold broadly beyond our theoretical analysis, and we empirically validate its predictions on various real and synthetic datasets. As an application, we derive sharp excess error rates under standard power-law assumptions of the spectrum and target decay. In particular, we provide a tight result for the smallest number of features achieving optimal minimax error rate.

Dimension-free deterministic equivalents and scaling laws for random feature regression

Abstract

In this work we investigate the generalization performance of random feature ridge regression (RFRR). Our main contribution is a general deterministic equivalent for the test error of RFRR. Specifically, under a certain concentration property, we show that the test error is well approximated by a closed-form expression that only depends on the feature map eigenvalues. Notably, our approximation guarantee is non-asymptotic, multiplicative, and independent of the feature map dimensionallowing for infinite-dimensional features. We expect this deterministic equivalent to hold broadly beyond our theoretical analysis, and we empirically validate its predictions on various real and synthetic datasets. As an application, we derive sharp excess error rates under standard power-law assumptions of the spectrum and target decay. In particular, we provide a tight result for the smallest number of features achieving optimal minimax error rate.

1 Introduction

At odds with classical statistical intuition, overparametrized neural networks are able to generalize while perfectly interpolating the training data. This observation, which defies the canonical mathematical understanding of generalization based on complexity measures and uniform convergence, appears surprising at first [Zhang et al., 2017]. However, recent progress in our mathematical understanding of generalization has taught us that this benign overfitting property of overparametrized neural networks is shared by a plethora of simpler learning tasks [Bartlett et al., 2021, Belkin, 2021]. Among them, the investigation of the following class of random feature models has been at the forefront of this progress:

Here x ∈ X denotes the inputs and W = ( w j ) j ∈ [ p ] a set of weight vectors which are taken to be random w j ∈ W ⊆ R d ∼ iid µ w . Hence, as the name suggests the feature map φ : X × W → R defines a random function. A few examples of random feature maps include the fully connected neural network features φ ( x , w ) = σ ( ⟨ w , x ⟩ ) , σ : R → R , and the convolutional features with global average pooling φ ( x , w ) = 1 / d ∑ d ℓ =1 σ ( ⟨ w , g ℓ · x ⟩ ) where σ : R → R and

g ℓ · x = ( x ℓ +1 , . . . , x d , x 1 , . . . , x ℓ ) is the ℓ -shift operator with cyclic boundary conditions (both with X = R d ).

Random features [Balcan et al., 2006, Rahimi and Recht, 2007] were originally introduced as a computationally efficient approximation for the limiting kernel:

Although this can reduce the computational cost of kernel methods, it introduces an approximation error. Rahimi and Recht [2008] showed that in a supervised setting with n samples, p = O ( n ) features are sufficient to achieve an excess risk O ( n -1 / 2 ) . Rudi and Rosasco [2017] improved this result under standard power-law assumptions on the asymptotic kernel spectrum, showing that for fast decays, less features are needed to achieve the minimax rate. In Section 4 we will revisit this question, where we will derive a tight result for the minimum number of features.

More recently, the random feature model has gained in popularity as a proxy model for studying the generalization properties of two-layer neural networks in the lazy regime of training [Jacot et al., 2018, Chizat et al., 2019]. Indeed, for particular choices of feature maps such as φ ( x , w ) = σ ( ⟨ w , x ⟩ ) , it can also be seen as a two-layer neural network with fixed first-layer weights. Exact asymptotic results for the generalization error of eq. (1) were derived for different supervised learning tasks under the proportional scaling regime n, p = Θ( d ) in [Mei and Montanari, 2022, Gerace et al., 2021, Dhifallah and Lu, 2020, Hu and Lu, 2023, Goldt et al., 2022, Loureiro et al., 2022, Bosch et al., 2023b,a, Schröder et al., 2023, 2024] and under more general polynomial scaling n, p = Θ( d κ ) in [Simon et al., 2023b, Aguirre-López et al., 2024, Hu et al., 2024]. As discussed above, these works played a fundamental role in our current mathematical understanding of the relationship between overparametrization and generalization, demystifying different phenomena such as double descent Belkin et al. [2019] and benign overfitting Bartlett et al. [2020]. It also led to fundamental separation results between lazy and trained networks [Ghorbani et al., 2019, 2020, Mei et al., 2022], recently motivating the investigation of corrections to the random limit [Ba et al., 2022, Dandi et al., 2023, Moniri et al., 2024, Cui et al., 2024].

With the exception of [Simon et al., 2023b], which is based on non-rigorous arguments, the results in all the works cited above are derived in the asymptotic limit of large data dimension. However, the relative scaling of the n, p, d is fundamentally artificial, and in practice it is hard to unambiguously define the regime of interest. Our main goal in this manuscript is to provide a dimension-free characterization of the generalization error allowing us to give tight answers to questions which cannot be addressed asymptotically. More precisely, our main contributions are:

(i) Under a concentration assumption on the feature map eigenfunctions, we prove a non-asymptotic deterministic approximation for the RFRR risk R test ≈ R n,p which is independent of the feature map dimension. More precisely, with high-probability over the input data and random weights:

where the deterministic equivalent R n,p can be computed by solving a set of self-consistent equations of the type x = f ( x ) , with f a contractive map. This result unifies the long list of asymptotic formulas in the RFRR literature, and proves a conjecture by Simon et al. [2023b]. We numerically validate the results on various real and synthetic datasets. The precise statement of the theorem and the assumptions are discussed in Section 3.

(ii) Leveraging our formula, we investigate the error scaling laws in a setting where the target function and feature spectrum decay as a power-law, also known as source and capacity conditions. We provide a full picture of the different scaling regimes and the cross-overs between them, summarized in Figure 2. Our result is closely related to the neural scaling laws literature [Kaplan et al., 2020], and provides the first rigorous, non-linear extension of [Bahri et al., 2024, Maloney et al., 2022].

(iii) We provide a sharp expression for the minimum number of features required to achieve the minimax optimal decay rate of Caponnetto and De Vito [2007], closing the gap of previous lower-bounds in the literature [Rudi and Rosasco, 2017].

Further related works Deterministic equivalents have been derived for a wide range of learning problems, such as ridge regression [Dobriban and Wager, 2018, Hastie et al., 2022, Cheng and Montanari, 2022, Wei et al., 2022], kernel regression [Misiakiewicz and Saeed, 2024], shallow [Liao

and Couillet, 2018, Mei and Montanari, 2022, Chouard, 2022, Bach, 2024, Atanasov et al., 2024] and deep random feature regression [Fan and Wang, 2020, Schröder et al., 2023, 2024, Chouard, 2023] and spiked random features [Wang et al., 2024]. Scaling laws under source and capacity conditions were studied by several authors in the context of kernel ridge regression [Bordelon et al., 2020, Spigler et al., 2020, Cui et al., 2022, Simon et al., 2023a, Li et al., 2023, Misiakiewicz and Mei, 2022, Favero et al., 2021, Cagnetta et al., 2023, Dohmatob et al., 2024] and classification [Cui et al., 2023].

2 Setting

In this work, we focus on the generalization properties of the random feature class F RF defined in eq. (1) in a supervised regression setting. More precisely, consider a data set D = { ( x i , y i ) i ∈ [ n ] } composed of n independent and identically distributed samples from a joint distribution µ x,y on X × R . Let f ⋆ ( x ) = E [ y | x ] denote the target function. We assume f ⋆ ∈ L 2 ( µ x ) , where µ x is the marginal distribution over X . Moreover, we assume the noise ε := y -f ⋆ ( x ) has zero mean and finite variance E [ ε 2 ] = σ 2 ε < ∞ . Note this is equivalent to:

Given the training data, we are interested in the properties of the minimiser:

where we have defined the feature matrix Z ij = p -1 / 2 φ ( x i ; w j ) and the label vectors y = ( y i ) i ∈ [ n ] . In particular, we are interested in its capacity of generalising to unseen data, as quantified by the excess population risk :

It will be convenient to decompose the excess risk above in terms of the standard bias and variance:

where:

Note that to simplify the exposition, we have explicitly taken an expectation over the training data noise ε = ( ε i ) i ∈ [ n ] . Indeed, it can be shown that the excess risk eq. (6) concentrates on its expectation over ε under mild assumptions (see for example Misiakiewicz and Saeed [2024]).

3 Deterministic equivalents

The excess risk eq. (6) is a function of the covariates X and the weights W , and therefore it is a random quantity. Our main result in what follows is a sharp characterization of the bias and variance in terms of a deterministic equivalent depending only on the model parameters and spectral properties of the features. Consider a square-integrable φ ∈ L 2 ( X × W ) , and define the Fredholm integral operator T : L 2 ( X ) →V ⊆ L 2 ( W ) :

where we define V = Im( T ) . This is a compact operator, and therefore can be diagonalized:

where ( ξ k ) k ≥ 1 ⊆ R are the eigenvalues and ( ψ k ) k ≥ 1 and ( ϕ k ) k ≥ 1 are orthonormal bases of L 2 ( X ) and V respectively:

Without loss of generality, we assume the eigenvalues are ordered in non-increasing absolute values | ξ 1 | ≥ | ξ 2 | ≥ . . . , and for simplicity of presentation we assume that all eigenvalues are non-zero, i.e., ker( T ) = { 0 } . Denote Σ = diag( ξ 2 1 , ξ 2 2 , . . . ) ∈ R ∞×∞ the diagonal matrix of the squared eigenvalues. Similarly, since f ⋆ ∈ L 2 ( µ x ) , it admits the following decomposition in ( ψ ) k ≥ 1 :

Our formal results will assume the following concentration property over the eigenfunctions.

Assumption 3.1 (Concentration of the eigenfunctions) . Denote the (infinite-dimensional) random vectors 1 ψ := ( ξ k ψ k ( x )) k ≥ 1 and ϕ := ( ξ k ϕ k ( w )) k ≥ 1 . There exists a constant C x > 0 such that for any deterministic p.s.d. matrix A ∈ R ∞×∞ , i.e. a linear operator acting on an infinite-dimensional Hilbert space, with Tr( Σ A ) < ∞ , we have

While Assumption 3.1 is restrictive and will not be satisfied by many non-linear settings, it covers a number of popular theoretical models studied in the literature: 1) independent sub-Gaussian entries, 2) verifying a log-Sobolev inequality or convex Lipschitz concentration (see Cheng and Montanari [2022]). We expect that Assumption 3.1 can be relaxed using the same procedure as in Misiakiewicz and Saeed [2024] to cover classical examples such as data and weights uniformly distributed on the sphere or hypercube. Such a relaxation is involved and we leave it to future work. We will further assume that:

Assumption 3.2. There exists m ∈ N such that

Furthermore, we will assume that for some C ∗ > 0 that we have

Assumption 3.2 is technical, and we believe it can be removed at the cost of a more involved analysis. For instance, eq. (16) is always satisfied for ξ 2 k ∝ k -α if we take m = O ( p 2 ) . Condition (17) was also considered in Cheng and Montanari [2022], and is satisfied in many settings of interest, for example under source and capacity conditions β k ≍ k -β and ξ 2 k ≍ k -α considered in Section 4.

Main result Our main result concerns a dimension-free characterization of the risk eq. (6) in terms of deterministic equivalents. We start by defining them.

Definition 1 (Deterministic equivalents) . Given integers n, p , covariance matrix Σ and regularization parameter λ ≥ 0 . Consider the parameter ν 2 ∈ R > 0 defined as the unique solution of the selfconsistent equation:

and ν 1 ∈ R > 0 is given by:

We introduce the short-hand:

Then, the deterministic equivalents for the bias, variance and test error are defined as:

Our main result provides precise conditions for when the deterministic equivalents defined in definition 1 are a good approximation for the test error eq. (6), as a function of the dimensions n, p , feature covariance Σ , and regularization λ > 0 . More precisely, the approximation rates will depend on them through the following quantities:

Below we denote C a 1 ,...,a k constants that only depend on the values of { a i } i ∈ [ k ] . We use a i = ' ∗ ' to denote the dependency on the constants in Assumptions 3.1 and 3.2.

Theorem 3.3 (Test error of RFRR) . Under Assumptions 3.1, 3.2 and for any D,K > 0 , there exist constants η ∗ ∈ (0 , 1 / 2) and C ∗ ,D,K > 0 such that the following holds. For any n, p ≥ C ∗ ,D,K , regularization λ > 0 , and target function f ⋆ ∈ L 2 ( µ x ) , if

then with probability at least 1 -n -D -p -D , we have

where R n,p ( β ∗ , λ ) has been defined in eq. (24) and:

and the approximation rate is given by

For typical settings, with regularly varying spectrum, ρ κ ( p ) ≲ log( p ) C /κ and ˜ ρ κ ( n, p ) ≲ log( n ∧ p ) C /κ . In this case, the approximation rate scales as E ( n, p ) = ˜ O ( n -1 / 2 + p -1 / 2 ) , which matches the optimal rates expected from local law fluctuations. A few remarks on this theorem are in order:

(a) Theorem 3.3 provides fully non-asymptotic approximation bounds for the population risk and its deterministic equivalent. They hold pointwise and for a large class of functions. In particular, they do not require probabilistic assumptions over the target function coefficients β ⋆ , as for instance in [Dobriban and Wager, 2018, Richards et al., 2021, Wu and Xu, 2020].

0

200

400

600

800

1000

1200

p

10

1

10 0

Excess Risk

0

200

400

600

800

1000

p

10 0

=10 4

=10 3

=10 2

=10 1

=

p

1

=

p

1/2

(b) They are not explicitly dependent on the feature map dimension.

(c) They are multiplicative, and therefore relative to the scale of the risk. In particular, they hold even if R≍ n -γ , which will be crucial to the discussion in section 4.

(d) Theorem 3.3 is considerably more general than previous results. First, it extends the dimension-free results of Cheng and Montanari [2022] for well-specified ridge regression and Misiakiewicz and Saeed [2024] for KRR (see p →∞ discussion below) to the case of feature maps φ : X × W → R , which, as discussed in Section 2, comprises several cases of interest in machine learning. Moreover, the deterministic equivalent recovers as particular cases the asymptotic results derived under proportional n, p = Θ( d ) [Mei and Montanari, 2022, Loureiro et al., 2022, Schröder et al., 2023] and polynomial n, p = Θ( d κ ) [Xiao et al., 2022, Hu et al., 2024, Aguirre-López et al., 2024] scaling.

(e) The bounds depend on λ -1 and λ -1 > m . Following similar arguments as in Cheng and Montanari [2022], Misiakiewicz and Saeed [2024], this assumption could be removed at the cost of a lengthier analysis and worse rates n -C + p -C with C < 1 / 2 .

Figure 1 illustrates Theorem 3.3 in two different settings with real and synthetic data. On the left, we show the population risk of learning a single-index target function with a spiked random features model. This model was recently shown to be equivalent to the first-step of training in a fully-connected two-layer network [Ba et al., 2022], and it was recently studied by several authors [Moniri et al., 2024, Cui et al., 2024, Wang et al., 2024]. On the right, we apply our formulas directly to a real data set. In both cases, the theoretical curves show excellent agreement with the numerical simulations. In Appendix C we present additional plots, together with a discussion of how these plots were generated.

Particular limits We now discuss some particular limits of interest of the deterministic equivalent eq. (24). First, note that at the interpolation threshold n = p , we have 1 -Υ( ν 1 , ν 2 ) ∼ √ λ . Therefore, the risk R n,p ∼ λ -1 / 2 diverges as λ → 0 + , a well-known behaviour known as the interpolation peak in the random feature literature [Hastie et al., 2022, Mei and Montanari, 2022, Gerace et al., 2021] and observed in neural networks [Spigler et al., 2019, Nakkiran et al., 2021].

Another limit of interest is p →∞ where, in the generic case, the features span an infinite-dimensional RKHS. Typically, the resulting kernel will be universal, implying it can approximate any function in L 2 ( µ x ) . In this limit, the risk bottleneck is given by the finite amount of data n .

Corollary 3.4 (Kernel limit) . In the p →∞ limit both ν 1 and ν 2 converge to a single ν K which is the unique positive solution to the following self-consistent equation

Moreover, the bias eq. (22) and variance eq. (23) terms simplify to:

We denote the corresponding test error R K ,n ( β ∗ , λ ) = B K ,n ( β ∗ , λ ) + V K ,n ( λ ) .

Note that eq. (23) exactly agrees with the dimension-free deterministic equivalents for kernel methods in Cheng and Montanari [2022], Misiakiewicz and Saeed [2024]. Finally, the third limit of interest is the n →∞ where data is abundant. In this case, the empirical risk eq. (5) converge to the population risk, and therefore the bottleneck in the risk is given by the capacity of the random feature class F RF eq. (1) to approximate the target f ⋆ .

Corollary 3.5 (Approximation limit) . In the n →∞ limit, we have ν 1 → 0 and ν 2 → ν A satisfying the following simplified self-consistent equation:

Moreover, the bias eq. (22) and variance eq. (23) terms simplify to:

We denote the risk in this case R A ,p ( β ∗ ) = B A ,p ( β ∗ ) , which as expected does not depend on λ .

4 Scaling laws

Our exact characterization of the excess risk in Theorem 3.3 shows that the bottleneck in the model performance stems either from its approximation capacity (as measured by the 'width' p ) and the availability of data (as measured by the number of samples n ). In other words, for a fixed data budget n , increasing p might not improve the error besides a certain point, yielding a waste of computational resources. This raises an important question: given a fixed data budged n , what is the optimal choice of model size p ⋆ ?

Context This is a fundamental question in the random feature literature, and was investigated already in the pioneering works of Rahimi and Recht [2007, 2008], who showed that to achieve an excess risk of O ( n -1 / 2 ) requires at most p = O ( n ) features. This upper bound was considerably refined by Rudi and Rosasco [2017] under classical power law scaling assumptions, also known as source and capacity conditions in the kernel literature:

where α ∈ (1 , ∞ ) and r ∈ (0 , ∞ ) , with the case r = 1 / 2 corresponding to f ⋆ belonging to the RKHS of the asymptotic random feature kernel eq. (2). The optimal minmax rate O ( n -2 αr 2 αr +1 ) for ridge regression under source and capacity conditions were obtained by Caponnetto and De Vito [2007]. Rudi and Rosasco [2017] showed that this optimal rate can be attained by the random feature hypothesis eq. (1) with p > p 0 = O ( n α -1+2 r 1+2 αr ) features. However, this is only an upper bound, and understanding how tight it is, as well as the full picture in the hard regime r ∈ (0 , 1 / 2 ) , remains an open question. In this section, we leverage our tight characterization of the excess risk in Theorem 3.3 to provide a sharp answer to this question.

Results Without loss of generality, we can assume the covariance is a diagonal matrix Σ = diag( ξ 2 k ) k ≥ 1 , and we consider the case where the exponents exactly saturate the source and capacity conditions eq. (37):

Further, we assume a relative scaling of the number of features p and the regularization λ with the number of samples n :

with q ≥ 0 and ℓ ≥ 0 .

ℓ

q

0

1

-

ℓ

α

2

ℓ

(

r

∧

1)

α

-

1

α

ℓ

+

q

1

-

q

αq

α

ℓ

∗

1

q

∗

ˆ

q

ℓ

q

0

1

-

q

1

-

ℓ

α

2

αqr

2

ℓr

α

ℓ

∗

1

q

∗

Theorem 4.1 (Excess risk rates) . Under source and capacity conditions eq. (38) and scaling assumptions eq. (39) , the deterministic equivalent eq. (24) rate is given by:

̸

where γ ( ℓ, q ) := γ B ( ℓ, q ) ∧ γ V ( ℓ, q ) for non-zero noise variance σ 2 ε = 0 , otherwise γ ( ℓ, q ) = γ B ( ℓ, q ) . The exponents γ B and γ V are respectively the decay rates of the bias and variance terms eqs. (22) and (23) , and are explicitly given by

Remark 4.1. Under the scaling in eqs. (38) and (39) , one can check that the approximation rates E ( n, p ) in Theorem 3.3 are vanishing for ℓ ≤ α + 1 / 12 if q ≥ 1 , and for ℓ ≤ q (( α + 1 / 16 ) ∨ 1 / 16( α -1) ) if q < 1 , which includes the optimal vertical line ℓ = ℓ ∗ . Hence, for these regions of scaling, Theorem 3.3 readily implies that the excess risk eq. (6) indeed has the decay rates described in Theorem 4.1. As discussed in the previous section, we expect that these approximation guarantees can be improved to include a larger region of decay rates, but we leave it to future work.

̸

A detailed derivation of the result above from the deterministic equivalent characterization from Theorem 3.3 is discussed in Appendix D. The expressions in eq. (41) are easier to visualise in a diagram. Figure 2 shows the excess risk exponent γ ( ℓ, q ) as a function of the parameters ℓ and q , in the case where σ 2 ε = 0 for r ≥ 1 / 2 (left) and r < 1 / 2 (right). Note that the key difference between the diagrams is the presence of an additional region for r ≥ 1 / 2 . 2 Defining the following shorthand:

we can identify two main regions in the ( ℓ, q ) plane, corresponding to a trade-off between the bias γ B and variance γ V terms:

(a) Variance dominated region ( γ V < γ B ): if ℓ > ℓ ⋆ , q > ˆ q and p > λ , the excess risk is dominated by the variance term, provided the number of samples is large enough n ≫ σ -1 / ( γ B ( ℓ,q ) -γ V ( ℓ,q )) ε . 3 Inside this region it is possible to further distinguish between two regimes:

slow decay regime (orange and brown): for ℓ < α and q < 1 ( p ≪ n ), γ V = 1 -( ℓ / α ∧ q ) , hence the decay depends on the interplay between regularization strength and number of random features and it is slower as ( ℓ / α ∧ q ) increases;

plateau regime (red): for ℓ ≥ α and q ≥ 1 ( p ≥ n ) the excess risk converges to a constant value and does not decay as n increases.

(b) Bias dominated region ( γ V > γ B ): if ℓ < ℓ ⋆ , q < ˆ q and p < λ , the excess risk is dominated by the bias term, whose decay is faster as ( ℓ / α ∧ q ) increases (cyan, emerald and teal).

Note that in the limit of large number of random features p →∞ , we recover the same rates found by Cui et al. [2022] for kernel ridge regression. Of particular interest is the rate for which the excess risk decays the fastest with the number of samples n , and what is the minimum number of random features p ⋆ required to achieve this rate.

Corollary 4.2 (Optimal rates) . The optimal excess risk rate achieved by the random features hypothesis eq. (1) under source and capacity conditions eq. (38) and scaling assumptions eq. (39) :

and it is attained for:

corresponding to the bold red line ( -) in Fig. 2. In particular, the minimal number of random features p ⋆ = n q ⋆ required to achieve the optimal rate γ ⋆ is given by:

and corresponds to the bold red dot ( · ) in Fig. 2.

A few comments on Corollary 4.2 are in place. 1) The optimal excess error rate eq. (44) is consistent with the minimax optimal rates for ridge regression from Caponnetto and De Vito [2007], as also discussed by Rudi and Rosasco [2017]. 2) The minimal number of random features p ⋆ = n q ⋆ in eq. (47) achieving the optimal rate eq. (44) in the r ≥ 1 / 2 regime is strictly smaller than the lower bound p > p 0 of Rudi and Rosasco [2017]. More precisely, letting p 0 = n q 0 , for r ∈ [ 1 / 2 , 1) :

Relationship to scaling laws The empirical observation that the performance of large scale neural networks decreases as a power law with respect to the number of samples, parameter and computing time has sparked a renewed wave of interest in the theoretical investigation of power laws [Kaplan et al., 2020]. Despite being a mature topic in the statistical learning literature, different recent works have turned to the study of linear models under source and capacity conditions as a playground to understand the emergence of different bottlenecks in the excess error rates [Bahri et al., 2024, Maloney et al., 2022].

The model studied in these works is given by ridge regression on data y i = ⟨ β ⋆ , x i ⟩ with x ∼ N ( 0 , diag(( d / k ) α )) and β ⋆ ∼ N ( 0 , 1 / d I d ) with a linear projection model ˆ f ( x , a ) = ⟨ a , Wx ⟩ , where W is an i.i.d. Gaussian matrix. Note this model is a particular case of the one studied here, corresponding to a linear feature map and random target function. Moreover, since the variance of the target is constant, the source is entirely determined by the capacity α of the asymptotic kernel, here controlled by the decay of the covariance of the input data.

The approximation limit from Corollary 3.5 and the kernel limit from Corollary 3.4 are known in this literature as Variance and Resolution limited regimes , respectively [Bahri et al., 2024]. They correspond precisely to the bottlenecks in the excess risk arising from the limited approximation

10 2

10 3

10 4

10 5

n

10

4

10

3

10

2

10

1

Excess Risk

=1.2,

r

=0.9,  =

* =0.38

q

=

* =0.38

q

=0.45

q

=0.53

q

=0.

q

* =0.62

q

=0.7

q

=0.8

n

2

*

r

n

q

* +

*

/

10 2

10 3

10 4

n

=3.5,

r

=1.5,

q = 1.25

=0.1

=0.3

=

* =0.44

=1

=2.5

=

=3.5

=4

n

2

r

n

2

*

r

n

1+(1

/

)

capacity of the random feature model or the limited availability of training data. As this model is a particular case of ours, the rates in the variance limited regime can also be obtained from Theorem 4.1, and correspond to particular cases in Fig. 2, see Appendix E for a detailed discussion. Contemporary to our work, Atanasov et al. [2024] has extended the analysis in this linear model to the case where β ⋆ also has a power-law decay, and provided a comprehensive discussion of the different scaling regimes for this model. Their rates can be put in a one-to-one correspondence with the rates derived in section 4. We refer the interested reader to Section VI.6 of Atanasov et al. [2024] for a detailed discussion of this relationship. We stress, however, that beside being rigorous, our results hold for features in infinite-dimensional Hilbert spaces and are not restricted to a particular asymptotic limit in the dimensions.

Complementary to the sample and model complexity bottlenecks, Kaplan et al. [2020] also observed the emergence of computational scaling laws in the risk as a function of flops used in training. A recent line of work has investigated this question on the aforementioned linear random feature model under different training algorithms, such as gradient flow [Bordelon et al., 2024] and SGD [Paquette et al., 2024, Lin et al., 2024]. Due to the simplicity of this setting, the risk of ridge regression with a particular choice of regularization λ is closely related to the risk of different descent algorithms for least-squares at a fixed running horizon [Ali et al., 2019, 2020, Sonthalia et al., 2024]. A similar analogy allows us to compare our results to the ones obtained in [Paquette et al., 2024, Lin et al., 2024]. In particular, our setting cover three of the phases identified by Paquette et al. [2024], and correspond to the result in Theorem 4.1 with λ = 1 ( ℓ = 1 ). Similarly, the rates of Lin et al. [2024] are obtained by taking λ to be the inverse of the learning rate. A detailed connection to this line of work is discussed in Appendix E.

5 Conclusion

In this paper, we have investigated the generalization properties of random feature models, deriving a non-asymptotic deterministic equivalent for the risk of random feature ridge regression-which recovers (and unifies) previous asymptotic findings as special limits. Our results provide a rigorous multiplicative approximation rate, enabling us to analyze error scaling laws under source and capacity conditions, and offers a complete view of the different scaling regimes and their cross-overs. Our analysis relies on Assumption 3.1 which, while popular in theoretical investigations, excludes more realistic random feature models, such as φ ( x , w ) = σ ( x ⊤ w ) with x , w Gaussian vectors and non-linear σ . Although restrictive, this assumption allowed us to derive tight multiplicative approximation bounds for a generic random feature model with infinite-dimensional features-which was essential for obtaining the rigorous excess risk rates that are the primary motivation of our work. We further note that numerical simulations in Figure 1 and Appendix C suggest that the predictions of Theorem 3.3 remain accurate much beyond Assumption 3.1. We consider lifting this technical condition-e.g., by following the approach in Misiakiewicz and Saeed [2024]-to be an important direction for future research.

References

Fabián Aguirre-López, Silvio Franz, and Mauro Pastore. Random features and polynomial rules. arXiv preprint arXiv:2402.10164 , 2024.

Alnur Ali, J Zico Kolter, and Ryan J Tibshirani. A continuous-time view of early stopping for least squares regression. In The 22nd international conference on artificial intelligence and statistics , pages 1370-1378. PMLR, 2019.

Alnur Ali, Edgar Dobriban, and Ryan Tibshirani. The implicit regularization of stochastic gradient flow for least squares. In International conference on machine learning , pages 233-244. PMLR, 2020.

Alexander B. Atanasov, Jacob A. Zavatone-Veth, and Cengiz Pehlevan. Scaling and renormalization in high-dimensional regression. arXiv preprint arXiv:2405.00592 , 2024.

Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. Highdimensional asymptotics of feature learning: How one gradient step improves the representation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 37932-37946. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ f7e7fabd73b3df96c54a320862afcb78-Paper-Conference.pdf .

Francis Bach. High-dimensional analysis of double descent for linear regression with random projections. SIAM Journal on Mathematics of Data Science , 6(1):26-50, 2024. doi: 10.1137/ 23M1558781. URL https://doi.org/10.1137/23M1558781 .

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences , 121(27):e2311878121, July 2024. doi: 10.1073/pnas.2311878121. URL https://www.pnas.org/doi/abs/10.1073/ pnas.2311878121 .

Maria-Florina Balcan, Avrim Blum, and Santosh Vempala. Kernels as features: On kernels, margins, and low-dimensional mappings. Machine Learning , 65(1):79-94, 2006.

Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020. doi: 10.1073/pnas.1907378117. URL https://www.pnas.org/doi/abs/10.1073/ pnas.1907378117 .

Peter L. Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: a statistical viewpoint. Acta Numerica , 30:87-201, 2021. doi: 10.1017/S0962492921000027.

Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation. Acta Numerica , 30:203-248, 2021. doi: 10.1017/S0962492921000039.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, August 2019. doi: 10.1073/pnas.1903070116. URL https: //www.pnas.org/doi/full/10.1073/pnas.1903070116 .

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1024-1034. PMLR, 13-18 Jul 2020. URL https://proceedings. mlr.press/v119/bordelon20a.html .

Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092 , 2024.

David Bosch, Ashkan Panahi, and Babak Hassibi. Precise asymptotic analysis of deep random feature models. In Gergely Neu and Lorenzo Rosasco, editors, Proceedings of Thirty Sixth Conference on Learning Theory , volume 195 of Proceedings of Machine Learning Research , pages 4132-4179. PMLR, 12-15 Jul 2023a. URL https://proceedings.mlr.press/v195/bosch23a.html .

David Bosch, Ashkan Panahi, Ayca Ozcelikkale, and Devdatt Dubhashi. Random features model with general convex regularization: A fine grained analysis with precise asymptotic learning curves. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , volume 206 of Proceedings of Machine Learning Research , pages 11371-11414. PMLR, 25-27 Apr 2023b. URL https://proceedings.mlr.press/v206/bosch23a.html .

Francesco Cagnetta, Alessandro Favero, and Matthieu Wyart. What can be learnt with wide convolutional neural networks? In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 3347-3379. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/cagnetta23a.html .

Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 7(3):331-368, 2007.

Chen Cheng and Andrea Montanari. Dimension free ridge regression. arXiv preprint arXiv:2210.08571 , 2022.

Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper_files/paper/2019/file/ ae614c557843b1df326cb29c57225459-Paper.pdf .

Clément Chouard. Quantitative deterministic equivalent of sample covariance matrices with a general dependence structure. arXiv preprint arXiv:2211.13044 , 2022.

Clément Chouard. Deterministic equivalent of the conjugate kernel matrix associated to artificial neural networks. arXiv preprint arXiv:2306.05850 , 2023.

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime. Journal of Statistical Mechanics: Theory and Experiment , 2022(11):114004, nov 2022. doi: 10.1088/1742-5468/ac9829. URL https://dx.doi.org/10.1088/1742-5468/ac9829 .

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Error scaling laws for kernel classification under source and capacity conditions. Machine Learning: Science and Technology , 4 (3):035033, aug 2023. doi: 10.1088/2632-2153/acf041. URL https://dx.doi.org/10.1088/ 2632-2153/acf041 .

Hugo Cui, Luca Pesce, Yatin Dandi, Florent Krzakala, Yue Lu, Lenka Zdeborova, and Bruno Loureiro. Asymptotics of feature learning in two-layer networks after one gradient-step. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 9662-9695. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/cui24d.html .

Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. How two-layer neural networks learn, one (giant) step at a time. arXiv preprint arXiv:2305.18270 , 2023.

Oussama Dhifallah and Yue M. Lu. A precise performance analysis of learning with random features. arXiv preprint arXiv:2008.11904 , 2020.

Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics , 46(1):247-279, 2018. ISSN 00905364, 21688966. URL https://www.jstor.org/stable/26542784 .

Elvis Dohmatob, Yunzhen Feng, and Julia Kempe. Model collapse demystified: The case of regression. arXiv preprint arXiv:2402.07712 , 2024.

Zhou Fan and Zhichao Wang. Spectra of the conjugate kernel and neural tangent kernel for linearwidth neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 7710-7721. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/572201a4497b0b9f02d4f279b09ec30d-Paper.pdf .

Alessandro Favero, Francesco Cagnetta, and Matthieu Wyart. Locality defeats the curse of dimensionality in convolutional teacher-student scenarios. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 9456-9467. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ 4e8eaf897c638d519710b1691121f8cb-Paper.pdf .

Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Generalisation error in learning with random features and the hidden manifold model. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124013, dec 2021. doi: 10.1088/1742-5468/ac3ae6. URL https://dx.doi.org/10.1088/1742-5468/ac3ae6 .

Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Limitations of lazy training of two-layers neural network. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/ paper_files/paper/2019/file/c133fb1bb634af68c5088f3438848bfd-Paper.pdf .

Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 14820-14830. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper_files/paper/2020/file/a9df2255ad642b923d95503b9a7958d8-Paper.pdf .

Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mezard, and Lenka Zdeborova. The gaussian equivalence of generative models for learning with shallow neural networks. In Joan Bruna, Jan Hesthaven, and Lenka Zdeborova, editors, Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference , volume 145 of Proceedings of Machine Learning Research , pages 426-471. PMLR, 16-19 Aug 2022. URL https://proceedings.mlr.press/v145/goldt22a.html .

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949 - 986, 2022. doi: 10.1214/21-AOS2133. URL https://doi.org/10.1214/21-AOS2133 .

Hong Hu and Yue M. Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory , 69(3):1932-1964, 2023. doi: 10.1109/TIT.2022.3217698.

Hong Hu, Yue M. Lu, and Theodor Misiakiewicz. Asymptotics of random feature regression beyond the linear scaling regime. arXiv preprint arXiv:2403.08160 , 2024.

Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper_files/paper/ 2018/file/5a4be1fa34e62bb8a6ec6b91d2462f5a-Paper.pdf .

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 , 2020.

Yan Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998. doi: 10.1109/5.726791.

Yicheng Li, haobo Zhang, and Qian Lin. On the asymptotic learning curves of kernel ridge regression under power-law decay. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 49341-49364. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/ 2023/file/9adc8ada9183f4b9a007a02773fd8114-Paper-Conference.pdf .

Zhenyu Liao and Romain Couillet. On the spectrum of random features maps of high dimensional data. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 3063-3071. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/liao18a.html .

Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee. Scaling laws in linear regression: Compute, parameters, and data. arXiv preprint arXiv:2406.08466 , 2024.

Bruno Loureiro, Cédric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacherstudent model. Journal of Statistical Mechanics: Theory and Experiment , 2022(11):114001, nov 2022. doi: 10.1088/1742-5468/ac9825. URL https://dx.doi.org/10.1088/1742-5468/ ac9825 .

Alexander Maloney, Daniel A. Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.16859 , 2022.

Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 75(4):667-766, 2022. doi: https://doi.org/10.1002/cpa.22008. URL https://onlinelibrary. wiley.com/doi/abs/10.1002/cpa.22008 .

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Generalization error of random feature and kernel methods: Hypercontractivity and kernel matrix concentration. Applied and Computational Harmonic Analysis , 59:3-84, 2022. ISSN 1063-5203. doi: https://doi.org/10. 1016/j.acha.2021.12.003. URL https://www.sciencedirect.com/science/article/pii/ S1063520321001044 . Special Issue on Harmonic Analysis and Machine Learning.

Theodor Misiakiewicz and Song Mei. Learning with convolution and pooling operations in kernel methods. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 29014-29025. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/ file/ba8aee784ffe0813890288b334444eda-Paper-Conference.pdf .

Theodor Misiakiewicz and Basil Saeed. A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator. arXiv preprint arXiv:2403.08938 , 2024.

Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 36106-36159. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr.press/v235/moniri24a.html .

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124003, dec 2021. doi: 10.1088/1742-5468/ac3a74. URL https://dx.doi.org/10.1088/1742-5468/ac3a74 .

Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+3 phases of computeoptimal neural scaling laws. arXiv preprint arXiv:2405.15074 , 2024.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems , volume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips.cc/paper_ files/paper/2007/file/013a006f03dbc5392effeb8f18fda755-Paper.pdf .

Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems , volume 21. Curran Associates, Inc., 2008. URL https://proceedings.neurips.cc/paper_files/paper/2008/file/ 0efe32849d230d7f53049ddc4a4b0c60-Paper.pdf .

Dominic Richards, Jaouad Mourtada, and Lorenzo Rosasco. Asymptotics of ridge(less) regression under general source condition. In Arindam Banerjee and Kenji Fukumizu, editors, Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 3889-3897. PMLR, 13-15 Apr 2021. URL https://proceedings.mlr.press/v130/richards21b.html .

Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/ file/61b1fb3f59e28c67f3925f3c79be81a1-Paper.pdf .

Dominik Schröder, Hugo Cui, Daniil Dmitriev, and Bruno Loureiro. Deterministic equivalent and error universality of deep random features learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 30285-30320. PMLR, 23-29 Jul 2023. URL https://proceedings. mlr.press/v202/schroder23a.html .

Dominik Schröder, Daniil Dmitriev, Hugo Cui, and Bruno Loureiro. Asymptotics of learning with deep structured (Random) features. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research , pages 43862-43894. PMLR, 21-27 Jul 2024. URL https://proceedings.mlr. press/v235/schroder24a.html .

James B. Simon, Madeline Dickens, Dhruva Karkada, and Michael R. DeWeese. The eigenlearning framework: A conservation law perspective on kernel regression and wide neural networks. arXiv preprint arXiv:2110.03922 , 2023a.

James B. Simon, Dhruva Karkada, Nikhil Ghosh, and Mikhail Belkin. More is better in modern machine learning: when infinite overparameterization is optimal and overfitting is obligatory. arXiv preprint arXiv:2311.14646 , 2023b.

Rishi Sonthalia, Jackie Lok, and Elizaveta Rebrova. On regularization via early stopping for least squares regression. arXiv preprint arXiv:2406.04425 , 2024.

S Spigler, M Geiger, S d'Ascoli, L Sagun, G Biroli, and M Wyart. A jamming transition from under- to over-parametrization affects generalization in deep learning. Journal of Physics A: Mathematical and Theoretical , 52(47):474001, oct 2019. doi: 10.1088/1751-8121/ab4c8b. URL https://dx.doi.org/10.1088/1751-8121/ab4c8b .

Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher-student paradigm. Journal of Statistical Mechanics: Theory and Experiment , 2020(12):124001, dec 2020. doi: 10.1088/1742-5468/abc61d. URL https://dx. doi.org/10.1088/1742-5468/abc61d .

Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning , 8(1-2):1-230, 2015.

Roman Vershynin. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv:1011.3027 , 2010.

Zhichao Wang, Denny Wu, and Zhou Fan. Nonlinear spiked covariance matrices and signal propagation in deep neural networks. arXiv preprint arXiv:2402.10127 , 2024.

Alexander Wei, Wei Hu, and Jacob Steinhardt. More than a toy: Random matrix models predict how real-world neural representations generalize. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 23549-23588. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/ wei22a.html .

Denny Wu and Ji Xu. On the optimal weighted \ell_2 regularization in overparameterized linear regression. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 10112-10123. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/ file/72e6d3238361fe70f22fb0ac624a7072-Paper.pdf .

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

Lechao Xiao, Hong Hu, Theodor Misiakiewicz, Yue Lu, and Jeffrey Pennington. Precise learning curves and higher-order scalings for dot-product kernel regression. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 4558-4570. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 1d3591b6746204b332acb464b775d38d-Paper-Conference.pdf .

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview.net/forum?id=Sy8gdB9xx .

Appendix

A Background on deterministic equivalents

We consider a feature vector f ∈ R q , q ∈ N ∪{∞} , with covariance matrix Σ = E [ ff T ] . We denote γ 2 1 ≥ γ 2 2 ≥ γ 2 3 ≥ · · · the eigenvalues of Σ in non-increasing order. In the case of infinite-dimensional features q = ∞ , we will further assume that Tr( Σ ) < ∞ , i.e., we consider Σ to be a trace-class self-adjoint operator.

We assume that the feature f satisfies the following assumption.

Assumption A.1 (Feature concentration) . There exist c ∗ , C ∗ > 0 such that for any p.s.d. matrix A ∈ R q × q with Tr( Σ A ) < ∞ , we have

Recall that we denote C a 1 ,...,a k constants that only depend on the values of { a i } i ∈ [ k ] . We use a i = ' ∗ ' to denote the dependency on the constants c ∗ , C ∗ from Assumption A.1.

We will further define the following two quantities.

Definition 2 (Effective regularization) . For an integer n , covariance Σ , and regularization λ ≥ 0 , we define the effective regularization λ ∗ associated to model ( n, Σ , λ ) to be the unique non-negative solution to the equation

Throughout this appendix, we assume that λ > 0 . The existence and uniqueness follow from noticing that the left-hand side is monotonically increasing in λ ∗ while the right-hand side is monotonically decreasing. We consider the change of variable µ ∗ := µ ∗ ( λ ) = λ/λ ∗ , such that µ ∗ is the unique non-negative solution of

Both µ ∗ and λ ∗ are increasing functions with λ .

Definition 3 (Intrinsic dimension) . For a covariance matrix Σ ∈ R q × q with eigenvalues in nonincreasing order γ 2 1 ≥ γ 2 2 ≥ γ 2 3 ≥ · · · , we define the intrinsic dimension r Σ ( k ) at level k ∈ N of Σ to be the intrinsic dimension of the covariance matrix Σ ≥ k = diag( γ 2 k , γ 2 k +1 , . . . ) , i.e., the covariance matrix projected orthogonally to the top k -1 eigenspaces, which is given by

The intrinsic dimension of Σ ≥ k captures the number of dimensions of Σ ≥ k that have significant spectral content, i.e., γ 2 j ≈ ∥ Σ ≥ k ∥ op (see [Tropp et al., 2015, Chapter 7] for further background).

We are given n i.i.d. features ( f i ) i ∈ [ n ] , and we denote F = [ f 1 , . . . , f n ] T ∈ R n × q the feature matrix. The train and test errors are functionals of the feature matrix F . In particular, they depend on the following resolvent matrix

In this section we consider functionals that depend on products of F , R

For a general p.s.d. matrix A ∈ R

and deterministic matrices. q × q , define the functionals

These functionals are well approximated by quantities proportional to

Without loss of generality, we can assume that Tr( A Σ ) < ∞ for Φ 1 , as otherwise Φ 1 ( F ; A , λ ) = Ψ 1 ( µ ∗ ; A , λ ) = ∞ almost surely, and Tr( A Σ 2 ) < ∞ for Φ 3 and Φ 4 , as otherwise Φ j ( F ; A , λ ) = Ψ j ( µ ∗ ; A , λ ) = ∞ , j = 3 , 4 , almost surely.

Our relative approximation bound will depend on the covariance matrix Σ through

where η ∗ ∈ (0 , 1 / 2) is a constant that will only depend on c ∗ , C ∗ and we used the convention that γ 2 ⌊ η ∗ · n ⌋ = 0 if ⌊ η ∗ · n ⌋ > q .

The following theorem gathers the approximation guarantees for the different functionals stated above, and is obtained by modifying [Misiakiewicz and Saeed, 2024, Theorem 4].

Theorem A.2 (Dimension-free deterministic equivalents) . Assume the features ( f i ) i ∈ [ n ] satisfy Assumption A.1 with some constants c x , C x , β > 0 . For any D,K > 0 , there exist constants η := η x ∈ (0 , 1 / 2) (only depending on c x , C x , β ), C D,K > 0 (only depending on K,D ), and C x,D,K > 0 (only depending on c x , C x , β, D, K ), such that the following holds. For all n ≥ C D,K and λ > 0 , if it holds that

then for any p.s.d. matrix A , we have with probability at least 1 -n -D that

Proof of Theorem A.2. The only difference between this theorem and [Misiakiewicz and Saeed, 2024, Theorem 4] comes from the definition of ρ λ ( n ) . This new definition is obtained by slightly modifying the proof bounding the operator norm of Σ 1 / 2 R Σ 1 / 2 from [Cheng and Montanari, 2022, Lemma 7.2] and [Misiakiewicz and Saeed, 2024, Lemma 1]. In particular, we will simply modify step 2 in the proof of [Misiakiewicz and Saeed, 2024, Lemma 1]. Consider F + = [ f + , 1 , . . . , f + ,n ] T ∈ R n × ( q -k ∗ ) where f + ,i correspond to the projection orthogonal to the top k ∗ := ⌊ η ∗ n ⌋ -1 eigenspaces with covariance matrix

Then, denoting S = ∑ i ∈ [ n ] S i with S i := f + ,i f T i, + , we have with probability at least 1 -n -D , for any i ∈ [ n ] ,

Denoting ˜ S = ∑ i ∈ [ n ] ˜ S i with ˜ S i := S i ✶ ∥ S i ∥ op ≤ L n , so that ˜ S = S with probability at least 1 -n -D , we obtain

Therefore, applying the matrix Bernstein's inequality with intrinsic dimension [Tropp et al., 2015, Theorem 7.3.1] to ˜ S gives that with probability at least 1 -n -D ,

Note that by the condition of our theorem, log( n ) ≤ K √ n , and therefore

Following the rest of the argument in [Misiakiewicz and Saeed, 2024, Lemma 1] we obtain ρ λ ( n ) in Eq. (52).

B Proof of the deterministic equivalent for RFRR

In this appendix, we prove the approximation guarantees stated in Theorem 3.3 between the test error of RFRR and its deterministic equivalent. We start in Section B.1 by introducing background and notations that we will use throughout the proof. Section B.2 introduces key results on the covariance matrix and the fixed points. We then leverage these results to prove deterministic equivalents for different functionals of Z = ( σ ( ⟨ x i , w j ⟩ )) i ∈ [ n ] ,j ∈ [ p ] ∈ R n × p conditional on ( w j ) j ∈ [ p ] in Section B.3, and functionals of F = ( ξ k ϕ k ( w j )) j ∈ [ p ] ,k ≥ 1 ∈ R p ×∞ in Section B.4. Given these deterministic equivalents, we prove our approximation guarantees for the variance term in Section B.5, and for the bias term in B.6. Finally, we deffer the proof of some technical results to Section B.7.

B.1 Preliminaries

Recall that throughout the paper, we will keep track of the parameters of the problem ( n, p, Σ , λ, σ 2 ε ) . For the other constants C ∗ , K, D , we will denote C a 1 ,a 2 ,...,a k constants that only depend on the values of { a i } i ∈ [ k ] . We use a i = ' ∗ ' to denote the dependency on the constant C ∗ appearing in Assumption 3.1 and Assumption 3.2.

Throughout this appendix, we will directly work in the 'feature space'

with distribution induced by x i ∼ µ x and w j ∼ µ w . We will denote the covariate feature and weight feature matrices by

We denote the random feature weight vector

and the associated feature matrix

Note that f has covariance matrix

We will further introduce the covariance matrix of z conditional on the weight feature matrix F (i.e., conditional on ( w j ) j ∈ [ p ] )

Note that under Assumption 3.1, the features z and f satisfy the following assumption.

Assumption B.1 (Concentration of the features z and f ) . There exists a constant C ∗ > 0 such that for any weight feature matrix F ∈ R p ×∞ and deterministic p.s.d. matrix A ∈ R p × p with Tr( ̂ Σ F A ) < ∞ , we have

and for any deterministic p.s.d. matrix B ∈ R ∞×∞ with Tr( Σ B ) < ∞ ,

We will assume in the rest of this appendix that Assumption B.1 holds. Using the notations introduced above, we restate our setting below. Recall that we consider learning a target function h ∗ ( g ) := g T β ∗ from i.i.d. samples ( y i , g i ) i ∈ [ n ] with

where ε i are independent noise with E [ ε i ] = 0 and E [ ε 2 i ] = σ 2 ε . Denote y = ( y 1 , . . . , y n ) the vector containing the labels. We fit this data using a random feature model with i.i.d. random weight features ( f j ) j ∈ [ p ]

We fit the parameter a using random feature ridge regression (RFRR)

The test error is then given by

where the bias and variance terms are given explicitly by

Note that both the bias and variance terms are random quantities that depend on the random matrices G , F . The goal of this appendix is to prove non-asymptotic and multiplicative approximation guarantees between these two terms and deterministic quantities that only depend on the parameters of the model ( n, p, Σ , β ∗ , λ, σ 2 ε ) , i.e., we will show that with high probability

where B n,p ( β ∗ , λ ) and V n,p ( λ ) are defined in Eqs (22) and (23), and the approximation rates ˜ O ( · ) are explicit in terms of the model parameters.

The proof of these approximation guarantees will proceed in two steps. We first show that the bias and variance terms conditional on F are well approximated by functionals that only depend on F , i.e.,

We then show that ˜ B ( β ∗ ; F , λ ) and ˜ V ( F , λ ) are well approximated by B n,p ( β ∗ , λ ) and V n,p ( λ ) with

For each of these two steps, we will apply general results showing deterministic equivalents for functionals of (possibly infinite-dimensional) random matrices proved in Misiakiewicz and Saeed [2024] (see Appendix A and in particular Theorem A.2 for some background). Note that when writing the proof, we will directly show Eq. (66) assuming that F is in some good event F ∈ A F , where P F ( A F ) ≥ 1 -p -D , so that we can immediately write functionals with regularization parameter that does not depend on ̂ Σ F , and therefore the rate will be directly ˜ O ( n -1 / 2 + p -1 / 2 ) . However, we can reorganize the proof to indeed get the separate contributions (66) and (67) to the approximation error rate.

The rest of Appendix B is devoted to implementing this proof strategy. We start in the next three sections by introducing key technical results which we will use in the analysis of the bias and variance terms.

B.2 Fixed points, feature covariance matrix, and tail rank

Recall that our deterministic equivalents will depend on the fixed points ( ν 1 , ν 2 ) ∈ R 2 > 0 stated in Definition 1. Furthermore, our approximation guarantees will depend on the covariance matrix Σ through ρ κ ( p ) and ˜ ρ κ ( n, p ) which we restate below for convenience

where η ∗ ∈ (0 , 1 / 4) is a constant that only depends on C ∗ appearing in Assumption B.1, and r Σ ( k ) is the intrinsic dimension of Σ at level k (see Definition 3)

Observe that ˜ ρ κ ( n, p ) is well defined for n → ∞ while p stays constant with ˜ ρ κ ( ∞ , p ) = 1 , and for p →∞ while n stays constant with ˜ ρ κ ( n, ∞ ) = ρ κ ( n ) (under the conditions in our setting that ρ κ ( p ) ≤ K √ p for some constant K ).

In this section, we introduce and prove properties on the fixed point ( ν 1 , ν 2 ) ∈ R 2 > 0 and the weight feature matrix F which we will use to prove deterministic equivalents.

Feature covariance matrix. The features z i ∈ R p conditional on F are i.i.d. random vectors with covariance ̂ Σ F = FF T /p . We first show that with high probability over F , this feature covariance matrix has eigenvalues and intrinsic dimensions bounded by the ones of Σ .

Denote ( ˆ ξ 2 1 , ˆ ξ 2 2 , . . . ˆ ξ 2 p ) the p eigenvalues of ̂ Σ F in nonincreasing order. Applying the definition of the intrinsic dimension at level k to ̂ Σ F , we have for any k = 1 , . . . , p ,

Applying Theorem A.2 directly to functionals of Z conditional on F , the approximation guarantees depend on

which simply corresponds to ρ λ defined in Eq. (52) applied to ̂ Σ F where we recall that ˆ ξ 2 ⌊ η ∗ · n ⌋ = 0 if ⌊ η ∗ · n ⌋ > p . The next lemma shows that with high probability over F , we have ̂ ρ λ ( n ) ≲ ˜ ρ λ ( n, p ) for all n ∈ N .

Lemma B.2 (Feature covariance matrix) . Assume the feature vectors { f j } j ∈ [ p ] satisfy Assumption B.1. Then for any D,K > 0 , there exist constants η ∗ ∈ (0 , 1 / 4) and C ∗ ,D,K > 0 such that the following holds. For any p ≥ C ∗ ,D,K , the event

holds with probability at least 1 -p -D .

We defer the proof of this lemma to Section B.7.1.

High-degree part of the feature matrix F . Recall that ( ν 1 , ν 2 ) ∈ R 2 > 0 are the solutions to the fixed point equations stated in Definition 1. In order to get approximation guarantees when pν 1 → 0 as n →∞ , we will analyze separately the top eigenspaces of F from the rest. For an integer m ∈ N , we split the feature vector f j = [ f 0 ,j , f + ,j ] where f 0 ,j ∈ R m corresponds to the top m coordinates with covariance

and f + ,j ∈ R ∞ corresponds to the high degree features orthogonal to the top m eigenspaces. We denote their covariance

We split the weight feature matrix intro F = [ F 0 , F + ] , where

We will use that for m chosen such that p · ξ m +1 ≪ Tr( Σ + ) , we have with high probability

and therefore

where we defined the function

To simplify the final statement of our results, we assume that we can choose m such that p 2 ξ 2 m +1 ≤ γ ( pλ/n ) . Note that ν 1 ≥ λ/n from the fixed point equations and γ ( pλ/n ) ≤ γ ( pν 1 ) (e.g., see Equation (76)). For convenience, we will further denote

Lemma B.3 (Concentration of high-degree part of F ) . Assume that ( f + ,j ) j ∈ [ p ] satisfy Assumption B.1 and m ∈ N is chosen such that p 2 ξ 2 m +1 ≤ γ ( pλ/n ) . Then for any D > 0 , there exists a constant C ∗ ,D > 0 such that with probability at least 1 -p -D ,

This lemma follows directly from [Misiakiewicz and Saeed, 2024, Proposition 9].

Effective regularization and fixed points. Conditional on F , the bias and variance are functionals of the random matrix Z which has n i.i.d. rows with regularization parameter λ and covariance ̂ Σ F . The deterministic approximations to these functionals depend on an 'effective regularization' ˜ ν 1 associated to ( n, ̂ Σ F , λ ) (see Definition 2 in Appendix A for background) which is given as the unique non-negative solution to the equation

Note that the right-hand side can be rewritten as

which is itself of functional of the random matrix F which has p i.i.d. rows with regularization parameter p ˜ ν 1 and covariance Σ . Note that ˜ ν 1 is a random variable depending itself on F . However we will show that it concentrates on a deterministic value ν 1 . We therefore introduce a second effective regularization ν 2 associated to ( p, Σ , pν 1 ) given as the unique non-negatice solution of the equation

The functional (75) is then well approximated by

This motivates to define ( ν 1 , ν 2 ) ∈ R 2 > 0 as the unique non-negative solutions to the coupled fixed point equations

Writing ν 1 as a function of ν 2 indeed produces the equations stated in Definition 1.

To show that ˜ ν 1 concentrates on ν 1 , we define the following fixed points (˜ ν 1 , ˜ ν 2 ) ∈ R 2 > 0 to be the unique positive solutions to the random equations

The following proposition shows that (˜ ν 1 , ˜ ν 2 ) is well approximated by ( ν 1 , ν 2 ) with high probability.

Proposition B.4 (Concentration of the fixed points) . Assume that ( f j ) j ∈ [ p ] satisfy Assumption B.1. Then for any D,K > 0 , there exist constants η ∗ ∈ (0 , 1 / 4) and C ∗ ,D,K > 0 such that the following holds. Let ρ κ ( p ) and ˜ ρ κ ( n, p ) be defined as per Eqs. (26) and (25) , and γ λ and γ + as per Eq. (74) . For any p ≥ C ∗ ,D,K and λ > 0 , if it holds that

then with probability at least 1 -p -D , we have

The proof of this proposition can be found in Section B.7.2.

To study functionals of Z conditional on F , we will assume that F belonds to the good event

where ˜ A F is defined in Lemma B.2. In particular, as long as p ≥ C ∗ ,D,K and condition (78) hold, then P ( A F ) ≥ 1 -2 p -D by Lemma B.2 and Proposition B.4.

where we defined

Truncated fixed point. As mentioned above, we will separate the analysis of the low-degree part of the feature matrix F 0 and the high-degree part F + . For the high-degree part, we will simply use the concentration stated in Lemma B.3. For the low degree part, we will study functional of F 0 with regularization γ + = pν 1 +Tr( Σ + ) . We therefore introduce an effective regularization ν 2 , 0 associated to the model ( p, Σ 0 , γ + ) , i.e., the unique positive solution to the equation

Intuitively, ν 2 , 0 will be closed to ν 2 as soon as λ max ( Σ + ) ≪ ν 2 as

and uniqueness of the positive solution ν 2 , 0 . This is formalized in the following lemma.

Lemma B.5 (Truncated fixed point) . Let m be chosen such that p 2 ξ 2 m +1 ≤ γ λ , then

Furthermore, there exists an absolute constant C > 0 such that

Proof of Lemma B.5. The first bound (81) follows directly from [Misiakiewicz and Saeed, 2024, Lemma 6]. For the second inequality, we decompose this difference into

where we used that ξ 2 m +1 /ν 2 , 0 ≤ γ + / ( p 2 ν 2 , 0 ) ≤ 1 /p by the assumption on m and identity (80).

B.3 Deterministic equivalents for functionals of Z conditional on F

As mentioned in Section B.1, we will first show that the test error concentrates over Z conditional on F on some quantity that only depends on ̂ Σ F . The bias and variance terms can be written in terms of the following three functionals of the feature matrix Z : for a general p.s.d. matrix A ∈ R p × p and positive scalar κ > 0 , define

where we recall that Z has i.i.d. rows with covariance ̂ Σ F = FF T /p . We show that these functional are well approximated by functionals of F that can be written in terms of the following functionals:

The following proposition gather the approximation guarantees for Φ 2 , Φ 3 , Φ 4 listed in Eq. (83).

Proposition B.6 (Deterministic equivalents for Φ( Z ) conditional on F ) . Under Assumption B.1 and assuming that F ∈ A F defined in Eq. (79) , for any D,K > 0 , there exist constants η ∗ ∈ (0 , 1 / 4) , C D,K > 0 , and C ∗ ,D,K > 0 such that the followings holds. Let ρ κ ( p ) and ˜ ρ κ ( n, p ) be defined as per Eqs. (26) and (25) . For any n ≥ C D,K and regularization parameter λ > 0 , if it holds that

then for any p.s.d. matrix A ∈ R p × p (independent of Z | F ), with probability at least 1 -n -D on Z conditional on F , we have

where the approximation rate is given by

Proposition B.6 is a consequence of [Misiakiewicz and Saeed, 2024, Theorem 4] (see Appendix A and Theorem A.2 for background) and Proposition B.4. We defer its proof to Section B.7.3. Note that the term ˜ O ( p -1 / 2 ) in the approximation rate E 1 ( n, p ) defined in Eq. (89) comes from comparing pν 1 with p ˜ ν 1 and is equal to ˜ ρ λ ( n, p ) · E ν ( n, p ) where E ν ( n, p ) is defined in Proposition B.4. If instead, we compared to functionals with regularization p ˜ ν 1 , then the approximation rate in Proposition B.6 would scale ˜ O ( n -1 / 2 ) as expected.

In the analysis of the bias term, we will further need to show deterministic equivalents in the case where A is itself a random matrix uncorrelated (but not independent) to Z | F . The following proposition gather these approximation guarantees and is a consequence of [Misiakiewicz and Saeed, 2024, Lemma 10] and Proposition B.4.

Proposition B.7 (Deterministic equivalents for Φ( Z ) , uncorrelated numerator) . Assume the same setting as Proposition B.6 and the same conditions (85) . Consider a deterministic vector v ∈ R p and a random vector u = ( u i ) i ∈ [ n ] with i.i.d. entries and E [ u i ] = 0 , E [ u 2 i ] = 1 , and E [ z i u i | F ] = 0 . Then with probability at least 1 -n -D on Z conditional on F , we have

where we denoted v := ̂ Σ -1 / 2 F v and the approximation rate is given by

We defer the proof of Proposition B.7 to Section B.7.3.

B.4 Deterministic equivalents for functionals of F

After replacing the bias and variance terms by their deterministic equivalents over the randomness in Z | F , we obtain functionals in terms of ˜ Φ 2 ( F ; pν 1 ) and ˜ Φ 5 ( F ; A , pν 1 ) listed in Eq. (84). As mentioned in Section B.2, we will analyze the low-degree and high degree part of the feature matrix separately. Using Lemma B.3, we can replace the high-degree part F + F T + by a deterministic matrix, which results in a regularization parameter γ ( pν 1 ) = pν 1 +Tr( Σ + ) .

The functionals of F 0 can be written in terms of the following quantities: for any deterministic matrix B ∈ R m × m , define

We show that these functionals can be well approximated by the deterministic functions that can be written in terms of

Recall that for κ = pν 1 , we denote γ + := γ ( pν 1 ) and ν 2 , 0 the effective regularization associated to model ( p, Σ 0 , γ + ) . The following proposition gather the approximation guarantees for ˜ Φ 1 , ˜ Φ 2 , ˜ Φ 3 , ˜ Φ 4 listed in Eq. (93).

Proposition B.8 (Deterministic equivalents for F 0 ) . Under Assumption B.1, for any D,K > 0 , there exist constants η ∗ ∈ (0 , 1 / 4) , C D,K > 0 , and C ∗ ,D,K > 0 such that the followings holds. Let ρ κ ( p ) be defined as per Eq. (26) . For any p ≥ C D,K and λ > 0 , if it holds that

then for any deterministic p.s.d. matrix B ∈ R m × m , with probability at least 1 -p -D , we have

where the approximation rate is given by

This proposition is obtained by directly applying Theorem A.2 with no modifications.

Again, in the analysis of the bias term, because we separated the analysis of the low-degree and highdegree parts F 0 and F + , we will further need deterministic equivalents when B is itself a random matrix uncorrelated but not independent to F . We gather the associated deterministic equivalents in the following proposition.

Proposition B.9 (Deterministic equivalents for F 0 , uncorrelated numerator) . Assume the same setting as Proposition B.8 and the same conditions (95) . Consider a deterministic vector v ∈ R m and a random vector u = ( u j ) j ∈ [ p ] with i.i.d. entries and E [ u j ] = 0 , E [ u 2 j ] = 1 , and E [ f 0 ,j u j ] = 0 .

Then with probability at least 1 -p -D , we have

where we denoted v := Σ -1 / 2 0 v and the approximation rate is given by

This proposition is a direct consequence of [Misiakiewicz and Saeed, 2024, Lemma 10].

Throughout the proofs, with a slight abuse of notations, we will denote functionals ˜ Φ i ( F 0 ; B , κ ) , i ∈ [4] , the functionals listed in Eq. (93) applied to the truncated feature matrix F 0 , with covariance Σ 0 , regularization γ ( κ ) , and deterministic matrix B ∈ R m × m , and ˜ Φ i ( F ; B , κ ) , i ∈ [4] , the functionals applied to the full feature matrix F ∈ R p ×∞ , where Σ 0 is replaced by Σ , the regularization parameter is κ , and the deterministic matrix B ∈ R ∞×∞ . Similarly, we will us the notation Ψ i ( ν 2 , 0 ; B ) , i ∈ [3] for the truncated deterministic functionals (94), and the notation Ψ i ( ν 2 ; B ) , i ∈ [3] to denote the full functionals with Σ 0 replaced by Σ and B ∈ R ∞×∞ .

B.5 Approximation guarantee for the variance term

Recall the expressions for the variance term

and its associated deterministic equivalent

We prove in this section an approximation guarantee between V ( G , F , λ ) and V n,p ( λ ) . For convenience, we state a separate theorem for this term.

Theorem B.10 (Deterministic equivalent for the variance term) . Assume the features ( z i ) i ∈ [ n ] and ( f j ) j ∈ [ p ] satisfy Assumption B.1, and the covariance Σ and target coefficients β ∗ satisfy Assumption 3.2. Then, for any D,K > 0 , there exist constants η ∗ ∈ (0 , 1 / 2) and C ∗ ,D,K > 0 such that the following holds. Let ρ κ and ˜ ρ κ be defined as per Eqs. (26) and (25) . For any n, p ≥ C ∗ ,D,K and λ > 0 , if it holds that

then with probability at least 1 -n -D -p -D , we have

where the approximation rate is given by

Proof of Theorem B.10. First, note that V ( G , F , λ ) can be written in terms of the functional Φ 4 defined in Eq. (83):

Recall that A F is the event defined in Eq. (79). Under the assumptions of Theorem B.10, we can apply Lemma B.2 and Proposition B.4 to obtain

Hence, applying Proposition B.6 for F ∈ A F and via union bound, we obtain that with probability at least 1 -p -D -n -D ,

where E 1 ( n, p ) is defined in Eq. (89) and we recall the expressions

Let us decompose ˜ Φ 6 ( F ; I , pν 1 ) into

From Lemma B.11 stated below, we have with probability at least 1 -p -D

where E 3 ( p ) is the approximation rate defined in Eq. (100). Note that

Combining the above displays, we deduce that

Using Eq. (120) in Lemma B.14 stated in Section B.7, we conclude that with probability at least 1 -p -D ,

Finally, by simple algebra, we have

Note that by Eq. (119) in Lemma B.14, we have

Hence rearranging the terms in Eq. (110) and using from conditions (106) and that p ≥ C ∗ ,D,K , we obtain with probability at least 1 -p -D that

Using an union bound and combining bounds Eqs. (108) and (111), we obtain with probability at least 1 -n -D -p -D , that

where we used Eq. (111) and conditions (106) in the last line. Replacing the rates E j by their expressions conclude the proof of this theorem.

Lemma B.11. Under the setting of Theorem B.10 and assuming the same conditions (106) , we have with probability at least 1 -p -D ,

where E 3 ( p ) is the approximation rate defined in Eq. (100) .

The proof of this lemma can be found in Section B.7.4.

B.6 Approximation guarantee for the bias term

Recall the expression for the bias term

and its associated deterministic equivalent

We prove in this section an approximation guarantee between B ( β ∗ ; G , F , λ ) and B n,p ( β ∗ , λ ) . For convenience, we state a separate theorem for this term.

Theorem B.12 (Deterministic equivalent for the bias term) . Assume the features ( z i ) i ∈ [ n ] and ( f j ) j ∈ [ p ] satisfy Assumption B.1, and that there exists m ∈ N such that p 2 ξ 2 m ≤ γ m ( nλ/p ) . Then, for any D,K > 0 , there exist constants η ∗ ∈ (0 , 1 / 2) and C ∗ ,D,K > 0 such that the following holds. Let ρ κ and ˜ ρ κ be defined as per Eqs. (26) and (25) , and recall that γ λ := γ m ( nλ/p ) and γ + := γ m ( pν 1 ) . For any n, p ≥ C ∗ ,D,K and λ > 0 , if it holds that

then with probability at least 1 -n -D -p -D , we have

where the approximation rate is given by

Before starting the proof, let us introduce some notations. First, define P F the projection onto the span of F , and P ⊥ , F the projection orthogonal to the span of F , i.e.,

We can decompose the feature g with respect to the orthogonal sum of these two subspaces

We define r := P ⊥ , F g and R = [ r 1 , . . . , r n ] T ∈ R n ×∞ . Similarly, we can decompose the target function

where we introduced

Note that in particular E [ z ⟨ r , β ⊥ , F ⟩ ] = 0 by orthogonality.

Proof of Theorem B.12. Step 0: Decomposing the bias term.

Note that using the notations introduced above, we can decompose the bias term into

where we denoted

We proceed similarly to the proof for the variance term, by first considering the deterministic equivalent over Z conditional on F , and then over F . We omit some repetitive details for the sake of brevity.

Step 1: Deterministic equivalent over Z conditional on F .

First note that, denoting ˜ A ∗ = ̂ Σ -1 / 2 F β F β T F ̂ Σ -1 / 2 F , we have

Furthermore, T 3 and T 2 correspond respectively to the terms (90) and (91) in Proposition B.7 with v = β F and u = Rβ ⊥ , F where

Thus, under the assumptions of Theorem B.12, we can apply Propositions B.8 and B.7 to obtain (via union bound) that with probability at least 1 -n -D -p -D ,

Hence we deduce that

Let us simplify these terms. Recall that

For the term involving ˜ A ∗ , we can rewrite it as

Hence the terms involving ∥ β ⊥ , F ∥ 2 2 cancel out and we obtain

Combining the above displays, we deduce that with probability at least 1 -n -D -p -D ,

Step 2: Deterministic equivalents over F .

Following the same steps as Eq. (110) in the proof of Theorem B.10, we have with probability at least 1 -p -D ,

Furthermore, from Lemma B.13 stated below, with probability at least 1 -p -D ,

where ˜ B n,p ( β ∗ , λ ) is defined in Eq. (118). Combining these two bounds and recalling conditions (114), we obtain

Noting that B n,p ( β ∗ , λ ) = ˜ B n,p ( β ∗ , λ ) / (1 -Υ( ν 1 , ν 2 )) , we can combine this bound with Eq. (116) to obtain via union bound that with probability at least 1 -n -D -p -D ,

Replacing the rates E j by their expressions conclude the proof of this theorem.

Lemma B.13. Under the setting of Theorem B.12 and assuming the same conditions (114) , we have with probability at least 1 -p -D ,

The proof of Lemma B.13 can be found in Section B.7.5.

B.7 Technical results

In this section, we prove the technical results that were deferred from the previous sections. We start with a lemma that gathers useful bounds on the deterministic functionals.

Lemma B.14. There exists a constant C ∗ > 0 such that

where we recall that Υ( ν 1 , ν 2 ) is defined as per Eq. (105) . Furthermore, under Assumption 3.2, we have

where ˜ B n,p ( β ∗ , λ ) is defined as per Eq. (118) in Lemma B.13.

Proof of Lemma B.14. Step 1: Equation (119) .

Note that we have

In particular, if n ≥ p/η ∗ , then we can simply write

For n ≤ p/η ∗ , note that using the first identity in Eq. (76), we have

Combining the previous two displays, we obtain Eq. (119).

Step 2: Equation (120) .

We rewrite the left-hand side as

where we uses the second of the identities (76) in the second line, and Assumption 3.2 in the third line. Hence,

Step 3: Equation (121) .

Similarly, we get again using Eq. (76) and Assumption 3.2 that

and therefore

which concludes the proof of this lemma.

B.7.1 Feature covariance matrix

Proof of Lemma B.2. Recall that we consider ̂ Σ F = p -1 FF T , with F = [ f 1 , . . . , f p ] T ∈ R p ×∞ and the f i are i.i.d. random vectors satisfying Assumption B.1. For any integers k 2 ≥ k 1 ≥ 1 , we split the weight feature matrix into F = [ F 1 , F 2 , F 3 ] , where F 1 = [ f 1 , 1 , . . . , f 1 ,p ] T ∈ R p × k 1 with f 1 ,j the first k 1 coordinates of the feature vector f j , F 2 = [ f 2 , 1 , . . . , f 2 ,p ] T ∈ R p × ( k 2 -k 1 ) with f 2 ,j the next k 2 -k 1 coordinates of f j , and F 3 = [ f 3 , 1 , . . . , f 3 ,p ] T ∈ R p ×∞ contains the rest of the coordinates. In other words, we split the feature vector into f j = [ f 1 ,j , f 2 ,j , f 3 ,j ] . Denote

We decompose the feature covariance matrix into

Step 1: Bounding the eigenvalues of F 1 and F 2 .

Introduce the whitened matrices

so that the feature vectors f 1 ,j and f 2 ,j have covariance I k 1 and I k 2 -k 1 respectively. We have

Denote Z j := ⟨ v , f 1 ,j ⟩ 2 -1 . Then we have from Equation (60) applied to B = Σ -1 / 2 1 vv T Σ -1 / 2 1 , for any integer q ≥ 1 ,

Hence we can apply Bernstein's inequality for centered sub-exponential random variable and obtain

Following the proof of [Vershynin, 2010, Theorem 5.39], we deduce that there exist constants C ∗ , C ∗ ,D > 0 such that with probability at least 1 -p -D ,

In particular, there exists η ∗ ∈ (0 , 1 / 4) and C ∗ ,D > 0 such that for p ≥ C ∗ ,D and via union bound, we have with probability at least 1 -p -D (reparametrizing D ), that for any k 1 ≤ ⌊ η ∗ · p ⌋ ,

From Eq. (122), we deduce that the k 1 -th eigenvalue of ̂ Σ F for k 1 < ⌊ η ∗ · p ⌋ is lower bounded by

and

Similarly for F 2 , we have with probability at least 1 -p -D that for any k 1 < k 2 ≤ ⌊ η ∗ · p ⌋ ,

and therefore

Step 2: Bounding the eigenvalues of F .

Equation (124) provides a lower bound on the eigenvalues ˆ ξ 2 k of ̂ Σ F up to k < ⌊ η ∗ · p ⌋ . Let's upper bound the p eigenvalues of ̂ Σ F . From now on, set k 2 = p ∗ -1 where p ∗ := ⌊ η ∗ · p ⌋ .

For the contribution of ∥ F + ∥ , we use the matrix Bernstein's inequality as in [Misiakiewicz and Saeed, 2024, Lemma 1] (see proof of Theorem A.2 in Appendix A) and obtain that for p ≥ C K , with probability at least 1 -p -D ,

By the min-max theorem, we have with probability at least 1 -p -D for all k 1 < p ∗ -1 ,

where we used Eq. (125). For k ≥ p ∗ , we simply use that

We deduce from the above two displays that with probability at least 1 -p -D , we have for any k ≤ p

where we recall that we defined

Step 3: Bounding r ̂ Σ F ( k ) for k ≤ p .

Recall that the intrinsic dimension r ̂ Σ F ( k ) at level k ≤ p is given by

First note that for p ≥ k ≥ ⌊ η ∗ · p ⌋ , we simply use that and the eigenvalues are nonincreasing to get

For k ≤ p ∗ -1 , we use that from Eq. (124) with probability at least 1 -p -D ,

Let's bound the second term on the right-hand side. Notice that

where the minimization is over V ∈ R p × ( ⌊ η ∗ · p ⌋-1) with V T V = I ⌊ η ∗ · p ⌋-1 and the second inequality is obtained by taking V orthogonal to the matrix [ F 1 , F 2 ] . We can rewrite

Introduce Z j := ∥ f 3 ,j ∥ 2 2 -Tr( Σ 3 ) . By Assumption B.1 with B = diag( 0 , I ) (identity on the subspace Σ 3 and 0 otherwise), we have

We therefore we can apply Bernstein's inequality (123) again and we get

Hence, for any p ≥ C ∗ ,D with probability at least 1 -p -D ,

Combining the above display with the previous inequalities yields with probability at least 1 -p -D that for any k ≤ ⌊ η ∗ · p ⌋ ,

We deduce that there exist a constant C ∗ > 0 such that with probability at least 1 -p -D , we have for any n ∗ := ⌊ η ∗ · n ⌋ ≤ p

where r Σ ( n ) is the effective rank associated to Σ .

Step 4: Concluding the proof.

For any n ∗ = ⌊ η ∗ · n ⌋ > p , we have ̂ ρ λ ( n ) = ˜ ρ λ ( n, p ) = 1 . Hence, we only need to consider the case n ∗ ≤ p . Using Eqs. (126) and (127), we get

which concludes the proof.

B.7.2 Concentration of the fixed points

Proof of Proposition B.4. Recall that (˜ ν 1 , ˜ ν 2 ) ∈ R 2 > 0 are the unique solution to the random fixed point equations (77). From the first equation, we have the following bounds on ˜ ν 1 :

From Lemma B.2, we have with probability at least 1 -p -D that ∥ ̂ Σ F ∥ op ≤ C ∗ ,D,K ≤ p . Hence, by the uniform concentration over κ ∈ [ pλ/n, ( p 3 + pλ ) /n ] in Lemma B.15 stated below and an union bound, we deduce that with probability at least 1 -p -D ,

Therefore, we can rewrite the fixed equations (77) as

where with probability at least 1 -p -D , we have | δ ( F ) | ≤ ˜ E ( p ) . Therefore, by condition (78) and p ≥ C ∗ ,D,K

and we can directly use Lemma B.17 stated below to obtain with probability at least 1 -p -D ,

This concludes the proof of this proposition.

For any κ ≥ 0 , denote ν 2 ( κ ) ∈ R > 0 the unique positive solution to

We will further define analogously to Eq. (80) the truncated fixed point

where we recall that we denoted γ ( κ ) = κ +Tr( Σ + ) . It will be convenient to recall the notations

and the deterministic equivalents

The next lemma show that ˜ Φ 2 ( F , κ ) concentrates on Ψ 2 ( ν 2 ( κ )) uniformly on an interval of κ .

Lemma B.15. Under the setting of Proposition B.4 and for any D,K ≥ 0 , there exist constants η ∗ ∈ (0 , 1 / 4) and C ∗ ,D,K > 0 such that the following holds. For any p ≥ C ∗ ,D,K and λ > 0 , if it holds that

then with probability at least 1 -p -D , we have for any κ ∈ [ pλ/n, p ( p 2 + λ ) /n ] ,

Proof of Lemma B.15. Throughout the proof we assume that we are working on the event

which happens with probability at least 1 -p -D by Lemma B.3 via union bound. Note that γ ( κ ) ≥ γ ( pλ/n ) for all the κ ∈ [ pλ/n, p ( p 2 + λ ) /n ] . Furthermore, we assume that p ≥ C ∗ ,D chosen large enough so that ∥ F + F T + -γ (0) I p ∥ op ≤ 1 / 2 · γ ( pλ/n ) so that

The proof will proceed via a standard union bound argument over κ in the interval [ pλ/n, p ( p 2 + λ ) /n ] . Let us first prove Eq. (131) for a fixed κ . We first simplify the functional by rewriting it as

where we denoted

Using again the above identity, we rewrite

We can now apply Eq. (55) in Theorem A.2: under the assumption of Proposition B.4 and recalling the conditions (130), we have with probability at least 1 -p -D ,

Combining the above displays, we obtain that with probability at least 1 -p -D

where we used in the first inequality identity (129) to get

Further note that using condition (130), we can simplify the right-hand side

Wecan now use Eq. (82) in Lemma B.5 applied to ν 2 ( κ ) and ν 2 , 0 ( κ ) to concluded that with probability at least 1 -p -D ,

We now consider a p -P -grid P n of the interval κ ∈ [ pλ/n, p ( p 2 + λ ) /n ] , which contains at most p P +3 points. We can use a union bound over κ ∈ P n and reparametrizing D ′ = D + P +3 , so that with probability at least 1 -p -D , Equation (132) holds for any κ ∈ P n . Then for any point κ 1 ∈ [ pλ/n, p ( Kp 2 + λ ) /n ] , denote κ 0 ∈ P n its closest point. Then by Lemma B.16 stated below, we have

where we used that | κ -κ | ≤ p . Taking P = CK +1 concludes the proof.

1 0 -P

Lemma B.16. Under the setting of Proposition B.4 and for any D,K ≥ 0 , there exist constants η ∗ ∈ (0 , 1 / 4) , C > 0 and C ∗ ,D > 0 such that the following holds. For any p ≥ 1 and λ > 0 , it holds that

then for any κ 0 , κ 1 ≥ pλ/n , we have

Furthermore, if p ≥ C ∗ ,D , then we have with probability 1 -p -D that for any κ 1 , κ 2 ≥ pλ/n ,

Proof of Lemma B.16. Using the identity (128), we can decompose the first difference into

From condition (133) and the proof of [Misiakiewicz and Saeed, 2024, Lemma 11], we have

Combining the above inequalities, we deduce that there exists a constant C > 0 such that

For the second inequality (135), we rewrite the difference as

Hence, recalling Lemma B.3 and condition (133), for any p ≥ C ∗ ,D , we get with probability at least 1 -p -D and for all κ 1 , κ 2 ≥ pλ/n ,

which concludes the proof of this lemma.

We consider ( ν ε 1 , ν ε 2 ) ∈ R ≥ 0 the unique positive solutions of the perturbed equations:

Lemma B.17. Let η ∗ ∈ (0 , 1 / 4) be chosen as in Proposition B.4. Then there exists C ∗ , C ′ ∗ > 0 such that for any ε ∈ R with

then

Proof of Lemma B.17. For convenience, we introduce the notations

We first consider the second equation (137) and subtract the identities for ( ν ε 1 , ν ε 2 ) and ( ν 0 1 , ν 0 2 ) to obtain

Hence, we obtain the first identity

We now turn to the first equation (136): we obtain similarly

Hence, rearranging the term and recalling the first identity (136), we get the second identity

From this identity, we directly have

Note that for n ≥ p/η ∗ , we simply have by Eq. (136) that

where we use that Tr( Σ ( Σ + ν 0 2 ) -1 ) ≤ p by Eq. (137). For n ≤ p/η ∗ , let's denote µ 0 1 = λ/ν 0 1 . Rewriting Eq. (136), we get that

Hence

where we use in the last inequality that ν 0 1 /ν 0 2 ≤ 1 and the definition of the effective rank. Rearranging the terms we obtain

Combining the above bounds we deduce that

By assumption C ∗ · ˜ ρ λ ( n, p ) · | ε | ≤ 1 / 2 and therefore ν ε 1 ≤ 2 ν 0 1 . We conclude the first inequality

Recalling Eq. (138), we have directly

which concludes the proof.

B.7.3 Proof of deterministic equivalents for functionals on Z

Proof of Proposition B.6. Recall that ̂ ρ λ ( n ) is defined in Eq. (72) and that for F ∈ A F , we have ̂ ρ λ ( n ) ≤ C ∗ ,D,K ˜ ρ λ ( n, p ) for all n ∈ N . Under the assumptions of Proposition B.6, we can apply Theorem A.2 to Z conditional on F to obtain that with probability at least 1 -n -D ,

where ˜ ν 1 is the solution of the fixed point equation (77). We conclude the proof using Lemma B.18 stated below and that by condition (85), we have C ∗ ,D,K · ˜ ρ λ ( n, p ) E ν ( p ) ≤ C ∗ ,D,K K .

Proof of Proposition B.7. Again, under the assumptions of the proposition and for F ∈ A F , we can apply [Misiakiewicz and Saeed, 2024, Lemma 10] to get

We conclude by combining these inequalities with Lemma B.18.

and

Proof of Lemma B.18. For convenience, we denote κ := ν 1 , κ ′ := ˜ ν 1 , and Γ := ̂ Σ F . For Eq. (140), we simply use that

Similarly, by simple algebra, we have

Furthermore, note that by the identity (77),

Furthermore, we have from the previous computation and simple algebra that

where we used the same argument as in the proof of Lemma B.17 in the last inequality as well as condition (85). The proof of Eqs. (142) and (141) from simple algebra from the above displays and (143).

B.7.4 Proof of Lemma B.11

Proof of Lemma B.11. For convenience, we introduce the notations

where we recall that we denoted γ + = γ ( pν 1 ) = pν 1 +Tr( Σ + ) . Recall that for the first approximation guarantee, we denote

and the deterministic equivalents

For the second approximation guarantee, recall that we denote

and their associated deterministic equivalents

We separate the analysis of the low-degree part F 0 from the high-degree part F + . To remove the dependency on the high-degree part F + , we recall that by Lemma B.3, we have with probability at least 1 -p -D

where we denoted ∆ := F + F + -Tr( Σ + ) · I p and we recall that γ + = γ ( pν 1 ) = pν 1 +Tr( Σ + ) . In particular, taking p ≥ C ∗ ,D , we have ∥ ∆ ∥ op ≤ 1 2 γ + and therefore

Step 1: Bound on | ˜ Φ 2 ( F ; pν 1 ) -Ψ 2 ( ν 2 ) | .

First note that we have the identity

where we denoted Θ = Tr( G 0 ) -Tr( G F ) . By Eqs. (145) and (146), we have

Using again the above identity, we have

Under the assumption of the lemma, we can apply Proposition B.8 and obtain with probability at least 1 -p -D that

where E 3 ( p ) is defined in Eq. (100). Furthermore note that using identity (80), we have

and that by Eq. (82) in Lemma B.5,

Thus combining Eqs. (147), (149) and (150), we obtain

with probability at least 1 -p -D via union bound. Using condition (106), we can simplify the right-hand side using identity (148) and bounds (149) and (150) to get

This concludes the proof of the first part of this lemma.

We proceed similarly as in the first part and omit some repetitive details. First note that we can rewrite

where

Again, by Proposition B.8, we get that with probability at least 1 -p -D that

Furthermore, note that by Lemma B.19 stated below, we have

Combining Eqs. (151), (152) and (153), we deduce via union bound that with probability at least 1 -p -D

Let us simplify the right hand side. First, from the proof of Lemma B.19, we have

Combining the above two displays with E 3 ( p ) ≤ K from conditions (106) yields

Finally, note that using Eq. (152) and again E 3 ( p ) ≤ K that

which concludes the proof of this lemma.

Lemma B.19. Assuming that p 2 ξ 2 m ≤ γ + , we have

Proof of Lemma B.19. For convenience, we introduce

Using that

we obtain by simple algebra and using identity (80) that

Following the proof of [Misiakiewicz and Saeed, 2024, Lemma 7], we get that

Recalling Eq. (82) in Lemma B.5, we can conclude the proof using simple algebraic manipulations similarly to [Misiakiewicz and Saeed, 2024, Lemma 7].

B.7.5 Proof of Lemma B.13

Proof of Lemma B.13. We will follow similar steps as in the proof of Lemma B.11. For the sake of brevity, we omit some repetitive details. Recall that we introduced the notations (144). We decompose the coefficient vector β ∗ = [ β 0 , β + ] with β 0 ∈ R m the first m coordinates of β ∗ (aligned with the top m eigenspaces), while β + ∈ R ∞ corresponds to the rest of the coordinates. Further introduce the matrices

and recall the expressions of the functionals

as well as

We further recall the expressions of the deterministic equivalents

and

We first rewrite our functionals into

and study each term separately.

Step 1: Bounding term ˜ Φ 1 ( F ; A ∗ , pν 1 ) .

Let us start by removing the dependency on the high-degree part F + in the denominator. Note that we can rewrite the matrix R in block matrix form

so that

Let us study each of these terms separately. Denote ˜ G + := F + F T + + pν 1 . By simple algebra, we have where

Furthermore, from Proposition B.8, we have with probability at least 1 -p -D that

Hence, combining the previous two displays and using that E 3 ( p ) ≤ K by conditions (114), we have

Similarly, denoting ˜ G 0 = ( F 0 F T 0 + pν 1 ) -1 , we can rewrite the third term as

where with probability at least 1 -p -D ,

where we used the same concentration argument as in Step 3 of the proof of Lemma B.2. Finally, we rewrite

Hence, using Eqs. (154) and (155) as well as conditions (114), we obtain

Finally, note that by Lemma B.20 stated below, we have

Combining Eqs. (154), (155), (156), and (157), we deduce that with probability at least 1 -p -D ,

Step 2: Bounding term ˜ Φ 4 ( F ; A ∗ , pν 1 ) .

We rewrite this term as

where

Let us decompose

For the first term, we have directly from Proposition B.8 that

For the two other terms, notice that they correspond to the terms (101) and (102) in Proposition B.9 with v = β 0 and u = F + β + , where

Hence, by Proposition B.9, we have with probability at least 1 -p -D that

Furthermore, by Lemma B.20 stated below

(162) Combining Eqs. (154), (155), (156), and (157), we deduce that with probability at least 1 -p -D ,

where we used that

Step 3: Combining the terms.

From Eqs. (158) and (163), we have with probability at least 1 -p -D that

First, it is straightforward to verify that indeed

Then by Eq. (121) in Lemma B.14, we conclude that with probability at least 1 -p -D ,

which concludes the proof of this lemma.

Lemma B.20. Assuming that p 2 ξ 2 m ≤ γ + , we have

Proof of Lemma B.20. This lemma follows from the same arguments as in the proofs of Lemma B.5 and Lemma B.19.

C Details of the numerical illustration

In this section we provide further examples of comparison between the excess risk computed using the deterministic equivalent in Theorem 3.3 and numerical simulations, together with details about their realization. Results from numerical experiments are obtained averaging over 20-50 seeds. The data dimension is d = 100 , with the exception of experiments involving real data (see Appendix C.4) and the ones realized considering the Gaussian design described in Appendix C.2.

10 0

10 1

10 2

10 3

10

3

10

2

10

1

10 0

n

1/2

relative error

10 0

10 1

10 2

10 3

10

3

10

2

10

1

10 0

p

1/2

relative error

n

p

C.1 Self-consistent equations

To solve Equations 18 and 19 numerically, the following approach has been employed. From equation 19,

Substituting this expression in equation 18, we obtain

Therefore, the parameters ν 1 and ν 2 have been computed by iterating

until a chosen tolerance ϵ was reached.

C.2 Gaussian design

Figures 3, 4 and 9 have been realized considering Gaussian design for the vectors in 'feature space' defined in Appendix B.1. In particular, fixed Σ and β ∗ , we have drawn

and consequently { y i = β ⊤ ∗ g i } i ∈ [ n ] . Then the random feature estimator can be computed according to eqs. (61) and (62). In the figures produced with this setting, the elements of β ∗ and diag( Σ ) follow power-laws truncated at the component 10 4 .

C.3 Empirical diagonalization

Whenever the data probability distribution µ x or the weights distribution µ w are unknown (e.g. in all cases involving real data), we estimated the matrix Σ and the vector β ∗ following the procedure described in this section and summarized in Algorithm 1. Consider a data set of N covariates

{ x i } i ∈ [ N ] drawn from µ x and N noiseless labels { y i = f ∗ ( x i ) } i ∈ [ N ] , and a set of P weights { w j } j ∈ [ P ] . In Figures 1, 5, 6, 7, 8, for which this procedure was used, we take both N,P = 10 4 (with the exception of Fig. 7 (right), where P = 8000 ) and approximate µ x and µ w respectively with the empirical distributions ˜ µ x = ∑ N i =1 N -1 δ ( x -x i ) and ˜ µ w = ∑ P j =1 P -1 δ ( w -w j ) . Then eq. (2) becomes

and since eq. (11) implies E x K ( x , x ′ ) ψ k ( x ) = ξ 2 k ψ k ( x ′ ) , defining the Gram matrix K emp ∈ R N × N with elements ˜ K ii ′ = K ( x i , x i ′ ) N -1 and the vectors ˜ ψ k = ( ψ k ( x 1 ) , . . . , ψ k ( x N )) ⊤ , we can write the following eigenvalue problems, for k ∈ [ N ] :

We then constructed the matrix ˜ Σ = diag( ˜ ξ 2 1 , . . . , ˜ ξ 2 N ) and used it as an approximation of Σ . One should note that in this situation E x ψ k ( x ) ψ k ′ ( x ) = δ kk ′ corresponds to ˜ ψ k ˜ ψ k ′ = Nδ kk ′ . Similarly, eq. (13) implies β ∗ ,k = E x [ f ∗ ( x ) ψ k ( x )] , which can be approximated by

Algorithm 1 Empirical diagonalization

˜

K

ii

′

end for

{

( ˜

ξ

k

,

˜

ψ

for

k

k

)

k

(

NP

)

-

∈

∈ {

1

˜

β

k

[

N

]

, . . . , N

∑

1

P

j

=1

eig(

}

K

do

N

1

y

˜

ψ

-

⊤

k

end for

Iterate eqs. (167-168) up to tolerance ϵ

Compute the deterministic equivalent for the excess risk (22-24)

C.4 Real data

We performed numerical simulations sampling the training data from the MNIST data set Lecun et al. [1998] and the FashionMNIST data set Xiao et al. [2017], standardizing both covariates and labels, reshaping the images into vectors with d = 748 . Results are shown in Figures 1 (right) and 7 (left).

C.5 Trained network

In Figure 7 (right), we apply the procedure described in Appendix C.3 to the trained weights of a two layer neural network with hidden layer of size p

At initialization, the weights are randomly drawn; then, after sampling a training dataset X tr ∈ R n tr × d , y tr ∈ R n tr , the weights of the first layer are trained using gradient descent, iterating for t = 1 , . . . , T the following

where η is the learning rate, ⊙ is the Hadamard product, ˆ f and φ are applied component-wise; finally, the weights of the second layer are minimized using ridge regression, as in eq. (5).

˜

φ

)

(

x

,

w

)

φ

(

x

′

,

w

)

i

j

i

j

0

200

400

600

800

1000

1200

10

1

10 0

Excess Risk

0

200

400

600

800

1000

1200

=10 5

=10 4

=10 3

=10 2

=10 1

n

p

0

200

400

600

800

1000

1200

n

10

1

10 0

10 1

Excess Risk

0

200

400

600

800

1000

1200

p

=10 5

=10 4

=10 3

=10 2

=10 1

0

200

400

600

800

1000

p

10 0

10 1

Excess Risk

=10 5

=10 4

=10 3

=10 1

25

50

75

100

125

150

175

200

n

10

1

10 0

RF

1 GD step

2 GD steps

3 GD steps

0

200

400

600

800

1000

1200

n

0.4

0.6

0.8

1.0

1.2

Excess Risk

0

200

400

600

800

1000

1200

p

=10 5

=10 4

=10 3

=10 2

=10 1

D Derivation of the rates

In this appendix we present a sketch of the derivation of the decay rates for the excess error given in Section 4. We choose

and we assume which follows the source and capacity conditions stated in (37). In order to simplify the derivation of the rates, we introduce the following notation:

For s + α ( δ -γ ) < 1 and in the limit ν → 0 , this term can be written as a Riemann sum as follows

Otherwise, if s + α ( δ -γ ) > 1 , we can write:

Hence, for ν → 0 ,

Rewriting (18-19) as follows, we study the dependence of the positive parameters ν 1 and ν 2 with n :

. In the limit n →∞ , we can distinguish the following regimes

In ( a ) we have used the fact that, for ν 2 constant or diverging with n , T 0 11 ( ν 2 ) is respctively O (1) or infinitesimal, while we have that T 0 11 ( ν 2 ) is diverging in both cases. Hence, ν 2 must be infinitesimal,

allowing us to use (173). In conclusion, as n →∞ ,

D.1 Variance term

Considering the results (176-178), we can write eq. (20) as

One could notice, using the integral approximation of the Riemann sum T 0 22 ( ν 2 ) given in (173), that 1 -Υ( ν 1 , ν 2 ) = O (1) for any choice of ℓ and q . Hence, the variance term given by (23) decays with n with rate

D.2 Bias term

Using again (176-178), we can compute the rate of χ ( ν 2 ) defined in eq. (21), as n →∞ :

Using the integral approximation given in (173), one could verify that p -T 0 22 ( ν 2 ) = O ( p ) for any choice of ℓ and q .

The deterministic equivalent for the bias term, given in eq. (22), can be written as

where we have used (175) to compute the scalings of the terms T 1 δγ and the fact that 1 -Υ( ν 1 , ν 2 ) = O (1) .

From eq. (183), and using the result (177), it is straightforward to see that the decay rate of the bias term is given by

Examples of the results of Theorem 4.1 and Corollary 4.2 are shown in Fig. 3 and 9.

in particular

10 2

10 3

10 4

10 5

10

3

10

2

10

1

Excess Risk

=3.0,

r

=0.4,  =

* =0.88

q

=0.55

q

=0.45

q

=

q

* =0.294

q

=0.25

n

2

rq

(bias)

n

2

*

r

(optimal)

n

1+

q

(variance)

10 2

10 3

10 4

10 5

=3.0,

r

=0.4,

q

=

q

*  = 0.294

=

* =0.88

=1.5

=2.5

least squares

n

2

*

r

(optimal)

n

n

D.3 Details of Remark 4.1

In order to extend the results of Theorem 4.1 and Corollary 4.2 to the excess risk defined in (6), in this section we compute the intervals for ℓ and q such that the assumptions of Theorem 3.3 hold and the approximation rates E ( n, p ) are vanishing, under source and capacity conditions. Given n , p = n q , λ -( ℓ -1) and ν 2 as in (177) assumption 3.2 is verified for m = n q + ℓ and C ∗ = O ( ν -1 2 ) . In fact

and the inequality in (16) holds if

The inequalities in (17) can be written as

Then, introducing η ∗ ∈ (0 , 1 / 2 ) , we can compute the following quantities of interest (introduced in eqs. (25) to (27)):

Similarly, considering ν 1 scaling as in eq. (176), we have that eq. (31)

The last step is a consequence of

Fixing K > 0 and considering the , we consider condition (28):

˜ ρ λ ( n, p ) 5 / 2 · log 3 / 2 ( n ) ≤ K √ n ⇐ = ℓ > ✶ [ q ≥ 1] ( α + 1 5 ) , (201) and, similarly, condition (29) is satisfied if, for q ≥ 1

while, for q < 1 , if

where the last step is a consequence of

Finally, the approximation rate defined in remark 4.1 is

where the second term vanishes under the conditions in (203) and (206), and the first term vanishes by further assuming, for q ≥ 1

E Comparison with neural scaling laws

In this appendix we discuss the relationship between our results and the recent literature of the theory of neural scaling laws with linear models. We adopt a notation close to ours, with dictionary to their notation given in Table 1 and Table 2.

Bahri et al. [2024] and Maloney et al. [2022] have considered a model where with Gaussian input data and linear target function:

The covariance matrix Λ = diag( λ k ) k ∈ [ d ] is taken to be diagonal, with eigenvalues following a power-law scaling:

with α > 1 and the target weights are assumed to be random Gaussian vectors β ⋆ ∼ N (0 , 1 / d I d ) . In particular, note that Tr Λ ∼ d for d → ∞ . Given the training data, they consider least-squares regression in the class of linear random features predictor:

where W ∈ R p × d is a Gaussian random matrix elements in N (0 , 1 / d I d ) . 5

This setting is a particular case of the one introduced in Section 2. In particular, it satisfies particular source and capacity conditions eq. (38). To see this, note that the feature population covariance is identical to the input data covariance:

Therefore, we can identify Σ = 1 / d Λ which has Tr Σ < ∞ for α > 1 in the limit d →∞ . Therefore, the features satisfy a capacity condition with scaling α . Moreover, the asymptotic kernel is simply the linear kernel:

Since the target variance is constant, this is equivalent to a source condition with:

Since α ∈ (1 , ∞ ) , we are always in the hard regime r ∈ (0 , 1 / 2 ) where the target does not belong to the RKHS H = R d . Indeed, since β ⋆ ∼ N (0 , 1 / d I d ) , we have:

which indeed diverges as d →∞ . Moreover, note that least-squares regression correspond to the case ℓ = ∞ .

From the discussion above, the bias term scalings from Bahri et al. [2024] (resolution limited regime) and Maloney et al. [2022] (underparametrized regime n ≫ p , i.e. q ≪ 1 , and overparametrized regime n ≪ p , i.e. q ≫ 1 ), correspond to a vertical cross-section on the large ℓ region of Fig. 2 (Right). Indeed, we recover exactly the rate of the label term in eqs. (167)-(168) of Maloney et al. [2022]:

Similarly, it is possible to recover the rates for the noise term (first two results in eq. (86) of Maloney et al. [2022]) as the vertical cross-section on the large ℓ region of Figure 2 for the rates of the variance term. In particular:

Comparison with the SGD rates from Paquette et al. [2024], Lin et al. [2024] Furthermore, in the linear noiseless target setting, lifting the condition in eq. (217), it is possible to compare our results to the compute-optimal rates for the risk obtained through stochastic gradient descent in Paquette et al. [2024]. In particular, we consider unitary batch-size and the correspondence between the number of iterations of stochastic gradient descent and the number of samples n in ridge regression. Then, defining ˆ γ the decay rate of the compute-optimal curves for the risk ˆ R ≍ n -ˆ γ in Paquette et al. [2024], corresponding to the compute-optimal number of features p ≍ ˆ p =: n ˆ q , 6

coincides with γ B ( ℓ = 1 , ˆ q ) in Theorem 4.1, i.e. with fixed regularization parameter λ = 1 ( ℓ = 1 ). In particular, consider the following regions in the phase diagram provided in their work: 7

Phase Ia ( r < 1 / 2 ):

Phase II ( r > 1 / 2 and r < 1 -1 / 2 α < 1 ):

Phase III ( r > 1 / 2 and r > 1 -1 / 2 α ):

We emphasize that, in Phases Ia and II, ˆ γ = max q γ B (1 , q ) , while, in Phase III, ˆ γ ≤ max q γ B (1 , q ) . Hence, the compute-optimal decay rate of the risk for stochastic gradient descent is equal or smaller than the largest rate achievable by RFRR with fixed regularization λ = 1 and therefore always smaller than the optimal one in Corollary 4.2.

Asimilar setting has been investigated by the recent work Lin et al. [2024], providing scaling laws for the excess risk obtained by stochastic gradient descent with stepsize schedule η t = η/ 2 t log n / n , for t = 1 , ..., n . 8 Under the same source and capacity conditions, assuming r ∈ (0 , 1 / 2 ) and η = O (1) , the result in their Theorem 4.2 may be rephrased as follows:

Hence, choosing p ≍ n q and η ≍ n ℓ -1 , i.e. η ≍ λ -1 , provided η = O (1) = ⇒ ℓ < 1 , this result recovers precisely the same rates as in our Theorem 4.1.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: All claims in the abstract and introduction are supported by mathematical proofs or numerical results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: All mathematical proofs include clear assumptions reflecting the scope of applicability. Numerical results are replicated for different data sets and choices of feature maps.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Clear assumptions are provided in an 'assumption environment'.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have included a detailed discussion of how all the numerical experiments were conducted in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We judge the code is too simple to be released, and that we give enough information for the reproducibility of the numerical plots. All data sets used in the numerical experiments are either synthetic or open source.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have added a detailed discussion of the experiments in the captions and in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The numerical agreement between theory and experiments is so good that error bars are unnecessary.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: The experiments are simple illustrations of the theorems. They are simple enough to be ran on a standard laptop in a a few hours.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work is of theoretical nature, and therefore has no major ethical implications.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work is of theoretical nature, and therefore has no relevant societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work is of theoretical nature, and therefore has no risk of misusage.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All resources used from other works are properly acknowledged.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets are introduced in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This work does not involve crowdsourcing.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work is of theoretical nature and does not have potential risks to the people involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a non-asymptotic bound (Theorem 3.3) on the test error of random feature ridge regression (RFRR) using dimension-free (in the sense of the feature space) deterministic equivalents, which are the solutions to some self-consistent equations, under a concentration assumption (Assumption 3.1) on the features. As a result, they recover (Corollary 3.5) the deterministic equivalents result on the test error of kernel ridge regression (KRR) from previous literature, and they prove the exact decay rate of the neural scaling law (Corollary 4.1), filling the gap from previous literature.\n\nSTRENGTHS:\nThis paper improves the results from previous literature in multiple ways: non-asymptotic over asymptotic bounds [Loureiro2022], RFRR over KRR [Misiakiewicz2024], more precise neural scaling law [Rudi2017, Cui2022]. The phase diagram (Figure 2) is novel, as far as I know, and it offers great insights for the test error in RFRR. \n\nReference:\n- *Bruno Loureiro, Cédric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher student model. Journal of Statistical Mechanics: Theory and Experiment, 2022(11):114001, nov 2022.*\n- *Theodor Misiakiewicz and Basil Saeed. A non-asymptotic theory of kernel ridge regression: deterministic equivalents, test error, and gcv estimator, 2024.*\n- *Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.*\n- *Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime. Journal of Statistical Mechanics: Theory and Experiment, 2022(11):114004, nov 2022.*\n\nWEAKNESSES:\nMy main concern, however, is the significance of the theoretical contribution of this paper. From the first sight, this paper is merely an extension of the paper [Misiakiewicz2024] with same techniques of deterministic equivalents. \n\nAlso, the result holds only under the strong concentration Assumption 3.1. As mentioned in the paper (line 119-121), I think that this paper could have been more significant if Assumption 3.1 had been relaxed in this paper.\n\nQUESTIONS:\nRegarding the above comments, \n\n1. Could the authors explain more how different it is to implement the deterministic equivalents techniques in RFRR than in KRR?\n\n2. How difficult it would be to relax Assumption 3.1 as in [Misiakiewicz2024]? And what datasets and random feature model could satisfy Assumption 3.1 empirically?\n\n3. Despite not essential, but it could help the readers a lot if the authors could explain more on the quantities and their intuitions in the paper, like Eq (21-27). From my point of view, Theorem 3.3 is not the easiest to read.\n\nBesides the questions, I found some potential typos in the appendix:\n\n- A paragraph (line 536 - 540) seems misplaced: they should be directly after Eq (50). \n\n- The equation below line 587 should be $\\mathbf{f}\\_j = ((\\xi\\_k\\phi\\_k(\\mathbf{w}\\_j)))\\_{j\\geq1}$ instead of $\\mathbf{f}\\_j = ((\\xi\\_k\\psi\\_k(\\mathbf{w}\\_j)))\\_{j\\geq1}$. \n\n- a ``\\rangle`` is missing in the equation under line 590: it should be $\\sigma(\\langle \\mathbf{w}\\_p,\\mathbf{x}\\_i \\rangle)$ instead of  $\\sigma(\\langle \\mathbf{w}\\_p,\\mathbf{x}\\_i )$.\n\nLIMITATIONS:\nThis paper is theoretical paper and the authors have addressed the assumptions and conditions explicitly in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nEDIT: updated my score after rebuttal. \n\nThe authors investigate the excess risk in random feature ridge regression. They give a deterministic expression that approximates the excess risk, with a controlled relative error. The dependence of the deterministic \"equivalent\" w.r.t. to key quantities in the problem is examined.\n\nSTRENGTHS:\n* Understanding the generalization ability of simple models is important, especially as our intuition of generalization has been challenged by recent results on overparametrized neural networks. Therefore, the paper is potentially impactful. \n* The proposed result generalizes and refines a string of previous results in a single deterministic equivalent, and the dependence of this equivalent to problem parameters yields interesting conclusions, e.g. on the minimal number of features needed to achieve a statistically satisfying rate.\n* There is clearly a lot of work that has gone in obtaining such a general result.\n\nWEAKNESSES:\n## General comments\nMy main concern is the format of the paper, which I think is more suited to a mathematical-minded ML/stats journal than a conference. The main paper is very dense but misses key details such as proofs or proof outlines, while the full proof of the main result is 28 pages long, which makes the full paper 50 pages long (i.e., appendices included). It is not reasonable to expect reviewers to go through such a long technical proof in the context (short time span and heavy workload) of NeurIPS reviews. As a journal submission, I would have the time required to go through the proofs and get to the heart of the proposed (interesting) result. \n\n## Major\n1. Eqn 4: don't you want $\\epsilon$ to be zero-mean as well?\n2. p3 L104; \"we define wlog $\\mathcal{V} = Im(T)$\": if this is indeed a definition, I don't see why we need to add \"wlog\". If this is a statement about the image of $T$ being the whole of $\\mathcal{V}$, we need to define $\\mathcal{V}$ beforehand.\n3. Eqn 13 is only true in $L^2(\\mu_x)$ so I would avoid involving $\\mathbf{x}$ in the statement of the equation, which suggests you mean a pointwise equality and requires a quantifier. If you actually meant a pointwise equality, then this should be explained, and further assumptions should likely be made on $\\varphi$.\n4. Assumption 3.1. I am unsure what is meant by an infinite matrix $A$. Especially when one needs to talk about the trace of $\\Sigma A$, so that I assume we want to guarantee that $\\Sigma A$ is a trace-class operator. Can you rephrase the assumption in terms of operators? \nSimilarly, the Frobenius norm for operators should be defined. \n5. p4 L117: can you give more details on why cases 1) and 2) are covered by your assumption? Same for p4 L128: can you detail why these power decays satisfy Assumption 3.2?\n6. Definition 1: For easier reading, I would define $\\nu_1, \\nu_2$ first, then $\\Upsilon$, and then only $B$ and $R$. Otherwise, the reader has to wait until Eqn (24) to understand Eqn (18), which requires a lot of buffer memory from the reader's brain.\n7. p5 L156 is there an implicit dependence on the feature map dimension? Can you explain where? \n8. Figure 1: I would keep the caption short and descriptive, and move the definition of the data generating process to the main text. This would allow to explain more, for instance, what you mean by $v$ has a fixed overlap with the teacher vector\".\n9. The bibliography needs to be harmonized. There are many missing journal/conference names (if it's an arxiv preprint, say so and give the arxiv number), and a few initials mixed with full first names.\n10. p9 A short discussion section summarizing the main points and limitations of the paper would be a good addition.  \n\n## Minor\n* p1 L23: here and in the rest of the paper, you use the notation $\\mu_w(\\mathcal{W})$ to indicate that $\\mu_w$ is a probability measure on $\\mathcal{W}$. I would say that is not standard notation, and I would rather keep $\\mu_w(\\mathcal{W})$ for the measure of the whole space $\\mathcal{W}$, i.e. 1. \n* p1 L25: I think $\\sigma$ has not been defined yet.\n* p2 L46 \"demystifying phenomena such as double descent and benign overfitting\". I would give a reference for each concept.\n* p2 L55 no need to boldface \"our main contributions\".\n* Eqn 9: I would write $\\mathrm{Var}$ instead of $\\mathrm{Cov}$.\n* p4 L127: settings\n* p4 L135: what is a \"self-consistent\" equation?\n* p5 L142 \"we use $a_i = '*'$ to denote [...]\": I don't understand the exact meaning of this statement.\n* Eqn 30: I would remind the reader that $R_{n,p}$ has been defined in Definition 1.\n* p5 L151 \"in place\" reads strangely. \"In order\", maybe?\n* p5 L152 what does \"fully\" mean in \"fully non-asymptotic\"?\n* p6 L171 what do you mean by \"single-index\"?\n* p6 L183 span, not spam!\n* p6 L183 infinite-dimensional\n* p7 L216 understanding\n* p8 L255 decays\n* Figure 3 is too small to read.\n\nQUESTIONS:\n* Item 4 in my major comments.\n\nLIMITATIONS:\nThis is fundamental work and does not have any immediate potentially harmful impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

comments: SUMMARY:
Prior work on random feature (ridge) regression study the test error in the high-dimensional asymptotic. However, ideally one would hope for a non-asymptotic deterministic characterization of the test error.   In this paper, the authors tackle this problem and show that under a concentration assumption,  the test error is well approximated by a closed-form expression that only depends on the feature map eigenvalues. They use this result to study various problem in random features regression.

STRENGTHS:
- This paper rigorously solves an important problem in the analysis of random features regression. I think Theorem 3.3 will be of independent interest as well.
- The main result of the paper does not require the random regression coefficient assumption and hold for deterministic beta_star.
- Theorem 3.3 has a very clean form. The bounds in Theorem 3.3 are multiplicative. Thus, they scale correctly with the risk. Taking particular limits (e.g., p \to \infty or \lambda \to zero), we easily recover already known phenomenon.
-  The authors derive sharp excess error rates under power-law assumptions and provide a tight result on the smallest number of features needed to achieve optimal minimax rate.
- The proofs seem correct and rigorous.

All in all, I really enjoyed reading this paper and I recommend acceptance.

WEAKNESSES:
I suggest the authors expand the discussion around Assumption 3.1 and provide more detailed examples for which this assumption holds.

QUESTIONS:
The paper is very well written and I have no particular question.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studied the non-asymptotic generalization error for random feature ridge regression (RFRR) models. By considering the eigendecomposition of random features with respect to data distribution and weight distribution, the authors proved a feature-dimension-free deterministic equivalence for the generalization error of RFRR, where the sample size and number of features bound the approximation error. From this deterministic equivalence, fixing the number of data points, this paper presented the minimal number of features for the optimal decay rate of the excess risk of RFRR when considering power law assumptions for data covariance and target. This analysis provides a clear picture of the generalization error scaling law for source and capacity conditions of RFRR.\n\nSTRENGTHS:\n1. The paper is well-motivated, and the mathematics appears correct to me. The writing is clear, and the authors do a good job of presenting results with many discussions and comparisons of related works.  \n\n2. The authors provide various empirical simulations to justify the theorems, including random synthetic data, real-world data, random weights, and weights trained by gradient descents. This offers more insights into theoretic results and the generality of the results in this paper.\n\nWEAKNESSES:\n1. One concern is how we can check Assumptions 3.1 and 3.2 for specific nonlinear feature models with some concrete data and weight distributions. The results of this paper rely on the eigen decomposition in (11) but can we get some specific examples of eigenvectors $\\psi_k$ and $\\phi_k$? For instance, in a classical high-dim statistics setting, if we consider a nonlinear RFRR model with i.i.d. sub-Gaussian dataset and weight matrix and a certain nonlinear teacher model, can we justify Assumptions 3.1 and 3.2 in this case?\n\n2. There should be more clarification for the notations and conditions of the main results, Theorem 3.3. For instance, what would be the meaning of (25-27) and (28-29)? Are these bounds and approximation rates necessary or due to some technical reasons? If we apply the bounds in (28-29) to (32) for the error bound of the deterministic equivalence, the bound of $\\mathcal{E}(n,p)$ seems to be loose and cannot get $\\tilde O(n^{-1/2}+p^{-1/2})$.\n\nQUESTIONS:\n1.  In the simulation, the authors provide an empirical diagonalization method for estimating $\\Sigma$ and $\\beta$ by considering $N=P\\gg n,p$ and computing the eigendecomposition of the empirical feature covariance. Is there any theoretical guarantee for this approximation? This approximation seems to indicate that we can still apply the asymptotic results for empirical feature covariance under the proportional or polynomial scaling regime for $n, p$, e.g. [Mei and Montanari, 2022, Gerace et al., 2021, Dhifallah and Lu, 2020, Hu and Lu, 2023, Hu et al., 2024], to analyze the deterministic equivalence and the generalization errors. \n\n2. In (10), do you need to assume the subsets $\\mathcal{X}$ and $\\mathcal{W}$ in $\\mathbb{R}^d$ are compactly supported?\n\n3. Typo (7): $h_*\\to f_*$\n\n4. What is $\\nu_2$ in (17) in Assumption 3.2? You did not introduce this notation until Definition 1.\n\n5. You did not introduce intrinsic dimension around (25).\n\n6. In line 147, why do you assume assumption 3.2 again?\n\n7. Line 183, typo: spam; Line 230: (37) $\\to$ (38)?\n\n8. In Definition 3, typo in the definition $r_\\Gamma(k)$: $p \\to q$. Line 546, $\\Sigma\\to \\Gamma$.\n\n9. In Theorem A.2, what is the typical order of $\\rho_\\lambda (n)$? From (53), we cannot claim the error term in (54) will be vanishing.\n\n10. How do you prove the last equation on page 16, for the upper bound the $||S_i||_{op}$? Can you explain it? And in the same proof, how do you apply the matrix Bernstein inequality for infinite dimension $\\tilde S$?\n\n11. Below Line 587, typo in $f_j$: $j\\ge 1\\to k\\ge 1$; typo in (60): $A\\to B$.\n\n12. In Line 634, you consider $\\eta_*\\in (0,1/4)$ but in the main results, you have $\\eta_*\\in (0,1/2)$. Can you clarify it?\n\n13. How do you prove $||\\hat \\Sigma_F||_{op}\\ge 1/2$?\n\n14. Below Line 800 and in Line 1033, you use $G$ to denote the resolvent which has been used as the data feature matrix as well.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-28"}