Title: Learning Curves for Deep Structured Gaussian Feature Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: random feature models

generalization

deep networks

ridge regression
Abstract: In recent years, significant attention in deep learning theory has been devoted to analyzing when models that interpolate their training data can still generalize well to unseen examples. Many insights have been gained from studying models with multiple layers of Gaussian random features, for which one can compute precise generalization asymptotics. However, few works have considered the effect of weight anisotropy; most assume that the random features are generated using independent and identically distributed Gaussian weights, and allow only for structure in the input data. Here, we use the replica trick from statistical physics to derive learning curves for models with many layers of structured Gaussian features. We show that allowing correlations between the rows of the first layer of features can aid generalization, while structure in later layers is generally detrimental. Our results shed light on how weight structure affects generalization in a simple class of solvable models.

Learning Curves for Deep Structured Gaussian Feature Models

Abstract

In recent years, significant attention in deep learning theory has been devoted to analyzing when models that interpolate their training data can still generalize well to unseen examples. Many insights have been gained from studying models with multiple layers of Gaussian random features, for which one can compute precise generalization asymptotics. However, few works have considered the effect of weight anisotropy; most assume that the random features are generated using independent and identically distributed Gaussian weights, and allow only for structure in the input data. Here, we use the replica trick from statistical physics to derive learning curves for models with many layers of structured Gaussian features. We show that allowing correlations between the rows of the first layer of features can aid generalization, while structure in later layers is generally detrimental. Our results shed light on how weight structure affects generalization in a simple class of solvable models.

1 Introduction

Characterizing how data structure and model architecture affect generalization performance is among the foremost goals of deep learning theory [1, 2]. A fruitful line of inquiry has focused on the properties of a class of simplified models that are asymptotically solvable: neural networks in which only the readout layer is trained and other weights are random, which are known as random feature models (RFMs) [3-21]. Though RFMs cannot capture the effects of representation learning on generalization in richly-trained neural networks [13, 22, 23], they have substantially advanced our understanding of how data structure and model architecture interact to give rise to a wide array of generalization phenomena observed in deep learning [1-5, 7-19, 24, 25].

Of particular interest is the question of when models overfit benignly, that is, when they generalize well despite having been trained to perfectly interpolate their training data. Here, much intuition has been gained by studying minimum-norm kernel interpolation-that is, the ridgeless limit of kernel ridge regression-with RFM kernels, for which precise generalization asymptotics can be computed using tools from random matrix theory. These asymptotics lead to a precise picture of how the spectrum of the random feature kernel and the structure of the task interact to determine generalization. These analyses are facilitated by universality results, often termed Gaussian equivalence theorems, that state that the generalization error of a nonlinear RFM is asymptotically equal to that of a linear Gaussian model with an effective noise term resulting from nonlinearity [3, 7, 10, 25, 26]. In the past few years, Gaussian equivalence theorems for ever more general classes of RFMs have been established: within this year Schröder et al. [20] and Bosch et al. [21] have established Gaussian equivalence theorems

for deep nonlinear RFMs with unstructured feature weights, while Cui et al. [27] have extended some of these results to the setting of deep Bayesian neural networks when the target is of the same architecture.

However, these analyses consider the effect only of correlations in the data, and do not address the possibility of correlations between the random weights. It is standard to assume that the elements of the weight matrices at each layer are independent and identically distributed Gaussian random variables, and to our knowledge all existing Gaussian equivalence theorems make use of this assumption [3-15, 19-21]. As a result, how weight anisotropy affects generalization in deep RFMs-in particular, if it can affect the asymptotic scaling of generalization error with dataset size and network width [16, 19, 28]-remains unclear.

In this note, we take the first step towards filling that gap in our theoretical understanding of RFMs by computing the asymptotic generalization error of the simplest class of deep RFMs with anisotropic weight correlations: models with linear activations. Our primary contributions are as follows:

Using the replica method from statistical mechanics [29], we compute the asymptotic generalization error of deep linear random feature models with weights drawn from general matrix Gaussian distributions. This computation is closely related to prior replica approches to product random matrix problems [13, 30].

We show that, in the ridgeless limit, structure in the weights beyond the first layer is detrimental for generalization.

We next consider the special case of power-law spectra in the weights and in the data, which was classically studied in kernel interpolation in the form of source-capacity conditions [31], and has recently attracted substantial interest in deep learning due to approximate power-law spectra present in real data [16, 19, 28, 32]. Using approximations for required spectral statistics derived in past works [19], we show that altering the power laws of the weight covariance spectra do not affect the scaling laws of generalization.

We finally show how our results can be extended from the ridge regression estimator to the Bayesian Gibbs estimator, an object of classic study in the statistical physics of learning [13, 33, 34]. For sufficiently large prior variance, structure can be beneficial for generalization with this estimator.

Taken together, these results are consistent with the intuition that representation learning at only the first layer of a deep linear model is sufficient to recover a single teacher weight vector [13, 35-37].

2 Preliminaries

We consider depthL linear RFMs with input x ∈ R n 0 and scalar output given by

where the feature matrix F ∈ R n 0 × n L is fixed and the vector v ∈ R n L is trainable. If L = 0 , corresponding to standard linear regression, the feature matrix is simply the identity: F = I n 0 . If L > 0 , we take the feature matrix to be defined by a product of L factors U ℓ ∈ R n ℓ -1 × n ℓ :

We draw the random feature matrices independently from matrix Gaussian distributions

for input covariance matrices Γ ℓ ∈ R n ℓ -1 × n ℓ -1 and output covariance matrices Σ ℓ ∈ R n ℓ × n ℓ , such that E [( U ℓ ) ij ( U ℓ ′ ) i ′ j ′ ] = δ ℓℓ ′ (Γ ℓ ) ii ′ (Σ ℓ ) jj ′ . Subject to the constraints of layer-wise independence and separability-which are required for the factors to be matrix-Gaussian distributed-this is the most general covariance structure one could consider. One might wish to relax this to include non-separable covariance tensors E [( U ℓ ) ij ( U ℓ ′ ) i ′ j ′ ] = δ ℓℓ ′ ( χ ℓ ) ii ′ ,jj ′ , but this would spoil the matrixGaussianity of the factors, and to our knowledge does not appear to be addressable using standard

methods [30, 38]. We generate training datasets according to a structured Gaussian covariate model, with p i.i.d. training examples ( x µ , y µ ) generated as

where the teacher weight vector w ∗ is fixed and the label noise follows

We collect the covariates into a matrix X ∈ R p × n 0 , and the targets into a vector y ∈ R p .

As in most works on RFMs [3-5, 8-21, 25], our focus is on the ridge regression estimator

where the positive-definite matrix Γ L +1 ∈ R n L × n L controls the anisotropy of the norm and the ridge parameter λ > 0 sets the regularization strength. This minimization problem has the well-known closed form solution

As motivated in the Introduction, we are chiefly interested in the ridgeless limit λ ↓ 0 , in which the ridge regression solution gives the minimum ℓ 2 norm interpolant of the training data. We measure performance of this estimator by the generalization error

which is a random variable with distribution induced by the training data and feature weights.

This leads us to a simple, but important observation: including structured input-input covariances is equivalent to transforming the feature-feature covariances. We state this formally as:

Lemma 2.1. Fix sets of matrices { Γ ℓ } L +1 ℓ =1 and { Σ ℓ } L ℓ =0 , and a target vector w ∗ . Let ϵ p,n 0 ,...,n L be the resulting generalization error as defined in (8) . Let

Let ˜ ϵ p,n 0 ,...,n L be the generalization error for these transformed covariance matrices and target. Then, for any λ > 0 , we have the equality in distribution ϵ p,n 0 ,...,n L d = ˜ ϵ p,n 0 ,...,n L .

Proof of Lemma 2.1. As the features and data are Gaussian, we can write X d = Σ 1 / 2 0 Z 0 and U ℓ d = Γ 1 / 2 ℓ Z ℓ Σ 1 / 2 ℓ for unstructured Gaussian matrices ( Z ℓ ) ij ∼ i.i.d. N (0 , 1) . Substituting these representations into the ridge regression solution (7) and the generalization error (8), the claim follows.

Therefore, we may take Γ ℓ = I n ℓ -1 without loss of generality. Moreover, thanks to the rotationinvariance of the isotropic Gaussian factors Z ℓ , we may in fact take the remaining covariance matrices Σ ℓ to be diagonal without loss of generality, so long as we then express ˜ w ∗ in the basis of eigenvectors of Σ 0 . An important qualitative takeaway of this result is that changing the covariance matrix of the inputs of the first layer Γ 1 is equivalent to modifying the data covariance matrix, which was in a simpler form observed in the shallow setting ( L = 1 ) by Pandey et al. [39].

3 Asymptotic learning curves

Having defined the setting of our problem, we can define our concrete objective and state our main results, deferring their interpretation to the following section. We consider the standard proportional asymptotic limit

which we will refer to as the thermodynamic limit. Our goal is to compute the limiting generalization error:

where E D denotes expectation over all sources of quenched disorder in the problem, i.e., the training data and the random feature weights. In the thermodynamic limit, we expect the generalization error to concentrate, which is why we compute its average in (13) [3-5, 8-21].

To have a well-defined thermodynamic limit, the covariances ˜ Σ ℓ and the teacher ˜ w ℓ must be in some sense sufficiently well-behaved. We consider the following conditions, which are the generalization to our setting of those assumed in previous work [4-7, 16-18, 40]:

Assumption 3.1. We assume that we are given deterministic sequences of positive-definite matrices ˜ Σ ℓ ( n ℓ ) and vectors ˜ w ∗ ( n 0 ) indexed by the system size, such that the limiting (weighted) spectral moment generating functions

are well-defined, for all ℓ = 0 , . . . , L .

We can now state our results. As a preliminary step, we first give an expression for the generalization error for a fixed teacher ˜ w ∗ at finite ridge λ . Then, we pass to the ridgeless limit, on which we focus for the remainder of the paper. At finite ridge, we have the following:

Proposition 3.1. Assume Assumption 3.1 holds. For λ > 0 , let ζ solve the self-consistent equation

In terms of ζ , let κ ℓ ( ζ ) solve

for ℓ = 0 , . . . , L , where E ˜ σ ℓ [ h (˜ σ ℓ )] = lim n ℓ →∞ n -1 ℓ ∑ n ℓ j =1 h (˜ σ ℓ,j ) denotes expectation of a function h with respect to the limiting spectral distribution of ˜ Σ ℓ , for ˜ σ ℓ,j its eigenvalues at finite size, and let

Then, the learning curve (13) at finite ridge for a fixed target is given by

Proof of Proposition 3.1. We defer the derivation of (18) to Appendix A. To compute the disorder average in (13), we express the minimization problem in (6) as the zero-temperature limit β →∞ of an auxiliary Gibbs distribution p ( v ) ∝ e -βL , and evaluate the average over the random data random feature weights using the non-rigorous replica method from the statistical mechanics of disordered systems [29, 33]. This computation is lengthy but standard, and is closely related to the approach used in our previous works on deep linear models [13, 30]. All of our results are obtained under a replica-symmetric Ansatz ; as the ridge regression problem (6) is convex, we expect replica symmetry to be unbroken [29, 41, 42].

From the self-consistent equation (15), we recognize that ζ is is up to a sign the spectral moment generating function of the feature Gram matrix K = XFF ⊤ X ⊤ /n 0 , which is a product-Wishart random matrix [30]:

This dependence falls out of the replica computation of the generalization error using an auxiliary Gibbs distribution; we emphasize that one could take an alternative approach in which the generalization error is first expressed in terms of M K -as, for instance, in Gerace et al. [25] or Hastie et al.

[5]-and then use results on the spectra of product-Wishart matrices to conclude the claimed result [30]. This approach would potentially have the advantage of giving a fully rigorous proof, rather than one that depends on the replica trick. However, one would still then be faced with the task of solving the self-consistent equation for the spectral moment generating function, and therefore would end up in the same place insofar as quantitative predictions are concerned.

In principle, we could now directly proceed to study how weight structure affects (18) for some fixed ridge λ . However, as long as there is structure in the weights and/or the data, the self-consistent equation (15) must generally be solved numerically [14, 30]. To allow us to make analytical progress, we therefore focus on the ridgeless limit λ ↓ 0 for the remainder of the present paper, and leave careful analysis of the λ > 0 case to future work. This follows the path of most recent studies of models with linear random features, and also the fundamental interest in interpolating models [3-17, 19-21]. We therefore emphasize that we state Proposition 3.1 merely as a preliminary result.

Before giving our result for the generalization error in the ridgeless limit, we warn the reader of an impending, somewhat severe abuse of notation: in Proposition 3.2 and for the remainder of the paper, we will re-define κ ℓ to be given by its value for the solution for ζ appropriate in the regime of interest. Moreover, we will simply write ϵ for lim λ ↓ 0 ϵ .

Proposition 3.2. Assume Assumption 3.1 holds, and let α min = min { α 1 , · · · , α L } . For ℓ = 0 , . . . , L , in the regime α ℓ > 1 , let κ ℓ be given by the unique non-negative solution to the implicit equation

In terms of κ ℓ , let

In the regime α min < α 0 , let κ min be the unique non-negative solution to the implicit equation

Then, the learning curve (13) for a fixed target in the ridgeless limit λ ↓ 0 is given by

Proof of Proposition 3.2. We derive (23) as the zero-ridge limit of Proposition 3.1 in Appendix A.

Before we analyze the effect of weight anisotropy in detail in Section 4, we note several simplifying special cases of Proposition 3.2 which recover the results of prior works. To facilitate this comparison, we provide a notational dictionary in Appendix D. The first important special case is

Corollary 3.1. If L = 0 , we have

This recovers the known, rigorously proved result for linear ridgeless regression [4-7, 16-18]. For larger depths, an important simplifying case of Proposition 3.2 is that in which the data and features are unstructured, in which case the generalization error is given by

Corollary 3.2. If ˜ Σ ℓ = I n ℓ for ℓ = 0 , . . . , L , we have, for any target satisfying ∥ ˜ w ∗ ∥ 2 = n 0 ,

(a)

(b)

1/

α

0

α

1

_

α

0

1

2

0

0

2

1

ε

0

5

η

=0

η

=1/2

ε

0

40

η

=0

η

=1/2

Proof of Corollary 3.2. We have M I n ℓ ( z ) = 1 / ( z -1) , hence κ ℓ = α ℓ -1 , µ ℓ = 1 -1 /α ℓ , and κ min = α 0 /α min -1 . Finally, for any fixed teacher vector satisfying ∥ ˜ w ∗ ∥ 2 = n 0 , we have ψ ( z ) = 1 / ( z +1) if ˜ Σ 0 = I n 0 . Substituting these results into (23), we obtain (25).

This recovers the result obtained in our previous work [13], and in the single-layer case L = 1 recovers results obtained by Rocks and Mehta [14, 15], and by Hastie et al. [5] (see Appendix D). In the slightly more general case of unstructured weights but structured features, we have

̸

Corollary 3.3. If ˜ Σ ℓ = I n ℓ for ℓ = 1 , . . . , L , but ˜ Σ 0 = I n 0 , we have, for any target satisfying ∥ ˜ w ∗ ∥ 2 = n 0 ,

Proof of Corollary 3.3. (26) follows from substituting the results of Corollary 3.2 into (23).

In the special case L = 1 , this recovers the result obtained using rigorous methods in contemporaneous work by Bach [40], posted to the arXiv one day after the first version of our work [43]. Here, as the data spectrum and target vector enter the generalization error in nearly the same way as in the case of linear regression, all of the intuitions developed in that case can be carried over [4-7, 16-18].

Another useful simplification can be obtained by further averaging over isotropically-distributed teachers ˜ w ∗ ∼ N ( 0 , I n 0 ) , which gives

Corollary 3.4. Let ¯ ϵ = E ˜ w ∗ ∼N ( 0 , I n 0 ) [ ϵ ] . Then, we have

Proof of Corollary 3.4. Observing that E ˜ w ∗ ψ ( z ) = -M ˜ Σ 0 ( -z ) , the claim follows from (23).

In the special case of a single layer of unstructured feature weights ( L = 1 , ˜ Σ 1 = I n 1 ), this recovers the result of recent work by Maloney et al. [19], who used a planar diagram method to the generalization error of single-hidden-layer linear RFMs with unstructured weights (see Appendix D).

Another important simplifying case of Proposition 3.2 is the limit in which the hidden layer widths are large, in which the generalization error of the deep RFM reduces to that of a shallow model, as given by Corollary 3.1. More precisely, we have a large-width expansion given by:

Corollary 3.5. In the large-width regime α 1 , . . . , α L ≫ 1 , assuming that the weight spectra have finite moments, the generalization error (23) expands as

in the regime α 0 > 1 ; if α 0 < 1 the generalization error does not depend on the hidden layer widths so long as they are greater than 1.

Proof of Corollary 3.5.

4 How does weight structure affect generalization?

The first salient feature of the learning curves given by Proposition 3.2 is that the addition of weight structure does not alter the phase diagram of generalization, which is illustrated in Figure 1. There are three qualitatively distinct phases present, depending on the data density and minimum layer width: the overparameterized regime α 0 , α min > 1 , the bottlenecked regime α min < 1 , α min < α 0 , and the overdetermined regime α 0 < 1 , α 0 < α min . This dependence on the narrowest hidden layer matches our previous work on models with unstructured weights [13] 1 , and can be observed in the solutions to the ridge regression problem for fixed data (Appendix C). As α ℓ ↓ 1 , κ ℓ ↓ 0 and µ ℓ ↓ 0 , and the generalization error diverges. Similarly, the generalization error diverges as α min ↑ 1 , or α 0 ↑ 1 in the presence of label noise. However, there are not multiple descents in these deep linear models, consistent with the qualitative picture of the effect of nonlinearity given by previous works [9, 10].

The second salient feature of Proposition 3.2 is that the matrices ˜ Σ ℓ enter the generalization error independently; there are no 'interaction' terms involving products of the correlation matrices for different layers. This decoupling is expected given that the features are Gaussian and independent across layers [30]. Moreover, under the rescaling ˜ Σ ′ ℓ = τ ℓ ˜ Σ ℓ for τ ℓ > 0 , we have κ ′ ℓ = τ ℓ κ ℓ and µ ′ ℓ = µ ℓ (we show this explicitly in Appendix B). Therefore, (23) is sensitive only to the overall scale of ˜ Σ 0 , not to the scales of ˜ Σ 1 , . . . , ˜ Σ L . This scale-invariance can be observed directly from the ridgeless limit of the ridge regression estimator (7).

We can gain intuition for the effect of having ˜ Σ ℓ ̸∝ I n ℓ for ℓ ≥ 1 through the following argument:

Lemma 4.1. Under the conditions of Proposition 3.2, in the regime α 0 , α min > 1 , we have

That is, the generalization error for a given ˜ Σ 1 , · · · , ˜ Σ L is bounded from below by the generalization error for ˜ Σ ℓ = I n ℓ for ℓ = 1 , . . . , L .

Proof of Lemma 4.1. In Appendix B, we show that µ ℓ ≤ 1 -1 /α ℓ for any weight spectrum, which implies that (1 -µ ℓ ) /µ ℓ ≥ 1 / ( α ℓ -1) . Substituting these bounds in to the general expression for the generalization error in this regime from (23), the claim follows.

̸

Therefore, having ˜ Σ ℓ = I n ℓ for ℓ = 1 , . . . , L cannot improve generalization in the α 0 , α min > 1 regime. This is consistent with the large-width expansion in Corollary 3.5, where we can apply Jensen's inequality to bound the weight-dependence of the correction as E ˜ σ ℓ [˜ σ 2 ℓ ] / E ˜ σ ℓ [˜ σ ℓ ] 2 ≥ 1 , with equality only when the weights are unstructured. In other regimes, ˜ Σ 1 , · · · , ˜ Σ L do not affect the generalization error. In contrast, a similar argument shows that anisotropy in ˜ Σ 0 can be beneficial in the target-averaged case, at least in the absence of label noise. We formalize this as:

Lemma 4.2. Under the conditions of Corollary 3.4, in the absence of label noise ( η = 0 ), we have

That is, ¯ ϵ for a given ˜ Σ 0 is bounded from above by the generalization error for a flat spectrum ˜ Σ 0 = E [˜ σ 0 ] I n 0 .

Proof of Lemma 4.2. In Appendix B, we show that κ 0 ≤ ( α 0 -1) E [˜ σ 0 ] . As its defining equation (22) is of the same form as (20), the corresponding bound for κ min follows immediately: κ min ≤ ( α 0 /α min -1) E [˜ σ 0 ] . Substituting these bounds into (27) with η = 0 , the claim follows.

If E [˜ σ 0 ] is not finite, then this bound is entirely vacuous: ¯ ϵ ≤ ∞ . If we do not average over isotropically-distributed targets, then the effect of anisotropy in ˜ Σ 0 is harder to analyze. Previous works have, however, analyzed the interaction of data structure with a fixed target in great detail for models with L = 0 or L = 1 , showing that targets that align with the top eigenvectors of ˜ Σ 0 are easier to learn [5, 16, 17, 42, 44].

5 Power law spectra

We can gain further intuition for the effect of weight structure by considering an approximately solvable model for anisotropic spectra: power laws [16, 19, 28]. Power law data spectra have recently attracted considerable attention as a possible model for explaining the scaling laws of generalization observed in large language models [16, 19, 28, 32]. Maloney et al. [19] proposed a single-hiddenlayer ( L = 1 ) linear RFM with power-law-structured data and unstructured weights as a model for neural scaling laws. Does introducing power law structure into the weights affect the scaling laws predicted by deep linear RFMs? We have the following result:

Corollary 5.1. At finite size, define each covariance matrix ˜ Σ ℓ such that its j -th eigenvalue is ˜ σ ℓ,j = ˜ ς ℓ ( n ℓ /j ) 1+ ω ℓ for some fixed scale factor ˜ ς ℓ > 0 and exponent ω ℓ > 0 . Then, the limiting target-averaged generalization error is approximately

Proof of Corollary 5.1. Using the dictionary of notation in Appendix D, we can plug the approximate solutions for κ ℓ and µ ℓ derived by Maloney et al. [19] into (27) to obtain (31).

Therefore, the power law exponents ω 1 , · · · , ω L of the weight covariances beyond the first layer, which enter only through their sum Ω L , do not affect the scaling laws of the generalization error with the dataset size and network widths. In particular, in the absence of label noise ( η = 0 ) we can approximate the scaling of (31) in the regimes of large or small hidden layer width by

which recovers the results found by Maloney et al. [19] for L = 1 with unstructured weights. This behavior, and the agreement of (31) with numerical experiments, is illustrated in Figure 2. Consistent with Lemma 4.1, generalization with power-law weight structure is never better than with unstructured weights, as can be seen by comparing (31) with (25).

6 Bayesian inference and the Gibbs estimator at large prior variance

Thus far, we have focused on ridge regression (6). Though this is the most commonly-considered estimator in studies of random feature models [3-17, 19-21], one might ask whether our qualitative findings-in particular, that feature weight structure beyond the first layer is generally harmful for generalization-carry over to other estimators. Our approach to Proposition 3.2 is easily extensible to the setting of zero-temperature Bayesian inference , which has recently attracted substantial interest [13, 27, 34, 37, 45-47], sparked by work from Li and Sompolinsky [34]. In this case, we take seriously the Gibbs distribution p ( v ) ∝ e -βL , which in the ridge regression case was simply a convenient tool, and interpret it as the Bayes posterior for a Gaussian likelihood of variance 1 /β and a Gaussian prior with covariance Γ L +1 / ( βλ ) . It is in this context conventional to fix λ = 1 /β , such that the prior variance does not scale with β . We can then study the average of the generalization error

(a)

(b)

1/

α

0

1

2

0

ε

0

30

15

η

=0

η

=1/2

α

1

/

α

0

0.50

0.75

1.00

1.25

1.50

1.75

2.00

α

1

/

α

0

0.50

0.75

1.00

1.25

1.50

1.75

2.00

1/

α

0

1

2

0

ε

0

30

15

(c)

ω

1

0.50

1.00

2.00

3.00

4.00

1/

α

0

1

2

0

ε

0

30

15

η

=1/2

(13) under this posterior in the zero-temperature limit β →∞ , which we refer to as the generalization error of the Gibbs estimator. We emphasize that this is not identical to the Bayesian minimum mean squared error (MMSE) estimator given by the posterior mean, which would coincide with the ridgeless estimator in the zero-temperature limit (see Appendix A).

For a deep RFM, this simply has the effect of adding a 'thermal' variance term to the generalization error of the ridgeless estimator, which we describe in detail in Appendices A and C. We have:

Proposition 6.1. With the same setup as in Proposition 3.2, the generalization error of the Gibbs estimator for a RFM is

where ϵ ridgeless is given by Proposition 3.2, and κ ℓ is defined as in (20) .

Proof of Proposition 6.1. We derive (33) alongside Proposition 3.2 in Appendix A.

The Gibbs estimator is sensitive to the scale of the random feature weight distributions through κ ℓ , while as noted above the ridgeless estimator is not sensitive to their overall scale. This direct dependence on κ ℓ means that the simple argument of Lemma 4.1 cannot be applied. Indeed, in the limit of large prior variance, where the thermal variance term dominates, structure can improve the performance of the Gibbs estimator. We make this result precise in the following lemma:

Lemma 6.1. In the setting of Proposition 6.1, consider Bayesian RFMs with weight covariances scaled as τ ℓ ˜ Σ ℓ for ℓ = 1 , . . . , L . Then, in the non-trivial regime α 0 , α min > 1 where the thermal variance is non-vanishing, we have

where the scalars κ ℓ are defined in terms of the un-scaled covariances ˜ Σ ℓ as in (20) and ς 2 ≡ ∏ L ℓ =1 E ˜ σ ℓ [˜ σ ℓ ] . Therefore, in the limit of large prior variance, including structure in the weight priors is generically advantageous for generalization. If E ˜ σ ℓ [˜ σ ℓ ] is not finite, then the bound is vacuous.

Proof of Lemma 6.1. The first part of (34) follows from (33) using the scaling properties of κ ℓ , while the bound follows from the bounds on κ ℓ derived as part of Lemma 4.2.

In contrast, weight structure is generally harmful for the Bayesian RFM in the limit of small prior variance, as its performance then coincides with the ridgeless RFM, as can be seen from the scaling

of κ ℓ . This example illustrates that there are cases in which, depending on the estimator used, weight structure in deeper layers can sometimes be helpful for generalization. However, whereas the ridgeless estimator is commonly used in practice, the Gibbs estimator is less standard, and the limit of large prior variance is certainly artificial. Therefore, we emphasize that we give this example to show that the behavior of the ridgeless estimator is not entirely general, not to show that weight structure can be helpful in practical settings.

7 Discussion

We have computed learning curves for models with many layers of structured Gaussian random features learning a linear target function, showing that structure beyond the first layer is generally detrimental for generalization. This result is consistent with the intuition that in deep linear models learning a single target direction it is sufficient to modify the representation only at the first layer [13, 36]. It will be interesting to investigate whether this intuition carries over to nonlinear networks learning complex tasks, particularly including multi-index targets [35, 48]. Moreover, we have considered only linear, Gaussian models. As mentioned in the Introduction, past works have established Gaussian equivalence theorems for nonlinear RFMs with unstructured Gaussian feature weights. It will be important to investigate the effect of feature weight structure on Gaussian equivalence in future work, and determine whether our qualitative results carry over to nonlinear RFMs in the proportional limit.

Though our results are obtained using the replica trick, and we do not address the possibility of replica symmetry breaking, they should be rigorously justifiable given the convexity of the ridge regression problem [29, 33, 41]. We note that the replica approach makes it straightforward to handle models of any finite depth [30]. The relevant averages could of course be computed with alternative random matrix theory techniques, which could allow for a fully rigorous proof [5, 19-21]. Another more challenging setting to study with either the replica trick or rigorous techniques would be that in which one allows for correlations between weights in different layers. This setting could qualitatively capture aspects of feature learning in deep networks, which induces couplings across depth [45].

̸

In closing, we note that RFMs with structured weights may also have relevance for biological neural networks. A recent study by Pandey et al. [39] considered RFMs with a single layer of random features ( L = 1 ) with correlated rows ( Γ 1 = I n 0 ). In several biologically-inspired settings, they showed that introducing this structure could improve generalization, consistent with our results. More broadly, biological neural networks are imbued with rich priors [49]; investigating what insights deep structured models can afford for neuroscience will be an interesting subject for further study.

Acknowledgments and Disclosure of Funding

We thank Alexander Atanasov, Blake Bordelon, Benjamin S. Ruben, and James B. Simon for helpful discussions and comments on a draft of our manuscript. JAZ-V and CP were supported by NSF Award DMS-2134157 and NSF CAREER Award IIS-2239780. CP received additional support from a Sloan Research Fellowship. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.

References

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machinelearning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019. doi:https://doi.org/10.1073/pnas.1903070116.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3): 107-115, 2021. doi:10.1145/3446776.

Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 75, 2019. doi:10.1002/cpa.22008.

Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020. doi:10.1073/pnas.1907378117. URL https://www.pnas.org/doi/abs/10.1073/ pnas.1907378117 .

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949 - 986, 2022. doi:10.1214/21-AOS2133. URL https://doi.org/10.1214/21-AOS2133 .

Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel 'Ridgeless' regression can generalize. The Annals of Statistics , 48(3):1329 - 1347, 2020. doi:10.1214/19-AOS1849. URL https://doi.org/10.1214/19-AOS1849 .

Hong Hu and Yue M. Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory , 69(3):1932-1964, 2023. doi:10.1109/TIT.2022.3217698.

Stéphane d'Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in double descent: Bias and variance(s) in the lazy regime. In International Conference on Machine Learning , pages 2280-2290. PMLR, 2020. URL https://proceedings.mlr.press/v119/ d-ascoli20a.html .

Stéphane d'Ascoli, Levent Sagun, and Giulio Biroli. Triple descent and the two kinds of overfitting: where & why do they appear? In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 3058-3069. Curran Associates, Inc., 2020. URL https://proceedings.neurips. cc/paper/2020/file/1fd09c5f59a8ff35d499c0ee25a1d47e-Paper.pdf .

Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning , pages 74-84. PMLR, 2020. URL https://proceedings.mlr.press/v119/ adlam20a.html .

Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine-grained biasvariance decomposition. In Advances in Neural Information Processing Systems , volume 33, pages 11022-11032, 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 7d420e2b2939762031eed0447a9be19f-Abstract.html .

Gabriel Mel and Jeffrey Pennington. Anisotropic random feature regression in high dimensions. In International Conference on Learning Representations , 2022. URL https://openreview. net/forum?id=JfaWawZ8BmX .

Jacob A. Zavatone-Veth, William L. Tong, and Cengiz Pehlevan. Contrasting random and learned features in deep Bayesian linear regression. Physical Review E , 105:064118, Jun 2022. doi:10.1103/PhysRevE.105.064118. URL https://link.aps.org/doi/10.1103/ PhysRevE.105.064118 .

Jason W. Rocks and Pankaj Mehta. Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models. Physical Review Research , 4:013201, Mar 2022. doi:10.1103/PhysRevResearch.4.013201. URL https://link.aps.org/doi/10.1103/ PhysRevResearch.4.013201 .

Jason W. Rocks and Pankaj Mehta. Bias-variance decomposition of overparameterized regression with random linear features. Physical Review E , 106:025304, Aug 2022. doi:10.1103/PhysRevE.106.025304. URL https://link.aps.org/doi/10.1103/ PhysRevE.106.025304 .

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1024-1034. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/bordelon20a.html .

Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature Communications , 12(1):2914, May 2021. ISSN 2041-1723. doi:10.1038/s41467-02123103-1. URL https://doi.org/10.1038/s41467-021-23103-1 .

James B. Simon, Madeline Dickens, Dhruva Karkada, and Michael R. DeWeese. The Eigenlearning framework: A conservation law perspective on kernel regression and wide neural networks. arXiv , 2022. doi:https://doi.org/10.48550/arXiv.2110.03922.

Alexander Maloney, Daniel A. Roberts, and James Sully. A solvable model of neural scaling laws. arXiv , 2022. doi:10.48550/ARXIV.2210.16859.

Dominik Schröder, Hugo Cui, Daniil Dmitriev, and Bruno Loureiro. Deterministic equivalent and error universality of deep random features learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 30285-30320. PMLR, 23-29 Jul 2023. URL https: //proceedings.mlr.press/v202/schroder23a.html .

David Bosch, Ashkan Panahi, and Babak Hassibi. Precise asymptotic analysis of deep random feature models. arXiv , 2023. doi:10.48550/ARXIV.2302.06210. URL https://arxiv.org/ abs/2302.06210 .

Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 15156-15172. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ ad086f59924fffe0773f8d0ca22ea712-Paper.pdf .

Alexander Atanasov, Blake Bordelon, Sabarish Sainathan, and Cengiz Pehlevan. The onset of variance-limited behavior for networks in the lazy and rich regimes. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum? id=JLINxPOVTh7 .

Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment , 2021(12):124003, 12 2021. doi:10.1088/1742-5468/ac3a74. URL https://dx.doi.org/10.1088/1742-5468/ac3a74 .

Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mezard, and Lenka Zdeborova. Generalisation error in learning with random features and the hidden manifold model. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 34523462. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/gerace20a. html .

Andrea Montanari and Basil N. Saeed. Universality of empirical risk minimization. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 4310-4312. PMLR, 02-05 Jul 2022. URL https://proceedings.mlr.press/v178/montanari22a.html .

Hugo Cui, Florent Krzakala, and Lenka Zdeborova. Bayes-optimal learning of deep random networks of extensive-width. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 6468-6521. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/ cui23b.html .

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv , 2021. doi:10.48550/ARXIV.2102.06701. URL https://arxiv. org/abs/2102.06701 .

Marc Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications . World Scientific Publishing Company, 1987. doi:https://doi.org/10.1142/0271.

Jacob A. Zavatone-Veth and Cengiz Pehlevan. Replica method for eigenvalues of real Wishart product matrices. SciPost Physics Core , 6:026, 2023. doi:10.21468/SciPostPhysCore.6.2.026. URL https://scipost.org/10.21468/SciPostPhysCore.6.2.026 .

A. Caponnetto and E. De Vito. Optimal rates for the regularized least-squares algorithm. Foundations of Computational Mathematics , 7(3):331-368, 07 2007. ISSN 1615-3383. doi:10.1007/s10208-006-0196-8. URL https://doi.org/10.1007/s10208-006-0196-8 .

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv , 2020. doi:10.48550/ARXIV.2001.08361. URL https://arxiv.org/abs/ 2001.08361 .

Andreas Engel and Christian van den Broeck. Statistical Mechanics of Learning . Cambridge University Press, 2001. doi:https://doi.org/10.1017/CBO9781139164542.

Qianyi Li and Haim Sompolinsky. Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization. Physical Review X , 11:031059, 09 2021. doi:10.1103/PhysRevX.11.031059.

Adityanarayanan Radhakrishnan, Daniel Beaglehole, Parthe Pandit, and Mikhail Belkin. Feature learning in neural networks and kernel machines that recursively learn features. arXiv , 2022. doi:10.48550/ARXIV.2212.13881. URL https://arxiv.org/abs/2212.13881 .

Haozhe Shan and Haim Sompolinsky. Minimum perturbation theory of deep perceptual learning. Physical Review E , 106:064406, Dec 2022. doi:10.1103/PhysRevE.106.064406. URL https://link.aps.org/doi/10.1103/PhysRevE.106.064406 .

Boris Hanin and Alexander Zlokapa. Bayesian interpolation with deep linear networks. Proceedings of the National Academy of Sciences , 120(23):e2301345120, 2023. doi:10.1073/pnas.2301345120. URL https://www.pnas.org/doi/abs/10.1073/pnas. 2301345120 .

Zdzisław Burda, Jerzy Jurkiewicz, and Bartłomiej Wacław. Spectral moments of correlated Wishart matrices. Physical Review E , 71:026111, Feb 2005. doi:10.1103/PhysRevE.71.026111. URL https://link.aps.org/doi/10.1103/PhysRevE.71.026111 .

Biraj Pandey, Marius Pachitariu, Bingni W. Brunton, and Kameron Decker Harris. Structured random receptive fields enable informative sensory encodings. PLOS Computational Biology , 18(10):1-28, 10 2022. doi:10.1371/journal.pcbi.1010484. URL https://doi.org/10.1371/ journal.pcbi.1010484 .

Francis Bach. High-dimensional analysis of double descent for linear regression with random projections. arXiv , 2023. doi:https://doi.org/10.48550/arXiv.2303.01372.

Jean Barbier, Dmitry Panchenko, and Manuel Sáenz. Strong replica symmetry for highdimensional disordered log-concave Gibbs measures. Information and Inference: A Journal of the IMA , 12 2021. ISSN 2049-8772. doi:10.1093/imaiai/iaab027. URL https://doi.org/ 10.1093/imaiai/iaab027 . iaab027.

Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborova. Learning curves of generic features maps for realistic datasets with a teacher-student model. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, 2021. URL https://proceedings.neurips.cc/paper/2021/hash/ 9704a4fc48ae88598dcbdcdf57f3fdef-Abstract.html .

Jacob A. Zavatone-Veth and Cengiz Pehlevan. Learning curves for deep structured Gaussian feature models. arXiv , 2023. doi:https://doi.org/10.48550/arXiv.2303.00564.

Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Out-of-distribution generalization in kernel regression. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 12600-12612. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/ paper/2021/file/691dcb1d65f31967a874d18383b9da75-Paper.pdf .

Jacob A Zavatone-Veth, Abdulkadir Canatar, Benjamin S Ruben, and Cengiz Pehlevan. Asymptotics of representation learning in finite Bayesian neural networks. Journal of Statistical Mechanics: Theory and Experiment , 2022(11):114008, nov 2022. doi:10.1088/1742-5468/ac98a6. URL https://dx.doi.org/10.1088/1742-5468/ac98a6 .

Jacob A Zavatone-Veth and Cengiz Pehlevan. Depth induces scale-averaging in overparameterized linear Bayesian neural networks. In Asilomar Conference on Signals, Systems, and Computers , volume 55, 2021. doi:10.1109/IEEECONF53345.2021.9723137.

S. Ariosto, R. Pacelli, M. Pastore, F. Ginelli, M. Gherardi, and P. Rotondo. Statistical mechanics of deep learning beyond the infinite-width limit. arXiv , 2022. doi:10.48550/ARXIV.2209.04882. URL https://arxiv.org/abs/2209.04882 .

Jacob A. Zavatone-Veth, Sheng Yang, Julian A. Rubinfien, and Cengiz Pehlevan. Neural networks learn to magnify areas near decision boundaries. arXiv , 2023. doi:10.48550/ARXIV.2301.11375. URL https://arxiv.org/abs/2301.11375 .

Lukas Braun, Clémentine Carla Juliette Dominé, James E Fitzgerald, and Andrew M Saxe. Exact learning dynamics of deep linear networks with prior knowledge. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , volume 35, 2022. URL https://openreview.net/forum?id=lJx2vng-KiC .

Robb J Muirhead. Aspects of Multivariate Statistical Theory . John Wiley & Sons, 2009. doi:10.1002/9780470316559.

A Derivation of Proposition 3.2

In this Appendix, we sketch our replica-theory approach to computing the learning curves, which leads to Proposition 3.2. Many of the steps of this calculation are all but identical to our previous works on replica approaches to the spectra of product Wishart random matrices [30], and on unstructured deep Gaussian random feature models [13], so we will sketch the major steps rather than spelling out all the details of the algebra.

A.1 Gibbs distribution and replica free energy

We start by introducing a Gibbs distribution at fictitious inverse temperature β associated with the ridge regression loss

where we denote by D all randomness in the problem. For any λ > 0 , in the zero-temperature limit β →∞ , this Gibbs distribution concentrates around the unique minimum of the loss E [29, 33].

For the purpose of the replica computation, it is convenient to consider instead the partition function of the posterior of a related Bayesian model, which corresponds to absorbing βλ into a redefinition of Γ L +1 , and treating the ridge penalty as a Gaussian prior

We can then recover the partition function of the ridge regression model by undoing the rescaling: Γ L +1 ← Γ L +1 / ( βλ ) . Without this re-scaling-i.e., in the case in which the prior variance is held fixed as the temperature goes to zero-this is the Gibbs estimator in the zero-temperature limit, i.e., a Bayesian model with Gaussian likelihood of vanishing variance [13, 34, 37, 45, 46].

This gives us the partition function

which is the extension to structured priors of the Gibbs estimator partition function considered in [13]. By standard arguments, we expect the quenched free energy

to be self-averaging in the thermodynamic limit, i.e., f = E D f almost surely [29, 33]. To compute the limiting quenched average, we use the replica trick, and write

where we evaluate the moments E D Z m for positive integer m , and assume that they can be analytically continued to m → 0 .

Following previous work [13, 30], we can compute the quenched averages and integrate out the weights by introducing order parameters

with partition function for ℓ = 0 ,

for ℓ = 1 , . . . , L -1 and

Under this Ansatz , we have

to leading order in m , where we recall the boundary condition q L +1 = 1 , c L +1 = 0 [30]. The resulting saddle point equations can be simplified to give a closed system for the replica non-uniform components,

along with corresponding Lagrange multipliers ˆ C ℓ , which yields

for

where we let C L +1 = I m and

for ℓ = 0 , . . . , L . We note that ⊗ here denotes the Kronecker product, and we use the convention that the standard matrix product has higher precedence than the Kronecker product, i.e., AB ⊗ C = ( AB ) ⊗ C . Importantly, the quantity of interest-the generalization error-is simply given by the diagonal elements of C 0 , i.e., ϵ = ( C 0 ) aa . Therefore, if we can solve for the order parameters at zero temperature, we will obtain the generalization error.

In the thermodynamic limit, the integral over these order parameters can be evaluated using the method of steepest descent. We make a replica symmetric Ansatz , and seek saddle points of the form

with the boundary condition q L +1 = 1 , and a linear system for the replica uniform components,

with the boundary condition c L +1 = 0 .

A.2 Converting between the Gibbs and maximum-likelihood estimators

As our primary aim is to study ridge regression, we must now account for the fact that the prior over the readout weights scales with the inverse temperature β . In particular, we have a prior with scaled covariance Γ L +1 / ( βλ ) , where Γ L +1 does not scale with β . If we perform this rescaling in (A.16) and (A.16), we can see that the re-scaled order parameters

obey an identical system of equations to the original order parameters in the Bayesian case at inverse temperature

Therefore, if we can solve the saddle point equations for the Gibbs estimator in the zero-temperature limit, we can simply read off the corresponding result for the ridge regression estimator in the ridgeless limit. The important difference is that the replica nonuniform component q 0 of C 0 is O (1 /β ) in the ridge regression case, hence only the replica uniform component c 0 contributes to the generalization error. We note that this allows one to read off the generalization error of a deep linear RFM with unstructured features from the results of our previous work [13] simply by setting the Bayesian prior variance σ 2 to zero.

A.3 Solutions for the generalization error

The replica-symmetric saddle point equations in (A.16) and (A.19) are nearly identical to those analyzed our computation of the maximum eigenvalue of a structured Wishart product matrix [30],

which in turn are related to those in our original paper on unstructured deep linear RFMs [13] by the replacement of the spectral moment generating function of the identity matrix with the appropriate spectral generating functions. Given this similarity, and the fact that we have provided extensive exposition of how to solve such systems in those previous works, we will merely state the results for the order parameters relevant to the computation of the generalization error.

Let for ℓ = 0 , . . . , L , where

satisfies the closed equation

For ℓ = 0 , . . . , L , let so that κ ℓ satisfies

In terms of κ ℓ , let

be the moment generating function of ˜ Σ ℓ , with functional inverse M -1 ˜ Σ ℓ ( z ) . Then, at finite temperature, after eliminating the Lagrange multipliers, the replica nonuniform components of the order parameters are given by

From [30], we recognize this as the self-consistent equation for the moment generating function M = A of the feature kernel K = XFF ⊤ X ⊤ /n 0 , evaluated at -1 /β . Even in the unstructured case, this equation must in general be solved numerically at finite temperature [30].

Given a solution to this equation, we can solve the system of linear equations (A.19) for c 0 , mirroring the computation of the extremal eigenvalues of structured product Wishart matrices in [30]. After eliminating the Lagrange multipliers, this calculation boils down to solving a three-term recurrence relation, which is detailed in previous works [13, 30]. We therefore simply state the result of this computation here. Let ζ = -A , which then satisfies

Viewing κ ℓ as a function of ζ , we may alternatively write the self-consistent equation for ζ as

We then finally have

Using the mapping of Appendix A.2 and again defining the weighted generating function

this yields Proposition 3.1.

We now want to extract the zero-temperature/ridgeless limit. As β →∞ , the self-consistent equation for ζ admits the solution

valid for α ℓ > 1 for all ℓ , which gives q 0 ∼ O (1) , the solution

valid for α 0 < 1 , α 0 < α 1 , . . . , α L , which gives q 0 ∼ O (1 /β ) , and, for ℓ ∗ = 0 , . . . , L , the solutions

̸

valid for α ℓ ∗ < 1 , α ℓ ∗ < α 0 , α ℓ ∗ < α ℓ for all ℓ = ℓ ∗ , which also give q 0 ∼ O (1 /β ) . These solutions mirror those found in the unstructured setting [13]. We remark that, as in [13], we can determine the regimes in which each solution is physical by demanding that the order parameters q ℓ are non-negative.

For the ζ → 1 solution, we immediately have

where by a minor abuse of notation we simply write κ ℓ and µ ℓ for the corresponding quantities evaluated at ζ = 1 .

If ζ → α ℓ , then κ ℓ ↓ 0 and µ ℓ ↓ 0 . We can then apply L'Hôpital's rule to evaluate the limit in A.39, which corresponds to extracting the most divergent terms on each side of A.39. For the ζ = α 0 solution, one finds that

Finally, for the solutions with ζ = α ℓ ∗ for ℓ ∗ = 1 , . . . , L , one finds that

where we must be careful to recall that κ 0 now satisfies

But, we recognize that α ℓ ∗ = α min = min { α 1 , . . . , α L } , so we will write

to avoid clashing with our notation for the ζ = 1 solution.

Therefore, recalling from Appendix A.2 that the generalization error for the ridge regression estimator in the ridgeless limit is simply given by c 0 , we have

as reported in (23), where we again define the weighted generating function

To obtain the average generalization error for the Gibbs estimator in the zero-temperature limit, we must account for the effect of q 0 in the regime α ℓ > 1 , as in all other regimes it is q 0 ∼ O (1 /β ) . But, we recognize that

from the definition above, hence we conclude that

A.4 Physical interpretation of the order parameters and thermal bias-variance decomposition

With these results in hand, we now comment on the interpretation of the replica uniform and replica non-uniform contributions to

At the saddle point, we have

̸

where ⟨·⟩ β denotes the expectation with respect to the replicated Gibbs measure at inverse temperature β . Under the replica-symmetric Ansatz , considering off-diagonal elements a = b , we can use the fact that the replicas are initially uncoupled and identical to write

Similarly, we have

where we write δ v = v -⟨ v ⟩ β . Therefore, at the saddle point, c 0 and q 0 correspond exactly to the bias and variance terms in the thermal bias-variance decomposition of the generalization error:

This makes concrete an argument which was presented only intuitively in [13]. As a result, if one considered the Bayesian MMSE estimator ˆ v = ⟨ v ⟩ β , the zero-temperature generalization error would simply coincide with that for the ridgeless estimator.

B Properties of the inverse generating functions

Here, we record a few useful properties of the inverse spectral generating functions

and their relatives

These results are used in the proofs of Lemmas

B.1 Dependence on width

Implicitly differentiating the self-consistent equation defining κ ℓ , we have

showing that κ ℓ is a decreasing function of 1 /α ℓ . As 1 /α ℓ ↓ 0 , we should have κ ℓ ↑ ∞ , while as 1 /α ℓ ↑ 1 , we should have κ ℓ ↓ 0 .

B.2 Behavior under rescaling

Consider the re-scaling ˜ Σ ′ ℓ = τ ℓ ˜ Σ ℓ for τ ℓ > 0 . Then, we have κ ℓ and ˜ κ ′ ℓ given by

respectively. We can then see that we should have

and

B.3 Bound on κ ℓ in terms of isotropic spectrum

We now prove that

Then, note that and

we must have

in the relevant regime α ℓ > 1 . For any z > 0 ,

is a concave function of ˜ σ ℓ ≥ 0 , hence Jensen's inequality implies that

are both decreasing functions of z ≥ 0 , and both are equal to 1 when z = 0 . Thus, if κ ℓ > 0 solves

as specified by its definition and ¯ κ ℓ > 0 solves

But, we can easily see that ¯ κ ℓ = ( α ℓ -1) E [˜ σ ℓ ] , hence the claim follows.

B.4 Bound on µ ℓ terms of isotropic spectrum

We next prove that

in the relevant regime α ℓ > 1 . By definition, we have

By Jensen's inequality and the definition of κ ℓ , we have

As α ℓ > 1 by assumption, this bound is always positive. Therefore, we conclude the desired claim.

C Simplifying the generalization error for fixed data

In this appendix, we show how the ridgeless generalization error can be simplified in each regime for fixed data. Using the solution to the ridge regression problem (6),

we have

Following our discussion in the main text, we may set Γ L +1 = I n L without loss of generality, as otherwise we may re-define Σ L . Then, we have

In the subsequent sections, we will simplify this expression in each regime.

For the Gibbs estimator, we must account for the additional contribution to the generalization error from thermal variance. Following our previous work [13], we may compute the bias and variance terms directly from the posterior moment generating function of the readout weight vector,

yielding

and

We then can see that

which is precisely in agreement with the conversion in Appendix A.2. Considering the thermal bias-variance decomposition of the generalization error for the Gibbs estimator,

we can then see that the bias term at zero temperature coincides exactly with the generalization error of the ridgeless estimator, as we found in Appendix A. The variance term is

In both the bias and variance terms, we can see that we may set Γ L +1 = I n L without loss of generality, as otherwise we may simply re-scale Σ L as discussed in Lemma 2.1. Then, we need only consider the thermal variance term

Here, we leave the thermodynamic limit implicit to allow the expression to fit on a single line.

C.1 The overparameterized regime

First, consider the regime p < min { n 0 , . . . , n L } . Here, we expect the kernel

to be invertible with probability one in the thermodynamic limit, and with overwhelming probability at large but finite size [50]. Applying the push-through identity and passing to the ridgeless limit, we have

Averaging over label noise, we have

Turning our attention to the Gibbs estimator, we can use the Woodbury identity to write the thermal variance term as

where the thermodynamic limit is implied [13]. Therefore, in this regime we do not expect the thermal variance term to vanish, consistent with Proposition 6.1.

C.2 The bottlenecked regime

If min { n 1 , . . . , n L } < min { n 0 , p } , then the situation is slightly more complicated. Let

be the index of the narrowest hidden layer. Then, let

Then, the matrices F ⊤ 1 X ⊤ XF 1 and F 2 F ⊤ 2 are invertible with probability one, and upon passing to the ridgeless limit we have

Averaging over the label noise,

Focusing on the label noise term, we have

Then, using the fact that we have

and such that

so

Then, as we expect the matrix F ⊤ 1 Σ 0 F 1 to be invertible with overwhelming probability, the standard formula for the mean of an inverse-Wishart distribution [50] gives

This proves that, in this regime, the label noise term does not depend on data structure, matching the result of our replica computation.

Considering the Gibbs estimator, we can see immediately that the thermal variance term is O ( β -1 ) because of the fact that F ⊤ 1 X ⊤ XF 1 and F 2 F ⊤ 2 are invertible with probability one. This is consistent with Proposition 6.1.

C.3 The overdetermined regime

Finally, consider the regime in which n 0 < min { p, n 1 , . . . , n L } . Then, both X ⊤ X and FF ⊤ are invertible with probability one, and we can easily compute

Then, as we found using replicas.

Here, again, we can see that the thermal variance term for the Gibbs estimator is O ( β -1 ) , matching Proposition 6.1.

D A notational dictionary

In this appendix, we show that special cases of our general result recover the results reported in previous works. This is largely a matter of translating notation, as the conventions used in different communities are often at odds with each other.

D.1 Shallow ridgeless regression

In the shallow case L = 0 , our general result for a fixed target (23) reduces to

where, writing expectation with respect to the limiting spectral distribution of Σ 0 as E σ 0 , we recall that κ 0 is determined by the implicit equation

Working in the eigenbasis of Σ 0 and assuming that ∥ w ∗ ∥ 2 = n 0 , we introduce the weighted density

We can now make contact with the result of Hastie et al. [5]. We note that those authors use an opposite definition for p and n : following the convention in the statistics literature, they use p for in terms of which we have

and that in terms of which we have

and

so using the formula for the mean of the inverse-Wishart [50] we have

the dimensionality and n for the number of examples, while we follow the convention in the physics literature of using n 0 for the dimensionality and p for the number of examples. Then, Hastie et al. [5]'s γ , defined such that, in our terms, n 0 /p → γ , is precisely our α 0 . Moreover, they use H ( z ) to denote the limiting spectral law of Σ 0 , and G ( z ) to denote the law corresponding to the weighted density we define above as ρ . We note also that their σ 2 is our η 2 . In these terms, their Theorem 2 gives the generalization error in the overparameterized regime α 0 > 1 as

where c 0 is defined by the implicit equation

Subtracting one from both sides, the implicit equation for c 0 gives

from which we can see that

Then, we have

which proves the equivalence of our results. This also shows that we recover the results of other works on ridgeless kernel interpolation [4, 6, 7, 16, 17] that are in this setting equivalent to the results of Hastie et al. [5].

D.2 Two-layer linear random feature models with unstructured weights and isotropic targets

Another special case in which we can make contact with prior work is that of a single hidden layer ( L = 1 ) and with target averaging. In this case, our general result (27) reduces to

and

where in this case we find it convenient to write κ 0 /α 0 and α 0 κ min /α min in terms of χ ( z ) , which solves

It is then easy to show that our result agrees with that of Maloney et al. [19]. Their notation is:

When M > N,T , their result is, in the absence of label noise,

and similarly for ∆ -1 ( T, M ) . To map this to our results, let us re-define

Then, we can see that, in our notation,

we can see that we recover their result in these regimes. We can also map their ∆ 0 to our µ 0 . For T < M , they let

where ∆ -1 ( N,M ) solves which then satisfies

or while

Then, noting that hence we can see that

with

(a)

(b)

α

0

α

1

(1)

(2)

(3)

0

1

2

2

1

1/

α

0

α

1

_

α

0

(1)

(2)

(3)

0

1

2

0

0

2

1

This mapping also enables our application of their interpolating approximate solutions for ∆ -1 and ∆ 0 in the case of power law spectra. For a finite-size spectrum

where we denote the exponent by ω rather than α as Maloney et al. [19] do to avoid clashing with our notation elsewhere, they obtain the approximate solution

for

which leads to the expression

Moreover, for T < M , they give the approximate solution

By applying these results, we obtain the result claimed in the main text, (31). We note that we fix σ -to be constant rather than σ + as Maloney et al. [19] do, which ensures normalizability of the limiting eigenvalue distribution at the expense of diverging moments.

D.3 Deep linear models with unstructured weights and data

In [13], we studied deep Bayesian linear models with unstructured features and data. 2 There, and in very recent work by Schröder et al. [20], a different parameterization for the thermodynamic limit

was used:

where we decorate ˜ α and ˜ γ ℓ with tildes to avoid confusion with parameters used elsewhere in the present work. The conversion to the parameterization used in the present work and in [30] is then given by

Though these parameterizations are mathematically equivalent, it is important to distinguish between them as they give phase diagrams that divide the plane identically but swap the locations of the phases, as is shown in Figure D.1. Moreover, though the parameterization used here is more convenient for the replica computation [30], that given in (D.42) is conceptually useful, as it is closer to what one does in practical machine learning settings: the input dimension n 0 is fixed by the task, and one can vary the dataset size p and the network widths n ℓ . This is why we plot the phase diagrams in Figure 1 in the (1 /α 0 , α 1 /α 0 ) plane.

E Large-width expansions

In this appendix, we consider the limit of large width, i.e., the limit in which α 1 , . . . , α L →∞ for fixed α 0 . Our first task is to determine how the quantities κ ℓ behave in this limit, as it is through these inverse generating functions that the hidden layer widths enter the generalization error.

Starting from the defining equation

Expanding, we have

we can see that κ ℓ should tend to infinity linearly with α ℓ as α ℓ →∞ . In particular, we should have

at large widths. Then, µ ℓ has limiting behavior

From this, we can see that in the infinite-width limit the generalization error of the random feature model in Proposition 3.2 reduces to that of shallow ridgeless regression as in Corollary 3.1, as we would expect.

We now want to compute the leading correction to this result. In the unstructured case, this is easy, because we have κ ℓ = ( α ℓ -1)˜ σ ℓ , hence there is an O (1) correction and nothing else. More generally, we assume Laurent series behavior of the form

hence, if we integrate term-by-term, we have

If we solve order-by-order, we again find that

while the coefficients of all higher-order terms in 1 /α ℓ must vanish. In particular, this gives

This computation assumes that the spectrum has finite moments, which is not the case for the heavy-tailed power law spectra considered in Corollary 5.1.

Then, we have and

Collecting our results, we have

Each term in these expansions has the expected behavior under rescaling: if we let ˜ Σ ′ ℓ = τ ℓ ˜ Σ ℓ for τ ℓ > 0 , we have κ ′ ℓ = τ ℓ κ ℓ and µ ′ ℓ = µ ℓ .

Then, substituting these expansions into (23), we find that the generalization error of an RFM in the ridgeless limit expands at large widths as

in the regime α 0 > 1 ; if α 0 < 1 the generalization error does not depend on the hidden layer widths so long as they are greater than 1.

For an RFM trained using the Gibbs estimator, as considered in Proposition 6.1, we find that

where we have defined

upon expanding the thermal variance term

Here, we denote by O ( w -2 ) all terms of O ( α -2 ℓ ) for a given layer ℓ = 1 , . . . , L or terms of O ( α -1 ℓ α -1 ℓ ′ ) for two different layers ℓ, ℓ ′ .

F Numerical methods

In this appendix, we describe the numerical methods used to produce Figures 1, 2. All simulations were performed using MATLAB 9.13 (R2022b; The MathWorks, Natick MA, USA; https://www. mathworks.com/products/matlab.html ) on a desktop workstation (CPU: Intel Xeon W-2145, 64GB RAM). They were not computationally intensive, and required less than an hour of compute time in total. Code to reproduce the figures is archived as part of the online supplemental material. Numerical computation of the solution to the ridgeless regression problem-the minimum-norm interpolant-was performed using the lsqminnorm solver ( https://www.mathworks.com/help/ matlab/ref/lsqminnorm.html ), which uses an algorithm based on the complete orthogonal decomposition of the design matrix.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the generalization performance of models utilizing multi-layered Gaussian random features. The study evaluates the impact of feature anisotropy, which is often overlooked due to common assumptions that features are generated using independent, identically distributed Gaussian weights. The findings demonstrate that correlations within the first layer of features can enhance generalization, but any structure beyond the initial layer proves generally detrimental. These insights provide valuable perspective on how weight structure affects generalization in random feature models with linear activations.\n\nSTRENGTHS:\n1. The paper is well-written and the theorems are constructed with a solid use of mathematical rigor.\n\n2. The idea is interesting, where how correlation between the rows of the first layer of features can improve generalization is a new theoretical result, which could be very interesting to the community.\n\n3. Visualization and numerical experiments are conducted along with the theoretical results. For instance, the experiments in Figure 2 nicely summarize the theoretical finding, implying the bounds are generally non-vacuous.\n\nWEAKNESSES:\n1. The discussion to previous works are limited. There are many RFM works recently. Although they may focus on a different perspective, such as inductive biases or behavior under SGD, but it would still be valuable to briefly discuss the relationship to these works.\n\n2. At the first glimpse, it is hard to parse how the structure can improve the generalization in some of the theorem or corollary. In particular, there are many constants and it is non-trivial to interpret their numerical property. Adding a simple sentence after the theorem to briefly summarize it could help reader better parse the results.\n\nQUESTIONS:\n1. Would figure 2 vary a lot if we modified the number of data points?  In particular, would the theory behaves differently under over-parameterized or under-parameterized regime?\n\n2. The theorems analyze the asymptotic learning curve, where the numerical experiments are conducted under a relatively small number of datapoints. Will this cause a gap between the theory and the experimental results?\n\n3. The constraint on the norm of the teacher vector (line 142) is not very intuitive. Can you briefly explain this?\n\n\n4. Does the result approximately hold for setting without closed form such as Lasso-regression?\n\nLIMITATIONS:\nThe authors did discuss the limitation and there is no obvious negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate the exact asymptotics characterization of deep Gaussian feature models. They analyze depth-L linear Random Feature Models where the feature matrix is built as a product of L factors, each drawn from a matrix Gaussian distribution. Thanks to the replica method, they compute the generalization error learning curves for the ridge regression estimator. They study the influence of the weight structure on the test performance, presenting numerical experiments backing up the theoretical claims.\n\nSTRENGTHS:\nThe authors consider the interesting case of deep RFMs with weight anisotropy. They compute the performance for both ERM and Bayes estimators in the high-dimensional proportional regime. They present numerical simulation and release the code to reproduce the main experiments.\n\nWEAKNESSES:\nThe primary weakness of this paper is the clarity of the presentation, more precisely I believe more effort should be put into guiding a non-expert reader with a more detailed introduction, and by creating connections between the different main results presented. See below for a more detailed discussion.\n\nQUESTIONS:\n1) The abstract is very concise. I would expand this section to help guide the reader, e.g., mention that you focus on interpolators,  describe what exactly solvable model for anisotropic spectra you considered (power law), briefly explain that the proposed formalism allows as well to cover analysis of Bayesian setting. \n2) On a similar note to the first point, the introduction section is too short. There is little explanation of why RFMs are interesting, e.g., it is not mentioned explicitly their relation with the limiting kernel. The analysis of power-law spectrum decay is standard in the kernel literature and goes under the name of source-capacity conditions [1]. There is no mention of the technique which is going to be used for the computation (the replica trick). Please mention why the interpolating (ridgeless) regime is interesting, with associated reference, e.g. ref [5] in the main text. There is no citation to previous work on the Bayesian settings, the reference [42] mentioned in the conclusion could be introduced as well there. A justification should be given for the Gaussian data assumption, e.g., by referring to the extensive line of work on the Gaussian Universality property, see [2]. \n3) Please before introducing the preliminaries and the setting on page 2, include a summary of your main contributions. \n4) Why is it untractable to study the $\\lambda > 0$ case? A numerical investigation of the behavior at finite regularization I believe would enhance the manuscript, if the problem is not untractable. If this is the case, please mention why in the main text. \n5) The results are nice but I believe some work is needed to glue them together. The authors may think to move in the appendix some proof and substitute them with explanations in plain words to help the reader build intuition, as the notation is quite heavy.\n6) In Sec. 4 remind the reader in which theorem are defined $(\\alpha_{min},\\mu)$. \n7) Can the authors explain in more detail why the exponents $(\\omega_1,\\dots,\\omega_L)$ do not affect the scaling laws if they enter only with the sum (Sec. 5)?\n\n[1] Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares algorithm, 2007.\n\n[2] Montanari, A. and Saeed, B. N. Universality of empirical risk minimization, 2022.\n\nLIMITATIONS:\nThe limitations are addressed in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors introduce correlations to weights of linear RFMs and study the generalization error.

This is done under the assumption data follows Gaussian distribution and linear model.

The authors provide a general expression for the limiting generalization error that recovers results from previous works as special cases.

STRENGTHS:
The present results present appear to be very general and well linked to previous research.

The introduction and preliminaries section are relatively clear.

WEAKNESSES:
The proofs could be included in the supplementary material which would allow for more text interpreting the results and providing the intuition for their importance.

The authors should explain the solutions (15) and (16).

The authors should explain in more detail their result, i.e. (18). 

The authors should be more explicit about the notation, e.g. carefully check the use of /tilde or $\kappa_0$.

In my opinion, the clarity of the paper could be improved.

QUESTIONS:
Do solutions (15) and (16) always exist? Can you elaborate on their form?

Do you have any intuition how quickly convergence to (13) happen?

Do you have any intuition about the limits when p is fixed?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the asymptotic risk of deep linear random features models (RFMs), with a high dimensional analysis based on the replica method. It extends previous work on this topic by relaxing the standard i.i.d assumption for the Gaussian weights in each layer. \n\nSeveral consequences of their anaysis are discussed: (i)  several known results in linear regression, isotropic models, infinite width RFM, are recovered as special cases ; (ii) it is shown that feature anisotropy is detrimental, in the sense that the risk of the isotropic model  lowerbounds the risk of the general model ; (iii) it is shown that feature anisotropy does not affect the scaling laws of the risk ;  (iv) going beyond ridge regression, the analysis allows for the derivation of the risk of the Gibbs estimator, where feature anisotropy is shown to be generally beneficial (resp detrimental) for  large (resp small) prior variance.\n\nSTRENGTHS:\n* Sound piece of theoretical work. \n\n* Relaxing the standard feature isotropy assumption seems to be a natural extension of the recent line of work on deep RFMs. Since the learned features of deep learning models often exhibit complex correlations, it could also provide insights on the effects of feature learning of deep networks in a controlled setting. \n\n* The paper is very clearly written and enjoyable to read.\n\nWEAKNESSES:\nMy main reservation regarding this paper is related to its scope -- and the significance of the results. \n\n* While the analysis presented in the paper is novel and fills a gap in the literature by relaxing a standard assumption, the technical innovations appear to be somewhat limited and incremental compared to previous work, such as References [28].\n\nI believe some related references were missed, see e.g., Mel & Pennington (ICLR 2022) -- which also investigates the effect of feature anisotropy  in random feature regression. I understand the main technical difference is that they work with shallow models with a Gaussian feature matrix, whereas the current paper work with Gaussian products (one may argue that the first setting is sufficient to capture the effects of anisotropy in RFMs). \n\n* The insights gained from this analysis also seem to have certain limitations. For instance, considering  that in setting (13), the risk is studied in expectation over rotation invariant Gaussian matrices $Z_\\ell$ at each layer, the result of Section 4 on the optimality of the isotropic case appears rather unsurprising to me. \n\nMoreover I believe that the studied setting may not allow for  significant insights into representation learning (not that the authors claim otherwise, but this is one of the motivations for the work i.m.o).  For example , the assumption of Gaussians requires layerwise independence, while in feature learning scenarios, one would expect learned features in different layers to be correlated with each other -- and with the underlying data structure. So from that poinf of view, I feel the assumptions underlying this work are still quite restrictive. \n\nOn a minor related note, the general-sounding statement found in Section 1, \"these results are consistent with the intuition that representation learning at only the first layer of a deep linear model is sufficient to achieve optimal performance\" is a bit puzzing, as it seems to contradict known results in the topic, e.g. those on the implicit sparsity bias in deep linear networks (which requires representation learning in multiple layers, see e g., Woodworth et al, 2020).\n\n**References**\n\nMel & Pennington (ICLR 2022). Anisotropic Random Feature Regression in High Dimensions. https://openreview.net/forum?id=JfaWawZ8BmX. \nWoodworth et al, 2020. Kernel and rich regimes in overparametrized models, https://arxiv.org/abs/2002.09277.\n\nQUESTIONS:\n* As the authors mention in the conclusion,  Gaussian equivalence theorems could potentially extend these results to more general activations (and losses).  Including this extension would considerably increase the paper's scope. Could the authors provide a bit more details about the extent of the technical gaps to be filled to achieve this ?\n\n* Could the authors comment on the comparisons with the results of Gerace et al, which seem to address the case of general RFMs with any (fixed) feature matrix F satisfying the balance condition (1.7)?\n\n**References**\n\nGerace et al, 2020.  Generalisation error in learning with random features and the hidden manifold model. https://arxiv.org/pdf/2002.09339.pdf\n\nLIMITATIONS:\nAdequately acknowledged.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper derives the learning curves (generalization error vs number of samples) for random linear deep networks with Gaussian weights with non-trivial correlations. The correlations at the first layer control the performance of those networks, while all other layers (possibly of different widths) are equivalent to a single layer of minimal width among them.\n\nSTRENGTHS:\n*\tQuality: the main result that in the noiseless case, correlations at the first layer improve performance, while correlations in the other layers degrade it (Lemmas 4.1 and 4.2) is interesting (and is not fully captured by the summary “representation learning at only the first layer of a deep linear model is sufficient to achieve optimal performance”). The additional result for generalization error of scale-free correlations, which converge with previously known results for uncorrelated weights, is very nice.\n\nWEAKNESSES:\n*\tQuality: it is not justified why the authors focus on the “ridgeless limit”; can’t the minimal generalization error be achieved at a finite value of lambda? Why not?\nThe model studied seems to collapse for large alpha0 (figure 1 a vs b), but the phenomena is not explained.\n *\tClarity: as a theory-heavy paper, the authors could have done a better job in keeping the notation clear. p was not properly defined; I assumed it was the number of samples. The spectral moments generating function from eq 15,17 is introduced only at eq 19. The functions (or scalars) phi and phi bar (evaluated at k0) are not introduced. The important “expectation with respect to the limiting spectral distribution” is not properly defined (“is defined in eq X of the SM” would have been fine as well). \n *\tSignificance: the results of sections 4 and 5 seem to suggest the depth of the network does not contribute anything, as only alpha_min enters the results. This is probably a limitation of the linear model, and thus the discussion on the behaviour of deep networks seems empty. Also, the focus on the noiseless case makes it hard to extrapolate what (if any) of the results hold under noise at an optimal finite lambda.\n\nQUESTIONS:\n* Can the minimal generalization error be achieved at a finite lambda value?\n * Is there any effect of depth when alpha_min is kept fixed?\n\nLIMITATIONS:\nThe authors are clear about the limitations of their work, where only the behaviour for the ridgeless case is studied (and hence the behaviour under noise is not well understood), and only linear networks are studied (and hence the depth does not play an effect), with random initialization (and hence there is no training).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}