SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jacob A. Zavatone-Veth<sup>1,2</sup> and Cengiz Pehlevan<sup>3,2,4</sup>
$^{1}$ Department of Physics,  $^{2}$ Center for Brain Science,
$^{3}$ John A. Paulson School of Engineering and Applied Sciences,
Kempner Institute for the Study of Natural and Artificial Intelligence,
Cambridge, MA 02138, USA
jzavatoneveth@g.harvard.edu, cpehlevan@seas.harvard.edu
In recent years, significant attention in deep learning theory has been devoted to analyzing when models that interpolate their training data can still generalize well to unseen examples. Many insights have been gained from studying models with multiple layers of Gaussian random features, for which one can compute precise generalization asymptotics. However, few works have considered the effect of weight anisotropy; most assume that the random features are generated using independent and identically distributed Gaussian weights, and allow only for structure in the input data. Here, we use the replica trick from statistical physics to derive learning curves for models with many layers of structured Gaussian features. We show that allowing correlations between the rows of the first layer of features can aid generalization, while structure in later layers is generally detrimental. Our results shed light on how weight structure affects generalization in a simple class of solvable models.
Characterizing how data structure and model architecture affect generalization performance is among the foremost goals of deep learning theory [1, 2]. A fruitful line of inquiry has focused on the properties of a class of simplified models that are asymptotically solvable: neural networks in which only the readout layer is trained and other weights are random, which are known as random feature models (RFMs) [3-21]. Though RFMs cannot capture the effects of representation learning on generalization in richly-trained neural networks [13, 22, 23], they have substantially advanced our understanding of how data structure and model architecture interact to give rise to a wide array of generalization phenomena observed in deep learning [1-5, 7-19, 24, 25].
Of particular interest is the question of when models overfit benignly, that is, when they generalize well despite having been trained to perfectly interpolate their training data. Here, much intuition has been gained by studying minimum-norm kernel interpolation—that is, the ridgeless limit of kernel ridge regression—with RFM kernels, for which precise generalization asymptotics can be computed using tools from random matrix theory. These asymptotics lead to a precise picture of how the spectrum of the random feature kernel and the structure of the task interact to determine generalization. These analyses are facilitated by universality results, often termed Gaussian equivalence theorems, that state that the generalization error of a nonlinear RFM is asymptotically equal to that of a linear Gaussian model with an effective noise term resulting from nonlinearity [3, 7, 10, 25, 26]. In the past few years, Gaussian equivalence theorems for ever more general classes of RFMs have been established: within this year Schröder et al. [20] and Bosch et al. [21] have established Gaussian equivalence theorems
for deep nonlinear RFMs with unstructured feature weights, while Cui et al. [27] have extended some of these results to the setting of deep Bayesian neural networks when the target is of the same architecture.
However, these analyses consider the effect only of correlations in the data, and do not address the possibility of correlations between the random weights. It is standard to assume that the elements of the weight matrices at each layer are independent and identically distributed Gaussian random variables, and to our knowledge all existing Gaussian equivalence theorems make use of this assumption [3-15, 19-21]. As a result, how weight anisotropy affects generalization in deep RFMs—in particular, if it can affect the asymptotic scaling of generalization error with dataset size and network width [16, 19, 28]—remains unclear.
In this note, we take the first step towards filling that gap in our theoretical understanding of RFMs by computing the asymptotic generalization error of the simplest class of deep RFMs with anisotropic weight correlations: models with linear activations. Our primary contributions are as follows:
- Using the replica method from statistical mechanics [29], we compute the asymptotic generalization error of deep linear random feature models with weights drawn from general matrix Gaussian distributions. This computation is closely related to prior replica approaches to product random matrix problems [13, 30].
- We show that, in the ridgeless limit, structure in the weights beyond the first layer is detrimental for generalization.
- We next consider the special case of power-law spectra in the weights and in the data, which was classically studied in kernel interpolation in the form of source-capacity conditions [31], and has recently attracted substantial interest in deep learning due to approximate power-law spectra present in real data [16, 19, 28, 32]. Using approximations for required spectral statistics derived in past works [19], we show that altering the power laws of the weight covariance spectra do not affect the scaling laws of generalization.
- We finally show how our results can be extended from the ridge regression estimator to the Bayesian Gibbs estimator, an object of classic study in the statistical physics of learning [13, 33, 34]. For sufficiently large prior variance, structure can be beneficial for generalization with this estimator.
Taken together, these results are consistent with the intuition that representation learning at only the first layer of a deep linear model is sufficient to recover a single teacher weight vector [13, 35-37].
We consider depth- $L$  linear RFMs with input  $\mathbf{x} \in \mathbb{R}^{n_0}$  and scalar output given by
$$
g (\mathbf {x}; \mathbf {v}, \mathbf {F}) = \frac {1}{\sqrt {n _ {0}}} (\mathbf {F} \mathbf {v}) ^ {\top} \mathbf {x}, \tag {1}
$$
where the feature matrix  $\mathbf{F} \in \mathbb{R}^{n_0 \times n_L}$  is fixed and the vector  $\mathbf{v} \in \mathbb{R}^{n_L}$  is trainable. If  $L = 0$ , corresponding to standard linear regression, the feature matrix is simply the identity:  $\mathbf{F} = \mathbf{I}_{n_0}$ . If  $L > 0$ , we take the feature matrix to be defined by a product of  $L$  factors  $\mathbf{U}_\ell \in \mathbb{R}^{n_{\ell-1} \times n_\ell}$ :
$$
\mathbf {F} = \frac {1}{\sqrt {n _ {1} \cdots n _ {L}}} \mathbf {U} _ {1} \dots \mathbf {U} _ {L}. \tag {2}
$$
We draw the random feature matrices independently from matrix Gaussian distributions
$$
\mathbf {U} _ {\ell} \sim \mathcal {M N} _ {n _ {\ell - 1} \times n _ {\ell}} (\mathbf {0}, \boldsymbol {\Gamma} _ {\ell}, \boldsymbol {\Sigma} _ {\ell}) \tag {3}
$$
for input covariance matrices  $\Gamma_{\ell} \in \mathbb{R}^{n_{\ell-1} \times n_{\ell-1}}$  and output covariance matrices  $\Sigma_{\ell} \in \mathbb{R}^{n_{\ell} \times n_{\ell}}$ , such that  $\mathbb{E}[(U_{\ell})_{ij}(U_{\ell'})_{i'j'}] = \delta_{\ell \ell'}(\Gamma_{\ell})_{ii'}(\Sigma_{\ell})_{jj'}$ . Subject to the constraints of layer-wise independence and separability—which are required for the factors to be matrix-Gaussian distributed—this is the most general covariance structure one could consider. One might wish to relax this to include non-separable covariance tensors  $\mathbb{E}[(U_{\ell})_{ij}(U_{\ell'})_{i'j'}] = \delta_{\ell \ell'}(\chi_{\ell})_{ii',jj'}$ , but this would spoil the matrix-Gaussianity of the factors, and to our knowledge does not appear to be addressable using standard
methods [30, 38]. We generate training datasets according to a structured Gaussian covariate model, with  $p$  i.i.d. training examples  $(\mathbf{x}_{\mu}, y_{\mu})$  generated as
$$
\mathbf {x} _ {\mu} \sim_ {\mathrm {i . i . d .}} \mathcal {N} (\mathbf {0}, \boldsymbol {\Sigma} _ {0}), \quad y _ {\mu} = \frac {1}{\sqrt {n _ {0}}} \mathbf {w} _ {*} ^ {\top} \mathbf {x} _ {\mu} + \xi_ {\mu}, \tag {4}
$$
where the teacher weight vector  $\mathbf{w}_{*}$  is fixed and the label noise follows
$$
\xi_ {\mu} \sim_ {\mathrm {i . i . d .}} \mathcal {N} (0, \eta^ {2}). \tag {5}
$$
We collect the covariates into a matrix  $\mathbf{X} \in \mathbb{R}^{p \times n_0}$ , and the targets into a vector  $\mathbf{y} \in \mathbb{R}^p$ .
As in most works on RFMs [3-5, 8-21, 25], our focus is on the ridge regression estimator
$$
\mathbf {v} = \underset {\mathbf {v}} {\arg \min } L \quad \text {f o r} \quad L = \frac {1}{2} \left\| \frac {1}{\sqrt {n _ {0}}} \mathbf {X F v} - \mathbf {y} \right\| ^ {2} + \frac {\lambda}{2} \| \boldsymbol {\Gamma} _ {L + 1} ^ {- 1 / 2} \mathbf {v} \| _ {2} ^ {2}, \tag {6}
$$
where the positive-definite matrix  $\Gamma_{L + 1}\in \mathbb{R}^{n_L\times n_L}$  controls the anisotropy of the norm and the ridge parameter  $\lambda >0$  sets the regularization strength. This minimization problem has the well-known closed form solution
$$
\hat {\mathbf {v}} = \frac {1}{\sqrt {n _ {0}}} \left(\lambda \boldsymbol {\Gamma} _ {L + 1} ^ {- 1} + \frac {1}{n _ {0}} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {X} \mathbf {F}\right) ^ {- 1} \mathbf {F} ^ {\top} \mathbf {X} ^ {\top} \mathbf {y}. \tag {7}
$$
As motivated in the Introduction, we are chiefly interested in the ridgeless limit  $\lambda \downarrow 0$ , in which the ridge regression solution gives the minimum  $\ell_2$  norm interpolant of the training data. We measure performance of this estimator by the generalization error
$$
\epsilon_ {p, n _ {0}, \dots , n _ {L}} = \mathbb {E} _ {\mathbf {x}} \left(g (\mathbf {x}; \hat {\mathbf {v}}, \mathbf {F}) - \mathbb {E} _ {\xi} [ y (\mathbf {x}) ]\right) ^ {2} = \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} (\mathbf {F} \hat {\mathbf {v}} - \mathbf {w} _ {*}) \| ^ {2}, \tag {8}
$$
which is a random variable with distribution induced by the training data and feature weights.
This leads us to a simple, but important observation: including structured input-input covariances is equivalent to transforming the feature-feature covariances. We state this formally as:
Lemma 2.1. Fix sets of matrices  $\{\Gamma_{\ell}\}_{\ell = 1}^{L + 1}$  and  $\{\pmb {\Sigma}_\ell \}_{\ell = 0}^L$  , and a target vector  $\mathbf{w}_{*}$  . Let  $\epsilon_{p,n_0,\dots,n_L}$  be the resulting generalization error as defined in (8). Let
$$
\tilde {\mathbf {I}} _ {\ell} = \mathbf {I} _ {n _ {\ell - 1}} \quad f o r \ell = 1, \dots , L + 1, \tag {9}
$$
$$
\tilde {\boldsymbol {\Sigma}} _ {\ell} = \boldsymbol {\Gamma} _ {\ell + 1} ^ {1 / 2} \boldsymbol {\Sigma} _ {\ell} \boldsymbol {\Gamma} _ {\ell + 1} ^ {1 / 2} \quad f o r \ell = 0, \dots , L, a n d \tag {10}
$$
$$
\tilde {\mathbf {w}} _ {*} = \boldsymbol {\Gamma} _ {1} ^ {- 1 / 2} \mathbf {w} _ {*}. \tag {11}
$$
Let  $\tilde{\epsilon}_{p,n_0,\dots,n_L}$  be the generalization error for these transformed covariance matrices and target. Then, for any  $\lambda > 0$ , we have the equality in distribution  $\epsilon_{p,n_0,\dots,n_L} \stackrel{d}{=} \tilde{\epsilon}_{p,n_0,\dots,n_L}$ .
Proof of Lemma 2.1. As the features and data are Gaussian, we can write  $\mathbf{X} \stackrel{d}{=} \pmb{\Sigma}_0^{1/2}\mathbf{Z}_0$  and  $\mathbf{U}_{\ell} \stackrel{d}{=} \Gamma_{\ell}^{1/2}\mathbf{Z}_{\ell}\pmb{\Sigma}_{\ell}^{1/2}$  for unstructured Gaussian matrices  $(Z_{\ell})_{ij} \sim_{\mathrm{i.i.d.}} \mathcal{N}(0,1)$ . Substituting these representations into the ridge regression solution (7) and the generalization error (8), the claim follows.
Therefore, we may take  $\Gamma_{\ell} = \mathbf{I}_{n_{\ell -1}}$  without loss of generality. Moreover, thanks to the rotation-invariance of the isotropic Gaussian factors  $\mathbf{Z}_{\ell}$ , we may in fact take the remaining covariance matrices  $\pmb{\Sigma}_{\ell}$  to be diagonal without loss of generality, so long as we then express  $\tilde{\mathbf{w}}_*$  in the basis of eigenvectors of  $\pmb{\Sigma}_0$ . An important qualitative takeaway of this result is that changing the covariance matrix of the inputs of the first layer  $\Gamma_1$  is equivalent to modifying the data covariance matrix, which was in a simpler form observed in the shallow setting ( $L = 1$ ) by Pandey et al. [39].
3 Asymptotic learning curves
Having defined the setting of our problem, we can define our concrete objective and state our main results, deferring their interpretation to the following section. We consider the standard proportional asymptotic limit
$$
p, n _ {0}, \dots , n _ {L} \rightarrow \infty , \quad \text {w i t h} \quad n _ {\ell} / p \rightarrow \alpha_ {\ell} \in (0, \infty), \tag {12}
$$
which we will refer to as the thermodynamic limit. Our goal is to compute the limiting generalization error:
$$
\epsilon = \lim  _ {p, n _ {0}, \dots , n _ {L} \rightarrow \infty} \mathbb {E} _ {\mathcal {D}} \frac {1}{n _ {0}} \| \boldsymbol {\Sigma} _ {0} ^ {1 / 2} \left(\mathbf {F} \mathbf {v} - \mathbf {w} _ {*}\right) \| ^ {2}, \tag {13}
$$
where  $\mathbb{E}_{\mathcal{D}}$  denotes expectation over all sources of quenched disorder in the problem, i.e., the training data and the random feature weights. In the thermodynamic limit, we expect the generalization error to concentrate, which is why we compute its average in (13) [3-5, 8-21].
To have a well-defined thermodynamic limit, the covariances  $\tilde{\Sigma}_{\ell}$  and the teacher  $\tilde{\mathbf{w}}_{\ell}$  must be in some sense sufficiently well-behaved. We consider the following conditions, which are the generalization to our setting of those assumed in previous work [4-7, 16-18, 40]:
Assumption 3.1. We assume that we are given deterministic sequences of positive-definite matrices  $\tilde{\Sigma}_{\ell}(n_{\ell})$  and vectors  $\tilde{\mathbf{w}}_*(n_0)$  indexed by the system size, such that the limiting (weighted) spectral moment generating functions
$$
M _ {\tilde {\mathbf {S}} _ {\ell}} (z) = \lim  _ {n _ {\ell} \rightarrow \infty} \frac {1}{n _ {\ell}} \operatorname {t r} \left[ \tilde {\mathbf {S}} _ {\ell} \left(z \mathbf {I} _ {n _ {\ell}} - \tilde {\mathbf {S}} _ {\ell}\right) ^ {- 1} \right] \quad a n d \quad \psi (z) = \lim  _ {n _ {0} \rightarrow \infty} \frac {1}{n _ {0}} \tilde {\mathbf {w}} _ {*} ^ {\top} \tilde {\mathbf {S}} _ {0} \left(z \mathbf {I} _ {n _ {0}} + \tilde {\mathbf {S}} _ {0}\right) ^ {- 1} \tilde {\mathbf {w}} _ {*} \tag {14}
$$
are well-defined, for all  $\ell = 0,\dots ,L$
We can now state our results. As a preliminary step, we first give an expression for the generalization error for a fixed teacher  $\tilde{\mathbf{w}}_*$  at finite ridge  $\lambda$ . Then, we pass to the ridgeless limit, on which we focus for the remainder of the paper. At finite ridge, we have the following:
Proposition 3.1. Assume Assumption 3.1 holds. For  $\lambda >0$ , let  $\zeta$  solve the self-consistent equation
$$
\lambda = \frac {1 - \zeta}{\zeta} \prod_ {\ell = 0} ^ {L} \frac {- \zeta}{\alpha_ {\ell}} M _ {\hat {\Sigma} _ {\ell}} ^ {- 1} \left(- \frac {\zeta}{\alpha_ {\ell}}\right). \tag {15}
$$
In terms of  $\zeta$ , let  $\kappa_{\ell}(\zeta)$  solve
$$
\mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} (\zeta) + \tilde {\sigma} _ {\ell}} \right] = - M _ {\tilde {\Sigma} _ {\ell}} (- \kappa_ {\ell} (\zeta)) = \frac {\zeta}{\alpha_ {\ell}} \tag {16}
$$
for  $\ell = 0,\dots ,L$ , where  $\mathbb{E}_{\tilde{\sigma}_{\ell}}[h(\tilde{\sigma}_{\ell})] = \lim_{n_{\ell}\to \infty}n_{\ell}^{-1}\sum_{j = 1}^{n_{\ell}}h(\tilde{\sigma}_{\ell ,j})$  denotes expectation of a function  $h$  with respect to the limiting spectral distribution of  $\tilde{\Sigma}_{\ell}$ , for  $\tilde{\sigma}_{\ell ,j}$  its eigenvalues at finite size, and let
$$
\mu_ {\ell} (\zeta) = - \frac {\alpha_ {\ell}}{\zeta} \kappa_ {\ell} (\zeta) M _ {\tilde {\mathbf {S}} _ {\ell}} ^ {\prime} (- \kappa_ {\ell} (\zeta)) = 1 - \frac {\alpha_ {\ell}}{\zeta} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \left(\frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} (\zeta) + \tilde {\sigma} _ {\ell}}\right) ^ {2} \right]. \tag {17}
$$
Then, the learning curve (13) at finite ridge for a fixed target is given by
$$
\left[ 1 + \left(\sum_ {\ell = 0} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) (1 - \zeta) \right] \epsilon = \left(\sum_ {\ell = 1} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) \kappa_ {0} \psi (\kappa_ {0}) - \frac {\kappa_ {0} ^ {2}}{\mu_ {0}} \psi^ {\prime} (\kappa_ {0}) + \left(\sum_ {\ell = 0} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) \zeta \eta^ {2}. \tag {18}
$$
Proof of Proposition 3.1. We defer the derivation of (18) to Appendix A. To compute the disorder average in (13), we express the minimization problem in (6) as the zero-temperature limit  $\beta \rightarrow \infty$  of an auxiliary Gibbs distribution  $p(\mathbf{v})\propto e^{-\beta L}$ , and evaluate the average over the random data random feature weights using the non-rigorous replica method from the statistical mechanics of disordered systems [29, 33]. This computation is lengthy but standard, and is closely related to the approach used in our previous works on deep linear models [13, 30]. All of our results are obtained under a replica-symmetric Ansatz; as the ridge regression problem (6) is convex, we expect replica symmetry to be unbroken [29, 41, 42].
From the self-consistent equation (15), we recognize that  $\zeta$  is up to a sign the spectral moment generating function of the feature Gram matrix  $\mathbf{K} = \mathbf{X}\mathbf{F}\mathbf{F}^{\top}\mathbf{X}^{\top} / n_{0}$ , which is a product-Wishart random matrix [30]:
$$
\zeta (\lambda) = - M _ {\mathbf {K}} (- \lambda). \tag {19}
$$
This dependence falls out of the replica computation of the generalization error using an auxiliary Gibbs distribution; we emphasize that one could take an alternative approach in which the generalization error is first expressed in terms of  $M_{\mathbf{K}}$  as, for instance, in Gerace et al. [25] or Hastie et al.
[5] and then use results on the spectra of product-Wishart matrices to conclude the claimed result [30]. This approach would potentially have the advantage of giving a fully rigorous proof, rather than one that depends on the replica trick. However, one would still then be faced with the task of solving the self-consistent equation for the spectral moment generating function, and therefore would end up in the same place insofar as quantitative predictions are concerned.
In principle, we could now directly proceed to study how weight structure affects (18) for some fixed ridge  $\lambda$ . However, as long as there is structure in the weights and/or the data, the self-consistent equation (15) must generally be solved numerically [14, 30]. To allow us to make analytical progress, we therefore focus on the ridgeless limit  $\lambda \downarrow 0$  for the remainder of the present paper, and leave careful analysis of the  $\lambda > 0$  case to future work. This follows the path of most recent studies of models with linear random features, and also the fundamental interest in interpolating models [3-17, 19-21]. We therefore emphasize that we state Proposition 3.1 merely as a preliminary result.
Before giving our result for the generalization error in the ridgeless limit, we warn the reader of an impending, somewhat severe abuse of notation: in Proposition 3.2 and for the remainder of the paper, we will re-define  $\kappa_{\ell}$  to be given by its value for the solution for  $\zeta$  appropriate in the regime of interest. Moreover, we will simply write  $\epsilon$  for  $\lim_{\lambda \downarrow 0} \epsilon$ .
Proposition 3.2. Assume Assumption 3.1 holds, and let  $\alpha_{\mathrm{min}} = \min \{\alpha_1,\dots ,\alpha_L\}$ . For  $\ell = 0,\ldots ,L$  in the regime  $\alpha_{\ell} > 1$ , let  $\kappa_{\ell}$  be given by the unique non-negative solution to the implicit equation
$$
\frac {1}{\alpha_ {\ell}} = - M _ {\tilde {\mathbf {S}} _ {\ell}} (- \kappa_ {\ell}) = \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}} \right]. \tag {20}
$$
In terms of  $\kappa_{\ell}$ , let
$$
\mu_ {\ell} = - \alpha_ {\ell} \kappa_ {\ell} M _ {\boldsymbol {\Sigma} _ {\ell}} ^ {\prime} (- \kappa_ {\ell}) = 1 - \alpha_ {\ell} \mathbb {E} _ {\tilde {\sigma} _ {\ell}} \left[ \left(\frac {\tilde {\sigma} _ {\ell}}{\kappa_ {\ell} + \tilde {\sigma} _ {\ell}}\right) ^ {2} \right]. \tag {21}
$$
In the regime  $\alpha_{\mathrm{min}} < \alpha_0$ , let  $\kappa_{\mathrm{min}}$  be the unique non-negative solution to the implicit equation
$$
\frac {\alpha_ {\operatorname* {m i n}}}{\alpha_ {0}} = - M _ {\tilde {\Sigma} _ {0}} (- \kappa_ {\operatorname* {m i n}}) = \mathbb {E} _ {\tilde {\sigma} _ {0}} \left[ \frac {\tilde {\sigma} _ {0}}{\kappa_ {\operatorname* {m i n}} + \tilde {\sigma} _ {0}} \right]. \tag {22}
$$
Then, the learning curve (13) for a fixed target in the ridgeless limit  $\lambda \downarrow 0$  is given by
$$
\epsilon = \left\{ \begin{array}{l l} \left(\sum_ {\ell = 1} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) \kappa_ {0} \psi (\kappa_ {0}) - \frac {\kappa_ {0} ^ {2}}{\mu_ {0}} \psi^ {\prime} (\kappa_ {0}) + \left(\sum_ {\ell = 0} ^ {L} \frac {1 - \mu_ {\ell}}{\mu_ {\ell}}\right) \eta^ {2}, & \alpha_ {0}, \alpha_ {\min } > 1 \\ \frac {\kappa_ {\min } \psi (\kappa_ {\min })}{1 - \alpha_ {\min }} + \frac {\alpha_ {\min }}{1 - \alpha_ {\min }} \eta^ {2}, & \alpha_ {\min } <   1, \alpha_ {\min } <   \alpha_ {0} \\ \frac {\alpha_ {0}}{1 - \alpha_ {0}} \eta^ {2}, & \alpha_ {0} <   1, \alpha_ {0} <   \alpha_ {\min }. \end{array} \right. \tag {23}
$$
Proof of Proposition 3.2. We derive (23) as the zero-ridge limit of Proposition 3.1 in Appendix A.
Before we analyze the effect of weight anisotropy in detail in Section 4, we note several simplifying special cases of Proposition 3.2 which recover the results of prior works. To facilitate this comparison, we provide a notational dictionary in Appendix D. The first important special case is
Corollary 3.1. If  $L = 0$ , we have
$$
\epsilon = \left\{ \begin{array}{l l} - \frac {\kappa_ {0} ^ {2}}{\mu_ {0}} \psi^ {\prime} \left(\kappa_ {0}\right) + \frac {1 - \mu_ {0}}{\mu_ {0}} \eta^ {2}, & \alpha_ {0} > 1 \\ \frac {\alpha_ {0}}{1 - \alpha_ {0}} \eta^ {2}, & \alpha_ {0} <   1. \end{array} \right. \tag {24}
$$
This recovers the known, rigorously proved result for linear ridgeless regression [4-7, 16-18]. For larger depths, an important simplifying case of Proposition 3.2 is that in which the data and features are unstructured, in which case the generalization error is given by
Corollary 3.2. If  $\tilde{\mathbf{S}}_{\ell} = \mathbf{I}_{n_{\ell}}$  for  $\ell = 0,\dots ,L$ , we have, for any target satisfying  $\| \tilde{\mathbf{w}}_*\|^2 = n_0$
$$
\epsilon = \left\{ \begin{array}{l l} \left(1 + \sum_ {\ell = 1} ^ {L} \frac {1}{\alpha_ {\ell} - 1}\right) \left(1 - \frac {1}{\alpha_ {0}}\right) + \left(\sum_ {\ell = 0} ^ {L} \frac {1}{\alpha_ {\ell} - 1}\right) \eta^ {2}, & \alpha_ {0}, \alpha_ {\min } > 1 \\ \frac {1 - \alpha_ {\min } / \alpha_ {0}}{1 - \alpha_ {\min }} + \frac {\alpha_ {\min }}{1 - \alpha_ {\min }} \eta^ {2}, & \alpha_ {\min } <   1, \alpha_ {\min } <   \alpha_ {0} \\ \frac {\alpha_ {0}}{1 - \alpha_ {0}} \eta^ {2}, & \alpha_ {0} <   1, \alpha_ {0} <   \alpha_ {\min }. \end{array} \right. \tag {25}
$$
Figure 1: Phase diagram of generalization in deep linear RFMs. For simplicity, we consider a model with a single hidden layer  $(L = 1)$ ; the picture for deeper models is identical if one considers the narrowest hidden layer [13]. (a). Generalization error  $\epsilon$  for unstructured data and features from (25) as a function of training data density  $1 / \alpha_0$  and hidden layer width  $\alpha_{1} / \alpha_{0}$  in the absence of label noise  $(\eta = 0$ ; left) and in the presence of label noise  $(\eta = 0.5$ ; right). (b). As in (a), but for power law structured data and weights, with  $\omega_0 = \omega_1 = 1$ , and  $\bar{\epsilon}$  given by (31). See Appendix F for numerical methods.