Year: 2021

Abstract:
In contrast to standard statistical wisdom, modern learning algorithms typically find their best performance in the overparameterized regime in which the model has many more parameters than needed to fit the training data. A growing number of recent works have shown that random feature models can offer a detailed theoretical explanation for this unexpected behavior, but typically these analyses have utilized isotropic distributional assumptions on the underlying data generation process, thereby failing to provide a realistic characterization of real-world models that are designed to identify and harness the structure in natural data. In this work, we examine the high-dimensional asymptotics of random feature regression in the presence of structured data, allowing for arbitrary input correlations and arbitrary alignment between the data and the weights of the target function. We define a partial order on the space of weight-data alignments and prove that generalization performance improves in response to stronger alignment. We also clarify several previous observations in the literature by distinguishing the behavior of the samplewise and parameter-wise learning curves, finding that sample-wise multiple descent can occur at scales dictated by the eigenstructure of the data covariance, but that parameter-wise multiple descent is limited to double descent, although strong anisotropy can induce additional signatures such as wide plateaus and steep cliffs. Finally, these signatures are related to phase transitions in the spectrum of the feature kernel matrix, and unlike the double descent peak, persist even under optimal regularization.

heading: 1 INTRODUCTION
text: While deep learning races ahead with its impressive list of practical successes, it has often done so by building larger models and training on ever larger datasets. For example, recent large language models leverage billions (Brown et al., 2020; Raffel et al., 2019) or even trillions (Fedus et al., 2021) of parameters. As the state-of-the-art pushes the computational frontier, model development, experimentation, and exploration become increasingly costly, both in terms of the computational resources and person-hours.
To facilitate the development of improved models, it is becoming ever more necessary to develop theoretical models that can help guide model development in these high-dimensional scenarios. Indeed, our theoretical understanding is so limited that even linear regression, the most basic machine learning method of all, is still yielding surprises in high dimensions (Hastie et al., 2019).
While linear regression can capture many salient features of high-dimensional data, it is not a reasonable model for high-dimensional models. In particular, there is no natural data-independent notion of model size: the number of parameters in the model is intimately tied (usually equal) to the dimensionality of the datapoints. This limitation means that linear regression may not accurately capture the effect of overparameterization, which appears to be a crucial ingredient in modern deep learning models (Zhang et al., 2016).
Beyond linear regression, the next simplest model is arguably the random feature model of Rahimi et al. (2007), which turns out to be tractable in high dimensions while also allowing for a natural data-independent lever to vary model size – the number of random features. Random feature models have also gained independent interest through their relationship to neural kernels and Gaussian processes (Neal, 1996; Lee et al., 2017; Novak et al., 2018; Lee et al., 2019; Jacot et al., 2018). While
recent works have examined the asymptotic performance of random feature models, most have done so in the restricted setting of isotropic covariates and target functions (Mei & Montanari, 2019; d’Ascoli et al., 2020; Adlam & Pennington, 2020a;b). Based on the conjectured Gaussian equivalence principle of Goldt et al. (2020b), a few recent works have begun relaxing these simplifying assumptions in an effort to describe more natural data distributions (Loureiro et al., 2021; d’Ascoli et al., 2021). In this work, we study high-dimensional random feature regression for anisotropic data and target functions, setting the results of d’Ascoli et al. (2021) on a more rigorous footing and revealing a host of novel phenomena related to the alignment of the data and the target weights.

heading: 1.1 CONTRIBUTIONS
text: Our primary contributions are to:
1. Derive theoretical predictions for the test error, bias, and variance of random feature regression for anisotropic Gaussian covariates under general linear target functions in the high-dimensional asymptotic limit (Section 3.1);
2. Prove that overparameterization reduces the total test error, bias, and variance, even in the presence of anisotropy (Section 3.2);
3. Define a partial order on the space of weight-data alignments and prove that generalization performance improves in response to stronger alignment (Definition 2.1, Section 3.3);
4. Demonstrate that anisotropy can induce sample-wise multiple descent, but prove that parameter-wise multiple descent is limited to double descent (Section 3.4);
5. Provide a theoretical explanation for steep cliffs observed in parameter-wise learning curves that persist under optimal regularization, and relate them to phase transitions in the spectrum of the feature kernel matrix (Section 3.4).

heading: 1.2 RELATED WORK
text: Our analysis builds on a series of works that have studied the exact high-dimensional asymptotics of the test error for a growing class of model families and data distributions. Early work in this direction examined minimum-norm interpolated least squares and ridge regression in the high-dimensional random design setting (Belkin et al., 2019; Dobriban et al., 2018; Hastie et al., 2019), finding surprising phenomena such as double descent and a negative optimal ridge constant (Kobak et al., 2020). These analyses were generalized in (Richards et al., 2021; Wu & Xu, 2020; Hastie et al., 2019) to allow for anisotropic covariance matrices and weights that generate the targets. This anisotropic configuration was studied in further detail in (Mel & Ganguli, 2021), uncovering nuanced behavior such as sample-wise multiple descent and a sequence of phase transitions in the performance and optimal regularization.
While linear regression can provide valuable insight into generalization performance in high dimensions, it is not sufficiently complex to faithfully reproduce the features of non-linear models, such as the effects of overparameterization. Random feature models, on the other hand, can capture many such features while still maintaining analytical tractability. Recent results have developed a detailed understanding of isotropic random feature regression in the high-dimensional asymptotic limit, providing a precise characterization of peaks in the test error and the benefits of overparameterization (Mei & Montanari, 2019; d’Ascoli et al., 2020; Adlam & Pennington, 2020a). The origin of these peaks was explained by means of a fine-grained decomposition of the bias and variance in (Adlam & Pennington, 2020b; Lin & Dobriban, 2020). While most prior work examining the high-dimensional asymptotics of random feature models have leveraged the isotropy of the covariate distribution, this assumption was recently relaxed in (Liao et al., 2020) for random Fourier feature models, in (Anonymous, 2021) in the study of covariate shift, and in a series of works based on the Gaussian equivalence principle of Goldt et al. (2020a;b) (Gerace et al., 2020; Loureiro et al., 2021).
Though most of these works did not pursue a detailed investigation of the anisotropy itself, while finalizing this manuscript we became aware of concurrent work (d’Ascoli et al., 2021) which does focus on the interplay of the anisotropic input features and target functions, as we do here. Whereas d’Ascoli et al. (2021) leverage the Gaussian equivalence principle and the replica method from statistical physics, we use a completely different set of techniques stemming from the literature
on random matrix theory and operator-valued free probability (Pennington & Worah, 2018; 2019; Adlam et al., 2019; Adlam & Pennington, 2020a; Louart et al., 2018; Péché et al., 2019; Far et al., 2006; Mingo & Speicher, 2017), which can be made completely rigorous. d’Ascoli et al. (2021) investigates the role of anisotropy not just under squared loss, as we do here, but also under logistic loss, and obtains good numerical support for the Gaussian equivalence conjecture in that setting. In contrast, our narrower and deeper focus on squared loss allows us not only to uncover a host of novel phenomena, including multiple descent, steep error cliffs, and spectral phase changes, but also to rigorously prove several propositions about their existence and properties. We provide a more in-depth discussion of these and other related works in App. A.
Our work relies heavily on the concept of alignment (see Def. 2.1) between the data covariance and the target vector coefficients. A completely analogous definition appears in Anonymous (2021) in the context of covariate shift. Owing to the similarity of these definitions as well as the underlying formulas, many of the techniques and analyses of Anonymous (2021) can be applied in our context, leading to a kind of mirroring of results. As the underlying implications and interpretations are otherwise quite different, we refrain from repeatedly commenting on these recurring connections.

heading: 2 PRELIMINARIES

heading: 2.1 PROBLEM SETUP AND NOTATION
text: We study random feature regression (Rahimi et al., 2007) as a model for learning an unknown linear function of the data,
y(xi) = >xi/ p n0 + ✏i , (1)
from m independent samples (xi, yi) 2 Rn0 ⇥ R, i = 1, . . . ,m, where the covariates are Gaussian, xi ⇠ N (0,⌃), and where ✏i ⇠ N (0, 2✏ ) is additive noise (that we take to be present on the training samples only).
We focus on the high-dimensional proportional asymptotics in which the input feature dimension n0, the hidden layer size n1, and the number of samples m all tend to infinity at the same rate, with := n0/m and := n0/n1 held constant. The overparameterization ratio / = n1m is a measure of the normalized complexity of the random feature model.
Interestingly, under this model in our high-dimensional setup, any nonlinear component of the signal to be learned behaves like additive noise, so the linear function in Eq. (1) does not sacrifice generality (Mei & Montanari, 2019; Adlam & Pennington, 2020a).
Following many works on high-dimensional regression (see e.g. Dobriban et al. (2018)), we assume the coefficient vector is random, ⇠ N (0,⌃ ), though allowing for deterministic would be a straightforward extension and would simply involve replacing ⌃ ! > throughout. Denoting the training dataset as X = [x1, . . . ,xm] and the test point as x, the random features are,
F := (WX/ p n0) and f := (Wx/ p n0) , (2)
where W 2 Rn1⇥n0 is a random weight matrix with i.i.d. standard Gaussian entries, and (·) : R ! R is an activation function that is applied elementwise. These random features define the kernel
K(x1,x2) := 1
n1 (Wx1/
p n0) > (Wx2/ p n0) , (3)
which can be used to compute the model’s predictions on a point x as
ŷ(x) = Y K 1Kx . (4)
Here we have defined the training labels as Y := [y(x1), . . . , y(xm)], and the regularized kernel matrix as K := K(X,X) + Im, as well as Kx := K(X,x) for 0. The training error is given by,
E⌃train = E E[(y(X) ŷ(X))2] = E[( >X/ p n0 Y K 1K(X,X))2] , (5)
and the test error (without label noise on the test point) can be written as
E⌃ = E ExE[(y(x) ŷ(x))2] 2✏ = ExE[( >x/ p n0 Y K 1Kx)2] , (6)
where the outermost expectation over has been suppressed since the quantity concentrates sharply around its mean and the inner expectation is computed over all the randomness from training, i.e. W , X , and ✏. The test error can be decomposed into its bias and variance components as
E⌃ = ExE[(E[ŷ(x)] y(x))2]| {z } B⌃ +Ex[V[ŷ(x)]]| {z } V⌃ . (7)
The bias and variance are computed with respect to all the randomness from training, i.e. W , X , ✏, which differs from common practice in the statistics literature, but which is necessary to obtain a decomposition that is intuitive and unambiguous when the predictive function relies on randomness from multiple sources (Adlam & Pennington, 2020b; Lin & Dobriban, 2020).
As noted by (Hastie et al., 2019; Wu & Xu, 2020; Mel & Ganguli, 2021), the test error of linear regression depends on the geometry of (⌃,⌃ ) (or (⌃, ) in the case of nonrandom ), and we will find the same to be true for random feature regression. As such, we decompose the training and covariance matrices into eigenbases as ⌃ = Pn0 i=1 ivivi > and ⌃ = Pn0 i=1 i v i v i > , where the eigenvalues are in nondecreasing magnitude, i.e. 1  2  . . .  n0 and 1  2  . . .  n0 . Following (Anonymous, 2021), we then define the overlap coefficients as
qi := v > i ⌃ vi =
n0X
j=1
(v j · vi) 2 j , (8)
to measure the alignment of ⌃ with the ith eigendirection of ⌃. Note that for deterministic this definition simply gives the square of the component of in the ith eigendirection, i.e. qi = (v>i )2.

heading: 2.2 ASSUMPTIONS
text: In order to define the limiting behavior of the test error in Eq. (6), it is necessary to impose some regularity conditions on ⌃ and ⌃ .1 As in (Wu & Xu, 2020; Anonymous, 2021), the limiting spectra of these matrices cannot be considered independently because their eigenspaces may align. This alignment is most conveniently described in an eigenbasis of ⌃. Assumption 1. We define the joint spectral distribution (JSD) as
µn0 := 1
n0
n0X
i=1
( i,qi) (9)
and assume it converges in distribution to some µ, a distribution on R2+ as n0 ! 1. We refer to µ as the limiting joint spectral distribution (LJSD), and emphasize that this defines the relevant limiting properties of the covariate and weight distributions 2 .
Often we use ( , q) for random variables sampled jointly from µ and denote the marginal of under µ with µdata. Since the conditional expectation E[q| ] is an important object in our study, we assume the following for simplicity. Assumption 2. µ is either absolutely continuous or a finite sum of delta masses and the expectations of and q are finite. Moreover, Eµ[ q] = 1.
The normalization condition Eµ[ q] = 1 is not necessary for our analysis, but, as in (Mel & Ganguli, 2021), it enforces consistent signal-to-noise ratios across different target functions and thereby facilitates meaningful comparisons of the test error.
As we will eventually consider the high-dimensional limit, it is convenient to define the asymptotic covariance scale as s = limn0!1 1 n0
tr(⌃) = Eµ[ ] under the limiting behavior specified in Assumption 1.
1In fact, the necessary assumptions are identical to those of Anonymous (2021), since ⌃ plays an analogous role to the test covariance ⌃⇤ of that work.
2The JSD depends not only on ⌃ and ⌃ but also on a choice of eigendecomposition for ⌃ when it has repeated eigenvalues; however, as in (Anonymous, 2021), all possible choices lead to the same formulas and conclusions.
One important case is when ⌃ = In0 , in which case the LJSD degenerates to µ; defined by µ;( , q) := µdata( ) 1(q) . (10)
Following Anonymous (2021), we also enforce the following standard regularity assumptions on the activation functions to ensure the existence of the moments and derivatives we compute. Assumption 3. The activation function : R ! R is assumed to be differentiable almost everywhere. We assume there exists a universal constant C such that, | (x)|, | 0(x)| = O(exp(Cx)).

heading: 2.3 A SIMPLE FAMILY OF ANISOTROPIC DISTRIBUTIONS
text: The assumptions above allow for a wide class of possible asymptotic covariance structures and spectra. To allow for concrete examples that have intuitive interpretations, we introduce the following simple family of distributions that capture multi-scale structure in the input, which we refer to as d-scale LJSDs. In particular, for d 2 N and real 0 < ↵  1 and ✓, we define
µd-scale↵,✓ := d 1X
j=0
1 d (C↵j ,D↵✓j), (11)
where C := ⇣
1 d 1 ↵d 1 ↵ ⌘ 1 and D := 1C ⇣ 1 d 1 ↵(✓+1)d 1 ↵(✓+1) ⌘ 1 enforce the normalization conditions
s = Eµd-scale↵,✓ [ ] = 1 and Eµd-scale↵,✓ [q ] = 1.
These simple covariance models capture the hierarchy of scales that often characterize the structure of natural datasets. Note that by setting ↵ = 1 and s = 1, the distribution reduces to the isotropic case with identity covariance. For the nontrivial setting in which ↵ < 1, the data exhibits a sequence of d distinct scales where each scale is a factor ↵ smaller than the previous one. The exponent ✓ parameterizes the strength of the weight-data alignment in an intuitive way. When ✓ = 0, there is no alignment, and ⌃ is proportional to the identity. When ✓ > 0, there is strong alignment, as the large eigendirections of the training distribution correspond to large eigendirections of ⌃ . When ✓ < 0, there is anti-alignment, as the large eigendirections of the training distribution correspond to small eigendirections of ⌃ .

heading: 2.4 DEFINITION OF THE STRENGTH OF WEIGHT-DATA ALIGNMENT
text: As the preceding discussion has suggested and as we will see explicitly in the following section, the LJSD µ captures all of the information about the pair of covariance matrices (⌃, ⌃ ) that is relevant for describing the asymptotic test error, bias, and variance. As the alignment between ⌃ and ⌃ can vary in strength among the different eigendirections, the concept of alignment is inherently multi-dimensional. Nevertheless, by requiring strong alignment along the larger eigendirections, it is possible to define the following natural partial order on the space of possible weight-data alignments3
Definition 2.1. Let µ1 and µ2 be LJSDs with the same marginal distribution of . If the asymptotic overlap coefficients are such that Eµ1 [ q| ] /Eµ2 [ q| ] = Eµ1 [q| ] /Eµ2 [q| ] is nondecreasing in , we say that µ1 is more strongly aligned than µ2 and write µ1  µ2. Comparing against the case of isotropic weight distribution, µ;, we say µ1 is aligned when µ1  µ; and anti-aligned when µ1 µ;.
The parameter ✓ of the d-scale model in Eq. (11) provides a quantitative measure of alignment under Definition 2.1, formalizing the intuition given in Section 2.3. As we will see in Section 3.3, the asymptotic generalization performance of random feature regression is strictly ordered under this definition of alignment strength.

heading: 3 MAIN RESULTS
text: Our main result characterizes the high-dimensional asymptotic limits of the test error, bias, and variance of the nonlinear random feature model of Section 2. Before stating the result, we introduce
3Note that this definition is nearly identical to that of Anonymous (2021), with the concept of hardness replaced by alignment.
Etrain/Etrain(✓=0) , and similarly for Bµ, Vµ). Same legend as (e). Constancy of the blue traces
illustrates how the train and test error respond identically to alignment. Simulations for m = 4000 (d,e; crosses) agree well with formulas.
some additional notation that captures the effect of the nonlinearity , ⌘ := Vz⇠N (0,s)[ (z)] , ⇢:= ( 1sEz⇠N (0,s)[z (z)])
2 , ⇣ := s⇢ , !:= s(⌘/⇣ 1) . (12) where, as above, s = limn0!1 1 n0
tr(⌃) = Eµ[ ]. The constant ! 0 is a measure of the degree of nonlinearity, with ! = 0 corresponding to linear activation functions (see Lemma B.1). Analogously to Anonymous (2021), we also introduce two functionals of µ, which capture all the relevant spectral information needed for describing the test loss,
Ia,b := Eµ ⇣ a ( + x ) b ⌘ and I a,b := Eµ ⇣ q a ( + x ) b ⌘ . (13)

heading: 3.1 FORMULAS FOR ASYMPTOTIC BIAS, VARIANCE, AND TEST ERROR
text: Theorem 3.1. Under Assumptions 1-3, as n0, n1,m ! 1 with = n0/m and = n0/n1 held constant, the training error E⌃
train tends toward the value of Eµ train where
Eµ train = 2 ⇣ @ (⌧1I 1,1) + 2"@ ⌧1 ⌘ , (14)
and the bias, variance, and test error B⌃ , V⌃ , E⌃ tend toward Bµ, Vµ, Eµ with
Eµ = Eµ train
2⌧21 2" , Bµ = I 1,2 , and Vµ = Eµ Bµ , (15)
where ⇢,! are defined in (12), Ia,b, I a,b are defined in (13), and x is the unique nonnegative real root of x = 1 ⌧1!+I1,1 with ⌧1 = p ( )2+4x /⇢+ 2 .
Remark 3.1. As noted in (Adlam & Pennington, 2020a; Hastie et al., 2019) for isotropic covariates,
the test error is simply related to the training error through the generalized cross-validation (GCV)
formula (Golub et al., 1979). Curiously, because ⌧1 is independent of , it follows that the test error depends on entirely through its effect on the training error; however, the bias-variance decomposition retains explicit dependence, even conditional on the training error.
The results in Theorem 3.1 depend on a single scalar self-consistent equation for x, x = 1 ⌧1!+I1,1 , which is in fact the same as the one appearing in Anonymous (2021), and which significantly simplifies the expressions relative to those recently obtained using the replica method (d’Ascoli et al., 2021). Owing to its simplicity, this equation admits straightforward analysis that we pursue in the Appendix, yielding numerous inequalities and bounds that ultimately prove the propositions presented throughout this section.
We will on occasionally refer to the ridgeless limit of Theorem 3.1, which follows by carefully expanding the result for small and is given in Corollary G.1.
Finally, as a consistency check, taking (x) = x and ! 0 in Theorem 3.1 yields an expression for the test error, bias, and variance for linear regression that agrees with the results of (Wu & Xu, 2020; Mel & Ganguli, 2021) (see Section E).

heading: 3.2 THE BENEFITS OF OVERPARAMETERIZATION
text: While there is abundant empirical evidence that overparameterization can improve the generalization performance of practical models (see e.g. (Zhang et al., 2016)), rigorous explanations for this behavior have been offered solely in the setting of isotropic covariates and target functions. It is therefore natural to wonder whether the benefits of overparmeterization persist in the presence of anisotropy, and indeed recent theoretical work has provided numerical evidence hinting that overparameterization is beneficial (d’Ascoli et al., 2021). The following two results prove this to be the case.
First, we show that the bias decreases (or stays constant) in response to an increase in the number of random features, regardless of any anisotropy in the covariates or target function weights.
Proposition 3.1. In the setting of Theorem 3.1, the bias Bµ is a nonincreasing function of overparameterization ratio / .
The same is true for the variance in the overparameterized regime. Note that our proof requires the ridgeless setting, but numerical simulations suggest that the result may hold generally (see Fig. 1). Proposition 3.2. In the ridgeless limit and in the overparameterized regime ( < ), the variance

heading: Vµ is a nonincreasing function of overparameterization ratio / .
text: In Fig. 1, Propositions 3.1 and 3.2 are illustrated. Moving left to right in panels (a)-(d) leads to models with more parameters. The monotonicity of the bias across the whole range of parameterization is evident, as is the monotonicity of the variance in the overparmaeterized regime.

heading: 3.3 WEIGHT-DATA ALIGNMENT REDUCES THE BIAS AND TEST ERROR
text: A number of recent works have confirmed the intuition that generalization performance should improve if the weights of the target function align with the large eigendirections of the training covariance, with formal theoretical arguments in the case of linear regression (Hastie et al., 2019; Mel & Ganguli, 2021), and with informal numerical simulations for random feature models (d’Ascoli et al., 2021). The following result proves that this informal observation holds in full generality, not only for the total test error, but for the bias and the bias-to-variance ratio as well. Proposition 3.3. Let µ1, µ2 be two LJSDs such that µ1  µ2 (see Definition 2.1). Then Bµ1  Bµ2 , Eµ1  Eµ2 , and Bµ1/Vµ1  Bµ2/Vµ2 .
We illustrate Proposition 3.3 in Fig. 1 for the 2-scale LJSD of Eq. (11). Following vertical lines upward in panels (a-c) or the x-axis left-to-right in (e,f) yields stronger alignment and a corresponding decrease in the bias and test error. Panel (f) shows the alignment-dependence of the bias, variance, and test error when normalized by the training error. As suggested by Remark 3.1, the normalized test error is independent of the alignment strength, and as intuition would suggest and as predicted by Proposition 3.3, the normalized bias decreases in response to stronger alignment.

heading: 3.4 ANISOTROPY INDUCES STRUCTURED LEARNING CURVES
text: Strong anisotropy has been observed to induce structured learning curves in the context of linear regression (Mel & Ganguli (2021)). It is natural to wonder whether these effects generalize to the case of nonlinear random features. Figure 2 shows that this is indeed the case for sample-wise learning curves, where 1 = m/n0 is varied while = n0/n1 is held fixed. Horizontal slices through Fig. 2 (a) exhibit sample-wise multiple descent. Fig. 2 (b) and (c) show that these peaks are associated with phase transitions at critical values of 1 where the spectrum of the kernel matrix 1n1F
TF acquires a new component. Increased nonlinearity ! attenuates this effect.
In contrast, the following proposition shows that parameter-wise multiple descent does not occur even in the presence of strong anisotropy, as long as µ is aligned. Proposition 3.4. If µ is aligned (see Definition 2.1), then, in the ridgeless limit, the test error has at most two interior critical points as a function of the overparameterization ratio / .
Remark 3.2. Since k-fold descent requires at least k critical points, we conclude that multiple (k > 2) descent does not occur, even in the presence of covariate anisotropy, so long as µ is aligned.
Fig. 3(a,b) shows parameter-wise learning curves for various values of ✓ in the 2- and 3-scale models. As predicted by Proposition 3.4, the curves for aligned LJSDs (✓ 0) have at most two critical points (not shown). In fact, even for anti-aligned LJSDs (✓ < 0), most curves also have at most two critical points; however, for small enough ✓, an additional critical point emerges, and indeed the upper-most traces in Fig. 3(b) showcase a second peak in the learning curves.
Even though parameter-wise multiple descent is not possible for aligned LJSDs, strong anisotropy can nevertheless induce other signatures, including wide plateaus and steep cliffs. To understand the origin of these effects, consider the ridgeless limit, for which the self-consistent equation for x can be written as,
! x + Eµ
+ x =
1
min ⇣ 1, ⌘ . (16)
For small !, the left side becomes relatively insensitive to changes in x whenever ⌧ /x ⌧ + for some wide spectral gap with boundary ( , +). It follows that, in the underparameterized regime ( < ), the derivative @( / )/@x is small, or, equivalently, @x/@( / ) is large. This strong sensitivity of x to changes in the overparameterization ratio / induces similarly strong changes to the error because @Eµ/@x is bounded. As a result, the parameter-wise learning curves exhibit sharp downward cliffs in the underparameterized regime for sufficiently small ! and sufficiently large spectral gaps +/ . We make this argument more precise in Appendix F.
Fig. 3(a) illustrates these cliffs in the context of the d = 2- and 3-scale LJSD. Along horizontal slices, the error turns sharply downward as a function of n1 / / at integer multiples of n0/d. As can be seen in Fig. 3(b), when the alignment is increased and overlaps more with large s, the first error cliff, corresponding to the largest scales in the data, strengthens relative to others (cliff at / = 1/2 in the 2-scale model and / = 1/3 in the 3-scale model). Fig. 3(c) shows that, unlike the peak at the interpolation threshold, cliffs persist under optimal regularization. Finally, Fig. 3(d) illustrates how steep decreases in the error are associated with the appearance of a new component in the spectrum of the kernel matrix at critical values of / .

heading: 4 CONCLUSIONS
text: We presented an asymptotically exact calculation of the test error, bias, and variance for random feature kernel regression in the presence of anisotropic covariates and target function weights. After defining a partial order over weight-data alignments, we proved that stronger alignment decreases the bias and test error. We also proved that the benefits of overparameterization persist in the anisotropic setting, and that weight-data alignment guarantees that parameter-wise multiple descent is limited to double descent. In contrast, we demonstrated that anisotropy can induce sample-wise multiple descent, and parameter-wise cliffs, and that their structure is dictated by the eigenstructure of the data covariance. Future directions include extending our results to the non-asymptotic regime, accommodating feature learning and more general neural network models, and investigating the impact of anisotropy for other loss functions.

Review summaries:
This work studies random features regression on a setting where data $(x_{i}, y_{i})\in\mathbb{R}^{n_{0}+1}$, $i=1, \cdots,n$ is generated from a noisy linear model $y_{i}=\beta^{\top}x_{i}+\epsilon_{i}$ with correlated covariates $x_{i}\sim\mathcal{N}(0, \Sigma)$, planted weights $\beta\sim\mathcal{N}(0,\Sigma_{\beta})$ and independent Gaussian noise $\epsilon_{i}\sim\mathcal{N}(0,\sigma_{\epsilon}^2)$. Estimation is given by $\hat{y}(x) = \hat{\beta}^{\top}\sigma\left(Wx\right)$ with $\sigma$ a real-valued activation function and $W\in\mathbb{R}^{n_{1}\times n_{0}}$ a Gaussian matrix with iid $\mathcal{N}(0, 1/n_{0})$ entries, and $\hat{\beta}\in\mathbb{R}^{n_{1}}$ minimising the ridge regression risk on the features $f = \sigma\left(Wx\right)\in\mathbb{R}^{n_{1}}$.  Its main theoretical contribution is to provide an asymptotic characterisation of the test MSE, the bias and the variance on the proportional regime $m,n_{0}, n_{1}\to\infty$ with fixed ratios $\phi=n_{1}/m, \psi=n_{0}/n_{1}$, as a function of the asymptotic spectral statistics of $(\Sigma, \Sigma_{\beta})$, the ratios $(\phi,\psi)$, the $\ell_2$ penalty strength $\gamma>0$, the noise variance $\sigma_{\epsilon}^2$ and the constants $\rho = \mathbb{E}\left[z\sigma(z)\right]$, $\eta = \rm{Var}\left[\sigma(z)\right]$, $z\sim\mathcal{N}(0,1)$, characterising the activation $\sigma$.   It provides numerical experiments confirming the validity of their formulas for finite but large dimension, and discusses the implications of the anisotropies $(\Sigma, \Sigma_{\beta})$  on the bias, variance and test error in different regimes. In particular, it shows that alignment between $\beta$ and the leading eigenvectors of $\Sigma$ reduces the test MSE, and that anisotropy can induce double-descent like behaviour in the learning curves.   

This paper studies nonlinear random feature regression of a linear response in the *anisotropic* setting where both the covariates and parameter have arbitrary covariance matrix. The training error and test error are computed explicitly as the dimension, sample size, and number of features jointly tend to infinity. The proof method relies on linearization and a technique from free probability involving the construction of linear pencils. The asymptotic formulas are used to theoretically describe (i) the beneficial effect of overparameterization on the bias and variance components of the test error and (ii) the number of critical points of the test error as a function of overparameterization. This paper graphs the asymptotic formulas for some specific models to demonstrate parameter-wise and sample-wise multiple descent phenomena. 

The paper studies random feature regression in high dimensional setting, when the data and model weights are assumed to be normal with anisotropic covariances. By an error decomposition argument, the exact formulas for the bias, variance and total error are provided. The results motivates the discussion on the multiple descent effect of the sample-wise test error curve, and the authors also prove that the error against overparametirazation only exhibits double descent. The phase transition is associated with the spectral structure of the approximated kernel matrix. Also, the error is improved when the data covariance aligns more with the model weight covariance.

This paper evaluates the training and generalization errors of the random feature (ridge) regression model, in the regime where the number of samples $m$, the input feature dimension $n_0$, and the hidden layer size $n_1$ tend to infinity at the same rate. More precisely, the authors consider * Gaussian data with zero mean and general covariance $x_i \sim \mathcal{N}(0, \Sigma)$; * label generated by $y(x_i) = \beta^\top x_i/\sqrt{n_0} + \epsilon_i$, for some additive noise $\epsilon_i \sim \mathcal{N}(0, \sigma_\epsilon^2 ) $ and coefficient vector $\beta \sim \mathcal{N} (0, \Sigma_\beta)$; * (random feature) kernel regression on $K(x_i, x_j) = \frac1{n_1} \sigma(W x_i/\sqrt{n_0})^\top \sigma(W x_j/\sqrt{n_0})$;  and evaluate the training and test performance as a function of the dimensionality, the nonlinear function (via a few scalar parameters), as well as the spectral behavior of $\Sigma$ and $\Sigma_\beta$.  As a consequence of this theoretical evaluation, the authors characterize sample-wise multiple descents, steep cliffs, and how they relate to, e.g., the spectrum of the (random feature) kernel matrices and the structure of coefficient vector $\beta$.