Title: Generalized equivalences between subsampling and ridge regularization
Venue: NeurIPS 2023
Accepted: yes
Keywords: subsampling

ridge regularization

asymptotic equivalences

proportional asymptotics
Abstract: We establish precise structural and risk equivalences between subsampling and ridge regularization for ensemble ridge estimators. Specifically, we prove that linear and quadratic functionals of subsample ridge estimators, when fitted with different ridge regularization levels $\lambda$ and subsample aspect ratios $\psi$, are asymptotically equivalent along specific paths in the $(\lambda,\psi)$-plane (where $\psi$ is the ratio of the feature dimension to the subsample size). Our results only require bounded moment assumptions on feature and response distributions and allow for arbitrary joint distributions. Furthermore, we provide a data-dependent method to determine the equivalent paths of $(\lambda,\psi)$. An indirect implication of our equivalences is that optimally tuned ridge regression exhibits a monotonic prediction risk in the data aspect ratio. This resolves a recent open problem raised by Nakkiran et al. for general data distributions under proportional asymptotics, assuming a mild regularity condition that maintains regression hardness through linearized signal-to-noise ratios.

Generalized equivalences between subsampling and ridge regularization

Abstract

We establish precise structural and risk equivalences between subsampling and ridge regularization for ensemble ridge estimators. Specifically, we prove that linear and quadratic functionals of subsample ridge estimators, when fitted with different ridge regularization levels λ and subsample aspect ratios ψ , are asymptotically equivalent along specific paths in the ( λ, ψ ) -plane (where ψ is the ratio of the feature dimension to the subsample size). Our results only require bounded moment assumptions on feature and response distributions and allow for arbitrary joint distributions. Furthermore, we provide a data-dependent method to determine the equivalent paths of ( λ, ψ ) . An indirect implication of our equivalences is that optimally tuned ridge regression exhibits a monotonic prediction risk in the data aspect ratio. This resolves a recent open problem raised by Nakkiran et al. [1] for general data distributions under proportional asymptotics, assuming a mild regularity condition that maintains regression hardness through linearized signalto-noise ratios.

1 Introduction

Ensemble methods, such as bagging [2, 3], are powerful tools that combine weak predictors to improve predictive stability and accuracy. This paper focuses on sampling-based ensembles, which exhibit an implicit regularization effect [4-6]. Specifically, we investigate subsample ridge ensembles, where the ridge predictors are fitted on independently subsampled datasets [7-9]. Recent work has demonstrated that a full ensemble (that is fitted on all possible subsampled datasets) of ridgeless [10] predictors achieves the same squared prediction risk as a ridge predictor fitted on full data [11-13].

To be precise, let ϕ be the limiting dataset aspect ratio p/n , where p is the feature dimension, and n is the number of observations. For a given ϕ , the limiting prediction risk of subsample ridge ensembles is parameterized by ( λ, ψ ) , where λ ≥ 0 is the ridge regularization parameter and ψ ≥ ϕ is the limiting subsample aspect ratio p/k , with k being the subsample size [11, 12]. Under isotropic features and a well-specified linear model, [11, 12] show that the squared prediction risk at (0 , ψ ∗ ) (the optimal ridgeless ensemble) is the same as the risk at ( λ ∗ , ϕ ) (the optimal ridge), where ψ ∗ and λ ∗ are the optimal subsample aspect ratio and ridge penalty, respectively. Furthermore, this equivalence of prediction risk between subsampling and ridge regularization is extended in [13] to anisotropic linear models. As an application, [13] also demonstrates how generalized cross-validation for ridge regression can be naturally transferred to subsample ridge ensembles. In essence, these results suggest that subsampling a smaller number of observations has the same effect as adding an appropriately larger level of ridge penalty. These findings prompt two important open questions:

(a) The extent of equivalences. The previous works all focus on equivalences of the squared prediction risk when the test distribution matches the train distribution. In real-world scenarios,

however, there are often covariate and label shifts, making it crucial to consider prediction risks under such shifts. In addition, other risk measures, such as training error, estimation risk, coefficient errors, and more, are also of interest in various inferential tasks. A natural question is then whether similar equivalences hold for such 'generalized' risk measures. At a basic level, the question boils down to whether any equivalences exist at the 'estimator level'. Answering this question would establish a connection between the ensemble and the ridge estimators, facilitating the exchange of various downstream inferential statements between the two sets of estimators.

(b) The role of linear model. All previous works assume a well-specified linear model between the responses and the features, which rarely holds in practical applications. A natural question is whether the equivalences hold for arbitrary joint distributions of the response and the features or whether they are merely an artifact of the linear model. Addressing this question broadens the applicability of such equivalences beyond simplistic models to real-world scenarios, where the relationship between the response and the features is typically intricate and unknown.

We provide answers to questions raised in both directions. We demonstrate that the equivalences hold for the generalized squared risks in the full ensemble. Further, these equivalences fundamentally occur at the estimator level for any arbitrary ensemble size. Importantly, these results are not limited to linear models of the data-generating process. Below we provide a summary of our main results.

1.1 Summary of contributions

Risk equivalences. We establish asymptotic equivalences of the full-ensemble ridge estimators at different ridge penalties λ and subsample ratios ψ along specific paths in the ( λ, ψ ) -plane for a variety of generalized risk functionals. This class of functionals includes commonly used risk measures (see Table 2), and our results hold for both independent and dependent coefficient matrices that define these risk functionals (see Theorem 1 and Proposition 2). In addition, we demonstrate that the equivalence path remains unchanged across all the functionals examined.

Structural equivalences. We establish structural equivalences in the form of linear functionals of the ensemble ridge estimators, which hold for arbitrary ensemble sizes (see Theorem 3). Our proofs for both structural and risk equivalences exploit properties of certain fixed equations that arise in our analysis, enabling us to explicitly characterize paths that yield equivalent estimators in the ( λ, ψ ) -plane (see Equation (5)). In addition, we provide an entirely data-driven construction of this path using certain companion Stieltjes transform relations of random matrices (see Proposition 4).

Equivalence implications. As an implication of our equivalences, we show that the prediction risk of an optimally-tuned ridge estimator is monotonically increasing in the data aspect ratio under mild regularity conditions (see Theorem 6). This is an indirect consequence of our general equivalence results that leverages the provable monotonicity of the subsample-optimized estimator. Under proportional asymptotics, our result settles a recent open question raised by Nakkiran et al. [1, Conjecture 1] concerning the monotonicity of optimal ridge regression under anisotropic features and general data models while maintaining a regularity condition that preserves the linearized signal-to-noise ratios across regression problems.

Generality of equivalences. Our main results apply to arbitrary responses with bounded 4 + µ moments for some µ > 0 , as well as features with similar bounded moments and general

covariance structures. We demonstrate the practical implications of our findings on real-world datasets (see Section 6). On the technical side, we extend the tools developed in [14] to handle the dependency between the linear and the non-linear components and obtain model-free equivalences. Furthermore, we extend our analysis to include generalized ridge regression (see Corollary 5). Through experiments, we demonstrate the universality of the equivalences for random and kernel features and conjecture the associated data-driven prediction risk equivalence paths (see Section 6).

1.2 Related work

Ensemble learning. Ensemble methods yield strong predictors by combining weak predictors [15]. The most common strategy for building ensembles is based on subsampling observations, which includes bagging [2, 3], random forests [16], neural network ensembles [17-19], among the others. The effect of sampling-based ensembles has been studied in the statistical physics literature [20-22] as well as statistical learning literature [2, 3]. Recent works have attempted to explain the success of ensemble learning by suggesting that subsampling induces a regularizing effect [4, 6]. Under proportional asymptotics, the effect of ensemble random feature models has been investigated in [23-25]. There has been growing interest in connecting the effect of ensemble learning to explicit regularization: see [26] for related experimental evidence under the umbrella of 'mini-patch' learning. Towards making these connections precise, some work has been done in the context of the ridge and ridgeless ensembles: [11-13] investigate the relationship between subsampling and regularization for ridge ensembles in the overparameterized regime. We will delve into these works in detail next.

Ensembles and ridge equivalences. In the study of ridge ensembles, LeJeune et al. [11] show that the optimal ridge predictor has the same prediction risk as the ridgeless ensemble under the Gaussian isotropic design, which has been extended in [12] for RMT features (see Assumption 2 for definition). Specifically, these works establish the prediction risk equivalences for a specific pair of ( λ ∗ , ϕ ) and (0 , ψ ∗ ) . The results are then extended to the entire range of ( λ, ψ ) using deterministic asymptotic risk formulas [13], assuming a well-specified linear model, RMT features, and the existence of a fixed limiting spectral distribution of the covariance matrix. Our work significantly broadens the scope of results connecting subsampling and ridge regularization. In particular, we allow for arbitrary joint distributions of the data, anisotropic features with only bounded moments, and do not assume the convergence of the spectrum of the population covariance matrix. Furthermore, we expand the applicability of equivalences to encompass both the estimators and various generalized squared risk functionals. See Table 1 for an explicit comparison and Section 3 for additional related comparisons.

Other ridge connections. Beyond the connection connections between implicit regularization induced by subsampling and explicit ridge regularization examined in this paper, ridge regression shares ties with various other forms of regularization. For example, ridge regression has been linked to dropout regularization [27, 28], variable splitting in random forests [2], noisy training [29, 30], random sketched regression [31-33], various forms of data augmentation [34], feature augmentation [9, 35], early stopping in gradient descent and its variants [36-39], among others. These connections highlight the pervasiveness and the significance of understanding various 'facets' of ridge regularization. In that direction, our work contributes to expand the equivalences between subsampling and ridge regularization.

2 Notation and preliminaries

Ensemble estimators. Let D n = { ( x i , y i ) : i ∈ [ n ] } be a dataset containing i.i.d. samples in R p × R . Let X ∈ R n × p and y ∈ R n be the feature matrix and response vector with x ⊤ i and y i in i -th rows. For an index set I ⊆ [ n ] of size k , let D I = { ( x i , y i ) : i ∈ I } be the associated subsampled dataset. Let L I ∈ R n × n be a diagonal matrix such that its i -th diagonal entry is 1 if i ∈ I and 0 otherwise. Note that the feature matrix and response vector associated with D I respectively are L I X and L I y . Given a ridge penalty λ > 0 , the ridge estimator fitted on D I consisting of k samples is given by:

The ridgeless estimator ̂ β 0 k ( D I ) = ( X ⊤ L I X /k ) + X ⊤ L I y /k is obtained by sending λ → 0 + , where M + denotes the Moore-Penrose inverse of matrix M . To define the ensemble estimator, denote

the set of all k distinct elements from [ n ] by I k = {{ i 1 , i 2 , . . . , i k } : 1 ≤ i 1 < i 2 < . . . < i k ≤ n } . For λ ≥ 0 , the M -ensemble and the full-ensemble estimators are respectively defined as follows:

Here { I ℓ } ℓ M =1 and I are simple random samples from I k . In (2), the full-ensemble ridge estimator ̂ β λ k, ∞ ( D n ) is the average of predictors fitted on all possible subsampled datasets. It is shown in Lemma A.1 of [13] that ̂ β λ k, ∞ ( D n ) is also asymptotically equivalent to the limit of ̂ β λ k,M ( D n ; { I ℓ } ℓ M =1 ) as the ensemble size M →∞ (conditioning on the full dataset D n ). This also justifies using ∞ in the subscript to denote the full-ensemble estimator. For brevity, we simply write the estimators as ̂ β λ k,M , ̂ β λ k, ∞ , and drop the dependency on D n , { I ℓ } ℓ M =1 . We will show certain structural equivalences in terms of linear projections in the family of estimators ̂ β λ k,M along certain paths in the ( λ, p/k ) -plane for arbitrary ensemble size M ∈ N ∪ {∞} (in Theorem 3). Apart from connecting the estimators, we will also show equivalences of various notions of risks (in Theorem 1), which we describe next.

Generalized risks. Since the ensemble ridge estimators are linear estimators, we evaluate their performance relative to the oracle parameter: β 0 = E [ xx ⊤ ] -1 E [ x y ] , which is the best (population) linear projection of y onto x and minimizes the linear regression error (see, e.g., [40-42]). Note that we can decompose any response y into: y = f LI ( x ) + f NL ( x ) , where f LI ( x ) = β ⊤ 0 x is the oracle linear predictor, and f NL ( x ) = y -f LI ( x ) is the nonlinear component that is not explained by f LI ( x ) . The best linear projection has the useful property that f LI ( x ) is (linearly) uncorrelated with f NL ( x ) , although they are generally dependent. It is worth mentioning that this does not imply that y and x follow a linear regression model. Indeed, our framework allows any nonlinear dependence structure between them and is model-free for the joint distribution of ( x , y ) . Relative to β 0 , we measure the performance of an estimator ̂ β via the generalized mean squared risk defined compactly as follows:

where L A , b ( β ) = Aβ + b is a linear functional. Note that A and b can potentially depend on the data, and their dimensions can vary depending on the statistical learning problem at hand. This framework includes various important statistical learning problems, as summarized in Table 2. Observe that the framework also includes various notions of prediction risks. One can use the test error formulation in Table 2 to obtain the prediction risk at any test point ( x 0 , y 0 ) , where y 0 = x ⊤ 0 β 0 + ε 0 , and ε 0 may depend on x 0 and have non-zero mean. This also permits the test point to be drawn from a distribution that differs from the training distribution. Specifically, when ε 0 = f NL ( x 0 ) but x 0 has a distribution different from x , we obtain the prediction error under covariate shift . Similarly, when ε 0 = f NL ( x 0 ) but x 0 and x have the same distribution, we get the prediction error under label shift .

̸

Asymptotic equivalence. For our theoretical analysis, we consider the proportional asymptotics regime, where the ratio of the feature size p to the sample size n tends to a fixed limiting data aspect ratio ϕ ∈ (0 , ∞ ) . To concisely present our results, we will use the elegant framework of asymptotic equivalence [12, 43, 44]. Let A p and B p be sequences of (additively) conformable matrices of arbitrary dimensions (including vectors and scalars). We say that A p and B p are asymptotically equivalent , denoted as A p ≃ B p , if lim p →∞ | tr[ C p ( A p -B p )] | = 0 almost surely for any sequence of random matrices C p with bounded trace norm that are (multiplicatively) conformable and independent of A p and B p . Note that for sequences of scalar random variables, the definition simply reduces to the typical almost sure convergence of sequences of random variables involved.

0.1

1

10

0.0

0.1

1

10

Ridge penalty

Estimation risk

0.1

1

10

Subsample aspect ratio

Training error

0.1

1

10

Prediction risk

0.1

1

10

Prediction risk (OOD)

0.0002

0.0003

0.0004

0.0005

2.2

2.4

2.6

2.8

3.0

3.2

3.4

0.10

0.15

0.20

0.25

0.30

0.35

0.075

0.100

0.125

0.150

0.175

0.200

0.225

3 Generalized risk equivalences of ensemble estimators

We begin by examining the risk equivalences among different ensemble ridge estimators for various generalized risks defined as in (3). To prepare for our upcoming results, we impose two structural and moment assumptions on the distributions of the response variable and the feature vector.

Assumption 1 (Response variable distribution) . Each response variable y i for i ∈ [ n ] has mean 0 and satisfies E [ | y i | 4+ µ ] ≤ M µ < ∞ for some µ > 0 and a constant M µ .

Assumption 2 (Feature vector distribution) . Each feature vector x i for i ∈ [ n ] can be decomposed as x i = Σ 1 / 2 z i , where z i ∈ R p contains i.i.d. entries z ij for j ∈ [ p ] with mean 0 , variance 1 , and satisfy E [ | z ij | 4+ ν ] ≤ M ν < ∞ for some ν > 0 and a constant M ν , and Σ ∈ R p × p is a deterministic and symmetric matrix with eigenvalues uniformly bounded between constants r min > 0 and r max < ∞ .

Note that we do not impose any specific model assumptions on the response variable y in relation to the feature vector x . We only assume bounded moments as stated in Assumption 1, making all of our subsequent results model-free. The zero-mean assumption for y is for simplicity since, in practice, centering can always be done by subtracting the sample mean. The bounded moment condition can also be satisfied if one imposes a stronger distributional assumption (e.g., sub-Gaussianity). Assumption 2 on the feature vector is common in the study of random matrix theory [45, 46] and the analysis of ridge and ridgeless regression [10, 47-49], which we refer to as RMT features for brevity.

Given a limiting data aspect ratio ϕ ∈ (0 , ∞ ) and a limiting subsample aspect ratio ψ ∈ [ ϕ, ∞ ] , our statement of equivalences between different ensemble estimators is defined through certain paths characterized by two endpoints (0 , ψ ) and ( λ, ϕ ) . These endpoints correspond to the subsample ridgeless ensemble and the (non-ensemble) ridge predictor, respectively, with λ being the ridge penalty to be defined next. For that, let H p be the empirical spectral distribution of Σ : H p ( r ) = p -1 ∑ p i =1 1 { r i ≤ r } , where r i 's are the eigenvalues of Σ . Consider the following system of equations in λ and v :

Existence and uniqueness of the solution ( λ, v ) ∈ [0 , ∞ ] 2 to the above equations are guaranteed by Corollary E.4. Now, define a path P ( λ ; ϕ, ψ ) that passes through the endpoints (0 , ψ ) and ( λ, ϕ ) :

We are ready to state our results. We consider two cases for the generalized risk (3) depending on the relationships between ( A , b ) and X . In the first case, when both A and b are independent of the data, Theorem 1 shows that the generalized risks are equivalent along the path (5).

Theorem 1 (Risk equivalences when ( A , b ) ⊥ ⊥ ( X , y ) ) . Suppose Assumptions 1-2 hold. Let ( A , b ) be independent of ( X , y ) such that nrow( A ) -1 / 2 ∥ A ∥ op and ∥ b ∥ 2 are almost surely bounded. Let n, p → ∞ such that p/n → ϕ ∈ (0 , ∞ ) . For any ψ ∈ [ ϕ, + ∞ ] , let λ be as defined in (4) . Then, for any pair of ( λ 1 , ψ 1 ) and ( λ 2 , ψ 2 ) on the path P ( λ ; ϕ, ψ ) as defined in (5) , the generalized risk functionals (3) of the full-ensemble estimator are asymptotically equivalent:

In other words, Theorem 1 establishes the equivalences of subsampling and ridge regularization in terms of generalized risk for many statistical learning problems, encompassing coefficient estimation,

coefficient confidence interval, and test error estimation. All of these problems fall in the category when ( A , b ) are independent of X . However, there are other statistical learning problems not covered in Theorem 1, such as the in-sample prediction risk and the training error, which correspond to the case when A = X . Fortunately, the equivalences also apply to them, as summarized in Proposition 2.

Proposition 2 (Risk equivalences when A = X ) . Under Assumptions 1-2, when A = X , the conclusion in Theorem 1 continue to hold for the cases of in-sample prediction risk ( b = 0 ) and training error ( b = -f NL ).

Both Theorem 1 and Proposition 2 provide specific second-order equivalences for the full-ensemble ridge estimators, in the sense that the quadratic functionals of the estimators associated with ( A , b ) are asymptotically the same. See Figure 1 for visual illustrations of both equivalences. These statements are presented separately because their proofs differ, with the proof for the dependent case being more intricate. We note that by combining the risk functionals in Table 2, it is possible to further extend the equivalences to other types of functionals not directly covered by statements above, using composition and continuous mapping. For example, it is not difficult to show that similar equivalences hold for the generalized cross-validation in the full ensembles [13]. This follows by combining the result for training error from Proposition 2 and for denominator concentration proved in Lemma 3.4 of [13].

Theorem 1 generalizes the existing equivalence results for the prediction risk [11-13] to include general risk functionals under milder assumptions. As summarized in Table 1, we allow for a general response model and feature covariance without assuming convergence of the spectral distributions H p to a fixed distribution H . This flexibility is achieved by showing equivalences of the underlying resolvents in the estimators rather than deriving the limiting functionals as done in previous works. To extend results allowing for a general response, we generalize certain concentration results by leveraging tools from [14] to handle the dependency between the linear and non-linear components.

As alluded to earlier, it is worth noting that the generalized risk equivalences only hold in the full ensemble. However, by using the risk decomposition obtained from Lemma S.1.1 of [12] for finite ensembles, one can obtain the following relationship along the path in Theorem 3: R ( ̂ β λ 1 ⌊ p/ψ 1 ⌋ ,M ; A , b , β 0 ) -R ( ̂ β λ 2 ⌊ p/ψ 2 ⌋ ,M ; A , b , β 0 ) ≃ ∆ /M, for some ∆ (independent of M ) that is eventually almost surely bounded. In other words, the difference between any two estimators on the same path scales as 1 /M when n is sufficiently large, as numerically verified in Figure F5.

4 Structural equivalences of ensemble estimators

The equivalences established in the previous section only hold for the full-ensemble estimators in a second-order sense. We can go a step further and ask if there exist any equivalences for the finite ensemble and if there are any equivalences at the estimator coordinate level between the estimators. Specifically, we aim to inspect whether each coordinate of the p -dimensional estimated coefficients asymptotically equals. This section establishes structural relationships between the estimators in a first-order sense that any bounded linear functionals of them are asymptotically the same.

Theorem 3 (Structural equivalences) . Suppose Assumptions 1-2 hold. Let n, p →∞ with p/n → ϕ ∈ (0 , ∞ ) . For any ψ ∈ [ ϕ, + ∞ ] , let λ be as in (4) . Then, for any M ∈ N ∪ {∞} and any pair of ( λ 1 , ψ 1 ) and ( λ 2 , ψ 2 ) on the path (5) , the M -ensemble estimators are asymptotically equivalent:

Put another way, Theorem 3 implies that for any fixed ensemble size M , the M -ensemble ridgeless estimator at the limiting aspect ratio ψ is equivalent to ridge regression at the regularization level λ . This equivalence is visually illustrated in Figure 2, where the data is simulated from an anisotropic and nonlinear model (see Appendix F.2 for more details). Furthermore, the contour path P ( λ ; ϕ, ψ ) connecting the endpoints (0 , ψ ) and ( λ, ϕ ) is a straight line, although it may have varying slopes. Along any path, all M -ensemble estimators at the limiting aspect ratio ψ and ridge penalty λ are equivalent for all ( λ, ψ ) ∈ P ( λ ; ϕ, ψ ) . The equivalences here are for any arbitrary linear combinations of the estimators. In particular, this implies that the predicted values (or even any continuous function applied to them due to the continuous mapping theorem) of any test point will eventually be the same, almost surely, with respect to the training data. Finally, we remark that in the statement of Theorem 3, the equivalence is defined on extended reals in order to incorporate the ridgeless case when ψ = 1 .

0.0

0.1

1

10

Ridge penalty

(a)

Uniform,

M

=1

Uniform,

M

=5

Uniform,

M

=50

0.1

1

10

0.0

0.1

1

10

Ridge penalty

(b)

Uniform,

M

=100

0.1

1

10

Subsample aspect ratio

Gaussian,

M

=100

0.1

1

10

Student's t,

M

=100

0.0015

0.0020

0.0025

0.0030

0.0035

0.0040

0.0045

0.0015

0.0020

0.0025

0.0030

0.0035

0.0040

0.0045

0.0015

0.0020

0.0025

0.0030

0.0035

0.0040

0.0045

0.0015

0.0020

0.0025

0.0030

0.0035

0.0040

0.0045

0.030

0.025

0.020

0.015

0.010

0.000

0.002

0.004

0.006

0.008

0.010

The paths (5) in Theorem 3 are defined via the spectral distribution H p , which requires knowledge of Σ . This is often difficult to obtain in practice from the observed data. Fortunately, we can provide an alternative characterization for the path (5) solely through the data, as summarized in Proposition 4.

Proposition 4 (Data-dependent equivalence path characterization) . Suppose Assumptions 1-2 hold. Define ϕ n = p/n . Let k ≤ n be the subsample size and denote by ψ n = p/k . For any M ∈ N ∪{∞} , let λ n be the value that satisfies the following equation in ensemble ridgeless and ridge gram matrices:

Define the data-dependent path P n = P ( λ n ; ϕ n , ψ n ) . Then, the conclusion in Theorem 3 continues to hold if we replace the (population) path P by the (data-dependent) path P n .

The term 'data-dependent path' signifies that we can estimate the level of implicit regularization induced by subsampling solely based on the observed data by solving (8). We remark that (8) always has at least one solution for a given triplet of ( n, k, p ) . This is because the right-hand side of (8) is monotonically decreasing in λ , and the left-hand side always lies within the range of the right-hand side. The powerful implication of the characterization in Proposition 4 is that it enables practical computation of the path using real-world data. In Section 6, we will demonstrate this approach on various real-world datasets, allowing us to predict an equivalent amount of explicit regularization matching the implicit regularization due to subsampling.

One natural extension of the results is to consider the generalized ridge as a base estimator (1):

where G ∈ R p × p is a positive definite matrix. The equivalences still hold, albeit on different paths:

Corollary 5 (Equivalences for generalized ridge regression) . Suppose Assumptions 1-2 hold. Let G ∈ R p × p be a deterministic and symmetric matrix with eigenvalues uniformly bounded between constants g min > 0 and g max < ∞ . Let n, p → ∞ such that p/n → ϕ ∈ (0 , ∞ ) . For any ψ ∈ [ ϕ, + ∞ ] , let λ be as defined in (4) with H p replaced ˜ H p , the empirical spectral distribution of G -1 / 2 Σ G -1 / 2 . Then, the conclusions in Theorems 1 and 3 continue to hold for the generalized ridge ensemble predictors.

Another extension of our results is to incorporate subquadratic risk functionals in Theorem 1 to derive equivalences of the cumulative distribution functions of the predictive error distributions associated

with the estimators along the equivalence paths. One can use such equivalences for downstream inference questions. Finally, while our statements are asymptotic in nature, we expect a similar analysis as done in [50] can yield finite-sample statements. We leave these extensions for the future.

5 Implications of equivalences

The generalized risk equivalences establish a connection between the risks of different ridge ensembles, including the ridge predictors on full data and the full-ensemble ridgeless predictors. These connections enable us to transfer properties from one estimator to the other. For example, by examining the impact of subsampling on the estimator's performance, we can better understand how to optimize the ridge penalty. We will next provide an example of this. Many common methods, such as ridgeless or lassoless predictors, have been recently shown to exhibit non-monotonic behavior in the sample size or the limiting aspect ratio. An open problem raised by Nakkiran et al. [1, Conjecture 1] asks whether the prediction risk of the ridge regression with optimal ridge penalty λ ∗ is monotonically increasing in the data aspect ratio ϕ = p/n . The non-monotonicity risk behavior implies that more data can hurt performance, and it's important to investigate whether optimal ridge regression also suffers from this issue. Our equivalence results provide a surprising (even to us) indirect way to answer this question affirmatively under proportional asymptotics.

To analyze the monotonicity of the risk, we will need to impose two additional regularity assumptions to guarantee the convergence of the prediction risk. Let Σ = ∑ p j =1 r j w j w ⊤ j denote the eigenvalue decomposition, where ( r j , w j ) 's are pairs of associated eigenvalue and normalized eigenvector. The following assumptions ensure that the hardness of the underlying regression problems across different limiting data aspect ratios is comparable.

Assumption 3 (Spectral convergence) . We assume there exists a deterministic distribution H such that the empirical spectral distribution of Σ , H p ( r ) = p -1 ∑ p i =1 1 { r i ≤ r } , weakly converges to H , almost surely (with respect to x ).

Assumption 4 (Limiting signal and noise energies) . We assume there exists a deterministic distribution G such that the empirical distribution of β 0 's (squared) projection onto Σ 's eigenspace, G p ( r ) = ∥ β 0 ∥ -2 2 ∑ p i =1 ( β ⊤ 0 w i ) 2 1 { r i ≤ r } , weakly converges to G . As n, p →∞ and p/n → ϕ ∈ (0 , ∞ ) , the limiting linearized energy ρ 2 = lim ∥ β 0 ∥ 2 2 and the limiting nonlinearized energy σ 2 = lim ∥ f NL ∥ 2 L 2 are finite.

Assumption 3 is commonly used in random matrix theory and overparameterized learning [10, 12, 13] and ensures that the limiting spectral distribution of the sample covariance matrix converges to a fixed distribution. Under these assumptions, we show in Appendix C that there exists a deterministic function R ( λ ; ϕ, ψ ) , such that R ( ̂ β λ k, ∞ ; A , b , β 0 ) ≃ R ( λ ; ϕ, ψ ) . Notably, the deterministic profile on the right-hand side is a monotonically increasing and a continuous function in the first parameter ϕ when limiting values of ∥ β 0 ∥ 2 2 and ∥ f NL ∥ 2 L 2 are fixed (i.e., when Assumption 4 holds). Moreover, the deterministic risk equivalent of the optimal ridge predictor matches that of the optimal fullensemble ridgeless predictor; in other words, min λ ≥ 0 R ( λ ; ϕ, ϕ ) = min ψ ≥ ϕ R (0; ϕ, ψ ) . The same monotonicity property of the two optimized functions leads to the following result.

Theorem 6 (Monotonicity of prediction risk with optimal ridge regularization) . Suppose the conditions of Theorem 1 and Assumptions 3-4 hold. Let k, n, p → ∞ such that p/n → ϕ ∈ (0 , ∞ ) and p/k → ψ ∈ [ ϕ, ∞ ] . Then, for A = Σ 1 / 2 and b = 0 , the optimal risk of the ridgeless ensemble, min ψ ≥ ϕ R (0; ϕ, ψ ) , is monotonically increasing in ϕ . Consequently, the optimal risk of the ridge predictor, min λ ≥ 0 R ( λ ; ϕ, ϕ ) , is also monotonically increasing in ϕ .

Under Assumptions 3 and 4, the linearized signal-to-noise ratio (SNR) is maintained at the same level across varying distributions as ϕ changes. The key message of Theorem 6 is then that, for a sequence of problems with the same SNR (indicating the same level of regression hardness), the asymptotic prediction risk of optimized ridge risk gets monotonically worse as the aspect ratio of the problem increases. This is intuitive because a smaller ϕ corresponds to more samples than features. In this sense, Theorem 6 certifies that optimal ridge uses the available data effectively, avoiding sample-wise non-monotonicity. Moreover, as the null risk is finite, this also shows that the optimal ridge estimator mitigates the 'double or multiple descents' behaviors in the generalization error [1, 5, 51].

Attempting to prove Theorem 6 directly is challenging due to the lack of a closed-form expression for the optimal risk of ridge regression in general. Moreover, the risk profile of ridge regression,

Uniform

Student's

t

Linear functionals

0.25

0.00

0.25

0.50

0.75

1.00

Relative value

Estimation risk

Prediction risk

Prediction risk

(OOD)

Genearlized risk functionals

Paths through

(

,

)

for varying

:

=0.01

=0.05

=0.1

=1

R ( λ ; ϕ, ϕ ) , does not exhibit any particular structure as a function of λ . On the other hand, the risk profile of the full-ensemble ridgeless, R (0; ϕ, ψ ) , has a very nice structure in terms of ψ . This, coupled with our equivalence result, allows one to prove quite non-trivial behavior of optimal ridge. See Figure F6 for illustrations.

6 Discussion

Motivated by the recent subsampling and ridge equivalences for the prediction risk [11-13], this paper establishes generalized risk equivalences (Section 3) and structural equivalences (Section 4) within the family of ensemble ridge estimators. Our results precisely link the implicit regularization of subsampling to explicit ridge penalization via a path P defined in (5), which connects two endpoints (0 , ψ ) and ( λ, ϕ ) regardless of the risk functionals. Furthermore, we provide a data-dependent method (Proposition 4) to estimate this path. Our results do not assume any specific relationship between the response variable y and the feature vector x . We next explore some extensions of these equivalences.

Real data. While our theoretical results assume RMT features (Assumption 2), we anticipate that the equivalences will very likely hold under more general features [50]. To verify this, we examine the equivalences in Proposition 4 with M = 100 on real-world datasets. Figure 3 depicts both the linear functionals in Theorem 3 and the generalized quadratic functionals in Theorem 1 computed along four paths, shown in different colors. We observe that the variation within each path is small, while different paths have different functional values. This suggests that the theoretical finding in the previous sections also hold quite well on real-world datasets. We investigate this further by varying ψ on the three image datasets, CIFAR-10, MNIST, and USPS. Figure F7 shows similar values and trends at the two points. These experiments demonstrate that the amount of explicit regularization induced by subsampling can indeed be accurately predicted based on the observed data using Proposition 4.

Random features. Closely related to two-layer neural networks [52], we can consider the random feature model, f ( x ; β , F ) = β ⊤ φ ( Fx ) , where F ∈ R d × p is some randomly initialized weight matrix, and φ : R → R is a nonlinear activation function applied element-wise to Fx . As from 'universality/invariance', certain random feature models are asymptotically equivalent to a surrogate linear Gaussian model with a matching covariance matrix [53], we expect the theoretical results in Theorems 3-1 to likely hold, although the relationship (4) will now depend on the non-linear activation functions. Empirically, we indeed observe similar equivalence phenomena of the prediction risks with random features ridge regression using sigmoid, ReLU, and tanh activation functions, as shown in Figure 4. Analogous to Proposition 4, we conjecture a similar data-driven relationship between ( λ n , ϕ n ) and (0 , ψ n ) for random features ˜ X = φ ( XF ⊤ ) :

Conjecture 7 (Data-dependent equivalences for random features (informal)) . Suppose Assumptions 12 hold. Define ϕ n = p/n . Let k ≤ n be the subsample size and denote by ψ n = p/k . Suppose φ satisfies certain regularity conditions. For any M ∈ N ∪ {∞} , let λ n be the value that satisfies

0.1

1

10

0.0

0.1

1

10

Ridge penalty

Sigmoid

0.1

1

10

Subsample aspect ratio

RuLU

0.1

1

10

1.04

1.06

1.08

1.10

1.12

1.14

1.16

1.09

1.10

1.11

1.12

1.13

1.14

1.15

1.16

1.06

1.08

1.10

1.12

1.14

Define the data-dependent path P n = P ( λ n ; ϕ n , ψ n ) . Then, the conclusions in Theorem 1, Proposition 2, and Theorem 3 continue to hold on ( φ ( XF ⊤ ) , y ) with P replaced by P n .

Kernel features. Wecan also consider kernel ridge regression. For a given feature map Φ : R p → R d , the kernel ridge estimator (in the primal form) is defined as:

Leveraging the kernel trick, the preceding optimization problem translates to solving the following problem (in the dual domain): ̂ α λ k ( D I ) = argmin α ∈ R k α ⊤ ( K I + kλ I k ) α + α ⊤ y I , where K I = Φ I Φ ⊤ I ∈ R k × k is the kernel matrix and Φ I = (Φ( x i )) i ∈ I ∈ R n × d is the feature matrix. The correspondence between the dual and primal solutions is simply given by: ̂ β λ k ( D I ) = Φ ⊤ I ̂ α λ k ( D I ) .

Figure F8 illustrate results for kernel ridge regression using the same data-generating process as in the previous subsection. The figure shows the prediction risk of kernel ridge ensembles for polynomial, Gaussian, and Laplace kernels exhibit similar equivalence patterns, leading us to formulate an analogous conjecture for kernel ridge regression (which when Φ( x ) = x gives back Proposition 4 with appropriate rescaling of features):

Conjecture 8 (Data-dependent equivalences for kernel features (informal)) . Suppose Assumptions 12 hold. Define ϕ n = p/n . Suppose the kernel K satisfies certain regularity conditions. Let k ≤ n be the subsample size and denote by ψ n = p/k . For any M ∈ N ∪ {∞} , let λ n be a solution to

Define the data-dependent path P n = P ( λ n ; ϕ n , ψ n ) . Then, the conclusions in Theorem 1, Proposition 2, and Theorem 3 continue to hold for kernel ridge ensembles with P replaced by P n .

The empirical evidence here strongly suggests that the relationship between subsampling and ridge regression, which we have proved for ridge regression, holds true at least when the gram matrix 'linearizes' in the sense of asymptotic equivalence [14, 54, 55]. This intriguing observation opens up a whole host of avenues towards fully understanding the universality of this relationship and establishing precise connections for a broader range of models [17, 18, 20, 21]. We hope to share more on these compelling directions in the near future.

Tying other implicit regularizations. Finally, as noted in related work, the equivalences between ridge regularization and subsampling established in this paper naturally offer opportunities to interpret and understand the other implicit regularization effects such as dropout regularization [27, 28], early stopping in gradient descent variants [36, 38, 39]. Furthermore, these equivalences provide avenues to understand the combined effects of these forms of implicit regularization, such as the effects of both subsample aggregating and gradient descent in mini-batch gradient descent, and contrast them to explicit regularization. Whether the explicit regularization can always be cleanly expressed as ridge-like regularization or takes a more generic form is an intriguing question. Exploring this is exciting, and we hope our readers also share this excitement!

Acknowledgments and Disclosure of Funding

We are indebted to Ryan Tibshirani, Alessandro Rinaldo, Arun Kuchibhotla, Yuting Wei, Matey Neykov, Edgar Dobriban, Daniel LeJeune, Shamindra Shrotriya, Yuchen Wu for many insightful conversations. We are also grateful to anonymous reviewers of the precursor [13] for encouraging us to work on several follow-up directions that make up parts of this successor. On a more personal note: The lack of proof of risk monotonicity of optimally-tuned ridge in general had been 'monotone bugging' (in a good way) the first author, ever since plotting Figure 8 (and other similar plots) in [51]. The resolution in Theorem 6 is of great personal significance to him and he is heartily appreciative of all the collaborators on related threads, especially the second author, for being a dedicated climbing partner on this technical 'ascent'. At the risk of over analogizing, the metaphor of letting a nut soak in water instead of forcibly cracking it open with a hammer, from Grothendieck 2 , may perhaps describe the proof of Theorem 6. Establishing subsampling and ridge equivalences first and using it to demonstrate ridge monotonicity, in retrospect, seems a much less arduous route than a direct brute-force attempt (with calculus, say). But we will let the readers decide!

This work used the Bridges2 system at the Pittsburgh Supercomputing Center (PSC) through allocations MTH230020 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program. The code for reproducing the results of this paper can be found at https://jaydu1.github.io/overparameterized-ensembling/equiv .

References

Preetum Nakkiran, Prayaag Venkat, Sham M. Kakade, and Tengyu Ma. Optimal regularization can mitigate double descent. In International Conference on Learning Representations , 2021.

Leo Breiman. Bagging predictors. Machine Learning , 24(2):123-140, 1996.

Peter Bühlmann and Bin Yu. Analyzing bagging. The Annals of Statistics , 30(4):927-961, 2002.

Abraham J. Wyner, Matthew Olson, Justin Bleich, and David Mease. Explaining the success of adaboost and random forests as interpolating classifiers. The Journal of Machine Learning Research , 18(1):1558-1590, 2017.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machinelearning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Lucas Mentch and Siyu Zhou. Randomization as regularization: A degrees of freedom explanation for random forest success. The Journal of Machine Learning Research , 21(1):6918-6953, 2020.

Arthur E. Hoerl and Robert W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics , 12(1):55-67, 1970.

Arthur E. Hoerl and Robert W. Kennard. Ridge regression: Applications to nonorthogonal problems. Technometrics , 12(1):69-82, 1970.

Trevor Hastie. Ridge regularization: An essential concept in data science. Technometrics , 62 (4):426-433, 2020.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949-986, 2022.

Daniel LeJeune, Hamid Javadi, and Richard Baraniuk. The implicit regularization of ordinary least squares ensembles. In International Conference on Artificial Intelligence and Statistics , 2020.

Pratik Patil, Jin-Hong Du, and Arun Kumar Kuchibhotla. Bagging in overparameterized learning: Risk characterization and risk monotonization. arXiv preprint arXiv:2210.11445 , 2022.

Jin-Hong Du, Pratik Patil, and Arun Kumar Kuchibhotla. Subsample ridge ensembles: Equivalences and generalized cross-validation. In International Conference on Machine Learning , 2023.

Peter L. Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: A statistical viewpoint. Acta numerica , 30:87-201, 2021.

Trevor Hastie, Robert Tibshirani, and Jerome H. Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction . Springer, 2009. Second edition.

Leo Breiman. Random forests. Machine learning , 45(1):5-32, 2001.

L.K. Hansen and P. Salamon. Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence , 12(10):993-1001, 1990.

Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. Advances in neural information processing systems , 1994.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems , 2017.

Michael Perrone. Putting it all together: Methods for combining neural networks. In Advances in Neural Information Processing Systems , 1993.

Anders Krogh and Peter Sollich. Learning with ensembles: How overfitting can be useful. In Advances in Neural Information Processing Systems , 1995.

Anders Krogh and Peter Sollich. Statistical mechanics of ensemble learning. Physical Review E , 55(6):811, 1997.

Stéphane d'Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in double descent: Bias and variance (s) in the lazy regime. In International Conference on Machine Learning , pages 2280-2290. PMLR, 2020.

Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine-grained bias-variance decomposition. In Advances in Neural Information Processing Systems , 2020.

Bruno Loureiro, Cedric Gerbelot, Maria Refinetti, Gabriele Sicuro, and Florent Krzakala. Fluctuations, bias, variance & ensemble of learners: Exact asymptotics for convex losses in high-dimension. In International Conference on Machine Learning , 2022.

Tianyi Yao, Daniel LeJeune, Hamid Javadi, Richard G. Baraniuk, and Genevera I. Allen. Minipatch learning as implicit ridge-like regularization. In International Conference on Big Data and Smart Computing , 2021.

Stefan Wager, Sida Wang, and Percy S. Liang. Dropout training as adaptive regularization. In Advances in Neural Information Processing Systems , 2013.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research , 15(1):1929-1958, 2014.

Andrew R. Webb. Functional approximation by feed-forward networks: A least-squares approach to generalization. IEEE Transactions on Neural Networks , 5(3):363-371, 1994.

Chris M. Bishop. Training with noise is equivalent to Tikhonov regularization. Neural computation , 7(1):108-116, 1995.

Gian-Andrea Thanei, Christina Heinze, and Nicolai Meinshausen. Random projections for large-scale regression. Big and Complex Data Analysis: Methodologies and Applications , pages 51-68, 2017.

Alessandro Rudi, Raffaello Camoriano, and Lorenzo Rosasco. Less is more: Nyström computational regularization. In Advances in Neural Information Processing Systems , volume 28, 2015.

Daniel LeJeune, Pratik Patil, Hamid Javadi, Richard G Baraniuk, and Ryan J. Tibshirani. Asymptotics of the sketched pseudoinverse. arXiv preprint arXiv:2211.03751 , 2022.

Chi-Heng Lin, Chiraag Kaushik, Eva L. Dyer, and Vidya Muthukumar. The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective. arXiv preprint arXiv:2210.05021 , 2022.

Marina Skurichina and Robert P. W. Duin. Regularization by adding redundant features. In Advances in Pattern Recognition: Joint IAPR International Workshops SSPR'98 and SPR'98 Sydney, Australia, August 11-13, 1998 Proceedings , pages 564-572. Springer, 1998.

Gergely Neu and Lorenzo Rosasco. Iterate averaging as regularization for stochastic gradient descent. In Conference On Learning Theory , pages 3222-3242. PMLR, 2018.

Arun Suggala, Adarsh Prasad, and Pradeep K. Ravikumar. Connecting optimization and regularization paths. In Advances in Neural Information Processing Systems , 2018.

Alnur Ali, Edgar Dobriban, and Ryan J. Tibshirani. The implicit regularization of stochastic gradient flow for least squares. In International conference on machine learning , 2020.

Alnur Ali, J. Zico Kolter, and Ryan J. Tibshirani. A continuous-time view of early stopping for least squares regression. In International Conference on Artificial Intelligence and Statistics , 2019.

A. Buja, R. Berk, L. Brown, E. George, E. Pitkin, M. Traskin, K. Zhang, and L. Zhao. A conspiracy of random predictors and model violations against classical inference in regression. arXiv preprint arXiv:1404.1578 , 2014.

Andreas Buja, Lawrence Brown, Richard Berk, Edward George, Emil Pitkin, Mikhail Traskin, Kai Zhang, and Linda Zhao. Models as approximations I. Statistical Science , 34(4):523-544, 2019.

Andreas Buja, Lawrence Brown, Arun Kumar Kuchibhotla, Richard Berk, Edward George, and Linda Zhao. Models as approximations II. Statistical Science , 34(4):545-565, 2019.

Edgar Dobriban and Yue Sheng. Wonder: Weighted one-shot distributed ridge regression in high dimensions. Journal of Machine Learning Research , 21(66):1-52, 2020.

Edgar Dobriban and Yue Sheng. Distributed linear regression by averaging. The Annals of Statistics , 49(2):918-943, 2021.

Zhidong Bai and Jack W. Silverstein. Spectral Analysis of Large Dimensional Random Matrices . Springer, 2010. Second edition.

Noureddine El Karoui. The spectrum of kernel random matrices. The Annals of Statistics , 38 (1):1-50, 2010.

Noureddine El Karoui and Holger Kösters. Geometric sensitivity of random matrix results: Consequences for shrinkage estimators of covariance and related statistical methods. arXiv preprint arXiv:1105.1404 , 2011.

Noureddine El Karoui. Asymptotic behavior of unregularized and ridge-regularized highdimensional robust regression estimators: rigorous results. arXiv preprint arXiv:1311.2445 , 2013.

Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics , 46(1):247-279, 2018.

Chen Cheng and Andrea Montanari. Dimension free ridge regression. arXiv preprint arXiv:2210.08571 , 2022.

Pratik Patil, Arun Kumar Kuchibhotla, Yuting Wei, and Alessandro Rinaldo. Mitigating multiple descents: A model-agnostic framework for risk monotonization. arXiv preprint arXiv:2205.12937 , 2022.

Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 75(4):667-766, 2022.

Hong Hu and Yue M. Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory , 69(3):1932-1964, 2023.

Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel 'ridgeless' regression can generalize. The Annals of Statistics , 2020.

Mojtaba Sahraee-Ardakan, Melikasadat Emami, Parthe Pandit, Sundeep Rangan, and Alyson K Fletcher. Kernel methods and multi-layer perceptrons learn linear models in high dimensions. arXiv preprint arXiv:2201.08082 , 2022.

Antti Knowles and Jun Yin. Anisotropic local laws for random matrices. Probability Theory and Related Fields , 169:257-352, 2017.

László Erd˝ os and Horng-Tzer Yau. A Dynamical Approach to Random Matrix Theory . American Mathematical Society, 2017.

Francisco Rubio and Xavier Mestre. Spectral convergence for a general class of random matrices. Statistics & probability letters , 81(5):592-602, 2011.

TorchVision. Torchvision: PyTorch's computer vision library. https://github.com/ pytorch/vision , 2016.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011.

Supplementary material for

'Generalized equivalences between subsampling and ridge regularization'

This document serves as a supplement to the paper 'Generalized equivalences betweensubsampling and ridge regularization.' The initial (unnumbered) section of the supplement provides a summary of the notation used in both the paper and supplement, followed by an organization for the supplement.

Notation. Below we provide an overview of the notation used throughout.

General notation:

Specific notation:

A note on indexing of sequences:

In the subsequent sections, we will prove the results for n, k, p being a sequence of integers { n m } ∞ m =1 , { k m } ∞ m =1 , { p m } ∞ m =1 . Alternatively, one can also view k and p as sequences k n and p n that are indexed by n . For notational brevity, we drop the subscript when it is clear from the context.

Organization. Below we outline the structure of the rest of the supplement. The technical lemmas refer to the main ingredients that we use to prove results in the corresponding sections.

A Proofs of results in Section 3

A.1 Proof of Theorem 1

Given an observation ( x , y ) , recall the decomposition y = f LI ( x )+ f NL ( x ) explained in Section 2. For n i.i.d. samples from the same distribution as ( x , y ) , we define analogously the vector decomposition:

where f LI = Xβ 0 and f NL = [ f NL ( x i )] i ∈ [ n ] . Let n A = nrow( A ) . Note that

By Theorem 3, the cross term vanishes, i.e., n -1 A b ⊤ A ( ̂ β λ k, ∞ -β 0 ) a.s. - - → 0 . We then have

It suffices to analyze R ( ̂ β λ k, ∞ ; A , 0 , β 0 ) .

For simplicity, we treat A as the normalized matrix n -1 / 2 A A to avoid the notation of nrow( A ) . Note that y = Xβ 0 + f NL + ε , where β 0 is the best linear projection of y on X , f NL is the nonlinear residual and ε is the independent noise. Let A λ k = E I ∼I k [( X ⊤ L I X /k + λ I p ) -1 X ⊤ L I /k ] and B λ k = I p -A λ k X = E I ∼I k [ λ ( X ⊤ L I X /k + λ I p ) -1 ] . We begin by decomposing the generalized risk for arbitrary ( λ, k ) into different terms:

When λ > 0 , from Lemma A.3, we know that the cross terms 2 f NL ⊤ ( A λ k ) ⊤ A ⊤ AB λ k β 0 vanishes as p tends to infinity. Further, by Lemma A.1 and Lemma A.2, it follows that when A ⊥ ⊥ ( X , y ) , as p/n → ϕ and p/k → ψ ,

where the function R p is defined as

and the nonnegative constants ˜ c p ( -λ ; ϕ, ψ, A ⊤ A ) and ˜ v p ( -λ ; ϕ, ψ, A ⊤ A ) are as defined in Lemma A.1.

When λ = 0 , as in the proof of Theorem 3, we again show that P n,λ -Q n,λ is equivcontinuous over Λ = [0 , λ max ] for any λ max ∈ (0 , ∞ ) fixed, where we define

̸

When ψ = 1 , it can be verified that | P n,λ | , | ∂P n,λ /∂λ | , | Q n,λ | , | ∂Q n,λ /∂λ | are bounded almost surely. Thus, by the Moore-Osgood theorem, it follows that, with probability one,

Note that ˜ c p ( -λ ; ϕ, ψ, A ⊤ A ) and ˜ v p ( -λ ; ϕ, ψ, A ⊤ A ) are functions of the fixed-point solution v p ( -λ ; ψ ) . By Lemma C.1 we have that for ψ ∈ [ ϕ, ∞ ] there exists a segment P such that for all ( λ 1 , ψ 1 ) , ( λ 2 , ψ 2 ) ∈ P , it holds that v ( -λ 1 ; ψ 1 ) = v ( -λ 2 ; ψ 2 ) as p/k 1 → ψ 1 and p/k 2 → ψ 2 . This implies that

Thus, by triangle inequality, we have

which completes the proof.

A.2 Proof of Proposition 2

When A = X and b = f NL , the generalized risk reduces to

The proof then follows analogously as in Theorem 1 by involving Lemma A.1 and Lemma A.2 with A = n -1 / 2 X .

When A = X and b = 0 , we have

Invoking Lemma A.2 for f NL ⊤ ( A λ k ) ⊤ ̂ Σ A λ k f NL instead of f NL ⊤ ( I p -XA λ k ) ⊤ ( I p -XA λ k ) f NL , the proof follows analogously.

A.3 Technical lemmas

Lemma A.1 (Bias of generalized risk) . Suppose the same assumptions in Theorem 1 hold with λ > 0 . Let B λ k = I p -A λ k X = E I ∼I k [ λ ( X ⊤ L I X /k + λ I p ) -1 ] . For any A with lim sup ∥ A ∥ op bounded almost surely, the following statements hold:

If A = n -1 / 2 X

Here the nonnegative constants v p ( -λ ; ψ ) , ˜ v p ( -λ ; ϕ, ψ, A ⊤ A ) and ˜ c p ( -λ ; ϕ, ψ, A ⊤ A ) are defined through the following equations:

Proof. Note that β 0 is independent of

where ̂ Σ 1 ∩ 2 = X ⊤ L I 1 ∩ I 2 X / | I 1 ∩ I 2 | and M j = ( X ⊤ L I 1 X /k + λ I p ) -1 for j = 1 , 2 . We analyze the deterministic equivalents of the latter for the two cases.

(1) A ⊥ ⊥ ( X , y ) .

From Lemma D.1 (2), we know that when A is independent to ( X , y ) , it follows that

Then, by the trace property of deterministic equivalents in Lemma E.3 (4), we have

(2) A = n -1 / 2 X .

From Lemma D.1 (4), it follows that

Then, by the trace property of deterministic equivalents in Lemma E.3 (4), we have

Lemma A.2 (Variance term of generalized risk) . Suppose the same assumptions in Theorem 1 hold with λ > 0 . Let A λ k = E I ∼I k [( X ⊤ L I X /k + λ I p ) -1 X ⊤ L I /k ] and B λ k = I p -A λ k X = E I ∼I k [ λ ( X ⊤ L I X /k + λ I p ) -1 ] . For any A with lim sup ∥ A ∥ op bounded almost surely, the following statements hold:

If A ⊥ ⊥ ( X , y )

Here the nonnegative constants v p ( -λ ; ψ ) and ˜ v p ( -λ ; ϕ, ψ, A ⊤ A ) are defined through the following equations:

Proof. The two cases are treated separately below.

A ⊥ ⊥ ( X , y ) .

We split the proof into three different parts.

Part (1) Intersection concentration. Let f NL = L I 1 ∩ I 2 f NL + L I 1 \ I 2 f NL + L I 2 \ I 1 f NL =: f 0 + f 1 + f 2 . Note that

where M j = ( X ⊤ L I 2 X /k + λ I p ) -1 . By conditional independence and Lemma S.8.5 of [51], the cross terms vanish

It remains to analyze the quadratic term of f 0 :

By conditioning on L I 1 ∩ I 2 X , from Lemma S.7.10 (1) of [12], we have

where ̂ Σ 0 = X ⊤ 0 X 0 /i 0 , X 0 = L I 1 ∩ I 2 X , C = ( k -i 0 ) /i 0 · ( v ( -λ ; γ 1 , Σ C 1 ) Σ + I p ) , C 1 = i 0 ( λ ( k -i 0 )) -1 ( ̂ Σ 0 + λ I p ) and i 0 = | I 1 ∩ I 2 | . Then we have

where X ′ 0 = X 0 ( I p + C ) -1 / 2 , A ′ = ( I p + C ) -1 / 2 A ⊤ A ( I p + C ) -1 / 2 , and M ′ = (( I p + C ) -1 / 2 ̂ Σ 0 ( I p + C ) -1 / 2 + λ I p ) -1 .

Part (2) Diagonal concentration. We next use a similar strategy as in the proof of Lemma A.16 in [14] by showing that the off-diagonal summation vanishes. Let Σ ′ = ( I p + C ) -1 / 2 Σ ( I p + C ) -1 / 2 . Since X 0 = Z 0 Σ 1 / 2 , we have X ′ 0 = Z 0 Σ ′ 1 / 2 . Then, the quadratic form becomes:

Note that from Lemma D.4, we have for any t > 0 ,

̸

Let U ∈ R n × n with U ij = [ f 0 ] i [ f 0 ] j 1 { i = j } . We then have

̸

For the first two terms, Lemma D.3 implies that

For the last term, note that ∥ B 2 ∥ op ≤ ∥ A ∥ 2 op ∥ ̂ Σ 0 ∥ op , where ∥ A ∥ op is almost surely bounded as assumed, and ∥ ̂ Σ 0 ∥ op ≤ r max (1 + √ ψ 2 /ϕ ) 2 almost surely as k, n, p →∞ and p/n → ϕ ∈ (0 , ∞ ) , p/k → ψ ∈ [ ϕ, ∞ ] (see, e.g., [45]). Also, ∥ U ∥ ∗ ≤ 2 ∥ f NL ∥ 2 2 a.s. - - → 2 ∥ f NL ∥ 2 L 2 < ∞ from the strong law of large numbers, Lemma D.5, and ∥ B 1 ∥ op ≤ λ -1 . Thus, the last term is almost surely bounded. It then follows that k -1 |⟨ B -1 1 B 2 B -1 1 , U ⟩| a.s. - - → 0 . Therefore,

Part (3) Trace concentration. From the results in [56], it holds that

Further, n -1 ∑ n i =1 f 2 i a.s. - - →∥ f NL ∥ 2 L 2 by strong law of large number and Lemma D.5. Therefore, we have

Combining (12), (13), and (16) yields that

By the trace property tr[( A λ k ) ⊤ A ⊤ AA λ k ] = tr[ A λ k ( A λ k ) ⊤ A ⊤ A ] , it remains to derive the deterministic equivalents of A λ k ( A λ k ) ⊤ A ⊤ A . Note that

where ̂ Σ 1 ∩ 2 = X ⊤ L I 1 ∩ I 2 X / | I 1 ∩ I 2 | and M j = ( X ⊤ L I 1 X /k + λ I p ) -1 for j = 1 , 2 . The above quantity is well-defined almost surely because | I 1 ∩ I 2 | converges to some positive quantity almost surely. Next, we analyze the trace term for the two cases.

From Lemma D.1 (3) we know that when A is independent to ( X , y ) , it follows that

We now have

where the second convergence is from Lemma S.8.3 of [12] and the trace property in Lemma E.3 (4).

(2) A = n -1 / 2 X .

Instead of working on (13), we use the following decomposition:

Then, repeating Part (2) and (3) as above, it suffices to derive the deterministic equivalents of ( A λ k ) ⊤ ̂ Σ A λ k . We decompose this term into:

For the last two terms of the right hand side, following the similar argument as in Part (1), it holds that

For the first term of the right hand side, notice that it is the variance term of the mean squared error computed on all samples, which is the variance term of the numerator of the generalized cross-validation (GCV) estimator in [13]. It has been shown in Proposition 3.6 of [13] that in the full ensemble, the GCV estimator is consistent with the prediction risk, which is also true for the variance term:

Finally, combining (14)-(16) with the above finishes the proof for the first convergence result of A = n -1 / 2 X .

Analogously, from (20), we also have

which finishes the proof for the second convergence result of A = n -1 / 2 X .

Lemma A.3 (Cross terms of generalized risk) . Suppose the same assumptions in Theorem 1 hold with λ > 0 . Let A λ k = E I ∼I k [( X ⊤ L I X /k + λ I p ) -1 X ⊤ L I /k ] and B λ k = I p -A λ k X = E I ∼I k [ λ ( X ⊤ L I X /k + λ I p ) -1 ] . For any A with lim sup ∥ A ∥ op bounded almost surely, it holds that

Proof. Note that

Let s 2 j be the singular value of ̂ Σ j . From the results in [45], we have lim sup ∥ ̂ Σ I ∥ op ≤ lim sup max 1 ≤ i ≤ p s 2 i ≤ r max (1 + √ ψ ) 2 almost surely as k, p → ∞ and p/k → ψ ∈ (0 , ∞ ] . On the other hand, n/k a.s. - - → ψ/ϕ and ∥ A ∥ 2 is uniformly bounded as assumed. Thus, we have lim sup n ∥ ( A λ k ) ⊤ A ⊤ AB λ k β 0 ∥ 2 2 is bounded almost surely.

Similar to the proof of Theorem 1, we decompose f NL as f NL = L I 1 ∩ I 2 f NL + L I 1 \ I 2 f NL + L I 2 \ I 1 f NL =: f 0 + f 1 + f 2 . We then have

where the first convergence is from Lemma S.8.5 of [51] and the second convergence is from (13), with X ′ 0 = X 0 ( I p + C ) -1 / 2 , A ′ = ( I p + C ) -1 / 2 A ⊤ A ( I p + C ) -1 / 2 , and M ′ = ( ̂ Σ ′ 0 + λ I p ) -1 and ̂ Σ ′ 0 = ( I p + C ) -1 / 2 ̂ Σ 0 ( I p + C ) -1 / 2 . From Lemma D.2, the first term in the above display vanishes. Analogous to (15), the second term also vanishes by splitting and applying Lemma D.2. Thus, we have for I 1 , I 2 ∼ I k ,

Finally, by Lemma G.5 (2) of [13], the conclusion follows.

B Proofs of results in Section 4

B.1 Proof of Theorem 3

We consider two cases.

(1) λ 1 , λ 2 > 0 .

We begin to prove for the full ensemble when M = ∞ . For j = 1 , 2 and I j ∼ I k j , define ̂ Σ j = X ⊤ L I j X /n and M j = ( X ⊤ L I j X /k + λ j I p ) + . Recall that y = Xβ 0 + f NL and

Then, for a ∈ R p with bounded l 2 norm, we have

Next, we analyze the three terms separately.

For the first term, from Lemma D.1 (1) we have for λ j > 0 ,

where v ( -λ j ; ψ j ) is as defined in (40). By Lemma C.1 we have that for ψ ∈ [ ϕ, ∞ ] there exists a segment P such that for all ( λ 1 , ψ 1 ) , ( λ 2 , ψ 2 ) ∈ P , it holds that v ( -λ 1 ; ψ 1 ) = v ( -λ 1 ; ψ 2 ) as p/k 1 → ψ 1 and p/k 2 → ψ 2 . By the definition of deterministic equivalents (Definition E.1), it follows that

For the second term, notice that by Lemma D.2,

From Lemma G.5 (2) of [13], it follows that

and T 2 a.s. - - → 0 .

Combining the above results and applying triangle inequality on (21) yields that

which completes the proof when M = ∞ . When M ∈ N , replacing the above expectation by the average over M simple random samples I 1 , I 2 ∼ I k completes the proof.

(2) λ 1 λ 2 = 0 .

When λ 1 = λ 2 = 0 , it is trivially true as k 1 = k 2 . Otherwise, without loss of generality, we assume λ 1 = 0 and λ 2 > 0 . We use the same decomposition (21) and first analyze T 1 . From part one we have that for λ > 0 , P n,λ -Q n,λ a.s. - - → 0 where

We next show that

by proving that the function P n,λ is equicontinuous family in λ over Λ = [0 , λ max ] for any λ max ∈ (0 , ∞ ) fixed. Note that for all λ ∈ Λ ,

and its derivative

are uniformly bounded in λ almost surely. In the above inequalities, the operator norm is bounded because ∥ ( X ⊤ L I 1 X /k 1 + λ I p ) -1 ̂ Σ 1 ∥ op ≤ s i / ( s i + λ ) ≤ 1 and ∥ ( X ⊤ L I 1 X /k 1 + λ I p ) -2 ̂ Σ 1 ∥ op ≤ s i / ( s i + λ ) 2 ≤ 1 / ( s i + λ ) ≤ 1 , by noting that lim sup ∥ ̂ Σ j ∥ op ≤ lim sup max 1 ≤ i ≤ p s 2 i ≤ r max (1 + √ ψ 1 ) 2 and lim inf ∥ ̂ Σ j ∥ op ≥ lim inf min 1 ≤ i ≤ p s 2 i ≥ r min (1 - √ ψ 1 ) 2 almost surely as k 1 , p → ∞ and p/k 1 → ψ 1 ∈ (0 , ∞ ) \ { 1 } [45]. On the other hand, since Q n,λ is a continuous function of v ( -λ ; ψ 1 ) , we have

When ψ 1 > 1 , ∂v ( -λ ; ψ 1 ) /∂λ is bounded over Λ [13, Lemma E.10 and Lemma E.11] and thus | ∂Q n,λ /∂λ | is also bounded almost surely. When ψ 1 < 1 , from Lemma E.12 of [13] we have

since v ( -λ ; ψ 1 ) ≤ v (0; ψ 1 ) = + ∞ . Therefore, | ∂Q n,λ /∂λ | is uniformly bounded over Λ for ψ 1 ∈ (0 , ∞ ) \ { 1 } . Thus, by the Moore-Osgood theorem, the convergence is uniform in λ and (22) follows. Finally, since T 2 is bounded analogously as in Part (1), the equivalence holds when λ 1 = 0 .

B.2 Proof of Proposition 4

From Lemma B.2, it follows that

By the continuity of function ϕ ↦→ v ( -λ ; ϕ ) from Lemma F.10 and F.11 of [13], we have v ( -λ ; ϕ n ) ≃ v ( -λ ; ϕ ) as ϕ n = p/n → ϕ . From Lemma C.1, there exists λ n such that

as ψ n → ψ and ϕ n → ϕ . Involving (23) on the both sides yields that

where { I 1 , . . . , I M } is a simple random sample from I k .

B.3 Proof of Corollary 5

We will use a structural equivalence (much more direct than the first-order equivalence considered in the paper) between generalized ridge predictor and the isotropic ridge predictor to prove the result.

The generalized ridge predictor (9), trained on the subsampled dataset D I , can be expressed as:

Observe that we can equivalently manipulate the generalized ridge estimator into:

Recalling from Assumption 2 that X = Z Σ 1 / 2 , where Z ∈ R n × p contains z ⊤ i in the i -th row for i ∈ [ n ] , we obtain

We now define Σ G = G -1 / 2 Σ G -1 / 2 and consider a transformed feature matrix X G = Z Σ 1 / 2 G . Denote the transformed dataset corresponding to the new feature matrix by D G (where we keep the response vector as is). Using (24), we can write relate the generalized ridge estimator fitted on the dataset D to the standard ridge estimator fitted on the data D G (both at the same scalar regularization level λ ) by:

Observe that the transformed dataset D G satisfies Assumption 2 as the eigenvalues of G are bounded away from 0 and ∞ . Further, note that both Theorem 1 and Theorem 3 are invariant to linear transformations of the estimator. Thus, we conclude that the results of both the theorems continue to hold. The equivalence path now use the modified Σ G , which in turn changes the spectral distribution H p to that of Σ G , in defining the end points via (4), given by:

where ˜ r i for i ∈ [ p ] are eigenvalues of Σ G . This completes the proof.

B.4 Technical lemmas

Lemma B.1 (Relationship between ̂ m and ̂ v ) . For X ∈ R n × p and ϕ n = p/n , define

It holds that

̸

Proof. Let r be the rank of the matrix X . Denote by s i , i = 1 , . . . , r , the non-zero eigenvalues of the matrix X ⊤ X . Note that these are the same non-zero eigenvalues of the matrix XX ⊤ . Define function S such that for z = 0 ,

For z = 0 , write out ̂ v and ̂ m in terms of S ( z ) as:

̸

We now expand the left-hand side of (25):

which finishes the proof.

Lemma B.2 (Equivalence of ̂ v and v ) . Suppose Assumptions 1-2 hold. Then it holds that

Proof. From Lemma B.1, we have

Substituting z = -λ yields that

From Corollary E.4, we have

This implies

where the last equality follows from (26). This concludes the proof.

C Proof of results in Section 5

C.1 Proof of Theorem 6

By Assumptions 3-4, for A = Σ 1 / 2 and b = 0 , R p ( λ ; ϕ, ψ ) , as defined in (11), converges to

where the nonnegative constants ˜ c ( -λ ; ϕ, ψ ) and ˜ v ( -λ ; ϕ, ψ ) are defined through the following equations:

From the proof of Theorem 1, we also have that R ( λ ; ϕ, ψ ) ≃ R p ( λ ; ϕ, ψ ) ≃ R ( ̂ β λ k, ∞ ; Σ , 0 , β 0 ) .

Since R (0; ϕ, ψ ) is a continuous function of ϕ and v (0; ψ ) and is increasing in ϕ for any fixed ψ , it follows that for 0 < ϕ 1 ≤ ϕ 2 < ∞ ,

where the first inequality follows because { ψ : ψ ≥ ϕ 1 } ⊇ { ψ : ψ ≥ ϕ 2 } , and the second inequality follows because R (0; ϕ, ψ ) is increasing in ϕ for a fixed ψ . Thus, min ψ ≥ ϕ R (0; ϕ, ψ ) is a continuous and monotonically increasing function in ϕ .

Finally, note that from Lemma C.1, for any λ , there exists ψ such that v (0; ϕ, ψ ) = v ( -λ ; ϕ, ϕ ) . This implies that R (0; ϕ, ψ ) = R ( λ ; ϕ, ϕ ) . Then we have min ψ ≥ ϕ R (0; ϕ, ψ ) ≤ min λ ≥ 0 R ( λ ; ϕ, ϕ ) . Conversely, since for any ψ , there exists λ such that v (0; ϕ, ψ ) = v ( -λ ; ϕ, ϕ ) , we also have min ψ ≥ ϕ R (0; ϕ, ψ ) ≥ min λ ≥ 0 R ( λ ; ϕ, ϕ ) . Combining the two inequalities yields the conclusion.

C.2 Technical lemmas

In this section, we gather results on certain analytic properties of the fixed-point solution v ( -λ ; ϕ ) defined in (40).

Lemma C.1 (Contour of fixed-point solutions) . As n, p →∞ such that p/n → ϕ ∈ (0 , ∞ ) , for any ψ ∈ [ ϕ, + ∞ ] , there exists a unique value λ ≥ 0 (or conversely for λ ∈ [0 , ∞ ] , there exists a unique value ψ ∈ [ ϕ ∨ 1 , ∞ ] ) such that for all ( λ, ψ ) on the path

where v p ( -λ ; ψ ) is as defined in (40) .

Proof. Since when ϕ < 1 , v (0; ψ ) = + ∞ for all ψ ∈ [ ϕ, 1] , we can restrict ourselves to ψ ∈ [ ϕ ∨ 1 , + ∞ ] . From Lemma E.11 (1) of [13], the function ψ ↦→ v (0; ψ ) is strictly decreasing over ψ ∈ [ ϕ ∨ 1 , ∞ ] with range

From Lemma E.12 (3) of [13], the function λ ↦→ v ( -λ ; ϕ ) is strictly decreasing over λ ∈ [0 , ∞ ] with range

Note that v (0; ϕ ∨ 1) = v (0; ϕ ) . For ψ ∈ [ ϕ ∨ 1 , ∞ ] , by the intermediate value theorem, there exists unique λ ∈ [0 , ∞ ] such that v ( -λ ; ϕ ) = v (0; ψ ) . Furthermore, when ψ ≤ 1 , λ = 0 is also the unique it holds that

value such that v (0; ψ ) = v ( -λ ; ϕ ) . Conversely, for λ ∈ [0 , ∞ ] , there also exists ψ ∈ [ ϕ ∨ 1 , ∞ ] such that v ( -λ ; ϕ ) = v (0; ψ ) .

Based on the definition of fixed-point solutions, it follows that

Then, for any ( λ, ψ ) = (1 -θ )( λ, ϕ ) + θ (0 , ψ ) on the path P , we have

Because v p ( -λ ; ψ ) is the unique solution to the fixed-point equation:

1

-

v

p

(

λ

;

ψ

)

1 +

it then follows that v p ( -λ ; ψ ) = v ( -λ ; ϕ ) = v (0; ψ ) .

D Asymptotic equivalents, concentration results, and other useful lemmas

D.1 Full-ensemble resolvents

Lemma D.1 (Full-ensemble resolvents) . Let ̂ Σ = X ⊤ X /n , ̂ Σ j = X ⊤ L I j X /k where I j ∼ I k . Let Σ 1 ∩ 2 = X ⊤ L I 1 ∩ I 2 X / | I 1 ∩ I 2 | and C ∈ R p × p with bounded operator norm almost surely. As n, p, k →∞ , p/n → ϕ , p/k → ψ , the following asymptotic equivalences hold:

Basic ridge resolvent:

Bias resolvent with C ⊥ ⊥ X :

Variance resolvent with C ⊥ ⊥ X :

Bias resolvent with C = ̂ Σ :

Here v ( -λ ; ϕ ) is as defined in (41) , and we let ˜ v p ( -λ ; ϕ, ψ ) = ˜ v p ( -λ ; ϕ, ψ, Σ ) ,

∫

r

(

-

=

λ

+

ψ

v

p

λ

;

ψ

)

d

H

p

(

r

)

,

The empirical distribution H n of eigenvalues can be replaced by the limiting distribution H whenever it exists.

Proof. The proofs for different parts is separated below.

(1) Basic ridge resolvent.

From Definition E.2, we know that λ ( ̂ Σ j + λ I p ) -1 ≃ ( v p ( -λ ; ψ ) Σ + I p ) -1 . By the definition of deterministic equivalents in Definition E.1, for any A ∈ R p × p that has bounded trace norm and is independent to X , we have

From Lemma G.5 (2) of [13], it follows that

which implies that

(2) Bias resolvent with C ⊥ ⊥ X .

Denote M j = ( ̂ Σ j + λ I p ) -1 for j = 1 , 2 . From Part (c) of the proof for Lemma S.2.4 in [12], it follows that for I 1 , I 2 ∼ I k ,

By the same argument as in (28), the conclusion follows.

(3) Variance resolvent with C ⊥ ⊥ X .

From Lemma E.8 (3) of [13], it follows that for I 1 , I 2 ∼ I k ,

By the same argument as in (28), the conclusion follows.

(4) Bias resolvent with C = ̂ Σ .

We begin by decomposing the object inside expectation into two terms:

where ̂ Σ 1 ∪ 2 = X ⊤ L I 1 ∪ I 2 X / | I 1 ∪ I 2 | and ̂ Σ (1 ∪ 2) c = X ⊤ ( I -L I 1 ∪ I 2 ) X / ( n -| I 1 ∪ I 2 | ) . Next, we analyze each of them.

For the first term, from Lemma D.6 (3) of [13] we have

where ˜ v p ( -λ ; ϕ, ψ ) = ˜ v p ( -λ ; ϕ, ψ, Σ ) .

For the second term, from Lemma E.8 (1) of [13] and the product rule of calculus in Lemma E.3 (3), we have

From Part (2) and Lemma E.3 (1), it follows that

Finally, for the coefficients of the two terms, from Lemma S.8.3 of [12], we have that

Combining (29)-(32) yields that

where the last equality follows by substituting

based on the fixed-point equation.

D.2 Convergence of random linear and quadratic forms

Lemma D.2 (Concentration of linear form with independent components and varying coefficients) . Let z i ∈ R p for i = 1 , . . . , n be a sequence of random vectors with i.i.d. entries z ij , j = 1 , . . . , p such that for each i, j , E [ z ij ] = 0 , E [ z 2 ij ] = 1 , E [ | z ij | 4+ α ] ≤ M α for some α > 0 and constant M α < ∞ . Let Z = [ z 1 , . . . , z n ] ⊤ ∈ R n × p be the random matrix formed by concatenating z i 's. Let g : R p → R be any measurable function such that E [ g ( z i )] = 0 and E [ z i g ( z i )] = 0 for i = 1 , . . . , n . Let a p ∈ R p be a sequence of random vectors independent of z p such that lim sup p ∥ a p ∥ 2 /p ≤ M 0 almost surely for a constant M 0 < ∞ . Let D be a positive semidefinite matrix such that lim sup ∥ D ∥ op ≤ M 0 almost surely as p →∞ for some constant M 0 < ∞ . Then, as n, p →∞ such that p/n → ϕ ∈ (0 , ∞ ) , we have

Proof. We will use the standard leave-one-out trick to break the dependence between the i -th component multiplier in the summation in (33) and g ( z i ) for each i = 1 , . . . , n . To that end, using the Woodbury matrix identity, first observe that

Plugging (34) back into (33), we expand the desired sum into:

where in step (35), we denote by:

and step (36) follows since d i ≥ 0 . It is easy to check that lim sup p ∥ b i ∥ 2 2 /p ≤ C 1 < ∞ , and d i a.s. - - → C 2 < ∞ for some constants C 1 and C 2 . Appealing to Lemma S.8.5 of [51], (36) almost surely converges to 0 . This finishes the proof.

Lemma D.3 (Concentration of sum of quadratic forms with independent components and independent varying inner matrices) . Let z p ∈ R p be a sequence of random vector with i.i.d. entries z pi , i = 1 , . . . , p such that for each i, E [ z pi ] = 0 , E [ z 2 pi ] = 1 , E [ | z pi | 4+ α ] ≤ M α for some α > 0 and constant M α < ∞ . Let Z = [ z 1 , . . . , z n ] ⊤ ∈ R n × p be the design matrix. Let g : R p → R be any measurable function such that E [ z i g ( z i )] = 0 and E [ g ( z i )] = 1 . Let D be a positive semidefinite matrix such that lim sup ∥ D ∥ op ≤ M 0 almost surely as p →∞ for some constant M 0 < ∞ . Then, as n, p →∞ such that p/n → ϕ ∈ (0 , ∞ ) ,

̸

̸

Proof. The strategy in the proof is to express each of the ij -the entry of the resolvent in (37) such that the dependence on ( z i , z j ) and the rest of ( z k : k = i, j ) is separated. One is then able to use the

uncorrelatedness of z and g ( z ) along with standard concentration of a quadratic form with respect to an independent matrix. Similar strategy has been used in [14] when analyzing kernel ridge regression under proportional asymptotics.

̸

Denote Y = ( ZDZ ⊤ /n + λ I n ) -1 . For each pair of i, j ∈ [ n ] and i = j , we let Z -( ij ) ∈ R ( n -2) × p be the matrix comprising the n -2 rows of Z excluding the i th and j th rows, and U ∈ R p × 2 be the matrix with columns U ij e i = z i , Ue j = z j . We finally define the matrices R -( ij ) ∈ R p × p as follows:

and let ˜ Y ij = { Y m,ℓ } m,ℓ ∈{ i,j } be the submatrix of Y . Then, using the block matrix inversion formula (see, e.g., Appendix A.1.4 of [45]) and the Woodbury matrix identity, one can show that

where d ij is given by:

̸

Since z i , z j , and R -( ij ) are mutually independent, by Lemma S.8.5 of [51], we have 〈 z i , R -( ij ) z j 〉 /n a.s. - - → 0 and the denominator concentrates on λ 2 . By Lemma G.5 (1) of [13], it follows that max 1 ≤ i = j ≤ n d ij a.s. - - → λ 2 . Thus, there exists N 0 ∈ N such that for all n ≥ N 0 , max 1 ≤ i = j ≤ n d ij ≥ λ 2 / 2 almost surely. Then, it follows that for n ≥ N 0 ,

̸

̸

̸

̸

where the last convergence is from Lemma S.8.5 of [51] and Lemma G.5 (2) of [13].

̸

Lemma D.4. For any two conforming matrices A and B and any t = 0 , we have

Proof. We recall the Woodbury matrix identity:

This holds for any conforming matrices A , U , C , and V . We will need to apply the Woodbury matrix identity twice below.

Applying the Woodbury identity for the first time with A = A , U = t , C = B , and V = 1 , we get

Rearranging, this yields

Applying the Woodbury identity the second time with A = t B , U = 1 , C = A , and V = 1 , we get

Multiplying by t B from the left yields

Now, multiplying by B on the right, notice that

From (38), observe that

Using (39), we then have

Rearranging, we arrive at the desired equality:

D.3 Concentration of energy of linear and nonlinear components

Lemma D.5. Under Assumptions 1-2, the best linear estimator β 0 and the nonlinear component f NL as defined in (10) satisfies that ∥ β 0 ∥ 2 and ∥ f NL ∥ L 4+ δ are bounded almost surely.

Proof. By Jensen's inequality and Assumption 1, we have E [ y 2 ] ≤ E [ y 4+ δ ] 2 / (4+ δ ) ≤ C 2 / (4+ δ ) 0 for some constant C 0 > 0 .

By the orthogonality, we have that E [ y 2 ] = E [( x ⊤ β 0 ) 2 ] + ∥ f NL ∥ 2 L 2 , which implies that E [( x ⊤ β 0 ) 2 ] and ∥ f NL ∥ 2 L 2 is bounded. Since by Assumption 2, the eigenvalues of Σ is lower bounded by r min > 0 , we have that E [( x ⊤ β 0 ) 2 ] = β 0 Σ β 0 ≥ r min ∥ β 0 ∥ 2 2 and thus ∥ β 0 ∥ 2 is bounded.

Since f NL = y -x ⊤ β 0 , by triangle inequality we have that

for some constant C > 0 . The second inequality of the above display is from Lemma 7.8 of [57]. Since ∥ y ∥ L 4+ δ and ∥ β 0 ∥ 2 are bounded, it follows that ∥ f NL ∥ L 4+ δ is also bounded.

E Asymptotic equivalents: background and known results

Preliminary background. In several proofs, we employ the concept of asymptotic equivalence of sequences of random matrices; see [12, 44, 49, 51]. This section provides a brief overview of the associated terminology and calculus principles.

Definition E.1 (Asymptotic equivalence) . Consider sequences { A p } p ≥ 1 and { B p } p ≥ 1 of (random or deterministic) matrices of growing dimensions. We say that A p and B p are equivalent and write A p ≃ B p if lim p →∞ | tr[ C p ( A p -B p )] | = 0 almost surely for every sequence of random matrices C p independent to A p and B p , with bounded trace norm such that lim sup p →∞ ∥ C p ∥ tr < ∞ almost surely.

The notion of asymptotic equivalence of two sequences of random matrices above can be further extended to incorporate conditioning on another sequence of random matrices; see [12] for more details.

Definition E.2 (Conditional asymptotic equivalence) . Consider sequences { A p } p ≥ 1 , { B p } p ≥ 1 and { D p } p ≥ 1 of (random or deterministic) matrices of growing dimensions. We say that A p and B p are equivalent given D p and write A p ≃ B p | D p if lim p →∞ | tr[ C p ( A p -B p )] | = 0 almost surely conditional on { D p } p ≥ 1 , i.e.,

for any sequence of random matrices C p independent to A p and B p conditional on D p , with bounded trace norm such that lim sup ∥ C p ∥ tr < ∞ as p →∞ .

Below we summarize the calculus rules for conditional asymptotic equivalence in Definition E.2. These are adapted from Lemma S.7.4 and S.7.6 of [12].

Lemma E.3 (Calculus of asymptotic equivalents) . Let A p , B p , C p and D p be sequences of random matrices. The calculus of asymptotic equivalents satisfies the following properties:

Equivalence: The relation ≃ is an equivalence relation.

Sum: If A p ≃ B p | E p and C p ≃ D p | E p , then A p + C p ≃ B p + D p | E p .

Product: If A p has bounded operator norms such that lim sup p →∞ ∥ A p ∥ op < ∞ , A p is conditional independent to B p and C p given E p for p ≥ 1 , and B p ≃ C p | E p , then A p B p ≃ A p C p | E p .

Trace: If A p ≃ B p | E p , then tr[ A p ] /p -tr[ B p ] /p → 0 almost surely when conditioning on E p .

Differentiation: Suppose f ( z, A p ) ≃ g ( z, B p ) | E p where the entries of f and g are analytic functions in z ∈ S and S is an open connected subset of C . Suppose for any sequence C p of deterministic matrices with bounded trace norm we have | tr[ C p ( f ( z, A p ) -g ( z, B p ))] | ≤ M for every p and z ∈ S . Then, we have f ′ ( z, A p ) ≃ g ′ ( z, B p ) | E p for every z ∈ S , where the derivatives are taken entrywise with respect to z .

Unconditioning: If A p ≃ B p | E p , then A p ≃ B p .

Substitution: Let v : R p × p → R and f ( v ( C ) , C ) : R p × p → R p × p be a matrix function for matrix C ∈ R p × p and p ∈ N , that is continuous in the first augment with respect to operator norm. If v ( C ) a.s. = v ( D ) such that C is independent to D , then f ( v ( C ) , C ) ≃ f ( v ( D ) , C ) | C .

Standard ridge resolvents and various extensions. In this section, we collect various asymptotic equivalents. The following corollary is a simple consequence of Theorem 1 in [58]. It provides deterministic equivalent for the scaled ridge resolvent.

Corollary E.4 (Deterministic equivalent for scaled ridge resolvent) . Suppose x i ∈ R p , 1 ≤ i ≤ n , are i.i.d. random vectors such that each x i = z i Σ 1 / 2 , where z i is a random vector consisting of i.i.d. entries z ij , 1 ≤ j ≤ p , satisfying E [ z ij ] = 0 , E [ z 2 ij ] = 1 , and E [ | z ij | 8+ α ] ≤ M α for some constants α > 0 and M α < ∞ , and Σ ∈ R p × p is a positive semidefinite matrix satisfying 0 ⪯ Σ ⪯ r max I p for some constant r max < ∞ (independent of p ). Let X ∈ R n × p the concatenated matrix with x ⊤ i , 1 ≤ i ≤ n , as rows, and let ̂ Σ ∈ R p × p denote the random matrix X ⊤ X /n . Let γ = p/n . Then, for z ∈ C + , as n, p →∞ such that 0 < lim inf γ ≤ lim sup γ < ∞ , and λ > 0 , we have

where v ( -λ ; γ ) > 0 is the unique solution to the fixed-point equation

Here H n is the empirical distribution (supported on R ≥ 0 ) of the eigenvalues of Σ .

Side remark: The parameter v ( -λ ; γ ) in Corollary E.4 is also the companion Stieltjes transform of the spectral distribution of the sample covariance matrix ̂ Σ . It is also the Stieltjes transform of the spectral distribution of the gram matrix XX ⊤ /n . This is essentially the characterization we use in proving Proposition 4.

The following lemma uses Corollary E.4 along with calculus of deterministic equivalents (from Lemma E.3), and provides deterministic equivalents for resolvents needed to obtain asymptotic bias and variance of standard ridge regression. It is adapted from Lemma S.6.10 of [51].

Lemma E.5 (Deterministic equivalents for ridge resolvents associated with generalized bias and variance) . Suppose x i ∈ R p , 1 ≤ i ≤ n , are i.i.d. random vectors with each x i = z i Σ 1 / 2 , where z i ∈ R p is a random vector that contains i.i.d. random variables z ij , 1 ≤ j ≤ p , each with E [ z ij ] = 0 , E [ z 2 ij ] = 1 , and E [ | z ij | 4+ α ] ≤ M α for some constants α > 0 and M α < ∞ , and Σ ∈ R p × p is a positive semidefinite matrix with r min I p ⪯ Σ ⪯ r max I p for some constants r min > 0 and r max < ∞ (independent of p ). Let X ∈ R n × p be the concatenated random matrix with x i , 1 ≤ i ≤ n , as its rows, and define ̂ Σ = X ⊤ X /n ∈ R p × p . Let γ = p/n . Then, for λ > 0 , as n, p → ∞ with 0 < lim inf γ ≤ lim sup γ < ∞ , the following statements hold:

Bias of ridge regression:

Variance of ridge regression:

Here v ( -λ ; γ, Σ ) > 0 is the unique solution to the fixed-point equation

and ˜ v b ( -λ ; γ, Σ ) and ˜ v v ( -λ ; γ, Σ ) are defined through v ( -λ ; γ, Σ ) by the following equations:

where H n ( · ; Σ ) is the empirical distribution (supported on [ r min , r max ] ) of the eigenvalues of Σ .

Though Lemma E.5 states the dependency explicitly, we will simply write H p ( r ) , v ( -λ ; γ ) , ˜ v b ( -λ ; γ, A ) , and ˜ v v ( -λ ; γ ) to denote H n ( r ; Σ ) , v ( -λ ; γ, Σ ) , ˜ v b ( -λ ; γ, Σ , A ) , and ˜ v v ( -λ ; γ, Σ ) , respectively, for simplifying various notations when it is clear from the context. When A = Σ , we simply write ˜ v b ( -λ ; γ ) = ˜ v b ( -λ ; γ, A ) .

F Experiment details

F.1 Reproducibility and compute details

The source code for generating all of our figures is included with the supplementary material. The source code also includes details about the computational resources used to run the code and other timing details.

F.2 Simulation details

The covariance matrix of an auto-regressive process of order 1 (AR(1)) is given by Σ ar1 , where ( Σ ar1 ) ij = ρ | i -j | ar1 for some parameter ρ ar1 ∈ (0 , 1) . Define β 0 := 1 5 ∑ 5 j =1 w ( j ) where w ( j ) is the eigenvector of Σ ar1 associated with the top j th eigenvalue r ( j ) . We generated data ( x i , y i ) for i = 1 , . . . , n from a nonlinear model:

where σ 5 = √ 5 / 3 is the standard deviation of t 5 distribution. The benefit of using the above nonlinear model is that we can clearly separate the linear and the nonlinear components and compute the quantities of interest because β 0 happens to be the best linear projection.

For the simulations, we set ρ ar1 = 0 . 5 . For finite ensembles, the risks are averaged across 50 simulations.

F.3 Risk comparisons along equivalence paths for finite ensembles

Figure F5 verifies the following linear (in M ) relationship along the path in Theorem 3 for finite ensembles: R ( ̂ β λ 1 ⌊ p/ψ 1 ⌋ ,M ; A , b , β 0 ) -R ( ̂ β λ 2 ⌊ p/ψ 2 ⌋ ,M ; A , b , β 0 ) ≃ ∆ /M, for some ∆ (independent of M ), which is eventually almost surely bounded.

1

10

100

10

4

10

3

Range

Estimation risk

1

10

100

Ensemble size

M

10

1

10 0

Training error

1

10

100

10

1

10 0

Prediction risk

1

10

100

10

1

10 0

Prediction risk (OOD)

F.4 Illustration of risk monotonicity of optimal ridge

0.1

0.3

1

3

10

Subsample aspect ratio

0.1

0.3

1

3

10

Data aspect ratio

increasing

0.1

0.3

1

3

10

Ridge penalty

increasing

1.1

1.2

1.3

1.4

1.5

1.6

1.1

1.2

1.3

1.4

1.5

1.6

F.5 Real-world datasets

We conduct experiments on real-world datasets to examine the equivalence in a more general setting. We utilized three image datasets for our experimental analysis: CIFAR 10, MNIST, and USPS [59]. For the CIFAR 10 dataset, we subset the images labeled as 'dog' and 'cat'. For other datasets, we subset the images labeled '3' and '8'. Then we treat them as binary labels y ∈ { 0 , 1 } and use the flattened image as our feature vector x . The training sample sizes, the feature dimensions, and the test sample sizes ( n, p, n te ) are (10000, 3072, 2000), (12873, 784, 2145), and (22358, 3072, 7981) for the three datasets, respectively.

For the first experiment in Figure 3, we fix ϕ = p/n and ψ = 4 ϕ using the training set of CIFAR-10. For λ ψ ∈ { 0 . 01 , 0 . 05 , 0 . 1 , 1 } , we compute the data-dependent value of λ based on Proposition 4. Each value of λ ψ gives a path between ( λ ψ , ψ ) and ( λ, ϕ ) .

For the second experiment in Figure F7, by varying the values of subsample aspect ratios ψ , we compare the random Gaussian linear projection of the estimators and the prediction risk at the two endpoints ( λ, ϕ ) and (0 , ψ ) , on CIFAR-10, MNIST, and USPS datasets.

)

2

4

6

8

10

Subsample aspect ratio

0.00

0.01

0.02

Value

Random linear projection

2

4

6

8

10

Subsample aspect ratio

0.6

0.8

1.0

Prediction risk

Dataset

CIFAR10

MNIST

USPS

Model

(

,

)

(0,

F.6 Random features model

The data ( x i , y i ) is generate from the nonlinear model

where x ij iid ∼ N (0 , 1) , and ε i ∼ N (0 , 1) . The prediction risk of the ridge ensemble ( M = 100 ) is computed based on the random feature φ ( Fx i ) and the response y i , where F ∈ R d × p is the random weight matrix with F ij iid ∼ N (0 , p -1 ) . Here, φ is a nonlinear activation function (sigmoid, ReLU, or tanh) that is applied entry-wise to Fx i . For the experiment, we set p = 250 , d = 500 , and ϕ = d/n = 0 . 1 .

F.7 Kernel ridge regression

For a given feature map Φ : R p → R d , the kernel ridge estimator is defined as:

By using the kernel trick, the above optimization problem is equivalent to solving:

where K I = Φ I Φ ⊤ I ∈ R k × k is the kernel matrix and Φ I = (Φ( x i )) i ∈ I ∈ R n × d is the feature matrix. Simple calculation shows that ̂ β λ k ( D I ) = Φ ⊤ I ̂ α λ k ( D I ) .

Using the same data-generating process as in the previous subsection, the results on kernel ridge regression are illustrated in Figure F8.

0.1

1

10

0.0

0.1

1

10

Ridge penalty

Polynomial

0.1

1

10

Subsample aspect ratio

0.1

1

10

Laplacian

1.230

1.232

1.234

1.236

1.238

1.08

1.10

1.12

1.14

1.16

1.10

1.12

1.14

1.16

1.18

1.20

1.22

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors investigate the problem of ridge regression, and prove equivalence results between ridge regularisation and ensambling of weak learners trained on subsamples of the original dataset.\nThe equivalences hold under very mild assumptions, and notably there is no requirement on the data model.\nTwo kind of equivalence are proven: i) equivalence at the level of a quite generic class of risks ii) equivalence at the level of the ridge estimator itself.\nThe equivalence basically say that one can trade a bit of subsampling for a bit of ridge regression without altering the performance of the estimator.\n\nThe equivalences hold on paths in the plane defined by the ridge regulariser and the subsampling ratio. \nThe authors provide both a \"theoretical\" characterisation of such paths, which requires knowledge of the population covariance of the features, and a \"data-driven\" characterisation, which requires only access to the sample covariance of the features.\n\nFinally, the authors discuss possible extensions of their results to real-data scenarios and random features regression.\n\nSTRENGTHS:\nThe works seems sound, relevant (answering open questions in the literature), well-motivated and well-presented.\nCode is available for reproducibility.\n\nI would like to highlight that the authors need basically no structural assumptions on the data model to prove the equivalence.\n\nWEAKNESSES:\nI did not identify any substantial weakness in the paper.\n\nQUESTIONS:\nI do not have any question of the authors. \nI only suggest the authors to improve Figure 3: the caption could additionally describe the difference in x-axis between left and right panel.\n\nLIMITATIONS:\nThe authors do not discuss explicitly limitations. I find that this is not strongly necessary, as the assumptions under which the results hold are clearly presented.\nThe only improvement I could suggest is for the authors to discuss whether they expect the equivalences they prove to break in some specified setting.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper shows an asymptotic equivalence between an ensembled+subsampled (E+S) version of ridge regression and the standard version, in the proportional asymptotic regime. The equivalence result shows that there exists a linear path in the space (ridge-parameter, aspect-ratio) along which all estimators yield essentially the same solution.

This equivalence also resolves an important open question regarding the behavior of optimally tuned ridge regularization. Specifically, the generalization error monotonically decreases with overparameterization (assuming the same level of SNR).

STRENGTHS:
This paper was delightful to read.

The problem considered is of very broad interest. The contribution is fundamental and is potentially path-breaking. At the very least, it yields a very satisfactory understanding of the effect of ridge regularization under both important settings.
- optimal tuning
- interpolation (lambda = 0)

WEAKNESSES:
The paper can do with a round of proof-reading. There are some minor issues.

Perhaps the title should read "ensembled subsampling" instead of just "subsampling".

A sketch of the proof is missing. It would be good to highlight in a couple of paragraphs the core ideas behind the proof of the main result.

QUESTIONS:
In Section 2, does M have to grow at a certain rate wrt n ?

It appears that the Conjecture regarding Kernel Ridge Regression may already be within the reach of this paper, following the equivalence result from Sahraee-Ardakan et al. https://arxiv.org/pdf/2201.08082.pdf

LIMITATIONS:
No negative impact envisioned.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the relationship between  ridgeless ensembles constructed from subsampled data and a ridge estimator in a setting with mild assumptions on the joint distribution $(Y,X)$.  They establish equivalences for a generalized class of risk functionals, which include quantities related to coefficient estimation and both in and out of sample errors. These equivalences are proven in the case where the ensemble includes estimators trained on all possible subsamples of a given size; the authors extend these results to equivalences between two finite ensembles. The authors use these equivalence results to settle a conjecture in a previous paper about risk monotonicity of ridge regression as a function of $p/n$.\n\nSTRENGTHS:\n- The authors substantially relax the distributional assumptions considered by previous work in the literature. This is conceptually important since it was not known how critical the linearity assumption is for these types of equivalences to hold. \n- The theoretical analysis is both novel and technical, invoking various concepts from random matrix theory.\n- Overall, the paper is well-written and well-organized.\n\nWEAKNESSES:\nWhile equivalences between ridge regression and subsampling are conceptually interesting, at this stage it appears that consequences for data analysis are a bit limited.  However, the authors generously discuss several potential extensions for which some of the tools developed in the paper may be helpful.\n\nQUESTIONS:\n- In Theorem 3, is the ensemble size $M$ fixed?\n- While the paper is well-written overall, some additional exposition/clarification in certain parts may be helpful to readers.  For example, the authors could elaborate more on what they mean by first-order and second-order.  In addition, the coefficient confidence interval case can be explained more.  Also, before theorem 3, it is stated that \"We can go a step further and ask if there exist any equivalences for the finite ensemble and if there are any equivalences at the estimator coordinate level between the estimators.\"  Do you mean that the conditions in this theorem have been previously shown to imply equivalences at the coordinate level and these conditions also imply equivalences with finite ensembles or do you mean something else?\n\nLIMITATIONS:\nThe authors are quite forthcoming about various limitations of their work and possible future extensions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission establishes equivalences between ridge regression (i.e. $\\ell_2$-penalized\nlinear regression) and ensembles of linear models trained on sub-sampled datasets.\nIn particular, the authors prove that for a fixed feature/sub-sample-size $d/k$, ratio, there \nexists a ridge-regression model with risk asymptotically equivalent to the full\nensemble, i.e. the average of all $k$ models trained on sub-samples of size $k$. \nMoreover, this equivalence holds for all convex combinations of the \nridge-regression and full sub-sampled model.\nNote that the asymptotics assume $d$, $k$, $n$ approach infinity such that\nthe ratios $d/k$ and $d/n$ are held constant.\nThis equivalence result is then extended to other metrics, such as training error\nand the weight estimation error, and to \"structural\" results which\nshow asymptotic equivalence of the weights of the models.\nThe authors leverage these equivalences to show that the prediction\nrisk of the best ridge regression model is monotone increasing in $d/n$.\n\nSTRENGTHS:\nThe main strength of this work is the theoretical contributions. In \nparticular:\n\n- The authors extend exist results on equivalent risk of ridge regression\n    models and sub-sampled ensembles to new metrics and to the model weights\n    themselves (\"structural results\").\n\n- The theorems are proved under relaxed conditions compared to previous work.\n    In particular, general data distributions are allowed provided a fairly\n    weak assumption on the moments holds.\n\n- The authors answer an open problem on the behavior of risk for ridge regression\n    models with the optimal regularization constant.\n\nThe paper is also well written, with very few typos. I congratulate the authors\non their polished manuscript. \nRelated work appears to be correctly cited, although this is not my research\narea so it is hard for me to check.\nNote that I did not check the theoretical derivations in the appendix so I\ncannot comment on their correctness.\n\nWEAKNESSES:\nThe greatest weakness of this paper is that the theoretical results are\nsomewhat incremental and unlikely to have an impact outside of learning theory.\nIn particular,\n\n- The connection between ridge regression and sub-sampled ensembles was\n    previously established, so that the main contributions of this work \n    are weakened conditions and new types of equivalences.\n\n- It's not clear how interesting it is to answer the conjecture from \n    Nakkiran et al. While the authors prove that the risk for the ridge\n    regression model with optimal regularization constant is monotone\n    increasing in the ratio $d/n$, this only applies to the asymptotic\n    regime and so its practical importance may be limited.\n\n- While the authors motivate their work by highlighting connections between\n    ridge regularization and dropout, noisy training, data augmentation,\n    and early stopping. However, these connections are not developed any further\n    and I am skeptical the asymptotic equivalences in this submission will\n    impact those areas.\n\nQUESTIONS:\nLine 93: Is this supposed to mean that Assumption 2 defines RMT features? \n    It's not obvious because the acronym RMT is not defined anywhere and \n    not used in Assumption 2.\n\nLine 176: I suggest Changing the name of Assumption 2 from \"Feature Vector Distribution\" \n    to \"RMT\" features as well as including a definition for the initialism/acronym, since\n    it isn't stated anywhere.\n\nLine 156 and Definition E.1: Does $C_p$ need to be bounded away from zero? \n    Otherwise the asymptotic\n    equivalence definition will be meaningless when $C_p = 0$ almost surely \n    for every $p$. Or perhaps when you say \"any sequence $C_p$\" you mean\n    \"every sequence\"?\n\nLine 183: Shouldn't this solution be $(\\\\lambda, \\\\bar{\\\\psi})$?\n\nFigures 1/2: The paths between equivalent models don't appear to be linear \n    in these figures, although Equation 5 seems to indicate that they are always\n    linear combinations. Is this because the figures are in log-log scale?\n\nTheorem 3: I don't understand what is \"structural\" or \"first-order\" in this\n    theorem compared to Theorem 1. Is this because the estimators themselves\n    are equivalent, rather than a risk functional of the estimators?\n    While is equivalence of risk functionals \"second order\"?\n    \nTheorem 3: \"this implies that the predicted values (or even any continuous function\napplied to them due to the continuous mapping theorem) of any test point will eventually be the same,\nalmost surely, with respect to the training data.\"\n\nThe asymptotic equivalence of parameters in Theorem 3 and this \nstatement seem to imply that Theorem 3 covers both Theorem 1 and Proposition 2 \nby taking $M = \\\\infty$ --- is this correct?\nIf yes, what is the novelty of these two previous results given Theorem 3?\n\nProposition 4: Is Equation 8 always guaranteed to admit a solution?\n    Moreover, what is the utility of this result when the equivalence only\n    holds asymptotically? That is, when $n \\rightarrow \\infty$ and the\n    root-finding problem defined by Equation (8) is impractical to solve.\n\nLIMITATIONS:\nAs mentioned in the \"Weaknesses\" section, I think the main limitation of this \nsubmission is the impracticality of the main theoretical results.\nSince they extend and generalize a previous result showing asymptotic equivalence\nof the risk, I do not see the theoretical derivations having an impact on practice.\nFurthermore, the connections to high-interest topics in ML like early stopping\nand dropout seem tenuous at best. \n\nSince I am not actively involved in learning theory research, I cannot\ncomment on the importance of this work for other members of this community.\nIt would be nice if the authors could provide additional context for their\nwork, including some comments on the novelty of their proof techniques and so\non. That way I can better understand the impact on this specific research community.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work compares the ridgeless ensemble and the ridge estimators in the proportional limit setting (i.e., $d/n\\to \\phi$). Prior works [11,12,13] show that these two estimators achieve the same out-of-sample risk. The main contribution of this paper has been to (1) weaken the assumptions and (2) broaden the equivalence from out-of-sample risk to other risks such as empirical risk, in-sample risk, and transfer learning risk. Based on this theory, this work also shows that the out-of-sample risk achieved by the optimal ridge estimator monotonically increases as a function of the data aspect ratio (i.e., $\\phi$).\n\nSTRENGTHS:\n+ Excellent presentation. I have enjoyed going through the paper. \n+ Compared to [11,12,13], this work has shown equivalence between ridgeless ensemble and ridge estimators in a broader sense and under weaker assumptions. Notably, in this work, the data is allowed to be misspecified and the limiting covariance spectrum needs not to exist, and the proved equivalence holds for transfer learning risk, in-sample risk, empirical risk beside out-of-sample risk.\n+ Section 5 is a neat addition to the main results, showing that the optimal ridge induces an out-of-sample risk increasing with respect to the data aspect ratio.\n\nWEAKNESSES:\n- The Conjecture 1 in [1] is stated for every finite $n$ and $p$. Section 5 in this work only applies to the proportional limit setting, where $p/n\\to \\phi, n,p\\to\\infty$ for a finite $\\phi$. While I think Section 5 is very interesting, I am not sure it is proper to claim that \"This resolves a recent open problem raised by Nakkiran et al. [1] under general data distributions and mild regularity conditions.\"\n\n\n- The theory is limited to the proportional limit regime. To what extent the theory holds in the finite sample regime is unclear. \n\n\n- The technique novelty could be further clarified. \n\n- I tried to read the proof and believe they are mostly correct. However, there are a number of typos that confuse me (and prevent me from spending more time checking the proof). For an incomplete list:\n\n1. The equation after Line 514. Missing a factor of 2 in the cross-term. \n2. Line 520. $B = AX - I$, missing $-I$.\n3. Line 534. $v(- \\lambda_1; \\psi_1) = v(- \\lambda_2; \\psi_2)$. $\\lambda_2$ instead of $\\lambda_1$. \n4. The first inequality after Line 622. Left hand side of the inequality, $R(0; \\phi, \\psi)$ should be $R(0; \\phi_1, \\psi)$?\n\nQUESTIONS:\nUpto my quick glance, it seems the proof methods are largely built upon existing works such as [12] and [13]. I understand that this work has derived broader equivalence results under weaker assumptions compared to [11,12,13]. Would you please clarify what are the new ingredients in this work that allow so?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}