SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pratik Patil
Berkeley, CA 94720, USA
pratikpatil@berkeley.edu
Jin-Hong Du
& Machine Learning Department
Pittsburgh, PA 15213, USA
jinhongd@andrew.cmu.edu
We establish precise structural and risk equivalences between subsampling and ridge regularization for ensemble ridge estimators. Specifically, we prove that linear and quadratic functionals of subsample ridge estimators, when fitted with different ridge regularization levels  $\lambda$  and subsample aspect ratios  $\psi$ , are asymptotically equivalent along specific paths in the  $(\lambda, \psi)$ -plane (where  $\psi$  is the ratio of the feature dimension to the subsample size). Our results only require bounded moment assumptions on feature and response distributions and allow for arbitrary joint distributions. Furthermore, we provide a data-dependent method to determine the equivalent paths of  $(\lambda, \psi)$ . An indirect implication of our equivalences is that optimally tuned ridge regression exhibits a monotonic prediction risk in the data aspect ratio. This resolves a recent open problem raised by Nakkiran et al. [1] for general data distributions under proportional asymptotics, assuming a mild regularity condition that maintains regression hardness through linearized signal-to-noise ratios.
Ensemble methods, such as bagging [2, 3], are powerful tools that combine weak predictors to improve predictive stability and accuracy. This paper focuses on sampling-based ensembles, which exhibit an implicit regularization effect [4-6]. Specifically, we investigate subsample ridge ensembles, where the ridge predictors are fitted on independently subsampled datasets [7-9]. Recent work has demonstrated that a full ensemble (that is fitted on all possible subsampled datasets) of ridgeless [10] predictors achieves the same squared prediction risk as a ridge predictor fitted on full data [11-13].
To be precise, let  $\phi$  be the limiting dataset aspect ratio  $p / n$ , where  $p$  is the feature dimension, and  $n$  is the number of observations. For a given  $\phi$ , the limiting prediction risk of subsample ridge ensembles is parameterized by  $(\lambda, \psi)$ , where  $\lambda \geq 0$  is the ridge regularization parameter and  $\psi \geq \phi$  is the limiting subsample aspect ratio  $p / k$ , with  $k$  being the subsample size [11, 12]. Under isotropic features and a well-specified linear model, [11, 12] show that the squared prediction risk at  $(0, \psi^{*})$  (the optimal ridgeless ensemble) is the same as the risk at  $(\lambda^{*}, \phi)$  (the optimal ridge), where  $\psi^{*}$  and  $\lambda^{*}$  are the optimal subsample aspect ratio and ridge penalty, respectively. Furthermore, this equivalence of prediction risk between subsampling and ridge regularization is extended in [13] to anisotropic linear models. As an application, [13] also demonstrates how generalized cross-validation for ridge regression can be naturally transferred to subsample ridge ensembles. In essence, these results suggest that subsampling a smaller number of observations has the same effect as adding an appropriately larger level of ridge penalty. These findings prompt two important open questions:
(a) The extent of equivalences. The previous works all focus on equivalences of the squared prediction risk when the test distribution matches the train distribution. In real-world scenarios,
Table 1: Comparison with related work. The marker “√°” indicates a partial equivalence result connecting the optimal prediction risk of the ridge predictor and the full ridgeless ensemble predictor.
Type of equivalence results	Type of data assumptions
pred. risk	gen. risk	estimator	response	feature	lim. spectrum
LeJeune et al. [11]	✓°	linear	isotropic Gaussian	exists
Patil et al. [12]	✓°	linear	isotropic RMT	exists
Du et al. [13]	✓	linear	anisotropic RMT	exists
This work	✓	✓	✓	arbitrary	anisotropic RMT	need not exist
however, there are often covariate and label shifts, making it crucial to consider prediction risks under such shifts. In addition, other risk measures, such as training error, estimation risk, coefficient errors, and more, are also of interest in various inferential tasks. A natural question is then whether similar equivalences hold for such "generalized" risk measures. At a basic level, the question boils down to whether any equivalences exist at the "estimator level". Answering this question would establish a connection between the ensemble and the ridge estimators, facilitating the exchange of various downstream inferential statements between the two sets of estimators.
(b) The role of linear model. All previous works assume a well-specified linear model between the responses and the features, which rarely hold in practical applications. A natural question is whether the equivalences hold for arbitrary joint distributions of the response and the features or whether they are merely an artifact of the linear model. Addressing this question broadens the applicability of such equivalences beyond simplistic models to real-world scenarios, where the relationship between the response and the features is typically intricate and unknown.
We provide answers to questions raised in both directions. We demonstrate that the equivalences hold for the generalized squared risks in the full ensemble. Further, these equivalences fundamentally occur at the estimator level for any arbitrary ensemble size. Importantly, these results are not limited to linear models of the data-generating process. Below we provide a summary of our main results.
(1) Risk equivalences. We establish asymptotic equivalences of the full-ensemble ridge estimators at different ridge penalties  $\lambda$  and subsample ratios  $\psi$  along specific paths in the  $(\lambda, \psi)$ -plane for a variety of generalized risk functionals. This class of functionals includes commonly used risk measures (see Table 2), and our results hold for both independent and dependent coefficient matrices that define these risk functionals (see Theorem 1 and Proposition 2). In addition, we demonstrate that the equivalence path remains unchanged across all the functionals examined.
(2) Structural equivalences. We establish structural equivalences in the form of linear functionals of the ensemble ridge estimators, which hold for arbitrary ensemble sizes (see Theorem 3). Our proofs for both structural and risk equivalences exploit properties of certain fixed equations that arise in our analysis, enabling us to explicitly characterize paths that yield equivalent estimators in the  $(\lambda, \psi)$ -plane (see Equation (5)). In addition, we provide an entirely data-driven construction of this path using certain companion Stieltjes transform relations of random matrices (see Proposition 4).
(3) Equivalence implications. As an implication of our equivalences, we show that the prediction risk of an optimally-tuned ridge estimator is monotonically increasing in the data aspect ratio under mild regularity conditions (see Theorem 6). This is an indirect consequence of our general equivalence results that leverages the provable monotonicity of the subsample-optimized estimator. Under proportional asymptotics, our result settles a recent open question raised by Nakkiran et al. [1, Conjecture 1] concerning the monotonicity of optimal ridge regression under anisotropic features and general data models while maintaining a regularity condition that preserves the linearized signal-to-noise ratios across regression problems.
(4) Generality of equivalences. Our main results apply to arbitrary responses with bounded  $4 + \mu$  moments for some  $\mu > 0$ , as well as features with similar bounded moments and general
covariance structures. We demonstrate the practical implications of our findings on real-world datasets (see Section 6). On the technical side, we extend the tools developed in [14] to handle the dependency between the linear and the non-linear components and obtain model-free equivalences. Furthermore, we extend our analysis to include generalized ridge regression (see Corollary 5). Through experiments, we demonstrate the universality of the equivalences for random and kernel features and conjecture the associated data-driven prediction risk equivalence paths (see Section 6).
Ensemble learning. Ensemble methods yield strong predictors by combining weak predictors [15]. The most common strategy for building ensembles is based on subsampling observations, which includes bagging [2, 3], random forests [16], neural network ensembles [17-19], among the others. The effect of sampling-based ensembles has been studied in the statistical physics literature [20-22] as well as statistical learning literature [2, 3]. Recent works have attempted to explain the success of ensemble learning by suggesting that subsampling induces a regularizing effect [4, 6]. Under proportional asymptotics, the effect of ensemble random feature models has been investigated in [23-25]. There has been growing interest in connecting the effect of ensemble learning to explicit regularization: see [26] for related experimental evidence under the umbrella of "mini-patch" learning. Towards making these connections precise, some work has been done in the context of the ridge and ridgeless ensembles: [11-13] investigate the relationship between subsampling and regularization for ridge ensembles in the overparameterized regime. We will delve into these works in detail next.
Ensembles and ridge equivalences. In the study of ridge ensembles, LeJeune et al. [11] show that the optimal ridge predictor has the same prediction risk as the ridgeless ensemble under the Gaussian isotropic design, which has been extended in [12] for RMT features (see Assumption 2 for definition). Specifically, these works establish the prediction risk equivalences for a specific pair of  $(\lambda^{*},\phi)$  and  $(0,\psi^{*})$ . The results are then extended to the entire range of  $(\lambda ,\psi)$  using deterministic asymptotic risk formulas [13], assuming a well-specified linear model, RMT features, and the existence of a fixed limiting spectral distribution of the covariance matrix. Our work significantly broadens the scope of results connecting subsampling and ridge regularization. In particular, we allow for arbitrary joint distributions of the data, anisotropic features with only bounded moments, and do not assume the convergence of the spectrum of the population covariance matrix. Furthermore, we expand the applicability of equivalences to encompass both the estimators and various generalized squared risk functionals. See Table 1 for an explicit comparison and Section 3 for additional related comparisons.
Other ridge connections. Beyond the connection connections between implicit regularization induced by subsampling and explicit ridge regularization examined in this paper, ridge regression shares ties with various other forms of regularization. For example, ridge regression has been linked to dropout regularization [27, 28], variable splitting in random forests [2], noisy training [29, 30], random sketched regression [31-33], various forms of data augmentation [34], feature augmentation [9, 35], early stopping in gradient descent and its variants [36-39], among others. These connections highlight the pervasiveness and the significance of understanding various "facets" of ridge regularization. In that direction, our work contributes to expand the equivalences between subsampling and ridge regularization.
Ensemble estimators. Let  $\mathcal{D}_n = \{(x_i, y_i) : i \in [n]\}$  be a dataset containing i.i.d. samples in  $\mathbb{R}^p \times \mathbb{R}$ . Let  $X \in \mathbb{R}^{n \times p}$  and  $\boldsymbol{y} \in \mathbb{R}^n$  be the feature matrix and response vector with  $\boldsymbol{x}_i^\top$  and  $y_i$  in  $i$ -th rows. For an index set  $I \subseteq [n]$  of size  $k$ , let  $\mathcal{D}_I = \{(x_i, y_i) : i \in I\}$  be the associated subsampled dataset. Let  $L_I \in \mathbb{R}^{n \times n}$  be a diagonal matrix such that its  $i$ -th diagonal entry is 1 if  $i \in I$  and 0 otherwise. Note that the feature matrix and response vector associated with  $\mathcal{D}_I$  respectively are  $L_I X$  and  $L_I Y$ . Given a ridge penalty  $\lambda > 0$ , the ridge estimator fitted on  $\mathcal{D}_I$  consisting of  $k$  samples is given by:
$$
\widehat {\boldsymbol {\beta}} _ {k} ^ {\lambda} \left(\mathcal {D} _ {I}\right) = \underset {\boldsymbol {\beta} \in \mathbb {R} ^ {p}} {\operatorname {a r g m i n}} \frac {1}{k} \sum_ {i \in I} \left(y _ {i} - \boldsymbol {x} _ {i} ^ {\top} \boldsymbol {\beta}\right) ^ {2} + \lambda \| \boldsymbol {\beta} \| _ {2} ^ {2} = \left(\frac {1}{k} \boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I} \boldsymbol {X} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \frac {\boldsymbol {X} ^ {\top} \boldsymbol {L} _ {I} \boldsymbol {y}}{k}. \tag {1}
$$
The ridgeless estimator  $\widehat{\beta}_k^0 (\mathcal{D}_I) = (X^\top L_I X / k)^+ X^\top L_I y / k$  is obtained by sending  $\lambda \rightarrow 0^{+}$ , where  $M^{+}$  denotes the Moore-Penrose inverse of matrix  $M$ . To define the ensemble estimator, denote
Table 2: Summary of various generalized risks and their corresponding statistical learning problems. The asymptotic equivalences between subsampling and ridge regularization are analyzed in Theorem 1 and Proposition 2 for generalized risks (3) defined through functionals  $L_{A,b}$  for various  $A$  and  $b$ .
Statistical learning problem	LA,b(β̂ - β0)	A	b	nrow(A)
vector coefficient estimation	β̂ - β0	Ip	0	p
projected coefficient estimation	aT(β̂ - β0)	aT	0	1
training error estimation	Xβ̂ - y	X	-fNL	n
in-sample prediction	X(β̂ - β0)	X	0	n
out-of-sample prediction	x0Tβ̂ - y0	x0T	-ε0	1
the set of all  $k$  distinct elements from  $[n]$  by  $\mathcal{I}_k = \{\{i_1, i_2, \ldots, i_k\} : 1 \leq i_1 < i_2 < \ldots < i_k \leq n\}$ . For  $\lambda \geq 0$ , the  $M$ -ensemble and the full-ensemble estimators are respectively defined as follows:
$$
\widehat {\beta} _ {k, M} ^ {\lambda} \left(\mathcal {D} _ {n}; \{I _ {\ell} \} _ {\ell = 1} ^ {M}\right) = \frac {1}{M} \sum_ {\ell \in [ M ]} \widehat {\beta} _ {k} ^ {\lambda} \left(\mathcal {D} _ {I _ {\ell}}\right), \quad \text {a n d} \quad \widehat {\beta} _ {k, \infty} ^ {\lambda} \left(\mathcal {D} _ {n}\right) = \mathbb {E} \left[ \widehat {\beta} _ {k} ^ {\lambda} \left(\mathcal {D} _ {I}\right) \mid \mathcal {D} _ {n} \right]. \tag {2}
$$
Here  $\{I_{\ell}\}_{\ell = 1}^{M}$  and  $I$  are simple random samples from  $\mathcal{I}_k$ . In (2), the full-ensemble ridge estimator  $\widehat{\beta}_{k,\infty}^{\lambda}(\mathcal{D}_n)$  is the average of predictors fitted on all possible subsampled datasets. It is shown in Lemma A.1 of [13] that  $\widehat{\beta}_{k,\infty}^{\lambda}(\mathcal{D}_n)$  is also asymptotically equivalent to the limit of  $\widehat{\beta}_{k,M}^{\lambda}(\mathcal{D}_n; \{I_{\ell}\}_{\ell = 1}^{M})$  as the ensemble size  $M \to \infty$  (conditioning on the full dataset  $\mathcal{D}_n$ ). This also justifies using  $\infty$  in the subscript to denote the full-ensemble estimator. For brevity, we simply write the estimators as  $\widehat{\beta}_{k,M}^{\lambda}$ ,  $\widehat{\beta}_{k,\infty}^{\lambda}$ , and drop the dependency on  $\mathcal{D}_n$ ,  $\{I_{\ell}\}_{\ell = 1}^{M}$ . We will show certain structural equivalences in terms of linear projections in the family of estimators  $\widehat{\beta}_{k,M}^{\lambda}$  along certain paths in the  $(\lambda, p / k)$ -plane for arbitrary ensemble size  $M \in \mathbb{N} \cup \{\infty\}$  (in Theorem 3). Apart from connecting the estimators, we will also show equivalences of various notions of risks (in Theorem 1), which we describe next.
Generalized risks. Since the ensemble ridge estimators are linear estimators, we evaluate their performance relative to the oracle parameter:  $\beta_0 = \mathbb{E}[\pmb{x}\pmb{x}^\top]^{-1}\mathbb{E}[\pmb{x}\pmb{y}]$ , which is the best (population) linear projection of  $y$  onto  $\pmb{x}$  and minimizes the linear regression error (see, e.g., [40-42]). Note that we can decompose any response  $y$  into:  $y = f_{\mathrm{LI}}(\pmb{x}) + f_{\mathrm{NL}}(\pmb{x})$ , where  $f_{\mathrm{LI}}(\pmb{x}) = \beta_0^\top \pmb{x}$  is the oracle linear predictor, and  $f_{\mathrm{NL}}(\pmb{x}) = y - f_{\mathrm{LI}}(\pmb{x})$  is the nonlinear component that is not explained by  $f_{\mathrm{LI}}(\pmb{x})$ . The best linear projection has the useful property that  $f_{\mathrm{LI}}(\pmb{x})$  is (linearly) uncorrelated with  $f_{\mathrm{NL}}(\pmb{x})$ , although they are generally dependent. It is worth mentioning that this does not imply that  $y$  and  $\pmb{x}$  follow a linear regression model. Indeed, our framework allows any nonlinear dependence structure between them and is model-free for the joint distribution of  $(x,y)$ . Relative to  $\beta_0$ , we measure the performance of an estimator  $\widehat{\beta}$  via the generalized mean squared risk defined compactly as follows:
$$
R (\widehat {\boldsymbol {\beta}}; \mathbf {A}, \mathbf {b}, \boldsymbol {\beta} _ {0}) = \frac {1}{\operatorname {n r o w} (\mathbf {A})} \| L _ {\mathbf {A}, \mathbf {b}} (\widehat {\boldsymbol {\beta}} - \boldsymbol {\beta} _ {0}) \| _ {2} ^ {2}, \tag {3}
$$
where  $L_{A,b}(\beta) = A\beta + b$  is a linear functional. Note that  $A$  and  $b$  can potentially depend on the data, and their dimensions can vary depending on the statistical learning problem at hand. This framework includes various important statistical learning problems, as summarized in Table 2. Observe that the framework also includes various notions of prediction risks. One can use the test error formulation in Table 2 to obtain the prediction risk at any test point  $(x_0, y_0)$ , where  $y_0 = x_0^\top \beta_0 + \varepsilon_0$ , and  $\varepsilon_0$  may depend on  $x_0$  and have non-zero mean. This also permits the test point to be drawn from a distribution that differs from the training distribution. Specifically, when  $\varepsilon_0 = f_{\mathrm{NL}}(x_0)$  but  $x_0$  has a distribution different from  $x$ , we obtain the prediction error under covariate shift. Similarly, when  $\varepsilon_0 \neq f_{\mathrm{NL}}(x_0)$  but  $x_0$  and  $x$  have the same distribution, we get the prediction error under label shift.
Asymptotic equivalence. For our theoretical analysis, we consider the proportional asymptotics regime, where the ratio of the feature size  $p$  to the sample size  $n$  tends to a fixed limiting data aspect ratio  $\phi \in (0,\infty)$ . To concisely present our results, we will use the elegant framework of asymptotic equivalence [12, 43, 44]. Let  $A_{p}$  and  $B_{p}$  be sequences of (additively) conformable matrices of arbitrary dimensions (including vectors and scalars). We say that  $A_{p}$  and  $B_{p}$  are asymptotically equivalent, denoted as  $A_{p} \simeq B_{p}$ , if  $\lim_{p\to \infty}|\operatorname{tr}[C_p(A_p - B_p)]| = 0$  almost surely for any sequence of random matrices  $C_p$  with bounded trace norm that are (multiplicatively) conformable and independent of  $A_{p}$  and  $B_{p}$ . Note that for sequences of scalar random variables, the definition simply reduces to the typical almost sure convergence of sequences of random variables involved.
Figure 1: Heat map of the various generalized risks (estimation risk, training error, prediction risk, out-of-distribution (OOD) prediction risk) of full-ensemble ridge estimators (approximated with  $M = 100$ ), for varying ridge penalties  $\lambda$  and subsample aspect ratios  $\psi = p / k$  on the log-log scale. The data model is described in Appendix F.2 with  $p = 500$ ,  $n = 5000$ , and  $\phi = p / n = 0.1$ . Observe that along the same path the values match well for all risks, in line with Theorem 1 and Proposition 2.
Subsample aspect ratio  $\psi$
3 Generalized risk equivalences of ensemble estimators
We begin by examining the risk equivalences among different ensemble ridge estimators for various generalized risks defined as in (3). To prepare for our upcoming results, we impose two structural and moment assumptions on the distributions of the response variable and the feature vector.
Assumption 1 (Response variable distribution). Each response variable  $y_{i}$  for  $i\in [n]$  has mean 0 and satisfies  $\mathbb{E}[|y_i|^{4 + \mu}]\leq M_\mu < \infty$  for some  $\mu >0$  and a constant  $M_{\mu}$ .
Assumption 2 (Feature vector distribution). Each feature vector  $\pmb{x}_i$  for  $i \in [n]$  can be decomposed as  $\pmb{x}_i = \pmb{\Sigma}^{1/2}\pmb{z}_i$ , where  $\pmb{z}_i \in \mathbb{R}^p$  contains i.i.d. entries  $z_{ij}$  for  $j \in [p]$  with mean 0, variance 1, and satisfy  $\mathbb{E}[|z_{ij}|^{4 + \nu}] \leq M_\nu < \infty$  for some  $\nu > 0$  and a constant  $M_\nu$ , and  $\pmb{\Sigma} \in \mathbb{R}^{p \times p}$  is a deterministic and symmetric matrix with eigenvalues uniformly bounded between constants  $r_{\min} > 0$  and  $r_{\max} < \infty$ .
Note that we do not impose any specific model assumptions on the response variable  $y$  in relation to the feature vector  $\pmb{x}$ . We only assume bounded moments as stated in Assumption 1, making all of our subsequent results model-free. The zero-mean assumption for  $y$  is for simplicity since, in practice, centering can always be done by subtracting the sample mean. The bounded moment condition can also be satisfied if one imposes a stronger distributional assumption (e.g., sub-Gaussianity). Assumption 2 on the feature vector is common in the study of random matrix theory [45, 46] and the analysis of ridge and ridgeless regression [10, 47-49], which we refer to as RMT features for brevity.
Given a limiting data aspect ratio  $\phi \in (0,\infty)$  and a limiting subsample aspect ratio  $\overline{\psi} \in [\phi ,\infty ]$ , our statement of equivalences between different ensemble estimators is defined through certain paths characterized by two endpoints  $(0,\overline{\psi})$  and  $(\bar{\lambda},\phi)$ . These endpoints correspond to the subsample ridgeless ensemble and the (non-ensemble) ridge predictor, respectively, with  $\bar{\lambda}$  being the ridge penalty to be defined next. For that, let  $H_{p}$  be the empirical spectral distribution of  $\boldsymbol{\Sigma}$ :  $H_{p}(r) = p^{-1}\sum_{i = 1}^{p}\mathbb{1}_{\{r_{i}\leq r\}}$ , where  $r_i$ 's are the eigenvalues of  $\boldsymbol{\Sigma}$ . Consider the following system of equations in  $\bar{\lambda}$  and  $v$ :
$$
\frac {1}{v} = \bar {\lambda} + \phi \int \frac {r}{1 + v r} d H _ {p} (r), \quad \text {a n d} \quad \frac {1}{v} = \bar {\psi} \int \frac {r}{1 + v r} d H _ {p} (r). \tag {4}
$$
Existence and uniqueness of the solution  $(\bar{\lambda}, v) \in [0, \infty]^2$  to the above equations are guaranteed by Corollary E.4. Now, define a path  $\mathcal{P}(\bar{\lambda}; \phi, \overline{\psi})$  that passes through the endpoints  $(0, \overline{\psi})$  and  $(\bar{\lambda}, \phi)$ :
$$
\mathcal {P} (\bar {\lambda}; \phi , \bar {\psi}) = \left\{(1 - \theta) \cdot (\bar {\lambda}, \phi) + \theta \cdot (0, \bar {\psi}) \mid \theta \in [ 0, 1 ] \right\}. \tag {5}
$$
We are ready to state our results. We consider two cases for the generalized risk (3) depending on the relationships between  $(A, b)$  and  $X$ . In the first case, when both  $A$  and  $b$  are independent of the data, Theorem 1 shows that the generalized risks are equivalent along the path (5).
Theorem 1 (Risk equivalences when  $(\mathbf{A},\mathbf{b})\perp (\mathbf{X},\mathbf{y}))$ . Suppose Assumptions 1-2 hold. Let  $(\mathbf{A},\mathbf{b})$  be independent of  $(\mathbf{X},\mathbf{y})$  such that  $\mathrm{nrow}(\mathbf{A})^{-1 / 2}\| \mathbf{A}\|_{\mathrm{op}}$  and  $\| \boldsymbol {b}\| _2$  are almost surely bounded. Let  $n,p\to \infty$  such that  $p / n\rightarrow \phi \in (0,\infty)$ . For any  $\overline{\psi}\in [\phi , + \infty ]$ , let  $\bar{\lambda}$  be as defined in (4). Then, for any pair of  $(\lambda_1,\psi_1)$  and  $(\lambda_{2},\psi_{2})$  on the path  $\mathcal{P}(\bar{\lambda};\phi ,\bar{\psi})$  as defined in (5), the generalized risk functionals (3) of the full-ensemble estimator are asymptotically equivalent: