SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Benjamin S. Ruben<sup>1</sup>, Cengiz Pehlevan<sup>2,3,4</sup>
<sup>1</sup>Biophysics Graduate Program
$^{2}$ Center for Brain Science,  $^{3}$ John A. Paulson School of Engineering and Applied Sciences,  $^{4}$ Kempner
Institute for the Study of Natural and Artificial Intelligence,
benruben@g.harvard.edu, cpehlevan@seas.harvard.edu
Feature bagging is a well-established ensembling method which aims to reduce prediction variance by combining predictions of many estimators trained on subsets or projections of features. Here, we develop a theory of feature-bagging in noisy least-squares ridge ensembles and simplify the resulting learning curves in the special case of equicorrelated data. Using analytical learning curves, we demonstrate that subsampling shifts the double-descent peak of a linear predictor. This leads us to introduce heterogeneous feature ensembling, with estimators built on varying numbers of feature dimensions, as a computationally efficient method to mitigate double-descent. Then, we compare the performance of a feature-subsampling ensemble to a single linear predictor, describing a trade-off between noise amplification due to subsampling and noise reduction due to ensembling. Our qualitative insights carry over to linear classifiers applied to image classification tasks with realistic datasets constructed using a state-of-the-art deep learning feature map.
Ensembling methods are ubiquitous in machine learning practice [1]. A class of ensembling methods (known as attribute bagging [2] or the random subspace method [3]) is based on feature subsampling [2-6], where predictors are independently trained on subsets of the features, and their predictions are combined to achieve a stronger prediction. The random forest method is a popular example [3, 7].
While commonly used in practice, a theoretical understanding of ensembling via feature subsampling is not well developed. Here, we provide an analysis of this technique in the linear ridge regression setting. Using methods from statistical physics [8-12], we obtain analytical expressions for typical-case generalization error in linear ridge ensembles (proposition 1), and simplify these expressions in the special case of equicorrelated data with isotropic feature noise (proposition 2). The result provides a powerful tool to quickly probe the generalization error of ensembled regression under a rich set of conditions. In section 3, we study the behavior of a single feature-subsampling regression model. We observe that subsampling shifts the location of a predictor's sample-wise double-descent peak [13-15]. This motivates section 4, where we study ensembles built on predictors which are heterogeneous in the number of features they access, as a method to mitigate double-descent. We demonstrate this method's efficacy in a realistic image classification task. In section 5 we apply our theory to the trade-off between ensembling and subsampling in resource-constrained settings. We
characterize how a variety of factors influence the optimal ensembling strategy, finding a particular significance to the level of noise in the predictions made by ensemble members.
- Using the replica trick from statistical physics [8, 11], we derive the generalization error of ensembled least-squares ridge regression in a general setting, and simplify the resulting expressions in the tractable special case where features are equicorrelated.
- We demonstrate benefits of heterogeneous ensembling as a robust and computationally efficient regularizer for mitigating double-descent with analytical theory and in a realistic image classification task.
- We describe the ensembling-subsampling trade-off in resource-constrained settings, and characterize the effect of label noise, feature noise, readout noise, regularization, sample size and task structure on the optimal ensembling strategy.
Related works: A substantial body of work has elucidated the behavior of linear predictors for a variety of feature maps [14, 16-30]. Several recent works have extended this research to characterize the behavior of ensembled regression using solvable models [24, 31-33]. Additional recent works study the performance of ridge ensembles with example-wise subsampling [34, 35] and simultaneous subsampling of features and examples [32], finding that subsampling behaves as an implicit regularization. Methods from statistical physics have long been used for machine learning theory [10-12, 26, 27, 30, 36, 37]. Relevant work in this domain include [38] which studied ensembling by data-subsampling in linear regression.
2 Learning Curves for Ensembled Ridge Regression
We consider noisy ensembled ridge regression in the setting where ensemble members are trained independently on masked versions of the available features. We derive our main analytical formula for generalization error of ensembled linear regression, as well as analytical expressions for generalization error in the special case of equicorrelated features with isotropic noise.
Consider a training set  $\mathcal{D} = \{\bar{\psi}^{\mu},y^{\mu}\}_{\mu = 1}^{P}$  of size  $P$ . The training examples  $\bar{\psi}^{\mu}\in \mathbb{R}^{M}$  are drawn from a Gaussian distribution with Gaussian feature noise:  $\bar{\psi}^{\mu} = \psi^{\mu} + \sigma^{\mu}$ , where  $\psi^{\mu}\sim \mathcal{N}(0,\Sigma_{s})$  and  $\pmb{\sigma}^{\mu}\sim \mathcal{N}(0,\Sigma_0)$ . Data and noise are drawn i.i.d. so that  $\mathbb{E}\left[\pmb{\psi}^{\mu}\pmb{\psi}^{\nu \top}\right] = \delta_{\mu \nu}\pmb{\Sigma}_s$  and  $\mathbb{E}\left[\pmb{\sigma}^{\mu}\pmb{\sigma}^{\nu \top}\right] = \delta_{\mu \nu}\pmb{\Sigma}_0$ . Labels are generated from a noisy teacher function  $y^\mu = \frac{1}{\sqrt{M}}\pmb{w}^{*\top}\pmb{\psi}^\mu +\epsilon^\mu$  where  $\epsilon^\mu \sim \mathcal{N}(0,\zeta^2)$ . Label noises are drawn i.i.d. so that  $\mathbb{E}[\epsilon^\mu \epsilon^\nu ] = \delta_{\mu \nu}\zeta^2$ .
We seek to analyze the quality of predictions which are averaged over an ensemble of ridge regression models, each with access to a subset of the features. We consider  $k$  linear predictors with weights  $\hat{w}_r \in \mathbb{R}^{N_r}$ ,  $r = 1, \dots, k$ . Critically, we allow  $N_r \neq N_{r'}$  for  $r \neq r'$ , which allows us to introduce structural heterogeneity into the ensemble of predictors. A forward pass of the model is given as:
$$
f (\psi) = \frac {1}{k} \sum_ {r = 1} ^ {k} f _ {r} (\psi), \quad f _ {r} (\psi) = \frac {1}{\sqrt {N _ {r}}} \hat {\boldsymbol {w}} _ {r} ^ {\top} \boldsymbol {A} _ {r} (\psi + \sigma) + \xi_ {r}. \tag {1}
$$
The model's prediction  $f(\psi)$  is an average over  $k$  linear predictors. The "measurement matrices"  $A_r \in \mathbb{R}^{N_r \times M}$  act as linear masks restricting the information about the features available to each member of the ensemble. Subsampling may be implemented by choosing the rows of each  $A_r$  to coincide with the rows of the identity matrix – the row indices corresponding to indices of the sampled features. The feature noise  $\sigma \sim \mathcal{N}(0, \Sigma_0)$  and the readout noises  $\xi_r \sim \mathcal{N}(0, \eta_r^2)$ , are drawn independently at the execution of each forward pass of the model. Note that while the feature noise is shared across the ensemble, readout noise is drawn independently for each readout:  $\mathbb{E}[\xi_r \xi_{r'}] = \delta_{rr'} \eta_r^2$ .
The weight vectors are trained separately in order to minimize an ordinary least-squares loss function with ridge regularization:
$$
\hat {\boldsymbol {w}} _ {r} = \underset {\boldsymbol {w} _ {r} \in \mathbb {R} ^ {N _ {r}}} {\arg \min } \left[ \sum_ {\mu = 1} ^ {P} \left(\frac {1}{\sqrt {N _ {r}}} \boldsymbol {w} _ {r} ^ {\top} \boldsymbol {A} _ {r} \bar {\psi} ^ {\mu} + \xi_ {r} ^ {\mu} - y ^ {\mu}\right) ^ {2} + \lambda_ {r} | \boldsymbol {w} _ {r} ^ {2} | \right] \tag {2}
$$
Here  $\{\xi_r^\mu\}$  represents the readout noise which is present during training, and independently drawn:  $\xi_r^\mu \sim \mathcal{N}(0,\eta_r^2)$ ,  $\mathbb{E}[\xi_r^\mu \xi_r^\nu ] = \eta_r^2\delta_{\mu \nu}$ . As a measure of model performance, we consider the generalization error, given by the mean-squared-error (MSE) on ensemble-averaged prediction:
$$
E _ {g} (\mathcal {D}) = \mathbb {E} _ {\psi , \sigma , \left\{\xi_ {r} \right\}} \left[ \left(f (\psi) - \frac {1}{\sqrt {M}} \boldsymbol {w} ^ {* \top} \psi\right) ^ {2} \right] \tag {3}
$$
Here, the expectation is over the data distribution and noise:  $\psi \sim \mathcal{N}(0,\Sigma_s)$ ,  $\sigma \sim \mathcal{N}(0,\Sigma_0)$ ,  $\xi_r \sim \mathcal{N}(0,\eta_r^2)$ . The generalization error depends on the particular realization of the dataset  $\mathcal{D}$  through the learned weights  $\{\hat{w}^*\}$ . We may decompose the generalization error as follows:
$$
\begin{array}{l} E _ {g} (\mathcal {D}) = \frac {1}{k ^ {2}} \sum_ {r, r ^ {\prime} = 1} ^ {k} E _ {r r ^ {\prime}} (\mathcal {D}) (4) \\ E _ {r r ^ {\prime}} (\mathcal {D}) \equiv \frac {1}{M} \left[ \left(\frac {1}{\sqrt {\nu_ {r r}}} \boldsymbol {A} _ {r} ^ {\top} \hat {\boldsymbol {w}} _ {r} - \boldsymbol {w} ^ {*}\right) ^ {\top} \boldsymbol {\Sigma} _ {s} \left(\frac {1}{\sqrt {\nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \hat {\boldsymbol {w}} _ {r ^ {\prime}} - \boldsymbol {w} ^ {*}\right) \right. (5) \\ \left. + \frac {1}{\sqrt {\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}} \hat {\boldsymbol {w}} _ {r} ^ {\top} \boldsymbol {A} _ {r} \Sigma_ {0} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \hat {\boldsymbol {w}} _ {r ^ {\prime}} + M \delta_ {r r ^ {\prime}} \eta_ {r} ^ {2} \right] \\ \end{array}
$$
Computing the generalization error of the model is then a matter of calculating  $E_{rr'}$  in the cases where  $r = r'$  and  $r \neq r'$ . In the asymptotic limit we consider, we expect that the generalization error concentrates over randomly drawn datasets  $\mathcal{D}$ .
2.2 Main Result
We calculate the generalization error using the replica trick from statistical physics, and present the calculation in Appendix F. The result of our calculation is stated in proposition 1.
Proposition 1. Consider the ensembled ridge regression problem described in Section 2.1. Consider the asymptotic limit where  $M, P, \{N_r\} \to \infty$  while the ratios  $\alpha = \frac{P}{M}$  and  $\nu_{rr} = \frac{N_r}{M}$ ,  $r = 1, \ldots, k$  remain fixed. Define the following quantities:
$$
\tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \equiv \frac {1}{\sqrt {\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {A} _ {r} \left[ \boldsymbol {\Sigma} _ {s} + \boldsymbol {\Sigma} _ {0} \right] \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \tag {6}
$$
$$
\boldsymbol {G} _ {r} \equiv \boldsymbol {I} _ {N _ {r}} + \hat {q} _ {r} \hat {\boldsymbol {\Sigma}} _ {r r} \tag {7}
$$
$$
\gamma_ {r r ^ {\prime}} \equiv \frac {\alpha}{M \left(\lambda_ {r} + q _ {r}\right) \left(\lambda_ {r ^ {\prime}} + q _ {r ^ {\prime}}\right)} \operatorname {t r} \left[ \boldsymbol {G} _ {r} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \boldsymbol {G} _ {r ^ {\prime}} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r ^ {\prime} r} \right] \tag {8}
$$
Then the terms of the average generalization error (eq. 5) may be written as:
$$
\begin{array}{l} \langle E _ {r r ^ {\prime}} (\mathcal {D}) \rangle_ {\mathcal {D}} = \frac {\gamma_ {r r ^ {\prime}} \zeta^ {2} + \delta_ {r r ^ {\prime}} \eta_ {r} ^ {2}}{1 - \gamma_ {r r ^ {\prime}}} + \frac {1}{1 - \gamma_ {r r ^ {\prime}}} \left(\frac {1}{M} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*}\right) \\ - \frac {1}{M \left(1 - \gamma_ {r r ^ {\prime}}\right)} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \left[ \frac {1}{\nu_ {r r}} \hat {q} _ {r} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r} ^ {- 1} \boldsymbol {A} _ {r} + \frac {1}{\nu_ {r ^ {\prime} r ^ {\prime}}} \hat {q} _ {r ^ {\prime}} \boldsymbol {A} _ {r ^ {\prime}} ^ {\top} \boldsymbol {G} _ {r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}} \right] \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*} \tag {9} \\ + \frac {\hat {q} _ {r} \hat {q} _ {r ^ {\prime}}}{M (1 - \gamma_ {r r ^ {\prime}})} \frac {1}{\sqrt {\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}}} \boldsymbol {w} ^ {* \top} \boldsymbol {\Sigma} _ {s} \boldsymbol {A} _ {r} ^ {\top} \boldsymbol {G} _ {r} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r r ^ {\prime}} \boldsymbol {G} _ {r ^ {\prime}} ^ {- 1} \boldsymbol {A} _ {r ^ {\prime}} \boldsymbol {\Sigma} _ {s} \boldsymbol {w} ^ {*} \\ \end{array}
$$
where the pairs of order parameters  $\{q_r,\hat{q}_r\}$  for  $r = 1,\dots ,K$  satisfy the following self-consistent saddle-point equations
$$
\hat {q} _ {r} = \frac {\alpha}{\lambda_ {r} + q _ {r}}, \quad q _ {r} = \frac {1}{M} \operatorname {t r} \left[ \boldsymbol {G} _ {r} ^ {- 1} \tilde {\boldsymbol {\Sigma}} _ {r r} \right]. \tag {10}
$$
Figure 1: Comparison of numerical and theoretical learning curves for ensembled linear regression. Circles represent numerical results averaged over 100 trials; lines indicate theoretical predictions. Error bars represent the standard error of the mean but are often smaller than the markers. (a) Testing of proposition 1 with  $M = 2000$ ,  $[\Sigma_s]_{ij} = .8^{|i - j|}$ ,  $[\Sigma_0]_{ij} = \frac{1}{10}(0.3)^{|i - j|}$ ,  $\zeta = 0.1$ , and all  $\eta_r = 0.2$  and  $\lambda_r = \lambda$  (see legend).  $k = 3$  linear predictors access fixed, randomly selected (with replacement) subsets of the features with fractional sizes  $\nu_{rr} = 0.2, 0.4, 0.6$ . Fixed ground-truth weights  $w^*$  are drawn from an isotropic Gaussian distribution. (b) Testing of proposition 2 with  $M = 5000$ ,  $s = 1$ ,  $c = 0.6$ ,  $\omega^2 = 0.1$ ,  $\zeta = 0.1$ , all  $\eta_r = 0.1$ , and all  $\lambda_r = \lambda$  (see legend). Ground truth weights sampled as in eq. 11 with  $\rho = 0.3$ . Feature subsets accessed by each readout are mutually exclusive (inset) with fractional sizes  $\nu_{rr} = 0.1, 0.3, 0.5$ .
Proof. We calculate the terms in the generalization error using the replica trick, a standard but non-rigorous method from the statistical physics of disordered systems. The full derivation may be found in the Appendix F. When the matrices  $\mathbf{A}_r(\boldsymbol{\Sigma}_s + \boldsymbol{\Sigma}_0) \mathbf{A}_r^\top$ ,  $r = 1, \dots, k$  have bounded spectra, this result may be obtained by extending the results of [31] to include readout noise, as shown in Appendix G.
We make the following remarks:
Remark 1. Implicit in this theorem is the assumption that the relevant matrix contractions and traces which appear in the generalization error (eq. 9) and the surrounding definitions tend to a well-defined limit which remains  $\mathcal{O}(1)$  as  $M\to \infty$
Remark 2. This result applies for any (well-behaved) linear masks  $\{A_r\}$ . We will focus on the case where each  $A_r$  implements subsampling of an extensive fraction  $\nu_{rr}$  of the features.
Remark 3. When  $k = 1$ , our result reduces to the generalization error of a single ridge regression model, as studied in refs. [36, 39].
Remark 4. We include "readout noise" which independently corrupts the predictions of each ensemble member. This models sources of variation between ensemble members not otherwise accounted for. For example, ensembles of deep networks will vary due to random initialization of parameters [24, 31, 36]. Readout noise is more directly present in physical neural networks, such as an analog neural networks [40] or biological neural circuits[41] due to their inherent stochasticity.
In Figure 1a, we confirm the result of the general calculation by comparing with numerical experiments using a synthetic dataset with  $M = 2000$  highly structured features (see caption for details).  $k = 3$  readouts see random, fixed subsets of features. Theory curves are calculated by solving the fixed-point equations 10 numerically for the chosen  $\Sigma_{s}$ ,  $\Sigma_{0}$  and  $\{A_r\}_{r=1}^k$  then evaluating eq. 9.
2.3 Equicorrelated Data
Our general result allows the freedom to tune many important parameters of the learning problem: the correlation structure of the dataset, the number of ensemble members, the scales of noise, etc. However, the derived expressions are rather opaque. In order to better understand the phenomena captured by these expressions, we examine the following special case:
Proposition 2. In the setting of section 2.1 and proposition 1, consider the following special case:
$$
\boldsymbol {w} ^ {*} = \sqrt {1 - \rho^ {2}} \mathbb {P} _ {\perp} \boldsymbol {w} _ {0} ^ {*} + \rho \mathbf {1} _ {M} \tag {11}
$$
$$
\boldsymbol {w} _ {0} ^ {*} \sim \mathcal {N} (0, \boldsymbol {I} _ {M}) \tag {12}
$$
$$
\boldsymbol {\Sigma} _ {s} = s \left[ (1 - c) \boldsymbol {I} _ {M} + c \mathbf {1} _ {M} \mathbf {1} _ {M} ^ {\top} \right] \tag {13}
$$
$$
\boldsymbol {\Sigma} _ {0} = \omega^ {2} \boldsymbol {I} _ {M} \tag {14}
$$
with  $c \in [0,1], \rho \in [-1,1]$ . Label and readout noises  $\zeta, \eta_r \geq 0$  are permitted. Here  $\mathbb{P}_{\perp} = \mathbf{I}_M - \frac{1}{M}\mathbf{1}_M\mathbf{1}_M^\top$  is a projection matrix which removes the component of  $\mathbf{w}_0^*$  which is parallel to  $\mathbf{1}_M$ . The matrices  $\{\mathbf{A}_r\}_{r=1}^k$  have rows consisting of distinct one-hot vectors so that each of the  $k$  readouts has access to a subset of  $N_r = \nu_{rr'}M$  features. For  $r \neq r'$ , denote by  $n_{rr'}$  the number of neurons sampled by both  $\mathbf{A}_r$  and  $\mathbf{A}_{r'}$  and let  $\nu_{rr'} \equiv n_{rr'} / M$  remain fixed as  $M \to \infty$ .
Define the following quantities:
$$
a \equiv s (1 - c) + \omega^ {2} \quad S _ {r} \equiv \frac {\hat {q} _ {r}}{\nu_ {r r} + a \hat {q} _ {r}}, \quad \gamma_ {r r ^ {\prime}} \equiv \frac {a ^ {2} \nu_ {r r ^ {\prime}} S _ {r} S _ {r ^ {\prime}}}{\alpha} \tag {15}
$$
The terms of the decomposed generalization error may then be written:
$$
\left\langle E _ {r r ^ {\prime}} \right\rangle_ {\mathcal {D}, \boldsymbol {w} _ {0} ^ {*}} = \frac {1}{1 - \gamma_ {r r ^ {\prime}}} \left((1 - \rho^ {2}) I _ {r r ^ {\prime}} ^ {0} + \rho^ {2} I _ {r r ^ {\prime}} ^ {1}\right) + \frac {\gamma_ {r r ^ {\prime}} \zeta^ {2} + \delta_ {r r ^ {\prime}} \eta_ {r} ^ {2}}{1 - \gamma_ {r r ^ {\prime}}} \tag {16}
$$
$$
I _ {r r ^ {\prime}} ^ {0} \equiv s (1 - c) (1 - s (1 - c) \nu_ {r r} S _ {r} - s (1 - c) \nu_ {r ^ {\prime} r ^ {\prime}} S _ {r ^ {\prime}} + a s (1 - c) \nu_ {r r ^ {\prime}} S _ {r} S _ {r ^ {\prime}}) \tag {17}
$$
$$
I _ {r r ^ {\prime}} ^ {1} \equiv \left\{ \begin{array}{l l} \frac {s (1 - c) \left(\nu_ {r r ^ {\prime}} - \nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}\right) + \omega^ {2} \nu_ {r r ^ {\prime}}}{\nu_ {r r} \nu_ {r ^ {\prime} r ^ {\prime}}} & i f 0 <   c \leq 1 \\ I _ {r r ^ {\prime}} ^ {0} & i f c = 0 \end{array} \right. \tag {18}
$$
and where  $\{q_r,\hat{q}_r\}$  may be obtained analytically as the solution (with  $q_{r} > 0$  ) to:
$$
q _ {r} = \frac {a \nu_ {r r}}{\nu_ {r r} + a \hat {q} _ {r}} \quad , \quad \hat {q} _ {r} = \frac {\alpha}{\lambda_ {r} + q _ {r}} \tag {19}
$$
In the "ridgeless" limit where all  $\lambda_r \to 0$ , we may make the following simplifications:
$$
S _ {r} \rightarrow \frac {2 \alpha}{a (\alpha + \nu_ {r r} + | \alpha - \nu_ {r r} |)}, \quad \gamma_ {r r ^ {\prime}} \quad \rightarrow \frac {4 \alpha \nu_ {r r ^ {\prime}}}{(\alpha + \nu_ {r r} + | \alpha - \nu_ {r r} |) (\alpha + \nu_ {r ^ {\prime} r ^ {\prime}} + | \alpha - \nu_ {r ^ {\prime} r ^ {\prime}} |)} \tag {20}
$$
Proof. Simplifying the fixed-point equations and generalization error formulas in this special case is an exercise in linear algebra. The main tools used are the Sherman-Morrison formula [42] and the fact that the data distribution is isotropic in the features so that the form of  $\tilde{\Sigma}_{rr'}$  and  $\tilde{\Sigma}_{rr'}'$  depend only on the subsampling and overlap fractions  $\nu_{rr}, \nu_{r'r'}, \nu_{rr'}$ . To aid in computing the necessary matrix contractions we developed a custom Mathematica package which handles block matrices of symbolic dimension, with blocks containing matrices of the form  $M = c_1 I + c_2 11^\top$ . This package and the Mathematica notebook used to derive these results are available online (see Appendix B)
In this tractable special case,  $c \in [0,1]$  is a parameter which tunes the strength of correlations between features of the data. When  $c = 0$ , the features are independent, and when  $c = 1$  the features are always equivalent.  $s$  sets the overall scale of the features and the "Data-Task alignment"  $\rho$  tunes the alignment of the ground truth weights with the special direction in the covariance matrix (analogous to "task-model" alignment [14, 27]). A table of parameters is provided in Appendix A. In Figure 1b, we test these results by comparing the theoretical expressions for generalization error with the results of numerical experiments, finding perfect agreement.
With an analytical formula for the generalization error, we can compute the optimal regularization parameters  $\lambda_r$  which minimize the generalization error. These may, in general, depend on both  $r$  and the sample size  $\alpha$ . Rather than minimizing the error of the ensemble, we may minimize the generalization error of predictions made by the ensemble members independently. We find that this "locally optimal" regularization, denoted  $\lambda^*$ , is independent of  $\alpha$ , generalizing results from [14, 43] to correlated data distributions (see Appendix H.3).
Figure 2: Subsampling alters the location of the double-descent peak of a linear predictor. (a) Illustrations of subsampled linear predictors with varying subsampling fraction  $\nu$ . (b) Comparison between experiment and theory for subsampling linear regression on equicorrelated datasets. We choose task parameters as in proposition 2 with  $c = \omega = \zeta = \eta = 0$ ,  $s = 1$ , and (i)  $\lambda = 0$ , (ii)  $\lambda = 10^{-4}$ , (iii)  $\lambda = 10^{-2}$ . All learning curves are for a single linear predictor  $k = 1$  with subsampling fraction  $\nu$  shown in legend. Circles show results of numerical experiment. Lines are analytical prediction.
3 Subsampling shifts the double-descent peak of a linear predictor
Consider a single linear regressor  $(k = 1)$  which connects to a subset of  $N = \nu M$  features in the equicorrelated data setting of proposition 2. Also setting  $c = 0$ ,  $s = 1$ , and  $\eta_r = \omega = 0$  and taking the limit  $\lambda \to 0$  the generalization error reads:
$$
\langle E _ {g} \rangle_ {\mathcal {D}, \boldsymbol {w} ^ {*}} = \left\{ \begin{array}{l l} \frac {\nu}{\nu - \alpha} \left[ (1 - \nu) + \frac {1}{\nu} (\alpha - \nu) ^ {2} \right] + \frac {\alpha}{\nu - \alpha} \zeta^ {2}, & \text {i f} \alpha <   \nu \\ \frac {\alpha}{\alpha - \nu} [ 1 - \nu ] + \frac {\nu}{\alpha - \nu} \zeta^ {2}, & \text {i f} \alpha > \nu \end{array} \right\} \tag {21}
$$
We thus see that double descent can arise from two possible sources of variance: explicit label noise (if  $\zeta > 0$ ) or implicit label noise induced by feature subsampling ( $\nu < 1$ ). As  $E_g \sim (\alpha - \nu)^{-1}$ , generalization error diverges when sample size is equal to the number of sampled features. Intuitively, this occurs because subsampling changes the number of parameters of the regression model, and thus its interpolation threshold. To demonstrate this, we plot the learning curves for subsampled linear regression on equicorrelated data in Figure 2. At small finite ridge the test error no longer diverges when  $\alpha = \nu$ , but still displays a distinctive peak.
4 Heterogeneous connectivity mitigates double-descent