SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Double-descent – over-fitting to noise in the training set near a model's interpolation threshold – poses a serious risk in practical machine-learning applications [22]. Cross-validating the regularization strength against the training set is the canonical approach to avoiding double-descent [17, 43], but in practice requires a computationally expensive parameter sweep and prior knowledge of the task. In situations where computational resources are limited or hyperparameters are fixed prior to specification of the task, it is natural to seek an alternative solution. Considering again the plots in Figure 2(b), we observe that at any value of  $\alpha$ , the double-descent peak can be avoided with an acceptable choice of the subsampling fraction  $\nu$ . This suggests another strategy to mitigate double descent: heterogeneous ensembling. Ensembling over predictors with a heterogeneous distribution of interpolation thresholds, we may expect that when one predictor fails due to over-fitting, the other members of the ensemble compensate with accurate predictions.
In Figure 3, we show that heterogeneous ensembling can guard against double-descent. We define two ensembling strategies: in homogeneous ensembling, each of  $k$  readouts connects a fraction  $\nu_{rr} = 1 / k$  features. In heterogeneous ensembling, the number of features connected by each of the  $k$  readouts are drawn from a Gamma distribution  $\Gamma_{k,\sigma}(\nu)$  with mean  $1 / k$  and standard deviation  $\sigma$  (see Fig. 3b) then re-scaled to sum to 1 (see Appendix C for details). All feature subsets are mutually exclusive  $(\nu_{rr'} = 0$  for  $r \neq r'$ ). Homogeneous and heterogeneous ensembling are illustrated for  $k = 10$  in Figs. 3 a.i and 3 a.ii respectively. We test this hypothesis using eq. 16 in 3c. At small regularization
$(\lambda = .001)$ , we find that heterogeneity of the distribution of subsampling fractions  $(\sigma > 0)$  lowers the double-descent peak of an ensemble of linear predictors, while at larger regularization  $(\lambda = 0.1)$ , there is little difference between homogeneous and heterogeneous learning curves. The asymptotic  $(\alpha \rightarrow \infty)$  error is unaffected by the presence of heterogeneity in the degrees of connectivity, which can be seen as the coincidence of the triangular markers in Fig. 3c, as well as from the  $\alpha \rightarrow \infty$  limit of eq. 16 (see Appendix H.5). Fig. 3c also shows the learning curve of a single linear predictor with no feature subsampling and optimal regularization. We see that the feature-subsampling ensemble appreciably outperforms the fully-connected model when  $c = 0.8$  and  $\eta = 0.5$ , suggesting the important roles of data correlations and readout noise in determining the optimal readout strategy. These roles are further explored in section 5 and fig 4.
We also test the effect of heterogeneous ensembling in the a realistic classification task. Specifically, we train ensembles of linear classifiers to predict the labels of imagenet [44] images corresponding to 10 different dog breeds (the "Imagewoof" task [45]) from their top-hidden-layer representations in a pre-trained ResNext deep network [46] (see Appendix E for details). We characterize the statistics of the resulting  $M = 2048$ -dimensional feature set in Fig. S1. This "ResNext-Imagewoof" classification task has multiple features which make it amenable to learning with a feature-subsampling ensemble. First, the ResNext features have a high degree of redundancy [47], allowing classification to be performed accurately using only a fraction of the available features (see Fig. 3d and S1c). Second, when classifications of multiple predictors are combined by a majority vote, there is a natural upper bound on the influence of a single erring ensemble member (unlike in regression where predictions can diverge). Calculating learning curves for the imagewoof classification task using homogeneous ensembles, we see sharp double-descent peaks in an ensemble of size  $k$  when  $P = M / k$  (Fig. 3e.i). Using a heterogeneous ensemble mitigates this catastrophic over-fitting, leading to monotonically decreasing error without regularization (Fig. 3e.ii). A single linear predictor with a tuned regularization of  $\lambda = 0.1$  performs only marginally better than the heterogeneous feature-subsampling ensemble with  $k = 16$  or  $k = 32$ . This suggests heterogeneous ensembling can be an effective alternative to regularization in real-world classification tasks using pre-trained deep learning feature maps.
Note that training a feature-subsampling ensemble also benefits from improved computational complexity. Training an estimator of dimension  $N_{r}$  involves, in the worst case, inverting an  $N_{r} \times N_{r}$  matrix, which requires  $\mathcal{O}(N_r^3)$  operations. Setting  $N_{r} = M / k$ , we see that the number of operations required to train an ensemble of  $k$  predictors scales as  $\mathcal{O}(k^{-2})$ .
5 Correlations, Noise, and Task Structure Dictate the Ensembling-Subsampling Trade-off
In resource-constrained settings, one must decide between training a single large predictor or an ensemble of smaller predictors. When the number of weights is constrained, ensembling may benefit generalization by averaging over multiple predictions, but at the expense of each prediction incorporating fewer features. Intuitively, the presence of correlations between features limits the penalty incurred by subsampling, as measurements from a subset of features will also confer information about the unsampled features. The equicorrelated data model of proposition 2 permits a solvable toy model for these competing effects. We consider the special case of ensembling over  $k$  readouts, each connecting the same fraction  $\nu_{rr} = \nu = 1 / k$  of all features. For simplicity, we set  $\nu_{rr'} = 0$  for  $r \neq r'$ . We assess the learning curves of this toy model in both the ridgeless limit  $\lambda \to 0$  where double-descent has a large effect on test error, and at 'locally optimal' regularization  $\lambda = \lambda^*$  for which double-descent is eliminated. In these special cases, one can write the generalization error in the following forms (see Appendix H.4 for derivation):
$$
E _ {g} (k, s, c, \eta , \omega , \zeta , \rho , \alpha , \lambda = 0) = s (1 - c) \mathcal {E} (k, \rho , \alpha , H, W, Z) \tag {22}
$$
$$
E _ {g} (k, s, c, \eta , \omega , \zeta , \rho , \alpha , \lambda = \lambda^ {*}) = s (1 - c) \mathcal {E} ^ {*} (k, \rho , \alpha , H, W, Z) \tag {23}
$$
where we have defined the effective noise-to-signal ratios:
$$
H \equiv \frac {\eta^ {2}}{s (1 - c)}, \quad W = \frac {\omega^ {2}}{s (1 - c)}, \quad Z = \frac {\zeta^ {2}}{s (1 - c)} \tag {24}
$$
Therefore, given fixed parameters  $s, c, \rho, \alpha$ , the value  $k^*$  which minimizes error depends on the noise scales,  $s$ , and  $c$  only through the ratios  $H$ ,  $W$  and  $Z$ :
$$
k _ {\lambda = 0} ^ {*} (H, W, Z, \rho , \alpha) \equiv \underset {k \in \mathbb {N}} {\arg \min } E _ {g} (k) = \underset {k \in \mathbb {N}} {\arg \min } \mathcal {E} (k, \rho , \alpha , H, W, Z) \tag {25}
$$
$$
k _ {\lambda = \lambda^ {*}} ^ {*} (H, W, Z, \rho , \alpha) \equiv \underset {k \in \mathbb {N}} {\arg \min } E _ {g} (k) = \underset {k \in \mathbb {N}} {\arg \min } \mathcal {E} ^ {*} (k, \rho , \alpha , H, W, Z) \tag {26}
$$
In Fig. 4a, we plot these reduced errors curves  $\mathcal{E}$ ,  $\mathcal{E}^*$  as a function of  $\alpha$  for varying ensemble sizes  $k$  and reduced readout noise scales  $H$ . At zero regularization learning curves diverge at their interpolation threshold. At locally optimal regularization  $\lambda = \lambda^{*}$ , learning curves decrease monotonically with sample size. Increasing readout noise  $H$  raises generalization error more sharply for smaller  $k$ . In Fig. 4b we plot the optimal  $k^{*}$  in various two-dimensional slices of parameter space in which  $\rho$  is fixed and  $W = Z = 0$  while  $\alpha$  and  $H$  vary. The resulting phase diagrams may be divided into three regions. In the signal-dominated phase a single fully-connected readout is optimal ( $k^{*} = 1$ ). In an intermediate phase,  $1 < k^{*} < \infty$  minimizes error. And in a noise-dominated phase  $k^{*} = \infty$ . At zero regularization, we have determined an analytical expression for the boundary between the intermediate and noise-dominated phases (see Appendix H.4.1 and dotted lines in Figs 4.b,c,d). The signal-dominated, intermediate, and noise-dominated phases persist when  $\lambda = \lambda^{*}$ , removing the effects of double descent. In all panels, an increase in  $\mathrm{H}$  causes an increase in  $k^{*}$ . This can occur because of a decrease in the signal-to-readout noise ratio  $s / \eta^2$ , or through an increase in the correlation strength  $c$ . An increase in  $\rho$  also leads to an increase in  $k^{*}$ , indicating that ensembling is more effective for easier tasks. Figs 4c,d show analogous phase diagrams where  $W$  or  $Z$  are varied. Signal-dominated, intermediate, and noise-dominated regimes are visible in the resulting phase diagrams at zero regularization. However, when optimal regularization is used,  $k^{*} = 1$  is always optimal. The presence of regions where  $k^{*} > 1$  can thus be attributed to double-descent at sub-optimal regularization or to the presence of readout noise which is independent across predictors. We chart the parameter-space of the reduced errors and optimal ensemble size  $k^{*}$  extensively in Appendix I. We plot learning curves for the "ResNext-Imagewoof" ensembled linear classification task with varying strength of readout noise in Fig. 4e, and phase diagrams of optimal ensemble size  $k$  in Fig. 4f, finding similar behavior to the toy model. See Figs. S3, S4, S5 and Appendix E.4.3 for further discussion.
In this paper, we provided a theory of feature-subsampled linear ridge regression. We identified the special case in which features of the data are "equicorrelated" as a minimal toy model to explore the combined effects of subsampling, ensembling, and different types of noise on generalization error. The resulting learning curves displayed two potentially useful phenomena.
First, we demonstrated that heterogeneous ensembling can mitigate over-fitting, reducing or eliminating the double-descent peak of an under-regularized model. In most machine learning applications, the size of the dataset is known at the outset and suitable regularization may be determined to mitigate double descent, either by selecting a highly over-parameterized model [22] or by cross-validation techniques (see for example [17]). However, in contexts where a single network architecture is designed for an unknown task or a variety of tasks with varying structure and noise levels, heterogeneous ensembling may be used to smooth out the perils of double-descent.
Next, we described a trade-off between noise reduction due to ensembling and noise amplification due to subsampling in a resource-constrained setting where the total number of weights is fixed. Our analysis suggests that ensembling is particularly useful in neural networks with an inherent noise. Physical neural networks, such as analog neural networks[40] and biological neural circuits [41] present such a resource-constrained environments where intrinsic noise is a significant issue.
Much work remains to achieve a full understanding of the interactions between data correlations, readout noise, and ensembling. In this work, we have given a thorough treatment of the convenient special case where features are equicorrelated. Future work should analyze subsampling and ensembling for codes with realistic correlation structure, such as the power-law spectra (see Fig. S1) [27, 30, 48, 49] and sparse activation patterns [50].
III
vii
iv
viii
e
Figure 3: Heterogeneous ensembling mitigates double-descent. (a) We compare (i) homogeneous ensembling, in which  $k$  readouts connect to the same fraction  $\nu = 1 / k$  of features, and (ii) heterogeneous ensembling (b) In heterogeneous ensembling subsampling fractions are drawn i.i.d. from  $\Gamma_{k,\sigma}(\nu)$ , shown here for  $k = 10$ , then re-scaled to sum to 1. (c) Generalization Error Curves for Homogeneous and Heterogeneous ensembling with  $k = 10$ ,  $\zeta = 0$ ,  $\rho = 0.3$  and indicated values of  $\lambda$ ,  $c$ , and  $\eta$ . Blue: homogeneous subsampling. Red, green, and purple show heterogeneous subsampling with  $\sigma = 0.25 / k$ ,  $0.5 / k$ ,  $1 / k$  respectively. Dashed lines show learning curves for 3 particular realizations of  $\{\nu_{11},\dots ,\nu_{kk}\}$ . Solid curves show the average over 100 realizations. Gray shows the learning curve for a single linear readout with  $\nu = 1$  and optimal regularization (eq. 193). Triangular marks show the asymptotic generalization error  $(\alpha \rightarrow \infty)$ , with downward-pointing gray triangles indicating an asymptotic error of zero. (d,e) Generalization error of linear classifiers applied to the imagewoof dataset with ResNext features averaged over 100 trials. (d)  $P = 100$ ,  $k = 1$  varying subsampling fraction  $\nu$  and regularization  $\lambda$  (legend). (e) Generalization error of (i) homogeneous and (ii) heterogeneous (with  $\sigma = 0.75 / k$ ) ensembles of classifiers. Legend indicates  $k$  values.  $\lambda = 0$  except for gray curves, where  $\lambda = 0.1$
e
f i
Figure 4: Task parameters dictate the ensembling-subsampling trade-off: (a-d) In the setting of proposition 2 in the special case where all  $\nu_{rr'} = \frac{1}{k}\delta_{rr'}$  so that feature subsets are mutually exclusive and the total number of weights is conserved. (a) We plot the reduced generalization errors  $\mathcal{E}$  (for  $\lambda = 0$ , using eq. 22) and  $\mathcal{E}^*$  (for  $\lambda = \lambda^*$  using eq. 23) of linear ridge ensembles of varying size  $k$  with  $\rho = 0$  and  $H = 0, 1$  (values indicated above plots). Grey lines indicate  $k = 1$ , dashed black lines  $k \to \infty$ , and intermediate  $k$  values by the colorbar. (b) We plot optimal ensemble size  $k^*$  (eqs. 25, 26) in the parameter space of sample size  $\alpha$  and reduced readout noise scale  $H$  setting  $W = Z = 0$ . Grey indicates  $k^* = 1$  and white indicates  $k^* = \infty$ , with intermediate values given by the colorbar. Appended vertical bars show  $\alpha \to \infty$ . Dotted black lines show the analytical boundary between the intermediate and noise-dominated phases given by eq. 214. (c) optimal readout  $k^*$  phase diagrams as in (b) but showing  $W$ -dependence with  $H = Z = 0$ . (d) optimal readout  $k^*$  phase diagrams as in (b) but showing  $Z$ -dependence with  $H = W = 0$ . (e) Learning curves for feature-subsampling ensembles of linear classifiers combined using a majority vote rule on the imagewoof classification task (see Appendix E). As in (a-d) we set  $\nu_{rr'} = \frac{1}{k}\delta_{rr'}$ . Error is calculated as the probability of incorrectly classifying a test example.  $\lambda$  and  $\eta$  values are indicated in each panel. (f) Numerical phase diagrams showing the value of  $k$  which minimizes test error in the parameter space of sample size  $P$  and readout noise scale  $\eta$ , with regularization (i)  $\lambda = 0$  (pseudoinverse rule) (ii)  $\lambda = 0.1$ .
CP is supported by NSF Award DMS-2134157, NSF CAREER Award IIS-2239780, and a Sloan Research Fellowship. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. BSR was also supported by the National Institutes of Health Molecular Biophysics Training Grant NIH/ NIGMS T32 GM008313. We thank Jacob Zavatone-Veth and Blake Bordelon for thoughtful discussion and comments on this manuscript.
[1] Gautam Kunapuli. Ensemble Methods for Machine Learning. Simon and Schuster, 2023.
[2] Robert Bryll, Ricardo Gutierrez-Osuna, and Francis Quek. Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets. Pattern recognition, 36(6):1291-1302, 2003.
[3] Tin Kam Ho. The random subspace method for constructing decision forests. IEEE transactions on pattern analysis and machine intelligence, 20(8):832-844, 1998.
[4] Yali Amit and Donald Geman. Shape quantization and recognition with randomized trees. Neural computation, 9(7):1545-1588, 1997.
[5] Gilles Louppe and Pierre Geurts. Ensembles on random patches. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part I 23, pages 346-361. Springer, 2012.
[6] Marina Skurichina and Robert P. W. Duin. Bagging, boosting and the random subspace method for linear classifiers. Pattern Analysis & Applications, 5(2):121-135, June 2002.
[7] Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume 1, pages 278-282. IEEE, 1995.
[8] Marc Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications. World Scientific Publishing Company, 1987.
[9] M Mezard, G Parisi, and M Virasoro. Spin Glass Theory and Beyond. WORLD SCIENTIFIC, November 1986.
[10] Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical review A, 45(8):6056, 1992.
[11] Andreas Engel and Christian Van den Broeck. Statistical mechanics of learning. Cambridge University Press, 2001.
[12] Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha Sohl-Dickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics, 11:501-528, 2020.
[13] Preetum Nakkiran. More data can hurt for linear regression: Sample-wise double descent. arXiv preprint arXiv:1912.07242, 2019.
[14] Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature communications, 12(1):2914, 2021.
[15] Fatih Furkan Yilmaz and Reinhard Heckel. Regularization-wise double descent: Why it occurs and how to eliminate it. In 2022 IEEE International Symposium on Information Theory (ISIT), pages 426-431. IEEE, 2022.
[50] B OLSHAUSEN and D FIELD. Sparse coding of sensory inputs. Current Opinion in Neurobiology, 14(4):481-487, August 2004.
[51] Luc Devroye. Non-Uniform Random Variate Generation. Springer New York, 1986.
[52] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[53] John R. Silvester. Determinants of block matrices. The Mathematical Gazette, 84(501):460-467, November 2000.
[54] Wolfram Research, Inc. Mathematica, Version 13.3. Champaign, IL, 2023.
A Table of Parameters from Proposition 2 and Figures 2,3,4
Data Scale	s
Correlation Strength	c
Data-Task Alignment	ρ
Readout Noise Scale	η
Feature noise scale	ω
Label Noise Scale	ζ
Ensemble Size	k
Subsampling Fractions	νrr r=1,...,K
Subsampling Overlap Fractions	νrr' r≠r'
Sample Size	P→∞
Data Dimensionality	M→∞
Sample Complexity	α≡P/M (finite)
Effective Noise-To-Signal Ratios	H≡η2/s(1-c), W≡ω2/s(1-c), Z≡ζ2/s(1-c)
B Code Availability and Compute
All Code used in this paper has been made available online (see https://github.com/benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-RidgeEnsembles.git). This includes code used to perform numerical experiments, calculate theoretical learning curves, and produce plots as well as the custom Mathematica libraries used to simplify the generalization error in the special case of equicorrelated data. The compute time required to do all of the calculations in this paper is approximately 3 GPU days.
C Homogeneous and Heterogeneous Subsampling
In this section, we describe the homogeneous and heterogeneous subsampling strategies, as used in this work, in detail. This sampling method was used in calculating the theoretical loss curves for heterogeneous subsampling experiments seen in main text Fig. 3, and in the heterogeneous subsampling experiments applied to the ResNext-features-based image classification task in Figs. 3e, S2.
In homogeneous ensembling, the subsampling fractions  $\nu_{rr} = N_r / M$  are chosen as  $\nu_{rr} = 1 / k$  for all  $r = 1,\dots ,k$ . In heterogeneous ensembling, the subsampling fractions  $\{\nu_{11},\ldots ,\nu_{kk}\}$  are generated according to the following statistical process:
1. Each fraction  $\nu_{rr}$  is drawn independently as  $\nu_{rr} \sim \Gamma_{k,\sigma}$ , where a  $\Gamma_{k,\sigma}$  represents a Gamma distribution with mean  $\frac{1}{k}$  and variance  $\sigma^2$ .
2. The fractions are re-scaled in order to sum to 1:  $\nu_{rr} \rightarrow \nu_{rr} / (\nu_1 + \dots + \nu_k)$
Equivalently, the subsampling fractions are drawn from a Dirichlet distribution parameterized by the ensemble size  $k$  and a chosen variance  $\sigma$  as  $(\nu_{11},\dots,\nu_{kk})\sim \mathrm{Dir}((\sigma k)^{-2},\dots,(\sigma k)^{-2})$  [51].
In main text Fig. 3c, we combine this sampling strategy with theoretical learning curves for the equicorrelated data model in a quasi-numerical experiment. At each trial of the experiment, we draw a particular realization of the subsampling fractions  $\{\nu_{11},\dots,\nu_{kk}\}$ , then use the analytical expression (eq. 16) to calculate the resulting learning curve. Dotted lines show the loss curves for 3 single trials, corresponding to three particular realizations of the subsampling fractions. The solid lines show the average over 100 trials.
Note that we have defined our own convention for the parameterization of the  $\Gamma$  distribution in which the inverse of the mean and the standard deviation are specified. In terms of the standard "shape" and "scale" parameters, we have:
$$
\Gamma_ {k, \sigma} \equiv \Gamma (\text {s h a p e} = (k \sigma) ^ {- 2}, \text {s c a l e} = k \sigma^ {2}) \tag {27}
$$
D Numerical Linear Regression with Synthetic Datasets
Numerical experiments were performed using the PyTorch library [52]. The code used to perform numerical experiments and generate plots has been made publicly available (see section B).
In numerical regression experiments, synthetic datasets with label noise are constructed as described in section 2.1, drawing data randomly from multivariate Gaussian distributions and adding label noise (see "DatasetMaker.py" in available code). Representing the training set in terms of a data matrix  $\Psi \in \mathbb{R}^{M\times P}$  in which column  $\mu$  consist of the training point  $\psi_{\mu}$ , and the labels with a column vector  $\pmb{y}$  such that  $y_{\mu} = y_{\mu}$ , the learned weights are calculated as:
$$
\hat {\boldsymbol {w}} = \boldsymbol {\Psi} \left(\boldsymbol {\Psi} ^ {\top} \boldsymbol {\Psi} + \lambda \boldsymbol {I} _ {p}\right) ^ {- 1} \boldsymbol {y} \tag {28}
$$
In the ridgeless case, a pseudoinverse is used:
$$
\hat {\boldsymbol {w}} = \boldsymbol {\Psi} ^ {\dagger} \boldsymbol {y} \tag {29}
$$
E Ensembled Linear Classification of Imagenet Images
In this section, we provide the details of numerical experiments which demonstrate that qualitative insights gained from our analysis of the linear regression task with Gaussian data carries over to a practical machine learning task. In particular, we apply ensembles of linear classifiers to datasets constructed using a pre-trained ResNext [46] a specific type of Convolutional Neural Network (CNN).
E.1 Dataset Construction