Title: Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles
Venue: NeurIPS 2023
Accepted: yes
Keywords: ridge regression

ensembling methods
Abstract: Feature bagging is a well-established ensembling method which aims to reduce
prediction variance by combining predictions of many estimators trained on subsets
or projections of features. Here, we develop a theory of feature-bagging in noisy
least-squares ridge ensembles and simplify the resulting learning curves in the special
case of equicorrelated data. Using analytical learning curves, we demonstrate
that subsampling shifts the double-descent peak of a linear predictor. This leads
us to introduce heterogeneous feature ensembling, with estimators built on varying
numbers of feature dimensions, as a computationally efficient method to mitigate
double-descent. Then, we compare the performance of a feature-subsampling
ensemble to a single linear predictor, describing a trade-off between noise amplification
due to subsampling and noise reduction due to ensembling. Our qualitative
insights carry over to linear classifiers applied to image classification tasks with
realistic datasets constructed using a state-of-the-art deep learning feature map.

Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles

Abstract

Feature bagging is a well-established ensembling method which aims to reduce prediction variance by combining predictions of many estimators trained on subsets or projections of features. Here, we develop a theory of feature-bagging in noisy least-squares ridge ensembles and simplify the resulting learning curves in the special case of equicorrelated data. Using analytical learning curves, we demonstrate that subsampling shifts the double-descent peak of a linear predictor. This leads us to introduce heterogeneous feature ensembling, with estimators built on varying numbers of feature dimensions, as a computationally efficient method to mitigate double-descent. Then, we compare the performance of a feature-subsampling ensemble to a single linear predictor, describing a trade-off between noise amplification due to subsampling and noise reduction due to ensembling. Our qualitative insights carry over to linear classifiers applied to image classification tasks with realistic datasets constructed using a state-of-the-art deep learning feature map.

1 Introduction

Ensembling methods are ubiquitous in machine learning practice [1]. A class of ensembling methods (known as attribute bagging [2] or the random subspace method [3]) is based on feature subsampling [2-6], where predictors are independently trained on subsets of the features, and their predictions are combined to achieve a stronger prediction. The random forest method is a popular example [3, 7].

While commonly used in practice, a theoretical understanding of ensembling via feature subsampling is not well developed. Here, we provide an analysis of this technique in the linear ridge regression setting. Using methods from statistical physics [8-12], we obtain analytical expressions for typicalcase generalization error in linear ridge ensembles (proposition 1), and simplify these expressions in the special case of equicorrelated data with isotropic feature noise (proposition 2). The result provides a powerful tool to quickly probe the generalization error of ensembled regression under a rich set of conditions. In section 3, we study the behavior of a single feature-subsampling regression model. We observe that subsampling shifts the location of a predictor's sample-wise double-descent peak [13-15]. This motivates section 4, where we study ensembles built on predictors which are heterogeneous in the number of features they access, as a method to mitigate double-descent. We demonstrate this method's efficacy in a realistic image classification task. In section 5 we apply our theory to the trade-off between ensembling and subsampling in resource-constrained settings. We

characterize how a variety of factors influence the optimal ensembling strategy, finding a particular significance to the level of noise in the predictions made by ensemble members.

In summary, we make the following contributions:

Using the replica trick from statistical physics [8, 11], we derive the generalization error of ensembled least-squares ridge regression in a general setting, and simplify the resulting expressions in the tractable special case where features are equicorrelated.

We demonstrate benefits of heterogeneous ensembling as a robust and computationally efficient regularizer for mitigating double-descent with analytical theory and in a realistic image classification task.

We describe the ensembling-subsampling trade-off in resource-constrained settings, and characterize the effect of label noise, feature noise, readout noise, regularization, sample size and task structure on the optimal ensembling strategy.

Related works: A substantial body of work has elucidated the behavior of linear predictors for a variety of feature maps [14, 16-30]. Several recent works have extended this research to characterize the behavior of ensembled regression using solvable models [24, 31-33]. Additional recent works study the performance of ridge ensembles with example-wise subsampling [34, 35] and simultaneous subsampling of features and examples [32], finding that subsampling behaves as an implicit regularization. Methods from statistical physics have long been used for machine learning theory [10-12, 26, 27, 30, 36, 37]. Relevant work in this domain include [38] which studied ensembling by data-subsampling in linear regression.

2 Learning Curves for Ensembled Ridge Regression

We consider noisy ensembled ridge regression in the setting where ensemble members are trained independently on masked versions of the available features. We derive our main analytical formula for generalization error of ensembled linear regression, as well as analytical expressions for generalization error in the special case of equicorrelated features with isotropic noise.

2.1 Problem Setup

Consider a training set D = { ¯ ψ µ , y µ } P µ =1 of size P . The training examples ¯ ψ µ ∈ R M are drawn from a Gaussian distribution with Gaussian feature noise: ¯ ψ µ = ψ µ + σ µ , where ψ µ ∼ N (0 , Σ s ) and σ µ ∼ N (0 , Σ 0 ) . Data and noise are drawn i.i.d. so that E [ ψ µ ψ ν ⊤ ] = δ µν Σ s and E [ σ µ σ ν ⊤ ] = δ µν Σ 0 . Labels are generated from a noisy teacher function y µ = 1 √ M w ∗⊤ ψ µ + ϵ µ where ϵ µ ∼ N (0 , ζ 2 ) . Label noises are drawn i.i.d. so that E [ ϵ µ ϵ ν ] = δ µν ζ 2 .

̸

We seek to analyze the quality of predictions which are averaged over an ensemble of ridge regression models, each with access to a subset of the features. We consider k linear predictors with weights ˆ w r ∈ R N r , r = 1 , . . . , k . Critically, we allow N r = N r ′ for r = r ′ , which allows us to introduce structural heterogeneity into the ensemble of predictors. A forward pass of the model is given as:

̸

The model's prediction f ( ψ ) is an average over k linear predictors. The 'measurement matrices' A r ∈ R N r × M act as linear masks restricting the information about the features available to each member of the ensemble. Subsampling may be implemented by choosing the rows of each A r to coincide with the rows of the identity matrix - the row indices corresponding to indices of the sampled features. The feature noise σ ∼ N (0 , Σ 0 ) and the readout noises ξ r ∼ N (0 , η 2 r ) , are drawn independently at the execution of each forward pass of the model. Note that while the feature noise is shared across the ensemble, readout noise is drawn independently for each readout: E [ ξ r ξ r ′ ] = δ rr ′ η 2 r .

The weight vectors are trained separately in order to minimize an ordinary least-squares loss function with ridge regularization:

Here { ξ µ r } represents the readout noise which is present during training, and independently drawn: ξ µ r ∼ N (0 , η 2 r ) , E [ ξ µ r ξ ν r ] = η 2 r δ µν . As a measure of model performance, we consider the generalization error, given by the mean-squared-error (MSE) on ensemble-averaged prediction:

Here, the expectation is over the data distribution and noise: ψ ∼ N (0 , Σ s ) , σ ∼ N (0 , Σ 0 ) , ξ r ∼ N (0 , η 2 r ) . The generalization error depends on the particular realization of the dataset D through the learned weights { ˆ w ∗ } . We may decompose the generalization error as follows:

̸

Computing the generalization error of the model is then a matter of calculating E rr ′ in the cases where r = r ′ and r = r ′ . In the asymptotic limit we consider, we expect that the generalization error concentrates over randomly drawn datasets D .

2.2 Main Result

We calculate the generalization error using the replica trick from statistical physics, and present the calculation in Appendix F. The result of our calculation is stated in proposition 1.

Proposition 1. Consider the ensembled ridge regression problem described in Section 2.1. Consider the asymptotic limit where M,P, { N r } → ∞ while the ratios α = P M and ν rr = N r M , r = 1 , . . . , k remain fixed. Define the following quantities:

Then the terms of the average generalization error (eq. 5) may be written as:

where the pairs of order parameters { q r , ˆ q r } for r = 1 , . . . , K , satisfy the following self-consistent saddle-point equations

a

b

Proof. We calculate the terms in the generalization error using the replica trick, a standard but non-rigorous method from the statistical physics of disordered systems. The full derivation may be found in the Appendix F. When the matrices A r ( Σ s + Σ 0 ) A ⊤ r , r = 1 , . . . , k have bounded spectra, this result may be obtained by extending the results of [31] to include readout noise, as shown in Appendix G.

We make the following remarks:

Remark 1 . Implicit in this theorem is the assumption that the relevant matrix contractions and traces which appear in the generalization error (eq. 9) and the surrounding definitions tend to a well-defined limit which remains O (1) as M →∞ .

Remark 2 . This result applies for any (well-behaved) linear masks { A r } . We will focus on the case where each A r implements subsampling of an extensive fraction ν rr of the features.

Remark 3 . When k = 1 , our result reduces to the generalization error of a single ridge regression model, as studied in refs. [36, 39].

Remark 4 . Weinclude 'readout noise' which independently corrupts the predictions of each ensemble member. This models sources of variation between ensemble members not otherwise accounted for. For example, ensembles of deep networks will vary due to random initialization of parameters [24, 31, 36]. Readout noise is more directly present in physical neural networks, such as an analog neural networks [40] or biological neural circuits[41] due to their inherent stochasticity.

In Figure 1a, we confirm the result of the general calculation by comparing with numerical experiments using a synthetic dataset with M = 2000 highly structured features (see caption for details). k = 3 readouts see random, fixed subsets of features. Theory curves are calculated by solving the fixed-point equations 10 numerically for the chosen Σ s , Σ 0 and { A r } k r =1 then evaluating eq. 9.

2.3 Equicorrelated Data

Our general result allows the freedom to tune many important parameters of the learning problem: the correlation structure of the dataset, the number of ensemble members, the scales of noise, etc. However, the derived expressions are rather opaque. In order to better understand the phenomena captured by these expressions, we examine the following special case:

Proposition 2. In the setting of section 2.1 and proposition 1, consider the following special case:

̸

with c ∈ [0 , 1] , ρ ∈ [ -1 , 1] . Label and readout noises ζ, η r ≥ 0 are permitted. Here P ⊥ = I M -1 M 1 M 1 ⊤ M is a projection matrix which removes the component of w ∗ 0 which is parallel to 1 M . The matrices { A r } k r =1 have rows consisting of distinct one-hot vectors so that each of the k readouts has access to a subset of N r = ν rr M features. For r = r ′ , denote by n rr ′ the number of neurons sampled by both A r and A r ′ and let ν rr ′ ≡ n rr ′ /M remain fixed as M →∞ .

Define the following quantities:

The terms of the decomposed generalization error may then be written:

where we have defined

and where { q r , ˆ q r } may be obtained analytically as the solution (with q r > 0 ) to:

In the 'ridgeless' limit where all λ r → 0 , we may make the following simplifications:

Proof. Simplifying the fixed-point equations and generalization error formulas in this special case is an exercise in linear algebra. The main tools used are the Sherman-Morrison formula [42] and the fact that the data distribution is isotropic in the features so that the form of ˜ Σ rr and ˜ Σ rr ′ depend only on the subsampling and overlap fractions ν rr , ν r ′ r ′ , ν rr ′ . To aid in computing the necessary matrix contractions we developed a custom Mathematica package which handles block matrices of symbolic dimension, with blocks containing matrices of the form M = c 1 I + c 2 11 ⊤ . This package and the Mathematica notebook used to derive these results are available online (see Appendix B)

In this tractable special case, c ∈ [0 , 1] is a parameter which tunes the strength of correlations between features of the data. When c = 0 , the features are independent, and when c = 1 the features are always equivalent. s sets the overall scale of the features and the 'Data-Task alignment' ρ tunes the alignment of the ground truth weights with the special direction in the covariance matrix (analogous to 'task-model' alignment [14, 27]). A table of parameters is provided in Appendix A. In Figure 1b, we test these results by comparing the theoretical expressions for generalization error with the results of numerical experiments, finding perfect agreement.

With an analytical formula for the generalization error, we can compute the optimal regularization parameters λ r which minimize the generalization error. These may, in general, depend on both r and the sample size α . Rather than minimizing the error of the ensemble, we may minimize the generalization error of predictions made by the ensemble members independently. We find that this 'locally optimal' regularization, denoted λ ∗ , is independent of α , generalizing results from [14, 43] to correlated data distributions (see Appendix H.3).

b

a

i

ii

iii

𝜆𝜆

= 0

𝜆𝜆

= .0001

𝜆𝜆

= .01

3 Subsampling shifts the double-descent peak of a linear predictor

Consider a single linear regressor ( k = 1 ) which connects to a subset of N = νM features in the equicorrelated data setting of proposition 2. Also setting c = 0 , s = 1 , and η r = ω = 0 and taking the limit λ → 0 the generalization error reads:

We thus see that double descent can arise from two possible sources of variance: explicit label noise (if ζ > 0 ) or implicit label noise induced by feature subsampling ( ν < 1 ). As E g ∼ ( α -ν ) -1 , generalization error diverges when sample size is equal to the number of sampled features. Intuitively, this occurs because subsampling changes the number of parameters of the regression model, and thus its interpolation threshold. To demonstrate this, we plot the learning curves for subsampled linear regression on equicorrelated data in Figure 2. At small finite ridge the test error no longer diverges when α = ν , but still displays a distinctive peak.

4 Heterogeneous connectivity mitigates double-descent

Double-descent - over-fitting to noise in the training set near a model's interpolation threshold - poses a serious risk in practical machine-learning applications [22]. Cross-validating the regularization strength against the training set is the canonical approach to avoiding double-descent [17, 43], but in practice requires a computationally expensive parameter sweep and prior knowledge of the task. In situations where computational resources are limited or hyperparameters are fixed prior to specification of the task, it is natural to seek an alternative solution. Considering again the plots in Figure 2(b), we observe that at any value of α , the double-descent peak can be avoided with an acceptable choice of the subsampling fraction ν . This suggests another strategy to mitigate double descent: heterogeneous ensembling. Ensembling over predictors with a heterogeneous distribution of interpolation thresholds, we may expect that when one predictor fails due to over-fitting, the other members of the ensemble compensate with accurate predictions.

̸

In Figure 3, we show that heterogeneous ensembling can guard against double-descent. We define two ensembling strategies: in homogeneous ensembling, each of k readouts connects a fraction ν rr = 1 /k features. In heterogeneous ensembling, the number of features connected by each of the k readouts are drawn from a Gamma distribution Γ k,σ ( ν ) with mean 1 /k and standard deviation σ (see Fig. 3b) then re-scaled to sum to 1 (see Appendix C for details). All feature subsets are mutually exclusive ( ν rr ′ = 0 for r = r ′ ). Homogeneous and heterogeneous ensembling are illustrated for k = 10 in Figs. 3 a.i and 3 a.ii respectively. We test this hypothesis using eq. 16 in 3c. At small regularization

( λ = . 001 ), we find that heterogeneity of the distribution of subsampling fractions ( σ > 0 ) lowers the double-descent peak of an ensemble of linear predictors, while at larger regularization ( λ = 0 . 1 ), there is little difference between homogeneous and heterogeneous learning curves. The asymptotic ( α →∞ ) error is unaffected by the presence of heterogeneity in the degrees of connectivity, which can be seen as the coincidence of the triangular markers in Fig. 3c, as well as from the α →∞ limit of eq. 16 (see Appendix H.5). Fig. 3c also shows the learning curve of a single linear predictor with no feature subsampling and optimal regularization. We see that the feature-subsampling ensemble appreciably outperforms the fully-connected model when c = 0 . 8 and η = 0 . 5 , suggesting the important roles of data correlations and readout noise in determining the optimal readout strategy. These roles are further explored in section 5 and fig 4.

We also test the effect of heterogeneous ensembling in the a realistic classification task. Specifically, we train ensembles of linear classifiers to predict the labels of imagenet [44] images corresponding to 10 different dog breeds (the 'Imagewoof' task [45]) from their top-hidden-layer representations in a pre-trained ResNext deep network [46] (see Appendix E for details). We characterize the statistics of the resulting M = 2048 -dimensional feature set in Fig. S1. This 'ResNext-Imagewoof' classification task has multiple features which make it amenable to learning with a feature-subsampling ensemble. First, the ResNext features have a high degree of redundancy [47], allowing classification to be performed accurately using only a fraction of the available features (see Fig. 3d and S1c). Second, when classifications of multiple predictors are combined by a majority vote, there is a natural upper bound on the influence of a single erring ensemble member (unlike in regression where predictions can diverge). Calculating learning curves for the imagewoof classification task using homogeneous ensembles, we see sharp double-descent peaks in an ensemble of size k when P = M/k (Fig. 3e.i). Using a heterogeneous ensemble mitigates this catastrophic over-fitting, leading to monotonically decreasing error without regularization (Fig. 3e.ii). A single linear predictor with a tuned regularization of λ = 0 . 1 performs only marginally better than the heterogeneous featuresubsampling ensemble with k = 16 or k = 32 . This suggests heterogeneous ensembling can be an effective alternative to regularization in real-world classification tasks using pre-trained deep learning feature maps.

Note that training a feature-subsampling ensemble also benefits from improved computational complexity. Training an estimator of dimension N r involves, in the worst case, inverting an N r × N r matrix, which requires O ( N 3 r ) operations. Setting N r = M/k , we see that the number of operations required to train an ensemble of k predictors scales as O ( k -2 ) .

5 Correlations, Noise, and Task Structure Dictate the Ensembling-Subsampling Trade-off

̸

In resource-constrained settings, one must decide between training a single large predictor or an ensemble of smaller predictors. When the number of weights is constrained, ensembling may benefit generalization by averaging over multiple predictions, but at the expense of each prediction incorporating fewer features. Intuitively, the presence of correlations between features limits the penalty incurred by subsampling, as measurements from a subset of features will also confer information about the unsampled features. The equicorrelated data model of proposition 2 permits a solvable toy model for these competing effects. We consider the special case of ensembling over k readouts, each connecting the same fraction ν rr = ν = 1 /k of all features. For simplicity, we set ν rr ′ = 0 for r = r ′ . We asses the learning curves of this toy model in both the ridgeless limit λ → 0 where double-descent has a large effect on test error, and at 'locally optimal' regularization λ = λ ∗ for which double-descent is eliminated. In these special cases, one can write the generalization error in the following forms (see Appendix H.4 for derivation):

where we have defined the effective noise-to-signal ratios:

Therefore, given fixed parameters s, c, ρ, α , the value k ∗ which minimizes error depends on the noise scales, s , and c only through the ratios H , W and Z :

In Fig. 4a, we plot these reduced errors curves E , E ∗ as a function of α for varying ensemble sizes k and reduced readout noise scales H . At zero regularization learning curves diverge at their interpolation threshold. At locally optimal regularization λ = λ ∗ , learning curves decrease monotonically with sample size. Increasing readout noise H raises generalization error more sharply for smaller k . In Fig. 4b we plot the optimal k ∗ in various two-dimensional slices of parameter space in which ρ is fixed and W = Z = 0 while α and H vary. The resulting phase diagrams may be divided into three regions. In the signal-dominated phase a single fully-connected readout is optimal ( k ∗ = 1 ). In an intermediate phase, 1 < k ∗ < ∞ minimizes error. And in a noise-dominated phase k ∗ = ∞ . At zero regularization, we have determined an analytical expression for the boundary between the intermediate and noise-dominated phases (see Appendix H.4.1 and dotted lines in Figs 4.b,c,d). The signal-dominated, intermediate, and noise-dominated phases persist when λ = λ ∗ , removing the effects of double descent. In all panels, an increase in H causes an increase in k ∗ . This can occur because of a decrease in the signal-to-readout noise ratio s/η 2 , or through an increase in the correlation strength c . An increase in ρ also leads to an increase in k ∗ , indicating that ensembling is more effective for easier tasks. Figs 4c,d show analogous phase diagrams where W or Z are varied. Signal-dominated, intermediate, and noise-dominated regimes are visible in the resulting phase diagrams at zero regularization. However, when optimal regularization is used, k ∗ = 1 is always optimal. The presence of regions where k ∗ > 1 can thus be attributed to double-descent at sub-optimal regularization or to the presence of readout noise which is independent across predictors. We chart the parameter-space of the reduced errors and optimal ensemble size k ∗ extensively in Appendix I. We plot learning curves for the 'ResNext-Imagewoof' ensembled linear classification task with varying strength of readout noise in Fig. 4e, and phase diagrams of optimal ensemble size k in Fig. 4f, finding similar behavior to the toy model. See Figs. S3, S4, S5 and Appendix E.4.3 for further discussion.

6 Conclusion

In this paper, we provided a theory of feature-subsampled linear ridge regression. We identified the special case in which features of the data are 'equicorrelated' as a minimal toy model to explore the combined effects of subsampling, ensembling, and different types of noise on generalization error. The resulting learning curves displayed two potentially useful phenomena.

First, we demonstrated that heterogeneous ensembling can mitigate over-fitting, reducing or eliminating the double-descent peak of an under-regularized model. In most machine learning applications, the size of the dataset is known at the outset and suitable regularization may be determined to mitigate double descent, either by selecting a highly over-parameterized model [22] or by cross-validation techniques (see for example [17]). However, in contexts where a single network architecture is designed for an unknown task or a variety of tasks with varying structure and noise levels, heterogeneous ensembling may be used to smooth out the perils of double-descent.

Next, we described a trade-off between noise reduction due to ensembling and noise amplification due to subsampling in a resource-constrained setting where the total number of weights is fixed. Our analysis suggests that ensembling is particularly useful in neural networks with an inherent noise. Physical neural networks, such as analog neural networks[40] and biological neural circuits [41] present such a resource-constrained environments where intrinsic noise is a significant issue.

Much work remains to achieve a full understanding of the interactions between data correlations, readout noise, and ensembling. In this work, we have given a thorough treatment of the convenient special case where features are equicorrelated. Future work should analyze subsampling and ensembling for codes with realistic correlation structure, such as the power-law spectra (see Fig. S1) [27, 30, 48, 49] and sparse activation patterns [50].

Γ𝑘𝑘

,

𝜎𝜎

(

𝜈𝜈

)

a

i

ii

c

i

ii

iii

iv

b

𝜈𝜈

d

e

ii

v

vi

vii

viii

i

ii

𝜎𝜎

e

i

iii

f

i

ii

b

a

c

d

p=0,A=0

p=0,入=入

0

0.0

1.0

2.0

3.0 ∞

0.0

1.0

2.0

3.0∞

p=0.6,=0

p=0.6,入=入*

0

20

30

p=0.6,A=0

0

3

0

p=0.6,A=0

P=0.6,A=A

0

k

1

2

u

3

4

二

5

10

15

20

25

30

n

100

200

5

5

5

0.1

5

5

5

References

Gautam Kunapuli. Ensemble Methods for Machine Learning . Simon and Schuster, 2023.

Robert Bryll, Ricardo Gutierrez-Osuna, and Francis Quek. Attribute bagging: improving accuracy of classifier ensembles by using random feature subsets. Pattern recognition , 36(6):12911302, 2003.

Tin Kam Ho. The random subspace method for constructing decision forests. IEEE transactions on pattern analysis and machine intelligence , 20(8):832-844, 1998.

Yali Amit and Donald Geman. Shape quantization and recognition with randomized trees. Neural computation , 9(7):1545-1588, 1997.

Gilles Louppe and Pierre Geurts. Ensembles on random patches. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part I 23 , pages 346-361. Springer, 2012.

Marina Skurichina and Robert P. W. Duin. Bagging, boosting and the random subspace method for linear classifiers. Pattern Analysis & Applications , 5(2):121-135, June 2002.

Tin Kam Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition , volume 1, pages 278-282. IEEE, 1995.

Marc Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin Glass Theory and Beyond: An Introduction to the Replica Method and Its Applications . World Scientific Publishing Company, 1987.

MMezard, G Parisi, and M Virasoro. Spin Glass Theory and Beyond . WORLD SCIENTIFIC, November 1986.

Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical review A , 45(8):6056, 1992.

Andreas Engel and Christian Van den Broeck. Statistical mechanics of learning . Cambridge University Press, 2001.

Yasaman Bahri, Jonathan Kadmon, Jeffrey Pennington, Sam S Schoenholz, Jascha SohlDickstein, and Surya Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physics , 11:501-528, 2020.

Preetum Nakkiran. More data can hurt for linear regression: Sample-wise double descent. arXiv preprint arXiv:1912.07242 , 2019.

Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature communications , 12(1):2914, 2021.

Fatih Furkan Yilmaz and Reinhard Heckel. Regularization-wise double descent: Why it occurs and how to eliminate it. In 2022 IEEE International Symposium on Information Theory (ISIT) , pages 426-431. IEEE, 2022.

Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in highdimensional ridgeless least squares interpolation. The Annals of Statistics , 50(2):949 - 986, 2022.

Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction , volume 2. Springer, 2009.

Jason W. Rocks and Pankaj Mehta. Bias-variance decomposition of overparameterized regression with random linear features. Physical Review E , 106:025304, Aug 2022.

Jason W. Rocks and Pankaj Mehta. Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models. Physical Review Research , 4:013201, Mar 2022.

Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics , 75, 2019.

Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machinelearning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Hong Hu and Yue M. Lu. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory , 69(3):1932-1964, 2023.

Stéphane D'Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in double descent: Bias and variance(s) in the lazy regime. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 2280-2290. PMLR, 13-18 Jul 2020.

Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine-grained biasvariance decomposition. In Advances in Neural Information Processing Systems , volume 33, pages 11022-11032, 2020.

Jacob A. Zavatone-Veth, William L. Tong, and Cengiz Pehlevan. Contrasting random and learned features in deep bayesian linear regression, 2022.

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1024-1034. PMLR, 13-18 Jul 2020.

James B. Simon, Madeline Dickens, Dhruva Karkada, and Michael R. DeWeese. The Eigenlearning framework: A conservation law perspective on kernel regression and wide neural networks. arXiv , 2022.

Francis Bach. High-dimensional analysis of double descent for linear regression with random projections, 2023.

Jacob A. Zavatone-Veth and Cengiz Pehlevan. Learning curves for deep structured gaussian feature models, 2023.

Bruno Loureiro, Cedric Gerbelot, Maria Refinetti, Gabriele Sicuro, and Florent Krzakala. Fluctuations, bias, variance & ensemble of learners: Exact asymptotics for convex losses in high-dimension. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 14283-14314. PMLR, 17-23 Jul 2022.

Daniel LeJeune, Hamid Javadi, and Richard Baraniuk. The implicit regularization of ordinary least squares ensembles. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , volume 108 of Proceedings of Machine Learning Research , pages 3525-3535. PMLR, 26-28 Aug 2020.

Pratik Patil and Daniel LeJeune. Asymptotically free sketched ridge ensembles: Risks, crossvalidation, and tuning, 2023.

Jin-Hong Du, Pratik Patil, and Arun Kumar Kuchibhotla. Subsample ridge ensembles: Equivalences and generalized cross-validation, 2023.

Pratik Patil, Jin-Hong Du, and Arun Kumar Kuchibhotla. Bagging in overparameterized learning: Risk characterization and risk monotonization, 2022.

Alexander Atanasov, Blake Bordelon, Sabarish Sainathan, and Cengiz Pehlevan. The onset of variance-limited behavior for networks in the lazy and rich regimes. In The Eleventh International Conference on Learning Representations , 2023.

Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Out-of-distribution generalization in kernel regression. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 12600-12612. Curran Associates, Inc., 2021.

Peter Sollich and Anders Krogh. Learning with ensembles: How overfitting can be useful. Advances in neural information processing systems , 8, 1995.

Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborova. Learning curves of generic features maps for realistic datasets with a teacher-student model. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Devon Janke and David V. Anderson. Analyzing the effects of noise and variation on the accuracy of analog neural networks. In 2020 IEEE 63rd International Midwest Symposium on Circuits and Systems (MWSCAS) . IEEE, August 2020.

A. Aldo Faisal, Luc P. J. Selen, and Daniel M. Wolpert. Noise in the nervous system. Nature Reviews Neuroscience , 9(4):292-303, April 2008.

Jack Sherman and Winifred J. Morrison. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics , 21(1):124-127, March 1950.

Preetum Nakkiran, Prayaag Venkat, Sham Kakade, and Tengyu Ma. Optimal regularization can mitigate double descent. arXiv preprint arXiv:2003.01897 , 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

J Howard. Imagenette. Accessed: 25-10-2025.

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks, 2017.

Diego Doimo, Aldo Glielmo, Sebastian Goldt, and Alessandro Laio. Redundant representations help generalization in wide neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Alexander Maloney, Daniel A. Roberts, and James Sully. A solvable model of neural scaling laws, 2022.

Blake Bordelon and Cengiz Pehlevan. Population codes enable learning from few examples by shaping inductive bias. eLife , 11, December 2022.

B OLSHAUSEN and D FIELD. Sparse coding of sensory inputs. Current Opinion in Neurobiology , 14(4):481-487, August 2004.

Luc Devroye. Non-Uniform Random Variate Generation . Springer New York, 1986.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

John R. Silvester. Determinants of block matrices. The Mathematical Gazette , 84(501):460-467, November 2000.

Wolfram Research, Inc. Mathematica, Version 13.3. Champaign, IL, 2023.

A Table of Parameters from Proposition 2 and Figures 2,3,4

̸

Data Scale Correlation Strength Data-Task Alignment Readout Noise Scale Feature noise scale Label Noise Scale Ensemble Size Subsampling Fractions Subsampling Overlap Fractions Sample Size Data Dimensionality Sample Complexity

s

c

ρ

η

ω

ζ

k

ν

rr

r

= 1

, . . . , K

ν

rr

′

r

=

r

′

P

→∞

M

→∞

α

≡

P

M

(finite)

Effective Noise-To-Signal Ratios

H

≡

η

2

s

(1

-

c

)

,

W

≡

ω

2

s

(1

-

c

)

,

Z

≡

ζ

2

s

(1

-

c

)

B Code Availability and Compute

All Code used in this paper has been made available online (see https://github.com/benruben87/Learning-Curves-for-Heterogeneous-Feature-Subsampled-RidgeEnsembles.git). This includes code used to perform numerical experiments, calculate theoretical learning curves, and produce plots as well as the custom Mathematica libraries used to simplify the generalization error in the special case of equicorrelated data. The compute time required to do all of the calculations in this paper is approximately 3 GPU days.

C Homogeneous and Heterogeneous Subsampling

In this section, we describe the homoegeneous and heterogeneous subsampling strategies, as used in this work, in detail. This sampling method was used in calculating the theoretical loss curves for heterogeneous subsampling experiments seen in main text Fig. 3, and in the heterogeneous subsampling experiments applied to the ResNext-features-based image classification task in Figs. 3e, S2.

In homogeneous ensembling, the subsampling fractions ν rr = N r /M are chosen as ν rr = 1 /k for all r = 1 , . . . , k . In heterogeneous ensembling, the subsampling fractions { ν 11 , . . . , ν kk } are generated according to the following statistical process:

Each fraction ν rr is drawn independently as ν rr ∼ Γ k,σ , where a Γ k,σ represents a Gamma distribution with mean 1 k and variance σ 2 .

The fractions are re-scaled in order to sum to 1: ν rr → ν rr / ( ν 1 + · · · + ν k )

Equivalently, the subsampling fractions are drawn from a Dirichlet distribution parameterized by the ensemble size k and a chosen variance σ as ( ν 11 , . . . , ν kk ) ∼ Dir(( σk ) -2 , . . . , ( σk ) -2 ) [51].

In main text Fig. 3c, we combine this sampling strategy with theoretical learning curves for the equicorrelated data model in a quasi-numerical experiment. At each trial of the experiment, we draw a particular realization of the subsampling fractions { ν 11 , . . . , ν kk } , then use the analytical expression (eq. 16) to calculate the resulting learning curve. Dotted lines show the loss curves for 3 single trials, corresponding to three particular realizations of the subsampling fractions. The solid lines show the average over 100 trials.

Note that we have defined our own convention for the parameterization of the Γ distribution in which the inverse of the mean and the standard deviation are specified. In terms of the standard 'shape' and 'scale' parameters, we have:

D Numerical Linear Regression with Synthetic Datasets

Numerical experiments were performed using the PyTorch library [52]. The code used to perform numerical experiments and generate plots has been made publicly available (see section B).

In numerical regression experiments, synthetic datasets with label noise are constructed as described in section 2.1, drawing data randomly from multivariate Gaussian distributions and adding label noise (see 'DatasetMaker.py' in available code). Representing the training set in terms of a data matrix Ψ ∈ R M × P in which column µ consist of the training point ψ µ , and the labels with a column vector y such that y µ = y µ , the learned weights are calculated as:

In the ridgeless case, a pseudoinverse is used:

E Ensembled Linear Classification of Imagenet Images

In this section, we provide the details of numerical experiments which demonstrate that qualitative insights gained from our analysis of the linear regression task with Gaussian data carries over to a practical machine learning task. In particular, we apply ensembles of linear classifiers to datasets constructed using a pre-trained ResNext [46] a specific type of Convolutional Neural Network (CNN).

E.1 Dataset Construction

To construct the dataset, we start with a set of n images { x µ } n µ =1 from a subset of C = 10 classes of the imagenet dataset. For each image x µ , we obtain a corresponding feature vector ψ µ ∈ R M as the last-hidden-layer activation of the ResNext [46], which has been pre-trained on the imagenet classification task [44]. The architecture we use produces M = 2048 features per image. These features will serve as the data input to the downstream linear classifier. The corresponding labels y µ ∈ R C are one-hot vectors.

We construct two datasets using this method, using images from the 'Imagenette' and 'Imagewoof' datasets [45]. For the 'Imagenette' task, we a construct a training set of size n tr = 9469 and a test set of size n test = 3925 containing features corresponding to images from 10 unrelated classes (tench, English springer, cassette player, chain saw, church, French horn, garbage truck, gas pump, golf ball, parachute). For the 'Imagewoof' task, we a construct a training set of size n tr = 9025 and a test set of size n test = 3929 containing features corresponding to images of 10 different dog breeds (Australian terrier, Border terrier, Samoyed, Beagle, Shih-Tzu, English foxhound, Rhodesian ridgeback, Dingo, Golden retriever, Old English sheepdog). The imagewoof classification task is naturally more difficult. The statistics of the resulting datasets are described in Fig. S1, where we plot the data-data covariance matrix, feature-feature covariance matrix, and the eigenvalue spectrum for both the 'Imagenette' and 'Imagewoof' tasks.

E.2 Model Training

At training time a dataset of P examples is constructed: D = { ψ µ , y µ } P µ =1 . We represent the training set with a 'design matrix' Φ ∈ R P × M and a label matrix Y ∈ { 0 , 1 } P × C . The loss function for each ensemble member is generalized to multi-class regression. With ridge regularization, the objective for each ensemble member r becomes:

Imagenette

a

b

c

Data Covariance Matrix

0.020

0.015

0.010

0.005

0.000

Data Covariance Matrix

0.020

0.015

0.010

0.005

0.000

where ∥ · ∥ F denotes the Frobenius norm and where the readout noise vector ξ µ r ∼ N (0 , η 2 I C ) is a C - dimensional readout noise which corrupts the model's prediction. The weights ˆ W r can be determined in closed form as follows:

where we have defined [ Ξ r ] µc = [ ξ µ r ] c . When λ = 0 we instead use a pseudoinverse rule. In all experiments presented here, we use measurement matrices A r which implement an ordinary subsampling of the features, so that the rows and columns of A r consist of one-hot vectors.

E.3 Model Prediction

Once trained, the learned weights may be used to predict the label of a new example ψ as follows. For r = 1 , . . . , k we calculate the prediction of each ensemble member by first assigning each class a 'score'. The scores of predictor r are stored in a vector f r ∈ R C :

Where ξ r ∼ N (0 , η 2 I C ) is drawn randomly at model evaluation. Each ensemble member's 'vote' corresponds to the class with the largest score. The prediction of the ensemble is then calculated as a majority vote of the ensemble members. Generalization error is then calculated as the probability of misclassifying an example from the test set.

E.4 Linear Classification Experiments

We apply the described majority-vote linear classifier ensembles to the ResNext-Imagenette and ResNext-Imagewoof tasks in three different experiments.

E.4.1 Reduncancy of ResNext Features

In the first experiment, we investigate the performance of a single linear classifier ( k = 1 ) as the fraction of features ν which it has access to varies. We set P = 1000 and vary ν over 50 values on a logarithmic scale from 10 -3 to 1 . We also vary the regularization strength over 0 and a logarithmic scale from 10 -3 to 10 4 . We average over 100 trials. At each trial the particular subset of P = 1000 training examples and the particular subset of ν ∗ M features is randomly re-sampled. We find that ResNext features are highly redundant - classification accuracy is very robust to subsampling of the features. For example, in the Imagewoof classification task with the best regularization tested ( λ = 0 . 1 ), test error increases from about 1% to about 2% as the subsampling fraction decreases from ν = 1 to ν = 0 . 1 (meaning 90% of the features are ignored) (see Fig. 3d). Similarly, for the imagenette task, test error increases from about 0.1% to about 0.2% as the subsampling fraction decreases from ν = 1 to ν = 0 . 1 (see Fig. S1c). Code used to run these experiments may be found in the folder 'DeepNet_Subsamp' in the GitHub repository.

E.4.2 Heterogeneous Ensembling Mitigates Double-Descent

In the second experiment, we compare learning curves for homogeneous ensembling and heterogeneous ensembling applied to the ResNext-Imagewoof classification task. In each trial, we train ensembles of k = { 1 , 4 , 8 , 16 , 32 } linear predictors whose subsampling fractions { ν rr , . . . , ν kk } are assigned either with Homogeneous ensembling or with Heterogeneous Ensembling with σ = 0 . 75 /k C. After subsampling fraction are assigned, the training set is randomly shuffled. We iterate over 50 sample sizes P logarithmically distributed from 400 to 4000 , and then add the values M/k for each k to the list of P values. We repeat for 100 trials for both λ r = 0 (pseudoinverse rule) and λ = 0 . 1 ,

ensembles

̸

which was found to mitigate double-descent by the parameter sweep in Fig. S1c. In main text Fig. 3e, we plot the mean learning curves over 100 trials. In Fig. S2 we show standard deviation over the 100 trials as shaded error bars. When λ = 0 , heterogeneous ensembling mitigates double-descent, leading to a monotonically decreasing learning curve for sufficiently high k . When λ = 0 . 1 , homogeneous and heterogeneous ensembles of size k perform similarly. Code used to run these experiments may be found in the folder 'DeepNet_HomVHet' in the GitHub repository.

E.4.3 Readout Noise Encourages Ensembling

In the third experiment, we test the effect of a readout noise which is independent across the members of the ensemble on generalization error. We do this by sweeping over the readout noise scale η as defined in sections E.2, E.3. For λ = 0 , 0 . 1 and ν = 0 , . 1 , . . . , 1 we compute the learning curves of linear predictors with k = 1 , 2 , 3 , 4 , 5 , 10 , 15 , 20 , 25 , 30 , 100 , 200 and all ν rr = 1 /k , averaged over 50 trials. These learning curves are shown in Fig. S3 for both the ResNext-Imagenette and ResNext-Imagewoof task. In Figs. S4a and S5a, we plot the value k ∗ which minimizes error as pase diagrams in the parameter space of α and η , analogous to the phase diagrams in Fig. 4b. We see that the qualitative shape of these phase diagrams is similar to the equicorrelated model. The differences may be attributed to the nonlinear nature of the classification task. Furthermore, we find

that, in general the optimal k ∗ tends to be higher with the Imagenette dataset than with the Imagewoof dataset, in agreement with our finding in the equicorrelated regression model that as ρ increases (making the classification task easier), optimal ensemble size tends to increase (Fig. 4b and S6, S7, S8). In Fig S3 we see that there are often a number of k values for which test error is at or near to its lowest. To quantify this, we also plot diagrams of the minimum and maximum values of k that are within an small margin ϵ of the minimum measured error. For the ResNext-Imagenette task, we use ϵ = 0 . 001 and for ResNext-Imagewoof ϵ = 0 . 01 . We see that there is a wide array of k values which bring error near-to-minimum in practice (Figs S4b,c, S5b,c). We also plot the minimum achieved error, and the difference between minimum errors at λ = 0 . 1 , 0 (Figs. S4d, S5d). Code used to run these experiments may be found in the folder 'DeepNet_PD' in the GitHub repository.

Imagenette

a

b

c

d

e

K"A=Oimagenette2

0.

0

0.

O5

47

5

5

5

05

11

KminA=Olmagehette2

一

0.

O

0.

KminA=0.1imagehette2

5

5

5

0

Kmax

=Oirmiagenettez

1

0.

0

0.

八

=0.11magenette2

5

5

Optimal E 入= 0.1 imagenette2

D

5

0.05

0.41

0.77

1.14

1.5

0

5

0

05

4.1

77

114

E(A =0.1)-E(A =0) imagenette2

0.05

0.41

0.77

1.14

1.5

a

b

c

d

e

K"A=Oimagewoofz

0

5

5

0

05

11

K"A=0.1imagewoofz

5

5

5

0

05

41

14

0

5

0

O5

4.1

0

5

5

5

0

O5

LL

Nmax

=Oirmagewoofz

0

5

0

5

5

0

0

=o.1imagewoof2

0

5

5

5

0

Optimal E 入 = 0.1 imagewoof2

5

0.05

0.41

0.77

1.14

1.5

OptimalEA=oimagewoolz

0

5

0

OE

E(入 =0.1)-E(入 =0)imagewoof2

0

5

0

0.05

0.41

0.77

1.14

1.5

F Generalization error of ensembled linear regression from the replica trick

̸

Here we use the Replica Trick from statistical physics to derive analytical expressions for E rr ′ . We treat the cases where r = r ′ and r = r ′ separately. Following a statistical mechanics approach, we calculate the average generalization error over a Gibbs measure with inverse temperature β ;

In the limit where β →∞ the gibbs measure will concentrate around the weight vector ˆ w r which minimizes the regularized loss function. The replica trick relies on the following identity:

where ⟨·⟩ D represents an average over all quenched disorder in the system. In this case, quenched disorder - the disorder which is fixed prior to and throughout training of the weights - consists of the selected training examples along with their feature noise and label noise: D = { ψ µ , σ µ , ϵ µ } P µ =1 . The calculation proceeds by first computing the average of the replicated partition function assuming n is a positive integer. Then, in a non-rigorous but standard step, we analytically extend the result to n → 0 .

F.1 Diagonal Terms

We start by calculating E rr for some fixed choice of r . This derivation partially follows section D.3 from [36], with the addition of readout noise and label noise. Noting that the diagonal terms of the generalization error E rr only depend on the learned weights w r , and the loss function separates over the readouts, we may consider the Gibbs measure over only these weights:

Next we must perform the averages over quenched disorder. We first integrate over { ψ µ , σ µ , ξ µ r , ϵ µ } P µ =1 . Noting that the scalars

are Gaussian random variables (when conditioned on A r ) with mean zero and covariance:

To perform this integral we re-write in terms of { H r µ } P µ =1 , where

Integrating over the H r µ we get:

Next we integrate over Q r and add constraints. We use the following identity:

Using the Fourier representation of the delta function, we get:

Inserting this identity into the replicated partition function and substituting E rr ( w a r ) = Q rr aa -ζ 2 we find:

In order to perform the Gaussian integral over the { w a r } , we unfold over the replica index a . We first define the following:

We then have for the integral over w

We can finally write the replicated partition function as:

We now make the following replica-symmetric ansatz:

which is well-motivated because the loss function is convex. We will verify that the chosen scalings are self-consistent in the zero-temperature limit where β →∞ . We may then rewrite the partition function as follows:

Where the effective action is written:

To determine the values of the order parameters we set the derivatives of g [ q, q 0 , ˆ q, ˆ q 0 ] , evaluated at zero source ( J = 0 ), to zero:

where γ ≡ α Mκ 2 tr [ ( G -1 ˜ Σ ) 2 ] and κ ≡ λ + q . We retroactively confirm that the chosen β -scalings are correct by noticing that the saddle-point equations permit a solution where all order parameters remain O (1) as β →∞ . The error is given as:

Where q, q 0 , ˆ q, ˆ q 0 are the solutions to the saddle-point equations 53, 54, 55, 57. Substituting eq. 57 for q 0 , we obtain:

F.2 Off-Diagonal Terms

̸

We now calculate E rr ′ for r = r ′ . We now must consider the joint Gibbs Measure over w r and w r ′ :

(62)

Where the h ra µ are defined as before. Next we must perform the averages over quenched disorder. To do so, we note that the h ra µ are Gaussian random variables with covariance structure:

To perform this integral we re-write in terms of { H µ } P µ =1 , where

Integrating over H µ we get:

Where we have defined the matrix Q so that:

Next we integrate over Q and add constraints. We use the following identity:

Using the Fourier representation of the delta function, we get:

Inserting this identity and the corresponding statements for Q rr ab and Q r ′ r ′ ab into the replicated partition function and substituting E rr ′ ( w a ) = Q rr ′ aa -ζ 2 we find:

Where sums over r 1 and r 2 run over { r, r ′ } .

In order to perform the Gaussian integral over the { w a r } , we unfold in two steps. We first define the following:

Unfolding over the replica indices, we then get:

Note that the dimensionality of T r 1 r 2 varies for different choices of r 1 and r 2 . Next, we unfold over the two readouts:

The integral over w then becomes:

We are now ready to make a replica-symmetric ansatz. The order parameter that we wish to constrain is Q rr ′ ab . Overlaps go between the weights from different replicas of the system as well as different readouts. The scale of the overlap between two measurements depends on their overlap with each other and with the principal components of the data distribution. An ansatz which is replica-symmetric but makes no assumptions about the overlaps between different measurements is as follows:

Next step is to plug the RS ansatz into the free energy and simplify. To make calculations more transparent, we re-label the paramters in the RS ansatz as follows:

In order to simplify log det ( λ I 2 n + β Q ) , we note that this is a symmetric 2-by-2-block matrix, where each block commutes with all other blocks. We may then use [53]'s result to simplify.

Collecting these terms, we may write the replicated partition function as follows:

Where the free energy is written:

-

[(

q

ˆ

q

+

β

ˆ

qQ

+

βq

ˆ

Q

)

+

(

r

ˆ

r

+

β

ˆ

rR

+

βr

ˆ

R

)

+2

(

c

ˆ

c

+

β

ˆ

cC

+

βc

ˆ

C

)]

(100)

where we have defined G ≡ [ G rr G rr ′ G r ′ r G r ′ r ′ ]

The saddle-point equations for the replica-diagonal order parameters are:

Note that when J = 0 , the saddle point equations 109, 110 are solved by setting c = ˆ c = 0 , and in this case the remaining saddle-point equations decouple over the readouts (as expected for independently trained ensemble members) giving: For readout r :

These are equivalent to the saddle-point equations for a single readout given in equation 53, 54 as expected for independently trained readouts. It is physically sensible that c = 0 when J = 0 , because at zero source, there is no term in the replicated system energy function which would distinguish the overlap between two readouts from the same replica from the overlap between two readouts in separate replicas (we expect that the total overlap between readouts is non-zero, as we may still have C > 0 ).

The saddle-point equations obtained by setting the derivatives ∂ g ∂q = ∂ g ∂ ˆ q = ∂ g ∂r = ∂ g ∂ ˆ r = 0 will similarly decouple to recover two copies of the diagonal case 57 55. We will not re-write the expressions here as they are not necessary to determine the off-diagonal error term E rr ′ .

The remaining saddle-point equations are obtained by setting ∂ g ∂c = ∂ g ∂ ˆ c = 0

and for readout r ′ :

Solving equations 115 and 116 for C , we obtain:

We can obtain the generalization error as:

We may then simplify the expression for the E rr ′ error as follows:

Re-labeling the order parameters: ˆ q → ˆ q r , ˆ r → ˆ q r ′ , γ → γ rr ′ and G rr → G r , we obtain the result given in the main text.

G Derivation of Proposition 1 from [31]

In the case where the data and noise covariance matrices Σ s and Σ 0 have bounded spectra, our main result may be derived using Theorem 4.1 from Loureiro et. al. [31], with a few additional arguments to incorporate a readout noise which varies across ensemble members, and to allow for the presence of label noise in the training set but not at test time. Rather than reproducing their very lengthy statements here, we direct the reader to theorem 4.1 and corollary 4.2 in [31].

G.1 Altered Expectations at Evaluation

In [31], labels are generated at both training and test time through the same statistical process y ∼ P 0 y ( y | ν ) where ν = ⟨ x | θ ⟩ √ d . However, their results may be easily extended to the case where data are generated through a different statistical process for the training set and at evaluation. This will allow the application of their results to the case where label noise is present during training but not at evaluation as studied in this work. We therefore introduce separate distributions of the labels y at training and evaluation. During training, we still have y ∼ P 0 y ( y | ν ) . At evaluation, we put y ∼ P g y ( y | ν ) . This leads to the updated formula:

when the expectation is over data-label pairs ( y, x ) at evaluation .

G.2 Rigorous Proof of Proposition 1

We now restate the the problem setup of our main theorem using notation consistent with [31]. We study generalization error in an ensemble of estimators { ˆ w k } , k = 1 , . . . , K . We say ˆ w k ∈ R p for all k = 1 , . . . , K . The weights are trained independently such that each minimizes a ridge loss function:

So that our results will correspond to the results of [31] after re-scaling the regularizations λ k → ν kk λ k . The training labels are drawn as y µ ∼ P 0 y ( y | 1 √ p θ ⊤ x µ ) where P 0 y ( y | x ) = N ( x, ζ 2 ) , ξ µ k ∼ N (0 , η 2 r ) and θ represent the ground truth weights. For k = 1 , . . . , k we have a 'measurement matrix' A k ∈ R N k × d , and we may set d = p ≥ max k N k (so that γ = d p = 1 ). The feature vectors u k ( x ) are then drawn as

where x ∼ N (0 , Σ s ) and σ ∼ N (0 , Σ 0 ) . For convenience, we have defined the auxilary matrices

By constructing the feature vectors as p-dimensional vectors with only N k non-zero components, we may apply the results of [31] while preserving structural heterogeneity (we may have N k = N k ′ for k = k ′ ) as is present in our main result. Because [ u k ] i = 0 for all i > N k , these auxiliary dimensions will not affect model predictions or generalization error. We then have µ k = 1 √ p u ⊤ k ˆ w k , and labels are generated at evaluation according to :

̸

̸

The generalization error may then be decomposed as E g = 1 K 2 ∑ k,k ′ E kk ′ where E kk ′ = E ( y, x ) [( y -µ k )( y -µ k ′ )] . We will apply eq. 121 separately to calculate E rr ′ in the cases where r = r ′ and r = r ′ .

̸

G.2.1 Off-Diagonal Terms

̸

Eq. 121 cannot be used to calculate E rr ′ directly in the case where r = r ′ due to the presence of noises ξ µ k which vary over elements of the ensemble (indexed by k ) in the loss function (eq. 122). We may argue, however, that the presence of readout noise noise has no effect on the expected value of E rr ′ when r = r ′ , then apply eq. 121 with η k = η k ′ = 0 . To see this, we examine the analytical form of the minimizer of the loss function (eq, 122):

̸

where we have defined the design matrices [ U k ] iµ = 1 √ p [ u k ( x µ )] i and the vectors [ y ] µ = y µ and [ ξ k ] µ = ξ µ k . We then have

Taking the expectation value over the readout noise in the training set we get:

̸

This is identical to E kk ′ when ξ k = 0 for all k . We may therefore calculate the off-diagonal error terms by setting ξ k = 0 in eq. 122, which gives a problem compatible with the theorem 4 of [31]. To calculate E rr ′ , we appeal to theorem 4.1 and corollary 4.2 of [31]. The following objects defined in 121 are given the following definitions:

Where ρ , m and Q are defined as in [31]:

What remains is to determine the values of the order parameters ρ , m and Q . We next define the covariance matrices which characterize the feature maps. The feature-feature covariance is:

where we have defined ν kk = N k p , ˜ Σ kk ′ = 1 √ ν kk ν k ′ k ′ A k ( Σ s + Σ 0 ) A ⊤ k ′ , and ¯ Σ kk ′ = √ ν kk ν k ′ k ′ [ ˜ Σ kk ′ 0 0 0 ] ∈ R p × p . Note that while in [31], all Ω kk must have strictly positive eigenvalues, their result can be easily extended to cover the case where some eigenvalues are zero by a continuity argument.

The feature-label covariance is given by:

Recalling ω := Q 1 / 2 ξ , we can now obtain an explicit form for the proximal h :

The proximal G should not arise in this special case.

Next, we will simplify the saddle-point equations. Simplifying where possible, we may write:

We can simplify the prior equation for ˆ V to the following set of equations:

̸

Where we have defined:

̸

Equations 146 and 148 are solved by setting ˆ V kk ′ = V kk ′ = 0 for all k = k ′ so that ˆ V kk ′ = ˆ V k δ kk ′ and V kk ′ = V k δ kk ′ . The diagonal components then satisfy

Which may be solved separately for each k = 1 , . . . , K . Simplifying the remaining channel equations we have:

Simplifying the prior equations we obtain (through some tedious but straightforward algebra):

Solving eq's 154,152 for Q , we obtain:

Combining these results, we arrive at a formula for E kk ′ :

̸

Where the order parameters ˆ V k , k = 1 , . . . , K satisfy the fixed-point equations given by eq's 149, 150.

G.2.2 Diagonal Terms

To calculate E rr , we cannot ignore the presence of readout noise in the loss function. However, as the loss function is separable over the readouts k , we may calculate E rr using the results of [31] in

̸

the special case where K = 1 , incorporating the readout noise into the label noise. Concretely, we may calculate E rr as the error of a single linear predictor under the same setup as the off-diagonal terms except that in the training set y µ ∼ P 0 y ( y | 1 √ p θ ⊤ x µ ) where P 0 y ( y | x ) = N ( x, ζ 2 + η 2 k ) . This may be recovered from the calculation for the off-diagonal terms by setting k = k ′ and re-scaling ζ 2 → ζ 2 + η 2 k , giving:

̸

where the definitions of γ kk , ρ , R k , and Λ kk can be inherited from the off-diagonal case, as well as the saddle-point equations 149, 150.

G.2.3 Full Error

The results obtained here are equivalent to the results of our main theorem, up to a reshuffling of additive constants η 2 k among the error terms, and a trivial re-scaling of the order parameters as follows: V k → 1 V k , ˆ V k → λ k ˆ V k

H Equicorrelated Data Model

To gain an intuition for the joint effects of correlated data, subsampling, ensembling, feature noise, and readout noise, we simplify the formulas for the generalization error in the following special case:

Here s is a parameter which sets the overall scale of the data and c ∈ [0 , 1] tunes the correlation structure in the data and ω 2 sets the scale of an isotropic feature noise. We consider an ensemble of k readouts, each of which sees a subset of the features. Due to the isotropic nature of the equicorrelated data model and the pairwise decomposition of the generalization error, we expect that the generalization error will depend on the partition of features among the readout neurons through only:

The number of features sampled by each readout: N r ≡ ν rr M , for r = 1 , . . . , k

The number of features jointly sampled by each pair of readouts n rr ′ ≡ ν rr ′ M for r, r ′ ∈ { 1 , . . . , k }

Here, we have introduced the subsampling fractions ν rr = N r M and the overlap fractions ν rr ′ = n rr ′ M

We will average the generalization error over readout weights drawn randomly from the space perpendicular to 1 M , with an added spike along the direction of 1 M :

The projection matrix may be written P ⊥ = I M -1 N 1 M 1 ⊤ M . The two components of the ground truth weights will yield independent contributions to the generalization error in the sense that

Calculating E rr and E rr ′ is an exercise in linear algebra which is straightforward but tedious. To assist with the tedious algebra, we wrote a Mathematica package which can handle multiplication, addition, and inversion of matrices of symbolic dimension of the specific form encountered in this problem. This form consists of block matrices, where the blocks may be written as aδ MN I M + b 1 M 1 ⊤ N , where a, b are scalars and δ MN ensures that there is only a diagonal component for square blocks (when M = N ). This package is included as supplemental material to this publication.

H.1 Diagonal Terms and Saddle-Point Equations

Here, we solve for the dominant values of q r and ˆ q r and simplify the expressions for E rr in the case of equicorrelated features described above. In this isotropic setting, E rr , q r , ˆ q r will depend on the subsampling only through N r = ν rr M . We may then write, without loss of generality A r = ( I N r 0 ) ∈ R N r × M where 0 denotes a matrix of all zeros, of the appropriate dimensionality.

We start by simplifying the saddle-point equations 53,54. Expanding 1 M tr ( G -1 r ˜ Σ rr ) and keeping only leading order terms, the saddle-point equations for q r and ˆ q r reduce to:

Defining a ≡ s (1 -c ) + ω 2 and solving this system of equations, we find:

We have selected the solution with q r > 0 because self-overlaps must be at least as large as overlaps between different replicas. This solution to the saddle-point equatios can be applied to each of the k readouts.

Next, we calculate E rr . Expanding γ rr ≡ α Mκ 2 tr [ ( G -1 ˜ Σ ) 2 ] to leading order in M , we find:

With the aid of our custom Mathematica package, we calculate the traces and contractions in these expressions and expand them to leading order in M , finding:

In the 'ridgeless' limit where λ → 0 , we obtain:

H.2 Off-Diagonal Terms

In this section, we calculate the off-diagonal error terms E rr ′ for r = r ′ , again making use of our custom Mathematica package to simplify contractions of block matrices of the prescribed form. By the isotropic nature of the equicorrelated data model, E rr ′ can only depend on the subsampling scheme through ν rr , ν r ′ r ′ , and ν rr ′ . We can thus, without loss of generality, write:

̸

where we have defined n s to be the number of features shared between the readouts, n r = N r -n s and n r ′ = N r ′ -n s and the count of remaining features l = M -n r -n r ′ -n s .

Then, to leading order in M , we find:

Averaging E rr ′ over w ∗ 0 ∼ N (0 , I M ) , we get:

Calculating these contractions and traces and expanding to leading order in M , we get:

Taking λ → 0 we get the ridgeless limit:

̸

where

H.3 Optimal Regularization

Here, we derive the 'locally' optimal regularization which minimizes the prediction error of the ensemble members independently. Equivalently we derive the optimal regularization for a single readout ( k = 1 ) or the regularization which minimizes the diagonal terms E rr of the generalization errror. By differentiating E rr with respect to the regularization λ , one can show that for the optimal regularization, we will have

It is easy to show that dS r dλ cannot be equal to 0 . Setting the term in brackets to zero and solving for λ , we find with the aid of the computer algebra system Mathematica [54]:

In the limiting case of isotropic data ( c = 0 ), the optimal regularization reduces to:

We note that while generalization error is discontinuous at c = 0 , this result can be quickly obtained from eq. 193 by noticing that the formula for the generalization error with 0 < c ≤ 1 reduces to the formula for generalization error with c = 0 when we set ρ = c = 0 (when c = 0 , the generalization error does not depend on ρ because there is no special direction in the data).

H.4 Homogeneous Ensembling with Resource Constraints

̸

We make further simplifications in the special case where λ r = λ , η r = η , ν rr = 1 k for all r = 1 , . . . k . For simplicity, we also set ν rr ′ = 0 for all r = r ′ so that ensemble members sample mutually exclusive subsets of the features. We will consider both the ridgeless limit λ → 0 and the case of 'locally optimal' regularization λ = λ ∗ (see eq. 193). Concretely, we show here that under these special cases, generalization error has the forms given in eqs. 25, 26.

We start by analyzing the saddle-point equations 111, 112. Note that in this special case the saddle point equations will be identical for all r = 1 , . . . , k . We will therefore suppress the r index. Solving this quadratic system of equations explicitly, we encounter the following radical, which we assign variable name x :

In order to simplify this radical to begin extracting the factor of s (1 -c ) that appears in eqs. 25, 26, we define a reduced regularization:

And substitute Λ , H , W , and Z (recall eq. 24) for λ , η , ω , and ζ , giving

making the same substitutions, we can then write

Continuing to make substitutions in this manner, we arrve at:

Finally, we may express the generalization error in a reasonably compact form. In the special case at hand, the generalization error is written:

Substituting 201 for S in eq. 16, and making further substitutions as necessary, we arrive at:

where

We also obtain where

Combining these, we obtain the error of the ensemble as:

These result are derived in the Mathematica notebook titled 'EquiCorrParameterReduction.nb' included with the available code. It follows from eq. 211 that when λ = 0 error can be written in the form of eq. 22. It follows from equations 211, 209 that at 'locally optimal' regularization λ ∗ , error can be written in the form of eq. 23. To see this, note that the reduced regularization Λ ∗ which minimizes E rr will only depend on the other arguments of E rr . Full expresions for the generalization error in the case λ → 0 and λ = λ ∗ can be found in the mathematica notebooks 'EquiCorrPhaseDiagram_ZeroReg.nb' and 'EquiCorrPhaseDiagram_LocalReg.nb' included with the available code. These equations are long and difficult to interperet - nor are they directly used in our code - and so are omitted here.

H.4.1 The Intermediate to Noise-Dominated Transition

The transition between the intermediate regime where 1 < k ∗ < ∞ and the noise-dominated regime where k ∗ = ∞ can be studied analytically relatively painlessly in the ridgeless limit λ → 0 . The strategy we employ to determine this phase boundary is to examine the largek asymptotic expansion of E k to determine whether the error approaches its asymptotic value from below or above. If E k approaches E ∞ from below, then k ∗ must be finite. If E k approaches E ∞ from above, then k ∗ may be infinite - however, there is still the possibility of k ∗ < ∞ if E k is non-monotonic in k . In practice, we check the values of E k for k = 1 , 2 , . . . , 100 and k = ∞ .

Setting Λ = 0 and expanding E k around k = ∞ , we find:

Setting the coefficient of k -1 to zero, we find the phase boundary as:

This equation explains the shapes of the boundaries between the intermediate and noise-dominated regions in the phase diagrams of fig. 4 with λ = 0 (see black dotted lines in panels b, c, d).

where:

An analytical formula for the boundary between the intermediate and noise-dominated regimes at locally optimal regularization λ = λ ∗ cannot be easily obtained. To understand why, we can asses the largek asymptotic expansion of the generalization error at locally optimal regularization:

This shows that at large k , when H > 0 , error always approaches its asymptotic value from above (recall that when H = 0 we always have k ∗ = 1 , so that there is no phase boundary unless H > 0 ). Thus, determining the phase boundary requires determining the value of k which minimizes E k , which is not analytically tractable.

H.5 Infinite Data Limit

In this section we consider the behavior of generalization error in the equicorrelated data model as α → ∞ while keeping the λ ∼ O (1) . For simplicity, we assume ν rr ′ = 0 for r = r ′ , isotropic features ( c = 0 ), no feature noise ( ω = 0 ) and uniform readout noise η r = η as in main text Fig. 3. This limit corresponds to data-rich learning, where the number of training examples is large relative to the number of model parameters. In this case, the saddle point equations reduce to:

̸

In this limit, we find that γ rr ′ → 0 . Using this, we can simplify the generalization error as follows:

Interestingly, we find that the readout error in this case depends on the subsampling fractions ν rr only through their mean. Therefore, with infinite data, there will be no distinction between homogeneous and heterogeneous subsampling.

I Theoretical Learning Curves and Optimal Subsampling Phase Diagrams

Here, we provide additional learning curves and phase diagrams of k ∗ such as those in Fig. 4a,b,c,d, exploring more parameter values for the task-model alignment ρ and the Reduced noises H , W , and Z . Generalization errors are calculated for a homogeneous ensemble of k linear regressors, as described in sections 5, H.4. We also show diagrams of generalization error E k ∗ .

a

b

p=0,A=0

P=0.3,A=0

P=0.6,A=0

0.0

1.0

2.0

3.0 ∞

α

p=0.6,入=入

米

V=v'0=d

P=0.3,A=A

p=0,A=0

P=0.3，A=

P=0.6,A=0

0.0

1.0

2.0

3.0 ∞

α

p=0.6,入=入

P-O,A一A

P-0.3,A-A

a

b

p=0,A=0

P=0.3，A=0

p=0.6,A=

0.0

1.0

2.0

3.0 ∞

α

p=0.6,入=入

Q

p=0,A=A

P=0.3，A=A

P=0,A=0

P=0.3，A=0

p=0.6,A=0

0.0

1.0

2.0

3.0 ∞

α

p=0.6,入=入

P-O,X-A

P-0.3,A-A

a

b

p=0.6,A=0

0.0

1.0

2.0

3.0∞

α

p=0.6,入=入

P-0,>-0

P-0.3,A-0

0

0

P-O,A-A

P=0，A=0

p=0.3，A=

p=0.6,A=0

0.0

1.0

2.0

3.0

α

p=0.6,入=入

P=O，A=X

P=0.3，A=A

-1

100

α

0.6,P

=

-1

100

α

0.6, 1

二

10-1

10-1

100

101

10-210-1

10°

101

10-2

α

α

p=0,H=1.0

入 =0, p=0.3, H=1.0

入=0,p

101

101

100

100

m.

10-1

10-1

10-1

100

101

10-210-1

100

101

10-2

α

α

p=0,H=2.0

入 = 0, p=0.3,H= 2.0

入=0,p

101

101

TTT

..

10-1

TTTm

lm....

TI

10-1

100

101

10-210-1

100

101

10-2

1

α

α

p=0, H=1.0

入=A*, p=0.3, H=1.0 =A*,

100

100

m..

10-1

m

10-1

10-1

100

101

10-210-1

100

101

10-2

1

α

α

p=0,H=2.0

入=A*,=0.3,H=2.0 入=A*,

100

100

TTW

-1

100

α

0.6,

二

l

-1

100

α

0.6,

二

..

10-1

10-1

10°

101

10-2

10-1

10°

101

10-2

α

α

p=0, W=1.0

入=0, p=0.3,W=1.0 入=0,β

101

101

100

100

10-1

10-1

10°

101

10-2

10-1

100

101

10-2

α

α

p=0, W=2.0

入=0, p=0.3,W=2.0 =0,β

101

101

T11

-1

10

α

0.6,

-1

10

α

0.6,

TTTW

..

10-1

TTTT

10-1

100

101

10-2

10-1

100

101

10-2

1

α

α

p=0, W=1.0 入=A*, p=0.3, W=1.0入=)*,p

100

100

10-1

10-1

10-1

100

101

10-210-1

100

101

10-2

1

α

α

'*=0=M'0='*= 0=M'0=

100

100

m..

-1

100

α

0.6,

二

1

100

α

0.6,

二

10-1

10-1100

101

10-2

10-1

10°

101

10-2

α

α

p=0, Z=1.0

入 =0, p= 0.3, Z=1.0

入=0,

101

101

100

100

10-1

10-1

10°

101

10-2

10-1

100

101

10-2

α

α

p=0, Z=2.0

入=0,p=0.3,Z=2.0

入=0,

101

101

-1

10

α

0.6,

-1

10

α

0.6,

m..

10-1

10-1

100

101

10-210-1

100

101

10-2

1

α

α

p=0, Z=1.0

入=*,p=0.3,Z=1.0 =*,

100

10°

10-1

10-1

10-1

100

101

10-210-1

100

101

10-2

1

α

α

0 =0, Z=2.0

入=*,p=0.3,Z=2.0 =*,

100

100

TW

1

100

α

0.6,

二

TTIII

-1

100

α

0.6,

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide a theoretical analysis of the case of ensemble learning with linear ridge regression for the case where heterogeneous feature subsampling is used. The authors make a number simplifying assumptions about the distribution of the data (Gaussian distribution and noise, linear function), and using the replica trick from statistical physics derive generalization error for the L2-regularized least-squares solution. In simulations, the authors demonstrate that the derived solution coincides with numerically calculated generalization error, and based on analyzing learning curves provide a number of insights about the behavior of ensembled ridge regression, suggesting a novel way for mitigating the double descent phenomenon through the use of heterogenous feature subsampling in ensembling.\n\nSTRENGTHS:\nThe work provides novel insights about how double-descent behaves based on noise, regularization, and subsampling and provides alternative strategies to regularization for avoiding the peak. These are quite central questions in modern machine learning theory and new ideas here can prove to be significant. The mathematical analysis seems rigorous and the replica trick used for the analysis appears to be a new type of tool for studying this phenomenon, though I do not have suitable mathematical background to follow all the proofs in detail. The basic setting and assumptions are clearly defined, related work seems to be covered well, and novelty of the contribution compared to related similar works is clearly established.\n\nWEAKNESSES:\nThe work makes many simplifying assumptions that are unlikely to match most real-world learning problems (e.g. Gaussian data, linear teacher function), and analyzes may not be in practice tractable beyond the simpler cases such as analyzed in Section 2.3. with globally correlated data. It is not so clear whether the results would offer practical tools for an analyst wishing to apply heterogeneous feature-subsampled ridge ensembles in an optimal way, though results in the supplementary materials about CIFAR10 Classification Task suggest the results can have also practical relevance.\n\nQUESTIONS:\nHow computationally demanding is the computation of the generalization error curves?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide theoretical results on generalization for ridge regression for the case of an ensemble of regression models trained on feature subsets, with noise, and correlation.\n\nThey relate this to the previously observed double-descent phenomena and characterize how the fraction of subsampled features, the regularization strength, and other properties affect the double descent phenomenon and where it occurs (i.e., how these impact generalization error).  \n\nFinding that double descent occurs at different places for different feature subset amounts, they also propose and analyze using a heterogenous ensemble of different feature subsampling sizes as a way to mitigate double descent without having to tune regularization or model architectures.\n\n\n****Update****\nRevised my score after the responses - see my response comment for rationale.\n\nSTRENGTHS:\nThe generalization bounds for the particular setting seem novel.  \n\nThe problem is a highly relevant one and the idea of using feature subset ensembles is interesting and potentially useful- as alluded to in the conclusion - as searching over many architecture and regularization settings for large neural nets may be prohibitive, so this kind of approach could provide an alternative for such cases with a fixed architecture, by creating a feature subsample ensemble instead - however this possibility is just alluded to and not tested, as only theoretical results for simple ridge regression problems are analyzed.\n\nThe authors provide thorough analyses of the results of the theorem for generalization error, to try to elucidate what different aspects of the ridge regression problem and choices for the modeling mean in terms of generalization error.\n\nWEAKNESSES:\n1) The paper is hard to follow and extract any useful meaning or conclusions from.\n\na) Some terms are not defined or given adequate explanation.  As particular example - it would be helpful to explain what \"readout noise\" and readout dimensionality mean in this paper when first introduced - this was not clear to me and I found its use to be confusing in the theoretical setup section and even the related work.  It's just mentioned without defining what it is or how it related to the model, and it would help to give an example of what it would correspond to.\n\nb) A lot of different parameters are introduced and hard to remember when trying to connect to the various results figures and analyses -  it may help to remind the reader and also include simple examples of models describing what the different parameters and values would correspond to.\n\nc) I feel the main results and conclusion need a simpler explanation and deciphering for the reader - as the various plots and analyses are still based on a large number of symbols / parameters that it's hard to remember, and the reader would have to dig through the text each time to recall what they are.  Some simplified example and conclusion, as well as a table may help.  It's hard to come away with any kind of recommendation or firm conclusion of what we could expect to work better in practice as well.\n\nd) \n\n2) It's not clear how useful and impactful the results from the paper and the derived theorems are.  \nIt seems the complicated expression derived is too complex to derive any useful properties, and requires knowing ground truth properties - so the benefit is not clear.\n\nEven in the simplified tractable case - it's not clear what interpretation comes out of it.\n\nIn generally it's not clear if any novel and useful findings have come out of this.\n\nI think a key part of the issue might be - after presenting each set of results / figures, some interpretations and conclusions and take-ways for the reader are needed and currently lacking - results are just presented with little explanation or analysis.\n\n\n3) Related to the previous points, I feel having real data and model experiments and results could really strengthen this work and also more fully explore the idea and benefit proposed of using heterogenous feature sub-sampling ensembles.  \n\n\n4) Novelty is also somewhat unclear.\nObviously, heterogenous feature subsampling ensembles have been studied and applied in the past, and for some specific models there are theoretical results on generalization error as well, such as tree ensembles or even linear model ensembles with different kinds of approaches such as random subspace models.  \nAdditionally, the authors point to some specific related work as well on theoretical study of generalization error as well.  However, clearly spelling out the differences in this work seems lacking.  For example, in related work - they mention other work also studying feature subsampling for ensembles and finding it provides an implicit regularization approach, in particular [20].  However, it's not stated what differentiates this work from this prior work - what novel aspect does this provide?  It's necessary to spell out what limitations of previous work this addresses and overcomes.\n\nI.e., overall it would help to clearly spell out how their work is different and novel with respect to prior work on feature-subsampling ensembles and what new conclusions and approaches result from it.\n\nQUESTIONS:\n\"...correlated data by ensembles with heterogeneous readout dimensionality.\"\nThis is not very clear - what is meant by \"readout dimensionality\"?  I'm assuming\nit means heterogeneous input data dimensionality - so why not just say that?  \"Readout dimensionality\" is a term I've almost never heard and sounds like it mean the dimension of the model output, which doesn't make sense in this case.\n\n\nIt would be helpful to explain what \"readout noise\" means in this paper when first introduced - this was not clear to me and I found its use to be confusing in the theoretical setup section.  It's just mentioned without defining what it is or how it related to the model, and it would help to give an example of what it would correspond to.  After several re-reads I concluded it probably corresponds to noisy model outputs during training when using something like dropout.  However, for the most cases in practice for regression, even if noise is used in training, when applying the models in practice deterministic outputs are used (no longer any \"readout noise\").\n\n\n2.6 is hard to follow - again what is mean by each type of \"readout\"?  What is a \"single fully-connected readout\" and what does it mean by \"multiple sparsely connected readouts\"?  I'm assuming the latter is the feature subsampling ensemble, and the former a single model with no feature-subsampling, but this is described in a confusing way so it's not clear and took some though to try to decipher this.\nWhat is mean by correlation code and correlation strength?  How are these defined, and how are they used to generate the models / how are they used in these analyses and results?\n\nAlso how do these conclusions differ from past studies of ensembling approaches?  \n\n\n\n\nWhy in all the examples is P <= M?  It would be nice to explain this as well, and also what happens when P > M.  I.e., after some thought I assume this is to focus on the over-parameterized regime, but this is never really stated and explained, and there are still many practical cases where P >> M.\n\nWhat impact does k have on generalization error?\n\nAlso wondering how the results compare to the non-ensemble case?  I.e., a single model with no feature subset sampling.\nI.e., this shifts the problem to tuning the subset sample fraction sizes\n\n\nAlso it's not clear to me from the results and figures if tuning lambda (regularization) can always result in better performance then using the heterogenous ensemble?\n\nLIMITATIONS:\nYes - doesn't seem needed in this case.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a theoretical investigation into ensembling methods applied to linear ridge regression with feature subsampling. It builds upon previous research in this area by extending the analysis to include scenarios with varying readout dimensionality.\n\nBy employing the replica method, the authors derive expressions for the average-case generalization error. These expressions capture the influence of both the data structure and the hyperparameters of the ensembling method employed.\n\nFurthermore, the paper focuses on a simplified scenario to examine how the degree of subsampling and heterogeneity affect the occurrence of the double descent phenomena in learning curves for generalization error. This analysis provides potential insights into the understanding of this well-known and recurring phenomenon in machine learning.\n\nSTRENGTHS:\n- The paper is clearly written, and the experiments presented are of high quality.\n\nWEAKNESSES:\n- The authors use as proof technique for the main theorem the replica method, a heuristic method coming from statistical physics. While this method has been proven to be rigorous in certain contexts, its general correctness has not yet been formally established.  Unfortunately, the paper fails to acknowledge this limitation, which undermines the overall credibility of the research findings.\n- In terms of technical results, the computation leading to preposition 1 appears to be a minor modification of the calculation presented in a previous work by Loureiro [2022]. However, unlike the present study, Loureiro's work accompanies the calculation with a rigorous proof. This discrepancy raises concerns about the rigour and robustness of the current paper's technical contributions.\n\n(Minor) Typos:\n- Line 4: the the\n- Line 53: studied studied\n- Line 212: eta\n\nQUESTIONS:\n- Would the authors consider revising the presentation of their results from Proposition 1 to reflect the heuristic nature of the findings? This would involve explicitly stating that the results are derived using a heuristic method, thereby providing clarity to readers about the level of mathematical rigour employed in the study. Such a revision would enhance transparency and enable readers to appropriately interpret the results.\n\nLIMITATIONS:\nWhile the authors do acknowledge certain limitations in their work and propose potential new directions for applying the techniques employed, there are a couple of crucial aspects that remain unaddressed. Firstly, the paper fails to explicitly mention that the replica method utilized lacks mathematical rigour, which is an important consideration for interpreting the validity and reliability of the results. Additionally, the authors do not provide any indications or suggestions on how to potentially establish a more rigorous foundation for their findings.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This article characterizes the asymptotic performance curve of a heterogeneous feature ensembling framework for ridge linear regression, in the limit of comparably large numbers of data samples and variables. For having different error peaks in a double-decent performance curve as a function of the data sample size, predictors built of heterogeneous feature sets give rise to an ensemble that is robust to the number of data samples, without a carefully tuned regularization parameter.

STRENGTHS:
* Clarity of the mathematical setup.
* Numerical validation of asymptotic results on finite data sets.
* Discussion of the theoretical implications of the analysis.

WEAKNESSES:
* The analysis applies only to the square loss while recent related work addressed already a large family of convex losses.
* Even though the interest of heterogeneous feature ensembling is explained as mitigating the double-descent phenomenon, this problem can be in fact settled by optimizing the regularization parameter on a cross-validation set.
* There is no experiment on real data sets to see how well the observed consequences such as the mitigated double-descent curve apply to real data.

QUESTIONS:
I wonder if the optimal learning performance is achieved at a uniform subsampling rate (i.e., homogeneous ensembles); and whether the feature ensembling can indeed improve the performance without readout noise, as is the case with readout noise (and under certain conditions on the data model parameters) according to the discussion of Section 2.6.

LIMITATIONS:
The limitations are discussed. There does not seem to be any negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an asymptotic analysis of ensembles of ridge regressors using varying numbers of subsampled features. The authors consider a Gaussian data model with feature noise and readout noise in addition to label noise. Using the replica method from statistical physics, they obtain precise asymptotics of the limiting ensemble risk for any collection of subsampling operators. They specialize this result in the case of globally correlated isotropic data and numerically plot double descent curves for various non-overlapping ensemble strategies. They find that in poorly regularized regimes, increased heterogeneity of feature subset sizes leads to a significant decrease of worst-case generalization risk, in other words dampening the double descent peak, albeit typically at the expense of increased risk in other data aspect ratios. Departing from heterogeneity to consider instead homogeneous non-overlapping ensembles, the authors further demonstrate that in the presence of readout noise, there are regimes in the phase space of readout noise level and global feature correlation in which it is optimal to use an ensemble rather than a single learner, although without readout noise it is never optimal to ensemble.\n\nSTRENGTHS:\nThe biggest strength of this paper in comparison to existing work on linear ensembles of random projections of features is that this analysis applies to arbitrary projection operators that need not be independent. In fact, for the specific cases the authors consider of non-overlapping feature subsets, which would correspond closely with distributed optimization practice (I recommend the authors emphasize this in their introduction), the projection operators are clearly very much not independent. Furthermore, the most general form of their results in Proposition 1 also makes no assumption on the relationship between the projection operators and the covariances, in contrast with most existing work which assumes them to also be independent.\n\nI am also not aware of any other work which explicitly considers heterogeneous linear ensembles, which is a novel contribution.\n\nWEAKNESSES:\nI think that the biggest weakness of this paper is the justification of the data model and of the advantage of using a (heterogenous) ensemble. I think that these could be improved by adding justifying and motivating comments throughout the paper, and if the authors would indicate how they would make such changes for the camera ready version, I would increase my score of the paper.\n\nData model:\n- The features are corrupted with additional noise that is independent of the label, which is an interesting but highly non-standard assumption. Even more non-standard is the assumption that this noise is drawn independently in every forward pass. This assumption really should be motivated---should the reader be imagining this as something like noise due to dropout in a neural network?\n- The model outputs are corrupted with readout noise independent across each member of the ensemble, again drawn independently each forward pass. This seems like an even stranger assumption to me; I could imagine some negligible computational noise, but I struggle to come up with a setting in which we would see the noise levels that are modeled in this paper. It is very important to motivate this, as the experiment in Figure 4 only justifies ensembles in settings where readout noise is high. Without good motivation, it seems like a strawman.\n\nEnsemble advantages:\n- This is a complaint I have for many other papers that deal with double descent, but applies equally to this paper. The \"correct\" solution to double descent is to properly regularize, and other strategies such as the ensembling proposed here can be seen as \"hacks\" that can mitigate some of the worst double descent effects but are typically suboptimal. We see this in this case as well. E.g., consider Figure 3, where the heterogenous ensembles only improve risk in the under-regularized setting. In the setting with more appropriate regularization (c.iv-vi), where the risk is lower than in (c.i-iii) for most values of $\\alpha$, the heterogeneous ensemble is strictly worse than the homogeneous ensemble, which I am sure itself is strictly worse than the optimally tuned full ridge regressor. \n- No comparisons are made to the full ridge regressor. Obviously, the ensemble would have to be worse, as ridge regression is typically the minimum mean squared error estimator in additive noise settings. However, this is a lost opportunity to demonstrate perhaps that bad double descent effects could be mitigated at a fraction of the computational cost (since the ensembles consider small subsets).\n- Essentially, the risks are a linear combination of the individual risks of each ensemble member, plus some cross-terms. Thus in the limit of zero regularization, these ensembles would still have infinte double descent peaks, one for each different subsampling ratio. So to an extent, the authors already rely on regularization to help make the ensemble robust to double descent, although the benefits are realized for fairly small regularization already. Still, the authors should justify the need to regularize even with ensembles and discuss tuning.\n- Related to the previous points, it's not clear why worst case risk is an important metric, since tuning of regularization in some way or another seems necessary.\n- Part of the issues of statistical performance may be due to the non-overlapping partition of features. In the related works, ensembles with optimality guarantees typically require many members with overlapping subsets of features, which the authors do not consider here. \n\nOthers:\nOverall, I found the plots to be very busy and difficult to decipher. I give one suggestion on how to improve Figure 4 in Questions.\n\nQUESTIONS:\nInstead of the replica method, do the authors think that the same result could be obtained by the more rigorous \"deterministic equivalences\" from random matrix theory [DS21]? Related work has recently successfully applied this technique to random projections, which should be directly applicable to ensembles [LPJBT22], which seems closely related to the future direction raised by the authors at the end of the paper.\n\nIn Figure 4.a.i, I cannot understand the meaning of the white dashed line. I thought it should indicate the boundary between when $k^* = 1$ and $k^* = 2$, but I cannot figure out why there is a purple region above the line, and why the caption says \"there is no region of the phase\nspace where $k^* = 2$.\" Another suggestion is to add markers in the (a) plots to indicate the points in phase space that correspond to the (b) plots.\n\n[DS21] E. Dobriban and Y. Sheng, Distributed linear regression by averaging, The Annals of Statistics, 49\n(2021), pp. 918 – 943, https://doi.org/10.1214/20-AOS1984.\n\n[LPJBT22] D. LeJeune, P. Patil, H. Javadi, R. G. Baraniuk, R. J. Tibshirani, “Asymptotics of the sketched pseudoinverse,” 2022, https://arxiv.org/abs/2211.03751\n\nOther remarks:\n- 1: \"Bagging\" is conventionally short for \"bootstrap aggregating\", but you only subsample, which is often called \"subbagging\" in the literature.\n- 2: Ensembles do reduce variance, but subsampling feature adds bias.\n- 4: \"the the\"\n- 212: \"eta\" should be $\\eta$\n\nLIMITATIONS:\nWhile I understand the usefulness of the replica method for obtaining quick and often accurate results, it is unfortunate that it is a non-rigorous approach. The authors have mentioned this in the supplemental material, but I think it needs mention somewhere in the main paper. This is important, as it is not clear whether some of the things done here are \"legal\"---for example, many random matrix theory results which could be applied here to perform similar analysis require covariance matrices with uniformly bounded operator norm, yet the globally correlated model violates this, which would typically require additional care to ensure that the result should hold, rather than simply plugging in the covariance. The reader deserves to know that while the general intuitions are likely true to hold, they should be careful in blindly applying the results.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-13"}