Theoretical Limitations of Ensembles in the Age of Overparameterization
Authors: Niclas Dern
John Patrick Cunningham
Geoff Pleiss
Venue: ICML 2025 oral (2025)
Keywords: Ensembles
Deep Ensembles
Uncertainty Quantification
Overparameterization
Random feature regression
Kernel regression
Primary area: theory->learning_theory
TL;DR: We theoretically characterize the generalization and uncertainty properties of overparameterized random feature regressors, proving a functional equivalence between ensembles and single (but larger) models under weak assumptions.

Abstract: Classic ensembles generalize better than any single component model. In contrast, recent empirical studies find that modern ensembles of (overparameterized) neural networks may not provide any inherent generalization advantage over single but larger neural networks. This paper clarifies how modern overparameterized ensembles differ from their classic underparameterized counterparts, using ensembles of random feature (RF) regressors as a basis for developing theory. In contrast to the underparameterized regime, where ensembling typically induces regularization and increases generalization, we prove with minimal assumptions that infinite ensembles of overparameterized RF regressors become pointwise equivalent to (single) infinite-width RF regressors, and finite width ensembles rapidly converge to single models with the same parameter budget. These results, which are exact for ridgeless models and approximate for small ridge penalties, imply that overparameterized ensembles and single large models exhibit nearly identical generalization. We further characterize the predictive variance amongst ensemble members, demonstrating that it quantifies the expected effects of increasing capacity rather than capturing any conventional notion of uncertainty. Our results challenge common assumptions about the advantages of ensembles in overparameterized settings, prompting a reconsideration of how well intuitions from underparameterized ensembles transfer to deep ensembles and the overparameterized regime.

Theoretical Limitations of Ensembles in the Age of Overparameterization
Niclas Dern" John P. Cunningham Geoff Pleiss
Abstract
1. Introduction
Historically, most machine learning ensembles aggregated component models that are simple by today's standards (e.g. Hansen & Salamon, 1990; Opitz & Maclin, 1999; Diet- terich, 2000). Common techniques like bagging (Breiman, 1996). feature selection (Breiman, 2001), random projec- tions (Kabán, 2014; Thanei et al., 2017), and boosting (Fre- und, 1995; Chen & Guestrin, 2016) were developed and analyzed assuming decision trees, least-squares regressors, and other underparumeterriedd component models incapable of achieving zero training error.
Classic ensembles generalize better than any sin- gle component model. In contrast, recent empiri- cal studies find that modern ensembles of (overpa- rameterized) neural networks may not provide any inherent generalization advantage over single but larger neural networks. This paper clarifies how modern overparameterized ensembles differ from their classic underparameterized counterparts, us- ing ensembles of random feature (RF) regressors as a basis for developing theory. In contrast to the underparameterized regime, where ensembling typically induces regularization and increases gen- eralization, we prove with minimal assumptions that infinite ensembles of overparameterized RF regressors become pointwise equivalent to (sin- gle) -nfinite-width RF regressors, and finite width ensembles rapidly converge to single models with the same parameter budget. These results, which are exact for ridgeless models and approximate for small ridge penalties, imply that overparame- terized ensembles and single large models exhibit nearly identical generalization. We further charac- terize the predictive variance amongst ensemble members, demonstrating that quantifies the ex- pected effects of increasing capacity rather than capturing any conventional notion of uncertainty. Our results challenge common assumptions about the advantages of ensembles in overparameter- ized settings, prompting a reconsideration of how well intuitions from underparameterized ensem- bles transfer to deep ensembles and the overpa- rameterized regime.
Researchers and practitioners have now turned to ensem- bles of overparameterized models, such as neural networks, which have capacity to memorize entire training datasets. Motivated by heuristics from classic ensembles (Mentch Hooker, 2016), some have argued that ensembles provide robustness to dataset shift (Lee al., 2015; Fort et al., 2019) and that the predictive variance amongst component models in these so-called deep ensembles is notion of uncertainty that can be used on downstream decision maasiinn-maiing tasks (Lak- shminarayanan et al., 2017; Gal et al., 2017: Gustafsson et al., 2020; Ovadia er al., 2019; Yu et al., 2020).
While few theoretical works analyze modern overparame- terized ensembles, recent empirical evidence suggests that intuitions from their underparameterized counterparts do not hold in this new regime. For example, classic meth- ods to increase diversity amongst component models, such as bagging, can be harmful for deep ensembles (Nixon et al., 2020; Jeffares et al., 2024; Abe et al., 2022a; 2024; Webb et al., 2021) despite being nearly universally benefi- cial for underparameterizee ensembles. Moreover, while established underparameterizee ensembling techniques of- fer well- well-founded quantifications of uncertainty (e.g. Mentch & Hooker, 2016; Wager et al., 2014), several recent studies question the reliability of the uncertainty estimates from deep ensembles (Abe et al., 2022b; Theisen et al., 2024; Chen et al., 2024).
"Work done while at the Vector Institute. 'School of Com- putation, Information and Technology, Technical University of Munich, Munich, Germany Department of Statistics. Columbia University, Zuckerman Institute, New York, USA 'Department of Statistics, University of British Columbia, Vancouver, Canada "Vector Institute, Toronto, Canada. Correspondence to: Niclas Dem <niclas.dern derisddemmmmmi..mmm.mm.. .mai..com>..
To address this divergence and verify recent empirical find- ings, we develop theoretical characterization of ensembles in the overparameterized regime, with the goal of contrast- ing against (traditional) underparameterized ensembles. We answer the following questions:
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
1. Do ensembles of overparametterized models provide generalization or robustness benefits over a single (very large) model trained on the same data? Does the capac- ity of the component models affect this difference?
(1996) and Williams (1996). Building on these ideas, ran- dom feature (RF) models were later introduced as scalable approximation to kernel machines (Rahimi & Recht, 2007; 2008a;b). RF regressors have seen growing theoretical in- terest as simplified models of neural networks (e.g. Belkin et al., 2018; 2019; Jacot et al., 2018; Bartlett et al., 2020; Mei & Montanari, 2022; Simon et al., 2024). Random fea- ture models can be interpreted as neural networks where only the last layer is trained (e.g. Rudi & Rosasco, 2017; Belkin et al., 2019) or as first-order Taylor approximations of neural networks (e.g. Jacot et al., 2018).
2. What does the predictive variance of overparameter- ized ensembles measure, and does it relate to classic frequentist or Bayesian notions of uncertainty?
To answer these questions, we analyze ensembles of over- parameterized random feature (RF) linear regressors, a theoretically-tractabbl approximation of neural networks. Unlike prior work on RF models, our analysis makes very few assumptions about the distribution of random features, which we will show-is crucial for highlighting the differences between ensemble variance versus more estab- lished notions of uncertainty. Our analysis focuses on the practically relevant regime where RF models are trained with little to no regularization. We verify and contextualize our theory with experiments on RF and neural networks ensembles.
Underparameterized random feature models and en- sembles. In this paragraph, we restrict our discussion to analyses of (ensembles of) underparameterizee RF regres- sors, where the number of random features (i.e., the width) is assumed to be far fewer than the number of data points. In the fixed design setting, infinite ensembles of unregular- ized RF regressors achieve the same generalization error as ridge regression on the original (umprojected) inputs (Kabán, 2014; Thanei et al., 2017; Bach, 2024b). We provide theoret- ical analysis in Appx. E that further demonstrates ridge-like behaviour of underpameterized RF ensembles.
1.1. Related Work
Deep ensembles. A primary motivation of this paper is to understand recent empirical findings about uncertainty quan- tification afforded by deep ensembles (Lakshminarayanan et al., 2017). Historically, variance amongst deep ensemble members has been proxy for epistemic uncertainty (c.g. Kendall & Gal, 2017; Gustafsson et al., 2020), i.e., the uncer- tainty that can be reduced by collecting more data. This view reflects a classical intuition of ensembles: ignoring effects of overparameterrization and inductive bias, all ensemble mem- bers should converge to the same prediction in the infinite data limit, and thus differing predictions suggest a region of the input space with insufficient data. However, recent empirical findings challenge this interpretation of ensemble variance (Abe et al., 2022b; Theisen et al., 2024). Most rel- evant to our work, Abe et al. (2022b) demonstrate strong correlation between ensemble variance and the expected improvement that results from increasing model capacity. Specifically, across numerous architectures and datasets, they demonstrate strong point-wise correlation between the predictions of an ensemble (e.g.. 4 ResNet-18s) and sin- gle larger model (c.g.. a WideResNeet15 18 with 4x the width) on both in-distribution and of-distribuutinn data. The authors conclude that ensemble variance is more reflective of sensitivity to model capacity rather than data availability, finding with significant implications for decision- making and robustness. We theoretically verify these findings in ensembles of overparameterized random feature models.
Overparameterized random feature models. Recent works on RF models have focused on the overparameterized regime, often using sigh-dimensional asymptotics to char- acterize generalization error (Adlam & Pennington, 2020; Bach, 2024b; Hastie et al., 2022; Loureiro et al., 2022; Mei & Montanari, 2022; Ruben et al., 2024). Many works rely on results derived assuming that the the marginal distribu- tions over the random features can be replaced by moment- matched Gaussians. While such approximations are well- founded for asymptotic results (c.g. Goldt et al., 2022; Hu & Lu, 2022; Montanari & Saeed, 2022; Tao, 2012), we argue that they may be harmful specifically for an analysis which aims to characterize the uncertainty properties of ensemble variance. Assuming Gaussianity results in an ensemble vari- ance that is proportional to the predictive variance of Gaus- sian process regression, often held as a gold standard for uncertainty quantification (Rasmussen & Williams, 2006; Lee et al., 2018; 2020; Ovadia et al., 2019). In contrast, our non-Gaussian analysis yields a characterization of en- semble variance that differs from this conventional notion of uncertainty, closely matching recent empirical studies of ensemble variance (Abe et al., 2022b; Theisen et al., 2024).
The benefits of overparameetrizzation and ensembling for out-of-distribution gemeralization in random feature mod- els have been analyzed by Hao et al. (2024), who provide lower bounds on OOD risk improvements when increasing capacity or using ensembles. Their work focuses on non- asymptotic guarantees under specific distributional shifts, while ours examines the equivalence of ensembles and sin-
Random feature models. The connection between in- finitely wide neural networks and kernel methods, partic- ularly Gaussian processes, was pioneered by Neal & Neal
2

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
1.a
1.5
RF Models
Ensemzilee Model
Kernel Model
Kernel Model
1.0
Training Data
1.0
Training Data
0.5
0.0
-0.5
1,0
-4
-2
-2
Figure 1: An infinite ensemble of overparameterized RF models is equivalent to a single infinite-width RF model. (Left) We show a sample of 100 finite-widti RF models (blue) with ReLU activations trained on the same N = 6 data points. Additionally, we show the single infinite-width RF model (pink). The finite-width predictions concentrate around the infinite-width model. (Right) We again show the single infinite-width RF model (pink) and the "infinite" ensemble of M = 10, 000 RF models (blue). We note no perceptible difference between the two in this setting, though extreme numerical conditions can break this equivalence (cf. Fig. 8).
gle large models under minimal assumptions. Coneurrent work by Ruben et al. (2024) also finds RF ensembles of- fer little advantage over larger single models, though their analysis uses optimal ridge tuning and Gaussian universality assumptions. Most related to our work is Jacot et al. (2020), who analyze the pointwise expectation and variance of ridge- regularized RF models with Gaussian process (GP) features, leveraging Gaussianity to simplify their analysis. We go beyond this prior work by significantly weakening the sumptions on the distribution of random features, enabling us to characterize differences between ensembles versus Gaussian models with respect to uncertainty and robustness properties. Moreover, we provide a finite-sample analy- sis as well as characterization of the transition from the ridgeless to ridge-regularized regimes, which the best of our knowledge are novel results for overparameterized RF ensembles.
equivalent single infinite-wide width RF model (ef. Fig. 1). We further show that this equivalence approximately holds in the small ridge regime (Theorem 3.5). Moreover, we extend these results to finite parameter budget, showing that the functional difference between the parameters of a larger single model and finite ensemble, each with the same total number of parameters, small with high prob- ability (see Sec. 3.2). We validate these theoretical results with supporting experiments on RF and neural network en- sembles, using synthetic data and the California Housing dataset (Kelley Pace & Barry, 1997) with various activation functions (detailed in Appx. A.1 and Appx. B).
To answer Question 2: we show that the predictive variance in an overparamettrrized ensemble generally does have frequentist or Bayesian interpretation, unlike uncertainty quantifications obtained from Gaussian processes. Instead, we find that the variance measures the expected squared difference between the predictions from a (finite-width) RF regressor and its corresponding kernel regressor (i.e., the infinite- width model) (see Sec. 3.3). Crucially, this finding relies on our non- -aaassan analysis of RF models.
1.2. Contributions
We consider ensembles of overparameterized RF regressors in both the ridgeless and small ridge regimes. Unlike prior work, we make minimal assumptions about the distribution of the random features and so our results are not restricted to high-dimensional asymptotics where Gaussian univer- sality might typically apply. Our results thus distinguish differences between RF ensembles and more traditional uncertainty-aware models like Gaussian processes. Con- cretely, make the following contributions:
A Altogether, these results support recent empirical findings that deep ensembles offer few generalization and uncer- tainty quantification benefits over larger single models (Abe et al., 2022h; Theisen et al., 2024). Our theory and experi- ments demonstrate that these phenomena are not specific to neural networks or Gaussian models but are more general properties of ensembles in the overparameterized regime.
To answer Question 1: we show that the average ridgeless RF regressor is pointwise equivalent to its corresponding ridgeless kernel regressor (Theorem 3.2), implying that an infinite ensemble of overparameterized RF models exactly
3

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
2. Setup
(w (w1,--., WD) ~ x DD) to construct an RF regres- sor hw(xx) = ( (.) TO. We train the regressor pa- rameters 0 to minimize the loss vD Two - y11 + 2|10112 for some ridge parameter À 2 0. When > 0 this opti- mization problem admits the closed-form solution @
We work in a regression setting. The training dataset D {(2,, 8:)) E (X' R)" is a fixed set of size N. The vector E R is the concatenation of all training responses.
We consider RF models adhering to the form (x) VD Ehs x],,, where 0, are learned parameters, W (w:) 12 E 22 are i.idd draws from some distribution (T.), and Φ 1 + IR is a feature extraction function. In the case of ReLU-based RF model with p- p-dimensional in- puts, we have X = Ω = Rd and pbw;, x) = max(0, w, x). Though RF models cannot fully explain the behaviour of neural networks (e.g. Ghorbani al., 2019; Li et al., 2021: Pleiss & Cunningham, 2021), they can be a useful proxy for understanding the effects of overparameterization and capacity on generalization (e.g. Belkin et al., 2019; Adlam & Pennington, 2020; Mallinar et al., 2022).
1 ww (t Pw + >1) y. Although the learning problem is underspecified when À = 0 (i.e in the ridge- less case), the implicit bias of (stochastic) gradient de- scent initialized at zero leads to the minimum norm in- terpolating solution 8(1N) = vD ( ( (b$$) - y.
VD
We denote the resulting ridge(less) regressors as hw (OLN ()) := 1
vD [dww) ', oW>, and () ==
vD [dWy] )1, W.A-
We also consider ensembles of M ridge(less) regressors. We assume that each is trained on different set of i..d. D > N random features Wi...., Was ~ x D but trained on the same training set. Thus, the only source of randomness in these ensembles comes from the random selection of features W, analogous the standard training procedure of deep ensembles (Lakshminarayananl
Notation. For any 2, x. E X, we denote the second mo- ment of the feature extraction function o(w,,) as k(xr, [,,[[( z))(w, z')]. which is a positive definite kernel function. We use the matrix K == [k,,,, 2y)ls E IR N for the kernel function applied all training data pairs and the matrix y == [ō(wy, 3;)lıy E NxD for the feature extraction function applied to all data/feature combinations. In this notation, F-yy refers to the entry in the i-th row and j-th column; if one index is omitted (e.g.. [u]s), it refers to the j-th element of a row- or column-vector, depending on the context. We drop the subscript W when the set of random features is clear from context. Furthermore, we assume that K is invertible.
al., 2017). The ensemble prediction is given by the arithmetic average of the individual models hw () Σ M
m=1 (1 (-).
M
Assumptions. A key difference between this paper and prior literature is the set of assumptions about the random feature distribution (r)). Most prior works assume that en- tries in the extended whitened feature matrix [W; [i] are i.i.d. draws from standard normal distribution (e.g. Adlam & Pennington, 2020; Jacot et al., 2020; Mei & Montanari, 2022; Simon et al., 2024) implying that o(w,,)) -) are draws from a Gaussian process with covariance k., While Gaus- sianity is appropriate in nigh-dimensional asymptotics, it essentially reduces analysis about the ensemble distribution to statement about Gaussian processes. A major focus of this work is to differentiate ensembles from Gaussian processes with regards to uncertainty quantification.
Throughout our analysis, it will be useful to consider the "whitened" fcatures W = R 4 E RNxD where R R-- = K is the Cholesky decomposition of the kernel matrix K. When considering a test point x² E X (or equivalently a set of test points), we extend the K, R, Φ, W notation by
K [k(x*,,;))s
k(a;, x*) [b k( x²)
R
R
c
0
r
0
r
Even if we were to relax the Gaussian assumption to a sub-Gaussian assumption, (as done by Bartlett et al., 2020; Bach, 2024a). the distribution of random features will still not accurately reflect common neural network features if the entries of [W; are assumed to be i.i.d. For instance, consider ReLUU features. If X C R. with p < N, the func- tion max(w" T x, 0) can be fully specified by a p-dimensional random variable. Thus, knowing N evaluations of w x; al- lows one to infer making w deterministic given W. We instead consider the following less restrictive assumptions on the distribution of random feature functions T-)):
W w
R 0
=
[ x )1
(1)
ri
For fixed training/test points, Ew [WWT] = D I,
Eu, [we w1] = and Ew,, [w: W = which can be directly derived from Eg ( " = D K (and similar properties for 6i" the vector of feature evaluations at x², i.e., [Ó(w,,22 )z))), Moreover, the columns [wy; wie] of [W; wi] are i.i.d. since they are affine transformations of the i.i.d. columns of +b
'I the entries of W, w. are i.i.d. Gaussian, then the 2th feature applied to train/test inputs ((t wi; c wi+ri wid) is multivariate Gaussian. This fact holds for any train/test data; thus the 8hh feature is a GP by definition (c.g. Rasmussen & Williams, 2006, Ch. 2).
Overparameterized ridge/ridgeless regressors and en- sembles. As our focus is the overparameterized regime, we assume a computational budget of > N features
4

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
1.6
0.16
1.4
0.15
1.2
10000 nom
N Total Total Parameters
N = Total Parametors
0.8
0.6
0.11
0.2
0.10
0.0
0.09
20 Number of features per model
5000
10000
15000
20000 25000
30000
35000 40000
Number of parameters per model
Figure 2: Overparamelerizeed ensembles equivalent to a single infinite-width model regardless of feature distri- bution, while underparameterized ensembles behave differently. We present the average absolute difference between large ensembles of models with D features versus a single large (or infinite) width model. (Left) RF ensembles with softplus activations, N = 12, using the California Housing dataset (Kelley Pace & Barry, 1997). (Right) Neural network ensembles with ReLU activations, N = 12, 000, on the same dataset. The shaded region shows the standard deviation. Both exhibit "hockey stick"-like pattern, less pronounced for neural networks, where the difference between underparameterized ensembles and the large model is substantial, but diminishes for D) >>> N.
Assumption 2.1 (Assumption of subexponentiality).
(LN) limit of low
[W|
D
1. wewsi (where wi is the 7th column of W) is sub- exponential Vi E f1.... D} and
2. An infinite ensemble of finite-width least norm predic- tors, (LLN) which is the almost sure limit of h Wy, as M + cc, with N < D < oo remaining constant.
(LN)
2. EP w is a.s. positive definite for any D Z N.
These limiting predictors not only approximate large ensem- bles and very large single models but also help characterize the variance and generalization error of finite overparame- terized ensembles, as discussed in Sec. 3.3.
The first condition of Assumption 2.1 ensures that the whitened random features do not have excessively "heavy tails", meaning their values are well-concentratee.. This is mild condition, satisfied if the individual feature compo- nents w, and w i are sub- Ga--asssian (but potentially depen- dent), which is true if the features come from activation func- tions with bounded derivatives and sub- Gaussian weights. The second condition is equivalent to having almost surely full rank, which is not true for ReLUs and leaky-ReLUs fea- tures but which is true for arbitrarily precise approximations thereof. ? Note make assumptions about the mean independence of the entries in given column of [W;w]
Define kN (+) - R as the vector of kernel evaluations with the training data (x)-)) [k(x1,))
A(ZN.-)
As D + x, the minimum norm interpolating model con- verges pointwise almost surely to the ridgeless kernel re- gressor by the Strong Law of Large Numbers:
(I.N) h (.) وق M (+),
1 ( = :(())==k)())) kN(+) K-
On the other hand, using W and w as introduced in Sec. 2 we can rewrite the infinite ensemble prediction T(66.)) (x*)) as (for a derivation of this, see Appx. C.I)
3. Main results
3.1. Equivalence of Infinite Ensembles and the Infinite-Widtl Single Models
R(LNN)
h(LNN) (x*)
To TWT (WWT) || R- R-Ty
+ ri Ew
(2)
We at first assume an infinite computational budget and consider the following two limiting predictors, for which we will show pointwise equivalence in predictions:
To prove the pointwise equivalence of the infinite ensem- ble and infinite- winth single model, we need to show that Ew, [w]] W (WWT)-) term in Eq. (2) is zero. Note that this result trivially holds when the entries of W and w are i.i.d. and zero mean, as assumed in prior work (e.g. Jacot et al., 2020). In the following lemma, we show that this
1. An infinite- width least norm predictor, (LN) the a.s.
TawTs), full-rank T. and de (w) 3000 ReL.U( z.).
B..g.. (w, x)) 1 log(1 + e 2), a > yields an a.s.
5

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
term is zero even when wi and W are dependent, which- as described in Sec. 2-is a more realistic assumption for neural network features:
becoming negligible as the number of features per ensemble member increases (for formal version, see Appx. C.2).
Theorem 3.3 (Non-asymptotic difference between ensem- bles and single models (informal version)), Under slightly stronger assumptions than Assumption 2.1, the La difference between a single neural network with M D features and an ensemble of M neural networks each with D features is, with probability 1 - ö, upper bounded by:
Lemma 3.1. Under Assumption 2.1, it holds that Ew,,, [w] (TWTTN (WWT)-1]] O.
(Proof: see Appx. C.I.) Combining Lemma 3.1 and Eq. (2) yields the pointwise equivalence of T. and hise :
Theorem 3.2 (Equivalence of infinite width single model and infinite ensembles). Under Assumption 2.1, the infi- nite ensemble of finite-width (but overparameterized) RF
1owa(()) (+) - WWinrr O |
O0l/oolg(1100 + O(IDD
regressors (LL is pointwise almost surely equivalent to the (single) infinite-width RF regressor hox (LL))
Theorem 3.3 is supported through a standard bias-variance decomposition of risk:
Theorem 3.2 implies that ensembling overparameterized RF models yields exactly the same predictions as simply increasing the capacity of a single RF model, regardless of the RF distribution (see Fig. for a visualization). Note that this result significantly generalizes prior characterizations of overparameterized RF models that have relied on Gaus- sianity assumptions or asymptotic analyses (e.g. Adlam & Pennington, 2020; Jacot et al., 2020), demonstrating that the ensemble/infinit--iinge mental property of overparameterization. Consequently, we should not expect substantial differences in generalization between large single models and overparameterized ensem- bles, consistent with recent empirical findings by (Abe et al., 2022b; 2024; Theisen et al., 2024).
EA [L (h.] = Enl ER ((h(z) - Ely =))²] L (BA (n)) + E. IV, (h(x)))
(3)
Since (LN) and hw ....., RWas share the same expected predictor (as established in Theorem 3.2), the only dif-
(LN)
ference in the generalization of ) and W arises from their variances. Due to the independence between ensemble members, we have that V WWa.r 1 (M (x) =
model equivalence is funda-
M Vw.,, (' (xr)]. Moreover, prior works such as (Adlam & Pennington, 2020) and empirical results (see Appx. A.3) suggest the variance of single RF model is inversely pro- portional to the number of features. As consequence, we have that Vw- W (x:) : Vy,,, ( x 1/M, further suggesting that the generalization of ensembles and single models should be similar under the same parameter budget.
(L
We emphasize a contrast with the underparameterized regime, where RF ensembles match the generalization error of kernel ridge regression (see Appx. E or Bach, 2024b, Sec. 10.2.2). Width controls the implicit ridge parameter in the underparameterized regime (see Sec. 1.1), whereas width does not affect the ensemble predictor in the overpa- rameterized regime. We confirm this difference in Fig. 2 which shows that RF ensembles are close to the ridgeless kernel regressor when D > N but not when D < N. The figure also illustrates similar behaviours when comparing deep ensembles versus single large neural networks.
Fig. 3 (left) and Appx. A.3 empirically confirm that RF ensembles versus single RF models obtain similar general- ization under fixed feature budgets. Moreover, Fig. 3 (right) depicts a similar trend for neural networks: deep ensembles perform roughly the same as larger single models under a fixed parameter budget. These results show that ensembles offer meaningful generalization advantage over (large) single models and, since the arguments hold for any test dis- tribution, align with empirical findings (Abe et al., 2022b) that ensembles provide no additional robustness benefits.
3.2. Ensembles versus Larger Single Models under Finite Parameter Budget
For modest parameter budgets (where our asymptotic re- sults are not applicable), we compare whether ensembles are more parameter efficient than larger single models. Specif- ically, given access to M D random features, we compare ensembles of M models each of which use D of the fea- tures (h = 1
3.3. Implications for Uncertainty Quantification
We now analyze the predictive variance amongst component models in an overparameterized RF ensemble, a quantity of- ten used to quantify predictive uncertainty in safety-critical applications (Lakshminarayanan et al., 2017). Before diving in a mathematical characterization, it is worth reflecting the qualitative characterization based on our existing results. Because the expected overparameterized RF model First, we provide non-asymptotic theorem showing that Wo and h (LN) behave similarly, with their difference
T7 Σm= (LN) against a single model that uses all MD features ((LN) (-) (i.e., here [Wml = D for all and [W*|- = MD)
"hhis rate is exact for Gaussian features and approximate for the general case.
6

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
Single model Kernel model
Single Large Model
Ensemble
Generalization error
21
Generallzation error
0.245
2.0
0.240
1.9
0.235
1.8
0.230
1.7
0.220
1000
2000
3000
4000
5000
6000
7000
6
Total number features used
Total number of parameters
Figure 3: The generalization error of overparameterizee ensembles and single large models scales similarly with the total number of features. We present the generalization error of ensembles compared to single large models of the same type with equivalent total parameter budgets. Both exhibit nearly identical dependence on the total feature budget. Left: RF ensembles/modees, = 12, ReLU activations, ensembles of models with D = 200 features. Right: neural networks ensembles/models, N 12, 000, ensembles of three-layer MLPs with width 256 in each layer. The shaded region shows the standard deviation.
is the width model, predictive variance is equal
We recognize this quantity as the Gaussian process posterior variance with prior covariance k(-,-) (e.g. Rasmussen & Williams, 2006). Thus, with Gaussian features, ensemble variance admits Bayesian interpretation in addition to the model capacity interpretation. In other words, Gaussianity assumptions justify the use of overparameterized ensemble variance in uncertainty quantification tasks.
Ewla (x'))
Vwl (LN) (x*)] = Ew (LN) (x*)- (x*) TLLEND (xt) )
Ew (kw - - (x)))
Le. the expected difference between finite- versus infinite- width RF model predictions. In other words, ensemble variance quantifies how predictions change if we increase model capacity. This characterization, which holds for all random feature distributions satisfying Assumption 2. is not a standard frequentist or Bayesian notion of uncertainty except under specific distributional assumptions.
Uncertainty quantification under general features. Un- fortunately, this Bayesian interpretation explicitly does carry over the general Assumption 2.1 case. Although Eq. (4) still holds for general feature distributions, it does not have a simple expression unless W and wi are inde- pendent. The variance depends on x" through both r as well as through a complicated expectation involving W and wi. In Appx. C.3 we demonstrate with a simple example that this expectation can indeed depend x", implying that ensemble variance does not correspond to scalar multiple of r" (i.e. the Gaussian process posterior variance).
Uncertainty quantification under Gaussian features. Using Theorem 3.2, the variance of the predictions of single RF model with respect to its random features can be expressed as (see Appx. C.3 for a derivation)
(V R Ew [WWWT) (WWTT-TWaa TWw Wwlh (LN) (x*)]= = (x'))
(4) In the special case where W and wi are i..d. standard normal, this expression simplifies to
w I [[WTTWWE (WW 1 R v).
In numerical experiments using ReLU random features, (Fig. 4 and Appx. A.3), we observe significant deviations be- tween the ensemble variance and the Gaussian process pos- terior variance, further suggesting that one cannot view en- sembles through classic framework of uncertainty. These discrepancies are particularly important for uncertainty esti- mation in safety-critical applications or active learning (e.g. Gal et al., 2017; Beluch et al., 2018), as the only meaningful interpretation of ensemble variance (the expected change in prediction from increasing capacity) may not yield reliabil- ity guarantees or useful exploration-exploiaae tradeoffs.
Wwin
(LN) (x*)) (x*))= (INSNN (EE) D-N-1
where || l represents the squared norm of (LN). in the RKHS defined by the kernel k(.,)). From this equation, we see that [w (1.N) (xx)) only depends on x" through the quantity r2. which by Eq. (1) equal to
2 k(x", x²) kx(x") -K-=kN(x>

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
0.016
Var., (3]
0.014
Traming Data Points
0.012
0.110
0.010
0.088
0.008
0.006
Training Data Points
0.006
0.044
0.004
0.002
0.000
0.00
-4
-2
0
2
4
-4
-2
0
4
Figure 4: RF ensemble variance (left) and Bayesian notions of uncertainty (right) can differ significantly. For N = 6 and D = 200 with ReLU activations, the overparameterized ensemble variance (left) and the posterior variance of Gaussian process with prior covariance .).,,) (right) differ substantially across the input range.
3.4. Equivalence of the Limiting Predictors in the Small Ridge Regime
our knowledge, this Lipschitz-continiity has not been es- tablished even under Gaussianity assumptions. We note that the bound by Jacot et al. (2020, Thm. 4.1), which characterizes the difference between ridge ensembles and infinite models with Gaussian random features, becomes vacuous as À + O. Since Theorem 3.2 ensures that hn & (RR) (xx) h 20.
Having established the equivalence between infinite ensem- bles and infthite-width single models the ridgeless regime, investigate whether this equivalence approximately persists in the practically relevant setting when small ridge regularization parameter À > 0 is introduced. More gener- ally, we aim to determine whether the transition from the ridgeless case to the small ridge regime smooth. While (RR) the infinite-widd limit of k as [W] = -> So.
(RR) (x*) = 0 for = O, we can conclude
that pointwise difference grows at most linearly with X. Specifically, we have that
(RR) (1)) SC.A. (R -1
almost surely converges to the kernel ridge regressor with ridge A, the infinite ensemble 7, (RR) Ew (P (ar) does not generally maintain pointwise equivalence with J, 20.A This divergence occurs under Gaussianity assumptions (Jacot et al., 2020). However, we hypothesize that the dif- ference between these limiting predictors is small when A close to zero, which is common in practical applications. To analyze this regime, we introduce a minor additional assumption, which weaker than Gaussianity:
for some constant C independent of x², provided that X' is compact. In practical terms, this result indicates that for sufficiently small values of X, the predictions of large ensem- bles and large single models remain nearly indistinguishable, reinforcing our findings from the ridgeless regime.
4. Conclusion
This work characterized overparameterized RF ensembles and contextualized theoretical findings with neural network experiments. We used weaker distributional assumptions than prior work (a) more faithfully approximate real- world models and (b) highlight differences between ensem- bles versus models with Gaussian behaviour.
Assumption 3.4. We assume that Ew (w "1 is fi- nite for all [W = D N.
Under Assumptions 2.1 and 3.4, we show that the difference between ridge-regularized ensembles and single models is Lipschitz-continuuus with respect to À (proof in Appx. D.1). Theorem 3.5 (The difference between ensembles and large single models is smooth with respect to X.). Under Assump- tions 2.1 and 3.4, the difference Th (R. (x=) (RR) (x*) between the infinite ensemble and the single infinite-width model trained with ridge X Z 0 is Lipschitz-continune in X. The Lipschitz constant is independent of x* for compact X.
For Question we demonstrated under weak conditions that infinite ensembles and single infinite-widie models are pointwise equivalent the ridgeless regime (Theorem 3.2) and nearly identical with small ridge (Theorem 3.5), sig- nificantly expanding on prior results. We further provide asmmttymttoicc characterization showing that ensembles and large single models with the same parameter budget are nearly equivalent (Theorem 3.3). These results verify recent empirical findings (e.g. Abe et al., 2022b) that much
This result is illustrated in Fig. 5, where the terms bound- ing the difference evolve smoothly with A. To the best of
8

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
1.5
16200 [1
0.5
0.0
-15
-2.0
0.0000
0,0002
0.0004
2,0006
0.0008
0.0010
0.0000
0.0002
0.0004
0,000s
0.0008
0.0010
Figure 5: Lipschitz continuity of predictions for an infinite ensemble and kernel regressor with respect to the ridge parameter. (Left) We plot 1hh (RR)) (x*) ()
(x*) as function of À for 500 test points. (Right) We show the evolution of [h (RR) (x*) 1//55) (x*)) for the same test points. Both plots use ReLU activation functions and the California Housing Dataset with = 12 and D 200. While the direct difference ( (x*) ) - 1 (RR) (x*)] is not shown (for reasons outlined in Appx. A.4), it bounded by the sum of the plotted quantities (see Appx. D.1). The evolution of these plotted bounding
terms thus illustrates that this direct difference is Lipschitz continuous in (proven in Theorem 3.5) and converges to zero A consequence of Theorem 3.5 and the ridgeless equivalence established in Theorem 3.2).
of the benefit attributed to overparameterized ensembles, such as improved predictive performance and robustness, can be explained by their similarity to larger single models. Notably, our analysis does not rely on Gaussianity, empha- sizing that these phenomena are fundamental properties of overparameterized models and not artifacts of specific feature assumptions.
consequences of our work, none which we feel must be specifically highlighted here.
A Acknowledgements
GP is supported by Canada CIFAR AI Chairs. JPC sup- ported by the Gatsby Charitable Foundation (GAT3708), the Simons Foundation (542963), the NSF AI Institute for Arti- ficial and Natural Intelligence (ARNI: NSF DBI 2229929) and the Kavli Foundation. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC: RGPIN-2024-06405, Resources used in preparing this research were provided, part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute.
In contrast, for Question 2, we found that uncertainty inter- pretations of ensemble variance are contingent on Gaussian- ity assumptions and fall apart under more general feature distributions. We characterize ensemble variance as the expected difference to a single larger model, which only corresponds (scaled) Bayesian notion of uncertainty un- der strong independence assumptions. With more realistic feature distributions, the ensemble variance does not corre- spond to any conventional notion of uncertainty, reinforcing recent empirical findings on the limitations of ensemble un- certainty quantification (Abe et al., 2022b). This deviation supplies further evidence that caution is needed when using ensembles in safety-critics settings.
References
Abe, T., Buchanan, E. K., Pleiss, G., and Cunningham, J. The best deep ensembles sacrifice predictive diversity. In I Can't Believe It's Not Better Workshop: Understanding Deep Learning Through Empirical Falsification, 2022a.
Overall, while our results do not contradict the utility of overparameterized ensembles, they suggest that their ben- efits may often be explained by their similarity to larger models and that further research is needed to improve un- certainty quantification methods.
Abe, T., Buchanan, E. K., Pleiss, G., Zemel, R., and Cun- ningham, J. P. Deep ensembles work, but are they nec- essary? In Advances in Neural Information Processing Systems, 2022b.
Impact Statement
Abe, T., Buchanan, E. K., Pleiss, G., and Cunningham, J. P. Pathologies of predictive diversity in deep ensembles. Transactions on Machine Learning Research, 2024.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal
Adlam, B. and Pennington, Understanding double de-

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
scent requires fine-grained bias-variance decomposition. In Advances in Neural Information Processing Systems, 2020.
Freund, Y. Boosting weak learning algorithm by majority, Information and Computation, 121(2):256-285, 1995.
Gal, Y., Islam, R., and Ghahramani, Z. Deep Bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1192, 2017.
Angelova, J. A. On moments of sample mean and variance. Int. J. Pure Appl. Math, 79(1):77-85, 2012.
Bach, F. High-dimensional analysis of double descent for linear regression with random projections. SIAM Journal on Mathematics of Data Science, 6(1):26-50, 2024a.
Ghorbani, B., Mei, S., Misiakiewicz, T., and Montanari, A. Limitations of lazy training of two-layers neural network. In Advances in Neural Information Processing Systems, 2019.
Bach, F. Learning Theory from First Principles. MIT Press, 2024b.
Goldt, S., Loureiro, B., Reeves, G., Krzakala, F., Mézard, M., and Zdeborová, L. The gaussian equivalence of gen- erative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning, pp. 426 071,471,200 2022.
Bartlett, P. L., Long, P. M., Lugosi, G., and Tsigler, A. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(49):30033-30777) 2020.
Belkin, M., Ma, S., and Mandal, S. To understand deep learning we need to understand kernel learning. In Inter- national Conference on Machine Learning, pp. 541-549, 2018.
Gustafsson, F. K., Danelljan, M., and Schon, T. B. Evaluat- ing scalable bayesian deep learning methods for robust computer vision. In Computer Vision and Pattern Recog- nition Workshops, pp. 318-331 118-119,40 2020.
Belkin, M., Hsu, D., Ma, S., and Mandal, Reconciling modern machine- leaaning practice and the classical bias- variance trade-off. Proceedings of the National Academy 2019.
Hansen, L. and Salamon, P. Neural network ensembles. Pattern Analysis and Machine Intelligence, 12(10):993- 1001, 1990.
of Sciences, 116(32):55499-155
Hao, Y., Lin, Y., Zou, D., and Zhang, T. On the benefits of over-parameterization for -iittii-iitiiittoonnn generaliza- tion. arXiv preprint arXiv:2403.1 17592, 2024.
Beluch, W. H., Genewein, T., Nurnberger, A., and Kohler, M. The power of ensembles for active learning in image classification. In Computer Vision and Pattern Recognition, 2018.
Hastie, T., Montanari, A., Rosset, S., and Tibshirani, R. J. Surprises in high-dimensionall ridgeless least squares in- terpolation. Annals of Statistics, 50(2):949, 2022.
Breiman, I. Bagging predictors. Machine Learning, 24(2): 123-140, 1996.
Hu, H. and Lu, M. Universality laws for high- ligh-dimensional learning with random features. IEEE Transactions on Information Theory, 69(3):1932-1964, 2022.
Breiman, L. Random forests. Machine Learning, 45(1): 5-32, 2001.
Chen, L., Lukasik, M., Jitkrittum, W., You, C., and Ku- mar, S. On bias-variance alignment in deep models. In International Conference on Learning Representations, 2024.
Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. Ad- vances in Neural Information Processing Systems, 2018.
Jacot, A., Simsek, B., Spadaro, F., Hongler, C., and Gabriel, F. Implicit regularization of random feature models. In International Conference on Machine Learning, 2020.
Chen, T. and Guestrin, C. Xgboost: A scalable tree boost- ing system. In International Conference on Knowledge Discovery and Data Mining, pp. 785-794, 2016.
Cho. Y. and Saul, L. Kernel methods for deep learning. In Advances in Neural Information Processing Systems, 2009.
Jeffares, A., Liu, T., Crabbé, J., and van der Schaar, M. Joint training of deep ensembles fails due to learner collusion. Advances in Neural Information Processing Systems, 2024.
Dietterich, T. G. Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems, pp. 1-15. Springer, 2000.
Kabán, A. New bounds compressive linear least squares regression. In Artificial Intelligence and Statistics, pp. 448 456.45, 2014.
Fort, S., Hu, H., and Lakshminarayanan, B. Deep en- sembles: A loss landscape perspective. arXiv preprint 191.0279/. 2019.
Kelley Pace, R. and Barry, R. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3):291-25 1997.
10

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
Kendall, A. and Gal, Y. What uncertainties do we need in Bayesian deep learning for computer vision? In Advances in Neural Information Processing Systems, 2017.
Nixon, J., Lakshminarayanan, B., and Tran, D. Why bootstrapped deep ensembles not better? In NeurlPS "I Can't Believe It's Not Better!" Workshop, 2020.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Process- ing Systems, 2017.
Opitz, D. and Maclin, R. Popular ensemble methods: An empirical study. Journal of Artificial Intelligence Re- search, 11:169-198, 1999.
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan. B., and Snoek, J. Can you trust your model's uncertainty? evaluat- ing predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019.
Lee, J., Bahri, Y., Novak, R., Schoenholz, S., Pennington, J., and Sohl-dickstein, J. Deep neural networks gaus- sian processes. In International Conference on Learning Representations, 2018.
Lee, J., Schoenholz, S., Pennington, J., Adlam, B., Xiao, L.,, Novak, R., and Sohl-Dickstein, Finite versus infinite neural networks: an empirical study. In Advances in Neural Information Processing Systems, volume 33, pp- 15156-15172, 2020.
Pleiss, G. and Cunningham, J. P. The limitations of large width in neural networks: A deep Gaussian process per- spective, Advances in Neural Information Processing Systems, 34:3349-3333 3363, 2021.
Rahimi, A. and Recht, B. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, 2007.
Lee, S., Purushwalkam, S., Cogswell, M., Crandall, D., and Batra, D. Why m heads are better than one: Training diverse ensemble of deep networks. arXiv preprint arXiv:1511.06314, 2015.
Rahimi, A. and Recht, B. Uniform approximation of func- tions with random bases. In 46th Annual Allerton Con- ference on Communication, Control, and Computing, pp. 555-561, 2008a.
Li, M., Nica, M., and Roy, D. The future is log-Gaussian: ResNets and their siinit--dptth-ndd--iee limit at ini- tialization. Advances in Neural Information Processing Systems, 34:7852-785 -7664, 2021.
Rahimi, A. and Recht, B. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In Advances in Neural Information Processing Systems, 2008b.
Loureiro, B., Gerbelot, C., Refinetti, M., Sicuro, G., and Krzakala, F. Fluctuations, bias, variance & ensemble of learners: Exact asymptoties for convex losses in high- dimension. In International Conference on Machine Learning, pp. 14283-14314, PMLR, 2022.
Rasmussen, C. E. and Williams, C. K. Gaussian Processes for Machine Learning. MIT Press, 2006.
Mallinar, N., Simon, J. B., Abedsoltan, A., Pandit, P., Belkin, M., and Nakkiran, P. Benign, tempered, or catastrophic: A taxonomy of overfitting. In Advances in Neural Infor- mation Processing Systems, 2022.
Ruben, B. S., Tong, W. L., Chaudhry, H. T., and Pehlevan, No free lunch from random feature ensembles. arXiv preprint wrXiv:2412.054166 2024.
Rudi, A. and Rosasco, L. Generalization properties of learn- ing with random features. Advances in Neural Informa- tion Processing Systems, 2017.
Mei, S. and Montanari, A. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75(4):667-766, 2022.
Simon, J. B., Karkada, D.,, Ghosh, N., and Belkin, M. More is better in modern machine learning: when infinite over- parameterization is optimal and overfitting is obligatory. In International Conference on Learning Representations, 2024.
Mentch, L.. and Hooker, G. Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. Journal of Machine Learning Research, 17(26): 1-41, 2016.
Tao, T. Topics in Random Matrix Theory. American Mathe- matical Soc., 2012.
Montanari, A. and Saeed, B. N. Universality of empirical risk minimization. In Conference on Learning Theory, pp. 4310-4312, 2022.
Thanei, G.-A., Heinze, C., and Meinshausen, N. Random projections for large-scale regression. Big and Complex Data Analysis: Methodologies and Applications, pp. 51- 68, 2017.
Neal, R. M. and Neal, R. M. Priors for infinite networks. Bayesian learning for neural networks, pp. 29-53, 1996.

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
Theisen, R., Kim, H., Yang, Y., Hodgkinson, L.,, and Ma- honey, M. W. When ensembles really effective? Ad- vances in Neural Information Processing Systems, 36, 2024.
Wager, S., Hastie, T., and Efron, B. Confidence intervals for random forests: The jackknife and the infinitesimal jackknife. Journal of Machine Learning Research, 15(1): 1625-1651, 2014.
Wainwright, M. J. High-Dimensional Statistics: A Non- Asymplotic Viewpoint. Cambridge University Press, 2019.
Webb, A., Reynolds, C., Chen, W., Reeve, H., lliescu, D., Lujan, M., and Brown, G. To ensemble or not ensem- ble: When does end-to-end training fail? In Machine Learning and Knowledge Discovery in Databases: Euro- pean Conference, ECML PKDD 2020, Ghent, Belgium, September 14-18, 2020, Proceedings, Part III, pp. 109- 123. Springer, 2021.
Williams, C. K. I. Computing with infinite networks. In Ad- vances in Neural Information Processing Systems, 1996.
Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, 2020.
12

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
0.25
Data Gesorating Pinetion
Training
0.75
0.00
0.50
-0.50
Data Geserating Function
Training Data
0.75
0.00
0.25
-125
-0.75
-1.75
-2
-4
-2
Figure 6: True function f(z) sin(5 6% x) with different random seeds. The blue line shows the true function, while red dots represent training samples for two distinct random seeds.
In the appendix, we will provide the following additional results:
1. In Appx. A, we will describe our experimental setup for RF models in more detail, difficulties we encountered when developing the experiments, and provide the results of additional experiments.
2. In Appx. of additional experiments.
will describe our experimental setup for neural network models in more detail, and provide the results
3. In Appx. C we will give the proofs for Secs. 3.1 and 3.3 in the main paper.
4. In Appx. we will give the proofs for Sec. 3.4 the main paper.
5. Finally, in Appx. E, we prove (under mild assumptions) that infinite underparameterized RF ensembles equivalent to kernel ridge regression under some transformed kernel.
The code to run all our experiments can be found on GitHub: https: / cww.nioiiiiiioooiooo nic-ndenidnn theoreti ca 1al1-11 1-1 simi tat iesiiins-oon wwwns-vverrrraaeeeeeee eensmbbl es. It contains README, md file that explains how to set up and run the experiments.
A. Experimental Setup and Additional Results for RF Models
A.1. Experimental Setup
We had two setups using which performed most of our experiments:
1. We generate training and test points uniformly random from [-5, 5j using the function f(x) sin(5 (((55-bTa is a vector (depending on the random seed) and the noise parameter is = 0.05 (we assume Gaussian noise with mean 0). In this setting, we use = 6, D = 200, and data from TR (i.e., = 1) if not specified otherwise. You can find plot of an example true function Fig.
bTxz) x), where
2.
We use the California Housing (Kelley Pace & Barry, 1997) dataset and sample distinct training and points from it (randomly permutating the dataset initially). In this setting, The data dimension is R here. In contrast to the first setting, we employ data normalization using max-min normalization on the entire dataset since we experimentally found this makes our methods more stable.
N 12, D 200 if not differently specified.
We calculate the generalization enror using = 1000 test points in both settings. In the first setting, we calculate the variance the predictions of a single model using 20, 000 models, while in the second setting, we use = 4, 000 models. Apart from Fig. 12 where we use 100, 000 samples, "infinite" ensembles consist of M 10, 000 models.
13

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
Troizing Points after tronsformation
Ling
Hyperplane of as
Hyperplane of
X
Figure 7: Visualization of hyperplanes separating training points. We illustrate how series of hyperplanes can separate growing subset of the training points, leading to triangular, invertible matrix structure as subset of tb
As distribution 7(-) of the elements w, W always use M/O,, f). As activation functions, use ReLU, the Gaussian error function, and the softplus function 3 log(1 exp(B w wTa) x)) with 3 = 1. For the first two activation functions, there exist atalalically calculatable limiting kernels, the arc-cosine kernel (Cho & Saul, 2009) and the erf-kerne (Williams, 1996). The closed forms for these are
karc-cosino (aac-cosine (I,
2π ||>|||>0||| (sin 0 (π 0) cos 0)
where / cos
cos-1
[ |>>||>>||||
and
2x T x'
Kerr(a x') sin
(11 + 211x1|2)(1 2|)21)2)
For the softplus function, we approximate the kernel by estimating the second moment k( x0) [e( (w,, >x, x") of the feature extraction using 10" samples from r(-). For sampling Gaussian features, described by Jacot et al. (2020).
the same approach as
Before training on data, we always append a 1 in the peroeth-dimension of the data before calculating the dot product with w (correspondingly, the dimension w + 1) and applying the activation function. In the ridgeless case, we use = 10-8 avoid numerical issues.
A.2. Notes on Stability
During our experiments, we encountered challenges related to both mathematical stability (i.e., matrices being truly singular rather than nearly singular) and numerical stability. This section outlines these issues and describes the steps we took to mitigate them.
Most importantly, the matrix Φ is not almost surely invertible when using the ReLU activation function, meaning that technically, the second condition of our Assumption 2.1 is not fulfilled. In numerical experiments, this results in cases where (2 b is nearly singular (though stabilized with À = 10 (0-8).
On the other hand, when is sufficiently large relative to N, 4w full rank with high probability, which implies that Ty invertible with high probability. Given our data transformation of appending a the zeroeth dimension, one can see this as there exists series of (non-zero probability sets of) hyperplanes separating an increasing subset of the training points, leading to subset of Tw's 's columns that form triangular, invertible matrix (see Fig. for visualization). Intuitively, higher data dimensionality and better separability of the points increase the probability of &w having full rank.
As an example of the discussed instabilities, see the adversarial scenario shown in Fig, 8, where N = 15 and many training points are placed very close to each other. In this case, individual RF regressors exhibit relatively high variance output values (due to numerical instabilities), which are not averaged out in the "infinite" ensemble. Similar issues were also observed when using the Gaussian error function as the activation function, although they were generally less pronouneed.
14

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
Ensemble Model
0
Kenel Model
Troining Thata
-2
-4
-6
-10
-12
RF Models
Kernel Model
Training Date
Figure 8: An adversarial example where the infinite ensemble overparameterized RF models numerically not equivalent single infinite-width RF model. (Left) We show sample of 100 RF models (blue) with ReLU activations trained on the same N 15 densely clustered data points. Additionally, we show the single infinite-width RF model (pink). (Right) We again show the single fhfnnte-wwdth RF model (blue) and the "infinite" ensemble of . 10, 000 RF models (pink). A significant difference between the two models is observed in this adversarial case, indicating instability.
N Total Parsmeters
1.4
0.8
12
1.0
N Total Parameters
0.6
0.5
0.6
0.4
0.4
0.3
20
50
10
20
40
50
Number of features per model
Number of features per model
Figure 9: Using softplus activations instead ReLU activations reduces instabilities in overparameterized RF ensembles. The plots show the average absolute difference between the predictions of an infinite ensemble and a single infinite-wiet width model for varying feature counts D. using N 12 training samples from the California Housing dataset. (Left) ReLU activations exhibit significant instability, especially for D > N, D N, and do not consistently show the expected pointwise equivalence between the infinite ensemble and the single infinite-width model. (Right) Softplus activations - as equivalently shown in Fig. 2 1--smoth out these instabilities and more consistently show the expected pointwise equivalence.
15

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
To alleviate these issues, we used the following approaches:
We used a relatively low number samples, N or N 12, compared to D 200. As shown in Fig. even with D 200, there still considerable amount of variance in the RF regressors (i.e., the individual RF regressors are yet closely approximating the limiting kernel ridge regressor).
We appended 1 in the zeroeth dimension of the data before calculating the dot product with w.
We performed additional experiments using the softplus function with 3 = as a smooth approximation of the ReLU activation function. This often helped stabilize the numerical computations, as seen in Fig. 9, where we repeated a part of the experiment from Fig. using the ReLU function as activation function which increased the numerical instability for low D values.
We used ridge term À = 10 8 in the ridgeless
stabilize the inversion of Ww
We used double precision for all computations and used the torch. dinalg.lstts function with the driver ge sd (for not-well-conddtioned matrices) to solve linear systems.
We applied max-min normalization to the entire California Housing dataset to improve stability.
A.3. Additional Experiments for the Ridgeless Case
To address the question of whether our findings are specific to normally distributed weights wi for the feature generating function, we supplement Fig. 1. Fig. 10 replicates that visualization using weights wi drawn from a Uniform(-10, 10) distribution and the softplus activation function. As can be seen, the equivalence between the infinite ensemble of overparameterized RF models and the single sdfinitewwitth RF model remains apparent, and no perceptible difference is observed.
2.5
RF Models
Finembbl Moriel
Kernel Model
Kernel Model
Training
1.0
Training Daia
0.5
0.0
-0.5
-05
-1.0
2
2
(a) Sample of 100 finite-width RF models (Uniform wi)
(b) Infinite ensemble (Uniform wi) infinite-width model
Figure 10: Replication of Fig. with Uniformly Distributed Weights w; Similar to Fig. 1, we demonstrate the equivalence of an infinite ensemble of overparameterized RF models to a single infinite-width RF model. Here, the weights wi for the Softplus activation functions are drawn from a Uniform(-10, 10) distribution. (Left) A sample of 100 finite-width RF models (blue) trained on N = 6 data points, with the single infinite-width RF model (pink). (Right) The infinite-width RF model (pink) and the "infinite" ensemble of M = 10, 000 models (blue). No perceptible difference observed, mirroring the findings with normally distributed weights.
Furthermore, to illustrate the convergence of finite ensembles to the infinite-width model prediction as the number of ensemble members M increases, Fig. 11 expands on the setting of Fig. 1. It shows that even with a small number of ensemble members, the average prediction begins to concentrate around the infinite width model, and this concentration improves M grows.
16

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
13
1.5
Ensemblee Model
Ezeeruble Model
Keraei Model
Kernel Model
1.0
Training Data
1.0
Troining Data
0.5
0.5
-0.5
-4
-2
2
4
(a) Ensembles of 5 members
(b) Ensembles M 10 members
1.5
1.5
Model
Kernel Model
Kernel Model
1.0
Training Data
Training Data
-0.5
(c) Ensembles M 20 members
(d) Ensembles M 40 members
Figure 11: Evolution of Ensemble Predictions with Increasing Number of Members (M). Following the setup of Fig. I (ReLU activations, normally distributed Wi, N data points), these plots show 10 sample ensemble predictions (blue lines) for varying ensemble sizes M. The single infinite-width RF model is shown in pink. As M increases, the ensemble predictions become more concentrated around the infinite-width model.
Additional experiments on the identity of infinite-width single model and infinite ensembles. In Fig. 12, we show that the term E[w W (WWT)-11 is consistently zero for both ReLU and the Gaussian error function activations consistetly with Lemma 3.1. To further demonstrate that this result is not dependent on Gaussian-lic like weight distributions, Fig. 13 shows this for softplus activation function, with weights drawn from a Uniform(-10, 10) distribution and Laplace(O., 1) distribution. The expectation of the term remains centered at zero, supporting the generality of our theoretical findings.
Additional experiments on the ensemble variance. We observed different behavior of the RF regressor variance and r1 as shown in Fig. 4 consistently across different random seeds and dimensions for both ReLU and the Gaussian error function activations as activation functions. In Fig. 14, we present additional examples for the Gaussian error function in one dimension and the ReLU activation two dimensions.
Additional experiment on generalization error and variance scaling. In Fig. 3, the generalization error decay for the ReLU activation function. To verify the consistency of this trend, we repeated the experiment using the Gaussian error function and the corresponding erf-kernel The result very similar, shown in Fig. 15. Furthermore, this figure shows that the variance of a single model with M D features decays as MD. matching the ensemble's behavior.
17

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
12000
300
Mosn: 0.00
Moan: 0.00
10000
8000
4000
2000
50
0.00
0.06
0.02
0.00
0.02
0.04
0.06
0.08
0.20
0.15
0.10
0.05
0.00
0.03
0.10
0.15
[w] (WWT) >- (whitened residual)
[w]
(WW T>- (whitened finite model residual)
Figure 12: Empirically, the term E[w] W [ (WWT)- is consistently zero. We plot the distribution of the first index of w. W (WWT) which captures the difference between the infinite-width single model and smaller overparameterized RF model (see Eq. (2)). (Left) We use ReLU as activation function, I4 E R, and N = 6, D = 200. (Right) We use the Gaussian Error activation function, the California Housing dataset and = 12, D 200.
40
Mosn:
35
30
30
25
20
20
10
5-
5-
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15
-0.20
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15
WWIWTWWWT
1), (whitened finite model residual)
[w
WIWT(WM (WW T) }}, (whitened finite model residual)
(a) Softplus activation, Uniform(-10 10, 10) weights
(b) Softplus activation, Laplace(O,, 1) weights
Figure 13: Empirical validation of E[w T (WWT)-1) Rs O for Softplus activation and non-Gaussian weights. Similar to Fig. 12, we plot the distribution of the first index w W (WWT)-+ Both plots use a softplus activation function, the California Housing dataset, and N = 12, 200. (Left) Weights w; are drawn from a Uniform(-10, 10) distribution. (Right) Weights w; are drawn from Laplace distribution with location 0 and scale In both cases, the distribution is centered at zero.
18

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
0.016
Varzg Iig (M) ()]
0.030
0.014
Training Data Points
0.025
0.012
0.010
0.020
e Training
0.008
0.015
0.006
0.110
0.055
0.002
0.000
0.000
Var,, 1b
Training Data Points
Trainin Data Points
0.8
0.4
0.7
0.6
0.3
0.5
0.2
0.1
-2
-4
Figure 14: Variance and r for different activations and dimensions. (Top left) Variance of RF model predictions across the input range for = 200 and N 6, using the erf activation function. (Top right) Corresponding r values across the input range using the erf kernel. (Bottom left) Variance RF model predictions across the input range for 200, p = 2, and N 12, using the ReLU activation function. (Bottom right) Corresponding r2 values across the input range using the arc-cosine kernel.
10a
Single model
Kernel model
Ensemble
Varr, Varr, (1)
Generalization
3.6
10
Point:
3.4
10-1
Point
Point
Point
1000
2000
3000
4000
5000
6000
7000
Number of features of random feature models
Total number of features used
Figure 15: Variance and generalization error scale similarly with the number of features, consistent with Fig. In (a), the variance of single model with MD features decays MD matching the ensemble's 's behavior. In (b), the generalization error of ensemble with M models and D 200 features shows similar decay that of single model with MD features. Results use the Gaussian error function, California Housing dataset, and N = 12.
19

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
1000
Mran: 0.00
Mean: -0.00
000
200
Frequency
Frequency
600
400
100 -
200
-0.20
-0.15
-0.05
0.00
0.000 0.005
0.010
[w] + TR 1/3
(WW
'R $1s
Figure 16: Empirically, the term Ew wi WIWT WT (WWT + D R "] [ is consistently zero. We show the E IR,, which captures the difference in predictions between c Ew, WWT (WWT + + R- and finite-sized overparameterized RF model (see Eq. (7)). We use 1.0 in both plots. (Left) We use a ReLU activation function, I; E R, and = 6, D 200. (Right) We use the Gaussian Error Function as activation function, the California Housing dataset, and N 12, D 200.
empirical distribution of an index of w WT (WWT + D -. R-
A.4. More Experiments for the Ridge Case
Additional experiments for the convergence of the expected value term. In Appx. D, we show that a variant of Lemma 3.1 also holds in the ridge case. More precisely, we show that
Ew
w W (WWT + +
under Assumption 2.1. We repeated the experiment from Fig. 12 for the ridge case to verify this experimentally, The results are shown in Fig. 16.
Additional notes. In Fig, 5, we illustrate the Lipschitz continuity of the predictions for an infinite ensemble and a kernel regressor with respect to the ridge parameter. Rather than directly presenting the difference oo, (xx)) h oc, (x*) we (RR)
(RR)
- show the evolution of h ( (z*) T.(L.8)) T1(5,)(('')) (a*) and ( (x*) - (1.8) (z*).. This choice was made because the upper
bound we obtained was not consistently tight for settings with large D. particular, the pointwise predictions of the infinite ensemble 7, (RR)) and the single infinite-width model h (. trained with ridge were already very close for non-zero A. We opted to display the upper bounds rather than the direct difference to avoid cherry-picking favorable settings.
Our best explanation for this phenomenon is that infinite ensembles under Assumption 2.1 the ridge regime often behave similarly to the single infinite- width model h. (RR) with an implicit ridge parameter \, which solves the equation 2o.
N d
x x
D
1=1 Ā
d;
where de are the eigenvalues of the kernel matrix K, as shown by Jacot et al. (2020) under Gaussianity. Intuitively and empirically, for large D, the implicit ridge À tends to he very close to the true ridge X. Using Lemma D.1, this suggests that for small values of X, the difference between the infinite ensemble and the infinite-width single model h oo. with ridge À is already minimal before À approaches zero.
(RR)
Interestingly, our findings (see Fig. 2) suggest that in the ridgeless case, the similarity to the ridge regressor with the implicit ridge only holds in the overparameterized regime. Note that this does not violate the results from Jacot et al. (2020) since the constants in their bounds blow up as - in both the underparameterized and overparameterized regimes.
20

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
B. Experimental Setup and Additional Results for Neural Network Models
B.1. Experimental Setup
For all our experiments with neural networks, we used a three-layer MLP with hidden layers of equal width and ReLU activations. Models were trained for 1000 epochs using SGD with momentum, a learning rate of 0.01, and a momentum decay of 0.9.
Training was performed the same set of 12,000 samples from the California Housing dataset, with validation set of 3,000 samples and a test set of 5,000 samples. Since the number of parameters scales quadratically with the hidden layer width, the overparameterized regime is reached at a width of approximately 80.
All reported results are based on the best checkpoint selected using validation performance over the 1000 training epochs.
B.2. Additional Results and Limitations of Our Experiments
The results in Fig. 2, which show the average absolute difference in predictions between large ensembles with increasing number of parameters (D) in the component models and a single large model (with MD parameters), are further supported by the increasing correlation of residuals between the ensemble and the single model, as shown Fig. 17. When the component models of the ensemble become overparameterized. ensemble residuals align better with those of a single large model, indicating that overparameterized ensembles make more similar errors large single model than their underparameterized counterparts,
Furthermore, the correlations of the residuals of large ensembles of overparameterized models and two large single models trained with different initializations were comparably high, with the average correlation between an ensemble and a single model even slightly higher. At the same time, the residual correlation between two large ensembles trained with different initializations was significantly higher than both of these correlations. The lower variance in predictions across multiple ensembles compared a single large model does not align with our theoretical expectations and experiments with random feature models. We hypothesize that this discrepancy arises from large single models being more unstable to train but did not investigate this limitation of our experiments in more detail.
C. Proofs for Overparameterized Ridgeless Regression
C.1. Equivalence of Infinite Ensemble and Infinite Single Model
We start by proving the equivalent formulation of the infinite ensemble prediction stated in Eq, (2) using the terms W and w as introduced in Sec. 2:
Proof. Defining ow = (o(w;,>")) DR,,, we have
(xxxxx) Ew Ho (t xw -11
Ew +riwi] WT R (R WWT R) R)- "0-1
Ew,wl [cTW +rwwir T (WWT) -1] RTTTV
-.R.- R Ty + B.E
w (WWT) )
R R-TT..
(6)
where c, R, are as defined in Eq. (1). The left term Eq. (6) is equal to h (LNN (x):
(LN) (x") = [k(rs, 2"]," ,- = RR y
21

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
Correlation of Residuals: 0.918
Correlation of Residuals: 0.957
Single (Width = 321) Residuals
Single (Width = 324) Residuals
-1
-2
-3
Ensemble (Width 20) Residuals
Ensemble (Width 50) Residuals
Correlation of Residuals: 0.958
Correlation of Residuals: 0.965
Single (Width = 329) Residuals
Single (Width = 336) Residuals
1-
-1
-2
-2
Ensemble (Width 80) Residuals
Ensemble (Width 110) Residuals
Figure 17: Correlation of residuals between ensembles and single large model. Scatter plots comparing the residuals
of single large model with MD parameters to those of ensembles with increasing component parameters (shown by the the width of the component models): (Top left) 20, (Top right) 50, (Bottom left) 80, and (Bottom right) 110. The correlation of residuals increases as the component parameter count grows. This suggests that overparameterized ensembles make more similar errors to single large model than (strongly) underparameterized ensembles.
22

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
In the of O, we similarly see that
h ( (x*) = lEw Ho ( Iw +1) "|
Ew,
[c + W R( R+ -
R
R
(cT++ruug WT (WWT- + D
R
Rー
=c =C_EW.
EW
WWT (WW DD -O-A-R-RR
R y
WIWT (WWT +D -
R-T y-
R
Note that the simplification demonstrated in Eq. (2) does not work as nicely in the underparameterizee case (D S N). This is because the weights, in this case, are given by 0 = (o) and thus the infinite ensemble prediction expands as:
11 = Ew (ow (w) - $
+r (W RR -1 W R] y.
= EW.wi
Here, RR lies inside the inverse, preventing the simplifications available in the overparameterized regime.
Next up, we show that the expected value Ew,wa wi WT (WWT)-") is zero under Assumption 2.1. This directly implies the pointwise equivalence of the infinite ensemble and the single infinite-wide width model (see Theorem 3.2).
Lemma 3.1 (Restated). Under Assumption 2.1, it holds that Ew, [w] W (WWT)-11)
Proof. Define A_, = (WWT -w,w,") ). Note that A-1 is almost surely invertible and positive definite by assumption Assumption 2.1.
By the Woodbury formula, for almost every WWT have that
(WWT)-1- (A_ -
wuy A 1+ tw.
which implies that
w WT (WWT)-1 -12 Wwtl
ww A AA_
1+t1 A ure
+w: T A J. A Aw
w/ A_ "uy A
Db. Wii W;
1+w7 A_ we
1+ A wi
E
D
wit
A_
14
For any positive definite matrix B E N and any vector E N |||||| = and any i (1,..., D), we have
mmw 1+w, Bws
E Ewin.
1tw, Bwy
1) dl
Itw; Bwi
| 2 (1 + wp Bwi) t dt
P [!M
dt
///a
<
2exp
2
di +
2 eex (-2) dl,
(8)
v22
23

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
where the last inequality is standard sub-exponential bound applied w1sW;- Note that we here use the fact that E[w 0 and the (v², ---ueeeeepoonnntall of [mw..!"
Since the last two integrals in Eq. (8) are finite, the expectation Ew [( ( D/(1 + w/ Bux)) is finite, By the weak law of large numbers, for iid. random variables w!!) and w Ii across different j's, we have
(3)
w (wn))) Ew 1=1 (w Ban?")
1+
Bwi
1
> - O,
for any / >> 0 and E R. such that ||a|| = 1 as M + we have that
At the same time, repeating the sub-exponential argument above,
P
M
M j=1
I+
w
v Bu?
SP
M
i-11
M
(1) Tol >1
2eepp
M1²
0 0cts
1
2v
2cxp
Mf
1>
2a
+
MM 00. Here we use the property that the sum of M (v2, ..s-subeexpoonattial random variables a--uu- exponential.
/((1))))))))))))) + wi Together, these results imply that Ww [(wIew] 1/11 matrix A_ is positive semidefinite, almost surely invertible (by the second half of Assumption 2.1), and independent of Wi, wiis we have that
Bw:)] for every positive definite B. Since the random
Bwi,W w WT (WWT) -1
EP Ew Ewi
wa A I+w, A
E!= EA,; Ewins
A
0.
1+m
this proof Ew,w; WIWT (WWT + D. X R RR
We remark
equivalently holds for the ridge-regression case, i.e., since the proof does not rely on the specific form of the matrix A_; other than it being positive definite. Thus by Eq. (7) directly get that under Assumption 2.1 it holds that
that
7, (RR) (x*) = Ew, [WWT (WWT + .D.A.RTTR-J) R RTy.
(9)
C.2. Ensembles versus Larger Single Models under a Finite Feature Budget
We now prove the formal version of Theorem 3.3.
Let's first restate the informal version of the theorem:
Theorem 3.3 (Restated). Under slightly stronger assumptions than Assumption 2.1, the L2 difference hetween a single neural network with MD features and an ensemble of M neural networks each with features is, with probability upper bounded by:
1. ( (-) RILN) ( { f1=== ( O(ID))
+
We now provide the formal version of this theorem, which uses the following definitions:
Definition C.I. Define E (x² - (x)) as Σʃ = Jx k(x, (z))))(
Definition C.2. For any fixed set of random features W !wl,...wp) .......)] of size D define Φw(x) ((,,,),),,,,,, ((WD, J. ()(W),)),,-..
and the approximated kemnel function kw(x,,) = ( Tow()) Using this, define Ew: : (2()) - (2)) as Ewf = kw(a, (z))dd(()
24

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
(We will drop the W subscript from Φw, kw. and Ew when the of random features is clear from context.) Now, state the assumptions that we need for the proof which stronger than the assumptions in Assumption 2.1:
Assumption C.3. We make the following assumptions:
The columns of & are subgaussian with constant L, i.e P([Xu) 2 s 2xxp (-P/L3) for all E IR with |o| S 1.
For any d1 (0, there exists C' < oo and Do(61) such that for all D Do(01) it holds that | 2 - with
probability 2 1 - ő1-
The feature extraction ((w,,) is almost surely square integrable over the data probability measure (i.c. Ex(o(w, ) < 00).
Theorem C.4 (Non-asymptotic bound on the L2 difference between ensembles and single models (informal version)).
Under Assumption C.3, there exist constants e1, C2, co such that for any ô1 E (0, 1) and all M, N, D with M . D 2 Da(ōi) and defining Amin = min(1, Amin (I)) it holds:
Amin If
2
12
+
> O and 82 M 02e c DoDDmin (aDmmink22.1n2 (n2))n + eze
< 1 - 51, where
/ )) then for any 83 E (0,1- 81 02) holds with probubility at least 1 51 62 83 that the Lz-norm of the difference between the larger, but finite-width single model and the finite ensemble with the same features is bounded by
ma
Amp 2T
{
M.D M.D )
} and N2j max(O, Amin 2-1
(
now
(2N)(() ()) hw 11 1 c O(1/))
25)
where e
V Aman log
Proof.
First step: Expressing as the difference in their parameter norms.
In the following, we define
r)) Using this definition we get that with 1 (ENS)TT where 0(ENS) are the parameters of the j-th component model, we get N E== vD -1), .., pWyDD x)] (EEN)) ( At the same time, can write d(x) T ((Single) (Singl)))
(xx) = VMB o(warn))))
(ENNS) 1 [0 (ENS) V
d(x) OIENS) ((EEN))
M
VMD [o(w), ..., (WWDD, x]] (
Using equivalence of norm argument, we get E. (o(E)) ((ENS) - ((x) (((ingll) ))²] s Cupper |[@(ENS) - ((Single ) ||
More precisely, for this argument, Cupper will be the operator norm of the matrix Ex [o()) Since this is positive semi-definite matrix, the operator norm is equal to its largest eigenvalue.
We can now define the operator T T.) + R.D, T()) = Jx (())(()) The adjoint operator T : R (x)) (y) El vio(wr..)
Let's see how T acts vector D:
TT T)
T (T))TT)
= / o(z)o(.) vdu(x)
J (())(() du(z)o
25

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
Thus, have that TT* Eze (x)
J. We know from linear algebra that 77* has the same eigenvalues T*T.
Thus, it is enough bound the eigenvalues of E. =
To bound the eigenvalues of Σ, we bound its difference in operator norm to the equivalent operator for the true kernel K,
Lee. Σf = 1 k(r, (x)))((()) Since we assume that for MD > Du(ó1) we have that - Σ - C with probability
2 1 81. we get that
| < |لح|| + C' with probability 2 1 - 61. This implies that with probability 2 - 61 Cupper is
bounded by a constant independent of M and D.
Second step: Using least norm geometry.
By least norm geometry, we get that || (Sigle || [[0() 1 ||0( || Furthermore, we directly get that
the norm of the ensemble as the sum of the norms of the component models, i.e. ||a(ENs) شド= ( (ENS)
Third step: bound the probability that all empirical kernel inverses admit a Taylor expansion and have bounded lower eigenvalues.
We now want to bound probability that the eigenvalues of MDD "4" - K-iie. the difference between the empirical kernel matrix and the true kernel matrix-are bigger than Amin. If the eigenvalues are less than Amin. then
1. the inverse of the matrix 1/2
K- 1/2 admits a Taylor expansion and M.D h TOrTK
2. the minimum eigenvalue of MD (ND001 "b"")= will be lower bounded.
We use the following concentration inequality to bound this probability:
Lemma C.5 (Wainwright (2019), Thm. 6.5). Let 21...., In Idd be i.idd L-subgaussian random variables with
A = B [appo]]] E Rdxd, Then for any 5 ' 0, there exists some C1 C22 C3 O that
P
n
1=
All2
2 C1
(
d +
d
+
S 02e
can 6,.6)
T2
n
Applying Lemma C.5 to the case of the random matrix M-D
TT K, we have that M D,d A K and x; is 62
the i-th column of Φ. Furthermore, we want to bound the probability that It set 61 max(0,
-K|12 -
is bigger than Amin Thus, we 21
L
{ M.D + M.D } > 0, then the probability that the eigenvalues the difference are bigger than Amia is at most Coe Og M min(x1.*33) 2
Amia 2
M.D + } and get that if Amin 2
13
Similarly, we get that the probability that the eigenvalues of the difference for single component model [ D K)
The probability that the eigenvalues of any of the component models are bigger than 2 is then by union bound bounded by M c2e D min min(n2, (ās) (again Amin 2
are bigger than سنسد is bounded by C2e -ca -rigDmin(raa (2)) where we define Kaj max(0, Amiy 2
+
{v N + D
(v + > 0).
We now define d2 M - 62t - DyD min (ray. + o2e D,D ())
Fourth step: assume that all empirical kernel matrices come from truncated distribution.
MD ΦΦ matrices conditioned on the fact that their inverse matrices admit Taylor expansion. With probability - 62. the D T matrices that form our ensemble and single model admit Taylor expansions. In other words, with probability - ò2 we can view the D 2. matrices as i.i.d. draws from 7D and we can view
Let iD and #MD be the distributions over 1 $.. and 1
MD ØT" as a draw from Tp-
Fifth step: bound the difference between the expected ensemble and single model inverses under the truncated distribution.
26

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
Under dD. we have that:
( (1- (1-K-1/(6000) k -1
BaD k2/2 (0,00,,0011111111100110111
BaD
Σ EED (1- (1-K-
(0 1K-1
The zero-th term in the Taylor expansion is L. Recognizing that (,44, is a sample mean random feature outer products
with expectation K, the first term in the Taylor expansion is O. Using formula (15) from (Angelova, 2012) we find that the second term is equal to D M2 for some constant M2 an all other terms are O ( Thus:
=I+ M2 +0 ( 1
ERD
K
D
D2
Following the same argument we have that
BRND
1/22
1
= =1
M2 +0
(11
1
MD
MD
(MD)²
Thus,
L
-1
-1
Σ ERn
D
1/
ERAD
1/
$
1/
MD
(MD) )1) K-1/2/
y 11122
[1+
D
M2
-
MD
M2 - 0
ky-
Sixth step: Using a Hoeffding bound.
T -'y and its expected value (over the truncated distribution). Equiva- Lastly, we bound the difference between y MD
>-bb)-11 lently, we have to do this for the ensemble terms 37 E y (+ y.
We first employ that we now that the operator norm of the difference #D DT K is bounded by Amin implying that the eigenvalues of MD 4Φ" T are bounded by Amii from below.
Thus, we have that 0 < y" ( ΦΦ - this holds for the ensemble terms.
|| is bounded a.s. under the truncated distribution. Equivalently, Amam
2_
MD
To bound the difference between the ensemble and single model, we can now employ Hoeffding bound. This gives that
M
P
(1 EaD
ml 2 S Sexp (-Me² Amin))
and for the single model term:
( - ExxeD [y' T (T +D T)
( ->y] 2 2e] S exp (
Setting exp (- M Ammi)) + exp (-2 (-emmin)) S 6g 2. 2xpxppppp (-c² Xmin) and solving for c gives that:
log
Xmin
8z
27

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
Seventh step: Taking everything together.
We can now employ union bound to get that the probability that all three conditions are satisfied is at least 1 - òj 82 83 and thus we get the bound on the difference between the ensemble and single model in L2 norm.
Note that the fact that we worked under a truncated distribution in the previous step is not a problem, since the corresponding events have lower probability under the the non-truncated distribution as long as we have already assumed the exclusion of all events where the difference between the empirical kernel matrix and the true kernel matrix is bigger than Amin
C.3. Variance of Ensemble Predictions
In the next step, we show the formula for the variance of a single model prediction under Gaussianity, Note that one could also get this result by slightly extending proofs by (Jacot et al., 2020).
Lemma C.6 (Variance of single model predictions), Under Gaussianity and assuming D N + 1, the variance of single model prediction at a test point x² given by
-11 12
12:00
(I.N)
Vw!Ww
(LN) (x
(10)
where 11x is norm defined by the RKHS associated with kernel k(.;)).
Proof.
We start by writing down the variance of the prediction of a single model:
(LN) Ww (x*)] Ew(hw (LN)
(a+)²)
Ew (n (x*)]²
Using Theorem 3.2, the definition of the prediction of a single model and the definition of W and w1, we can expand this expression:
Ewlow W Tw) ww (W) T w (x))))(()=)))))))))))))))))))(
= -E,,, [[r_w] + tc T.TWWWTT R (R WW R) -1 yy T WW R) -T R W(r W))
- -(h(LN) (I)))
E.www [[]] W +c WWWT (WWT)-1 R R- my R- (WWT)-1 -T W(riw +c-
-(H(LN)
R-Tyy²² (=TT))2((tt) ( (x)) (x))²
.r] rI [WT ) "y R
+ r1 Bw.w [wIW COwITT(WWW - yy T -
Now we can see that the first two terms cancel out (since h ((LN)()) (x) T R T 7) and the third term is zero by Lemma 3.1. We are left with the fourth term, which we can slightly rewrite:
(x Ww(h (I.N)
= r E.. [ WT (WWT)-1 yy T (WW
= =r R Bw,..
WT (WWT)-1)
(11)
Using the tower rule for conditional expectations, we have:
Wwlhi (LN) (x:) = = R B. - (WWT)) W Wa w W =riv T R 'Ew (WWT)- -TWE, | (wo w [W]W (WWT)-111)
Since the Gaussianity assumption implies W and wn are independent, we get:
Ww (LN) (z)) -ri=" T R 'w (TW Mllmwn TWWWTTTTTTTT (
28

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
Moreover, since by Gaussianity tT and W multivariate Gaussians with the identity matrix covariance, we get (via the expected value of a Wishart and inverse Wishart distribution; note that for getting this expected value, need to assume that D > N + 1):
Vw ww
(LN) (1:] = r R ' Bw((WW ( ((WWT)
-riv" R 'Ewl(ww (WW(
.1 NT BRIRRT
D-N-1
=rib-N- K K-yy
Recognizing that y'K' KK-y || 1
11 (e.g. Wainwright, 2019, Ch. 12) completes the proof.
An equivalent argument does not work under the more general Assumption 2.1 since w and W not necessarily independent. Even in the case of independence, Ew [WW ) might not be known.
Counterexample for subexponential case. uncorrelatedness between W and w the term
We now give explicit counterexample showing that when only assuming
E Ew w ((WWT) Ww (WWWW)-))
from Eq. (11) depends on x" implying that the variance does not only depend on x" via r?.
Let us assume N = D = and let W be uniformly distributed across the set { V12.5'
Then we
V12.3' v12.5' V12.5
have E[W] = 0 and E 2 = 2 12.5
16
9
1.
12.5
Now consider an x= that produces a so that w.. 2 when W {
3
3 and w. 0 otherwise. Then we
V12.5*
V12.5
have [[w] W| 0 and [[wi] 1. The value of E is now 12.5
Furthermore, consider an x² that produces wi so that wi /2 when W
VI2.5' vI2.5 } and w1 0 otherwise.
Then we have B[w W] 0 and B[w 1. The value of E is now 12.5
16
D. Proofs for Overparameterized Ridge Regression
D.1. Difference between the Infinite Ensemble and Infinite Single Model
We begin with a lemma, which shows that the prediction of kernel regressors is Lipschizz-contnuous in À for any x* and
O. We will denote the kernel ridge regressor with regularization parameter À h (RR). introduced in Sec. 3.4.
Lemma D.1 (Bound on the difference between the kernel ridge regressors). Let X, X' 0 be two regularization parameters. Then, for any x² E X it holds that:
(RR) Ih 00.
- X1 VyTK-A (x*)) RR (x*) S Vn. - -
where we assume k(r,, x²)== I C for all i [N].
(RT) Proof. We can write the kernel ridge regressors h 2e.A (x*) = E 01, and n (x*) =E: 02,h(1;; with coefficients an and a2 given by:
a1 = (K ++)
022 = (K + X'))-1y
We write y in the orthonormal basis of the eigenvectors of K, i.e. y Ei= aeu. We call the corresponding eigenvalues of K di..... d,, 0.
The matrix (K + AI)-1 has the same eigenvectors as K and the eigenvalues are 0 0< d,+> A. Thus, we can write
1 < 1
01 = E 0: xx 24 and 02 Ei= 04 a Ve.
29

---

Theoretical Limitations of Ensembles in the Age of Overparameterizattin
In the step, bound ||a
Using the orthonormality of the eigenvectors, we get:
|| a2l1 =E== (a:
- [ xxx-xxx) 2
Now we bound
1 X+di
1
$
X' -
|A'
which gives us:
X+d;
-x++2--''d,+td
- S ~10"-
||ı - a2ll Ei=
Using this result, we can bound the difference between the predictions of the two kernel regressors at a single point x":
(RR) (x*) (R.) (x*) -o (2,)k(( x²)| E an, - 02,4 x*)
Since k(,;, x²) S C₁, we get (using the relation between the I-norm and the 2-norm):
1fa(=") - fx Ci E. Jar, - a2.1 s Cilla1 - asllav vn. C1 IX Xa -
Using similar arguments, now show that the expected prediction of RF regressors, i.e., the prediction of the infinite ensemble of RF regressors, is Lipschitz-continuour for any x* and À > 0:
Lemma D.2 (Bound on the difference between expected RF Regressors). Under Assumption 2.1 and Assumption 3.4, the expected value of the prediction of RF regressors Lipschitz-continuuus in for any x² and X 2 0, ie., for any x² it holds that:
h ( (xx) - (B.)) (z") = R D1//////
where C2 is constant depending the distribution of Φ.
We use the characterization of h (RR) (x*) from Eq. (9), which gives the difference
Proof.
T Ew
[WWT
NOWWTTDDX.. (WWWT+D-X-P) R. -WWw + +D -D------T=-)
1 TT
We can now reverse some steps we made to get this characterization and write it in terms of t again:
c R TEw W (1ow + D. X. fw + + D D- -
m/
And now, using Jensen's 's inequality and the convexity of the two-norm, we can pull out the expected value to the outside of the difference:
|c R T -Ew Ilowo (lowoN+) + - - T +D (D-A-1)-)) wll
Similarly to the proof of Lemma D.I, write y in the orthonormal basis of the eigenvectors of Φ (note that drop the subscript W for notational simplicity), i.c. y = a,8. Furthermore define the cigenvalues of ΦΦ as d1 dn O. The matrix (ΦT7 + D AI)-1 again has the same eigenvectors as tb T and the eigenvalues are
0 Z.D < D
Multiplying with ΦΦ ( D. -D and ΦΦ (ΦOT +D XI) then gives
+Φ T ($T T +D- DD- Σx= a dit+Dx U3
कΦ ( + D. DT) ly= En +T
30

---

Theoretical Limitations of Ensembles in the Age of Overparameterriattio
We calculate the difference of these two vectors using the orthonormality of the eigenvectors:
| T ( DD- DD- ... ( +D D -D) yyl2 = (as (a.+bx a.Fb.x )
Now look the difference between the two coefficients and see that for each have:
de d D-A
D-1X-XI
d.+D-N
de
Thus, we have that the difference is bounded by:
( T + +D-XI) y- y- -0 (# + +D -
D X12
1ly
dy
All together, we can now bound the difference of the expected values the predictions of RF regressors via:
[1 1 (x*) -1 (R (+ [=")]= < ||e R "IllelI - AlEdd
1
Since tr(( Ex=1 a, and the trace is linear operator, we can write:
Edd dN
Ew (tr(fw w-)] tr(Ew [(ww C2
which is finite whenever Ew [( [] is finite, i.e. Assumption 3.4 holds.
Using Lemma D.I and Lemma D.2 we can now show that the difference between the infinite ensemble where each model has ridge Å and the infinite single model with ridge À is Lipschtiz- in À for À 2 0:
Theorem 3.5 (Restated). Under Assumptions 2.1 and 3.4, the difference 1T ( (x*) (RR)) (x*) between the infinite
ensemble and the single fhfintte-witth model trained with ridge À 2 0s Lipschitz-comtinnonn in A. The Lipschitz constant is independent of x² for compact X.
Proof. We bound difference normal triangle inequality:
(RR) 20.'' (x*) (RR) (x*) - 1 (x*) (RR) (x*) by using first the inverse, then the
hh (1 (x*) - (1.)) (xx) ] 16 (x") - 1 (RR) (x*))
(RH)
1h (RR) (x*) (((RR) (x*) + h (x*) - h (RR) (x*)
(RR) hh xc,' (x*) )- (RRR) (x*)) + (R()) (x*) (RT) (x*))
Using the bound from Lemma D.1 and Lemma D.2 (and summarizing the the corresponding constants e₁ and (2) we can bound this by:
(RR) Th (RR (x*) - 1 xc, (x*)] 17 (RR) (x*) (RR)) (a*) s sal2 -| +2/2 -X
Thus we have Lipschitz-comtinuity in À for 2 0.
The Lipschitz constant is independent of x= for X' compact since the Lipschitz constants from Lemma D.1 and Lemma D.2 depend on x" in a continuous fashion.
Note that an equivalent argument in combination with (Jacot et al., (rroooonnnnnnooonn 4.2), i.e. Às 7-1 X, directly gives the in À for À 2 0 for the difference between the infinite ensemble and the infinite- width single model with effective ridge in the overparameterized regime.
31

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
E. Underparameterized Ensembles
Here, we offer a proof that infinite, unregularized, underparameterized RF ensembles are equivalent to kernel ridge regression under transformed kernel function. We emphasize the difference from the overparameterized case the central focus of our paper-in which the infinite ensemble equivalent ridgeless kernel regressor. Thus, underparameterized ensembles induce regularization, while overparameterizze ensembles do not.
Other works have explored the ridge behavior of underparameterizee RF ensembles (Kahán, 2014; Thanei et al., 2017; Bach, 2024b); however, these works often focus on an equivalence in generalization eπrоr whereas we establish pointwise equivalence. To the best of our knowledge, the following result novel:
Lemma E.1. If the expected orthogonal projection matrix Eww [RT W (W RRTT) WTR) is well defined, and
contraction (i.e., singular values strictly less than 1), then the infinite underparameterized RF ensemble Tibb (x*) (x*) is equivalent to kernel ridge regression under some kernel function k(.,-).
Proof. When D < N, the infinite ridgeless RF ensemble is given by
Ew [
d(w,, x²) E ( o ww) ww y
Ew.wi [ + c W) (W RR w) -1 w w-] Ry.
(12)
where W. WL.FI,C c, are defined in Sec. Defining the following block matrices:
W
R R= 0
c r
W
((N+I)
R R=
R(N+I)
E (N++1)
we can rewrite Eq. (12) as
RI =
(Ew W (W RR WRT)) (eETW)-WWWT "T]] Ry.
By adding and subtracting RR inside the outer parenthesis, we can massage this expression into kernel ridge regression in transformed coordinate system:
h ((LN) = = RR..T + Ew W (WT RR w) - [T])=²
.RR
Ry.
- R R y.
(13)
Applying the Woodbury inversion lemma to -1 we have:
Ä- = Ew W (WT RR RRR w) WT]
(14)
+ Ew W (WT
RR RRTTR w) T R (I-Enp (w-E)(Pwl)) Ew R W (W RR
RRR.RR W) -1 w WT].
where Pw is the (random) orthogonal projection matrix onto the span of the columns of R W:
Pw = R W (W RR w) W R.
Because Pw is an orthogonal projection matrix, we have that ||Pw ll2 = 1, and thus (by Jensen's inequality) [|Ew[Pw]]e < 1. If this incquality is strict so that / - [Pw] is invertible, we have by inspection of Eq. (14) that Á is positive definite. Therefore, the block matrix
RR π
7 R ब =
T A カTĀ- ČTÂ-IR جاA
(15)
32

---

Theoretical Limitations of Ensembles in the Age of Overparameterization
also positive definite and thus the realization of some kernel function k(;,, i.e.
1((1,
k(), x(x1,x")
R A- R A a
x1) _k=",,, .i)
K(IN,IN) x²)) k( ,NN) x(x",x*)
Note that if Á then by Eq. (1) we recover the original kernel matrix
(()
k(x1,=*)
R R a R R
k(xN,21) k(x", 21)
x(NN,NN)
M(NN
((N) k(
Thus, the underparameterrizd ensemble in Eq. (13) simplifies to
(k(21,11)
k(11,N))
-1
(x)) = [k(=*,11)
x[x",xx]]
+1
(()
k(IN,EN)
which is kernel ridge regression with respect to the kernel k(-,-).
33


Reviews:
[{"review_id": "bNDZZQpgj9", "reviewer": "Reviewer_CuSq", "summary": "This paper proves that in the random feature (RF) regression, the ensemble estimator is approximately equivalent with the simple regressor, as long as the model size is sufficiently great. This result can be applied not only to ridgeless models, but also to models with small ridge parameters. \nBesides, the paper also demonstrate an interpretation of the predictive variance among ensemble members. It turns out that, in contrast with traditional models, the predictive variance of wide random feature ensembles quantifies the expected effects of increasing capacity rather than uncertainty. \nThese results implies that in this case, ensembles do not provide additional benefits over a simple model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and feedback on our paper. Below, we address the concerns and questions raised.\n\n**1\\. Concerns about Theorem 3.3:**\n\n> \"Firstly, there is some problem with Theorem 3.3. The bound term on the right-hand side does not converge to zero as the number of features increases, provided that $\\\\delta$ is fixed. Thus, Theorem 3.3 is not sufficient to conclude that in the case of finite ensemble budget, the RF ensemble regressor and the RF simple regressor are asymptotically equivalent with high probability as D goes to infinity.\"\n\nWe believe there is a misunderstanding regarding this theorem. Theorem 3.3 does not aim to establish exact asymptotic equivalence. We proved asymptotic equivalence holding almost surely in our main result (Theorem 3.2). Thus, Theorem 3.3 is simply an analogous finite-sample guarantee showing that the single large model and the ensemble behave similarly with high probability at finite scales.\n\n**2\\. Generality and novelty of Theorem 3.2:**\n\n> \"On the other hand, Theorem 3.2 is impressive but not surprising. The linear random feature regression in the case of infinite width simply coincides with kernel interpolation, hence the infinite-width assumption together with the linearity of the model helps simplify the computations”\n\nWe believe there is a misunderstanding regarding the key contribution of Theorem 3.2. The main contribution of the theorem is to investigate an *infinite* ensemble of *finite-width* random feature regressors rather than the infinite-width model. To our knowledge, no prior work has demonstrated that such an ensemble converges to the kernel interpolator, except in the special case of zero-mean Gaussian random features.\n\n**3\\. Question about the essential aspect preventing ensemble advantages:**\n\n> \"Traditionally, many ensemble methods such as random forest are based on simple but non-linear methods, and with the inspiration of this work, we are curious about what is the essential aspect of random feature model that prevents ensembles from bringing extra advantages? Is it the linearity?\"\n\nAs suggested by our title, the key attribute preventing ensembles of overparameterized random feature models from providing additional advantages is the **overparameterization** of the ensemble members. This claim is supported by our theoretical and empirical results contrasting ensembles of underparameterized and overparameterized random feature models (see Figure 2 and Appendix E) as well as other recent work (e.g., Abe et al., 2022).\n\nWe hope these responses address your concerns and clarify the contributions of our work. If you have further questions or suggestions, we would be happy to address them.", "claims_and_evidence": "The statements of this paper are clear. A number of references and experimental data are provided to make the claims more convincing. \nThe main results of this paper warns that it is unwise to assume without theoretical guarantees the additional advantages of ensembles over simple models. In the background that more and more recent studies concentrate on random feature models, the contribution of this paper to our understanding of random feature learning is impressive and inspiring.", "methods_and_evaluation_criteria": "There are some limitations in this paper. \nFirstly, there is some problem with Theorem 3.3. The bound term on the right hand side does not converge to zero as the number of feature $D$ increases, provided that $\\delta$ is fixed. Thus, Theorem 3.3 is not sufficient to conclude that in the case of finite ensemble budget, the RF ensemble regressor and the RF simple regressor are asymptotically equivalent with high probability as $D$ goes to infinity. \nOn the other hand, Theorem 3.2 is impressive but not surprising. The linear random feature regression in the case of infinite width simply coincides with kernel interpolation, hence the infinite-width assumption together with the linearity of the model helps simplify the computations, as is shown in section 3.1 and appendix C. \nThus, we are concerned that these results and examples are not general and sophisticated enough.", "theoretical_claims": "We have checked the proofs and found no evident mistakes.", "experimental_designs_or_analyses": "The details of experimental designs and results are clearly demonstrated in this paper.", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Traditionally, many ensemble methods such as random forest are based on simple but non-linear method, and with the inspiration of this work, we are curious about what is the essential aspect of random feature model that prevent ensembles to bring extra advantages? Is it the linearity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "pbof0EnkIv", "reviewer": "Reviewer_hF3L", "summary": "This paper presents a theoretical analysis of ensembles of overparametrized models (more parameters than training data) in which authors claim that an Infinite Ensemble is equivalent to an Infinite-Width Single Model. This analysis is done by using the equivalence between random feature regressors (RF) and neural networks. In particular, the authors aim at answering two key questions:\n\n- Do ensembles of overparameterized models provide generalization or robustness benefits over a single (very large) model trained on the same data?\n- What does the predictive variance of overparameterized ensembles measure?\n\nIn order to tackle these questions, the authors compares the Infinite Ensembles and the Infinite-Width Single Models for the RF model. In simple terms, the model resulting from aggregating (i.e., computing the mean over the distribution of $\\omega$) infinitely many models is pointwise equivalent to the infinite-width model. This result is based on Assumption 2.1 asking $\\Phi$ to be full rank and controlling the dependence with W and $\\phi(w_i,x*)$. It is important to underline that Theorem 3.2 is independent of the RF distribution. In order to complete the comparison and analysis on ensemble methods, in Section 3.2 authors assume limited computational budget. Here, in Theorem 3.3 they bound the distance between M models with D features and one single of MD random features. Figure 3, exhibits the theoretical finding, underlying the small difference in the generalization error. Since an important feature of ensemble methods is their ability to give measures of uncertainty, authors study in section 3.3. how the predictive variance of the two compared approaches behaves, giving details of Gaussian and general features. They conclude their analysis in section 3.4 accounting for the role of $\\lambda$ (the ridge parameter in the loss function) for the infinite ensemble and infinite-width model. As it was predictable, allowing $\\lambda$ to get higher values, affects the distance, since it is a parameter controlling for parametrization.\n\n## Update after rebuttal\nI thank the authors for integrating some comments in order to better clarify the work. I appreciate your efforts in answering my doubts and comments, which are convincing and clear. I therefore confirm to accept the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable comments and feedback on our paper. Below, we address the concerns and questions raised and outline the changes we plan to make to the manuscript based on your suggestions.\n\n**Regarding the Experiments:**\n\nWe agree that explicitly including the experimental setup in the main text will help clarity. Thus, we will move these details from the appendix into the main body of the paper. \n\n**Additional literature:**\n\nThanks for pointing out the additional papers. We will discuss \\[1\\], \\[2\\] and \\[3\\] in our related works section. Furthermore, we will add relevant references in the field of uncertainty quantification to our related works section and we will address the distinction between epistemic and aleatoric uncertainty.\n\n**Specific Comments:**\n\n* **C2 (Line 297, L2-norm notation):** We will correct the notation as suggested.  \n* **C3 (Connection between Figure 5 and Theorem 3.5):** We will add a short explanation of and a reference to Figure 5 in the main text.   \n* **C4 (Quantifying the difference in Figure 1):** We will include a more detailed plot showing the evolution with increasing ensemble size $M$ in the appendix and reference this in the figure description.  \n* **C5 (Feasibility and intuitiveness of assumptions):** We will add a brief intuitive explanation of Assumption 2.1, noting that these assumptions closely hold in most relevant scenarios including e.g., for distributions with bounded support.   \n* **C6 (Stability vs. robustness in Figure 3):** Indeed, the observed stability in Figure 3 arises from averaging effects. As suggested by reviewer e9Q9 we will add shaded regions for variance across instantiations of single models/ensembles  \n* **C7 (Counterexamples in Appendix A.2):** Due to length constraints, we cannot incorporate Figure 8 directly into the main text but will ensure it is more prominently referenced in the main text.  \n* **C8 (Experiments beyond standard normal distributions):** To clarify, even though the weights of feature generating function (e.g. $\\\\omega$) are normally distributed, the random features themselves (i.e. $\\\\mathrm{ReLU}(\\\\omega^\\\\top x)$ are not, and thus our experiments make use of the generality of our theorem. Regardless, we will supplement Figure 1 and Figure 13 with $\\\\omega$ drawn from heavy tailed distributions.   \n* **C9 (Invertibility assumption with ReLU activations):** We investigated other activations which fulfill the invertibility, including softplus (see e.g. Fig. 2 (left) and Fig. 9; note that using softplus activations, we can approximate ReLU activations arbitrarily precisely, see footnote 2 on page 4\\)  and the Gaussian error function activation function (see e.g., Fig. 10 on the right hand side and Fig. 11 & 12), which satisfy the invertibility assumption.   \n* **C11 (Double descent phenomena):** We are not entirely sure what aspect of the double descent phenomena you are referring to. If you clarify further, we would be happy to discuss it.\n\n**Responses to Questions:**\n\n- **Q1 (Training assumption):** Yes, in our theoretical analysis, we assume that the training procedure converges to the unique least norm solution, guaranteed for standard SGD initialized at zero. In our empirical analyses, we also always trained all models with the same training algorithms.\n- **Q2 (Epistemic vs. aleatoric uncertainty):** We admit that a single random feature model cannot decouple epistemic and aleatoric uncertainty. This decoupling is a possible advantage of ensembles, though our analysis in Section 3.3 implies that standard ensembles do not cleanly deliniate these sources either.  \n- **Q3 (Feasibility of Assumption 2.1):** As discussed in the paper, the first condition is fulfilled whenever $w\\_{\\\\perp i}$ and $w\\_i^{\\\\top}$ are sub-Gaussian, which is true when the features come from activation functions with bounded derivatives and sub-Gaussian weights (which we would argue is a very weak assumption). For the second condition we expect that most nonlinear activation functions, such as sigmoid, tanh, sine, or cosine, satisfy this condition if we assume i.i.d. weights from a distribution with a density function. However, rigorously proving this is beyond the scope of this work.  \n- **Q4 (Impact of regularization parameter $\\\\lambda$):** Intuitively, the training outcome is continuously dependent on the ridge parameter $\\\\lambda$; thus, small variations in $\\\\lambda$ typically result in minor deviations in predictions. The proof of our theorem effectively provides a rigorous confirmation for this intuition.\n\nWe hope these responses and adjustments clarify our contributions and address your feedback. If you have further questions or suggestions, we would be happy to address them.", "claims_and_evidence": "All claims are well supported and conclusions are realistic. This is a mainly theoretical paper in which claims are always supported by some kind of experiment.", "methods_and_evaluation_criteria": "Since the paper is mainly theoretical, the experiments carried are not very extensive. Only synthetic data and a single real world dataset (California Housing) are used, what limits a bit the validity of the evidences. However, all experimental result are suitable for the claim aiming to support and provide a good proof of the theoretical results.\nAuthors claim that this method is independent of the $\\omega$ distribution but only use synthetic data from a Normal one and one single real world dataset.\nThe idea of using RF regressors as Neural Networks, even not novel, is correctly chosen.", "theoretical_claims": "All theoretical claims seem to be well suited and properly derived. Appendix shows all demonstrations. The important claims have been checked and properly understood.", "experimental_designs_or_analyses": "I think they could include more cases a part from the synthetic and California Housing. Specially, I think the experimental setup should be explicitly mentioned in the main text. \nDespite that, this paper is mainly theoretical and I think the experiments support well their findings.", "relation_to_broader_scientific_literature": "According to comparing ensembles with single models, the paper presents good literature. However, literature related to uncertainty quantification is a bit scarce and they do not consider the fact that a single model cannot distinguish between epistemic and aleatoric uncertainty.", "essential_references_not_discussed": "Even though they cite [1] I think they missed the opportunity to introduce the term joint (or end-to-end) training, which gives a good vision of why ensembles can be similar to single models depending on how we train them.\nI think they missed [2] and [3], which showed similar results with less theoretical derivations and explained consequences of joint training, respectively.\n\n[1] Jeffares, A., Liu, T., Crabbé, J., & van der Schaar, M. (2023). Joint training of deep ensembles fails due to learner collusion. Advances in Neural Information Processing Systems, 36, 13559-13589.\n\n[2] Webb, A., Reynolds, C., Chen, W., Reeve, H., Iliescu, D., Lujan, M., & Brown, G. (2021). To ensemble or not ensemble: When does end-to-end training fail?. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part III (pp. 109-123). Springer International Publishing.\n\n[3] Abe, T., Buchanan, E. K., Pleiss, G., & Cunningham, J. P. (2022). The best deep ensembles sacrifice predictive diversity. In I Can't Believe It's Not Better Workshop: Understanding Deep Learning Through Empirical Falsification.", "other_strengths_and_weaknesses": "The paper is quite technical and I admit I am not an expert in the subject of kernel functions. However, the mathematical details convinced me and results are coherent with their goals. Moreover, they approached the comparison of the single model and the aggregated model in many ways, resulting in a complete and detailed analysis. The conclusions correctly reflects their finding suggesting that the benefits of overparametrized model may be explained by their similarity to larger models.\nI find the conclusion section very well written.", "other_comments_or_suggestions": "C1- The whole paper is well written and no substantial error was found. \n\nC2- In line 297 the notation for the L_2 norm is quite unusual and $\\_2$ would fit better.\n\nC3- It is not clear the connection between Figure 5 and Theorem 3.5 as there is no reference in text to Figure 5 and apparently one is dealing with $\\bar{h}^{LS}\\_{\\infty}(x^*)$ and the other $\\bar{h}^{RR}\\_{\\infty,\\lambda}(x^*)$. Even though it is mentioned the reasoning in Appendix 4, I suggest a more detailed and clear explanation as the evolution w.r.t $\\bar{h}^{RR}\\_{\\infty,\\lambda}(x^*)$ is never shown. \n\nC4- In Figure 1, authors write “We note no perceptible difference between the two.” which is quite vague. It would be more appropriate to quantify the distance. Also would it good to actually see the evolution of the graph, when increasing M, instead of just two values?\n\nC5- Assumptions are more or less explained, even though an intuitive idea of them would help the reader in understanding the main results. In the text, there is no mention of how feasible are these assumptions.\n\nC6- I understand that when authors say in line 295 (2nd column) that ensemble provide no additional robustness benefits they talk from a variance point of view when the number of parameters is fixed. However, in the right plot of Figure 3 we can see that ensembles show more stability to the total number of parameters. Is this only a consequence of averaging the models or has other explanations?\n\nC7- I know the length restriction of the paper is limiting, but counterexamples like Figure 8 in Appendix A.2. and an explanation to them would reinforce the findings.                                           \n\nC8- In the experimental section, authors explain to take parameters $\\omega$ from the standard normal distribution. This is a bit contrasting with their claim of generality of their results with respect to the parameter distribution. Also, part of the novelty of this paper is exactly the generality of results under a certain distribution $\\pi(\\cdot)$, hence it would have been better to provide a small example with another distribution.                                                                              \n\nC9- In Appendix A.2. authors claim that $\\Phi_{\\Omega}\\Phi^T_{\\Omega}$ is not almost surely invertible when using the ReLU activation function, hence contrasting assumption 2 in 2.1. I understand the approach to tackle this issue, but what about considering another activation function? Would it be interesting to analyze the impact of several activation functions on this assumption? \n\nC10- Line 80 repeats “model” twice.\n\nC11- Have you considered the effect of overparametrization in the double descent phenomena?", "questions_for_authors": "Q1- The comparison between Deep ensembles and single large models assumes that the training process is the same, right? Models in Deep ensembles are trained independently because one seeks diversity among models. It is true that a set of models could be trained end-to-end since joint function is fully-differentiable, but as [1] demonstrates this is, effectively, equivalent to a single wide model.\n\nQ2-  A part from the predictive variance, how could the single model decouple epistemic and aleatoric uncertainty?                                                                                                                           \n\nQ3- How feasible are the assumptions 2.1 necessary to prove the main Results? Since the matrix $\\Phi_{\\Omega}\\Phi^T_{\\Omega}$ is not invertible (hence non-positive definite) in the case of ReLU, it is reasonable to consider other activation functions? What would be their impact on the results? Would the assumption be satisfied?                                                                                \n\nQ4- Besides the technical proof, how can you justify the statement “In practical terms, this result indicates that for sufficiently small values of λ, the predictions of large ensembles and large single models remain nearly indistinguishable” regarding the impact of the the regulariser parameter?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ug7bR0Xryv", "reviewer": "Reviewer_e9Q9", "summary": "This paper studies ensembles of M random feature networks when the number of features D is greater than the number of data points N (overparameterized regime). Large ensembles are found to be asymptotically equivalent to a single large network and convergence bounds are given for finite M. There are numerical experiments that illustrate the key results and implications for real networks are discussed.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review, as well as their suggestions for improving our paper. Below, we address the concerns and questions raised, and outline the changes we will make in response.\n\n**Claims And Evidence:**\n\n- **Clarification of \"hockey stick\" pattern in Fig. 2:** We agree and will adjust our wording in the revised manuscript to reflect that the \"hockey stick\" pattern is less pronounced for neural networks, likely due to effects that cannot be captured from an RF analysis.\n- **Interpretation of performance gap and variability in Fig. 3:** Thank you for highlighting this point. Indeed, the variability of the curve on the right arises due to variance across single model instantiations. We will update Fig. 3 accordingly, adding shaded regions to represent this variance. Regarding why the gap at the left grows, this is likely due to a) the discrepancy between the finite and infinite cases, and b) numerical instabilities when working with ReLU models (for a more precise explanation, see Appx. A.2). As an example of an experiment where this does not happen, compare Fig. 12 (right), which we deliberately did not include in the paper since this uses the Gaussian Error function.\n- **Plotting suggestion for Fig. 4:** Thank you for this suggestion – we will update Fig. 4 to use shared axes.\n\n**Relation to Prior Work and Additional References:**\n\nWe will add a brief summary of Abe et al. (2022) and cite Neal (1996) and Williams (1997).\n\n**Other Comments or Suggestions:**\n\nWe appreciate your edits and suggestions for clarity, which we will incorporate into our revised version. We address a few specific points below. \n\n- **Clarification of Assumption 2.1:** We indeed intended the product of the vector and scalar (which produces a vector again). Following standard extensions of subexponentiality to vector valued random variables, our theory relies on any linear combination of the entries of $w\\_i w\\_{\\\\bot i}$ being subexponential. We will clarify this point in the revision.\n- **Phrasing on line 250, right column:** We agree and will revise this phrasing accordingly.\n- **Clarification of Assumption 3.4:** We require that the entries of the expected value of the inverse matrix are finite, i.e., that the corresponding expected value exists. This obviously also requires that, for almost all instantiations of the matrix $\\\\Phi\\_{\\\\mathcal{W}}$, the matrix $\\\\Phi\\_{\\\\mathcal{W}} \\\\Phi\\_{\\\\mathcal{W}}^\\\\top$ is invertible.\n\n**Questions for Authors:**\n\n- **Relation to Ruben et al. (2024):** We agree that Ruben et al. (2024) should be discussed in more detail and will include this in the revision; thank you. We would also note that this work is concurrent to ours and should be read as such. Both works conclude that ensembles of overparameterized random feature models do not outperform a single larger model with an equivalent total feature budget. However, we explicitly focus on the overparameterized and zero/small ridge regime, while Ruben et al. primarily analyze generalization under optimal ridge parameters in both the over- and underparameterized regimes, with only brief consideration of our regime. Additionally, our analysis also explicitly considers uncertainty quantification, and we do not rely on Gaussianity assumptions, contrasting their Gaussian universality assumption.\n- **Ensemble variance and Bayesian uncertainty:** As briefly mentioned in Sec. 3.3, only \"with Gaussian features, ensemble variance admits a Bayesian interpretation,\" by which we mean that the ensemble variance matches the posterior variance of a GP with the same limiting kernel. We will ensure this is clearer in the revised version.\n- **Different ridge parameters in Sec. 3.4:** We appreciate this suggestion and agree it would be interesting to explore. Note, however, that when investigating the finite-width regime, we assumed $\\\\lambda \\= 0$, while in Sec. 3.4, we specifically discuss the infinite-width limit. Additionally, our current focus was explicitly on the small ridge regime, where any implicit regularization parameters (such as $\\\\tilde{\\\\lambda}$ from Jacot et al. (2020)) are expected to be very small; therefore, variations in this parameter are likely to have only a minor impact in our setting.\n\nWe hope these responses and adjustments clarify our contributions and address your feedback. If you have further questions or suggestions, we would be happy to address them.", "claims_and_evidence": "The main \"claim\" is that large ensembles become equivalent to large-width models. Besides the theory (reviewed below), numerical experiments were run with random features as well as neural networks (Figs 2, 3). Overall, I found the claims fairly clear and well-supported. There seemed to be some exaggeration of how well the numerical results fit with the theory. For instance:\n* Fig 2 right is said to show a \"hockey stick\" pattern, however this isn't nearly as obvious on the right (neural networks) versus at left (with RF ensemble). \n* Fig 3 (left and right) both show ensembles outperforming single models despite the main \"result\" being an argument that these are equivalent as the networks/ensembles get larger. In fact, the gap at left seems to be growing as the network grows, which isn't inconsistent with the theory and in fact the ensemble outperforms the kernel, its infinite limit. I think the issue here is the interpretation of the theory; the theory tells about the infinite limit but isn't very relevant in the finite case where ensemble will have a variance-limiting effect. In this plot it would help to show both means and shaded regions for variance across instantiations of single models/ensembles. My guess is the variability of the curve at right is due to variance across single networks. Can you explain this discrepancy?\n* Fig 4: I found this convincing; it might be better if plotted on the same axes.", "methods_and_evaluation_criteria": "The datasets and methods seem fine for a theoretical study. There wasn't much discussion of the effect of input dimension on the results. I don't think this affects things, but is there a high-dimensional regime when the input dimension is large where these results break down?", "theoretical_claims": "I read the main paper and found the results well-explained and intuitive. However, I did not check the proof supplement in detail.\nI have some comments on the clarity of the mathematical presentation that I will list below.", "experimental_designs_or_analyses": "I read the supplemental description of the experimental setup and didn't have any issues with that.", "relation_to_broader_scientific_literature": "I am not sure if the current results are super \"novel\" in that I think some of these results are known (e.g. Ruben et al, 2024). The idea that the variance across random features isn't indicative of a Bayesian measure of uncertainty, to me, wasn't surprising since I've always thought of this variance as being due to ensemble randomness rather than randomness in the training data. For instance, the theory behind using orthogonal random features or the FastFood random feature method was mostly concerned with keeping this variance low. \n\nI don't think this means the current work isn't important. I think the generality of the assumptions taken here make these results nicely applicable in settings that haven't been considered before.\n\nThe discussion of the paper mentions the work of Abe et al. (2022) as well as in the introduction, but to understand what \"recent empirical findings\" the current paper is supposed to reproduce, it would be helpful for the reader to get a quick overview of those.", "essential_references_not_discussed": "Early work by Radford Neal (1996) that was the first to connect random feature networks with kernels. I usually cite that and CKI Williams (1997) when writing papers in this area.", "other_strengths_and_weaknesses": "I found the paper overall clear and the results interesting. My main issues are with some of the clarity in presentation.", "other_comments_or_suggestions": "Typos/small points/clarification needed:\n* Eqn (1): Notation $[ . ]_j$ should be explained since it seems to refer to both column Nx1 and 1xN row vectors\n* pg 3: Similar to above, MATLAB-like [W; w] notation should be described.\n* pg 3: $\\phi^*$ isn't defined\n* pg 4 line 178: order of \"ridge(less)\" is reversed from least-norm and RR expressions that come later; I suggest reversing their order for clarity.\n* Assumption 2.1: The expression $w_i w_{\\perp i}$ is unclear, since $w_i$ is a vector and the other term is a scalar. Do you really mean their product or do you mean a column vector that comes from concatenating them? I suspect a typo here.\n* Line 250 right column: \"RF ensembles are equivalent to the ridgeless...\" I think you can, at best, say they are \"close to\" here given the evidence you present.\n* Lemma C.5: $z_i$ seems like it should be $x_i$.\n* Inconsistent notation \"Var\" and $\\mathbb{V}$ used for variance\n* Figure 5 doesn't seem to be referred to in the paper itself\n* Assumption 3.4: Can you clarify what you mean by finiteness of the matrix here? It would require the matrix to be invertible but also finite in expectation?", "questions_for_authors": "* Can you explain the discrepancy between the curves in Fig 2? This should be discussed in the paper\n* I am somewhat familiar with the results in the paper by Ruben et al (2024) that also study random feature ensembles using a different theory. These related results aren't discussed in much detail, although the conclusion \"NO FREE LUNCH FROM RANDOM FEATURE ENSEMBLES\" is close to those of the current paper. Can you discuss?\n* Can you discuss in more detail, perhaps at the end of section 3.3, when we expect the RF ensemble variance to capture Bayesian uncertainty or not? Does it ever work?\n* In Sec 3.4 you make the point about ensembles with the same $\\lambda$ converging. There are some results out there that show network width acts as an effective regularization (RF models at finite width are close to the kernel predictor with a modified ridge parameter, c.f. Bordelon, Canatar, Pehlevan, 2020). Wouldn't it then be best to use a different ridge parameter to compare ensembles with width $D$ to a single network of width $MD$?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper offers a clear theoretical analysis showing that in overparameterized regimes, ensembles of random feature regressors provide minimal advantages compared to a single larger model, supported by thorough proofs and corroborating experiments. Reviewers commended the work’s strong theoretical base and its integration of relevant literature. Despite some concerns about finite-sample bounds, slight gaps between theory and practice, and potential expansions to additional activation functions, the authors’ rebuttal and clarifications were well received, and the empirical results on both synthetic and real data are sufficiently compelling. The paper’s insights into ensemble variance, its role in uncertainty quantification, and its demonstration that ensembling mainly replicates capacity expansion rather than offering robust uncertainty estimates advance our understanding of deep ensembles, which is important for ongoing work in kernel-based learning theories. Overall, the reviewers agreed that the paper’s contributions, especially its concrete analysis of how classical ensemble intuition may fail in highly overparameterized scenarios, are timely and valuable, and I recommend acceptance.
Decision: Accept (oral)