Agree to Disagree: Demystifying Homogeneous Deep Ensembles through Distributional Equivalence
Authors: Yipei Wang
Xiaoqian Wang
Venue: ICLR 2025 Poster (2025)
Keywords: deep ensemble
trustworthiness
distributional equivalence
Primary area: interpretability and explainable AI
TL;DR: We discover a distributional equivalence property of homogeneouly trained DNNs, and derive rigorous theoretical results to reveal the true mechanism of deep ensembles, enabling various applications.

Abstract: Deep ensembles improve the performance of the models by taking the average predictions of a group of ensemble members. However, the origin of these capabilities remains a mystery and deep ensembles are used as a reliable “black box” to improve the performance. Existing studies typically attribute such improvement to Jensen gaps of the deep ensemble method, where the loss of the mean does not exceed the mean of the loss for any convex loss metric. In this work, we demonstrate that Jensen’s inequality is not responsible for the effectiveness of deep ensembles, and convexity is not a necessary condition. Instead, Jensen Gap focuses on the “average loss” of individual models, which provides no practical meaning. Thus it fails to explain the core phenomena of deep ensembles such as the superiority to any single ensemble member, the decreasing loss with the number of ensemble members, etc. Regarding this mystery, we provide theoretical analysis and comprehensive empirical results from a statistical perspective that reveal the true mechanism of deep ensembles. Our results highlight that deep ensembles originate from the homogeneous output distribution across all ensemble members. Specifically, the predictions of homogeneous models (Abe et al., 2022b) have the distributional equivalence property – Although the predictions of independent ensemble members are point-wise different, they form an identical distribution. Such agreement and disagreement contribute to deep ensembles’ “magical power”. Based on this discovery, we provide rigorous proof of the effectiveness of deep ensembles and analytically quantify the extent to which ensembles improve performance. The derivations not only theoretically quantify the effectiveness of deep ensembles for the first time, but also enable estimation schemes that foresee the performance of ensembles with different capacities. Furthermore, different from existing studies, our results also point out that deep ensembles work in a different mechanism from model scaling a single model, even though significant correlations between them have been observed.

Published as a conference paper at ICLR 2025
AGREE TO DISAGREE: DEMYSTIFYING HOMOGENEOUS DEEP ENSEMBLE THROUGH DISTRIBUTIONAL EQUIVA- LENCE
Yipei Wang, Xiaoqian Wang"
Elmore Family School of Electrical and Computer Engineering Purdue University
West Lafayette, IN 47907, USA
{wang4865, jow.ang]wwwapppommm por www.wung/puuuuueeee..
ABSTRACT
Deep ensembles improve the performance the models by taking the average predictions of a group of ensemble members. However, the origin of these capabil- ities remains a mystery and deep ensembles are used as a reliable "black box" " to improve the performance. Existing studies typically attribute such improvement to Jensen gaps of the deep ensemble method, where the loss of the mean does not exceed the mean of the loss for any convex loss metric. In this work, we demonstrate that Jensen's 's inequality is not responsible for the effectiveness of deep ensembles, and convexity is not necessary condition. Instead, Jensen Gap focuses on the "average loss" of individual models, which provides no practical meaning. Thus fails to explain the core phenomena of deep ensembles such as their su- periority to any single ensemble member, the decreasing loss with the number of ensemble members, etc. Regarding this mystery, we provide theoretical analysis and comprehensive empirical results from statistical perspective that reveal the true mechanism of deep ensembles. Our results highlight that deep ensembles originate from the homogeneous output distribution across all ensemble members. Specifically, the predictions of homogeneous models (Abe et al. 2022b) have the distributional equivalence property Although the predictions of independent ensemble members are point-wise different, they form an identical distribution. Such agreement and disagreement contribute to deep ensembles" "magical power". Based on this discovery, we provide rigorous proof of the effectiveness of deep ensembles and analytically quantify the extent to which ensembles improve perfor- mance. The derivations not only theoretically quantify the effectiveness of deep ensembles for the first time, but also enable estimation schemes that foresee the performance of ensembles with different capacities. Furthermore, different from existing studies, our results also point out that deep ensembles work a different mechanism from model scaling a single model, even though significant correlations between them have been observed. The code to re- re-implement all the experiments is open source
INTRODUCTION
In the studies of deep learning models, pursuing high performance is always one of the ultimate goals, where an enormous body of approaches regarding model architecture, training regimes or
data augmentations, etc. have been proposed (Krogh & Hertz 1991 Dietterich] 2000 Mnihh et al.
2013. Simonyan Zisserman 2014: Kingma Ba 2014 Lee al. 2015: He et al. [016]] Vaswani
et al. Tolstikhn et al.) 2021). 1). Among these approaches, ensembling a group of models has
been demonstrated very effective in improving the performance of models (Schapire) 1990; Breiman
1996 2001 Chen & Guestrin 2016 Ganaie et al. (2022). Different from traditional boosting/bagging methods, deep ensembles, on the other hand, form the ensemble models where the differences between ensemble members only caused by the randomness of the stochastic gradient descent (SGD) optimizations, i.e., the random initialization of models and the order of the presence of training
corresponding author

---

Published as a conference paper at ICLR 2025
data. (eee et al. 2015 Huang et al., 2017 Lakshminrayyanan et al. 2017). This approach is not only
simple but also very effective in improving the performance of models (Fort et al. 2019 Kondratyuk
et al. 2020) and the robustness (Chen et al. 2023 Huang et al. 2023). It also sheds light on various
applications such as bioinformatics (Cao et al. 2020), geoscience (Zeng et al. 2023), etc.
However, effective and powerful it is, the source of the capability of deep ensembles still remains a mystery, Many existing works thereby aim at exploring the true mechanisms of the deep ensemble method. Some studies focus on the mechanisms of deep ensembles via the relations among ensemble members, including their diversities, agreements, etc. (Fort et al. 2019 Theisen et al. 2023) Mathematical modeling for the relation between the calibrated negative log- likelihood (NLL) and the number of ensemble members is also proposed (Lobacheva et al.) 2020). Discouragingly. in recent work, it is discovered that the ensemble diversity neither responsible for improved uncertainty quantification nor the improved robustness (Abe et al., 2022b). Their findings conclude that deep ensembles still remain a reliable "black-box" method to improve the performance of DNNs. Whether the performance of large single models and ensemble of smaller models originate from the same
source remains unknown, too (Lakshminarayanan et al. 2017 Fort et al. 2019 Lobacheva et al.
2020 Kondratyuk et al. 2020 Abe et al. 2022a b).
Different from existing work, we study this problem from a statistical, and global perspective. Model Distribution: Because of the over-parameter nature, modern DNNs are capable of overfitting the training set (Goodfellow et al. 2014 Allen-Zhu et al.) 2019 Nakkiran et al., (021). Given a fixed model structure, stochastically trained models form specific distributions over the functional space. Training single models is then equivalent to uniformly drawing samples (i.e. trained models) from this distribution (Lakshminarayanan et al. 2017: Gawlikowski et al. (023). As consequence, deep ensembles, by taking the mean of the predictions of all M ensemble members, are implicitly estimating the expectation over such distributions. Different models make different predictions given the same samples, resulting in performance improvement in their means. Data Distribution: On the other hand, predictions for different samples also differ for given model, forming distribution of predictions over the output space. Existing work focuses on the diversity brought by the models, but overlooks the importance of the distribution caused by the stochasticity in the data distribution. The resulting point-wise evaluation of the model diversity Var [FF(() fails to reveal the true mechanism of deep ensembles. We consider the stochasticity from the joint distribution of both models and data and how the output distributions are affected. Through this approach, we provide rigorous and theoretical analysis of deep ensembles. Beyond the derivations, we point out that existing studies suffer from insufficient sample size (i.e. the number of ensemble members) in studying the statistics of nigh-dimensional and complex distributions. To resolve this issue and provide a comprehensive empirical validation of theoretical results, we study deep ensembles via M 100 ensemble members across various datasets and model structures. We also incorporate the long-standing debate between the effectiveness of deep ensembles and large single models (Lakshminarayaanan et al., 2017 Fort
et al. 2019 Lobacheva et al., 2020 Kondratyuk et al., 2020; Abe et al. 2022a|b).
Previous work has attributed the improvement in deep ensembles to the Jensen gap (e.g. (Abe et al., 2022b)), which is the difference between the loss of the ensemble (average) of individual models and the average loss of all individual models. When the loss metric is convex (e-g. Brier score, negative log-likelihood, etc.), this value is positive according to Jensen's 's inequality, which means that the loss of the ensemble is guaranteed to be smaller than the "average loss". However, we argue that Jensen's 's inequality is insufficient in explaining the predictive improvement of deep ensembles. This is because the average loss of a collection of models has no practical meaning what makes deep ensembles effective is the superiority to the loss of any single model instead of the average loss. And this is not affected by the Jensen gap. Besides, it is also consistently observed that increasing the number of ensemble members improves the performance, which is also irrelevant the Jensen gap. In this work, we uncover property of trained models in deep ensembles called distributional equivalence. That is, the homogeneous ensemble members, although differ in the predictions of every sample, form the same distributions over the entire data distribution. We thus provide both thcoretical analysis and comprehensive empirical results to demonstrate that it is this distributional equivalence across ensemble members that contributes to all the observed behaviors of deep ensembles. The paper is organized as follows. In section 2 we first review the recent efforts in uncovering the mechanism of deep ensembles and summarize their differences with our work. Then we introduce the setups and summarize the deficiency of existing understanding of deep ensembles in section 3 Afterward, we
introduce the main theoretical findings of distributional equivalence in sections 4 4nddd] and delve
2

---

Published as a conference paper at ICLR 2025
into the essence of deep ensembles. For the compactness, all proofs are deferred to appendix B The main contributions of this work can be summarized as follows:
We focus on understanding the mechanism of deep ensembles, and point out the problems of the existing approach that is based on Jensen's inequality.
We reveal the distributional equivalence property of trained models and provide theoreti- cally proven explanation for the effectiveness of deep ensembles.
Our theoretical findings provide schemes to accurately estimate the ensemble performance regarding the number of ensemble members and to use only two models to foresee the asymptotic loss of deep ensembles with infinite capacity.
All theoretical findings are verified through extensive experiments carried out in this work.
RELATED WORK
Intrigued by the simplicity of implementations and the impressive performance, many studies have been carried out focusing on the internal mechanism of deep ensembles and trying to explore the reason why works. We hereby summarize these most related works and how our approach and discovery are connected and different from them.
Lobacheva et al. (2020) focus on the pattern of the negative log-likelihood (NLL) of deep ensembles in classification tasks regarding the number of ensemble members. An empirically verified power law is proposed for the calibrated NLL. The power law provides a guideline for designing the ensemble system and determining the most efficient number of ensemble members. However, one drawback of this work is that it's fully empirical, which can give rise to trustworthiness issues in the results. Abe al. (202a b) provide insightful studies of deep ensembles the improved uncertainty quantification ( (UQ) and the improved robustness. However, it is concluded that the Jensen gap measures the expected predictive improvement through ensembling, which we demonstrate in the later section is insufficient. Besides, Abe et al. (2022b) also admits that deep ensemble is still a "reliable 'black box' to improve the performance of DNNs. Another similar but slightly different ensembling scheme is through majority vote (Masegosa et al., 2020 Theisen et al. 2023), where the final prediction is obtained by the vote of all ensemble members instead of the average of them. However, the theoretical studies for this variant are limited by its generality - when all the ensemble members are considered general functions, the conclusions become very weak. For example, [Theisen al. 2023) point out that the theoretical error of the majority vote classifier can be even worse than the average error rate, which is far from the empirical observations. This indicates that important conditions satisfied by the distribution of the ensemble members and the data are overlooked, which we highlight in this work. As for the empirical study, the majority vote typically requires training a larger number of ensemble members compared with standard deep ensembles.
Many existing works also point out the similarity between performance improvement from (1) increas- ing the capacity of a single model and (2) ensembling many small models (e.g. (Lakssminnaayaaaal et al. 2017 Abe et al. 2022b), etc.), where empirical comparisons between them are carried out. Geiger et (2020); Kobayashi et al. (202)) carry out empirical studies and discover specific bias- variance trade-off curve for deep ensembles, where increasing the number of ensemble members after the optimal ensemble underscores the ensemble performance. This phenomenon also explained by the distributional equivalence property highlighted in our work.
DEFICIENCIES IN EXISTING STUDIES
3.1 SETUPS
Notations. In this work, we focus on the classification problem with the dataset D X' ), where X' C IRd and y = {1..... c). Here d, denote the input dimension and the number of classes, respectively. Let f Rdd - - denote the model that maps from the input space to the standard (c - s)-simelex. Such models are optimized stochastically with the objective ming (e Xhaia ((()))) w). where 0 denotes the weights of ʃ and omitted for simplification. Specifically, let fy : IRd + [0, 1] denote the predicted probability of the target class, where the subscript y is omitted in notations for compactness. Based on the model f. the metrics are defined as é (0, 1) + R,, which is determined by the specific type of metrics. For example, the Brier score defined as (f((()))
(1 f()))² And for negative log-likelihood (NLL),
'The original Brier score is defined over the entire prediction [[())) 1.112. Here we focus on the attribute to the target class. The differences are discussed appendix 1.10
3

---

Published as a conference paper at ICLR 2025
*('(
Performance (Lose)
x[pi0]
fr
a T 6l00
1fooo)
2000
Non-Monotonic Metric
Monotonic Metric
10000
Input Indices
(a) Non- Non-Monotonic Metric vs Monotonic Metric.
(b) Point-Wise LG JG
Figure I: (a) An illustration of JG vs LG in the M = scenario. Since (G(x) 2 0 always holds, provides no insights into how deep ensembles work. On the other hand, (G(x) 0 possible only
for non-monotonic metrics. (b) The point-wise prediction of the best single model (blue), ensemble (orange), and the average error (green). Samples are sorted by the ensemble predictions. The results are generated from the CNN model with & = 10 and CIFAR-10.
(n)()())) - log f(x) The models are evaluate on Mest Visse throughout the paper. Thus for brevity, we omit the subscripts.
Homogeneous Ensembles.
The deep ensemble approach is implemented by taking the average of a group independently optimized models as the final predictions. Homogeneous ensembles further refer to the variant where all ensemble members have identical structures and training hyper- parameters and only differ in the training stochasticity. Therefore, a trained ensemble member F ~ PF follows distribution pF. which is probability measure the functional space F. And supp(pr) C F. denotes the support of pr. Formally, homogeneous ensembles take set of M models {F = f(9) } which are considered i.i.d. in pr, and predict in the following way:
1 1 Σ (() (x))
f(x) Ep-ps F(x));
f(x;
M
Surprisingly, such simple scheme can drastically improve the performance (i.e., decrease the loss value). Since all members of the ensemble share identical structure, the optimization process can be viewed as random sampling from the model distribution pF.
Experiment Setups. To empirically validate our theoretical findings, we carry out comprehensive experiments, which are presented along with theoretical results. Each model structure determines model family F and the distribution pF over F. Varying the model architectures in CNNs and ResNets and capacities in width determined by & E {10, 20, 40, 80, 160), we include a total of 10 pF in experiments. For each pr, train M 100 models for three datasets: 1IFAR-10/100 and Tinylmagenet. i.e., total of 2 x x 3 x M 3000 trained models. The training parameters follow the suggestions in Nakkiran al. 2021 The setup for experiments detailed in appendix
3.2 UNDERSTANDING DEEP ENSEMBLES THROUGH JENSEN'S S INEQUALITY
Existing analysis regarding deep ensembles attributes the improvement to the Jensen Gap. Let æ e X be input data. Then given the functional distribution pr of the homogeneous models and a convex metric ø, it is according to Jensen's 's inequality that
o (Er~pr [F(x)]) S [0
(2)
That is, for any single input X ~ Px, the "error" of the ensemble no greater than the average error among all the members of the ensemble. Then the Jensen gap JG(x) = EF-py (f(()))) d(Fr<pF [F))] > is used as a measurement for the "predictive improvement" of the ensemble scheme. For example, Abee et al. (2022b) point out that for Brier score Φg, the Jensen gap is equivalent to the wise variance of the model distribution PF at X x.
3.3 THE FAILURE OF EVALUATING THE "AVERAGE Loss"
While eq. (2) provides seemingly reasonable and plausible explanation for the effectiveness of the deep ensemble scheme, it fails in practice. An important deficiency is that the average loss E. Ep-pe (x:)) does not possess any practical meaning, and thus having lower loss values than
4

---

Published as a conference paper at ICLR 2025
Table 1: A demonstration of the three explanations for the effectiveness of deep ensembles. The point-wise Jensen gap always holds, but has no practical meaning. The point-wise lower gap being positive can be strong evidence for the effectiveness of deep ensembles. However, never holds. The global gap defines why deep ensembles are always preferred. And it is proved in this work.
Types
Fornula
Posiivvit
Signilicance
Feusibility
Gap
Epwpy [o([(=)) dFE [F()))
Always 2
Lower Gap
minsez [d)) deR (F(m)))
Always < 0
Global Gap
min (Bx (o(](=)))] Bx (xle(Ex(P(X)))
Proved to be 2
this mean does not contribute to the success of deep ensembles. Jensen's 's inequality always holds, regardless of the instantiations of the ensemble members. Thus using Jensen Gap to explain deep ensembles' effectiveness suggests that for any group of models { Ji-1 the ensemble of them is always desired. This is trivially false. On the contrary, Jensen's inequality eq. (2) allows the possibility that 3f E F s.t. (x)) < ((EF~pp [F()]]). In such scenarios, some single ensemble members may outperform the ensemble itself. Note that ensemble is useful only if it outperforms all ensemble members. Therefore, practice, what really matters is the difference between the best individual (minimized loss) and the ensemble, which is denoted the lower gap (LG):
I.((r) min {d[[(=))) < (Er-pe [F(+)) JEF
LG provides reasonable intuition for the ensemble. It suggests that deep ensembles desired if they outperform all individuals. This provides practical meanings compared with JG.
3.4 THE FAILURE OF POINT- WISE EVALUATION
Point-Wise Gap. Note that although LG provides more practical understanding compared with JG, LG(x) 2 0 can be an exceptionally strong condition to satisfy. In fact, LG(x) 2 0 is possible only if the metric o(-) is not monotonic. Formally, we prove the following proposition in appendix B.
Proposition 3.1. LG(=) S 0 always holds for any monotonic metric 6. That is, given input sample x, deep ensemble always underperforms the best individual member.
The M = 2 scenario is visualized in fig. (a). It can be observed that the monotonicity of the metric Φ prohibits (=(x)) from being positive since the average prediction is always no greater than the highest prediction (i.e. higher loss). fig. (b)) shows an empirical result of the NLLs of CIFAR-10 and CNN models. The 10000 testing samples are sorted by the ensemble prediction for better presentations. It verifies that LG(x) < 0 and JG(æ) 2 0 always hold.
Global Evaluation. Although deep ensembles always underperform at least one individual member for any input sample, the ensemble scheme is still plausible in practice. This shows the deficiency of the wise evaluation. What is always overlooked in the studies of deep ensembles is the importance of the distributions of the data px, whose support is X. Ensemble is desired if the Global Gap (GG) is always positive:
GG = mmin fE3
(x(())) Ex-ax (F(X)) 2 0
However, universally one only prove that both RHS and LHS less equal to Ex-ex [E/ (x())) which means eq. (4) does not hold in the universal sense. Instead, is the distribution of (X, F) ~ px @ Pr that determines whether deep ensembles are beneficial.
The three types of explanations compared more straightforwardly in table I
DEMYSTIFYING DEEP ENSEMBLES
4.1 THE DISTRIBUTIONAL EQUIVALENCE PROPERTY
We start the investigation of px @ PF by introducing the distributional equivalence property, where the prediction forms identical distributions. Since F PF, X ~ px, the predicted probability é = F(X) is also a random variable & ~ Pt. Distributional equivalence refers to the property that for any two models f()) f(2) E F. the conditional variable f|F = f()) and f = f(2) follow the same distribution. In fig. a).
a). the kernel density estimation of F = f()) for all 10000 samples of CIFAR- 10 are visualized. There are 100 individual models, distinguished by the colors of curves.
can be qualitatively observed that P( (() P( f() for any f(0), f( F.
5

---

Published as a conference paper at ICLR 2025
1.-
Prodictedd Protabiee
Mesiiced Porabkay oreh Darger Cass
4200
6020
10020
(a) Distributional Equivalence
(b)
Distinction
Figure 2: The demonstration of (a) the distributional equivalence property and (b) the point-wise distinction property. The kernel density estimations F f()) ( 1, .M for M 100 models are plotted in (a). Conditioning The point-wise predictions are presented in (b), where 10000 testing samples are arranged according to the prediction of the ensemble (black curve). The point-wise predictions of the first 5 individual models are plotted in different colors. Each point represents specific prediction f(0(x) (æz). It is -
different models results in the identical distribution of predictions.
demonstrated clearly that although models tend agree on certain samples, the point-wise predictions vary significantly. The models are all CNN with k = 40, and the dataset is CIFAR-10.
Court
(a) KS Test
(b) The distribution of Wasserstein distances
Figure 3: The quantitative results in validating the distributional equivalence property. (120) = 4950 model pairs are evaluated. (a) shows the joint distribution of the p-values and KS statistics, where the red dashed line shows the level of p = 0.05. (b) shows the distribution of the 4950 Wasserstein distances between model pairs
We employ two metrics to validate the distributional property of F = The Kolmogorov-Smirroy test evaluates the null hypothesis that the conditional distributions F = f(()) and &|F = f() drawn from the same underlying distribution. While the KS is powerful, it's important to note that the null hypothesis becomes overly strong with the increases in sample size, potentially leading to statistically significant results for practically insignificant differences in large datasets. To complement this, we carry out the KS test over the first 1000 testing samples and also utilize the Wasserstein distance to quantify the minimal cost of transforming one probability distribution into another. This metric provides a continuous measure of dissimilarity between distributions, offering a more nuanced comparison of the conditional distributions than the binary outcome of the KS test. The results are presented in fig. 3 Based on M 100 models sampled from pF, there (100) 4950 pairs in KS tests and Wasserstein distances. It is observed from (a) that the statistics are all extremely small, with almost all p-values greater than 0.05. From (b), Wasserstein distances are very small compared with the domain of probability [0. 1]. Both (a)(b) verifies the assumption that F f() P({ f()) for any f() f() F. These results demonstrate that within a fixed model distribution pF (i.e. fixed training data and stochastic training schemes), the trained models have the same predicted distribution F f. This is weaker than point-wise equivalence, but much stronger than the equivalence under metrics (e.g. NLLs, Brier scores, etc.).
Point-Wiis Distinction. Note that distributional equivalence does not imply point-wiss equivalence. On the contrary, the point-wise diversity Var Fupp (F(z)) among ensemble members is always
considered important factor (Fort al. 2019 Abe al. 2022a Theisen et al. 2023). As the

---

Published as a conference paper at ICLR 2025
point-wise diversity approaches zero, all the ensemble members become point- wise identical and thus the ensemble becomes equivalent to each model. Therefore, it is the point wise distinction and the distributional equivalence together that contribute to ensembles' power. The point-wise predictions are visualized in fig. b)). The 10000 testing samples are sorted by the predicted probability of the target class, where the black curve represents the ensemble prediction. The scatter points represent the predictions of 5 individual ensemble members. It is observed that different models have distinct point-wise predictions but share similar trends. More figures are shown in appendix D
4.2 THE DISTRIBUTION OF MODELS
Although point-wise diversity plays an important role in ensembles, being fixated on a single a E X fails explore the true mechanism of ensembles. This is because globally, the distributions of é == f(X are always affected by the neural collapse phenomenon. According to the overfitting property of modern deep models, trained model f ~ py achieves 100% accuracy on the training set and the training loss becomes sufficiently small. The studies of Neural Collapse (NC) further demonstrate that the terminal phase of training, discriminative models tend to collapse to the exact dimension of the target space [Papyan et al. 2020). For classification tasks, the predicted probability collapses and converges to either zeros or ones as the model becomes overconfident in both correct and wrong predictions (Hein et al. 2019).
As a result, we consider the trained model distribution pp under complete neural collapse. That is, F=f((z) = 1 (arg max f(m)=)) If E F). The corresponding distribution defined on f is written as Dr(F=f) Pr([f FI (arg max f(X)=))- = =f)). This can be limit of pr pr as the training procedure continues infinitely. Due to the infeasibility of the continuous distribution of / F(X). we utilize pr as surrogate to pr in some theoretical derivations. It should be noted that this does not sacrifice the practicality of the theoretical analysis because of the high resemblance between pF and pp. In fact, all empirical results are carried out using pF, demonstrating the transferability between pr and Pr. The ensemble under the complete neural collapse is defined as
approximation of the
f()) = Ep-pr [F(x)] = EF-pe [lrg max F(m)
(5)
Thus for the collapsed approximation fr of models, the conditional distributions of the prediction un- der fixed sample a or fixed model f can collapse to Bernoulli distributions: F Bernoulli(p).
X = x ~ Bernoulii (f (z)). As result, the point-wise diversity can be written as
Var (F(x)) -EF [F(x)²] - EF-Ør [xx)]
(6)
-EF-ge [F(x:) Er-ar )(x))) f(x))
This shows that for modern DNNs with neural collapse, the point-wise diversity is minimized either when most models make the correct prediction or when most models make the wrong prediction. Without considering the distributions pv @ px, such strong correlation between point-wiss diversity and point-wise prediction cannot be discovered and the analysis on point-wise diversity deviates from the practice. For example, Abe et al. (2022a) discovered that the point-wise diversity equals to the Jensen gap of the Brier score, which falls short due to the drawbacks discussed section 3 has been demonstrated in section 4.1 that F = f are identical regardless of f. We thus assume the following condition throughout the derivations.
Distributional Equivalence: v FF f(0) P({)F (())
This leads to the distributional equivalence of Pr: Vf f() F. P(FF =
4.3 THE DISTRIBUTION OF DATA
According to the distributional equivalence across models, we note that the point-wise difference can be mitigated when we consider the entire data distribution px. All the models have the same expected predictions over the entire dataset. Based on this, we then prove the following theorem that explains the improvement of ensembles.
Theorem 4.1. (Guaranteed Improvement) Given convex metric O we have:
Exxpx [EFr))[F(X)]]
< min feF
Ex Exx [
(8)
7

---

Published as a conference paper at ICLR 2025
024
050
8.5
Z-46
034
2.40
612
23
60
330
20
40
102
(a) CIFAR-10
(b) CIFAR-100
(c) Tinylmagenet
Figure 4: The estimation (regions) and the empirical results (solid curves) of the Brier score of original models. The z-axis represents the number of ensemble members (a.k.a. ensemble capacity), and the y-axis represents the Brier score of the ensemble. Dashed lines represent the standard deviations. The tested model CNN.
The proof presented appendix B.3 This shows the condition eq. (4 where deep ensembles are preferred under neural collapse. It demonstrates that deep ensembles always outperform any individual ensemble members (instead of the average loss).
4.4 THE FACTORS OF THE ENSEMBLE IMPROVEMENT
Although it has been proved above that deep ensembles are guaranteed to outperform any single member, the extent to which the improvement and what factors contribute to the improvement remains unknown. Therefore, delve deep into the quantification analysis of the improvement.
Instantiating & to the Brier score őB. we have the following theorem to bound the performance shift. Theorem 4.2. (The Brier Score Improvement) The ensemble performance improvement of the Brier score is determined by:
A B min Ex px j0 ( (f(x)) - -Ex~px JoB (EF~br (F(X)) p D0--p-Vapxx [(()] (9) feF
where p = Ex [f) for ff E F. AB = 0 holds if and only if point-wise equivalence holds.
An = p- p² holds if and only if the ensemble predicts constantly for all samples.
The proof is deferred to appendix B.5 The result shows that when the expected prediction of pr is determined (as p). then the global diversity Varx~px [XX] is responsible for the improvement. To our best knowledge, previous work has been focusing on point-wise diversity (i.e. variance over PF) while the importance of the global diversity (i.e. variance over Px) has not been recognized before. Now it suffices determine how the global diversity changes regarding the joint distribution Px < pr. Recall from eq. that if individual models tend to agree more each sample, the point-wise predictions become closer to either or This will lead to a higher global diversity since the ensembled predictions are deviated from the center. Such agreement can be measured by the probability measure of the shared correct prediction between two models. In fact, We have the following theorem proved in appendix B.6
Theorem 4.3. Let F-1(1) = {a E F(x) = 1) denote the subset of X that F can predict correctly, then Varx=px [f(x) = E
This quantifies that the global diversity of pr decreases with p and increases with the expected agreement across models. We emphasize that this is not counterintuitive since the globalness is regarding px instead of pp. One important significance of theorem 4.3 that due to the distributional equivalence property, trained models F Py demonstrate great symmetry, leading to very small variance in Px (FT n F5- (1)). Thus it can be estimated by only two models with small errors. foresee how the ensemble will perform with only two models. Besides, due to the associative and commutative nature of multiplication, theorem 4.3 can be migrated to the original function space F. We prove the following theorem in appendix B.7
As result,
Theorem 4.4. The Brier score of the ensemble f can be estimated by
Ex-px ( (bn((( (xX)) Ep [Ex=px (F [X)) 2Er<pF Exxpx [F(X)]] +1 (10)
8

---

Published as a conference paper at ICLR 2025
Due to the distributional equivalence across all models, estimate this by f(), F:
E Ex~px (xx(f(X)) xe Ex-px
2Ex=px (x)) (x) +1
(II)
This is verified empirically in fig. 4 by using all 9900 possible pairs to estimate the ensemble performance. The mean and standard deviation of the estimation are visualized the regions, while the solid curves represent the loss of the ensembles. Therefore, we can use only two models to accurately estimate the asymptotic loss of deep ensembles. Using this scheme, practitioners can make judicious and efficient choices in the application of deep ensembles and balance the trade-off between training budget and performance requirements.
Bias-Var Variance Trade-Off of Ensembles. Another intriguing yet unsolved phenomenon of deep ensembles its bias-variance trade-off the single-model capacity of each ensemble member increases (Geiger et al.] 20200] Kobayashi et al. 2021). Note that be written as a function of p = Ex [XX) for any f E F as S(p) [x[ 2Ex [fX)] +1 p² 2p 1+ [f From fig. 5 we know that [f tend to increase with p. Thus the Brier score S(p) starts increasing before p reaches 1 (as the ungle-model capacity increases). This leads to the bias-variance trade-off observed for deep ensembles.
0.15
0.14
the ensemble performance under the Brier
0.12
-H Clb, C CfR-JC
0.08
Cls, CBAR-D0
-96- CW,, Thy rageret
Rahe: O1A4-20
0.2
0.4
0.6
0.8
5 ENSEMBLES OF LIMITED BUDGETS.
In practice, approximation with M models is utilized. It has been observed many existing works that the loss decreases as M increases. This phenomenon has not been understood ei- ther. Previous work simply terms M as the "ensemble capacity", which does not contribute the understanding of the internal mechanism given the M models are trained completely indepen- dently. It is also observed that increasing the ensemble capacity yields similar performance improvement compared with increas- ing the capacity of single model, as shown in fig. (More results are shown in appendix E).
Figure 5: An illustration of the relation between [x]]] and Varx[f(X)]. All six structure- dataset combinations are tested here, P is controlled by the single- model capacity k of ensemble members. For each curve, p in- creases the width parameter k.
To determine the efficiency of ensembles regarding M and also unveil the connection between mongle-model capacity and ensemble capacity, we prove the following theorem in appendix B.
Theorem 5.1. The expected Brier score loss of M ensemble members is determined by
Ep Parffr [
1 M
E
M
F((X))
-E -Ex~px [xn(f(X)]] Exxpx
VarF= (F(x))
(12)
M + E Ex [f(X)
p² M
2p 2pp+ +1
(13)
M
The expected NLL of M ensemble members is estimated by
M
1 1-f(x) 2f(x) M
Ep.... Ep......xurrrr @nnll
Σ (((X))]
Exxpx ((nu(F(x)) + Ex
(14)
M
This theorem explains the constantly observed phenomenon where increasing the number of en- sembles results in improved performance. Demonstrated by eqs. (12) and (14). the loss can be decomposed the loss of the convergence result the population means and another positive term controlled by M. Thus the loss decreases in an tnverseprroportione number M of ensemble members, i.e. the ensemble capacity. As M + xc, the loss converges to the global loss of the population ensemble. Note that all values are determined by pF except for the ensemble budget M. As a consequence, this theorem presents a closed-fome expression for the performance of deep ensembles. On the other hand, demonstrated in appendix D scaling up single model changes p instead, i... the prediction a single model. This theoretically verifies that both single-model capacity and ensemble capacity contribute to performance improvement but in
manner we increase the

---

Published as a conference paper at ICLR 2025
Baeck-30 20
Ens e- 66
257
Ens
Ens 66
2.0
dta:k 200
Model
5.1
0.
3.73
111
0.-
1.33
0.4
1.25
32a
10'
100
32a
14
30'
22"
10
20'
20
10
10"
157
Ructeveters
Fammettes
Pootecters
(a) CIFAR-10
(b) CIFAR-100
(c) CIFAR-10
(d) CIFAR-100
Figure 6: The performance comparison between the scaling of single model (black dashed curves) and increasing the number of ensemble members (colorful solid curves). In each ensemble (i.e. each colorful curve), M varies from 100. The results generated using CNN models.
642
B+30
2.5
4.0
6:54
610
(a) CIFAR-10
(b) CIFAR-100
(c) CIFAR-10
(d) CIFAR-100
Figure 7: The verification theorem 5.1 The theoretical results (dashed curves) are compared with the empirical results of ensembles using M models for (a)(b) the Brier score and (c)(d) the negative log-likelihood. The ensemble curves are computed using pp. Results are generated using CNNs.
completely distinet ways. We validate these theoretical results in fig. 7 where the empirical loss and the theoretical decomposition are compared. Note that although the theorems are derived for f with complete neural collapse, it generalizes to FF due to the already existing neural collapse nature in also carried out for F to demonstrate the practical
trained models. Therefore the experiments significance. It is observed that the curves almost completely overlap. The estimation of NLL deviates
little the beginning due to Taylor's expansion approximation. Here clip (1 j ((x))((2((X)) by an upper bound of 5 to avoid explosion caused by difficult samples to avoid overflow issues.
Besides, inspired by theorem 4.4 the Brier score be rewritten as eq. 13 where the first term be estimated by the agreement between two single models. We thus estimate the entire curve using only two models, and plot the standard deviation the colorful regions fig. ab) This provides even further and deeper understanding compared with fig.
CONCLUSIONS
In this work, focus the important problem of demystifying the "mysterious effectiveness" of deep ensembles. We diagnose and demonstrate the problem of existing ways of understanding deep ensembles and the inconsistency in practice. Then we reveal the distributional equivalence property of model distributions through comprehensive experimental results. Based on such property, we provide theoretical analysis with rigorously proved theorems to (1) demonstrate the guarantee of deep ensembles' effectiivnnsss; (2) propose a scheme to accurately estimate the asymptotic performance of infinitely many models using only two models; and (3) lucidly uncover the essence of increasing the ensemble capacity and scaling up single model. In conclusion, our work discloses the true mechanism of mysterious deep ensembles for the first time. The derivations also provide valuable insights into the understanding of the behaviors of models through the perspective of the joint distribution between data and models. We admit that the distributional equivalence property is an observational conclusion. It remains intriguing question why the probability measure of the testing data mastered by any single model retains an identical level across different models. We deduce it's due to certain properties of distribution shift and leave that for future explorations.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was partially supported by the EMBRIO Institute, contract #2120200, a National Science Foundation (NSF) Biology Integration Institute, and NSF IIS #1955890, IIS #2146091, IIS #2345235.
REFERENCES
Taiga Abe, E Kelly Buchanan, Geoff Pleiss, and John Patrick Cunningham. The best deep ensembles sacrifice predictive diversity. In Can't Believe s Not Better Workshop: Understanding Deep Learning Through Empirical Falsification, 2022a.
Taiga Abe, Estefany Kelly Buchanan, Geoff Pleiss, Richard Zemel, and John P Cunningham. Deep ensembles work, but are they necessary? Advances in Neural Information Processing Systems, 35: 33646-33660, 2022b.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. International conference on machine learning, pp. 242-252. PMLR, 2019.
Leo Breiman. Bagging predictors. Machine learning, 24:123-140, 1996.
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
Yue Cao, Thomas Andrew Geddes, Jean Yee Hwa Yang, and Pengyi Yang. Ensemble deep learning in bioinformatics. Nature Machine Intelligence, 2(9):500- -50000055,,-00 2020.
Huanran Chen, Yichi Zhang, Yinpeng Dong, and Jun Zhu. Rethinking model ensemble in transfer- based adversarial attacks. arXiv preprint arXiv:2303.09105, 2023.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pp. 1-15. Springer, 2000.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv preprint arXiv:1912.0275:: 2019.
Mudasir A Ganaie, Minghui Hu, AK Malik. M Tanveer, and PN Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:105151, 2022.
Jakob Gawlikowski, Cedrique Rovile Njieutcheu Tassi, Mohsin Ali, Jongseok Lee, Matthias Humt, Jianxiang Feng, Anna Kruspe, Rudolph Triebel, Peter Jung, Ribana Roscher, al. A survey of uncertainty in deep neural networks. Artificial Intelligence Review, 56(Suppl 1):1513-1589 2023.
Mario Geiger, Arthur Jacot, Stefano Spigler, Franck Gabriel, Levent Sagun, Stéphane d' A'scooli Giulio Biroli, Clément Hongler, and Matthieu Wyart. Scaling description of generalization with number parameters deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2020(2):033001, 2020.
Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv preprint 1412::411 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
11

---

Published as a conference paper at ICLR 2025
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 41-50, 2019.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train get for free. arXiv preprint arXiv:/704.001/ 2017.
Hao Huang. Ziyan Chen, Huanran Chen, Yongtao Wang, and Kevin Zhang. T-sea: Transfer-basee self-ensemble attack on object detection, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20514 15514-20523 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /4/2.69800 2014.
Seijin Kobayashi, Johannes von Oswald, and Benjamin F Grewe. On the reversed bias-variance tradeoff in deep ensembles. ICML,, 2021.
Dan Kondratyuk, Mingxing Tan, Matthew Brown, and Boqing Gong. When ensembling smaller models is more efficient than single large models. arXiv preprint arXiv:2005.005 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv:1511.063 14, 2015.
Ekaterina Lobacheva, Nadezhda Chirkova, Maxim Kodryan, and Dmitry P Vetrov. On power laws in deep ensembles. Advances In Neural Information Processing Systems, 33:2375-2385, 2020.
Andrés Masegosa, Stephan Lorenzen, Christian Igel, and Yevgeny Seldin. Second order pac-bayesian bounds for the weighted majority vote. Advances in Neural Information Processing Systems, 33: 9263-5273 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Prectum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 202 1240:3 2021.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Robert E Schapire. The strength of weak learnability. Machine learning, 5:197-227, 1990.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1509.156,, 2014.
Ryan Theisen, Hyunsuk Kim, Yaoqing Yang. Liam Hodgkinson, and Michael W Mahoney. When are ensembles really effective? arXiv preprint 3rXiv:2305.12313. 2023.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34:24261- 24272, 2021.
12

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Yipei Wang and Xiaoqian Wang. "why other classes?": Towards class-contrastive back- propagation explanations. Advances in Neural Information Processing Systems, 35:9085-9097, 2022.
Taorui Zeng, Liyang Wu, Dario Peduto, Thomas Glade, Yuichi S Hayakawa, and Kunlong Yin. Ensemble learning framework for landslide susceptibility mapping: Different basic classifier and ensemble strategy. Geoscience Frontiers, 1416)1001445, 2023.
13

---

Published as a conference paper at ICLR 2025
A EXPERIMENT SETUPS
To carry out empirical demonstration simultaneously with the theoretical analysis, we introduce the detailed experimental setup here. Experiments are carried out on Intel(R) Xeon(R) Gold 6226R CPU 2.90Hz with NVIDIA RTX A5000 GPUs.
The experiments carried out three datasets, CIFAR- 10/100 (Krihhessy al. 2009) and Tinylmagenet (Deng et al. 2009). For different model architectures, following a similar setup NNakkiran et al.) (021), ), consider both CNNs and ResNets(He et al. 2016). On the one hand, for each pr. M models are trained to obtain a comprehensive understanding of the influence of the ensemble size. The empirical studies of the properties of distributions of models require a large number of samples (i.i.d. trained models) due to the high nonlinearity and dimensions. However, in previous work, this has been effectively addressed with sufficiently large population (c.g. M = 4 in (Lee et al., 2013). , M 5 in (Abe et al. 2022b), M 14 in (Fort et al. 2019), etc.). To study the distribution of a homogeneous ensemble of models, we train M 100 models with distinct random seeds. On the other hand, note that increasing the ensemble size and increasing the capacity of single models both contribute to an improved performance. However, the relation between the mechanisms of these two distinct approaches remains unknown. Some empirical studies on the correlations even suggest that their mechanisms agree with each other. As a result, we set multiple different capacities for each architecture. Specifically, we through the width. CNNS contain a 4-layer convolutional layer with output channels [A: 2k, 4k, 8k), each of which is followed by batch normalization layer and ReLU activation. We also add max-pooling layer after the 2k, 4k, 8k convolutional layer, with strides 2. 2, 8. (For Tiny Imagent,, we use 2, 4, 8 instead.) Finally, linear classification head takes the 8k-dimensional embedding and maps it to the output space. Regarding ResNets. we scaled the width of ResNet-18 18 as suggested by (Nakkiran et al. 202I). The channel of each layer is linearly determined by where = 64 results in the original R In order to explore the influence of model capacities, we train DNNs with E K, where K = {10, 20, 40, 80, 160). Note that for single model capacities, the number of trainable parameters increases quadratically with respect to the factor k, while the trainable parameters of the entire ensemble increases only linearly with respect to the number of ensembles M.
factor i to control the capacity
As suggested by the essence of deep ensembles, SGD is used as the solver with momentum or data augmentation. Let i denote the number of epochs; we use the learning rate x(t) = 2g where Ao = 0.1 is the initial learning rate.
V/.
B
PROOFS
B.1 PROOF OF PROPOSITION 3.1
Proposition 3.1. LG(x) S 0 always holds for any monotonic metric Φ. That is, given input sample a, deep ensemble always underperforms the best individual member.
Proof. When & is monotonically increasing. Then
min()((())) = {f
(15)
fer
Jer
Since minser (x())) < EF [F))] and Φ is monotonically increasing, we have
@(min (x))) S ((Er~pp (F(x)))
(16)
fEF
Combining them results in
min min(o(f(z))) < @(E [F(x))) JeF
(17)
Therefore
L.(()) = min min()(((=))) - (E (F(x)) S 0
(18)
/EF
In the opposite, when @ is monotonically decreasing. Then (max((())))
min(o((()))} o (max [())))
(19)
JEF
14

---

Published as a conference paper at ICLR 2025
Since max 2 Ep-pr [[(x]] and o is monotonically decreasing, we have
(f I @(Br~pr (F())) JEF
(20)
Combining them results in
min min()(](a))) S @EEr>pp [F(x))) EEF
(21)
Therefore
T.G(a:)) = min min((([(a))) (Errpr (F(=))) S 0
(22)
/EF
This proves the proposition. _
B.2
LEMMA B.1
For the model ff FF with complete collapse, the prediction is either 0 1. Thus we have the following lemma:
Lemma B.1. (Prediction Distributional Equivalence) 3p E [0, x. vf E ft, - E
Rd f(x) = f then (U = p.
Proof. According to the distributional equivalence condition, for arbitrary f.g E E P(f(x)) = P()(x)) holds for Vaz E X, and thus Ex-px [f()] = (x()) Therefore, 3p E [0, 1] s.t.
for arbitrary f E F, Ex-xxxxxx [f(X) = p. Then
1 px(x)) [ 1 px /x
0 px(x))d
(23)
IxyU
1 (x)xxx(x) Exapx (X)] =p
(24)
This proves the lemma.
B.3 PROOF OF THEOREM 4.1
Theorem 4.1
(Guaranteed Improvement) Given a convex metric é, we have:
Ex-px ((Ers))(((X)) < min JEF
Ex-ex [
(25)
Proof. It suffices to show that for an arbitrary 9 E F
Ex-px (Eprgr [(()))) s Exxpx [o(O(X))]
(26)
Note that according to theorem B.
RHS
((0(z))px (x)dx
(27)
px(x))(1)dz + J, px()))(0)d
(28)
x(1)p (00 p)
(29)
20(1 ++ (1 p)) d(p)
(30)
On the other hand, the LHS can be written as
LHS = =Ex-px ((FEF [F(X)))
(31)
so Ex Exxpx [EFNDE [XX)]]
(32)
=0 (Ermpr [ EXxxxx [f(x)]] =((p) RHS
(33)
Therefore, the statement is proven.
15

---

Published as a conference paper at ICLR 2025
This result shows that for any individual model drawn from pr, subspace U of correctly predicted samples have the probability measure p under px. And such measure is irrelevant to f or U. Instead, it is determined by the global view of px @ pr.
B.4 LEMMA B.2
Distributional equivalence ensures not only the identical mean prediction but also identical average loss under any metric Φ. Formally, have:
Lemma B.2. (Performance Equivalence under Metrics) Ve E C([O, 1l), then (i) Vβ.9 F. Ex-px [x(((X)) = (Ex-xx(0(((X))) (i)) E F, Ex-px [o( (x)() = Ex-ex (Xg()))
Proof. (i) The equivalence of performance for two random models f.9 with complete collapse can be proved as:
Ex-ex (xxxx(((((x))] ((f(x))px
(34)
px(x))o(1)2z + SAY,, px (x)))((0)) Ju,
(35)
PT((1) +11 (1-1))(0)
(36)
px(xx))6(1)) 1) + / S.Nc, px (2))) JuU,
(37)
Ju,
o(g(z))px(z)de Ex-px [x((()))
(38)
(ii) Let l,g F be two arbitrary model. According to the distributional equivalence property, we have P( = f) = P(EF g). Let μy be pushforward measure such that VI S [0. 1].
(I( P(f(V) I), Mg is defined similarly. According the distributional equivalence condition, e (11)),1((1)) =
(1)))) (I). Therefore:
Ex-px (d([(X)) ((f(x))) (æ)dæ
(39)
(dt))
(40)
= ((1))g(dt)) = Exxpx [xxpx((((X))
(x(o()))
(41)
As result, all individual models have identical performance under arbitrary metric
B.5 PROOF OF THEOREM
Theorem 4.2
(The Brier Score Improvement) The ensemble performance improvement of the Brier score tightly bounded by:
min Ex-px feF
0z (f())) Ex-px ( (F s p²
(42)
where p = Ex-px [x)] for vf F. Ag = holds if and only if point-wise equivalence holds.
Ag = p p2 holds if and only if the ensemble f predicts constantly for all samples. Besides. Varx~px [x)) is responsible for the performance improvement.
Proof. From theorem
B.2 all individual models have identical performance. Thus vf E F.
min E Exxpx (((((X)) = Exxpx (x)(
(43)
ge
16

---

Published as a conference paper at ICLR 2025
Therefore, the ensemble performance improvement in Brier score is
AB -xx [x([(X))] Ex-px (6 (F(X))))((( [F(X)I]
(44)
=(1 ())())(0) pb(1) Exxxx 111 ((X))
(45)
=1 1-p-- Ex- [xx)² 20(2))+)
(46)
=2Ex~px [f(x)] Ex~px ((x))]-p
(47)
-2 [ Ex-px [xx))²] Ex-xx [f(x))
(48)
=Ex-px [f(x) - Exxxxxxxxxx [x()
(49)
Ex-ax [f()) - f(x))) 20
(50)
This proves the lower bound. And the lower bound holds if and only if vo E X. (1=) 0.
This means that for any x E X. f() is either or 1. As a result, Væ E X, either vf E F, f(x) = or vf E f(x) = 1. Therefore, the point-wise equivalence hold.
As for the upper bound, note that E. Ex=px [f(x)] p. then
AB = Exxxx [(x)] Ex-xx [xx)]]+ + Exxxx [mm)]]2 Ex-xx [f())
(51)
Ex Exxxx [()) Ex [xx]]] ((())
(52)
=p 1² Varx=px (0r)
(53)
Since Varx~px [xx)] 2 O, we have AB p p², and the equality holds if and only
Varx~py [xx) = 0. That is, the ensemble predicts constantly for all input samples. Thus the theorem is proved.
B.6 PROOF OF THEOREM 4.3
Theorem 4.3 Let F F-1(1) = far E X(F((x) = 1} denote the subset of that can predict correctly, then Varx [f(X) E EF (px_==xxxFFFF (F ( p²
Proof. By definition, the global diversity (LHS) can be written follows:
Varx~px [(x)] = =Exxpx [i(x) - Exxpx 1(()))
(54)
=Exxpx 11(x) p2
(55)
Thus it suffices to show that Ex-px [f(x)* = EEr,Paropp (V) the RHS, the expectation of the probability measure of the intersection can be re-written as:
Starting from
E B (0 (1)) EA.FOreE ( Px (x)
(56)
Note that ( ( 11
(1) fo e x(F(()) 1), we have
(1) =(x E = =1
(57)
=(x E (x) =
(58)
17

---

Published as a conference paper at ICLR 2025
Thus according to Fubini's Theorem:
Ep Er (x (Px F (1)) =R Pamir 1/8 (1(x)))((())x(())a
(59)
( Px
(60)
/ (Brapr(F(x)) px (x)j
(61)
J (x)))x(x))d
(62)
= =Ex Exxpxx [fx))
(63)
Thus the theorem is proved.
B.7
PROOF OF THEOREM
Theorem 4.4 The Brier score of the ensemble f can be estimated by
Ex-ex (fx))( EF [Ex=px (X - 2EF [xxxx [F()]]] + (64)
Proof. The ensemble performance can written as
(oxxx((((((X)) =Ex=px (1 f(x))
(65)
=Exxpx EFNPE (F(X)] XExxpx [Bpxp [F(X]]]
(66)
=xx=xx Eanp F2 - 2F [Ex=px [F(X]]] +1 (67)
-Ertra-ps E Ex-px [X)) - -2E [x [X))]]]
+1
(68)
Thus the theorem is proved.
B.8
PROOF OF THEOREM 5.1
Theorem 5.1
The expected Brier score loss of M ensemble members is determined by
EF FM
ØB
Σ
M
M + M
Ex-nx
Ex
[(()
p² M
2p +
(69)
M
The expected NLL of M ensemble members is estimated by
1 11-((x) 1
ΦB Σ
E((X))] zs Exxpx ((x)()((X)) + Ex-px
EF FANDE SFi.:,,Frrpp
(70)
2f(X)
M
Proof. Let U; = fxr ( = 1} px
xx (())))-((1)) And define an indicate Ix(x)) as
{
if E U,
I((=)
(0 if & U.
(71)
Then the ensemble of M members can be written
Ex-ax
(6 1 M
Σ (0
Σ
df )
Σ o
px (1m)
(72)
M
m=0
where Ω C X' such that
M 2m = fo x X
Σ [())= = m
(73)
That is, the set of samples that exactly members predict correctly.
18

---

Published as a conference paper at ICLR 2025
Consider M i..dd models Fi,.... FMs ~ Pr. we notice that Σ Ií(x)) (inomial(M, f(z)).
Thus P(E (x:) m) = (T f(z))) As result, the expected performance of an ensemble of M models can be written as
M Σ
EF Farmti
Ex-nx
nr (X)]
(74)
M
2m
-EF FF Fasmii
de px (2i))
(75)
m
BF
PaSNDo [px [px(52m) (22m]
(76)
M
m=0
M
m
Ex~px
M (M))
((x)m(1-((x)) - j(X))
(77)
M
m
m=0
M
m
M
((x) - ((x))
Exxxx
(78)
M
m
m=0
=Ex<px
E
Bi
(79)
For Brier score, this can be written
7
27 7
Ex-px
Ez (1
= =E
+
(80)
M
M
M
- + Mf(x)( - +P
=E =Exxpx
(81)
M²
(1 (1-)(x))+ ((x)1-((x))
= Exxxp
(82)
M
Exxpx [11 - }(X))7} Exapx ((x)(1-f(X))
(83)
M
= E (xn()(x))] + Exxpx (arr (F(x))
(84)
M
Note that
Ex~px -xxp))]]===xxxxxx>>> 2p+1 +1
(85)
and from theorem
we also have
Varx<px [(()] = Exxxx1f(x)1 p²
(86)
Thus the expected performance can be written as
2 Ez (1 M ]
Exxpx [ox(f(x))] + Exapx Varr=er (F(x))
Ex-px
(87)
M
Ex-px [[(X)) ,2
Ex-px [(x)} - -2p 1
(88)
M
M +
E -px [[(x}]
12
2p +
(89)
M
M
For negative likelioo,,,odd, we have
27
2
(
Z
Ex~px
F. Fz.-Binomaa(
-xxxxx
Ez
log
(90)
M
19

---

Published as a conference paper at ICLR 2025
Using Taylor's expansion around Z MM Mf((), we have
Z - -MM(
(Z- ((x))
log
~ log(
(91)
M
Mf(x)
(Mf(()))
Taking the expecation over Z, we have
0 -(Xx)
Ex-px
Ez
log
2 M
Ex ~px (
(92)
Mf(X)
(Mf(x))2
)(x((x) 1
= Exx [dn](((X)]] + Ex-px
(93)
2f(X) M
Thus the theorem is proved.
B.9 PROOF OF THEOREM WITHOUT SURROGATE MODELS f
Here present brief analysis of theorem 5.1 complementary to appendix B.8 We show that the original distribution pF defined over the function space JF also satisfies a more generalized version of the theorem compared with Py. We also demonstrate why the surrogate py is important in the proof. For any X ~ px, we have a distribution of the prediction of the ensemble: f(X) ~ Pp where f(x) = Ex~px [Epapr [F()]]] Then instead of the discrete Qm we have infinitely many i- (x) = r} for E [o, 1]. And thus px = Pr(r).
2 for E XI Ø7 Ee
As result, the expected loss can be written as
1 M
Ex Ex-px
0 M
Σ
F(X))
-
o(r)px (oxpxx(x)dcr (x)dr
(94)
Ja.
(1,))r
(95)
Taking the expectation over i..dd Fi,... FM ~ pF, we have
M
Er... -mmppl [[ (()))((2)))))
Epy. Pa Exapx 0 M
Σ ((())
(96)
f ((.) Furpp [px (1
(97)
suffices to determine EF,... PPrrpp [px (92))]. Note that for any single model f E F and X px. we denote by Pe, = Pe the density function of f(X). According to the distribution equivalence property, such a PDF shared across all functions F. Therefore, we have
E EF...., Parop Ipx (2,)) =EF, PPa S/, Px px(x)dæ
(98)
-EF,," PPrp"
I ia (EM. Px
(99)
Ep,.. B.p.....urpp
[L HEM ((())rr))XX (x)dx
(100)
x
Consider that for given & and F ~ pF, the random variable F(x) follows distribution Pt, And let pM, denote the distribution of the average of the distributions of 1 Σ.M, (x)). Then
E BP
[1 ( EM ((())=r)) (r)
(101)
And then
/ PM (T) (r)
E Ery Fxrope [px (Ωt
(102)
= =IE Exxox (r).((r)
(103)
20

---

Published as a conference paper at ICLR 2025
Substitute this back to eq-
(97 have
Ex~px [0(
Σ
M
[(X))]]
- -f¹ Bxx)x [PM (r)
(104)
M 1=1
=Ex<px If( (()) (Mr))))
(105)
where (pc.x #.... xp.xx is the M-fold convolution of Pe,x. Now we abuse the notation and denote m Mr [0, M). then
[0( ( Σ F(()))
=Exxpx 11/ o( ( (pex +Pt.x)(m)dm)
E Ex-px
(106)
-Exxpx Emmipexx pe [o )1]
(107)
For models with neural collapse, we have Pe,x Bernoulli. This naturally leads to the previous results with PF in eq. (79). And since Pe,x is already very close to Bernoulli distribution, theorem 5.1 the conclusions of theorem 5.] transfer to pF without requiring any additional modifications, This also shows the use of Pr is purely to avoid the retractable continuous distributions If pe.x is modeled as continuous distribution whose mass is mostly located l = 0 and l 1, the M-fold convolution becomes infeasible to track for closed-form analysis.
B.10 TARGET-CLASS BRIER SCORES
The use of the output of the target class is based on the following reasons: (1) Because of the neural collapse phenomenon, the predicted probability of the target class is either very close to or very close to When it's close to the prediction of all other classes is naturally zeros. And when it's close to O, there is usually one other dominant class. Therefore, taking the target class only is already sufficient for the analysis.
(2) Negative log-like li-elikiihood only considers the target class. Therefore, creates unnecessary inconsistency across the analysis. (3) Due to the softmax layer, the output of the target class already contains information from the prediction of all classes (Wang & Wang 2022).
Specifically, discriminative models do not directly predict the likelihood vector p [0, 1]s. Instead, they predict the logits y E RR, and then a softmax activation is applied to to obtain the likelihood. Here is the number of classes. As result, let denote the ground truth class, then the target-class Brier score is
izft e/
(1 pt)
)2
(108)
cily
2/=1
On the other hand, the all-class Brier score is
1 - 1el/ = Σ ps (1 (1- (1-p))2- 2=) >{
eVi
+
Enet e
litt
)2
(109)
2j=1 elli
2==1 Ed
c
iff
istt
Eiffe e/ e2 22 (2 Eigee eVi)2
(110)
(25=1 ca
The numerators are Exyze e and Eiste e
Eisr 22%, That is, the all-class Brier score increases the weights of the non-target class by 1. The differentiation between them is one of extent rather than essence.
Besides, by Cauchy- Schwartz inequality, we have
(1 - p.) < y 1,11 2(1 2
(111)
that tightly bounds the all-class Brier score with the single-class
21

---

Published as a conference paper at ICLR 2025
Person's R: 0.8624, (p=0.000)
Person's R: 0.7861, (p=0.000)
1.00
1.00
0.75
(Another) Single k10 Single k10
0.75
Single k160 Single k10
0.25
-1.00
-1.00
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
Ensemble k10 Single
Ensemble k10 Single
Large Single Small Single Difference vs. En-(b) Small Single Small Single Difference vs. En- semble Small Single Difference
semble Small Single Difference
Figure 8: The comparison between the single model-single model performance difference (y-axis) and the ensemble-single model performance difference (æ-axiis) for all testing data. In (1) large single model is used for the y-axis (large single model-small single model). In (2), two independent small single models are used.
C ENSEMBLES VS. SINGLE MODELS
Single models and ensembles are compared by comparing the results of
<((()) (((())) <(g(x)) ((f()))
(112)
where f are small models while g are large models in single-model capacity. Following previous work (Abe et al., 2022b), implement the results in fig. 8 a). The y-axis represents the difference between large and a small single model. The z-axis represents the difference between an ensemble (of small models) and small single model. Note that since the values represent the Brier score differences, a negative value indicates a performance improvement.
Based this result, previous work reported that the performance improvement from increasing model capacity and ensembling are very similar. However, we argue that this should be reconsidered carefully. Our analysis demonstrates that these results along with the high Pearson's is R values, and almost-zero p-values, are all caused by the distributional equivalence with point-wise distinction phenomenon revealed in our work. For k 160 single model g and a 10 single model f. know that g predicts correctly for set U, X' while f predicts correctly for Uy. We know that [Uyl Uú almost surely because of the increasing capacity. However, U, C Ug almost never holds because of point-wise distinction
Reasoning. The points in fig. 8 8a) that have negative values are those x E U, (predicted cor- rectly by g but wrongly by f). Given that these points misclassified by f. due to the distributional equivalence property, they are more likely to be predicted correctly by other fs in F, resulting in also negative z-values. On the other hand, the points that have negative y values (predicted correctly by f but wrongly by g). And because they are already predicted correctly by f. due to the distributional equivalence property, they are more likely to be predicted wrongly by other fs in F, resulting in also positive z-values. These two scenarios lead to the positive correlation between of (f(())) (x)) and ((g(z)) - ((f())).
those Uy UÜ,
Note that there are infinitely many large model gs such that U, C Ug. but compared with G, such gs' measure should be extremely small. For sanity check, we compare Uy for all M = 100 CNN models with h 10 and Ug for all M 100 CNN models with & 160. No examined of (f.9 pair satisfies the subset relationship Uy Ug-
22

---

Published as a conference paper at ICLR 2025
Here show counter-example The justification above does not require |0,] 1U That is, 9 does not have to be a larger single model compared with f. We can obtain the same positive correlation with two & 10 CNN models. This counterexample is shown in fig. [)). For the y-axis, instead of using a large single model we use another small model with 10. We can observe that the exact same trend holds in both fig. 8 ab).
D ADDITIONAL FIGURES FOR DISTRIBUTIONAL EQUIVALENCE
In addition to the demonstrative figure in fig. we carry out comprehensive experiments verify the distributional equivalence property. For model structure, we investigate the direct connection like CNNs and the skip connection like ResNets. We three datasets, including small models such as CIFAR-I0, and CIFAR-100 and more complex dataset like TinyImagence For each dataset and model structure, we scale the sanle-model capacity through width parameter k in {10, 20, 40, 60, 160). As result, there are total of 2 x 3 x 5 x M = 3000 models. The results for CNNs and ResNets are presented in fig. and fig. TO respectively. It can be clearly observed that all models satisfy the distributional equivalence property. That is, within each subplot, all M = 100 independently trained models have the equivalent distribution of the predictions for the output class. Besides, almost all the mass of the distributions falls at two end points {0, 1}. This also provides support for the neural collapse assumptions.
Furthermore, in each row, from left to right are increase in the data complexity. Since the model capacity remains the same, the prediction level p (i.e. Ex~px [f()) decreases drastically. On the other hand, in each column, from top bottom are increase in the model capacity, while the dataset stays invariant. It can be observed that for the same px, increases with the single-model capacity. This naturally leads to performance improvement under monotonic metrics such as the NLL or the Brier score.
The criteria identical the results of fig. ((b). Only the first five models (seed from 1 to 5) are plotted for better visualization. It can be clearly observed that models differ significantly regarding each input. Combined with figs. and 10 the distributional equivalence property is comprehensively verified.
Additional results of point-wise distinctions are shown in figs. 11 and 12
E ADDITIONAL FIGURES EMPIRICAL RESULTS
Here we show additional experiments as complementary to the manuscript. fig. 14 shows additional results to fig. Jon ResNet instead of CNNs. In the manuscript, figs. Gndd 7 show the results of CNNs on CIFAR- 10 and CIFAR-100. Here we present other results such as CNNs on all datasets in figs. [5]
and 17
And the results of ResNets on all datasets are presented in figs. [6 and [8] In other words,
figs. 15 and 16 are the additional results to fig. 6 and figs. 17 and 18 are additional results to fig. 7
We also plot the distributions of the testing accuracy of all models fig. 13 he top row shows the results of CNNs while the bottom shows the results of ResNets. Each color represents the distribution from models of identical width. It's observed that when the models only differ in the training stochasticity from SGD, their testing performances are very similar.
Besides, to further validate the discovered distributional equivalence property, we include models trained using schemes other than the standard SGD. Here test SGD with momentum. We set the learning rate as 1r=1e-3, 3, and momentum as 0 9. For another variant, we further include a weight decay 5e-4. The results are shown in fig. 19 Here we CNN models with 20 and CIFAR-10 10 dataset.
We also present the error rates of all the single models and ensembles in fig. 20 a complementary
result to figs. 15 and 16
23

---

Published as a conference paper at ICLR 2025
60
62
25
04
65
82
95
30
CIFAR-IO, 10
(b) CIFAR-100, A 10
(c) Tinylmagenct, l 10
2.0
0A
12
-22
2.5
04
26
90
92
0.6
CIFAR-IO, 20
CIFAR-100, 20
(1) Tinylmagenet, k 20
2.80
6:75
02
WHOOM PIEAdRty TH
CIFAR-10, h: 10
(h) CIFAR-100, & 40
(i) TinyImagenet, A 40
Pedceereaarry 4 Srgee
(j) CIFAR 10, A 80
(k) CIFAR- 100, A 80
(1) Tinylmagenet, &: 80
3400
(m) CIFAR- 10, 160
(n) CIFAR-100, & 160
(o) Tinylmagenet, 160
Figure The demonstration of the KDE of the prediction the target class. The density estimated using all testing samples. Here CNN models with different single-model capacities (widths) and three datasets are tested. For each figure, all M 100 models are plotted.
24

---

Published as a conference paper at ICLR 2025
33
12
-22
60
25
50
CIFAR-IO, 10
(b) CIFAR- 100, 10
(c) Tinylmagenct, l 10
30
2.5
04
04
17
8-6
17
(d) CIFAR-IO, 20
CIFAR-100. 20
(1) Tinylmagenet, k 20
(g) CIFAR-10, 40
(h) CIFAR-106 100, 40
(i) TinyImagenct, 40
1.5
veddted afre MERLM
(i) CIFAR -EFAR-10,K
(k) CIFAR- 100, 80
(1) Tinylmagenet, 80
(m) CIFAR-10, 160
(n) CIFAR-100, 160
(o) Tinylmagenet, 160
Figure 10: The demonstration of the KDE of the prediction of the target class. The density is estimated using all testing samples. Here ResNet models with different single-model capacities (widths) and three datasets tested. For each figure, all M 100 models are plotted.
25

---

Published as a conference paper at ICLR 2025
(a) CIFAR-10, k = 10
(b) CIFAR-100, A = 10
(c) TinyImagenet, k = 10
(d) CIFAR-IO 10, & = 20
(e) CIFAR-100, & = 20
( TinyImagene k = 20
CIFAR-10, k 40
(h) CIFAR-100, k 40
(i) Tinylmagenet, k 40
CIFAR-10, 80
(k) CIFAR-100, k 80
TinyImagenet, h == 80
(m) CIFAR-10, 10, 160
(n) CIFAR-100, & 160
(0) Tinylmagenet,
Figure 11: The point-wise prediction of the target class. Here CNN models with different single- model capacities (widths) and three datasets are tested. For each figure, we plot the first 5 models (seed from 1 to 5). Each point represents specific prediction (z,)) It is demonstrated clearly that although models tend to agree on certain samples, the point-wise predictions vary significantly.
26

---

Published as a conference paper at ICLR 2025
(a) CIFAR- 10, k 10
(b) CIFAR-100, k 10
Tinylmagenet, k = 10
(d) CIFAR-1O, k 20
(c) CIFAR-100, 20
Tinylmagenet, k 20
CIFAR-IO, A 40
(h) CIFAR-100, K 40
Tinylmagenet, k 40
( CIFAR 10, 80
(k) CIFAR-100, 80
Tinylmagenet, 80
(m) CIFAR-10, = 160
(n) CIFAR-100, & 160
(o) Tinylmagenet, & = 160
The point-wise predictions of the target class. All testing samples are presented. Here ResNet models with different single-model capacities (widths) and three datasets are tested. For each figure, we plot the first models (seed from I to 5). Each point represents a specific prediction (x,) It is demonstrated clearly that although models tend to agree on certain samples, the point-wise predictions vary significantly.
Figure 12:
27

---

Published as a conference paper at ICLR 2025
3.50
0.65
0.25
(a) CIFAR-IO 10, CNN
(b) CIFAR-100, CNN
(c) TinyImagenet, CNN
42
160
0325 0e50 0LTS
57000 8725 0756 0775 Axuwy
0.30
2.25
0.40
0.42
3.0
320
0.25
0.30
0.35
0.40
Accurncy
(d) CIFAR-10 10, ResNet
(c) CIFAR-100, ResNet
(f) Tinylmagenct, ResNet
Figure 13: The illustration of the distribution of the testing accuracy of all single models. The top row shows the results of CNNs while the bottom shows the results of ResNets. Each color represents the distribution from models of identical width.
2+ 00
2+
2+00
2-
275
2.000
Ductek
245
2.0
0225
6:6
0250
106
20
100
(a) CIFAR-10
(b) CIFAR-100
(c) Tinylmagenet
Figure 14: The estimation (regions) and the empirical results (solid curves) of the Brier score of original models. The z-axis represents the number of ensemble members (a.k.a. ensemble capacity). and the y-axis represents the Brier score of the ensemble. Dashed lines represent the standard deviations. The tested model ResNet.
28

---

Published as a conference paper at ICLR 2025
0.24
Ens: k=
Enst 10
Ens:
Enk k= 20
Ens: k
0.70
Ens: km
0.22
Ens: k 80
Ens:
Ens: K
Bier Score
0.20
160
0.50
=60
Brier Score
0.65
k= 160
0.18
Single Model
Brier
0.45
Single Model
0.80
Single Model
0.18
0.40
0.35
0.35
0.50
10P
10F
107
109
10P
10
10F
1.07
109
10
10
10F
100
109
10
Parameters
Parameters
(a) CIFAR-10
(b) CIFAR-100
(c) TinyImagenet
Ens:k
Ens = 20
2.75
Ens:k
Negetive Log Likelhood
Ens :k 40
Ens I 40
Log Likellhoodd
k=
Ens
2.50
1.0
Eiss I 160
Ens = 160
3.5
Ens: k 160
Single
Nogation Log-
2.25
Single Model
Single
0.8
2.00
3.0
1.75
0.6
1.50
10
10a
107
10
10'
10"
10'
20"
20
Parometeess
of Parameters
8 of Parameters
(d) CIFAR-10
(e) CIFAR-100
Tinylmagenet
Figure 15: The performance comparison between the scaling of single model (black dashed curves) and increasing the number of ensemble members (colorful solid curves). In each ensemble (i.e. each colorful curve), M varies from 100. The results are generated using CNN models.
29

---

Published as a conference paper at ICLR 2025
0.32
Ens: k =
0.70
Ens: 10
Enc: =0
0.30
k=
K-
0.75
Ens: k-
Ens: = 80
Enk: K 80
0.70
Ens: K 80
Brier Scoee
0.26
Brier Score
0.60
Brier Score
Single Model
+ Singli Model
Single Model
0.65
0.55
0.22
0.60
0.10
0.55
0.45
0.16
10a
10a
107
100
10a
10a
10
107
10
10
10a
10
107
10a
10
Parameters
Parameters
(a) CIFAR-10
(b) CIFAR-10 100
Tinylmagenet
Ers =
Ens:k= 10
Ers :
Ens:k
Regative Log Eilliloood
2.00
Ens:k-40
Negalive Log- Likellhhadd
Ers :k
k-
Ens k 40
1.75
Ens:km 80
Ent kk 160
Eris k 160
1.50
Single Model
Single Model
4.0
Single Model
1.25
3.5
1.00
3.0
0.75
2
2.5
0.50
2.0
10a
10"
10'
10a
10a
10P
10c
107
10
10
10a
10"
10"
10
10
Parameters
Parameters
Parametors
(d) CIFAR-10
(c) CIFAR-100
Tinylmagenet
Figure 16: The performance comparison between the scaling of single model (black dashed curves) and increasing the number of ensemble members (colorful solid curves). In each ensemble (i.e. each colorful curve), M varies from 100. The results are generated using ResNet models.
0.32
1-
0.24
20
6.22
0.55
6.22
Decempatiien
K- 100
F-160
Decompstioo Ersembl
1-0
Ensemble
STD
645
6.14
6:44
0.55
0.2
0.35
20
40
60
100
20
60
60
20
100
20
80
300
(a) CIFAR-10
(b) CIFAR-100
(c) TinyImagenet
3,2
kw440
3.75
Rlgroveezg-- Lolathoed
2w40
A-20
163
3.5
Enoewilee
2.25
Drakmbb bis
bncempadtt
Encemake
0.8
2.00
0.6
Eraemble
Members
Disumate Mumber
Eraumble
Vamberr
(d) CIFAR-10
(c) CIFAR-100
Tinylmagenct
The theoretical results (dashed curves) are compared with the empirical results of ensembles using M models for (a)(b)(c the Brier score and (f) the negative log- likelihooodd The ensemble curves are computed using pr. Results are generated using CNNs.
Figure 17: The verification of theorem 5.1
30

---

Published as a conference paper at ICLR 2025
10
K-10
0.55
160
6,70
160
Decompadoo
Sree Stor
Decorpsdor
6.50
Ereamods
Ext STD.
Ersemtie
6.25
0.001
0:55
0.179
20
20
120
60
60
20
100
20
60
20
100
(a) CIFAR-10
(b) CIFAR-100
(c) TinyImagenet
2m
2.00
Decenpsdtiio
2=000
2-160
Decempoation
30
(40
1.30
4.0
Braunidte
1.00
3.
830
Stuarbte Membeera
2: Encenbile Memtbers
Crauntri Vembera
(d) CIFAR-10
(c) CIFAR-100
Tinylmagenct
The theoretical results (dashed curves) compared with the empirical results of ensembles using M models for (a)(b)(c the Brier score and (f)(e)(f) the negative log-likelihood. The ensemble curves computed using pr. Results generated using ResNets.
Figure 18: The verification of theorem 5.1
4.0
4.0
3.5
3.5
3.0
Denotty
2.5
2.0
3.5
3.0
-0.2
0.0
0.2
0.8
-02
1.0
Predicted Probaliiity the Class
Predicted Probabillty
(a) SGD w/ momentum
(b) SGD w/ momentum & weight decay
Figure 19: The demonstration of the KDE the prediction of the target class. The density is estimated using all testing samples. For each figure, all M 100 models are plotted. The models are CNNs trained with (1) momentum and (2) momentum & weight decay. All testing samples are presented.
31

---

Published as a conference paper at ICLR 2025
1
0.60
k-20
10
0.70
Ens:
0.26
Enst =
Ens: = 40
Ens: è 40
0.55
0.05
Enter tate
0.22
Ens: 160
Single Model
0.50
Ens: 160
Single Model
Ens: k 160
Single Model
Eroor
0.60
0.45
0.10
0.40
0.35
0.12
10c
107
10P
10F
104
107
10P
10
10
10'
10
of Parameters
of Parameters
of Parameters
(a) CIFAR-IO+CNN
(b) IIFAR-IN
(c) Cinylmagenet+CNN
Ens: 1
Ens: 1010
Ens: k
Ens: R- 20
0.70
Ens: k
Enc: k
0.32
0.75
Ens: = 80
Ens: 80
Enc 2 80
==
0.70
160
0.28
Single Model
Single Model
Single Model
Ero
Emtor
0.55
0.24
0.60
0.50
0.35
0.20
0.45
10F
10F
107
100
10P
10F
107
10P
109
10F
10F
107
109
109
Parameters
Parameters
Parameters
(d) CIFAR-10-ResNet
(e) CIFAR-100+ResNG
(1) Tinylmagenet+ReeNee
Figure 20: The performance comparison between the scaling of single model (black dashed curves) and increasing the number of ensemble members (colorful solid curves). In each ensemble (i.e. each colorful curve), M varies from to 100. The results are generated using CNN models.
32


Reviews:
[{"review_id": "EEOljQF1e7", "reviewer": "Reviewer_wGn1", "summary": "This paper develops a theory to explain why homogeneous deep ensemble can improve the performance.", "questions": "Please address my questions in Weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The experiments of distributional equivalence property in Sections 4.1 and 4.2 are interesting.", "weaknesses": "- The setting and assumptions of the distribution $p_F$ which is crucial are not clear. All mentioned is $F \\sim p_F$ is a trained model. I believe that this is important for the solidness of the developed theory. At least, we should have something to connect the min over $\\mathcal{F}$ and the member function $F \\sim p_F$. Otherwise, the theory is loosing. \n- The proof of Proposition 3.1 can not be found.\n-  It seems to me that the definition of $\\hat{f}(x)$ is not meaningful because it is associated with a function $f \\in \\mathcal{F}$ and it returns 1 if $f(x)$ is a correct prediction and $0$ otherwise. Therefore, $E_{F\\sim \\hat{p}_F}[F(X)]$ seems to be the percentage of the number of correct predictions when $F \\sim p_F$. What is the meaning of applying $\\phi$ to this average. Similar question to $\\phi(\\hat{f}(x))$ in Eq. (8).\n-  I have checked the proof of Lemma B1 and Theorem 4.1. They require $E_{x \\sim p_x}[\\hat{f}(x)]$ = $E_{x \\sim p_x}[\\hat{g}(x)]$ for every $\\hat{f}, \\hat{g} \\in \\hat{\\mathcal{F}}$ or the accuracies are equal for every $f,g \\in \\mathcal{F}$ which is weird for a general function space $\\mathcal{F}$. Certainly, we can partly assume this with $f,g \\sim p_F$ which is the distribution of well-trained models. However, the support of $p_F$ is only a subset of $\\mathcal{F}$. Therefore, I reckon that the minimization in Theorem 4.1 should be over the support of $p_F$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eFL50QhcNn", "reviewer": "Reviewer_fvho", "summary": "This paper challenges the common explanation for deep ensemble success, which attributes their effectiveness to Jensen’s inequality. The authors argue that this explanation is insufficient and propose a new perspective based on the **distributional equivalence property** of trained models. According to this view, although ensemble members may differ in their individual predictions, their output distributions (based on the predicted probability for the target class) are statistically identical across the dataset.\n\nThrough both theoretical analysis and empirical evidence, the paper demonstrates how this distributional equivalence property helps explain why deep ensembles consistently outperform single models. Additionally, the authors introduce estimation schemes to predict ensemble performance based on the number of members and model capacity, offering insights into the relationship between ensemble capacity and single-model scaling.", "questions": "1. Section 4.1’s findings on distributional equivalence focus on the predicted probability for the target class, without considering the full class probability distribution. Could the authors provide an additional analysis that incorporates more information, such as the difference $\\(l = p(y) - p(\\hat{y})\\)$, where $\\(\\hat{y}\\)$ is the predicted class and $\\(p(\\hat{y})\\)$ is the highest probability across all classes? This could give a broader perspective on the consistency of distributional equivalence.\n\n2. Which loss function was used to train the models in Section 4.1? My hypothesis is that the loss function has a significant influence on these findings. For instance, cross-entropy loss focuses on the probability of the target class, so models trained with this objective might naturally exhibit distributional equivalence on this specific metric. In contrast, a loss function such as label smoothing, which considers probabilities across all classes, might lead to different observations, potentially limiting the generalizability of the results in Section 4.1. Could the authors discuss or provide empirical insights on how these findings might vary with alternative loss functions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper offers practical insights into the relationship between global diversity and ensemble performance improvement, as well as the distinct contributions of ensemble capacity versus single-model scaling. The estimation schemes, which allow practitioners to approximate ensemble performance using just two models, provide a computationally efficient approach that could inform more cost-effective system design.\n- Beyond merely acknowledging the diversity of ensemble members, the paper provides an in-depth analysis of both **point-wise** and **global diversity**, emphasizing the latter as a key factor often overlooked in prior research.", "weaknesses": "- The main contribution, introducing distributional equivalence, may lack generalizability to other loss functions. Please refer to the question section for more details.\n- The distributional equivalence condition, a foundational assumption underlying all theorems, seems overly restrictive. Specifically, in the proof of Theorem B.1, lines 692–695, the authors assert that “for arbitrary $\\(\\hat{f}, \\hat{g} \\in \\hat{F}, P(\\hat{f}(x)) = P(\\hat{g}(x))\\)$” holds  $\\(\\forall x \\in X\\)$, which is a strong assumption that I am still not convinced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HSXSlIpgvE", "reviewer": "Reviewer_C32A", "summary": "The paper discovers that members of homogeneous ensembles have the distributional equivalence property. Based on this property, the paper answers the fundamental question that why deep ensembles work. Additionally, a way to estimate the performance of deep ensembles is proposed. The paper further discusses the bias-variance tradeoff in deep ensembles and compares the effectiveness of scaling a single model versus scaling deep ensembles. Comprehensive theoretical and empirical analyses are provided to support the claims in the paper.", "questions": "Please see the weakness part.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. Theoretical findings are fruitful and solid.\n2. Insightful discussions about related works can effectively engage readers.", "weaknesses": "1. In Line 135 of the paper, aurthors mention that the unsatisfactory results of the majority vote classifier inspire them to explore overlooked conditions, which turns out to be the distributional equivalence property. Then, why is there no further effort to explain the majority vote classifier after finding the overlooked condition?\n2. The distributional equivalence property is observed on CIFAR-10 dataset. However, on such small datasets, it's known that the trained DNNs tend to be quite overconfident, especially in the case where weak or even no regularization is adopted. Combined with the fact that DNNs are highly accurate on small datasets, the observed distributional equivalence is not surprising at all. To justify the distributional \nequivalence property in general cases, larger datasets and/or stronger regularization settings need to be considered. \n3. In Section 4.4, results are mainly derived for the Brier score, while those for NLL are missing.\n4. In Fig. 6, it's better to include plots for error rate in additon to losses.\n5. In Line 676, authors note that SGD is used without momentum or data augmentation. However, these techiques are crucial for DNNs to reach high performance. If they are not considered, the results of the paper could have limited practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yhWYlhDHLQ", "reviewer": "Reviewer_eaji", "summary": "Previous theory of Deep Ensemble owing the benefit of ensembling to the diversity of ensemble members with \"trivial\" Jasen Gap of the losses without considering the data distribution. Instead, this paper introduces the distributional equivalence of the trained members from global data distribution, which further explains Deep Ensemble works with the equivalent output distribution globally but with distinct sample-wise predictions for a group of samples. Under this perspective, the paper further provides the theory of how sample-wise diversity and the number of members contribute to the performance. Besides, it provides a method to predict the ensemble performance with only two models.", "questions": "See Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is easy to follow and well-organized. \n2. The introduced \"distributional equivalence\" that all members produce similar prediction distribution across a group of samples is novel, and has been neglected by previous works.\n3. The theory is technically sound and the prediction of ensemble performance with only two models is also interesting.", "weaknesses": "1. What does the \"only differ in the training stochasticity\" actually mean? i.e., where does stochasticity come from? from the shuffled dataset? or random initialization?\n2. An interesting question is why the homogeneous members demonstrate the \"distributional equivalence\" property. Does it come from the fixed dataset bias, fixed architecture, etc.?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eAH9Y4BrLm", "reviewer": "Reviewer_CxeF", "summary": "This paper examines why deep ensembles are effective by proposing distributional equivalence, the idea that ensemble members produce predictions with identical distributions. The authors argue this property is key to the performance gains of deep ensembles, offering a new explanation beyond the Jensen gap. They validate this theory through experiments and introduce an estimation scheme that allows ensemble performance to be predicted using just two models, reducing evaluation costs and highlighting the unique benefits of deep ensembles over scaling single models.", "questions": "1. The experiments have included the CIFAR-10 and TinyImageNet, which may not represent the complexity of real-world data, such as text data or multi-modal data. Could the authors discuss how dataset complexity might affect the distributional equivalence property, especially for datasets with higher-dimensional or more variable features?\n2. How sensitive is the distributional equivalence property to various training hyperparameters, such as learning rate or initialization seed? An analysis of this could help in understanding the robustness of it across different training configurations.\n3. The paper introduces a method for performance estimation using two models. Could the authors comment on the potential for scaling this approach for ensembles of significantly larger capacity.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper introduces a fresh theoretical perspective on deep ensembles by proposing the concept of distributional equivalence among ensemble members. This approach offers an alternative to the usual Jensen gap explanation, providing insight into why deep ensembles work by analyzing homogeneous model distributions.\n2. The proposed method of estimating ensemble performance using only two models is a practical contribution that could streamline the application of ensembles in resource-constrained settings.\n3. The authors have conducted comprehensive experiments across multiple datasets and architectures, providing strong empirical support for their theoretical claims.", "weaknesses": "1. The theoretical framework relies on assumptions about neural collapse, which may not apply universally across various datasets and model configurations. This reliance could limit the generalizability of the findings, especially in cases with limited data or high variability.\n2.Deep ensembles are frequently applied to enhance uncertainty estimation, the paper mainly emphasizes performance improvements, leaving the effect of distributional equivalence on uncertainty quantification undiscussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)