(Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Models
Authors: Andreas Kirsch
Venue: Accepted by TMLR ()

Abstract: Epistemic uncertainty is crucial for safety-critical applications and data acquisition tasks. Yet, we find an important phenomenon in deep learning models: an epistemic uncertainty collapse as model complexity increases, challenging the assumption that larger models invariably offer better uncertainty quantification. We introduce implicit ensembling as a possible explanation for this phenomenon. To investigate this hypothesis, we provide theoretical analysis and experiments that demonstrate uncertainty collapse in explicit ensembles of ensembles and show experimental evidence of similar collapse in wider models across various architectures, from simple MLPs to state-of-the-art vision models including ResNets and Vision Transformers. We further develop implicit ensemble extraction techniques to decompose larger models into diverse sub-models, showing we can thus recover epistemic uncertainty. We explore the implications of these findings for uncertainty estimation.

Published in Transactions on Machine Learning Research (05/2025)
(Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Models
Andreas Kirsch
Blackho@gmailcoom com
Reviewed on OpenReview: https: openrevie.. pormm? id- QEVQV
Abstract
Epistemic uncertainty is crucial for safety-critical applications and data acquisition tasks. Yet, we find an important phenomenon in deep learning models: an epistemic uncertainty collapse as model complexity increases, challenging the assumption that larger models invariably offer better uncertainty quantification. We introduce implicit ensembling as a possible explanation for this phenomenon. To investigate this hypothesis, experiments that demonstrate uncertainty collapse in explicit ensembles of ensembles and show experimental evidence of similar collapse in wider models across various architectures, from simple MLPs to state-of-the-art vision models including ResNets and Vision Transform- ers. We further develop implicit ensemble extraction techniques to decompose larger models into diverse sub-models, showing we can thus recover epistemic uncertainty. We explore the implications of these findings for uncertainty estimation.
provide theoretical analysis and
1 Introduction
Bayesian deep learning provides us with a principled framework for quantifying uncertainty in complex machine learning models (MacKay, 1992; Neal, 1994). A wide range of applications relies on accurate uncertainty estimates. These include active learning (Settles, 2009; Osband al., 2022), where uncertainty guides data acquisition; anomaly detection (Tagasovska & Lopez-Par, 2019; Mukhoti el al., 2023), where uncertainty can signal confidence is crucial for responsible deployment (Band et al., 2021).
inputs; and safety-critical systems, where understanding model's
A key concept in this framework is epistemic uncertainly, which represents model's uncertainty about its predictions due to limited knowledge or data (Smith & Gal, 2018; Der Kiureghian Ditlevsen, 2009). This form of uncortainty is distinct from aleatoric uncertainty, which captures inherent noise or randomness the data (Kendall & Gal, 2017). Epistemic uncertainty usually quantified using the mutual information between the model's predictions and its parameters (Houlsby et al., 2011; Gal et al., 2017). For deep ensembles (Lakshminarayaneel et al., 2017), which consist of multiple independently trained neural networks, this amounts to the mutual information between the ensemble member index and the predictions. This is equivalent to the disagreement among ensemble members on a particular input (Smith & Gal, 2018): if the models in the ensemble produce widely varying predictions, this is indicative of high epistemic uncertainty.
As deep learning models grow in size and complexity, the quality of their uncertainty estimates generally improves (Tran et al., 2022). However, our work provides evidence for simple yet important phenomenon: when constructing higher-order ensembles, ensembles of ensembles, collapse". This collapse occurs because individual ensembles, given sufficient size and training, converge to similar predictive distributions, causing inter-ensemble disagreement to vanish as the ensemble size grows.
observe epistemic uncertainty
We initially observed this bohavior Ensemble-of Ease--eeeeeee--eeee Cassebble--ppistenic--e Cestteni--Dccetaantty -aa---DD--aaaaaa oar.:0000000a0000000000222111110 and have since confirmed it. in multiple independent experiments that we share here.
2021, published informally https://blackhc wtt.lacccc..niion.....ooo notton..site

---

Published in Transactions on Machine Learning Research (05/2025)
True Function, Data, and Ensemble Predictions
Epistemic Uncertainty for Different Sub-Ensemble Sizes
Epistemic Uncertainty vs. Sub-Ensembl Size
0176
9150
1
0196
Su--ri-
0100
Saver
9670
12"
9000
ddbers Ubc 04
0000
Sub Encembe
Figure 1: Epistemic Uncertainty Collapse in Toy Regression Problem. As the sub-ensemble size increases, epistemic uncertainty vanishes. Ensembles of 10 sub-ensembles with different sub-ensemble sizes. Left: True function, data, and ensemble predictions. Middle: Epistemic uncertainty across input space. Right: Mean epistemic uncertainty vs. sub-ensemble size.
We hypothesize that, similar to ensembles of ensembles, implicit ensembling might occur within large neural networks, potentially leading to significant underostination of epistemic uncertainty for traditional uncertainty estimators that rely on the final logits. Hence, similar to deep ensembles that have been found to offer better calibration (Ovadia et al., 2019), implicit ensembling may explain why larger models also appear more calibrated (TTran al., 2022). A recent preprint by Fellaji & Pennerath (2024) provides additional evidence of this phenomenon occurring even in simple parammaamtte MLPs trained on standard benchmark datasets but does not offer explanation.
We introduce implicit ensembling as possible explanation for this phenomenon. Our theoretical analysis demonstrates uncertainty collapse in ensembles of ensembles, and we present experimental evidence of similar collapse in wider models. Additionally, we show that our proposed implicit ensemble extraction technique can recover epistemic uncertainty from large models.
Our work is of interest because epistemic uncertainty estimation is crucial for safety-critical applications and decision-making scenarios (Wen et al., 2021), where understanding model limitations can prevent costly errors. While the potential underestination of epistemic uncertainty in large wer-parameterized neural networks could lead to overconfident predictions and compromised decisions, our proposed implicit ensemble extraction technique offers a promising approach to recover and properly quantify this uncertainty.
In the remainder of this paper, we examine epistemic uncertainty collapse in ensembles of ensembles and in wider neural networks. We offer theoretical justification for the former and propose implicit ensembling as a potential mechanism in the latter. We provide empirical evidence of such collapse in wider models through experiments with toy examples and MLPs trained on MNIST. Vice-versa, extraction methods, which decompose larger models into sub-models, recover hidden ensemble structure and epistemic uncertainty in both simple ML.Ps and state-of-the-art vision models based on ResNets (He et al., 2015) and Vision Transformers (Dosovitskiy et al., 2021).
show that implicit ensemble
2 Theoretical Framework
The quantification of uncertainty is crucial for robust and reliable predictions. We begin by distinguishing between two important types of uncertainty (Der Kiureghian & Ditlevsen, 2009):
1. Aleatoric uncertainty: Often referred to as statistical uncertainty, it represents the inherent noise in the data. This type of uncertainty is irreducible given the current set of features and cannot be reduced by collecting more data.
2. Epistemic uncertainty: Also known as model uncertainty, it stems from our lack of knowledge about the true data-generating process. This uncertainty can, in principle, be reduced by gathering more data or by using more expressive models.
Traditional Neural Networks vs. Bayesian Neural Networks. Traditional Neural Networks (NNs), while powerful function approximators, typically provide point estimates outputs and inherently do not.
2

---

Published in Transactions on Machine Learning Research (05/2025)
capture model uncertainty. This limitation means that regular NNs cannot easily distinguish between alcatoric and epistemic uncertainty.
In contrast, Bayesian Neural Networks (BNNs) infer a distribution over model parameters instead of single- point estimates. This probabilistic approach allows BNNs to capture epistemic uncertainty. When a BNN encounters data similar to its training distribution, the predictive distributions across the parameter distribution tend to be more concentrated, indicating lower epistemic uncertainty. For out-of-distribution data, on the other hand, distributions become more diffuse, indicating greater epistemic uncertainty.
Bayesian Model Average. The Bayesian Model Average (BMA) usually lies at the core of Bayesian deep learning. Let p(0) be the distribution over model parameters O We are interested in the posterior distribution p(0 D) given some observed data D, but this is not necessary for the rest of this work. Thus, let us use p(0) as the distribution interest over model parameters: here, it will be the distribution of possible model parameters that we obtain by running a training procodure given the training data and different. random initializations.
Let Y be the predicted output with likelihood p(y x). Then, the predictive distribution for a new input x" is:
p(y" 1101x'] xx) - / p(v" xxx,00 ,0, p(0) d0.
(1)
Quantities. To formally quantify and differentiate between aleatoric and epistemic uncertainty, we can use an information-thoorcti decomposition (Smith & Gal, 2018). Let Y be the predicted output, and I be the model paramcters. We define:
1. Total Uncertainty as the entropy of the predictive distribution of the BMA:
H|Y x] = -a-=- ff DDY
xx) pog |x)dY.
(2)
2. Epistemic Uncertainty 11Y;0) x!) the matual information which estimates the expected reduction in uncertainly about the prediction Y that would be obtained if we knew the model parameters O:
|| = H[Y x] Epco) [H[Y 0.
(3)
It has been shown be an effective measure of epistemic uncertainty (Houlsby al., 2011; Gal al., 2017; Smith & Gal, 2018).
While the choice of uncertainty measures remains an active research area (Hüllermeier & Waegeman, 2021; Wimmer et al., 2023; Bengs et al., 2022; Shen et al., 2024a), mutual information has strong theoretical and empirical support. The mutual information between predicted outputs and model parameters directly captures the core idea of epistemic uncertainty: how much additional data collection could reduce our predictive uncortainty. This interprctation usod in both active learning (Houlsby et al., 2011; Gal et al., 2017) and Bayesian optimal experimental design (Rainforth et al., 2024), where it is known the expected information gain. The effectiveness of this approach requires only that the training procedure is approzimately Bayesian in its treatment of new data. This property has been demonstrated across many machine learning methods (Mlodozeniec et al., 2024; Kirsch, 2023). Recent studies confirm the strong empirical performance of this mutual information compared to alternative uncertainty measures (Schweighofer et al., 2024). Other approaches based on variance decomposition via the law of total variance have been proposed (Kendall & Gal, 2017); we briefly compare these methods S.A.1.
2.1 Deep Ensembles
Deep ensembles, introduced by Lakshninarayanan el al. (2017), have emerged as a simple yel ellective method for approximating Bayesian inference in deep learning (Wilson & Izmailov, 2020). For a deeper discussion of the connection between deep ensembles and approximate Bayesian inference, see SA.2. They have been shown to offer superior performance compared more complex Bayesian approximations on many tasks (Ovadia et al., 2019), including calibration measurements (Guo et al., 2017), out-of-distributiio detection,
3

---

Published in Transactions on Machine Learning Research (05/2025)
and classification with rejection (Ashukha et al., 2020). The core idea is to train multiple neural networks independently using different random initializations. The BMA of deep ensemble of M models is:
M 2 ==1
1 M
P(Y
x)
P(Y x,0,);
(4)
where 0, p(0)) are the parameters of the j-th model in the ensemble, drawn from the distribution of possible trained models p(0). Given uniform index distribution p(j), we can estimate the epistemic uncertainty via the mutual information between the predicted class and the model index (Beluch al., 2018):
I[; θ x] H(Epp)) x, 0,))) PCC) [H x, 0,11-
(5)
This is just a Monte-Carlo approximation of the mutual information between the predicted class and the parameter distribution where the ensemble members are seen as samples from p(0).
3 Epistemic Uncertainty Collapse for Ensembles of Ensembles
Let us formally investigate a paradoxical phenomenon: the collapse of epistemic uncertainty when construct- ing ensembles of ensembles. A standard deep ensemble comprises M independently trained models that approximate Bayesian Model Average (BMA). A different way to "scale up" doep ensemblesis to ereate multiple deep ensembles and then average their predictions. One might expect that this ensemble of ensembles would yield more robust epistemic uncertainty estimates. However, our analysis has the opposite outcome: as we increase the size cach individual ensemble, the disagroement between these ensembles and consequently, our estimate of epistemic imcertainty-vaiiie
3.1 Setup and Predictive Distribution
Let 5 be the number sub-ensembles, and let each sub-ensemble & contain M models. Denote the parameters the j-th model in the i-th sub-ensemble by 0i, ~ p(0). Each sub-ensemble provides a predictive distribution:
M
ply x, 0.g). p = M
Σ 1=1
(6)
Once we collect sub-ensembles into E1s = EE, Es}, the aggregated predictive distribution
p④y x, E1.8)
Σ ply E;)
(7)
S
M
1
ply x, 0c.)).
(8)
SM
Viewed as a single (larger) ensemble, this model a mixture over all S x M members. Denote by I (1,.... S) S} the random variable for the choice of sub-ensemble, and by E [I....,M] the random variable for the choice of model within that sub-ensemble. Then each 0G, chosen with probability shr yielding a Bayesian mixture over members. The empirical epistemic uncertainty within sub-ensemble given by I[Y; x) and the empirical epistemic uncertainty across sub-ensembles is given by I[Y; x].
3.2 Limit of Increasing Sub-Ensemble Size
A key observation is that growing M, holding S fixed, causes all sub-ensembles to converge to identical predictive distributions, any disagreement across sub-ensembles vanishes. Formally, for M - 0o,
1 M
p(y x Ex) = M
Σ T=1
ply x, B15) - Ence)p(()) x)]
(9)
4

---

Published in Transactions on Machine Learning Research (05/2025)
since the LHS is Monte-Carll estimator of the RHS term. This implies that sub-ensemble indices i become irrelevant: all sub-ensembles yield identical outputs in the limit. Consequently, the epistemic uncertainly of the ensemble of ensembles the mutual information between the ensemble index (4) and the predicted label y collappses to zero as the predictions of Y become independent of the ensemble index. In other words,
+ as M +c. xc.
(10)
We call this epistemic uncertainty collapse.
Epistemic Uncertainty Collapse
As the size of the sub-ensemble in an ensemble of ensembles increases, the epistemic uncertainty of the overall ensemble approaches zero, and we observe an epistemic uncertainty collapse. This collapse occurs because the individual ensembles converge to similar predictive distributions, leading to a reduction in the mutual information between the predicted class and the ensemble index.
See $A.3 for a more detailed and formal argument.
3.3 Chain Rule Decomposition
To understand how the epistemic uncertainty single ensemble relates to that an ensemble of ensembles,
note that
Or; [x] & x] + 1[y x).
(11)
In words, the epistemic uncertainty of large ensemble can be broken into two parts: disagreement across sub-ensembles & x] plus disagreement within each sub-ensemble 0r, [1, As M > co, cach sub-ensemble individually captures a near-complete BMA, so there is negligible disagreement across sub- ensembles.
4 Implicit Ensembling
Having established the epistemic uncertainty collapse for explicit ensembles of ensembles, this raises the question whether similar phenomena occur within individual large neural networks. We hypothesize that as neural networks increase width, they might naturally exhibit ensembling internally, leading reduced epistemic uncertainty estimates. This would have important implications for uncertainty quantification in large models, including modern foundation models.
Concretely, we propose that large neural networks inherently function as "implicit ensembles," where operations like matrix multiplication (or average pooling) effectively average across many units similar to averaging predictions in explicit ensembles. Under this hypothosis, wider networks would exhibit stronger intornal ensembling effects, and ensembles of such networks would behave like ensemblee of ensembles. This suggests that the epistemic uncertainty collapse demonstrated in $3.2 would manifest as model width increases, potentially limiting uncortainty quantification benefits from simply scaling model size.
This implicit ensembling hypothesis guides our empirical investigations in the subsequent section. We examine whether epistemic uncertainty decreases as MLP width increases despite stable accuracy, and whether we can extract ensemble-like structures from single large models to recover the collapsed uncertainty. We also investigate average pooling in vision models as another potential form of implicit ensembling, where activations from different. receptive fields are aggregated in ways that might contribute uncertainty collapse.
4.1 Connection to Neural Tangent Kernel Theory
The hypothesized epistemic uncertainty collapse in larger models connects elegantly with Neural Tangent Kernel (NTK) theory (Jacot et al., 2018; Loe et al., 2019; Yang, 2019). NTK theory characterizes infinitely wide neural networks trained with gradient descent, showing that under appropriate conditions (initialization and learning rate), these networks converge to Gaussian Process predictors determined by the NTK. That.
5

---

Published in Transactions on Machine Learning Research (05/2025)
3.055
0.905
241
0.980
12G
0.975
Eplstomic Uncorainty [[8:7
2.005
0,970
3.000
0.0
10
Sub-Ensemble
(a) Mutual Information Histogram on SVHN (OoD) (b) OoD Detection for CIFAR10 vs. SVHN (OoD)
Figure 2: Ensemble of Ensemble Results for CIFAR10 (iD) vs. SVHN (OoD). Different configurations of 24 ResNet-50 models trained on CIFAR-10. (a) As the sub-ensemble size increases, the epistemic uncertainly on SVHN as OoD dataset collapses. (b) The area under the receiver-operating characteristic (AUROC f) for OoD detection using mutual information slowly deteriorates as the sub-ensemble size increases.
DnyMMST
FastionMMM1
(MM
0.5
0223
0,233
0.12
0.41
6:148
0.77
0/288
0.91
0.114
0410
X2
0.5
0.13
6.155
0.4
(34 53.55))
0.2
X.65
60
0.
00
133
(3HL)
(a) Mutual Information ECDF Different MLP Widths
Information
Entropy
3.0
1.0
0.6
0.5
A/WOC (Mon Stoee DP1
AURGE Jaonn Scem (.F)
0.2
K2. 17hh 21
k1/00 1rg
0.2
89 (8.00)
25%
85.97% 21.171
04 con 50
05.5% 20
84.20% 21101
0.0
54 10.00)
0.0
0.2
0.4
0.6
0.8
3.0
3.0
0.2
0.4
0.6
0.8
3.0
FRR (Risz OoD DetectionsID Samptes)
(b) MNIST vs. FashionMNIST OoD Detection AUROC
Figure 3: Epistemic Uncertainty Collapse on MNIST via Implicit Ensembling. (a) Mutual Information Empirical Cunulative Distribution Function (ECDF) for Different MLP Widths. As MLP size increases, mutual information decreases while accuracy remains stable. This trend persists across training and other distributions. (b) MNIST us. Fashion-MNISS OoD Detection AUROC Curves. The mean difference in uncertainty scores between in-distribution and out-of-distribution samples (in parentheses) also decreases with width, further evidencing epistemic uncertainty collapse, while the AUROC for OoD detection slightly improves across both uncertainty metrics.
is, for a fixed architecture, networks converge to identical predictive functions regardless of their random initialization in the infinite-width limit. Different initializations can follow different optimization paths but ultimately reach the same functional form. This has profound implications for uncertainty estimation.
In an ensemble of infinitely wide networks, each member converges to the same predictor function. The disagrocment between ensemble members, which is the foundation of epistemic uncertainty estimation, therefore vanishes completely. Any uncertainty measure based on this disagreement, such as mutual information between predictions and ensemble membership, collapses to zero. This directly parallels our earlier findings
6

---

Published in Transactions on Machine Learning Research (05/2025)
in $3.2: as sub-ensemble size increases, each subeensesble's predictive distribution approaches the common NTK-governed predictor, eliminating inter-ensemble disagreement.
Thus, NTK theory offers a theoretical framework that could be related to our proposed implicit ensembling hypothesis, though the connection remains speculative. As neural networks grow wider, their behavior increasingly approaches the NTK limit where all initializations converge to identical predictive distributions. We hypothesize that individual network components may still maintain considerable internal diversity, but this diversity might become progressively masked at the output level due to averaging effects in wider layers and pooling operations. If this hypothesis is correct, width increases, such implicit averaging mechanisms would grow stronger, causing the network's collective behavior to approach the NTK-governed predictor despite diverse internal representations. While NTK characterizes the infinite-width limit where uncertainty vanishes at the output, the implicit ensembling framework might help explain how this phenomenon occurs in practical finite-width networks and suggest how hidden uncertainty could potentially be recovered by accessing these diverse internal components. We present initial evidence for this through implicit ensomble extraction in $5.1.
5 Empirical Results
In the following section, we present a series of experiments that demonstrate the epistemic uncertainty collapse in explicit ensembles of ensembles and show similar patterns in the behavior of wide neural networks While these patterns are consistent with the implicit ensembling hypothesis we have proposed, we note that other explanations might also account for these observations. Details on the models, training setup, datasets, and evaluation are provided in $B and in &C in the appendix.
Toy Example. To illustrate the epistemic uncertainty collapse in ensembles of ensembles, we present a one-dimensional regression task with the ground-truth function [(x) = sin(x) + e, where 6~ (0, We generate a small training dataset of four points over [-5,5] and fit this data using ensembles of neural networks with one hidden layer of 64 units and tanh activations. The output layer also uses a tanh activation, scaled by a factor of 2 to allow for a output range that surely covers any noised sample.
We create ensembles of 10 sub-ensembles each, with sub-ensemble sizes of 1, 2, 4, 8, 16, 32, and 64 members. Figure presents the results across three pancls. The left panel shows ensemble predictions converging to the true function as sub-ensemble size increases, with narrowing uncertainty bands. The middle panel illustrates decrcasing epistemic uncertainty across the input space for larger sub-ensembles between training points. The right panel quantifies the inverse relationship between sub-ensemble size and mean epistemic uncertainty.
These results demonstrate the epistemic uncertainty collapse phenomenon: the systematic reduction of epistemic uncortainty, visible as reduced prediction variance and thus higher overall confidence, even in regions far from the training data, This loy example highlights how the proposed implicit ensembling within wide neural networks could lead to underestimating epistemic uncertainty, particularly in data-sparse regions.
In $A.4 in the appendix, we observe that epistemic uncertainty collapse also occurs in ensembles of random forests (Breiman, 2001), fundamentally different model class from neural networks. As we discuss in $6, this distinguishes epistemic uncertainty collapse from phenomena like feature collapse (Van Amersfoort et al., 2020) and neural collapse (Papyan et al., 2020) that are specific to deep neural networks and their training dynamics.
Explicit Ensemble of Ensemble. In Figure 2, we construct a deep ensemble comprising of 24 Wide- Res.Net-28-1 models (Zagoruyko & Komodakis, 2016; He et al., 2015) trained on CIFAR-10 (Krizhevsky et al., 2009), which we then partition into ensembles of ensembles with varying sub-ensemble sizes (24 x 1, 12 x 2, x 3,6 x 4, 4 x 6, 3 x 8, 2 x 12). Figure 2(a) illustrates the epistemic uncertainty of these ensemble configurations. As the sub-ensemble size incrcases, we observe clear epistemic uncertainty collapse, manifested by the mutual information concentrating on smaller values. Figure 2(b) presents the AUROC for oul-of-distribbttion (OoD) detection for different sub-ensemble sizes, comparing CIFAR-10 (in-distribution) with SVHN (OoD) (Netzer et al., 2011) using epistemic uncertainty as the detoction metric. The AUROC shows a deterioration as the sub-ensemble size increases, directly resulting from the epistemic uncertainty collapse. Crucially, while the decrease in AUROC may appear modest, it is large enough to make the difference wetween state-of-the-art.
7

---

Published in Transactions on Machine Learning Research (05/2025)
performance and baseline methods, such confidence entropy-basee approaches (Hendrycks & Gimpel, 2017), e.g, compare with the results in Mukhoti et al. (2023).
Implicit Ensembling on MNIST. Surprisingly, the effect of the epistemic uncertainty collapse is even visible when training relatively small MLP models of varying width on MNIST in a controlled setting. We reproduce the results from Fellaji & Pennerath (2024) in Figure 3. That is, we expand the size of the inner linear of a 2-hidden-layer MLP by a varying factors and train 10-model ensembles for 100 epochs each.
To quantify uncertainty, we employ two metrics that capture different aspects of the model's predictive distribution. Specifically, we utilize: (1) The predictive entropy of the ensemble, H[Y |x) which measures the overall uncertainty in the prediction (Hiendrycks & Gimpel, 2017); and (2) the mutual information between the predicted class and the ensemble index, as an empirical estimate for I[Y; O x]. which quantifies the epistemic uncertainty or model uncertainty (Malinin, 2019).
These results demonstrate epistemic uncertainty collapse in wider neural networks. This pattern is consistent with our implicit ensembling hypothesis, though other explanations may also be possible. As the width of the MLP increases, we observe clear trend of decreasing mutual information across all datasets: MNIST (LeCun & Cortes, 1998), Dirty-MNIST (Mukhoti et al., 2023), and Fashion-MNIST (Xiao et al., 2017). In Figure 3(a), the effect is most pronounced for the in-distribution MNIST test data, but the collapse is also clearly visible for out-of-distribution data (Fashion-MNIST & SVHN test sets). The decrease in mutual information indicates a reduction in the model's epistemic uncertainty as it grows larger, despite maintaining similar accuracy. In Figure 3(b), the mean difference in uncertainty scores between in-distribution (MNIST) and out-of-distribuutin (Fashion-MNIST) samples also decreases with increasing model width, further corroborating the collapse of epistemic uncertainty. However, the AUROC for OoD detection using different uncertainly metrics slightly improves the model width increases, which is in line with the results by Fellaji & Pennerath (2024), who report a significant deterioration of OoD performance for such MLP ensembles trained on CIFAR-10 but not MNIST.
This shows that epistemic uncertainty collapse might be pervasive phenomenon in neural networks, even in relatively simple MLP models.
5.1 Implicit Ensemble Extraction
To further investigate the hypothesis that implicit ensembling might contribute to epistemic uncertainty collapse, we demonstrate that it is possible to mitigate this collapse by decomposing larger models into constituent sub-models. This approach, which we term implicit ensemble extraction, allows us to recover the diversity of an ensemble from a single large model, potentially reversing the collapse of epistemic uncertainty. This not only offers insights into the nature of epistemic uncertainty collapse but also points towards potential practical strategies for improving model robustness and didoof--istribution detection.
Extracted Implicit Ensemble from a Single MNIST MLP. First, we extract implicit ensembles from the MLPs trained on MNIST above. For a given model, ensemble member with the largest width factor, 64, we train boolean masks on the weights such that we recover 10 individual models with maximally different. masks and low individual loss on MNIST's training set. We find that the resulting deep ensemble performs as well as the ensemble of smaller models, even though we started from a single model.
Concretely, we add binary mask to each linear layer of the network which we relax to probabilities of binomial variables by applying sigmoid activations, effectively sclecting subsets of the original weights We optimize separate 1D masks for its rows and columns. The outer product of these 1D masks determines which weights from the original layer are included in each sub-model among the resulting sub-models by regularizing the mutual information I[mask; M] between the masks and sub-model index M while minimizing the loss on the training set. This allows us to extract an implicit ensemble from a single trained network.
dense sub-matrix. We maximize the diversity
In Figure 4, we see the effectiveness of this implicit ensemble extraction technique. The results demonstrate that this decomposition can recover much of the epistemic uncertainty of an ensemble from a single model, providing support for our hypothesis about implicit ensembling:
8

---

Published in Transactions on Machine Learning Research (05/2025)
FashionMISST
Noa1
1.2
Mear
2.0
Mean
2.0
6:223
0.212
0.615
3.2
5.100
1,000
0.017
0:182
0.005
N.t.alnforrttoo
0.4
IN
0.4
0.4
1.0
044
0.5
120,03%1
196.1955
0.31
0.25
3.0
1.0
0.
2.0
a score)
(a) Mulual Information ECDF for Extracted Implicit Ensemble
Mutuel Intormaioo
Errap
Determrrdtl: Decomposed Entropy
1.0
0.5
3.5
26
AURDIC
TPR (Thw
1287% (317)
RLIET (007
MLAPS 2.15
83.9% 2211
IDEIN 2.1)
52
9.6
5.0
522
0.6
1/3
20
02
04
04
50
TST abe Del Detard emoC Sampllee)
(b) MNIST vs. MNSS OoD Detection AUROC for Extracted Implicit Ensemble
Figure 4: Recovering Epistemic Uncertainty through Implicit Ensemble Extraction. (a) The extracted implicit ensemble (dashed line) largely recovers the mutual information scores of fully trained ensemble of the same width, supporting the hypothesis of latent ensemble structures in large neural networks. The 10-member implicit ensemble is extracted from a single MLP with width factor 64 ($5.1). The regular 10-member ensembles comprise MLPs with width factors 4 and 64 trained MNIST. Ensembles are evaluated on MNIST, Dirty-MNIST, and Fashion-MNIST test sets. (b) The extracted implicit ensemble shows comparable AUROC scores across all metrics relative to a fully trained deep ensemble of the same width. OoD detection is performed using mutual information or entropy scores. The final panel compares the softmax entropy the original wide MLP with the predictive entropy of its extracted implicit ensemble. The mean entropy difference between and OoD samples is larger for the extracted ensemble. At the same time, the OoD performance does not match the single wider MLP.
Effectiveness of Model Decomposition. The dashed lines in both subfigures of Figure 4 represent the extracted ensemble, as described the previous section. Remarkably, this decomposed model nearly recovers the mutual information scores of fully trained ensemble with the same width factor despite being based on the weights of single model, and its OoD detection performance approaches that of a narrower model. The fact that we succossfully extract ensemble structure suggests that diverse functional components may exist within larger models, which is consistent with our implicit ensembling hypothesis.
Predictive Entropy of the Extracted Ensemble vs. the Softmax Entropy of the Original Model. The last panel of Figure 4(b) compares the softmax entropy of the original large model with the predictive entropy of the decomposed ensemble. While we cannot compare the OoD detection performance of the extracted ensemble to the single wider MLP using mutual information (as the latter does not have well-defined mutual information), we that the extracted ensemble does not match the performance of the single wider MLP when using its predictive entropy. We speculate this might be because the masks do not fully cover the weights of the original model, and thus the extracted ensemble does not capture the full predictive entropy of the original ensemble.
The initial success of this model decomposition technique in recovering epistemic uncertainty suggest a promising dircction for improving uncertainty quantification and out-of-distribution detection individual over-pate pre-paaramtterzeed deep learning models. Collectively, this demonstrates that epistemic uncertainty collapse occurs even in relatively simple MLP models.
9

---

Published in Transactions on Machine Learning Research (05/2025)
5.050
9.25
UIKEES POOK
6.903
0.20
EYPOW
2x2
6.893
0.15
3.10
1.993
Cpidamnc Uncoraaddy Accapaa Trruheld
Quitercc Accuptanca Thrahall
Epkitarnc Uncartarry Bedat
(a) ResNet models with temperature 1.0
0.015
0.935
UIEET my PADi
Crasmble
3.16
Natud atarratin
3+2
6.015
0.032
6.22
2.00
774
5.803
04
1.0
1.2
3.0
2.5 Fpltteeu Unnertairyy Rudiet horegge
Epidenic Uncemsliyy Acceptance Treddald
qp-tev tiseeniy Acceptance Thinduld
(b) ResNet models with optimal temperature
0.912
0.956
Oree
Uncamante Maric
Dramns
2xg
Welgttd
22x03
12x12
18x14
13x25
Opicarts Uncorbaryy Acoplanca Tirenoo
Esitani www.raatt Accedtanc Trasstd
Caloani Unovrtart Rucast Anaraee
(c) Max model with weighted mutual information and optimal temperature
0.94
Calornnn Kror
Uncemaato Perk
129
0.92
22613
0.50
75T
35x35
1x-2 DIISDERR Aocptaay Thinced
(d) VaxViT model with mutual information and optimal temperature
Figure 5: Classification with Rejection for Implicit Ensemble Extractions from Pre-Trainch Models. Each subfigure shows three performance metrics (Accuracy, Negative Log-Liiklihood, and Calibration Error) as a function of epistemic uncertainty quantiles for different ensemble sizes. Solid lines represent extracted ensembles of increasing size (from 2 to 7/16), while the dashed black line represents the original single model. (a) The mutual information between predictions used the epistemic uncertainty measure for ensembles, while entropy is used for the single model. As the ensemble size increases, observe improved performance for the area under curve (AUC), which indicates better epistemic uncertainty calibration (with the notable exception of the calibration error). This demonstrates that extracting larger eusembles from a single tre-treined model can enhance performance and uncertainty quantification. (b) Temperature scaling improves epistemic uncortainty calibration in general but benefits the original model most. Accuracy and NLL for extracted epistemic uncertainty only benefit in the low-uncertainty regime. (c) For VIT models, we find that a mutual information weighted by the logit sum each ensemble performs better than the mutual information ((c) vs (d) with mutual information).
Extracting Ensembles from Pro-Trained Vision Models. Second, we explore implicit ensemble extraction from pre-trained vision models based on ResNet (He et al., 2015) and Vision Transformer (Dosovitskiy et al., 2021) model architectures. Leveraging the common use of average pooling in these models to aggregate spatial information, we extract implicit ensembles without optimizing masks. Coneretely, we remove the global average pooling layer. This allows us to obtain per-tile class logits, which we average with different target sizes to create differently-sized ensembles.
10

---

Published in Transactions on Machine Learning Research (05/2025)
We evaluate these models in-distributtin the ImageNet-v2 dataset (Recht al., 2019), which serves as more challenging test set for ImageNet-trained models (Russakovsky et al., 2015). Specifically, we compare pre-trained ResNet-152 (He et al., 2015), Wide ResNet-101-2 (Zagoruyko & Komodakis, 2016), ResNeXt-101- 64x4d (Xie et al., 2017), and ViXViT (Tu et al., 2022) models, with the pre-trained weights retrieved from Py Torch's torchvision (maintainers contributors, 2016) and timm (Wighlman, 2019), respectively. Our evaluation pipeline computes various uncertainty metrics and performance measures for different ensemble sizes, ranging from 2 x to 7 x sub-models (respectively, 16 16 for MaxViT) extracted from a single pre-trained network. We compare these extracted ensembles against the original single model performance, using mutual information as the primary uncertainty metric for ensembles and entropy for the single model.
Figure 5 shows three key performance metrics accuracy, negative log-likelihood (NLL), and calibration error snoorppooted against epistemic uncertainty quantiles of various extracted ensemble sizes for the original models and for temperature-scaled models (Guo et al., 2017) The solid lines represent extracted ensembles of increasing size, while the dashed black line represents the original single model. For ResNet- ensembles, we use mutual information between predictions as the measure of epistemic uncertainly, whereas for the single model, we use entropy. For MaxViT, we use a weighted mutual information between predictions and ensemble size as the mcasure of epistemic uncertainty, which assign a weight to cach ensemble member basod on the logit sum of the member as it performs better.
For ResNet models, we find that the largest extracted ensembles (7x7) have a better response overall for accuracy and NLL, while the calibration error is worse throughout. For temperature-scales model, we observe an almost linear response in accuracy and NLL, with better sensitivity in the low-uncertainty regime, but otherwise worse performance, except for lower calibration error. For MaxViT, find that the weighted mutual information performs better than the mutual information. The effects are less pronounced for temperature-scaled models, likely because these models already are better calibrated. Overall, the results are mixed but promising.
Limitations. These exploratory results provide initial evidence that ensemble extraction technique can unlock meaningful epistemic uncertainty, which is useful for downst ream tasks. While this is consistent with the implicit ensembling hypothesis, it does not prove that implicit ensembling is the mechanism responsible for uncertainty collapse in wide networks. At the same time, already see that mutual information is not always the best uncertainly metric, the comparative performance changes depending the model temperature, and the pool size of the best implicit ensemble is not always the same for different metries and model architectures. More results that show this are shown in $E.
6 Related Work
Our study epistemic uncertainty collapse in large neural networks and ensemble extraction intersects with several recent works adjacent areas.
Epistemic Uncertainty Collapse. A recent preprint by Fellaji & Pennerath (2024) observed decreasing epistemic uncertainty as model size and dataset size vary, even in simple MLPs. They termed the decrease in epistemic uncertainty the "epistemic uncertainty hole" but left its explanation to future work. Our study proposes an explanation through a theoretical framework for ensembles of ensembles, the hypothesis of implicit ensembling for wider models, and additional empirical evidence from experiments on ensemble extraction. A more detailed comparison can be found in SD.
Extracting Sub-Models from Larger Network. The concept of extracting sub-models from a larger network shares similarities with several existing approaches in the literature. The "lottery ticket hypothesis" (Frankle & Carbin, 2018) proposes that dense, mandomly-initilizeee networks contain sparse subnetworks capable of training to similar accuracy. However, our approach differs in that we do not retrain the subnetworks, but rather identily diverse substructures within the pre-trained model. Our method is more closely related to "sub-network ensembles" (Durasov et al., 2021), where multiple subnetworks are extracted from a single trained network form an ensemble. Unlike previous work that primarily focused on pruning for efficiency, our approach aims to recover epistemic uncertainty. We introduce a novel mutual information- objective to obtain diverse masks, emphasizing diversity rather than pruning. This allows us to extract an ensemble
11

---

Published in Transactions on Machine Learning Research (05/2025)
that better captures the model's internal epistemic uncertainty. This goal is more similar to Wortsman et al. (2020) which is concerned with discovering sub-networks during training for continual learning of diverse tasks without catastrophic forgetting. Our approach is only applied post-training, however.
Learning from Underspecified Data. Our work can also related to efforts in learning from underspecified data, such as the approach by Lee et al. (2022) diversify and disambiguate model predictions. While their focus is on training strategies, our method extracts diverse sub-models from already trained networks, offering complementary approach.
In contrast to these works, this work addresses fundamentally different research questions. We propose a novel, unified explanation for epistemic uncertainty collapse in large models and hierarchical ensembles, a phenomenon not previously explored in depth. Furthermore, we introduce and examine implicit ensemble extraction to mitigate epistemic uncertainty collapse.
Relationship to Feature and Neural Collapse. Our findings on epistemic uncertainty collapse share interesting connections with, but remain distinct from, two related phenomena in deep learning. Feature collapse, introduced by Van Amersfoort et al. (2020), describes how intermediate feature representations become increasingly similar during training, leading to overconfident predictions. Neural collapse (Papyan et al., 2020) characterizes how features and classifiers converge to specific geometric structure in the terminal phase of training.
While these phenomena involve forms of representation collapse, our observed epistemic uncertainty collapse operates at a different level. Rather than focusing on feature geometry or representation similarity, we study how predictive uncertainty estimates degrade in two key settings: (1) ensembles of ensembles, where individual ensembles converge to similar predictive distributions despite diverse training, and (2) large models through the proposed implicit ensembling, where we can extract diverse sub-models that reveal hidden uncertainty structure. The distinction becomes particularly clear in regression tasks (Figure 1), where neural collapse theory does not apply due to its reliance on classification-spcciii geometric propertics.
7 Conclusion
This work has analyzed the collapse of epistemic uncertainty in large neural networks and hierarchical ensembles. Our thcoretical framework and empirical results demonstrate this phenomenon across various architectures and datasets, from simple MLPs to state-of-the-arl vision models.
To further explore this phenomenon, we examined implicit ensemble extraction, method to decompose larger models into sub-models. This approach can recover diverse predictive distributions from a single model and improve epistemic uncertainty estimates. The effectiveness of this extraction method is consistent with our implicit ensembling hypothesis, though it does not definitively establish it as the underlying mechanism of epistemic uncertainty collapsc.
Our findings challenge the assumption that more complex models invariably offer better uncertainty quantifi- cation out of the box. They open new avenues for research into uncertainty estimation in large-scale machine learning models, particularly for safety-critical applications and out-of-distribution detection tasks. Future work should investigate the exact mechanisms behind epistemic uncertainty collapse in large models and develop robust methods to maintain reliable uncertainty estimates as model complexity increases, ensuring the continued advancement of trustworthy Al systems.
Acknowledgments
We thank the anonymous TMLR reviewers and Freddic Bickford Smith for their helpful foedback. Alex Evans provided valuable comments on an earlier draft, and Midjourney generously supported this personal research last summer. Jishmu Mukhoti trained the model checkpoints used in the initial explicit ensemble of ensemble experiments on CIFAR-10 as part of Mukhoti et al. (2023). Discussions with Lisa Schut helped shape the intuitions about NTK and implicit ensembling. We also appreciate Jannik Kossen and Tom Rainforth's 's insights following an early presentation at OATMI..
12

---

Published in Transactions on Machine Learning Research (05/2025)
References
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty ostimation and ensembling in deep learning. In International Conference on Learning Representations, 2020. URL https://openrevier http://rwweee.eetttormmiiiiaaaee net/forrumiiiBDbI55gHt
Neil Band, Tim G. J. Rudner, Qixuan Feng, Angelos Filos, Zachary Nado, Michael W Dusenberry, Ghassen Jerfel, Dustin Tran, and Yarin Gal. Benchmarking bayesian deep learning on diabetic retinopathy detection tasks. In Thirty-fijh fifth Conference on Newral Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https://openrevian wwt.:/oorumndd-iyy4l.......
William H Beluch, Tim Genewein, Andreas Nürnberger, and Jan M Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference on computer vision ond pattern recognition, pp. 9368 9377, 2018.
Viktor Bengs, Eyke Hüllerreier, and Willem Waegeman. Pitfalls of epistemic uncertainty quantification through loss minimisation. Advances in Neural Information Processing Systems, 35:29205 29216, 2022.
Leo Breiman. Random forests. Machine learning, 45:5 32, 2001.
Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? Does it matter? Structural Safety, 31(2):
105-112, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL. https://opeerevee// http://www..ttoomu/dd////ooaaaaa
Nikita Durasov, Timur Bagautdinox, Pierre Baque, and Pascal Fua. Masksembles for Uncertainty Estimation. In 2021 LEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13534 13543. IEEE Computer Society, 2021.
Mohammed Fellaji and Frédéric Pennerath. The Epistemic Uncertainty Hole: an issue of Bayesian Neural Networks. Xix preprinl arXiv:2407.01988, 2024.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arXiv preprint Xix: 02757, 2019.
Jonathan Frankle and Michael Carbin. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In International Conference on Learning Representations, 2018.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning, pp. 1183 1192. PMLR, 2017.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1333 PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arxiv e-prints. arXiu preprint arXiv:1512.0053 10, 2015.
Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. In International Conference on Learning Representations, 2017. URL https: //openreview.nn nott/oormiid
Neil Houlsby, Ferenc Huszar, Zoubin Ghahramani, and Máté Lengyel. Bayesian Active Learning for Classifi- cation and Preference Learning. Xiv preprint, abs/1112.5745,, 2011.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Eyke Hüilermeeer and Willem Waegeman. Alcatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learnöng, 110(3):457-500 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in newral information processing systems, 31, 2018.
Alex Kendall and Yarin Gal. What Uncertainties Do we Need in Bayesian Deep Learning for Computer Vision? In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems, 2017.
Andreas Kirsch. Black-box batch active learning for regression. or preprinl arXiv:2302.08981, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Balaji Takshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Yann LeCun and Corinna Cortes. The mnist database of handwritten digits. 1998. URL https: //api. semanticscholas orgo/o////oooooo It:///oorII::::::::0
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstinn,, and
Jeffrey Pennington. Wide neural networks of any depth evolve as lincar models under gradient descent. Advances in neural information processing systems, 32, 2019.
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from underspecified data. ICML 2022: Workshop on Spurious Correlations, Invoriance and Stability, 2022.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL
David JC MacKay, A practical bayesian framework for backpropagation networks. Neurol computation, 4(3): 448-472, 1992.
TorchVision maintainers and contributors. Torchvision: Pytorch's computer vision library. https://githu:
con/pytorch/visior 2016.
Andrey Malinin. Uncertainty estimation in deep learning with application spoken language assessment. PhD thesis, University of Cambridge, 2019.
Bruno Modozeniee, David Krueger, and Richard Turner. Implicitly bayesian prediction rules in deep learning. In Symposium on Advances in Approzimate Bayesian Inference, pp. 79 110. PMLR, 2024.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deep Deterministic Uncertainty: A New Simple Bascline for Uncertainty Estimation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pottern Recognition, pp. 24384-24394 2023.
Radford M Neal. Bayesion learning for neural networks. PhD thesis, University of Toronto, 1994.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and stanford. Unsupercised Feature Learning 2011, 2011. URL ord edl/houseeumbers/nipp//// housenumbers, pdf.
Ian Osband, Scyed Mohammad Asghari, Benjamin Van Roy, Nat McAleese, John Aslanides, and Gcoffrey Irving. Fine-tuning language models via epistemic neural networks. arXiv preprint arXiv:2211.00 0..022211.1168. 2022.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Scbastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? Evaluating predic- tive uncertainty under dataset shift. In Н. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https: cttps://proceeding" httppoooeeinn...nuuipppppep nwwrins..rcoooo fttpppppr.ille/pppp//////////// 85581800011776221111188868778888
pdf.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):246522244)) -4666366322 2020.
Tom Rainforth, Adam Foster, Desi R. Ivanova, and Freddie Bickford Smith. Modern bayesian experimental design. Statistical Science, 39(1):100-114, 2024.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt,, and Vaishaal Shankar. Do ImageNet classifiers gen- eralize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, html.
pp. 5389 5400. PMLR, 09 15 Jun 2019. URL https://ppett ptttps:/prooeeedngg.. mlr, press/v97/rechoe
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Kranse, Sanjoev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Adilya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
Kajetan Schweighofer, Lukas Aichberger, Mykyta lelanskyi, and Sepp Hochreiter. On information-theoree measures of predictive uncertainty. arXiv preprint arXiv:2410.1072: 2024.
Burr Settles. Active learning literature survey. 2009.
Maohao Shen, Jongha Jon Ryu, Soumya Ghosh, Yuheng Bu, Prasanna Sattigeri, Subhro Das, and Gregory W Wornell. Are uncertainty quantification capabilities of evidential deep learning a mirage? In The Thirly- eighth Annual Conference on Neural Information Processing Systems, 2024a.
Yucsong Shen, Nico Dahcim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas MöllenhofT. Variational learning is effective for large deep networks. In International conference on machine learning, 2024b. URL attps://openreview http://pparreeeeaaeaeaoooaaaaaaa wtt.:e/ooormn/ddxxeB///0000
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. arXiv preprinl X88: 08533, 2018.
Mandt Stephan, Matthew D Hoffman, David M Blei, et al. Stochastic gradient descent as approximate bayesian inference. Journal of Machine Learning Research, 18(134):1-35, 2017.
Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. Advances in neural information processing systems, 32, 2019.
Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint ar xrx::22277004 11, 2022.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. MaxViT: Multi-axis vision transformer. In European conference on computer vision, pp. 479. Springer, 2022.
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning, pp. 9690-9700. PMLR, 2020.
Zheng Wen, Tan Osband, Chao Qin, Xiuyuan Lu, Morteza Tbrahimi, Vikranth Dwaracherla, Mohammad Asghari, and Benjamin Van Roy. From predictions to decisions: The importance of joint predictive distributions. arXio preprint arXi::2107.0922,, 2021.
Ross Wightman. Pytorch image models. https:/
2019.
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. A Advances in neural information processing systems, 33:4697-4777 -406, 2020.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Htilermmier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Uncertainty in Artificial Intelligence, pp. 2282 2292. PMLR, 2023.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Anirdddaa Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173 15184, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arxiv preprinl, abs/1708.007477, 2017. URL http://arxivv.rrg/ wttp:////or//a/////aa/a/aasss/// http//110..077744...
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pp. 5987-5995. Institute of Electrical and Electronics Engineers Inc., 2017.
Greg Yang. Wide feedforward or recurrent neural networks of any architecture are gaussian processes. Advances in Neural Information Processing Systems, 32, 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
16

---

Published in Transactions on Machine Learning Research (05/2025)
A Theoretical Framework
A.1 Variance-Based Epistemic Uncertainty
A convenient way to obtain tractable upper bound for the mutual information I[Y; x uses the law of total variance the random variable Y. Recall the law of total variance:
VarlY x] [[[ar[Y x, θ]] + Var[E[| x, θ]].
(12)
It is sometimes used an uncertainty decomposition in its own right (Kendall & Gal, 2017), where the first term on the right E|Var[Y| x, O]] is the average conditional variance across models, considered the alcatoric part, while the second term Var[E[Y x, θ]] quantifies how the conditional mean changes with θ considered the epistemic part.
To connect this decomposition to O[]; assume each conditional distribution p(Y |x, θ) is (approximately) Gaussian with constant covariance, and approximate p(Y | x) using a Gaussian as maximum entropy distribution. The differential entropy of a Gaussian random variable with variance o² allows us to approximate ![ x] by comparing the entropy of these Gaussians:
2 log(2re This
0 x = H x] - H[Y x, 6]
(13)
= =H(Y x| - F p()) [H[ x, 0]
(14)
1
(2iee VarlY xx) E / log(2Tre VarlY x, 0)]
(15)
2
1
Var Y x] E[Var|Y x, θ]]
$
log
(16)
2
1
1+ Var[E[y x, θ]]
log
(17)
2
E[Var[Y x,
The numerator reflects epistemic variability (how much the mean shifts between models), and the denominator captures the typical conditional variance. This ratio then appears within logarithm because differential entropy grows with the log of variance in the Gaussian setting.
From this, we can see that varianco-based epistemic uncertainty can be loosely matched to mutual information when each conditional distribution p(y] x, 0) is close to Gaussian with roughly constant variance, making total variance reflect the same factors that drive the mutual information. Outside this regime particnlaryy with multimodal, heavy-tailed, or strongly heterosoedastic predictive distributions the variance-based approach masks richer dependencies in p(y] x, 0), and the two metrics diverge as mutual information accounts for more complex, higher-order e
A.2 Deep Ensembles as Approximate Posterior Sampling
The theoretical justification for using deep ensembles to estimate epistemic uncertainty stems from their connection to Bayesian inference (Wilson & Izmailov, 2020). When training neural networks with differ- ent random initializations and stochastic optimization, the resulting ensemble can be viewed as drawing approximate samples from the posterior distribution over model parameters (Stephan et al., 2017; Shen et al., 2024b).
This interpretation arises because the training process combines:
Random initialization, which provides diverse starting points in parameter space (Garipov et al., 2018) Stochastic gradient descent, which introduces noise optimization ion trajectories (Stephan al., 2017)
Weight decay regularization, which acts as an implicit prior over parameters (Loshchilov & Hutter, 2019)
from 2022: Lau-of tar-of-Totall Vaaaaaaceeeeeeee Information har.
2This
section
is based on Informal note
https://blackhc..no www.iiii...nooiinooooo..
17

---

Published in Transactions on Machine Learning Research (05/2025)
While deep ensembles do not perform exact Bayesian inference, they capture important aspects of posterior uncertainty through the diversity of solutions found (Fort et al., 2019). The independent training runs explore different modes of the loss landscape (Garipov et al., 2018), approximating the kind of parameter diversity we would expect from true posterior samples. This provides practical foundation for using ensembles to estimate epistemic uncertainly, even without explicit variational approximations or MCMC sampling (Wilson & Izmailov, 2020).
The quality of this approximation improves with ensemble size, more independent training runs better characterize the range of plausible models given the data (Lakshminarayanan et al., 2017). This theoretical connection motivates why deep ensembles often provide better uncertainty estimates than methods that make stronger assumptions about the form of the approxinate posterior (Ovadia et al., 2019).
A.3 Epistemic Uncertainty Collapse in Ensembles of Ensembles
The rather informal exposition in $3.2 does not differentiale between empirical mutual information which measures the disagreement between the predictions of the ensemble members and the mutual information between the predictions and the parameter distribution where the ensemble members are scen as samples from p(0). is not necessary to understand the effect and the empirical results. Let us make this distinction clearer here for completeness' sake.
The empirical epistemic uncertainty of specific sub-ensemble & is I[Y; 01,2 1 x]. The epistemic uncertainty of the paramter distribution itself is I[Y: θ x]. We have:
T[: 01.1 & x] x, K,0,)))) H(E5[p(Y E, H(((( x, (.,))),
(18)
where p(j) follows uniform distribution. We can write this more explicitly as:
M P(Y x, 02,))
M Σ 1=1
IY;01,J E,, x] H(
Σ M
H(p(Y | x, 01,)).
(19)
M
The entropy is a contimuus function, so we can take the limit as the the number of models in the sub-ensemble M goes to infinity and swap the order of limit and entropy, where we use that 1 E p(y x, Bi,)) - Ep (p(o)(((y) M7
x, 8]] following (9) and similarly M ΣØ H(p(Y x, (.y)) + ((E)()) x, A)]. That is, for M oo, we
have:
M P(Y x, 02.))
1 M
1f; | = H(
M Σ j=1
Σ M
H(p(Y |x, 0x5))
(20)
==1
H(Ep(e))p(y x, 0)) E0(o0) [H(((v x, 0))
(21)
H(p(y xx) Epto) H(p(V x 0))
(22)
I[Y θ x].
(23)
Now, crucially, we make the same argument for the epistemic uncertainty of the ensemble comprising all models in all the sub-ensembles 0r, x], and we obtain the same result:
Of; |x] + xx=
(24)
With the chain rule of mutual information(11) we obtain:
0r, xx E[ xx] + 1[, [),
(25)
and in the limit of infinitely-sized sub-ensembles, the epistemic uncertainty across the sub-ensembles is thus sandwiched to 0:
& xx] II; x] - -TY: E,,x 1Y;0|x 1 = =0
(26)
18

---

Published in Transactions on Machine Learning Research (05/2025)
True Function, Data, and RF Predictions
Epistemic Uncertainty for Different Forest Sizes
Epistemic Uncertainty ve. Forest Size
Training den Tiae on
Wod
Mour Spnamm Unomeere
10
Treat per Randert Forest
Figure 6: Epistemic Uncertainty Collapse in Random Forest Ensembles. The panels show the collapse of epistemic uncertainty as the number of trees per forest increases. Left: Model predictions and uncertainty bands (=2 standard deviations) for different random forest sizes, showing convergence to the true function with reduced uncertainty as the random-forest ensemble size grows. Middle: Epistemic uncertainty across the input space for different raneom-forest ensemble sizes, demonstrating systematic reduction in epistemic uncertainty between the random-forees ensemble size increases. Right: Relationship between random-fores: forest ensemble size and mean epistemic uncertainty, with shaded regions showing 68% and 95% confidence intervals, revealing clear inverse relationship.
A.4 Epistemic Uncertainty Collapse in Random Forests
To further validate the epistemic uncertainty collapse phenomenon, we examine its manifestation in random -ooests-noother ensemble learning method with well-understood theoretical properties. An ensemble of random forests (Breiman, 2001) naturally forms an ensemble of ensembles, where each random forest an ensemble of decision trees.
We replicate the toy regression problem from earlier, training random forest ensembles with varying numbers of trees (1, 4, 16, 32, and 64) and crcating ensembles of ensembles of 10 random forests for cach configuration. Each random forest uses decision trees with maximum depth of to prevent overfitting while maintaining sufficient model capacity.
The results in Figure 6 mirror our findings with noural networks. As the number of trees in each random forest increases:
the ensemble predictions converge more tightly the true function, with narrowing uncertainty bands; the epistemic uncertainty decreases systematically across the input space, particularly in regions between training points; and
the mean epistemic uncertainty shows clear inverse relationship with forest size
This parallel between random forests and noural networks underscores that epistemic uncortainty collapse is a general phenomenon ensemble methods, rather than being specific neural networks. The observation that this effect appears in random carsss--hichh have stronger theoretical guarantees and more interpretable behavior than neural networks provides additional evidence for the fundamental nature of this phenomenon.
B Model and Dataset Details
B.1 MNIST Experiments
For the MNIST experiments, use simple Multi-Layer Perceptron (MLP) architecture with two hidden layers. The model structure is as follows:
Input layer: 784 units (28x28 flattened MNIST images)
First hidden layer: 64 units multiplied by a width multiplier (ranging from 0.5x to 256x)
Second hidden layer: 32 units multiplied by the same width multiplier
Output layer: 10 units (one for each digit class)
Activation function: ReLU after each hidden layer
19

---

Published in Transactions on Machine Learning Research (05/2025)
Dropout layers: Applied after each hidden layer with p=0.1
We train these models using the following configuration:
Optimizer: SGD
Learning rate: 0.01
Batch size: 128
Epochs: 100
Loss function: Cross-entropy loss
We create ensembles of 10 models for each width configuration. The width multipliers used are 1x, 2x, 4x, 8x, 32x, 64x, and 128x the base width.
Datasets used:
MNIST (LeCun Cortes, 1998): Standard handwritten digit dataset (in-distribution)
Pashion-MNIST (Xiao al., 2017): Clothing item dataset (out-of-distributtin)
Dirty-MNIST (Mukhoti et al., 2023): MNIST with added noise (high aleatoric uncertainty)
B.2 CIFAR-10 Experiments
For the CIFAR-10 experiments, we use -ide-ResNet-28-1 models (Zagoruyko & Komodakis, 2016). We create ensemble of 24 independently trained models, which we then partition into sub-ensembles of various sizes, following the training details of (Mukhoti et al., 2023).
Training configuration:
Optimizer: SGD with momentum (0.9)
Learning rate: 0.1, decayed by factor of 10 at epochs 150 and 250
Weight decay: 5e-4
Batch size: 128
Epochs: 350
Loss function: Cross-entropy loss
Datascts used:
CIFAR-10 (Krizhevsky al., 2009): 10-class image classification dataset (in-distribution) SVHN (Netzer et al., 2011): Street View House Numbers dataset cutt-of-distribution)
B.3
ImageNet Experiments
For the ImageNet experiments, use pre-trained models from the torchvision and timm libraries:
ResNet-152 (He et al., 2015)
Wide RosNet-101-2 (Zagoruyko & Komodakis, 2016)
ResNext--10-64x4d (Xie et al., 2017)
MaxViT (Tu al., 2022)
These models evaluated the ImageNet-v2 datasct (Recht et al., 2019), which serves more challenging test set for InageNeet-triieed models.
B.4 Implicit Ensemble Extraction
For the implicit ensemble extraction experiments on MNIST, we use the following approach:
Starting model: MLP with width factor 64
Extraction method: Optimizing binary masks for each layer
Number of extracted sub-modcls: 10
Optimization objective: Maximize mask diversity (mutual information between masks and sub-model
index) while minimizing the cross-entropy loss training set.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Mask diversity weight: 2.0
For the pre-trained vision models, we extract implicit ensembles by:
Removing the global average pooling layer
Obtaining per-tile class logits
Averaging these logits with different target sizes (from 2x2 to 7x7 for ResNets, and up 16x16 for MaxViT)
Table 1: AUROC and Mean Difference for Different Metrics and MLP Widths
AUROC
Mean MI Difference
OoD Metric
MI
Entropy
MI
Entropy
MLP Width
x1
0.824
0.833
0.188
0.482
x2
0.835
0.841
0.194
0.420
x4
0.840
0.845
0.169
0.362
x8
0.842
0.847
0.144
0.329
x32
0.847
0.851
0.099
0.285
x64
0.848
0.851
0.104
0.280
x1
0.848
0.851
0.100
0.275
C
Evaluation Details
This section provides detailed information about evaluation metrics, with particular focus the weighted mutual information and calibration error calculations.
C.1 Weighted Mutual Information
For the MaxViT model, we introduce a weighted mutual information metric to measure epistemic uncertainty. This metric assigns a weight to each ensemble member based on the logit sum of that member. The weighted mutual information is calculated as follows:
1. For cach input, we compute the logits for all ensemble members.
2. We calculate the sum logits for each ensemble member.
3. We normalize these sums to create weights for each member.
4. We compute the mutual information between the predictions and the ensemble index, weighting each member's contribution by its normalized logit sum.
Formally, let 4. be the logit sum for the i-th input and j-th ensemble member. The weight wy for this member is:
Σ, exp(l..t) k. exp(lix))
The weighted mutual information then computed using these weights in place of the uniform weights used in standard mutual information calculations.
21

---

Published in Transactions on Machine Learning Research (05/2025)
C.2 Calibration Error
We compute the calibration error as the absolute difference between the mean confidence and mean accuracy. This is calculated by first computing the softmax of the logits to get probabilities, then taking the maximum probability as the confidence for each prediction. We then compare the predictions to the true labels to determine accuracy. The calibration error is the absolute difference between the mean confidence and mean accuracy across all samples.
C.3 Metric Computation by Uncertainty Score
To analyze how different metries vary with uncertainty, we compute metrics for different quantiles of an uncertainty score. This process involves:
1. Sorting the imputs based on the provided uncertainty scores.
2. Dividing the sorted inputs into quantiles.
3. Computing the specified metric for each quantile ("Bucket Average") or up to the given quantile (*Accep- tance Threshold").
This approach allows us to observe how metrics like accuracy, negative log-likelihood, or calibration error change as function of the model's uncertainty.
C.4 Other Evaluation Metrics
In addition to the above, we used several standard evaluation metrics:
1. Accuracy: The proportion of correct predictions.
2. Negative Log-Likelihood (NLL): The negative log-likelihood of the true labels under the model's predictions.
3. Entropy: The entropy of the model's predictive distribution, used baseline uncertainty measure for single models.
4. Mutual Information: For ensembles, use the mutual information between the predicted class and the ensemble index as measure of epistemic uncertainty.
5. AUROC: The Area Under the Receiver Operating Characteristic curve, used for evaluating out-of- distribution detection performance.
These metrics were computed across different uncertainty quantiles to analyze how model performance and uncertainty estimates correlate. For the AUROC calculations, used the uncertainty scores (entropy for single models, (weighted) mutuua information for ensembles) as the ranking criterion to distinguish between in-distribution and out-of-distribation samples.
D Comparison with Fellaji & Pennerath (2024)
While we initially observed and documented the epistemic uncertainty collapse phenomenon in 2021, Fellaji & Pennerath (2024) independently discovered similar effects Bayesian neural networks, terming it the "epistemic uncertainty hole". Our study offers several key extensions and insights:
Theoretical Framework: We provide theoretical explanation for the epistemic uncertainty collapse through the lens of ensembles of ensembles and propose the hypothesis of implicit ensembling for wider models, offering a mechanistic understanding of why this phenomenon occurs:
Ensembles of Ensembles: Our work introduces the concept of ensembles of ensembles, showing how the epistemic uncertainty collapse manifests hierarchical ensemble structures. This provides a novel toy setting for the phenomenon not explored in the other work.
Implicit Ensemble Extraction: We propose and evaluate a novel technique for mitigating the epistemic uncertainty collapse through implicit ensemble extraction. This practical approach addressing the issue goes beyond the observational nature of Fellaji & Pennerath (2024)'s work.
22

---

Published in Transactions on Machine Learning Research (05/2025)
Broader Model Architectures: While Fellaji & Pennerath (2024) primarily focus on MLPs, we demonstrate that this phenomenon extends to more complex architectures, including state-of-the-art vision models based on ResNets and Vision Transformers.
Thus, while Fellaji & Pennerath (2024) observe an epistemic uncertainty collapse, our work provides a more comprehensive theoretical and empirical investigation of this phenomenon. We not only confirm their findings across a broader range of models and datasets but also offer new insights into the mechanisms behind this effect and potential strategies for mitigation. The proposed implicit ensemble extraction represents an initial step towards addressing the practical challenges posed by the epistemic uncertainty collapse in real-world applications.
E
Additional Results
Table Mean Mutual Information, Accuracy and NLL for Different MLP Widths and Datasets.
Mean MI
Accuracy
NLL
Dataset
MNIST
Dirty-MNIST
Fashion-MNIST
MNIST
Dirty-MNIST
MNIST
Dirty-MNIST
MLP Width
x1
0.0397
0.183
0.228
98
76.1
0.00216
0.477
x2
0.0305
0.176
0.225
98.4
77.5
4.81c-05
0.531
x4
0.0233
0.149
0.192
98.6
77.7
1.53e-06
0.471
x8
0.0182
0.125
0.162
98.5
77.5
1.73e-06
0.415
x32
0.011
0.0838
0.11
98.7
77.2
2.64e-07
0.402
x64
0.0104
0.0882
0.114
98.6
77.1
3.73e-08
0.46
x x128
0.00956
0.0805
0.11
98.5
76.6
7.08c-08
0.351
23

---

Published in Transactions on Machine Learning Research (05/2025)
Table 3: Covariance between the mutual information as uncertainty metric and performance metrics for different ResNet models and extracted ensemble sizes. Higher absolute values indicate stronger relationships. The arrows denote which (neg) covariance is to be preferred.
Neg. Bucket Covariance
Neg. Acceptance Covariance
Performance Metric
Calibration Error
Accuracy
Neg. Log-Likelihood
Model
Uncertainty Metric
Ensemble Size
Entropy
Original
0.046
Mutual Information
2x2
-0.016
0.050
-0.179
3x3
-0.030
0.011
-0.150
4x4
-0.033
0.053
-0.189
-0.035
0.054
-0.192
6x6
-0.037
0.053
-0.185
7x7
-0.031
0.054
-0.203
Wide Resuet101 2-V2 Entropy
Original
-0.030
0.044
-0.141
Mutual Information
2x
-0.019
0.049
-0.174
3x3
-0.031
0.039
-0.139
4x4
-0.038
0.051
-0.187
0.052
-0.188
6x6
-0.039
0.052
-0.191
7x7
.0.36
0.050
-0.194
Resnext101_64XAM
Entropy
Original
-0.018
0.046
-0.147
Mutual Information
-0.011
0.049
-0.162
3x3
-0.019
0.042
-0.145
4x4
-0.022
0.012
-0.156
-0.021
0.013
-0.156
-0.024
0.044
-0.159
7x7
-0.021
0.013
-0.156
24

---

Published in Transactions on Machine Learning Research (05/2025)
Table 4: Covariance between the weighted mutual information as uncertainty metric and performance metrics for different ResNet models and extracted ensemble sizes. Higher absolute values indicate stronger relationships. The arrows denote which (neg) covariance to preferred.
Neg. Bucket Covariance
Neg. Acceptance Covariance
Performance Metric
Calibration Error
Accuracy
Neg. Log-Likelihood
Model
Uncertainty Metric
Ensemble Size
Resnet152-V2
Entropy
Original
-0.030
0.046
-0.144
Weighted MI
2x2
-0.032
0.036
-0.134
3x3
-0.039
0.038
-0.151
4x4
0.037
0.038
-0.14
5x5
-0.042
0.047
-0,178
-0.038
0.044
-0.159
7x7
-0.035
0.020
-0.062
Wide Resnett111 2-V2
Entropy
Original
-0.030
0.044
-0.141
Weighted MI
2x2
-0.039
0.040
-0.138
3x3
-0.099
0.039
-0.148
4x4
-0.044
0.043
-0.159
5x5
-0.047
0.042
-0.163
6x6
-0.043
0.049
-0,180
7x7
-0.037
0.016
-0.073
Resnext101 64X4D
14X44
Entropy Weighted MI
Original
-00118
0.046
-0.147
2x2
-0.043
0.036
-0.131
3x3
00.42
0.045
00.60
4x4
-0.039
0.043
-0.150
-0.042
-0.157
6x6
-0.040
0.049
-0.171
7x7
-0.023
0.009
-0.034
25

---

Published in Transactions on Machine Learning Research (05/2025)
Table 5: Covariance between the mutual information as uncertainty metric and performance metrics for the MaxVit model and extracted ensemble sizes. Higher absolute values indicate stronger relationships. The arrows denote which (neg) covariance to be preferred.
Neg. Bucket Covariance
Neg. Acceptance Covariance
Performance Metric
Calibration Error
Accuracy 1
Neg. Log-Likelihood |
Model
Uncertainty Metric
Ensemble Size
Timm-Maxvit
Entropy
Original
-0.124
0.043
-0.228
Mutual Information
10x10
-0.098
0.032
-0.187
11x11
-0.097
0.031
-0.183
12x12
-0.101
0.033
-0.197
13x13
-0.102
0.034
-0.199
14x14
-0.104
0.035
-0.203
15x15
-0.105
0.036
-0.207
-0.211
2x2
-0.083
0.040
-0.217
3x3
-0.103
0.034
-0,200
4x4
-0.084
0.020
-0.130
5x5
-0.100
0.029
-0.176
6x6
-0.094
0.025
-0.155
7x7
-0.100
0.031
-0.184
8x8
-0.096
0.033
-0.193
9x9
-0.161
26

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6: Covariance between the weighted matual information uncertainty metric and performance metrics for the Max Vil model and extracted ensemble sizes. Higher absolute values indicate stronger relationships The arrows denote which (neg) covariance is to be preferred.
Nog. Bucket Covariance
Neg. Acceplance Covariance
Performance Metric
Calibration Error |
Accuracy f
Neg. Log-Lik Lig-Liielhood
Model
Uncertainty Metric
Ensemble Size
Timm-Maxvit
Entropy
Original
-0.124
0.043
-0.228
Weighted MI
10x10
-0.089
0.041
-0.236
11x11
-0.089
0.041
-0.235
12x12
-0.086
0.041
-0.236
13x13
-0.088
0.041
-0.237
14x14
-0.086
0.041
-0.236
15x15
-0.085
0.041
-0.235
16x16
-0.082
0.043
-0.233
2x2
-0.093
-0.213
3x3
-0.115
0.042
-0.237
4x4
-0.101
0.041
-0.236
5x5
10.
0.041
-0.236
6x6
-0.095
0.040
-0.234
7x7
-0.094
0.040
-0.234
8x8
-0.085
0.041
-0.238
9x9
-0.091
0.041
-0.237
27

---

Published in Transactions on Machine Learning Research (05/2025)
6.923
0.10
5.990
0:08
2x2
Welg fod M1
0.500
0.00
6.023
0.04
0.22
5.775
044
6.5
0.8
0.4
3.0
Caldersi wntarairyrrrrrryyy....o Acceptatio Thruhald
Estmark: Uncera Acciptance Themmed
Epinaree Uncertanty Avorgge
(a) ResNet models with weighted mutual information with optimal temperature
Uncersitt Mack
3+0
22x32
22x13
24636
Fx-teed Diceaiit Aecepan Trrechal
Baiteril Unireriiryy Nerrage
(b) MaxVIT model with weighted mutual information and temperature 1.0
Bireaan Pelk
140
24
32133
24124
35x33
0.00
Eritent: Uncetehly Accidtncc Thrand
Exitend: Uncerany Theareds
Estrertt Buctat
(c) MaxViT model with mutual information and temperature 1.0
6.30
5.24
0.93
0.33
Uscertam Mide
Eractble Sic
Eereay
2.0
5
0.35
6.33
0.5
TRandull
(d) ResNet models with weighted mutual information and temperature 1.0
Figure 7: Complementary Plots of Performance metrics for different ensemble sizes extracted from pre-trained models.
28


Reviews:
[{"review_id": "tqDFxUFqcY", "reviewer": "Reviewer_GeaP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe Empirical results are adequate to convince the significance of the issue of current ensembles based methods.  \n\nWeakness:\n\n1. The derivation from (11) to (12) is incorrect. Since entropy is a concave function of its input, the order of expectation and entropy cannot be swapped in the second term. As the number of ensembles approaches infinity, the estimate will converge to the true mutual information, provided that each ensemble is independently and identically distributed (i.i.d.) sampled. This convergence follows from the Law of Large Numbers rather than the Central Limit Theorem.\n\n2. The left-hand side of equation (8) is not well-defined, or at least, it is unclear why the two quantities are equal. It seems that the authors are attempting to use a plug-in estimator to approximate the quantity on the right-hand side, but this requires further clarification.\n\n3. As a result, regardless of the empirical observations, the claimed explanation for epistemic uncertainty collapse does not appear to be valid. The phenomenon may stem from other underlying factors that have not been accounted for in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates epistemic uncertainty collapse, a phenomenon where larger deep learning models underestimate uncertainty due to implicit ensemble within their architectures. The authors provide a “theoretical explanation” for this effect, which I believe is incorrect. Through extensive experiments on MLPs, ResNets, and Vision Transformers, they empirically confirm this collapse and propose Implicit Ensemble Extraction, a method that decomposes large models into diverse sub-models to recover epistemic uncertainty. Their findings highlight the limitations of uncertainty estimation in large models and offer a method to restoring uncertainty estimates without retraining, with implications for safer and more reliable AI systems.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To my understanding, the observed epistemic uncertainty collapse may be attributed to the following reasons: 1) The ensembles obtained from multiple training processes might not be truly i.i.d., leading to biased plug-in estimators that fail to converge properly. 2)The plug-in estimator itself may introduce substantial bias, which could be mitigated through improved estimation techniques. In addition, there is a long-standing body of literature on mutual information estimation, though the paper does not sufficiently engage with these established methods. It might be an interesting path to explore.\n\nSome other recent literature on understanding the issue of epistemic uncertainty\n\nBengs, Viktor, Eyke Hüllermeier, and Willem Waegeman. \"Pitfalls of epistemic uncertainty quantification through loss minimisation.\" Advances in Neural Information Processing Systems 35 (2022): 29205-29216.\n\nBengs, Viktor, Eyke Hüllermeier, and Willem Waegeman. \"On second-order scoring rules for epistemic uncertainty quantification.\" In International Conference on Machine Learning, pp. 2078-2091. PMLR, 2023.\n\nShen, Maohao, Jongha Jon Ryu, Soumya Ghosh, Yuheng Bu, Prasanna Sattigeri, Subhro Das, and Gregory W. Wornell. \"Are Uncertainty Quantification Capabilities of Evidential Deep Learning a Mirage?.\" In The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BVSMlKHn2J", "reviewer": "Reviewer_2swD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n**S1**: The paper investigates an important and under-explored topic of epistemic uncertainty in very large neural networks. This brings attention to an important issue with practical implications.\n\n**S2**: The paper is clearly written and well organized, containing numerous helpful figures and diagrams. It introduces many concepts well which makes it accessible to a broad audience.\n\n**S3**: Experiments use a variety of datasets, model architectures to verify their claims and provides an analysis to explain the observed phenomenon.\n\n**Weakness**\n\nMy main concern with this paper is the loose connection between \"large models\" and \"ensembles of ensembles\", as argued on page 5.\nWhile the paper's interpretation of successive layers in large models as \"ensembles of ensembles\" is interesting, it doesn't fit the formal definition of an ensemble. Traditionally, each member of the ensemble processes the same input. In contrast, successive layers in a neural network operate on the transformed outputs of previous layers, not a shared raw input. This definition is also shared by the author as per **Equation 7** in Section 3.\n\nThis subtle difference weakens the theoretical connection between the paper's explicit ensembles of ensembles experiments and the hypothesized implicit ensembling within large neural networks. \nSpecifically, the paper partitions an ensemble of $K\\times M$ models into $K$ ensembles of $M$ models and observes a decrease in epistemic uncertainty.\nThis experimental setup doesn't align with the implicit ensembling within a single large model because each member layer does not share the same input. To convincingly support their claim, the authors should ideally show uncertainty collapse across a growing pool of $N$ models that take the same input, which would be more consistent with the implicit ensembles in large models. \n\nAnother way to address this would be to explain why hidden transformations of the input are consistent with Equations 7-13, which currently assumes the same raw input to every model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper challenges the common convention that larger neural networks are better at estimating epistemic uncertainty. The author's core assumption is that large models are simply \"implicit ensemble of ensembles\". Based on this key assumption, the authors demonstrate theoretically and empirically that larger ensembles of ensembles are worse at estimating epistemic uncertainty.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As discussed in the weakness section, either show:\n1. Toy experiment comparing epistemic uncertainty using a growing number of $N$ ensembles that take the same input which would be more consistent with implicit ensembles in large models.\n2. Theoretically demonstrate how hidden transformations of the input work with Equations 7-13, instead of only raw inputs.\n\nMinor changes:\n1. The font of $\\mathcal{M}$ is inconsistent in places, for example in Equation 6 ($M$) and 7 ($\\mathcal{M}$).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dVAXefydZj", "reviewer": "Reviewer_VnwL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The paper is well-written.\n2. The figures look good.\n\nCons:\n\n1. The author(s) should avoid using “We initially observed this behavior in 2021” in footnote 1, as the followed link could potentially give the reviewer the information of the authors and thus violate the double-blind review requirement.\n2. The observation of epistemic uncertainty collapse and the implicit ensembling of the neural network models is very likely to be the consequence of neural collapse observed by [1] and explored widely in the follow-up works ([1] has 500+ citations reported by Google Scholar). The author(s) should provide clear evidence that these two phenomena are different to justify the contribution of the paper.\n3. The mathematical definition of epistemic uncertainty is not unique, the paper used the definition in [3] while [2] has a different definition. I would recommend that the author(s) provide a more detailed discussion of the existing mathematical definitions of epistemic uncertainty to avoid confusion.\n4. The approximation in Eq. 5 is not convincing. Firstly, [4] does not provide formal mathematical proofs to justify the approximation in Eq. 5. Secondly, in the deep ensemble approach, the ensemble members are trained individually over the train set with different initializations. Therefore the $p(\\theta_m|D)$ used after Eq. 4 has a very different meaning from the Monte-Carlo dropout approach or sampling $\\theta$ from the Bayesian prior. What is the practical meaning of $p(\\theta_m|D)$ in the ensemble approach? Why is valid to use $p(\\theta_m|D)$ when computing the mutual information? These questions require detailed clarification to justify the theoretical contribution of the paper.\n5. Eq.9 holds if the approximation in Eq. 5 is correct and Eq. 10,11,12,13 looks a bit trivial.  It would be better if this theoretical evidence could be further explored.\n\nReference:\n\n[1] “Prevalence of neural collapse during the terminal phase of deep learning training” Papyan et al., 2020\n\n[2] “What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? ” Alex Kendall and Yarin Gal, 2017\n\n[3] “Understanding Measures of Uncertainty for Adversarial Example Detection” Lewis Smith and Yarin Ga, 2018\n\n[4] “The power of ensembles for active learning in image classification” Beluch et al., 2018", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors observe the collapse of epistemic uncertainty when using deep ensemble in the regression and classification task. They argue that large neural network models can suffer from implicit ensemble problem and may have a poor estimation of epistemic uncertainty.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Provide clear evidence that the \"epistemic uncertainty collapse\" and \"implicit emsemble\" problem are not consequences of neural collapse observed by [1].\n2. Provide a more detailed discussion on different mathematical evaluations of epistemic uncertainty.\n3. Clarify the meaning of $p(\\theta_m|D)$ used in Eq. 5 and further justify the approximation with mathematical evidences.\n\nP.S. From my experience, the \"epistemic uncertainty collapse\" or \"implicit ensembling\" is very likely to be the consequence of the \"neural collapse\". Therefore, please provide clear evidence to differentiate the two phenomena.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Please See the Original Decision and Commentary Below

UPDATE: I have conducted further consultation with two of the reviewers. The feedback from both was that the paper could be edited to an acceptable form if the authors made a MAJOR change to the claims in the introduction and abstract (and in a couple of sentences discussing results). While this is a major change to the claims, it would still constitute a minor change to the paper. I have therefore adjusted the recommendation to "Accept with minor revision".

The reviewers wrote: "this paper is very well written and has interesting results" and that they "would support the paper if the author modified their claims", and that "if 1) the argument in the manuscript on the connection between “epistemic uncertainty collapse” and “implicit emsembling” is revised to be not so strong ... I am fine for the acceptance of the paper." There were reiterated concerns that "Theoretical analysis on the problem is a bit trivial (and not rigorous)" but there was not an argument that the theoretical analysis was incorrect.

If the claims are modified carefully throughout the paper to the following, then the paper could be accepted: CR1: We introduce implicit ensembling as a possible explanation for the phenomenon of uncertainty collapse in wide/large models. We provide theoretical analysis and experiments that demonstrate uncertainty collapse in ensembles of ensembles and experimental evidence of collapse in wide/large models. CR2: We provide a method for extracting an ensemble from a model and show that this extraction procedure can recover the uncertainty.

The current following claims are problematic because the evidence and analysis in the paper does not support them sufficiently. They should be removed or rewritten: C1: “We provide a possible explanation for this phenomenon through our theoretical analysis and experiments demonstrating implicit ensembling, along with methods to recover epistemic uncertainty by extracting implicit ensembles from large models”. - while most of this claim is supported (it is a “possible” explanation), it is not clear that any experimental results do genuinely demonstrate implicit ensembling. C2: “we provide evidence for both epistemic uncertainty collapse and implicit ensembling” - There is no clear experimental evidence of implicit ensembling. C3: [Experiments]: “These results provide compelling evidence for the epistemic uncertainty collapse due to implicit ensembling in neural networks.” and “This shows that epistemic uncertainty collapse due to implicit ensembling might be a pervasive phenomenon in neural networks, even in relatively simple MLP models.” - The results do not provide clear evidence that the epistemic uncertainty collapse is due to implicit ensembling (or even that implicit ensembling is definitey occurring), so these conclusions are not supported.

In light of this additional feedback from the reviewers, I support acceptance provided that the authors modify their claims as outlined above (to CR1 and CR2, removing/editing C1-C3.) 

I will carefully check the modified paper to ensure that the claims have been suitably adjusted.

_______________________________________
________________________________________
---------------------------------------------------
The reviewers appreciated the authors’ effort in explaining the results and revising the paper and there were comments that the paper was well-written. The reviewers acknowledged that the revisions had addressed initial concerns regarding the theoretical framework. 

However, all three reviewers recommended rejection of the paper, expressing concerns that the connection between wide networks and “ensembles of ensembles” is insufficiently strong to justify the claims and conclusions of the paper. 

The reviewers acknowledged that the experiments demonstrated epistemic uncertainty collapse. The consensus view is that the theoretical framework justifies the claim that ensembles of ensembles lead to collapsed epistemic uncertainty, but all the reviewers considered that this is fairly obvious and the derivation of this results is relatively straightforward.

All reviewers were concerned that the paper’s theoretical development and experimental results do not provide compelling evidence that the underlying reason for the uncertainty collapse phenomenon is the presence of ensembles of ensembles in larger networks. All the reviewers considered that the connection is not sufficiently established, and there would need to be additional theoretical analysis or carefully designed experiments to establish this.  

Although my personal view of the paper is more positive, it is not reasonable to override the recommendations of three reviewers. 

Section 3, the theoretical part of the paper, establishes that ensembles-of-ensembles lead to uncertainty collapse. Section 3.4 provides the hypothesis that “large neural networks could be viewed as implicit ensembles” and that “an ensemble of such large neural networks could thus form an ensemble of implicit ensembles”. There is no theoretical support for this hypothesis, but the introduction is written in such a way that it suggests that the paper does provide sufficient evidence for this hypothesis. We are thus left to consider whether the empirical results in Section 4 are sufficiently compelling to support the claim. 

Section 4 commences by showing experimentally that ensembles of ensembles do lead to uncertainty collapse (Figures 1 and 2). The paper then explores the training of small MLP models of varying width on MNIST. The results demonstrates that as the width of the MLP increases, there is a clear trend of decreasing mutual information. The paper the claims that these results “provide compelling evidence for the epistemic uncertainty collapse due to implicit ensembling”. This is a key component of the paper where the conclusion appears to overreach the experimental evidence. The experiments do not explicitly investigate or establish implicit ensembling. The results show only that as the width increases, the mutual information decreases. This could be due to implicit ensembling but there could be many other explanations. 

Section 4.1 provides results showing that it is possible to mitigate the collapse by decomposing the larger model into constituent submodels. Although these results do provide some support for the implicit ensembling hypothesis, the extraction process does involves steps such as “maximiz[ing] the diversity among the resulting sub-models by regularizing the mutual information”, suggesting that the connection between the hypothesized implicit ensemble in the original model and the extracted ensemble is not so simple. 
Indeed, the paper can only conclude that the results “provide support for our hypothesis” rather than “establish our hypothesis”. This contrasts with the stronger statement in the introduction that “we offer both theoretical justification and empirical evidence for the epistemic uncertainty collapse phenomenon”. Perhaps the reviewers’ concerns with the paper can be addressed by toning down the introductory claims. I would strongly encourage the authors to revise and resubmit the work, since the hypothesis and supporting results are very interesting.
Decision: Accept (poster)