On Joint Regularization and Calibration in Deep Ensembles
Authors: Laurits Fredsgaard
Mikkel N. Schmidt
Venue: Accepted by TMLR ()

Abstract: Deep ensembles are a powerful tool in machine learning, improving both model performance and uncertainty calibration. While ensembles are typically formed by training and tuning models individually, evidence suggests that jointly tuning the ensemble can lead to better performance. This paper investigates the impact of jointly tuning weight decay, temperature scaling, and early stopping on both predictive performance and uncertainty quantification. Additionally, we propose a partially overlapping holdout strategy as a practical compromise between enabling joint evaluation and maximizing the use of data for training. Our results demonstrate that jointly tuning the ensemble generally matches or improves performance, with significant variation in effect size across different tasks and metrics. We highlight the trade-offs between individual and joint optimization in deep ensemble training, with the overlapping holdout strategy offering an attractive practical solution. We believe our findings provide valuable insights and guidance for practitioners looking to optimize deep ensemble models.

Published in Transactions on Machine Learning Research (10/2025)
On Joint Regularization and Calibration in Deep Ensembles
Laurits Fredsgaard Department of Applied Mathematics and Computer Science Technical University of Denmark
laula@dta. dk
Mikkel N. Schmidt
mmsc@dtu.d dk
Deportment Applied Mathematics and Computer Science
Technical University of Denmark
Reviewed on OpenReview: https: // openreview. net/ met/ forum? id= 62qV
Abstract
Deep ensembles are a powerful tool machine learning, improving both model performance and uncertainty calibration. While ensembles are typically formed by training and tuning models individually, evidence suggests that jointly tuning the ensemble can lead better performance. This paper investigates the impact of jointly luning weight decay, temperature scaling, and early stopping on both predictive performance and uncertainty quantification. Additionally, we propose partially overlapping holdout strategy as a practical compromise between enabling joint evaluation and maximizing the use of data for training, Our results demonstrate that jointly tuning the ensemble generally matches or improves performance, with significant variation in effect size across different tasks and metrics. We highlight the tradc-offs between individual and joint optimization in deep ensemble training, with the overlapping holdout strategy offering an attractive practical solution. We believe findings provide valuable insights and guidance for practitioners looking to optimize deep ensemble models. Code is available at:
1 Introduction
Deep ensemblee are a simple and practical method that combines multiple independently trained models to enhance predictive accuracy, improve robustness, and provide uncertainty estimates (Lakshminarayanal
ct al.] 2017). Their offectiveness relics on having diverse members that have uncorrelated errors. which reduces variance and minimizes the impact of individual model mistakes (Hansen Salamon, 1990 Krogh
& &Solich 1997).
While individual models in ensemble may differ in architecture, training sct, and other factors, common practice is to train ensembles using the same model architecture, where the only differences are the initial- izations and the order in which the training examples are presented. This also offers simple and effective method for selecting regularization hyperparameterss such as weight decay and dropout: These settings can be optimized for a single model, typically through grid search, and then used to train the ensemble members independently. Similarly, if post-hoc calibration or early stopping used, often applied to each ensemble member independently.
This approach simplifies the tuning process, but while ensemble of well-regularized and well-calibrated models will generally perform well, it may not be the optimal strategy. This potential mismatch, which we term the ensemble optimality gap, arises because what is optimal for a single model is not necossarily optimal for the final ensemble. The theoretical basis for this is that the expected loss of an ensemble is fundamentally different from the average loss of its members, often involving a beneficial diversity term (Wood et al.) 2023; Krogh & Sollich 1997). Consoquently, using a single model's validation performance as a proxy for the final ensemble's test performance creates a generalization mismatch. This flawed validation process prevents the discovery of hyperparameters that might make beneficial trade-offs, such sacrificing marginal individual

---

Published in Transactions on Machine Learning Research (10/2025)
model performance for larger gain in diversity. While other research focuses explicitly inducing diversity, our work igvestigates simpler premise: closing the optimality gap by ensuring the validation objective correctly mirrors the final deployment objective the ensemble itself.
Previous work has shown that allowing individual models within an ensemble to overfit to a certain extent can lead to improvements both prediction accuracy (Sollich & Krogh 1995) and calibration (Wu & Gales 2021]. In practice, however, this is often disregarded because tuning the complete ensemble by holdout cross-validation can be considerable computational expense or does not scamlessly fit into existing workflows. Hyperparameter tuming (such grid search) requires training an entire ensemble for each parameter combination, scaling the computational cost with the ensemble size. However, methods like early stopping can be validated during parallel training with minimal added cost, whereas post hoc techniques like temperature scaling can be evaluated on the ensemble without additional expense.
In this paper, we systematically explore the ensemble optimality gap across three key aspects of deep ensemble training and calibration: weight decay luning, temperature scaling, and early stopping. Our objective is to assess the magnitude of this effect in common settings and demonstrate how it can be mitigated by optimizing for ensemble performance. In particular, examine how the optimality gap influences model accuracy, uncertainty calibration, and predictive likelihood, as well as investigate its impact on ensemble diversity.
To enhance ensemble diversity, a well-known approach is to train the ensemble using a k-fold cross-validation strategy, where each ensemble member validated on separate holdout data. While this approach can improve the estimation of generalization error for single model, it prevents direct. validation of the full ensemble performance,
a common validation data must be held out for all ensemble members. This leads to a choice between increasing ensemble diversity and having the ability to tune the ensemble a whole. Both strategies have been demonstrated to lead to improved performance. We explore strategy that balances these factors by using partially overlapping holdout sets across ensemble members.
Finally, when training standard deep ensembles is impractical, techniques like batch ensembles or multiple- input multiple-output (MIMO) ensembles offer viable alternatives. In these approaches, the ensemble is formed by sub-models with partially shared parameters within a single, larger model that is trained in run. We demonstrate how our overlapping holdout strategy can be applied in the batch ensemble setting and compare its performance across different initialization strategies.
In summary, our work addresses the following aspects:
We demonstrate several settings in which the ensemble optimality gap is significant, and show to which extent it can be mitigated by jointly tuning the ensemble.
We propose a novel overlapping holdout validation strategy that sits between using a common shared holdout set and using independent holdouts as in k-fold voss-validation.
We present case study based on batch ensembling that demonstrates how an ensemble can be jointly trained and tuned in single run with the overlapping holdout validation strategy.
We validate our study empirically using four diverse benchmark tasks spanning image, graph, tabular, and text classification. Our results demonstrate clear benefits from validating the ensemble jointly, especially for carly stopping and temperature scaling, compared to validating individual models, while joint weight decay luning shows more nuanced effects predominantly related to calibration. We assess the utility of the overlapping holdout strategy in different settings and also provide key insights regarding initialization choices for efficient batch ensembles. Collectively, these findings offer conerete guidance for practitioners on navigating the trado-offs between individual and joint optimization when training and calibrating deep ensembles.
2

---

Published in Transactions on Machine Learning Research (10/2025)
2 Background
2.1 Ensemble Methods: Foundations and Diversity
Ensemble methods improve prediction and robustness by combining multiple models (Dietterich, 2000 The core principle relies on combining outputs from diverse members with uncorrelated errors, thus reducing variance and improving generalization (Hansen & Salamon, 1990; Krogh & Sollich, 1997). In this paper, we focus on deep ensembles (Lakshminarayanan et al.) (017), a simple and effective technique for deep neural networks. For classification lasks, we consider the common approach where the ensemble prediction the arithmetic mean of the softmax probabilities from individual members,
p(yl))
Σ p(ylz,0m)) = M
1
(1)
corresponding to uniform mixture over the ensemble members (see e.g. Tassi (022) for a discussion of the pros and cons of this strategy).
While effective, standard deep ensembles incur substantial costs, training, storing, and rumning M inde- pendent models leads roughly to an M-fold increase in computation time at training and inference. This scalability challenge has spurred the development of more efficient ensemble methods. These approaches often reduce the computational or parameter costs through techniques like parameter sharing (e.g., BatchE ble (Wen et al. 2020b)), ; creating implicit ensembles (e.g.; MTMO (Havasi et al. 2021). Early Exits (Qendro let al.) 2021)), leveraging stochastic inference (e.g., MC Dropout (Gal & Ghahramani 2016)). ) or developing efficient Bayesian approximations (c.g., Rank-1 BNNs (Dusenberry et al. 2020
Implicit and Explicit Diversity. Standard Deep Ensembles typically use identical architectures trained independently. Diversity is achieved implicitly, primarily through different random weight initializations, which serve as the main source of functional diversity, although using distinct stochastic batches during training also contributes to a lesser extent (Fort et al.] 2020) Data resampling techniques such as Bag- ging (Breiman 1996) can promote beneficial diversity for traditional models but are often detrimental for al. 2017). This is largely because deep models are sensitive training data size, and the reduction in data per bagged member significantly weakens the in- dividual predictors. Furthermore, rogularization techniques applied during training (such as weight decay early stopping), while improving individual model generalization, may also implicitly constrain the di- versity among ensemble members. Although this implicit diversity (influenced by initialization, data splits, stochastic batches, and regularization) is often sufficient, explicit diversity-enhaaninng techniques can also lead to improvements, e.g., by modifying training losses or adding regularization (see e.g., Liu & Yao 1999) Pagliardini al. 2023 Jain et al. 2020), but their necessity and benefit, especially for large models, de- bated (Ae et al. 2022 2024). In some cases, joint training methods can lead to learner coltusion (Jeffares et al. 2023), a phenomenon where the ensemble members increase their diversity in way that does not improve generalization.
deep networks (Lee et al. 2015 Lakshminarayanan
Quantifying Diversity. Quantifying the diversity among ensemble members provides key insights into their collective behavior and prediction characteristics. For probabilistic predictive models, a useful informationn-heerrtiic metric for ensemble diversity is the difference between the entropy of the average predictive distribution, Pt, and the average entropy of individual member predictions, p,". This is equivalent to the average KT, divergence (p (P.)
(see Appendix for details):
M Σ M m=
D, = H(p))
H(P.))
(2)
where H denotes the Shannon entropy. In the classification setting with p. defined as the geometric mean, this expression has natural interpretation in the form of bias, variance, diversity decomposition of the
3

---

Published in Transactions on Machine Learning Research (10/2025)
expected loss (Wood et al., 2023]; however, in this work, use the arithmetic mean, as it is more commonly applied. For overview of alternative diversity metrics, see, e.g., Kuncheva & Whitaker (2003) Heidemann et al. 2021p.
2.2 Calibration Deep Learning Models and Ensembles
Beyond predictive accuracy, the reliability of a model's confidence estimates is crucial for dependable decision- making, particularly risk-sensitive applications (Niculescu- Mizil Caruana 2005). A model is considered well-calibrated if its predicted probabilities accurately reflect the true likelihood of correctness (e.g., predic- tions made with 80% confidence correct 80% of the time). While modern deep neural networks achieve high accuracy, they are often found to be poorly calibrated, typically exhibiting overconfidence in their predictions (Gu et al. 2017).
Calibration is commonly evaluated using metrics such as the expected calibration error (ECE), which mea- sures the discrepancy ween confidence and accuracy across prediction bins (Nacini et al. 2015), and the negative logliikel (NLL) of the true classes. NLL is a proper scoring rule, meaning it is uniquely min- imized when predicted probabilities match the true underlying probabilities, thus rewarding both accuracy and calibration (Gneiting & Raltery 2007
Temperature Scaling. scaling [Guo et al. 2017). It involves rescaling the model'ss output logits xxx) by single positive scalar parameter, the temperature T, before applying the softmax function according to the formula
A simple yet effective post-hoc technique for improving calibration is temperature
x(x) T
p(:; T) softmax
A temperature T softens the probability distribution (inereasing entropy, reducing confidence), while T sharpens it. The optimal T is typically found the value that minimizes some calibration metric on held-out validation dataset D Using the NLL as the metric, the optimal temperature is given by
arg min T>0
Σ (D)
logg(
(4)
where T) denotes the predicted probability for the true class yy. Since T only rescales logits before the softmax, does not change individual models' accuracics.
Individual vs. Joint calibration When applying temperature scaling to an ensemble of M models, two main stralegies arise, differing primarily in how the temperabure parameter(s) are optimized and applied. It is important to note that any strategy involving temperature scaling applied before averaging the outputs of the non-lincar softmax function can potentially alter the final classification outcome (i.c., the arg max of the averaged probabilities) compared to averaging unscaled probabilities. The two main implementation strategies are:
Individual Temperature Scaling: separate temperature Tm is optimized for each ensemble member m, typically using its own validation set ((m) The final ensemble prediction is the average of these individually calibrated probability vectors, given by
M
(2m(2r)
P gindividual
(x)
softmax
(5)
M m==1
T
Here, the potential impact on the classification outcome inffuenced by the use of different scaling factors T across members.
4

---

Published in Transactions on Machine Learning Research (10/2025)
Joint Temperature Scaling: A single, shared temperature Tjoin is optimized for the entire en- semble using a suitable joint validation set Di This shared temperature Tjoint is applied to the logits 2m (x) of each member before the softmax activation, and the rosulting calibrated probabilities are then averaged according to the formula.
1
M Σ
Zm (x)
(
M softmax
(6)
T Tjoint
Although the scaling Toone is uniformly applied (preserving the arg max of individual members), the ensemble class prediction can change as averaging happens after the probability distributions are tempered. The optimal Tjom is found by minimizing a calibration metric (such as NLL) of this final averaged prediction ploin(())) on a joint validation set D
A relevant alternative is the Pool- Then- Cali--aaee strategy, where temperature applied after averaging probabilities, which preserves the final prediction (Rahaman & Thiéry 2021). This contrasts with our method of scaling logits before averaging, which allows for diagnostic checks on individual members. Despite these conceptual differences, our results confirm the findings of Rahaman & Thiéry (2021) that both joint scaling methods perform comparably in practice (see Appendix D for comparison).
The relationship between individual member calibration and overall ensemble calibration is non-trivial. Importantly, Wu Gales 2021 showed that ensembling individually calibrated models does not guaranlee a well-calibrated ensemble and can lead under-confidence, advocating instead for calibration strategies that consider the ensemble effect. Their work focused specifically on optimizing temperature scaling parameters by minimizing ECE, whereas optimization based on proper scoring rules like NLL represents an alternative calibration objective commonly used for model training and evaluation.
2.3 Hyperparameter Tuning for Ensembles
Selecting appropriate hyperparameters, such as regularization strengths (e.g., weight decay) or learning pa- rameters, is critical for training performant deep learning models. For ensembles, this presents a fundamental choice regarding the optimization objective: Should hyperparameters be tuned to optimize the performance of individual members, the performance of the ensemble whole?
A common practice, largely due to simplicity and significantly lower computational cost, involves luning hyperparameterss for a single model (e.g., via grid search or random search (Bergstra & Bengio 2012)) and then applying the selected configuration uniformly to all ensemble members during their independent training (Lakshminarayaaan et al. 2017). Directly tuning for the ensemble objective, by contrast, would necessitate training and evaluating the entire ensemble for each candidate hyperparameter setting, incurring a computational cost that typically scales with the ensemble size and often prohibitively expensive.
Beyond searching for single optimal setting to apply uniformly, alternative strategies exist that leverage the models generated during hyperparameter exploration or explicitly use hyperparameter diversity. For instance, Wenzel et al. (2020) proposed hyper-deep ensembles, a method that explicitly combines models resulting from different hyperparameter settings (found via random search and greedy selection) and different random weight initializations. This combination of diversity sources was shown to improve robustness and uncortainty quantification compared to ensembles relying solely on random initialization. Similarly, Jin & Wu (2024) construct ensembles from models saved during learning rate schedule tuning runs, arguing this efficiently reuses computational effort and enhances diversity, reporting strong performance. A limitation of directly using models from tuning runs, however, is that the validation data used for hyperparameter selection is not incorporated into the training data for the final models, differing from standard workflows where models are typically retrained on combined data after tuning.
While these alternative construction mothods exist, the common practice remains to tune single configura- tion for standard deep ensembles. This standard practice implicitly assumes that hyperparameters optimal for single model are also (close to) optimal for the ensemble, which might not hold true in practice,
5

---

Published in Transactions on Machine Learning Research (10/2025)
issue also noted by Gorishniy al. (2025). However, there appears to be limited research directly comparing individual versus ensemble-based iyperparameter tuning for standard deep ensembles.
2.4
Early Stopping Ensembles
Early stopping is another widely used regularization technique that prevents overfitting by monitoring perfor- mance validation set and terminating training when performance on this set ceases to improve (Prechelt Alternatively, the stopping point can be guided by estimators of generalization error that do not re- quire held-out data, such as those derived from bootstrap ensembles (Hansen et al. 1997). When applied to ensembles, the main approaches are:
1998
Individual Early Stopping: Each ensemble member m is trained and stopped independently based on its own performance, monitored validation set Dím) Training for member halts when its validation metric fails to improve for a predefined patience period.
The performance of the entire ensemble (e.g., NLJ of the average predic- tion) is monitored on a joint validation set Diolue. Training for all members stops simultaneously when the ensemble's performance has not improved for the specified patience.
Joint Early Stopping:
While individual early stopping optimizes each member in isolation, early ensemble theory suggests that allowing individual members to overfit slightly may be beneficial for the final ensemble performance (Sollich & Krogh 1995). Joint early stopping might naturally facilitate this by potentially allowing longer Training compared to the point where individual models start overfitting on their respective validation sets. Despite its conceptual appeal, the comparative benefits of individual versus joint stopping strategies for modern deep ensembles appear less explored. The feasibility and specific mechanism for evaluating ensemble performance for joint early stopping depend critically the chosen validation data strategy (Section 3.3). Importantly, unlike potentially costly joint hyperparameter grid scarches, joint carly stopping can often be implemented with minimal computational overhead compared to individual early stopping, especially when members are trained in parallel.
3 Methodology
This section details the methodology used to empirically evaluate the ensemble optimalily gop. We begin by formally defining the problem and then describe the specific experimental settings used across our study.
3.1 Formalizing the Ensemble Optimality Gap
Our investigation centers comparing individual and joint optimization strategies. This comparison be framed general problem of model selection, where the goal is to select the best hypothesis, h, from a set of candidate hypotheses, Heand, by evaluating loss function, L, on a validation set, D A hypothesis h can represent a specific hyperparameter configuration, such as a weight decay value, a temperature for calibration, or a training epoch for carly stopping.
The core of this issue lies in the validation objective. Using a single model's validation performance as a proxy for the final ensemble's performance creates a mismatch: The quantity being optimized (single-model loss) does not align with the ultimate deployment goal (ensemble loss). This fiscrepancy-whie term the ensemble oplimalily gap arises because the loss of an ensemble's prediction is not merely the average of its members' individual losses.
This relationship is formally captured by the following dccomposition.
Ambiguity Decomposition Assuming convex loss, such as squared error or cross-entropy, the ensemble loss can be decomposed into two parts parts-the average individual model loss and an ambiguity term:
6

---

Published in Transactions on Machine Learning Research (10/2025)
C
fm
((f)) -mbiguitty
(7)
Eusemble Loss
Average Loss
The Ambiguity term quantifies the beneficial diversity within the ensemble, and its exact form depends on the loss. For instance, with the squared error loss, the Ambiguity term is the average squared difference of each model's prediction from the ensemble's mean prediction, directly measuring their disagreement. For cross- entropy loss with an arithmetic mean combiner, the term is more complex and becomes dependent the true label, capturing the divergence between the arithmetic and geometric mcans of the model probabilities (Wood et al. 2023). The Ambiguity is always non non-negative which follows from Jensen's inequality.
We have a model fa. with trainable parameters and hyperparameters which define conditions such as weight decay, temperature scaling, and stopping epoch.
Given the hyperparameters, h, optimal model parameters found by minimizing the loss training data, c.g. using stochastic gradient descent.
(h) arg min Cirain (fo,h)
(8)
and the optimal hyperparameters are chosen by minimizing a validation loss,
h arg min Leat (fal(fe(()).)))
(9)
kef
In practice, weight decay typically luned using grid search; temperaturs ure scaling may use either grid search or convex optimization; and early stopping is usually implemented as greedy procedure that halts training when the validation error longer improves, possibly after a patience period.
To make the following specific ensemble validation strategies easier to parse, we will use a convenient short- hand. We denote the fully trained m-th model with hyperparameters as fm) for, (h),
The Optimality Gap The optimality gap emerges from how we choose to define the validation loss when searching for the optimal hyperparameter h". There are two distinct strategies based on the decomposition:
Joint validation directly evaluates the final objective of interest: the performance of the entire ensemble. It finds hans by using the complete ensemble loss for validation, thereby accounting for both the average performance of the models and the ambiguity between them:
h arg min Lval
1 M
Σ Fm(())
(10)
NEW
Individual validation uses a practical but incomplete proxy. finds Wind by using the aver- age individual loss (or a single model's loss) for validation. This common approach simplifies the optimization but is structurally blind to the ambiguity term:
Wind = arg min BER
1
Σ (mm)(((())
(11)
M
Since these two strategics optimize different objectives-one one the complete objective and the other simpli- fied proxy-they generally yield different. lyperparamelers (hina # h:ns). This may lead to a measurable performance difference on the final test set.
We formally define the ensemble optimality gap as the difference in the test loss between the ensemble tuned with the individual-loss proxy and the one tuned with the joint. objective:
M
M
Optimality Gap = Crest
1
Σ
Sm (hina)) Lrest
Σ m=l
fm fm (brnns))
(12)
M
M
m=

---

Published in Transactions on Machine Learning Research (10/2025)
A positive gap demonstrates tangible benefit to the joint strategies. Our experiments are designed to empirically measure the magnitude of this gap under various conditions.
3.2 Datasets and Base Models
To assess the generality of our findings ensemble optimization, conducted experiments across four distinct and commonly used benchmark domains, differing significantly in data modality, task complexity, data scale, and model size:
Image Classification (CIFAR-10 / WRN-16-4): Our first domain uses the CIFAR-10 dataset [Krzhhvsky 2009), widely-used 10-class image classification benchmark with 50,000 train- ing images. We pair this with a Wide ResNet-16-4 (WRN-16-4) model (Zagoruyko & Komodakis, 2016), a common CNN architecture for this task containing approximately 2.7 million parameters. This combination represents a setting where a high-capacity model operates on moderately sized dataset, suggesting a significantly overparameterized regime. This charactoristic makes particu- larly suitable for studying the interplay between ensemble methods and factors like regularization (e.g., weight decay, early stopping) and multi-class calibration.
Graph Classification (NCI1 / GCN); As a contrasting setting with structured data, we used the NCI1 graph classification benchmark (Shervashidze el al. 2011 Wale et al. 2008), binary classi- fication task with significantly fewer data points (4,110 graphs total, 3,288 used for training), with four-layer Graph Convolutional Network (GCN). This pairing allows us to test our hypotheses on different data modality and an architecture with substantially fewer parameters (24,204). De- spite the vastly different scale, regularization remains important for the GCN's generalization on this dataset, allowing us to examine ensemble optimization effects in a different modeling context. Furthermore, the limited data availability may emphasize the impact of data allocation in different validation holdout strategies.
Tabular Classification (Covertype MLP): To cover third modality and explore larger ensemble sizes, we include the Covertype dataset (Blackard & Dean 1999) from the UCI repository. This is large-scale tabular benchmark with approximately 581,012 samples, 54 features, and 7 classes. We use throe-hidden-layer Multilayer Percoptron (MLP) and an ensemble size M = 8 for this lask. This setting allows us to lest. our conclusions a high-data, non-convolutional regime and investigate trends with larger number of ensemble members.
BiLSTM): Our final domain involves text classification using the 2015). which consists of 120,000 (raining samples and 7,600 test We employ a Bidirectional Long Short-Term Memory (BiLSTM) 1997 Schuster & Paliwal 1997/ which processes inputs pre- pared by the GPT-2 tokenizer (Radford al. (2019). This task introduces a sequential data modality and allows us to evaluate our findings on common natural language processing architecture with an ensemble size of M-8.
Text Classification (AG News AG News dataset (Zhang et al.) samples across balanced classes.
network (Hochreiter & Schmidhuber
3.3 Validation Data Strategies for Ensemble Evaluation
How validation data is assigned to ensemble members impacts training and evaluation, particularly when considering joint ensemble objectives versus individual member training needs. We define D' as the available data excluding the final test set, (mm) as the validation set for member m, and (m) as its training set. We
consider three primary strategics for an ensemble of size M:
Shared Holdout: All members use the same split: D(m) train Dirain and D D(m) Dval for all m. val Dval not used by any member during training. This allows direct evaluation of the full ensemble on D
8

---

Published in Transactions on Machine Learning Research (10/2025)
Disjoint Holdout: Each ensemble member uses its own unique validation set, D(m)) C D', of predefined size. These validation sets mutually disjoint, and their combined size cannot exceed the total available data [D'|. Since member m trains on all data except its own validation set (i.c., D(m)) D' ) the data points used validate it are necessarily included in the training data for all other members, maximizing training data utilization across the ensemble. However, this structure prevents direct evaluation of the full ensemble.
Each model gets unique validation set (m) composed of two distinct parts (halves). Each half is shared with one neighboring model in cyclical manner. For instance, half of Model 2's validation data is shared with Model 1, the other half with Model 3. This allows pairwise joint evaluation on the shared halves. (Formally, using M data portions Sş that partition D' then D(m) S. U Sm+1 (mod M) and D(m), D' \ D(m)). While this strategy does not per- mit estimating the validation performance of the entire ensemble on held-out data, it allows joint estimation of model pairs as an approximation.
Overlapping Holdout:
Choosing among these strategies involves balancing the trade-off between maximizing data utilization for training and enabling joint evaluation of the ensemble. The disjoint holdout strategy maximizes data usage but precludes any joint evaluation. Conversely, the shared holdout stralegy enables direct, full-ensemble evaluation but at the cost of reserving portion of data that is never used for training by any model. The overlapping holdout strategy offers practical compromise. While it does not permit evaluating the entire ensemble, it allows for pairwise joint evaluation. Given that the most substantial gains in ensemble performance are often observed when moving from a single model to an ensemble of two, this pairwise evaluation can serve as a proxy for full ensemble performance, making it a potentially useful option in data-scarce settings.
4
Experiments
We conduct a series of experiments to empirically measure the ensemble optimality gap across three key areas: hyperparameter tuning, temperature scaling, and carly stopping.
4.1 Hyperparameter Tuning (Weight Decay)
Purpose This experiment aims to investigate how model performance affected by different strategies for tuming the weight decay hyperparameter. Specifically, we compare two approaches: optimizing weight decay for single model, and optimizing it to maximize the full ensemble's performance. This comparison aims to quantify the ensemble optimality gap and shed light whether tuning for the ensemble yields better results.
Method Optimal weight decay was determined via grid scarch, minimizing validation NLL using a shared holdout set. We compared selecting the best value based on the average individual model NLL versus the NLL of the ensemble's average prediction. Models were trained using stochastic gradient descent (SGD) with momentum and cosine annealing (experimental parameters are detailed in Appendix A.3)

---

Published in Transactions on Machine Learning Research (10/2025)
Classification Error
NLL
ECE
7%
0.30
0.03
WRN-16-4 16-4 (CIFARI0)
6%
0.25
0.02
0.20
5%
0.01
0.15
4%
0.0
0.3
3.2 10.0
0.0 0.3 1.0
30.0
0.0
0.3
1.0 3.2 10.0
Weight Decay (x10-4)
Weight Decay (x10-4)
Weight Decay (x10-
30%
0.7
0.10
0.08
GCN
(NCHL)
25%
0.6
0.06
20%
0.5
0.04
#
//
0.0
1.0
3.2
10.0
0.0 1.0
3.2
10.0
0.0 1.0
3.2
10.0
Weight Decay (x k1100
Weight Decay (x10-1 3)
Weight Decay (x10 3)
0.100
0.0100
(COVERTYPE)
3.6%
0.095
0.0075
MLP
3.4%
0.090
0.0050
3.2%
0.085
0.0025
0.0
0.1
1.0
10.0
0.0 0.1
1.0
10.0
0.0
0.1
1.0
10.0
Weight Decay (x10-5)
Weight Decay (x10-5)
Weight Decay (x10-5)
0.100
12%
1.00
(AG AEGS
BILSTM
0.075
0.75
10%
0.050
0.50
8%
0.025
0.25
#
0.0
0.1
0.3
1.0
3.2
0.0
0.1
0.3
1.0
3.2
0.0 0.1
0.3
1.0
3.2
Weight Decay (x10-1)
Weight Decay (x10-1)
Weight Decay (x10-1)
Performance for the NLL-optimal weight decay
Ensemble size:
Ensemble size: 4
Ensemble size:
Ensemble size: 8
Figure 1: Validation performance across varying weight decay values for WRN-16-4 CIFAR-10, GCN
NCI1, an MLP on Covertype, and BiLSTM on AG News. The plots show results for eusemble sizes 1 to 4 (WRN and GCN) and 1 to 8 (MLP and BiLSTM). The weightl decay for each ensemble size is selected based on the lowest average NLL. (WRN: Wide ResNet; GCN: Graph Convolutional Network; MLP: Multi- Layer Perceptron; BiLSTM: Bidirectional Long Short-Term Memory; NLL: negative log-like;;hoo;; ECE: expected calibration error).
10

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Test performance of the full ensemble, where optimal weight decay (WD*) was tuned by minimizing validation NLL. Values in bold are not statistically different from the best result (within 1.96 SEM). The ensemble size is for WRN-16-4 and GCN, and 8 for MLP and BiLSTM.
Single-Model Opt.
Ensemble Opt.
Model / Dataset
WD*
Metric
Ensemble
Ensemble
Err. (%) +
4.09
+0.07
4.12
+0.04
WRN-16 164 CIFAR10
S: 7.50e-04
E: 5.62e-04
NLL +
0.129
10.000
0.127
10.001
ECE 4
0.009
2:0000
0.006
20.000
GCN
S: 3.16e-03 E: 2.74e-03
Err. (%) +
19.22 10.16
18.47
20.18
NCI1
NLL +
0.443
20.002
0.435
20.002
ECE +
0.034
#0.002
0.034
#0.001
MLP
S: 5.62e-05 E: 1.78e-05
Err. (%) +
3.31
+0.01
3.16
+0,02
COVERTYPE
NLL +
0.084
10.000
0.083
10.000
ECE +
0.001
20.000
0.004
20.000
BILSTM
S: 5.62c-05 E: 5.62e-05
Err. (%) +
7.55
#0.06
7.55
20.08
AG NEWS
NLL +
0.232
+0.002
0.232
+0.002
ECE
0.020
10.001
0.020
10.001
Results The validation sweeps in Figure show that for the WRN, GCN, and MLP models, the optimal weight decay (in terms of NLL) tends to shift lowards less regularization the ensemble size increases. This trend is consistent with the theory that allowing individual members more flexibility can benefit the collective. For the BiLSTM, however, the optimal value remains consistent regardless of ensemble size.
The final test performance, summarized Table reveals a nuanced picture the trade-offs between the tuning strategies. Optimizing for the ensemble's NLL is a principled approach that can yield targeted improvements. For instance, results in the best calibration (ECE) for WRN and improves the classification error for the GCN and MIP compared to the single-model optimum. However, this strategy does not uniformly outperform the others. The final ensemble consistently surpasses the performance of its constituent parts in all settings. This robustness is highlighted by the trend towards less regularization (Figure 1 and is most stark when no weight decay is used at all (Appendix Table 4. We include the average individual model performance in Appendix Table as a diagnostic metric; this provides context on how changes to the ensemble's behavior relate to the performance of its underlying members.
Conclusions Our findings suggest that while tuning weight decay on a single model provides a strong baseline, it can be suboptimal proxy for the final ensemble's performance. Optimizing the full ensemble's validation objective oflers path to improved results, but these benefits are modest and not guaranteed across all metrics.
The gains from ensemble-level tuning, when present, are often specific, improving either classification error (GCN) or calibration (WRN) depending on the context. Practitioners must therefore weigh the potential for a targeted performance boost against the increased computational cost of tuning the entire ensemble. Crucially, the choice of tuning metric shapes the outcome; optimizing for NLL, as we do here, directly improves that metric but does not guarantee the best ECE, exposing clear trade-off between model's predictive accuracy and its calibration.
11

---

Published in Transactions on Machine Learning Research (10/2025)
4.2 Temperature Scaling for Calibration
This experiment investigates how different temperature scaling stralegies impact the calibration and overall performance of deep ensembles. We specifically compare optimizing temperature(s) for individual models versus optimizing a single temperature for joint ensemble prodiction, evaluating these approaches under different validation holdout strategies and varying validation set sizes.
Purpose
Method Base models were first trained using parameters from single-model weight decay tuning. Sub- sequently; to compare individual and joint temperature scaling stralegies across shared and overlapping validation holdouts, we optimized the temperature(s) using T-BFFG to minimize validation NLL (experi- mental parameters are detailed in Appendix A.3
Results The results, presented in Figure reveal clear trade-off related to the validation set size across all four experimental settings: using too little data can lead to unstable temperature estimates, while using too much reduces the available training data and degrades overall performance. This trade-off is modulated by the holdout strategy. The overlapping holdout strategy demonstrates greater robustness to large validation percentages compared the shared holdout, particularly for the GCN and MLP models. This is likely due to its more efficient use of data for training. However, for most configurations, using smaller validation set (e.g., 1-5%) with the shared holdout strategy yields the most competitive results.
Regarding the scaling methods, our results confirm that strategics optimal for individual models are not necessarily optimal for the ensemble. For WRN-16-4 on CIFAR-10, applying lemperature scaling to each member individually degrades the final ensemble's calibration (ECE), even though improves the calibration of the single models (dctailed in Appendix E]. Conversely, the effect of joint scaling on ensemble ECE inconsistent in our experiments. While it improved calibration for the WRN, MLP, and BiLSTM configurations, found it could be detrimental for the GCN model, with the magnitude of this effect also varying significantly. The complete numerical data for these findings are available in Appendix E (Tables 5 through 10).
Conclusions Our findings confirm that joint temperature scaling is preferable to individual scaling. This result is consistent with previous work (Wu & Gales 2021 While joint scaling offers a path to better calibration, particularly on multi-class tasks like WRN CIO,R-TO, the improvements are often modest.
The most critical factor for practitioners is the trade-off in validation set size, as reserving too much data demonstrably harms overall model performance by shrinking the training set. While the overlapping holdout strategy provides robustness against this performance degradation, especially in data-scarce scenarios, the most effective approach in our experiments was typically to use smaller validation set (1-5%) with standard shared holdout.
12

---

Published in Transactions on Machine Learning Research (10/2025)
Classification Error
NLL
ECE
0.15
0.015
WRN-16-5 (CIFARIO)
4.4%
0.14
0.010
4.2%
0.13
4.0%
0.005
.1
2
.5
5
10
20
11
2
5
1
2
5
10
20
2
.5
2
10
20
21%
0.50
0.08
0.48
(NCI))
20%
0.06
GCN
0.46
19%
0.04
0.44
1
2
10
20
50
5
10
20
50
2
5
10
20
50
3.6%
0.006
(COVERTYPE)
3.5%
0.090
0.004
MLP
3.4%
0.002
0.085
3.3%
X
.1
2
5
1
2
5
10
20
1
2
5
1
2
5
10
20
2
.5
2
5
10
20
8.0%
0.06
0.26
BILSTM (AG. (G.NEWS)
7.8%
0.25
0.04
7.6%
0.24
X
0.02
7.4%
0.23
.1
5
5
10
20
11
2
.5
1
2
5
10
20
2
.
2
10
20
Validation Percentage (%)
Validation Percentage (%)
Validation Percentage (%)
No scaling
Shared
Individual scaling
Overlapping
Joint scaling
Temperature scaling test results for the full ensemble (M = 4 for WRN and GCN; M = 8 for MLP and BiLSTM). This plot compares different scaling approaches across varying validation percentages and holdout strategies. (WRN: Wide ResNet; GCN: Graph Convolutional Network; MLP: Multi-Layer Percoptron; BILSTM: Bidirectional Long Short-Tern Memory; NLL: negative log-likelihood; ECE: expected calibration error).
Figure 2:
13

---

Published in Transactions on Machine Learning Research (10/2025)
4.3 Early Stopping
This experiment aims to compare the effectiveness of individual versus joint early stopping strate- gies for deep ensembles. We assess their impact on key metrics including NLL, classification error, ECE, training duration, and ensemble diversity, utilizing different validation holdout methods and varying valida- tion set sizes.
Purpose
Method We compared individual and joint carly stopping based on validation NLL using the Adam optimizer without weight decay and palience of 10 epochs. Shared, disjoint, and overlapping holdout strategies were compared across various validation set percentages. To allow fair comparison of training duration across experiments with different validation set sizes, report stopping times in step-normalized epochs (experimental paramcters are detailed in Appendix A.3
Results The results in Figure show that joint early stopping consistently outperforms individual stopping, leading to lower classification error and NLL for the final ensemble across all holdout strategies. As shown in Figure [ this performance gain attributed to longer training duration. The impact on ensemble calibration (ECE) is also largely positive, with improvements for the WRN, MLP, and BiLSTM models, while the GCN model's ECE is maintained.
This extended training highlights the ensemble optimality gap: while the longer training under joint stopping also improves the classification error of the average individual model, it generally comes at the cost of worse calibration and higher NLL at the individual level (see Appendix[[) F). The effect of longer training on ensemble diversity is inconsistent across our experiments; is lower for the MLP, higher for the BiLSTM, and largely unchanged for the WRN and GCN.
Our experiments also underscore the trade-off in validation set size, where using too much data harms performance by reducing the training set, and using too little can lead to premature stopping. Here, the choice of holdout strategy is important. The disjoint holdout strategy, which prevents joint evaluation, consistently performs the worst. In contrast, for the GCN and MLP models at high validation percentages, the overlapping holdout strategy with joint stopping outperforms the shared holdout. This suggests its superior data utilization and a potential increase in ensemble diversity can mitigate the performance loss from a large validation set.
Conclusions Monitoring the entire ensemble's performance on a validation set to guide early stopping provides clear advantage over stopping members individually. This strategy allows individual members to train beyond their individual optima, benefiting collective ensemble generalization and demonstrating a clear ensemble optimality gap. While this can introduce an accuracy-calibration trade-off for the individual members, the final ensemble's performance is consistently improved.
From a practical standpoint, we recommend joint early stopping with a shared holdout set, as it is not significantly more complex to implement than individual stopping when models are trained on the same device. However, in data-scarce settings that necessitate a large validation percentage, the overlapping holdout strategy presents valuable alternative that is worth exploring. We also find that applying post-hoc temperature scaling provides no significant additive benefit to ensembles already regularized with joint early stopping, as detailed in Appendix
14

---

Published in Transactions on Machine Learning Research (10/2025)
Classification Error
NLL
ECE
0.03
0.26
WRN-16-4 (CIFARIO)
8%
0.24
0.02
7%
0.22
0.01
1
5
5
10
20
.22
5
1
2
5
10
20
2
10 20
26%
0.045
0.52
GCN
(NCI1)
24%
0.040
0.50
22%
X
0.035
0.48
1
2
10
20
50
5
10
20
50
2
5
10
20
50
0.100
MLP (COVERTYPE)
3.6%
0.015
0.095
3.4%
0.090
0.010
3.2%
0.085
0.005
1
2
5
2
5
10
20
12
5
1
2
5
10
20
.1
2
.
2
5
10
20
0.040
8.5%
0.26
0.035
BIL STM (AG. NEWS)
8.0%
0.25
0.030
X X
x
X
7.5%
0.025
0.24
1
5
10
20
11
2
5
1
2
5
10
20
2
5
10 20
Validation Pet (%)
Validation Pet (%)
Validation Pet (%)
Individual stopping
Shared holdout
Joint stopping
Overlapping holdout
Disjoint holdout.
Figure 3: Early stopping test performance for the full ensemble (M = 4 for WRN and GCN; M = 8 for MLP and BiLSTM), comparing different early stopping strategies across all holdout types. (WRN: Wide ResNet; GCN: Graph Convolutional Network; MIP: Multi-Layer Perceptron; BiL.STM: Bidirectional Long Short-Term Memory; NLL: negative log-likelihood; ECE: expected calibration error).
15

---

Published in Transactions on Machine Learning Research (10/2025)
Step-Normalin Epochs
Ensemble Diversity
Entropy
0.11
0.30
WRN-16-4 (CIFARIO)
40
0.10
0.25
-X-
0.09
0.20
-x
11
5
5
10
20
.22
5
1
2
5
10
20
.22
10 20
0.04
20
0.50
GCN
(NCI1)
X
0.03
X
10
X
0.45
0.02
1
2
10
20
50
1
5
10
20
50
2
5
10
20
50
100
0.0225
MLP (COVERTYPE)
0.12
75
0.0200
50
00.115
0.10
1
2
5
2
5
10
20
12
5
1
2
5
10
20
2
5
2
5
10
20
25
0.10
20
0.30
BIL BILSTY STM (AG NEWS)
15
0.08
0.25
10
1
5
10
20
11
2
5
1
2
5
10
20
2
5
10 20
Validation Pet (%)
Validation Pet (%)
Validation Pet (%)
Individual stopping
Shared holdout
Joint stopping
Overlapping holdout
Disjoint holdout.
Figure 4: Additional insights into the early stopping strategies for the full ensemble. We show the stopping epoch (normalized by training steps), alongside the resulting test set ensemble diversity and predictive entropy.(WRe) Wide ResNet; GCN: Graph Convolutional Network; MIP: Multi-Layer Perceptron; BiLSTM: Bidirectional Long Short-Term Memory).
16

---

Published in Transactions on Machine Learning Research (10/2025)
4.4 Validation Strategies and Data Leakage in BatchEnsembles
Purpose While standard deep ensembles consist of independent models, parameter--fflcieet methods like BatchEnsemble (Wen et al. 2020b) utilize extensive parameter sharing (see Appendix This structural difference raises a critical question: Can non-shared holdouts (disjoint or overlapping) be safely used, or does parameter sharing lead to data leakage, where information from one member's validation set influences another? We investigate this potential failure mode and hypothesize that the risk linked to the initialization of the member-specific parameters, which governs the functional diversity of the ensemble.
Method We implemented BatchEnsemble (M 4) on CIFAR-10 using WRN-16-4 architecture. This method employs shared slow weights modulated by member-specific, rank-1 fast weights (details in Ap- pendix A.1.5 We investigated three fast weight initialization strategies (Gaussian a= 0.1, o = 0.5, and random sign +1) across shared, disjoint, and overlapping holdout structures (2% validation split). A key difference from standard ensembles is that BaschEnsemble's shared parameters necessitate simultaneous training termination for all members. For the shared and overlapping holdouts, which permit joint evalua- tion, we used an early stopping criterion based on the joint ensemble's s NLL. However, as the disjoint strategy lacks common validation set, we instead stopped training based on the average of individual NLLs across their respective validation sets. Following best practices (Wen et al. 2020a), separate batch normalization layers were used for each member (experimental parameters are detailed in Appendix A.3
Results Our findings, presented in Table 2 reveal that the fast weight initialization strategy is the critical factor determining both performance and the validity of using non-shared holdouts. Among the BatchEnsem- ble configurations, random sign initialization consistently performs the best, achieving substantially better NLL, ECE, and much higher ensemble diversity than the Gaussian initializations. While it docs not fully match the performance of the standard Deep Ensemble baseline, represents significant improvement over single model.
Figure 5 explains why the Gaussian initializations are particularly unreliable with non-shared holdouts. For these configurations, the validation and training metrics for individual models closely align, while the test metrics diverge significantly. This lack of a typical generalization gap is strong evidence of data leakage, where information from one member's validation set influences others through the shared parameters, leading to poor generalization. In contrast, random sign initialization exhibits a more typical and desirable gap between training, validation, and test performance, indicating that it successfully mitigates this leakage risk.
Conclusions Our investigation confirms that non-shared holdout strategies can be used with BatchEnsem- ble, but this is critically dependent on the initialization promoting sufficient functional diversity. Random sign initialization is strongly preferable, as it mitigates data leakage and allows the model to behave more like a standard deep ensemble. Conversely, Gaussian initializations lead to unreliable performance and poor calibration when used with non-shared validation sets.
This highlights a crucial consideration for practitioners: parameter-efficient ensembles like Batembnsemble can have hidden complexities and may not behave ically to their standard counterparts. The choice of seemingly minor detail, like the initialization of member-specific parameters, can determine the validity of validation strategy. These findings underscore that more work is needed to understand the dynamics of efficient ensembles and to establish robust. best practices that ensure they provide reliable improvements.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: Test set performance of BatchEnsemble (M 4, WRN-16644 on CIFAR-10) compared to a standard Deep Ensemble (DE) bascline. The table compares holdout structures and fast weight initializations for BatchEnsemble. Bold values are the best among the BatchEnsemble configurations (within 1.96 SEM). The DE baseline is not included in the comparison.
Deep Eusemble
N(1.0.1)
(11,0.5)
Random sigu (=1)
Metric
(Shared Holions)
Shared
Overlapping
Disjeiat
Shared
Overlapping
Disjeint
Shared
Overlapping
Disjoint
Err. (%)
6.7
11.0
8.2 uua
8.0 aea
10.2
7.8 loa
7.8 loa
7.9
lna
8.1
8.0 1iu
NLL
0.221
0.440
0.456
0.448
0.363
0.430
0.432
0.258
0.259
0.269
ECE
0.014 iaees
0.058 1ea
0.055
0.051
Loam
0.011
0.050 1a
0.050 tean
0.020
Jam
0.022
0.023
Diversity
0.066
0.007
0.012
0.011
0.026
0.017
0.017
0.106
0.106
0.104
Eatrogy
0.182 Lance
0.137
0.064
0.002
0.100
0.067 1en0s
0.067
0,210
0.219
0.252
Norm. Epochs
44.50
12.45
03.12
52.45
14:21
08.02
59.19
39.00
37.04
27.34
Classification Error
Test
Validation
Train
10%
0%
0.5
NLL
0.0
ECE
0.05
0.00
Shared
Overlapping
Disjoint
Shared
Overlapping
Disjoint
Shared
Overlapping
Disjoint
Initialization method for fast weights
N(L,o= 0.1)
N(1,0=0.5)
Random sign (=1)
Figure 5: Average individual model performance within BatchEnsemble (M 4, WRN-16-4 on CIFAR- 10), comparing different initialization strategies for fast weights across shared, overlapping, and disjoint holdout strategies. The results for classification error, NLL, and ECE are shown for the test, validation, and training sets. Notably, for Gaussian initialization with overlapping and disjoint holdouts, the close alignment of validation and training performance (as opposed to test performance) suggests potential data leakage between ensemble members. (WRN: Wide ResNet; NLL: negative log-likelihood; FCF:: expected calibration error).
18

---

Published in Transactions on Machine Learning Research (10/2025)
5 Discussion and Conclusion
This study investigated the practical impact of adopting an ensemble-aware perspoctive when tuning hy- perparameters and applying calibration or regularization techniques, specifically focusing on the potential mismatch between individually optimal settings- the ensemble optimality gap. Our experiments across weight decay tuning, temperature scaling, early stopping, different validation strategies, and Batchlnsemble provide several insights into optimizing doep ensemble performance.
Our findings regarding weight decay tuning confirm that the optimal value for a single model provides a strong baseline. However, for most of the benchmarks, joint tuning revealed that a slightly lower level of regularization was beneficial for the ensemble, though the specific balance between improving classification error and calibration (ECF) was dependent on the model and dataset. While jointly tuning the entire ensemble for weight decay is computationally expensive, our results suggest that practitioners can consider the single-model optimum as a good starting point and explore slightly lower values.
For post-hoc calibration via temperature scaling, our findings show that a joint optimization strategy is preferable to calibrating members individually. This consistent with prior work (Rahaman & Thiory 2021; Wu & Gales 2021). While applying a single temperature to the joint ensemble generally improved calibration over an uncalibrated baseline, the magnitude of this improvement was often modest and dask-depen.nnt. Critically, these experiments highlight the need for a sufficiently large validation set for robust temperature estimation; however, using significantly more data than necessary yielded little further improvement in calibration while negatively impacting overall model performance due to the reduced training set size.
Perhaps the clearest advantage for joint optimization was observed with early stopping. Monitoring the performance of the entire ensemble and stopping only when its collective performance ceased to improve consistently led to better ensemble NL.L and classification error compared to stopping members individually based on their own optima. This aligus with early ensemble theory (Sollich & Krogh 1995), suggesting that allowing individual members to train longer (potentially slightly past their individual optimal stopping points) benefits the ensemble's generalization capacity. Although such extended training can negatively impact the calibration of individual members considered in isolation, the final ensemble calibration was often improved (e.g., WRN-16-4, MLP, BiLSTM) or maintained (e.g., GCN) compared to using individual stopping.
Conversely, tuning on individual models offers significant practical benefits that explain its widespread use. The approach is simple, requires no changes to standard single-model training pipelines, and allows for modular development where members can be trained and debugged independently. Adopting a joint optimization strategy, in contrast, introduces complexities. It can increase implementation overhead, require more sophisticated resource management to handle multiple models simultancously during validation, and necessitales a carefully chosen validation strategy. Despite these added hurdles, our findings show that the performance gains from joint optimization can be compelling. For computationally inexpensive procedures like carly stopping, in particular, the tradc-off is often favorable, the modest increase in implementation complexity can unlock clear improvements in generalization.
The choice of validation data strategy critically determines the feasibility of joint optimization techniques and involves fundamental trade-offs. A shared holdout enables straightforward joint evaluation of the full ensembles but requires reserving a common validation set that is never used for training by any member. Conversely, the disjoint holdout strategy maximizes the utilization of the available non-test data pool for training across the ensemble-ensuring every data point contributes to training all but one member-but completely precludes joint evaluation on validation data unseen by all members being evaluated. Our results for early stopping and temperature scaling suggest that the benefits derived from enabling robust joint evaluation outweigh the potential advantages of maximizing this training data utilization. The proposed overlapping holdout strategy offers middle ground: similar to disjoint, it ensures all non-test data is used for training somewhere within the ensemble, but by creating specific overlaps within the validation sets assigned to each member, it permits pairwise joint evaluation. This makes it a practical compromise, particularly in low-data regimes where reserving a fully shared holdout might be too costly because it prevents the entire portion of data from being used in training by any model.
19

---

Published in Transactions on Machine Learning Research (10/2025)
The importance of considering ensemble interactions also extends to parameter-effcient methods like Batchensemble. Our investigation underscores the paramount importance of the initialization strategy for the member-specific fast weights (the trainable rank-1 vectors), a detail potentially overlooked. We found that random sign initialization (assigming #1 values) proved crucial for achieving high ensemble diversity and good calibration, behaving much like a standard ensemble. This aligns with some earlier observations regarding its effectiveness (Wenzel et al. 2020). In stark contrast, Gaussian initialization (i.c., sampling from o)), although used in some large-scale applications (Tran et al., 2022 Dehghani et al. 2023), performed poorly in our experiments, showing lower diversity and signs of data leakage with non-shared holdouts. The potential confusion regarding initialization practices is highlighted by reports using ambiguous descriptions like "random sign initialization... of -0.5" (meaning Gaussian initialization) at large scale (Dehghani et al, 2023). Given the stark difference we observed, and the lack of direct comparisons at scale in cited works, further investigation into the impact of initialization on BatchEnsemble's effectiveness seemss warranted. Our findings strongly suggest that proper (random sign) initialization leads to robust. BatchEnsemble behavior, crucially mitigating data leakage issues with non-shared holdouts and thereby enabling the effective and reliable use of different validation strategies, including meaningful joint evaluation when the structure allows (like overlapping holdouts).
Across these diverse experiments, a unifying principle emerges: the significance of the ensemble optimality gap and the practical value of adopting an ensemble-aware perspective. Evaluating and optimizing based on the joint ensemble's behavior during validaaioondeependett procedures consistently led to ensembles that were more accurate, better calibrated, or both. Our results suggest practitioners should prioritize robust joint evaluation strategies-cspeciilly for computationally inexpensive procedures like early stopping and temperature scaling-rather than solely relying individually tuned components or potentially complex methods aimed at explicitly maximizing diversity during training, which may not always be necessary or beneficial, particularly for large models (Abe et al.) [022] 2024). Ultimately, the ensemble is the final predictor, and evaluating it directly during optimization yields better results.
Based on these findings, we offer the following practical recommendations for practitioners training deep ensembles:
Weight Decay: Start by finding the optimal weight decay for a single model, as this provides a strong baseline. To account for the ensemble optimality gap, consider also evaluating the ensemble performance with nearby weight decay values (particularly slightly lower ones), as the optimal setting for the ensemble may differ depending on the model, dataset, and target metric.
Temperature Scaling: Always optimize and apply temperature scaling jointly based on the en- semble's performance (e.g., minimizing NLL) on a validation set. Avoid calibrating members in- dividually. Crucially, use only a reasonably small validation set; dedicating too much data can unnecessarily harm overall model performance.
Early Stopping: Monitor the validation performance of the entire ensemble to determine the stopping point. This allows the ensemble to train longer and achieve better performance compared lo stopping based on individual member optima. Ensure the validation set size is appropriate.
Validation Strategy: Use a shared holdout set whenever possible to allow for direct joint evalu- ation. If data is extremely limited, making shared holdout prohibitively costly, the overlapping holdout strategy offers a viable alternative that retains some joint evaluation capability while max- imizing data use.
Bnsembbe Strongly prefer random sign inilialization for the fast weights over Gaussian initialization to maximize ensemble diversity, calibration, and robustness, particularly when using disjoint or overlapping holdout strategics.
While our findings show consistent trends across four diverse benchmarks (image, graph, tabular, and text) and multiple ensemble sizes (M 4 and M 8), this study has limitations that open avenues for future work. We did not investigate our hypotheses on very large-scale models and datasets, where the computational
20

---

Published in Transactions on Machine Learning Research (10/2025)
cost of joint tuning might be prohibitive and ensembling dynamics could differ. Similarly, whether Gaussian initialization for BatchEnsemble could yield sufficient diversity in such large models remains open question, given the lack of direct comparisons in the literature we surveyed. The implications of our findings for other types of ensembles (c.g., BNNs, MC-dropout) also warrant invostigation. Future work could explore these joint oplimization dynamics in other domains, develop more cost-ellective heuristics for joint tuning, investigate variations of the overlapping holdout strategy such as structures enabling higher-order joint evaluation (e.g,, among member triplets) and further study the interplay between model scale and the optimal validation strategy.
In conclusion, this work highlights the practical importance of considering ensemble effects during the tuning and calibration process. By adopting an ensemble-awar perspective and leveraging joint evaluation, partic- ularly for computationally efficient techniques like early stopping and temperature scaling, practitioners can build more accurate and reliable deep ensemble models.
Acknowledgments
The authors acknowledge support from the Novo Nordisk Foundation under grant no NNF220C0076658 (Bayesian neural networks for molecular discovery).
We acknowledge the Danish e-infrastructure Consortium (DeiC) for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium.
References
Taiga Abe, E. Kelly Buchanan, Gooff Plciss, Richard Zemel, and John Patrick Cunningham. Deep Ensembles Work, But Are They Necessary? In Advances in Neural Information Processing Systems (NeurIPS 2022), volume 35, October 2022.
Taiga Abe, Estefany Kelly Buchanan, Geoff Pleiss, and John P. Cunningham. Pathologies of Predictive Diversity in Deep Ensembles. Transactions on Machine Learning Research, 2024.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, (3(10):281-3000 2012.
Jock A. Blackard and Denis J. Dean. Comparative accuracies of artificial neural networks and discrimi- nant analysis in prodicting forest cover types from cartographic variables. Computers and Electronics in Agricullure, 24(3):131-DD 151, December 1999.
Leo Breiman. Bagging predictors. Machine Learning, 24(2):123 140, August 1996.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, An- dreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer, Michael Tschannen, Amurag Arnab, Xiao Wang, Carlos Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evci, Manoj Kumar, Sjoerd Van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Ma- hendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 7480-7512 PMLR, July 2023.
Thomas G. Dietterich. Ensemble Methods in Machine Learning. In Multiple Classifier Systems, volume 1857, pp. 1-15. Springer, Berlin, Heidelberg, 2000.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lak- shminarayanan, and Dustin Tran. Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 2782-2792. PMLR, November 2020.
21

---

Published in Transactions on Machine Learning Research (10/2025)
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep Ensembles: A Loss Landscape Perspective. arXiv preprint or Xiv: 1f::1912.00 2911..027700 2.0777, June 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48, pp. 1050 1059. PMLR, June 2016.
Tilmann Gneiting and Adrian Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477):359 378, March 2007.
Yury Gorishniy, Akim Kotelnikov, and Artem Babenko. TabM: Advancing tabular deep learning with parameter--fficien ensembling. In The Thirteenth International Conference on Learning Representations (ICLR 2025), 2025.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning (ICML 2017), volume 70, pp. 1321-1330. PMLR, 2017.
Lars Kai Hansen and Peter Salamon. Neural Network Ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10):993-1011, October 1990.
Lars Kai Hansen, Jan Larsen, and Torben Fog. Early stop criterion from the bootstrap ensemble. In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 197), volume 4, pp. 3205-3208, Munich, Germany, April 1997. IEEE Computer Society.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snock, Balaji Lakshmi- narayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnet for robust pre- diction. In 9th International Conference on Learning Representations (ICLR 2021), 2021.
Lena Heidemann, Adrian Schwaiger, and Karsten Roscher. Measuring Ensemble Diversity and Its Effects Model Robustness. In Proceedings of the Workshop on Artificial Intelligence Safety 2021 Co-Located with the Thirlieth International Joinl Conference Arlificial Intelligence (IJCAI 2021), volume 2916 of CEUR Workshop Proceedings, Virtual, 2021. -S.og..
Sepp Hochreiter and Jürgen Schmidhuber. Long Short- Mhor--Term Memory. Neural Computation, 988):1735-1780 November 1997.
Siddhartha Jain, Ge Liu, Jonas Mueller, and David Gifford. Maximizing Overall Diversity for Improved Un- certainty Estimates in Deep Ensembles. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 2264-4271. AAAI Press, 2020.
Alan Jeffares, Tennison Liu, Jonathan Crabbé, and Mihaela van der Schaar. Joint Training of Deep Ensembles Fails Due to Learner Collusion. In Aduances in Neural Information Processing Systems (NeurIPS 2023), volume 36, New Orleans, LA, USA, November 2023.
Hongpeng Jin and Yanzhao Wu. Boosting Deep Ensembles with Learning Rate Tuning. orXiv preprint N.Niv:210007755 October 2024.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Technical report, University of Toronto, 2009.
Anders Krogh and Peter Sollich. Statistical mechanics of ensemble learning. Physical Review E, 55(1): 811-825, January 1997.
Ludmila Kuncheva and Christopher J. Whitaker. Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy. Machine Learning, 51(2):181-207, 2003.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell Simple and Scalable Predictive Uncer- tainty Estimation using Deep Ensembles. In Advonces in Neural Information Processing Systems (NeurIPS 2017), volume 30, pp. 6402 6413, Long Beach, CA, USA, December 2017.
22

---

Published in Transactions on Machine Learning Research (10/2025)
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks. arviv preprint 11.06314 November 2015.
Yong Liu and Xin Yao. Ensemble learning via negative correlation. Neural Networks, 12(10):1399-1404, December 1999.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining Well Calibrated Probabilities Using Bayesian Binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2901 2907. AAAI Press, February 2015.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilities with supervised learning. In Proceedings of the 22nd International Conference on Machine Learning (ICML 2005), pp. 625-632, New York, NY, USA, August 2005, Association for Computing Machinery.
Matteo Pagliardini, Martin Jaggi, François Fleuret, and Sai Praneeth Karimireddy. Agree to Disagree: Diversity through Disagreement. for Better Transferability. In The Fleventh International Conference Learning Representations (ICLR 2023), 2023.
Lutz Prechelt. Automatic early stopping using cross validation: Quantifying the criteria. Newral Networks, 11(4):761 767, June 1998.
Lorena Qendro, Alexander Campbell, Pietro Liò, and Cecilia Mascolo. Early Exit Ensembles for Uncertainty Quantification. In Machine Learning for Health (ML411@N 2021), volume 158 of Proceedings of Machine Learning Research, pp. 181-195. PMLR, December 2021.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Πlya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019.
Rahul Rahaman and Alexandre H. Thióry. Uncertainty Quantification and Deep Ensembles. In Advances in Neural Information Processing Systems (NewrIPS 2021), volume 34, pp. 20063 20075, December 2021.
Mike Schuster and Kuldip K. Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673-22 26822261, November 1997.
Nino Shervashidze, Pascal Schweitzer, Erik Jan van Leeuwen, Kurt Mehlhorn, and Karsten M. Borgwardt. Weisfeller-lehman graph kernels. Journal of Machine Learning Research, 12(77):2539- 2569221611, 2011.
Peter Sollich and Anders Krogh, Learning with ensembles: How overfitting can be useful. In Advances in Newrul Information Processing Systems (NeurIPS 1995), volume 8. MIT Press, 1995.
Yongduo Sui, Xiang Wang, Jiancan Wu, Min Lin, Xiangnan He, and Tat-Seng Chua. Causal Attention for Interpretable and Generalizable Graph Classification. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1696 1705, Washington DC USA, August 2022. ACM.
Cedrique Rovile Njieutcheu Tassi, Jakob Gawlikowski, Auliya Unnisa Fitri, and Rudolph Triebel. The impact of averaging logits over probabilities on ensembles of neural networks. In CEUR Workshop Proceedings, volume 3215. CEUR-WS, 2022.
Dustin Tran, Jeremiah Liu, Michael W. Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Marict, Huiyi Hu, Neil Band, Tim G. J. Rudnor, Karan Singhal, Zachary Nado, Joost van Amers- foort, Andreas Kirsch, Rodolphe Jenatton, Nithum Thain, Honglin Yuan, Kelly Buchanan, Kevin Murphy, D. Sculley, Yarin Gal, Zoubin Ghahramani, Jasper Snoek, and Balaji Takshminarayaaaan Plex: Towards Reliability using Pretrained Large Model Extensions. arXie preprint arXiw:2207.07411 July 2022.
Nikil Wale, Ian A. Watson, and George Karypis. Comparison of descriptor spaces for chemical compound retrieval and classification. Knowledge and Information Systems, 14(3):347-31) 375, March 2008
23

---

Published in Transactions on Machine Learning Research (10/2025)
Yeming Wen, Ghassen Jerfel, Rafael Muller, Michael W. Dusenberry, Jasper Snock, Balaji Lakshmi- narayanan, and Dustin Tran. Combining Ensembles and Data Augmentation Can Harm Your Calibration. In Bighth International Conference on Learning Representations (ICLR 2020), October 2020a.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchhnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning. In Eighth International Conference Learning Representations (ICLR 2020). April 2020b.
Florian Wenzel, Jasper Snock, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robust- ness and uncertainty quantification. In Advances in Newral Informalion Processing Systems (NeurIPS 2020), volume 33, pp. 6514-6527, December 2020.
Danny Wood, Tingting Mu, Andrew M. Webb, Henry W. J. Reeve, Mikel Luján, and Gavin Brown. unified thoory diversity in ensemble learning. Journal of Machine Learning Research, 24(1):359, January 2023.
Xixin Wu and Mark Gales, Should Ensemble Members Be Calibrated? arXiv preprint arXiv:2101.05397, January 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In Proceedings the British Machine Vision Conference 2016 (BMVC 2016). BMVA Press, 2016.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Classifica- tion. In Advances in Neural Information Processing Systems (NeurIPS 2015), volume 28, Montreal, QC, Canada, 2015.
24

---

Published in Transactions on Machine Learning Research (10/2025)
A Experimental Setup Details
The code to reproduce the experiments presented in this paper is publicly available at httpp:://github. lll.aarrtttt/sssmmmmme-ppppaaala
All experiments were conducted the LUMI supercomputer, where each ensemble was trained on single Graphics Compute Die (GCD) LUMI-G node, which is equipped with AMD MI250x GPUs.
A.1
Model Details
A.1.1 Wide ResNet-16-4 (WRN-16-4)
For our experiments on the CIFAR-IU 10 dataset, we utilize the WRN-16-4 architecture (Zagoruyko & Ko- modakis 2016) as one of our base models for deep ensembles. WRN architectures are characterized by their wider convolutional layers and reduced depth compared to traditional ResNet. architecturen Specifically, WRN-1664 denotes WRN with 16 convolutional layers and widening factor of 4. In our implementation, we use the WRN variant where the placement of batch normalization, ReLU activation, and convolution layers follows the order: batch normalization ReLU convolution. We do not employ dropout regularization in our models.
A.1.2 Graph Convolutional Network (GCN)
For our experiments the NCT1 dataset, we utilize four-layer GCN, based the architecture presented in in et al. 20222) but extended to four GCN layers. The network consists of an initial feature transformation layer, followed by four GCN layers, and finally two fully connected layers.
We use ReLU activations after the initial feature transformation and each of the four GCN layers. Batch normalization is applied to the imput features before the initial transformation, after each of the four GCN layers, and before each of the two fully-connected layers.
Global sum pooling is applied after the final GCN layer obtain a graph-level representation. This represen- tation is then passed through sequence of two fully-connected layers. The first fully connected layer applics batch normalization, followed by ReLU activation and linear transformation. The second fully-connected layer is the classification layer and produces the final output logits.
A.1.3 Multilayer Perceptron (MLP)
For the Covertype dataset, for each hidden block consists of a linear layer without bias, followed by batch normalization and ReLU activation. The output layer is a linear transformation to the 7 classes of the dataset.
an MLP with three hidden layers, each with 1024 units. The architecture
A.1.4 Bidirectional LSTM (BiLSTM)
For the AG News classification task, we use BiLSTM network. The model's architecture consists of three main components: initial embedding layer that converts tokens into 128-dimensional vectors, a single BiLSTM layer with a hidden size of 256 units for each direction (resulting in a 312-dimensional output vector for each token), and a self-attention mechanism that computes a weighted average over the BiLSTM's output sequence to create a single context vector.
This context vector is then passed to a final linear layer that maps to the 4 output classes, The input text is processed using the pretrained GPT-2 tokenizer.
A.1.5 BatchEnsemble Implementation
BatchEnsemble (Wen et al. 2020b) modifics base network to efficiently train an ensemble using shared weights W (slow weights). For each shared weight W, member-specific weights W, (i= 1..M)) are generated using trainable rank-1 vectors r; and s; (fast weights) W; W (r:s?), where o denotes the Hadamard
25

---

Published in Transactions on Machine Learning Research (10/2025)
(element-wist) product. Crucially, this rank-1 structure allows the forward pass computations for all M en- semble members to be efficiently vectorized into a single operation, enabling parallel execution on hardware accelerators (Wen et al. 2020b). The initialization of these fast weights is known to be important for perfor- mance. As invostigated in Section 4.4 our experiments specifically compare thrce initialization strategies for r, and 8y: initializing elements from Gaussian distribution with mean and standard deviation (either a o= 0.1 or a 0.5), versus initializing clements randomly as +1 (random sign / Rademacher distribution) Consistent with recommendations for achieving good performance with BatchEnsemble (Wen et al. 2020a use separate batch normalization layers for each ensemble member throughout experiments.
A.2 Dataset Details
A.2.1
CIFAR-10
We conduct experiments on the CIFAR-10 dataset (Krizhevsky 2009), a widely used benchmark for image classification. CIFAR-10 consists of 60,000 32x32 color images in classes, with 6,000 images per class. There are 50,000 training images and 10,000 test images, and we use the original test set for final performance evaluation in all experiments.
Following common practice for CIFAR-10, apply the following preprocessing steps: random cropping to size 32 with padding of 4 pixels, random horizontal flipping with a probability of 0.5, and normalization. The normalization constants (mean and standard deviation for each color channel) are calculated based on the actual training split to prevent contamination from the Lest and validation data.
A.2.2 NCI1
We conduct experiments on the NCI1 dataset Shervashidze et al. 2011 Wale et al. 2008 graph clas- sification dataset with two classes. There are 4110 graphs. We randomly split the dataset into a training set (80%) and a test set (20%) using stratified sampling to ensure class balance, resulting in 3288 training graphs and 822 test graphs. The test kept fixed across all experiments. We do not apply any specific preprocessing to the node features graph structure beyond what is inherent in the dataset.
A.2.3 Covertype
We use the Covertype dataset (Blackard & Dean 1999) from the UCI repository, a large-scale tabular bench- mark for predicting forest cover type from cartographic variables. The dataset contains 581,012 samples, which we split into 464,809 for training and 116, 6,3 for testing. It includes 54 features and 7 classes. We perform input normalization on the features, using the mean and standard deviation calculated from each model's respective training data.
A.2.4 AG News
Our text classification experiments use the AG News dataset (Zhang et al. 2015 which consists of 120,000 training samples and 7,600 test samples for classifying news articles into balanced classes. The input text is processed using the pretrained GPT-2 tokenizer (Radford et al. 2019) with a max length corresponding the longest sequence in the dataset.
A.3 Experimental Setup Details
Across all experiments, the batch size was set 128. The ensemble size was M for the WRN-16-4 and GCN models, and M 8 for the MT.P and BiL.STM models.
Hyperparameter Tuning (Weight Decay) For the weight decay tuning experiments, models were trained using SGD with a momentum of 0.9 and a cosine anncaling learning rate schedule. We performed a grid search over a range of log-spaced weight decay values, always including 0. We used 100 epochs (200 for GCN), an initial learning rate of 0.1 (0.3 for MLP),, and 5 random seeds (20 for GCN).
26

---

Published in Transactions on Machine Learning Research (10/2025)
Temperature Scaling For post-hoc temperature scaling,, used base models trained with the optimal single-model weight decay. The temperature parameter was optimized using the L-BFGS algorithm with a maximum of 100 iterations. We used 10 seeds (5 for MLP and BiLSTM) and validation percentage range of 0.1-20% (1-50% for GCN).
Early Stopping The early stopping experiments were conducted using the Adam optimizer with a patience of epochs and no weight docay, using the validation NLL as the stopping criterion. The learning rate 1 x 10-3 (1 x 10-1 for BiLSTM) and experiments were on 10 seeds (50 for GCN). The range of validation percentages 0.1-20% (1-50% for GCN).
Ensemble The Bnsembbe experiment was performed with a WRN-16-4 on CIFAR-10 with an ensemble size of M = 4. We used the Adam optimizer with a learning rate of 1 x 10-3, no weight decay, and an early stopping patience of 10 epochs on a 2% validation split. The experiment was run across 10 seeds. We tested three fast weight initializations: Gaussian (W(p = 1, = 0.1) and N(u== 1, a = 0.5)) and random sign (+1). Consistent with best practices, a separate batch normalization layer was used for each ensemble member.
B Diversity Metric
quantifies the disagreement among The ensemble diversity metric used in this paper (Sections 2.1
4.3 4.4)
ensemble members for a given input data point i. It is defined the difference between the entropy of the average predicted probability vector p. M Em=1 P and the average entropy of the individual member predictions p
M
M
1
D; = H()))
Σ H(P))
M
m=1
where H(p) = - plc] log is the Shannon entropy.
This measure directly related to the Kullback-Leible (KL) divergence. can shown that the diversity defined above is equal the average KI. divergence from the individual member predictions p. to the mean ensemble prediction ps:
1 D, M Σ (kL(p"")p))
where DKL (pll)) = E. pic) logppc/lp//) Therefore, this diversity metric intuitively quantifics how much, on average, the probability distribution predicted by an individual member diverges from the consensus prediction the ensemble. Higher values indicate greater disagreement among ensemble members.
C Supplementary Results for Weight Decay Tuning
with a detailed breakdown of the average test performance for the individual models that constitute the ensembles. Table 3 presents the classification error, NLL, and ECE for these individual models, comparing the optimized weight decay strategies: using the value optimized for a single model's performance (Single-Model Opt.) versus the value optimized for the final ensemble's 's performance (Ensemble Opt.).
This section supplements the weight decay tuning experiments (Section 4.1
For completeness, Table details the performance when no weight decay is applied. The results directly compare an average single model against the full ensemble, highlighting that the ensemble is significantly more robust to the complete lack of regularization, where it consistently outperforms its individual components.
27

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Average test performance of the individual models that constitute the ensembles. Optimal weight docay (WD*) values determined by minimizing validation NLL for either single model or the full ensemble. For each metric, values in bold are not stalistically different from the best result (within 1.96 standard errors of the mean).
Single-Model Opt.
Ensemble Opt.
Model / Dataset
WD*
Metric
Single Model
Single Model
WRN-16-4 CIFAR10
7.50c-04
Err. (%) 4
4.91
20.06
4.88
+0.07
NLL $
0.160
0.157
E: 5.62o-04
ECE
0.012
10.001
0.013
1000001
Err. (%) 4
20.10
19.28
GCN NCII
3.16e-03 E: 2.74e-03
NLL $
0.480 10,000
0.485
1.00000
ECE
0.042 +0.002
0.051
+000002
MLP
S: 5.62e-05 1.78e-05
Err. (%) 4
3.39 20:00
3.27
1:0.01
COVERTYPE
NLL |
0.088
+00000
0.088
+00.000
ECE
0.003
+00.00
0.006
BILSTM
5.62e-05 5.62c-05
Err. (%) 4
9.96
20009
9.96
::0.00
AG NEWS
NLL 1
0.341
1.0.004
0.341
ECE
0.045
1.0.002
0.045
1:0.000
Table 4: Test performance comparison between an average single model and the full ensemble when no weight decay (WD= = 0) is used.
No Weight Decay
Model / Dataset
Metric
Avg. Single Model
Full Ensemble
WRN-16-1 CIFARIO
Err. (%) 4
8.08
$0.05
6.64
NLL +
0.447
0.217 10.001
ECE J
0.058
$0,001
0.018
$0.000
Err. (%) 4
19.02
18.12
GCN NCT1
NLL 1
1.072 +0.017
0.741 +0.00s
BCE +.
0.143
10.000
0.105
1.0.000 000
MLP
Err. (%) 4
3.25
20.00
3.11
2.001
COVERTYPE
NLL. +
0.092
1.0 000
0.082
2.0 000
ECE
0.007
0.004
Err. (%) 4
12.09
7.77
BILSTM
AG NEWS
NLL +
1.085
J:0 009
0.328
ECE 1
0.103
+00000
0.029
+.0.003
28

---

Published in Transactions on Machine Learning Research (10/2025)
D Comparison of Joint Temperature Scaling Methods
To address the alternative joint calibration method proposed by Rahaman & Thiéry 2021 this section provides a detailed comparison of the two primary joint temperature scaling strategies. The experiment was run using a 5% shared holdout set for validation based on a single seed (0), but otherwise follows the setting of the main temperature scaling experiments.
In the terminology of Rahaman & Thiéry (2021), our main approach, "Joint (on model logits), corresponds to their method (C), where models are simultancously aggregated and calibrated. The alternative, "Pool- (method D), involves first aggregating the model outputs and then calibrating the pooled estimate. Using our notation, the Tool-hhen-Calibraee prediction is given by:
phoot-theu calibaaeeeecateeeeeeeeeeoeeeeeee (a) = softmax
log p(æ))
where p(xx)
1 M sollmax(Zm((1))
Σ
(13)
T pool
The results of our comparison are presented in Figure
E Supplementary Results for Temperature Scaling
To supplement the main results in Section 4.2 this section provides view of the temperature scaling performance. While individual scaling can improve the calibration of the individual models (e.g., for WRN- 16-4), as shown in Figure 7 this benefit does not transfer to the final ensemble, whose performance is shown in the main paper Figure 2
Tables through I0 provide the numerical results for both the ensemble and the average of individual models across all metrics.
29

---

Published in Transactions on Machine Learning Research (10/2025)
WRN--6-4
Classification Faroor
4.16
0.15
4.14
0.4
NLL
0.10
ВCE
4.12
0.05
0.2
4.10
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Temperature (T)
GCN
Classification Eiror
19.2
0.15
1.5
19.1
0.10
1.0
0.05
19.0
0.5
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Temperature (T)
MLP
Classifiction Error
0.075
0.3
3.380
0.050
ECB
0.2
3.375
0.025
0.000
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Temperature (T)
BILSTM
Classification Error
7.8
0.10
0.6
ECB
7.7
0.05
0.4
7.6
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Temperature (T)
Temperature Scaling Strategy Joint (on model logits)
Joint (on ensemble log-probs)
Figure A comparison of two joint temperature scaling strategies. The solid line represents scaling model logits before averaging (method C in Rahaman & Thiery (2021)), while the dashed line represents scaling the ensemble prgpaabbasilities after averaging (method D). The vertical lines indicate the temperature that minimizes NLL for each strategy, which are nearly identical. Scaling the model logits can slightly alter the classification error, whereas scaling the log-irobabillties preserves it. Despite this, the overall performance on NLL and ECE is nearly identical for both methods.
30

---

Published in Transactions on Machine Learning Research (10/2025)
Classification Error
NLL
ECE
5.5%
0.18
0.020
WRN-16-4
(CIFARIO))
5.2%
0.17
X
0.015
5.0%
0.16
4.8%
0.010
4.5%
0.15
0.10.2
0.5
5
10
20
0.10.2
0.5
1
2
5
10
20
0.10.2
0.5
2
10
20
22%
0.55
0.10
(NCI))
21%
GCN
0.50
20%
0.05
19%
0.45
2
10
20
50
5
10
20
50
2
5
10
20
50
3.7%
0.006
0.095
(COVERTYPH)
3.6%
MLP
0.004
3.5%
0.090
0.002
3.4%
0.10.2
0.5
2
5
10
20
0.10.2
0.5
1
2
5
10
20
0.10.2
0.5
2
5
10
20
11.0%
0.04
BILSTM (AG. NEWS
10.5%
0.35
0.03
10.0%
0.30
0.02
9.5%
0.10.2
0.5
10 20
0.10.2
0.3
1
2
5
10
20
0.10.2
0.5
10 20
Validation Percentage (%)
Validation Percentage (%)
Validation Percentage (%)
No scaling
Shared holdout
Individual scaling
Overlapping holdout.
Joint Scaling
Figure 7: Test performance for the average of individual models within the ensemble, comparing differ- ent temperature scaling approaches. This figure shows results corresponding to the ensemble performance presented Figure 2 (WRN: Wide ResNet; GCN: Graph Convolutional Network; MLP: Multi-Layer Per- ceptron; BILSTM: Bidirectional Long Short- Term Memory; NLL: negative likelioo;hhoo; ECE; expected calibration error).
31

---

Published in Transactions on Machine Learning Research (10/2025)
Table 5: Ensemble temperature scaling results for Classification Error (%). Bold values are not statistically different from the best in each row (within 1.96 SEM).
Ensemble: Classification Error (%)
No scaling
Individual scaling
Joint scaling
Model
Dataset
Val. %
Shared
Overlapping
Shared
Overlapping
Shared
Overlapping
0.1%
4.09
10.00
4.13
20.02
4.11
10.00
13.74
4.11
20.03
4.15
200.02
0.2%
4.07
20:02
4.06
2004
4.06
20.00
4.09
20.00
4.12
20.04
4.08
20.04
0.5%
4.08
4.11
4.08
4.12
4.08
4.11
WRN-16-4
1%
4.08
+0.00
=0.04
+0.00
4.11
th0.04
4.09
20.02
4.10
10.04
(CIFARIO)
2%
4.13 2:0.00
4.12
20.00
4.14
20.00
4.13
20000
4.13
20.04
4.12
20.02
5%
4.26
#0.02
4.17
-0.04
4.26
+0002
4.17
+0.04
4.26
4.16
+0.04
10%
4.35 20.08
4.28
20.08
4.34
20.00
4.27
$0.00
4.35
20
4.27
20.08
20%
4.65
4.51
4.65
4.51
+0.02
4.66
4.51
1%
19.18
1.0.8
19.10
20.24
31.27
18.11
25.36
26.39
25.22
1:0.11
19.14
-0.23
2%
18.94
18.93
19.05
19.03
18.99
18.91
GCN
5%
19.22
20..5
19.00
20.10
19.21
-0.15
19.01
1:022
19.18
20.14
18.97
-0.17
(NCN)
10%
19.18
20:18
19.03
20:38
19.26
19.25
20.14
19.23
20.17
19.03
20%
18.63
18.89
50.21
18.59
-0.24
18.86
1:0.22
18.61
10.24
18.88
50%
20.77
20.88
18.86
20:10
20.83
$0.04
18.89
10.92
20.82
20:40
18.83
20:16
0.1%
3.30
$0000
3.30
20.03
3.31
$0.01
3.31
$00001
3.30
20.01
3.30
20.03
0.2%
3.31
2:0001
3.30
20.01
3.31
20.01
3.31
200001
3.31
20.01
3.30
20.01
0.5%
3.30
10.01
3.29
20.01
3.30
20.01
3.29
10.01
3.30
20.01
3.29
10000
MLP
1%
3.31
2:0.01
3.31
20.01
3.31
20.01
3.31
$0.00
3.31
20.01
3.31
20.01
(COVERTYPE)
2%
3,33
3.31
3.33
3.31
+0.00
3.33
3.31
5%
3.35
20000
3.33
20.01
3.35 20.01
3.33
20.00
3.36
20.00
3.33
20.00
10%
3,42 +0.01
3.38
-0.02
3.42 +0.01
3.38
+0.01
3.42 =0.03
3.38
+0.03
20%
3.55
3.41
3.55
3.41
3.55
3.41
0.1%
7.55
2:0.00
7.58
20.05
7.56
20000
7.58
2:0.08
7.57
7.58
20.04
0.2%
7.60
1.0000
7.60
20.04
7.61
20.04
7.59
:00.0
7.60
20000
7.58
1:0.0
0.5%
7.52
::0.08
7.58
20.04
7.52
20.05
7.58
20000
7.52
20.00
7.57
20.08
BILSTM
1%
7.55
7.55
7.56
20.04
7.52
10.00
7.55
7.53
NE_N)
2%
7.49
200004
7.53
20000
20.04
7.52
20.00
7.48
20.00
7.52
20.06
5%
7,53
$0.05
7.47
20.05
7.53
1:0.06
7.46
100000
7.54
-0.00
7.47
10.04
10%
7.62
7.54
7.57
7.53
7.62
7.52
20%
7.89
#0.07
7.66
-0.06
+-0.08
7.62
+00.00
-0.00
7.64
10.00
32

---

Published in Transactions on Machine Learning Research (10/2025)
Table 6: Ensemble temperature scaling results for NLL. Bold values not statistically different from the best in each row (within 1.96 SEM).
Ensemble: NLL +
No scaling
Individual scaling
Joint scaling
Model
Dataset
Val. %
Shared
Overlapping
Shared
Overlapping
Shared
Overlapping
0.1%
0.129
t-0 oon
0.129
+0.001
0.133
=0.002
0.351
#0.218
0.544
0.136
#0.000
0.2%
0,130
#00001
0.127
#0.001
0.131
+0.002
0.130
+0.001
1.052
0.129
#0.001
0.5%
0.129
0.129
0.130
+000002
0.131
+0.002
0.129
0.129
WRN-16-4
1%
0.129
1.0.000
0.129
1.0.000
0.131
1.0000 000
0.131
10.001
0.129
100.00
0.129
100.00
(CIFARIO)
25
0.130
1:00000
0.129
2000001
0.132
0.131
10.001
0.129
100.000
0.128
1000001
5%
0.134
+0.000
0.132
0.136
+00.002
0.134
+000002
0.133
0.131
10%
0,136
+0.001
0.134
+oopod
0.138
+00002
0.136
+00.000
0.135
#o.0ox
0.133
+000000
20%
0.145
100000
0.142
2000001
0.147
1.00000 001000
0.145
20.001
0.144
20.001
0.141
1.00001
1%
0.443
Ho.00a
0.440
+00.001
0.505
+00054
0.473
+.0.025
0.480
Hoc.cas
0.452
+00000
25
0.440
2000001
0.440
200-001
0.449
0.448
0.456
20:010
0.444
GCN
5%
0.440
0.439
0.443
0.451
0.443
0.443
(NCI))
10%
0.440
2000000
0.441
200000
0.448
1:0 0004
0.449
2:0 003
0.442
220,000
0.443
2:0.002
20%
0.449
110000
0.434
1.0.003
0.449
#:0.002
0.444
+-0.002
0.443
0000002
0.434
1.0.002
50%
0.559
0.441
0.482
0.454
0.478
0.432
0.1%
0.085
3:0.000
0.085
3:0.000
0.085
1.0.000
0.084
200000
0.085
3:0.000
0.084
1:0.000
0.2%
0,086
0.085
0.085
0.084
0.085
0.084
0.5%
0.085
3:00000
0.085
$00000
0.085
$00,000
0.084
20.000
0.085
$0.000
0.084
20.000
MLP
1%
0.085
3:00000
0.085
$00000
0.085
$000000
0.085
20.000
0.085
3:00000
0.085
2000000
(COVERTYPE)
2%
0,085
#:00000
0.085
$500000
0,085
0.085
0.085
0.084
5%
0.086
1:0.000
0.085
1.0,000
0.086
1.0.000
0.085
20,000
0.086
2.0.000
0.085
1000000
10%
0.088
1.0.000
0.087
1.0.000
0.088
1.0.000
0.086
20.000
0.087
.00.000
0.086
1.0.000
20%
0,092
0.087
$000000
0.092
0.087
$00,000
0.092
0.087
d:0
0.1%
0,235
1.0.001
0.235
1.0.001
0.250
1.0.0000 1.0.004
0.246
1000002
0.235
1.0.001
0.238
1.0 001
0.2%
0.234
t:0.
0.235
0.243
+00.004
0.243
+0.005
0.236
#00.001
0.237
0.5%
0.234
#00001
0.233
0.241
+000002
0.241
+00000
0.233
#000001
0.233
BILSTM
1%
0.235
1000001
0.235
1.0.000
0.239
1.0 002
0.240
100.001
0.233
2000001
0.234
1.0.000
(AG NEWS)
2%
0.234
1000001
0.234
10.000
0.239
0.242
10.001
0.234
1.00001
0.235
1.0.000
5%
0.235
0.235
0.242
0.244
0.234
0.235
10%
0.236
+0.001
0.236
#0 00.0001
0.244
+10.002
0.246
+00,001
0.236
+0.000
0.238
#0 001
20%
0,248
+3000s
0.242
+00.001
0.258
+.0.004
0.256
+00.002
0.247
#0.002
0,246
+00000 001
33

---

Published in Transactions on Machine Learning Research (10/2025)
Table 7: Ensemble temperature scaling results for ECE. Bold values are not statistically different from the best in each row (within 1.96 SEM).
Ensemble: ECE
No scaling
Individual scaling
Joint scaling
Model
Dataset
Val. %
Shared
Overlapping
Shared
Overlapping
Shared
Overlapping
0.1%
0.009
1:0. 200.000
0.009
#:00000
0.012
#0.00s
0.019
+0.000
0.012
#0.002
0.010
#:0 0.00002
0.2%
0.009
4:0.000
0.009
#00000
0.009
+o
0.011
+0.002
0.009
0.008
0.5%
0.009
+00000
0.009
+0.000
0.011
+00.002
0.011
+0.002
0.006
0.007
WRN-16-4
1%
0.009
1000000
0.009
10.000
0.012
1. 001
0.012
20.001
0.006
1.0.000
0.007
2:0 001
(CIFARIO)
256
0.009
1.0000
0.009
10.000
0.012
0.012
10.001
0.006
1,00000
0.007
2:0
5%
0.009
+0.000
0.010
+0u000
0.013
+-0.002
0.013
+00,002
0.005
+0.000
0.007
#0.001
10%
0,008
+00000
0.010
+ou000
0.012
+000000
0.013
+00,000
0.005
+0.000
0.007
+0.000
20%
0.008
4.00000
0.011
1000000
0.013
2.0
0.016
200000
0.006
1:0.000
0.009
2:0
1%
0.034
H00002
0.027
+00.001
0.113
+.0054
0.083
+-0027
0.084
#o.ca6
0.055
+00000
2%
0.033
2000002
0.033
2000009
0.053
0.063
0.049
20.007
0.052
GCN
5%
0.028
0.028
0.048
0.070
0.039
0.052
(NCI))
10%
0.034
200.000
0.034
2000001
0.065
2:0 007
0.068
2:0 005
0.047
20 000
0.054
2:0
20%
0.039
1.00003
0.033
1:0.002
0.064
+10.006
0.072
+-0.003
0.048
+00.00
0.053
+60.00
50%
0,086
0.043
0.057
0.092
0.041
0.052
0.1%
0.003
1:00000
0.003
1:0.000
0.004
1:0.003
0.002
200,000
0.003
3:0.00
0.002
1:0.001
0.2%
0.004
0.003
1.0.000
0.003
0.002
0.003
0.001
0.5%
0.004
200000
0.003
$500000
0.002
20.000
0.002
200000
0.003
a0.c0m
0.002
3000000
MLP
1%
0.003
3000000
0.003
$000000
0.002
$000000
0.002
$0.0000
0.002
20.000
0.002
3:0.000
(COVERTYPE)
2%
0,003
0.003
$50,000
0,002
0.002
0.002
0.001
de0
5%
0.003
1:0.000
0.003
1.0.000
0.002
1.0.000
0.002
1.00000
0.002
200.000
0.001
1.0.000
10%
0.002
1.0.000
0.004
1.0.000
0.002
1.0.000
0.004
1000000
0.002
3.0.000
0.002
1.0. 00.000
20%
0.002
0.004
$00,000
0,002
0.005
0.001
0.003
0.1%
0.019
1.0.001
0.019
1.0.001
0.052
1.0 007
0.047
10.0004
0.028
20.000
0.034
1.0 000
0.2%
0.018
0.019
0,041
0,040
+0.0007
0.021
0.029
0.5%
0.020
1:0 100.000
0.020
0.041
+00004
0.040
+000004
0.022
#0.000
0.026
BILSTM
1%
0.020
1.0.001
0.020
100.001
0.036
1.0 002
0.039
10.000
0.020
1.0.001
0.026
3.0 001
(AG NEWS)
2%
0.020
10.000
0.021
1.0.001
0.038
0.042
10.000
0.021
20.001
0.027
5%
0.021
0.023
0.042
0.043
0.021
0.028
10%
0.022
100 001
0.023
#0.001
0.042
+00.002
0.046
+-0.001
0.022
+00.003
0.030
#0.001
20%
0.028
1-0.00 004
0.027
+00.001
0.051
+10.005
0.056
+-0.002
0.026
+00.002
0.037
+0.001
34

---

Published in Transactions on Machine Learning Research (10/2025)
Table 8: Average individual model temperature scaling results for Classification Error (%). Bold values are not statistically different from the best in cach row (within 1.96 SEM).
Average Individual: Classification Error (%) 1
No scaling
Individual scaling
Joint scaling
Model
Dataset
Val. %
Shared
Overlapping
Shared
Overlapping
Shared
Overlapping
0.1%
4.77
10.01
4.81
20.02
4.77
20.01
14.34
29.02
4.78
20.01
4.81
100.02
0.2%
4.77
20.00
4.72
20.02
4.77
20.02
4.72
20.02
4.78
20:00
4.72
20.02
0.5%
4.79
4.78
4.79
4.78
4.79
4.78
WRN-16-4
1%
4.77
100001
=0.02
4.78
+0.01
4.76
+10.02
4.78
=0.01
4.76 1.002
(CIFARIO)
2%
4.80 2:0001
4.81
20.02
4.80
20.01
4.81
20.02
4.80
20.01
4.81
20.02
5%
4.98
10.02
4.93
-0.03
4.98
+0.02
4.93
+0.00
4.98
+0.02
4.92 +0.03
10%
5.04
$0.00
5.01
20.02
5.04
20.09
5.01
$0.00
5.04
20.02
5.01
20.02
20%
5.37
5.34
-0.02
5.37
5.34
+0.02
5.37
5.34
1%
20.26
10.15
20.04
10.17
31.86
17.86
27.21
$5.00
26.05
1:3.8
20.05
-0.17
2%
20.04
20.05
20.06
20:18
20.07
20.05
20.05
GCN
5%
19.80
20.14
19.87
20010
19.79
=0.14
19.87
20.16
19.79
200.14
19.86
-0.15
(NCN)
10%
19.96
2:0.09
20.03
20:14
19.98
20.08
20.33
19.97
20.000
20.03
20%
19.56
19.70
30.35
19.55
-0.13
19.70
20.16
19.56
19.70
-0.15
50%
21.66
21.19
200.4
21.67
2:0.0
21.20
20.14
21.67
21.19
20:18
0.1%
3.43
20000
3.42
200000
3.43
20.01
3.42
20.00
3.43
20.00
3.42
2:000
0.2%
3.51
3.42
3.51
3.42
$0.00
3.51
3.42
20000
0.5%
3.46
2:0.03
3.42
20.00
3.46
3.42
$00000
3.46
3.42
1:0.00
MLP
1%
3.44
2:0.00
3.46
0M
3.44 $0.00
3.46
20.00
3.44
20.00
3.46
20.00
(COVERTYPE)
2%
3,47
3.45
3.47
3.45
+00000
3.47
3.45
5%
3.49
20000
3.48
20001
3.49 20.01
3.48
200001
20:000
3.48
2:000
10%
3,55
+00001
3.59
-0.03
3.55 +0.01
3.59
+10.0
3.55
+0.0x
3.59
+00.03
20%
3.68
3.67
3.69
3.67
3.69
3.67
0.1%
9.52
9.54
20.08
9.52
20000
9.54
2:0.00
9.52
9.54
200008
0.2%
9.53
1.0.04
9.60
20.00
9.53
20.04
9.60
:00.07
9.53
20.04
9.60
10000
0.5%
9.56
250.08
9.66
20.00
9.56
20000
9.66
1:0.08
9.56
200008
9.66
20.08
BILSTM
1%
9.57
110.00
9.52
9.58
+0.08
9.52
+0..0a
9.57
9.52
NE_N)
2%
9.58
20000
20.04
9.58
20.04
9.62
20.00
9.58
20.00
9.62
20.04
5%
9.67
1.0002
9.64
=0.00
9.67
+0.02
9.64
10.00
9.67
20.02
9.64
10.00
10%
9.73
9.73
9.72
9.73
9.73
9.73
20%
10.48
-0.24
10.24
+0.03
10.47
+o.24
10.24
+.0.0
10.48
to.2s
10.24
-0.05
35

---

Published in Transactions on Machine Learning Research (10/2025)
Table 9: Average individual model lemperature scaling results for NLL. Bold values not statistically different from the best in each row (within 1.96 SEM).
Average Individual: NLL +
No scaling
Individual scaling
Joint scaling
Model
Dataset
Val. %
Shared
Overlapping
Shared
Overlapping
Shared
Overlapping
0.1%
0,153
to ou1
0.155
#0.001
0.185
#0.02s
0.438
#0.215
1.170
mo.ees
0.178
0:0000 +0.000
0.2%
0,155
#0.001
0.152
#0.001
0.162
+0.00s
0.171
+0.015
2.528
0.161
#0.000
0.5%
0,154
0.154
0.154
to #0.001
0.156
+0.002
0.161
0.158
WRN-16-4
1%
0.154
10.000
0.154
1000000
0.154
10.000
0.154
20.000
0.161
20.000
0.155
2:0 001
(CIFARIO)
25
0.155
20.001
0.154
10.000
0.155
10.000
0.154
20.000
0.161
20.000
0.156
5%
0.161
+0.001
0.159
0.161
+000002
0.159
0.167
#0.000
0.161
#0.001
10%
0,162
#0.001
0.161
+0.000
0.162
+00002
0.161
-0.000
0.167
0.162
+0.001
20%
0.173
20.001
0.172
20.001
0.172
20.000
0.172
20.001
0.178
20.001
0.173
1%
0.474
+0.003
0.472
+00.00a
0.523
+ou0ss
0.523
+0.022
0.513
Hoc.cas
0.482
+00.007
25
0.472
2.00002
0.477
1000000
0.480
20.000
0.474
2000002
0.500
200000
0.469
2:0.000
GCN
5%
0.472
0.478
0.465
0.474
0.473
0.468
(NCI))
10%
0.476
200000
0.493
100.00
0.467
20.000
0.479
2:0 008
0.466
220,000
0.476
2:
20%
0,493
010000
0.495
1:0.004
0.468
+.00002
0.470
=0.003
0.470
-10.003
0.467
110.000
50%
0.665
0.634
110.000
0.500
0.491
0.505
0.496
0.1%
0.088
3.0.000
0.088
3:0.000
0.088
1.0.000
0.089
1:0.000
0.089
a:0.00
0.088
0.2%
0.090
0.088
0.090
0.088
0.091
0.088
0.5%
0.089
0000011
0.088
200000
0.089
$00000
0.088
20.000
0.090
20.000
0.088
$000000
MLP
1%
0.088
2:0.000
0.089
d00000
0.088
$000000
0.089
$0.001
0.089
20.00m
0.089
d: 000
(COVERTYPE)
2%
0.089
0.088
$:00000
0,089
$00.002
0.088
0.090
20.000
0.088
de
5%
0.090
1:0.000
0.089
100.000
0.090
1.0.000
0.089
20.000
0.090
20.000
0.089
1.0.000
10%
0.091
1.0.000
0.092
110.000
0.091
1000000
0.092
1.00001
0.091
1.0.000
0.093
1.0 000.001
20%
0,096
0.095
0.096
$00,000
0.095
0.096
0.095
0.1%
0.323
1.00009
0.325
1.0.001
0.289
1.0.002
0.292
2.0,001
0.295
2.0.005
0.290
1.0.000
0.2%
0.324
#0.002
0.323
+50.00
0.292
0.295
0.321
0.302
0.5%
0.324
#0.001
0.325
0.288
0.290
+-0.002
0.309
0.299
BILSTM
1%
0.324
1.00001
0.322
1.0.000
0.288
1.0.00
0.286
20.000
0.324
2:0.009
0.292
1.0 1.0.009
(AG NEWS)
25
0.321
1.0000
0.322
1.0.001
0.288
20.008
0.289
20.001
0.319
20:010
0.294
5%
0.322
0.321
0.289
0.289
0.314
0.296
10%
0.321
#0.001
0.322
+ropoo
0.292
+0.000
0.292
=0.003
0.319
#co.002
0.297
#0.001
20%
0.347
+0.000
0.339
+ro.0ox
0.314
+10.00
0.307
-0.001
0.352
#0.015
0.313
+0.001
36

---

Published in Transactions on Machine Learning Research (10/2025)
Table 10: Average individual model temperature scaling results for ECE. Bold values not statistically different from the best in each row (within 1.96 SEM).
Average Individual: ECE +
No scaling
Individual scaling
Joint scaling
Model
Dataset
Val. %
Shared
Overlapping
Shared
Overlapping
Shared
Overlapping
0.1%
0.011
1:0 oo0
0.011
+0.000
0.015
#0.002
0.026
+0.005
0.023
mo..0s
0.020
#:0 200003
0.2%
0,011
0.010
#00000
0.014
+0.001
0.014
+0.001
0.023
+oo.os
0.015
0.5%
0.011
+0.000
0.011
0.010
to 20.001
0.011
0.016
0.014
WRN-16-4
1%
0.011
1000000
0.010
10.000
0.010
00.001
0.009
20.000
0.017
200001
0.012
2:0 001
(CIFARIO)
25
0.011
1000000
0.011
10.000
0.009
100000
0.009
20.001
0.016
20.001
0.013
5%
0.012
+0.000
0.012
+0u00d
0.009
0000000
0.010
-0.000
0.017
0.014
10%
0.011
+0.000
0.011
+oup00
0.009
+0.000
0.009
-0.000
0.016
+o.com
0.013
+0.000
20%
0.012
1.00000
0.013
1:0.000
0.010
20.000
0.010
200000
0.017
20.000
0.014
1%
0.040
+000001
0.039
+00.001
0.111
+0.052
0.101
+0.023
0.085
Ho..cs
0.061
+00.000
2%
0.039
200001
0.041
100000
0.054
0.063
20.004
0.054
20.009
0.050
GCN
5%
0.039
0.041
0.050
0.065
0.046
0.049
(NCI))
10%
0.043
220 000
0.044
2000001
0.058
1:0 000
0.063
2:0 004
0.046
200.004
0.048
2:0
20%
0,052
H 0.0002
0.052
#10.000
0.056
+.0.003
0.061
+-0.002
0.045
+00.002
0.045
+00.002
50%
0.112
0.102
1:0.001
0.054
0.060
0.045
0.046
0.1%
0.001
3.0.000
0.001
3.0.000
0.004
1.0.003
0.004
1:0.000
0.004
3:0.00
0.004
1:0
0.2%
0.002
0.001
0.003
0.003
0.006
0.002
0.5%
0.001
200000
0.001
20.000
0.003
$0.000
0.002
20.000
0.005
20.001
0.003
200000
MLP
1%
0.001
5:0. 0000
0.001
$0.000
0.002
2:00000
0.002
20.000
0.005
20.001
0.003
8:0
(COVERTYPE)
2%
0,001
0.001
0,002
0.002
$00,000
0.005
0.003
5%
0.001
10.000
0.001
1000000
0.002
1.0.000
0.002
1.00000
0.004
20.001
0.003
2.00000
10%
0.001
1.0.000
0.002
1.0.000
0.002
1.0.000
0.002
1.0.000
0.004
1000000
0.003
3.0 1.0000
20%
0.002
0.002
$50000
0.002
0.002
0.004
0.004
0.1%
0.040
1:00000
0.040
1.0.000
0.025
1.0 001
0.024
1.00002
0.024
20.000
0.019
1.0.000
0.2%
0,040
#00.000
0.040
450001
0.024
0.024
0.036
0.026
0.5%
0.040
#000000
0.040
+00000
0.018
t: 001
0.018
=0.003
0.033
#0.00s
0.026
6600002
BILSTM
1%
0.040
1.00000
0.039
1.0.000
0.018
1.0.001
0.017
20.000
0.039
2,00000
0.023
1.0 001
(AG NEWS)
2%
0.039
1.0.001
0.039
1.0.001
0.018
001
0.016
20.000
0.037
20.000
0.023
5%
0.038
0.038
0.016
0.016
0.035
0.023
10%
0.037
+0.001
0.038
+rupos
0.017
to 000
0.017
-1o -0.000
0.036
#c..cox
0.022
#0.000
20%
0.040
+00000
0.039
+o0.000
0.019
+0.000
0.019
-0.001
0.041
#0.002
0.024
+ 00.000
37

---

Published in Transactions on Machine Learning Research (10/2025)
F Supplementary Results for Early Stopping
This section shows the early stopping performance for the average of the individual models within the ensem- ble. As shown in Figure 8 the extended training duration under joint stopping can lead to a degradation in the performance and calibration of the individual models, even as the final ensemble's performance (shown in Figure 3) improves. This highlights the ensemble optimality gap, where the best strategy for the ensemble is not necessarily the best for its individual components.
G Interaction Between Early Stopping and Temperature Scaling
To assess whether lemperature scaling offers an additive benefit after joint early stopping, we performed a follow-up experiment using a 5% validation split across random seeds. single shared holdout set was used first to determine the stopping point for the ensemble via joint carly stopping, and then the same set used to optimize the temperature for post-hoc calibration. We compare the performance of full ensembles (M 4 for WRN and GCN; M 8 for ML.P and BiLSTM) before and after applying temperature scaling. The results in Table show no significant or consistent improvement, suggesting that for these well-regularized models, further post-hoc calibration has a negligible effect.
Test performance ensembles regularized with Joint Early Stopping on a shared holdout, com- paring results with and without subsequent Joint Temperature Scaling. The experiment was run with a 5% validation set. The results show no significant additive benefit. Bold values indicate the best performance for each metric and model (within 1.96 SEM).
Table 11:
Model
Metric
No Scaling
Joint Scaling
GCN (NCI)
Classification Error (%)
20.71
20.68
=0 .9.
NLL 4
0.4729
20.0044
0.4755
ECE
0.0402
20.0047
0.0395
2:0 000.0029
BILSTM NEWS)
Classification Error (%) 4
7.60
7.59
NLL 4
0.2406
20000000
0.2392
20000010
ECE
0.0257
20000004
0.0273
0006
MLP (COVERTYPE)
Classification Error (%)
3.23
100.01
3.23
10.001
NLL +
0.0849
100,0001
0.08449
2.000002
ECE
0.0046
0005
0.0031
10.0002
WRN-16-4 (CIFAR1O)
Classification Error (%) 4
6.83
10.18
6.79
10.18
NLL 4
0.2233
10.0000
0.2133
_00.00c
ECE 4
0.0149
10..0os
0.0188
1.0.0017
38

---

Published in Transactions on Machine Learning Research (10/2025)
Classification Error
NLL
ECE
0.055
11%
0.400
WRN-16-4 (CIFARIO)
0.050
10%
0.375
0.045
0.350
x
9%
0.040
11
2
5
5
10
20
1
2
5
1
2
5
10
20
.2
55
2
10
20
26%
0.56
GCN
(NCI1)
0.05
24%
0.54
0.04
22%
1
2
10
20
50
1
10
20
50
2
5
10
20
50
0.010
4.5%
0.115
MLP (COVERTYPE)
0.008
0.110
4.0%
0.006
0.105
0.004
1
2
5
2
5
10
20
1
2
5
1
2
5
10
20
2
5
2
5
10
20
0.06
0.45
11.0%
0.05
BIL STM (AG. NEWS)
0.40
0.04
10.5%
0.35
0.03
5
5
10
20
.1
2
.5
1
2
5
10
20
2
.
2
10
20
Validation Pet (%)
Validation Pet (%)
Validation Pet (%)
Individual stopping
Shared holdout
Joint stopping
Overlapping holdout
Disjoint holdout.
Figure 8: Test performance for the average of individual models within the ensemble, comparing different early stopping strategics. This figure shows results corresponding to the ensemble performance presented in Figure 3 (WRN: Wide ResNet; GCN: Graph Convolttional Network; MT.P: Multi-Layer Perceptron; BILSTM: Bidirectional Long Short-Term Memory; NLL: negative logeliio;;hoo; ECE: expected calibration error).
39


Reviews:
[{"review_id": "0cyO9P6g1Q", "reviewer": "Reviewer_YiaQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe paper is well-written and easy to follow. Moreover, it provides an evaluation that, at least for BatchEnsembles, is somewhat insightful. The results are well-presented, and the ablations appear to be well-executed. \n\n# Weaknesses\nI believe the main weakness of this work is that the insights gained aren't particularly convincing, except for the experiments on BatchEnsembles. This is unfortunate as the work is otherwise well executed, and while there are some mild differences to be observed, it raises the question if they are sufficient to give strong arguments for either the joint objectives or the individual ones. I will discuss this in more detail below. Moreover, as also pointed out by the authors, the current study is very limited in scope regarding the model classes (only two models) and datasets (only two datasets). This potentially further limits the conclusions that can be drawn from the experiments.\n\n# Detailed Comments\n- General, it would be helpful to add equation numbers to all equations.\n- The authors state that the paper examines the impact of hyperparameters on the *ensemble optimality gap*. However, it is never clearly stated what this optimality gap is or how it may be formalised.\n- Page 5, typo: \"objective: should\" -> \"objective: Should\"\n- I think it would help to formalise the two approaches (e.g., page 7) that are evaluated against, meaning the performance of an ensemble (equation on page 2) and the average performance of individual members. The latter is clearly a less meaningful metric as it does not correspond to any sensible probabilistic model.\n- I did like the results discussed in 4.1, but found that the benefits obtained through shared weight-decay to be marginal, even though the authors state *strong benefits*, which makes me think that this is overselling of the actual improvements (difference between 0.008 and 0.006 in ECE).\n- Similarly, I found the improvements in 4.2 to be modest at best, as accuracy and NLL are essentially equivalent for the ensemble, and only slight improvements in ECE can be observed.\n- Again, same story for 4.3. Here, I think the authors again oversell the improvements and even the statement that one can find \"joint stopping generally outperforms disjoint holdouts\" is overselling at best. At least to my eye, there is no visible improvement or difference at all.\n- I was slightly confused by the wording on page 12, saying \"overall ensemble's validation performance\". Is the performance showing the validation performance or the test performance in the figures of the paper? I believe it should be the latter.\n- The ablation on BatchEnsembles is somewhat ad hoc and doesn't really connect well to the rest. Unfortunately for the rest, those results seem to be the most interesting ones. I believe this section would need better integration into the rest of the paper.\n- The ablation on BatchEnsembles is somewhat too short, considering it reveals some patterns while the other ablations aren't very insightful. It would be interesting to study the effect of initialisation in similar approaches in more detail.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors discuss hyperparameter optimisation in deep ensembles, with a focus on the resulting negative log likelihood (NLL), expected calibration error (ECE) and accuracy. For this, the authors provide ablations for weight decay, temperature scaling, and early stopping in deep ensembles. Additionally, they examine the initialisation of *fast weights* in BatchEnsembles, which aims to amortise the members' weights by reconstructing them from a shared weight matrix and local fast weights. The results for deep ensembles are generally in favour of joint optimisation strategies, although improvements are very mild, and show that BatchEnsembles are brittle concerning the initialisation strategy. Finally, the authors provide a set of *best practices* for hyperparameter tuning in ensembles.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Section 4.4 needs better integration into the paper, and I would generally suggest overhauling the paper's structure to achieve better integration of the ablation.\n- The results are currently rather limited, and additional ablations (using more models and datasets) would help strengthen the points.\n- As the current hyperparameter ablations show rather mild improvements at best, I believe that it would be necessary to tone down the claims in the submission and explore why those improvements are so mild.", "broader_impact_concerns": "There are no ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tKJUJR78hm", "reviewer": "Reviewer_BWK3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n\n \n\nThe paper offers a thorough and well-organized review of prior work in ensemble tuning, calibration, and validation strategies. \n\nThe empirical results suggest that jointly optimizing ensemble-level hyperparameters can lead to improved calibration and generalization, especially in resource-constrained settings. \n\nWeakness:   \n1. Limited technical novelty \n* The primary techniques used—weight decay, temperature scaling, early stopping, etc—are well-established. The idea of applying them jointly at the ensemble level is sensible but constitutes an incremental rather than conceptual innovation. \n\n* The \"partially overlapping holdout\" has limited gains on data utilization but a notable drawback: it does not allow estimation on the entire ensemble but only pair-wise estimations. There is a simple alternative is to use m+1 fold validation dataset (by holding out 1 fold as D_eval for the full ensemble and using the other m folds for m individual models). This way, we can do full ensemble validation.  \n\n2. Insufficient Experimental Coverage:.  \n* The author only performed experiments on two datasets, CIFAR10 and NCI. Both are very small datasets. This cannot support the claim that \"jointly tuning the ensemble matches or improves performance across all conditions\". Experiments on more and larger datasets are expected. \n* It relies on visual inspection and without statistical testing or quantitative analysis. \n\n3. Method section lacks rigor.  \n* The current method section reads more like an experiment setup section. It lacks algorithmic or theoretical formalism that would clarify the generalizability of the proposed framework. \n\n* Further methodological justification is needed for the design and effectiveness of the partially overlapping holdout.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed an approach to tune the ensemble models for regularization and calibration: Optimizing weight decay, temperature scaling, early stopping on the ensemble jointly instead of individually on each model. The authors tried to evaluated the ensemble optimality gap by comparing the performance with the baseline approach. Experiments show that on CIFAR 10 and NCI, the proposed approach leads to improved NLL, ECE and classification error. In addition, the authors also proposed \"partially overlapping holdout\" where each model uses two parts of validation data that is shared with its two \"neighboring\" models respectively. This aimed at reaching a balance between shared holdout and overlapping holdout.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1.Revise Method section:  \n* Reframe the Methods section to explicitly define the joint optimization strategy, possibly as a formal algorithm or framework. Highlight how it improves upon conventional independent tuning. Add rigorous analysis. Carefully construct algorithms or rigor methodologies.  \n\n* Provide more rigorous justification of \"partially overlapping holdout\", and analyze its impact of invalidating full ensemble evaluation. \n2.  Strengthen experiments: conduct experiments on more and larger datasets (e.g. ImageNet). Add more quantitative analysis.  \n3. The author can also discuss the limitation of jointly optimizing the ensembles. Tuning on individual models has practical benefits.  \n\nSee \"Strengths And Weaknesses\" for more details.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gHmG9ejN6x", "reviewer": "Reviewer_3Apb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-----------\n\n* The paper addresses a practical and important problem in ensemble learning.\n* It is commendable that the authors focus on empirical findings rather than introducing unnecessary theoretical results. The decision to let practical experiments drive the contributions is well-justified.\n* Experiments are well expecuted\n* The proposed overlapping holdout strategy is simple, effective, and motivated by real-world constraints.\n* The study highlights a relatively under-explored issues such as the ensemble optimality gap and initialization sensitivity in BatchEnsemble.\n\nWeaknesses:\n-----------\n* The study is based on only two model–dataset pairs, which limits the generality of the findings. Given the purely empirical nature of the work, a broader set of architectures and datasets would strengthen the conclusions.\n* The ensemble size is fixed at K=4, which is relatively small. In practice, ensembles often use larger K to achieve more stable performance, so it would be valuable to explore whether the reported trends hold for larger ensembles.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies how to improve the performance and calibration of deep ensembles by tuning key procedures such as weight decay, temperature scaling, and early stopping at the ensemble level, rather than treating each model separately before finally ensembling them. It shows that tuning based on ensemble validation performance leads to more accurate and better-calibrated predictions. The authors also propose an overlapping validation strategy that allows for joint evaluation while making efficient use of available data. Their experiments demonstrate that ensemble-aware tuning is effective and practical (particularly in low-data settings). The paper also shows that initialization choices in parameter-efficient methods like BatchEnsemble can significantly affect performance and calibration.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The authors demonstrate that both joint temperature scaling and joint early stopping lead to improved ensemble performance. However, it is not entirely clear whether they recommend applying both techniques simultaneously. For example, training longer through joint early stopping may lead individual models to become slightly overconfident, which is acceptable due to the averaging effect of the ensemble. But if ensemble-level temperature scaling is applied afterward, is the benefit from longer training still necessary? It would be helpful if the authors clarified whether using both methods together provides additional value, and if so, how they recommend combining them in practice.\n\n* The ensemble size is fixed at K=4, which is relatively small. Since larger ensembles are commonly used in practice to improve stability and robustness, it would be helpful if the authors could comment on whether they expect their findings to generalize to larger K. If possible, running at least a limited experiment with a larger ensemble size, or providing justification for the chosen value, would strengthen the paper.\n\n* Investigations of temperature scaling at the ensemble learning is investigated in [1], with several variants and metrics. It would be interesting for the authors to discuss and possibly mention how their findings complements the recommendations described in [1].\n\n* The authors are encouraged to expand the experimental evaluation to include additional model–dataset pairs. Given the empirical focus of the work, broader coverage would help demonstrate the robustness and generality of the reported findings. Even a small number of additional settings could strengthen the paper’s contributions and improve its practical relevance.\n\n[1] Rahaman R, Thiery AH. Uncertainty quantification and deep ensembles. Advances in neural information processing systems. 2021", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)