LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks
Authors: Dominik J. Mühlematter
Michelle Halbheer
Alexander Becker
Dominik Narnhofer
Helge Aasen
Konrad Schindler
Mehmet Ozgur Turkoglu
Venue: Accepted by TMLR ()

Abstract: Numerous real-world decisions rely on machine learning algorithms and require calibrated uncertainty estimates. However, modern methods often yield overconfident, uncalibrated predictions. The dominant approach to quantifying the uncertainty inherent in the model is to train an ensemble of separate predictors and measure their empirical variance. In an explicit implementation, the ensemble has a high computational cost and memory footprint, especially if the base model itself is already large, like modern transformers. This motivates efforts to develop implicit ensemble methods that emulate the ensemble without explicitly instantiating all its members. We introduce LoRA-Ensemble, a parameter-efficient ensembling method for self-attention networks. It is based on Low-Rank Adaptation (LoRA), originally developed for efficient LLM fine-tuning, and extends it into an implicit ensembling scheme, where all ensemble members share the same, pre-trained self-attention network, but have individual low-rank matrices for the attention projections. The resulting method not only outperforms state-of-the-art implicit techniques like BatchEnsemble, but even matches or exceeds the accuracy of an Explicit Ensemble, while at the same time achieving superior calibration.

Published in Transactions on Machine Learning Research (02/2026)
LoRA- Efficient Uncertainty Modelling for Self-Attention Networks
Dominik J. Mühlematter*
dmuchlemaBetha.cc
Photogrammetry and Remote Sensing, ETH Zurich
Michelle Halbheer
hamich@etth..ck
Photogrammetry and Remote Sensing, ETH Zurich
Alexander Becker
alezonder. bww.errgedddedbeaaeee www....h....cee....
Photogrammetry and Remote Sensing, ETH Zurich
Dominik Narnhofer
dnarnhofer@ethe.cck
Photogrammetry and Remote Sensing, ETH Zurich
Helge Aasen
Melge..asen8ogrocoppe.ooo
Earth Observation of Agroecosystems, Agroscope
Konrad Schindler
schindler@ethz..ck
Phologrammetryy and Remote Sensing, ETH Zurich
Mehmet Ozgur Turkoglu" Earth Observation of Agroecosystems, Agroscope
moturkogju@gmaiil.com com
Reviewed OpenReview: https: o/penrvee/peeeveeee. net/ pormf id= 3/XXXXXXPPMP
Abstract
Numerous real-world decisions rely on machine learning algorithms and require calibrated uncertainty estimates. However, modern methods often yield overconfident, uncalibrated predictions. The dominant approach to quantifying the uncortainty inherent the model train an ensemble of separate predictors and measure their empirical variance. In an explicit implementation, the ensemble has high computational cost and memory footprint, especially if the base model itself is already large, like modern transformers. This motivates efforts to develop implicit ensemble methods that emulate the ensemble without explicitly in- stant ing all its members. We introduce LoRA-Ensemble, a parameter-effciint ensembling method for self-attention networks. It is based on Low-Rank Adaptation (LoRA), originally developed for efficient LLM fine-tuning, and extends it into an implicit ensembling scheme, where all ensemble members share the same, pre-trained self-attention network, but have individual low-rank matrices for the attention projections. The resulting method not only outperforms state-of-the-art implicit techniques like BatchEnsemble, but matches or exceeds the accuracy of an Explicit Ensemble, while at the same time achieving superior calibration.
1 Introduction
Machine learning models are increasingly applied also in fields where incorrect estimates can have severe consequences, e.g., autonomous driving, medical diagnosis, (extreme) weather event prediction, or decision support for agriculture. such applications, well-calibrated predictive uncertainties are crucial enable
"Equl contribution.

---

Published in Transactions on Machine Learning Research (02/2026)
self-diagnosis. Uncertainty can be separated into two components. Aleatoric uncertainty, a.k.a. irreducible noise, is inherent in the data. In contrast, epistemic uncerlainty stems from a lack of knowledge about certain regions of the input space, due to a lack of training data (Der Kiureghian & Ditlevsen 2009).
Quantification of epistemic uncertainty in large machine learning models is non-trivial. Analytical compu- tation usually intractable; thus research has focused on efficient approximations (Graves 2011 Blundel]
et al. 2015; WWelinng et al. 2011). To date, probabilistic ensembles remain the best-performing approach [Lakshminarayaaan et al., 2017]. In a naive implementation, such an ensemble consists of multiple inde- pendently trained models. Individual models are interpreted as Monte Carlo samples from the posterior weight space and are used to obtaín an unbiased estimator of the posterior distribution. To achieve a low correlation between ensemble members, one can capitalize on the stochastic nature of the training process and start from different initial weights, and/or sample different random batches of data. The basic principle is that the predictions of different ensemble members will agree near observed training samples, whereas they may vary far away from the training data. Their spread therefore serves as a measure of epistemic uncertainty. Empirically, even small ensembles often capture the uncertainty well (in expectation), i.e., they are well calibrated.
An issue with naive ensembles is that their computational cost and memory footprint grow proportional to the number of ensemble members. For smaller models, the added cost. and energy use may be acceptable. But for modern neural networks with up to several billion parameters, hardware restrictions render the naïve approach intractable, in particular, one can no longer hold the entire ensembll in memory. Consequently, research has focused on ways to create ensembles implicitly, without requiring multiple copies of the full base model (Wen et al. 2020 Wenzel et al. 2020 Huang et al., 2017 Turkoglu et al., 2022). Unfortunately, most of these paraameter-fficient ensembling techniques are not readily applicable to the newest generation of neural networks. Transformer networks (Vaswani et al. 2017) have become popular due to their superior ability to capture complex structures in data, However, implicit ensembling schemes tend to underperform for transformers, as demonstratod in our experiments, or are architecturally incompatible with them. These methods were developed primarily for MLP and CNN archilectures and there is no forwigh way to port them to transformers due to fundamental differences in computation structure. For instance, FiLM- Ensemble (Turkoglu et al. 2022) modulates feature maps through Batch Normalization layers, which are absent in standard transformer architectures that rely on Layer Normalization instead (see Appendix R.7 for details). Similarly, SNGP [Liu et al., 2020 enforces bi-Lipschitz constraints via spectral normaliza- tion to ensure distance-aware uncertainty; however, dot-product self-attention has an unbounded Lipschitz constant (Km et al. )021), violating SNGP's core theorctical assumptions (sec Appendix R.9 for details).
Several studies have shown that modern neural networks are heavily overparametrized and that their results have low intrinsic dimension (Li et al.] 2018& Aghajanyan et al. 2020). This led [Hu et al. (2021) to propose Low-Rank Adaptations (LoRAs) as way of fine-tuning Large Language Models (LLMs) for different tasks while avoiding the prohibitively large memory and compute requirements of retraining them. It turns out that the weight matricos in such models can be factorized to have very low rank, with hardly any loss in prediction performance.
ledHu
We show that LoRA can also serve as a basis for a novel, parameter-efficiett ensemble method tailored to the transformer architecture. In line with the trend towards parametcr-cfficiet fino-tuning, our method uses a pre-trained transformer model, which is expanded into an implicit. ensemble by varying the LoRA factorization, while keeping the backbone weights frozen. In this way, our method requires a small number of additional parameters to turn existing transformer model into diverse ensemble whose performance across various tasks is comparable to an Explicit Ensemble. In summary, our contributions are:
We introduce networks.
a parameter-eflcient probabilistic ensemble method for self-attention
LoRR-Ensemble can be readily combined with a wide range of standard pre-trained transformer architectures that employ the standard self-attention mechanism: it simply replaces the linear pro- jection layers in the attention module with LoRA layers.
2

---

Published in Transactions on Machine Learning Research (02/2026)
We apply LoRA-Ensemble to different classification tasks, including conventional image labeling, skin lesion classification in dermatoscopic images, fine-grained image classification, sound classification, out-of-distribution (OOD) detection, and language modeling; and demonstrate significant gains in accuracy and uncertainty modeling.
We demonstrate that LoRA-Ensemble outperforms traditional Explicit Ensembles by fostering greater diversity among members, both in their learned functions and in weight space.
We conduct extensive empirical analyses of how LoRA rank, initialization scheme, model scale, and parameter-sharing strategies impact performance, and we adapt LoRA-Ensemble for convolutional neural networks (CNNs) to demonstrate its broad applicability.
2
LoRA-Ensemble
The Low-Rank Adaptation (LoRA) technique makes it possible to use a pre-trained model and fine-tune it without having to retrain all its parameters. This is particularly beneficial for modern neural networks with large parameter spaces. The underlying principle is to freese the pre-trained model weights Wo E R and instead constrain the updates to a low-rank decomposition. This can be expressed mathematically as:
W Wo + AW Wo + B.A.
(1)
Here B E gexr and A E xx are two trainable low-rank matrices, where < min(d, W and AW are then multiplied by the same input x, which yields the following modified forward pass:
h = Wo x + AW z = Wo x B.A.x.
(2)
LoRA applics its low-rank adaptation scheme exclusively to the weight matrices of the self-attention modules in transformer, while leaving the interleaved MLP layers untouched. Concretely, the adapted weights are Wg Wx. and We, which project the input into query, key, and value representations, along with W.. which merges the oulputs of the atlention heads. As in Hu et al. (2021), the projection matrices are treated as single units, disregarding their typical partitioning into multiple attention heads. In Appendix G we provide additional ablations on the placement of LoRA layers within the transformer, as well as on ensemble design choices, illustrating how these factors impact predictive performance, calibration, and efficiency.
Although not designed with uncertainty calibration in mind, the LoRA concept fulfills all the requirements of an implicit deep ensemble: By modifying the weights of the highly nonlinear self-attention mechanism one is able to generate a diverse collection of networks with the same architecture and objective. By learning an additive, low-rank update AW = B. A rather than directly tuning the weight matrices, the expansion into a model ensemble adds only a small number of parameters and efficient. In detail, we start from a single, pre-trained model with frozen parameters Wo and expand it with a set of trainable low-rank matrices AW, Vi = N. At each transformer block, there now different forward pass per ensemble member i, illustrated in Fig.
h2 = Wo x + AW, x Wo x A
(3)
leading to N different predictions To, (X) for a given input X. From those individual predictions, we compute the ensemble mean and variance in the standard manner:
||
To,(X) Var[Y|
Σ
1 - N
Σ
(4)
respectively, for implementation, training, and initialization details. We publicly release the py Torch implementation of Fnsemble,,,, along with pre-trained weights, on GitHull
Refer to Appendices K
L and
https://github.. http:stuubb.com/prretttl/ltt/t Ensemble
3

---

Published in Transactions on Machine Learning Research (02/2026)
ENSEMBLE
Tox
[E[||X]
W 0
To
Concat
Scaled Dot-Product Attention
140
W
W
W
Figure 1: schema a LoRA-Ensemble. The computation structure of the multi-head self-attention module (right), and LoRA-Ensembll module (bottom left). denotes the actual input, and æ represents the intermediate input representation.
3 Experiments
3.1 Experimental Setup
We evaluate LoR.--nseemble across multiple datasets spanning three modalities in terms of predictive curacy, uncertainty calibration, and computational efficiency. Unless stated otherwise, reported results correspond to the mean over five independent runs with different random initializations, with 1-a error bars. For NNtturalist, we report results over three runs due to computational cost.
We ovaluated each method's predictive performance using classification accuracy and F1 score, and its calibration quality through Expected Calibration Error (ECE), Negative Log- Likelihood (NLL), and Brier score. The ECE measures how far predicted confidences deviate from observed error rates, i.e., perfect calibration occurs when the estimated uncertainties match the actual likelihood of misclassification. The definitions of all metrics are given in Appendix S
The primary hyperparameter introduced by our method the LoRA rank, r, which controls the dimension- ality of the low-rank adaptation matrices and thus directly affects both model capacity and computational overhead. For each dataset, we specily the selected rank in the corresponding experimental subsection. comprehensive sensitivity analysis of the LoRA rank, together with practical guidelines for its selection, is provided in Appendix
3.2 Baselines
LoRA-Ensemble is evaluated against range of baselines. As a sanity check, we include a single Vision Transformer (ViT) model as well as a ViT model augmented with LoRA in the attention modules. These models lack dedicated mechanisms for uncertainty calibration and rely solely class-conditional likelihoods to quantify uncertainty.
As gold-standard reference for uncertainty estimation, we compare against an Explicit Ensemble, where the full set of weights is independently trained for every member starting from the same pre-trained backbone (see Appendix R.2 for details).
In addition, we evaluate several common implicit approaches: (i) Monte Carlo Dropout (MC Dropout)
implemented in (Li et al. (023), (ii) Batch-Ensembi (Wen al.) 22220). (iii Snapshot Ensemble (Huang et al. 2017), and (iv) Last-layer Ensemble, similar to MTMO (Havasi al. 2020). Implementation details
4

---

Published in Transactions on Machine Learning Research (02/2026)
are provided in Appendix R.3 transformers, we refer to Appendix
R.4 and R.5 For challenges related to adapting certain implicit methods to
R.7
We also include Epistemic Neural Networks (EpiNet) (Osband et al., 2022 which augment a base net- work with a small auxliaay network that takes random epistemic indices as input, enabling ensemble-like uncertainty estimation without maintaining multiple full models (see Appendix R.6 for details).
For the CIFAR-100 100 experiments, we additionally compare with Speraiaa--nommalizedd Neural Gaussian Process (SNGP) (Liu et al. 2020), efficient single-model uncertainty estimation method. For HAM10000, we include two recent. adaptations designed to satisfy SNGP's bi-Lipschitz assumptions in transformers: T.2 (Kim) et al. 2021) and LRFormer Yee et al.) 2023) (see Appendix R.9 for details). For the SST-2 experiments, we further compare against Bayesian-LoRA (Yang el al., 2024). a method for improving uncertainly calibration in fine-tuned large language models.
3.3 Compute Cost
LoRA-Ensemble achieves substantial efficiency gains over Explicit Ensembles: with 16 members on the CIFAR-100 dataset, it requires approximately 14 times fewer paramoters, 9 times less inference memory, and delivers over times faster inference (when batch size 1). See Tab. g and refer to Appendix A.9 for more details. Training time comparable between the two methods, as LoRA--nsemble processes all members jointly while the Explicit Ensemble trains members sequentially LoRA-Ensemble lies in its memory efficiency, enabling deployment of large ensembles resource-constrained hardware where Explicit Ensembles would be infeasible. We note that Explicit. Ensembles can be parallelized across multiple GPUs by distributing members to separate devices; however, this requires proportionally more hardware.
single GPU. The primary advantage of
3.4 CIFAR-100
As a first sandbox experiment, we perform image classification for the popular, widely used CIFAR-100 benchmark [Krizhevsky] (209) (see Appendix A for CIFAR-10 experiments). The dataset consists of 100 object classes, each with 600 samples, for a total size of 60 000 images. From that set, 10 000 images are designated as test data, with all classes equally distributed between the training and testing portions.
Quantitative results are summarized in Tab. [ Reliability diagrams, along with plots depicting classification accuracy and ECE as function of ensemble size, are provided in Appendix A.3 The LoRA rank was set to 8 for this experiment.
MoRA-Ennemble consistently reaches higher accuracy than MC Dropout, and Snapshot Ensemble, with a notable edge of approximately 5 percentage points. Last-Layer Ensemble improves calibration but at the cost of a substantial accuracy drop, which we altribute to the limited number of trainable paramelers. SNGP fails to converge on the ViT backbone, reaching only approximately 32% accuracy; we provide a detailed analysis in Appendix R.9 EpiNet approaches the accuracy of Explicit Ensemble while its calibration falls short in comparison. Despite its conceptual similarity to the LoRA-Ensemble the
is the weakest performer among all methods when applied to transformers. Appendix RR3 examines this finding in detail and outlines key distinctions between the two approaches. Surprisingly, Ensembblee also consistently surpasses the Explicit Ensemble by about 2 percentage points, apparently a consequence of the fact that already a single ViT model, and thus every ensemble member, benefits from the addition of LoRA.
The LoRA-Ensemble also achieves better-calibratee predictive uncertainties than all implicit ensembling methods and the Explicit Ensemble. Interestingly, although a single LoRA network is already very well calibrated, forming ensemble slightly degrades its calibration, an effect not observed for the NLL Brier score (Tab. D.. The reliability diagram in Fig. 5 in the appendix somewhat clucidates this unexpected behavior: LoRA- E-Esemblle is under-confident on CIFAR- 100, i.e., its predictions are more accurate than its confidence suggests. As noted by (Rahaman & Thiery 2020), ensembling under-confident models can worsen calibration since accuracy grows faster than confidence. While under-confidence may be preferable safety- critical settings, where esttnnain uncertainty safer than being over-confident, we show in Appendix] that simple post-hoc Temperature Scaling effectively corrects this and yields near-perfect calibration.
5

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Model performance on the CIFAR-100 dataset for the compared methods. members. Best score for each metric in bold, second-best underlined.
Ensembles have
Method
Accuracy (1)
F1 (t)
ECE (1)
NLL (4)
Brier (4)
Single Network
0.3
0.3
0.145 = 0.004
1.181 = 0.019
0.370 = 0.004
Single Net w/ LoRA
79.6
0.2
0.014 0.003
0.671 = 0.005
0.286 + 0.003
MC Dropout
0.5
0.4
0.055 0.002
1.138 = 0.014
0.336 = 0.005
Last-layer Ensemble
0.0
730
0.093 0.000
0.978 = 0.000
0.376 = 0.000
Snapshot Ensemble
0.1
0.2
0.123 0.002
4.416 0.046
1.614 0.007
SNGP
0.4
0.4
0.072 = 0.004
2.744 0.010
0.817 = 0.002
ENN (EpiNet)
7.2
0.2
0.128 + 0.003
1.016 = 0.015
0.323 = 0.005
Batch-Ensemble
68.8 0.1
0.1
0.102 + 0.002
1.093 = 0.002
0,437 + 0.001
Explicit Ensemble
79.8 + 0.1
79.8 4: 0.2
0.100 0.001
0.745 + 0.003
0.284 + 0.002
LoRA-Fnsennn
82.5 + 0.1
82.5 + 0.1
0.035 + 0.001
0.587 + 0.001
0.253 0.000
Beyond transformers, we also adapt LoRA-Ensomblc to convolutional neural networks, ovaluating a RosNet- backbone on CIFAR-100; results are reported Appendix
3.5 HAM10000 Lesion Classification
The HAM10000 dataset was proposed for the Human Against Machine with 10 000 training images study (Tschanl et al. 2018). It consists of 10015 dermatoscopic images of pigmented skin lesions, collected from different populations. We use stratified 80/20 split at the image level, without prior near-duplicate filtering, patient- or lesion-level grouping. The dataset was initially assembled to compare machine learning methods against medical professionals on the task of classifying common pigmented skin lesions. Comparod to CIFAR- 100, this arguably a more relevant lest bed for our method: in the medical domain, uncertainly calibration is critical, due to the potentially far-reaching consequences of incorrect diagnoses and treatment planning.
We therefore evaluate the same group of models on this task, focusing on both predictive accuracy and calibration. The results are summarized in Tab. 2 The LoRA rank was set to 4 for this experiment. Similar to the CIFAR-100 evaluation, LoRA-Ensemble outperforms all other methods by a clear margin, with respect to both classification accuracy and calibration.
The experiments also further support the above discussion of confidence vs. ensemble size (Sec. 3.4 For HAM10000, LoRA-Ensemble slightly over-confident (just like the Explicit Ensemble) and, indeed, its calibration error decreases with ensemble size in this case, see Appendix A.4
We conducted further experiments on HAM10000 using different backbone architectures (DeiT) with varying numbers of parameters. See Tab. 10 in Appendix B LoRA-Ensemble generalizes smoothly across different backbones, and as the number of parameters in the backbone increases, its advantage over the Explicit Ensemble becomes more pronounced, in both accuracy and calibration.
3.6 Large-Scale Fine-Grained Image Classification with iNaturalist.
To demonstrate that LoRA-Ensemble scales large, fine-grained, real-world datasets, we apply to iNat- uralist 2017 (Van Horn et al. 2018), comprising 675 170 images across 5 089 species, order of magnitude larger than CIFAR-100. Severe class imbalance, high intra-class variability, and subtle inter-class differences make uncertainty quantification especially challenging due to the difficulty to avoid overconfident errors among similar species, and to flag uncertain predictions for rare species. The LoRA rank was set. to 64 for this experiment.
As shown in Tab. 3 our GoRA-Ensembll almost matches the Explicit Ensemble accuracy, while substan- tially improving the calibration, using only fraction of the parameters and compute. This demonstrates

---

Published in Transactions on Machine Learning Research (02/2026)
Model performance on the HAM10000 dataset for the compared methods. members. Best score for each metric in bold, second- best underlined
Table 2:
Ensembles have
Method
Accuracy (1)
F1 (t)
ECE (1)
NLL (4)
Brier (4)
Single Network
84.1 0.3
71.4 0.7
0.139 + 0.004
1.138 + 0.040
0.291 + 0.009
Single Net w/ LoRA
8.7
70.7
0.085 + 0.004
0.569 = 0.027
0.256 = 0.011
LRFormer
1.9
3.2
0.053 = 0.022
0.737 = 0.014
0.354 = 0.011
L2
1.
5..7
0.065 + 0.024
0.766 = 0.036
0.360 + 0.021
MC Dropout
8.4
7..0
0.099 = 0.007
0.631 = 0.023
0.270 0.009
Snapshot Ensemble
0.3
7.9
0.058 0.004
0.431 = 0.007
0.217 0.004
Batch-Ensemble
76.8 1.6
58.4
0.064 0.021
0.651 = 0.003
0.332 + 0.002
Explicit Ensemble
85.8 + 0.2
74.6 0.4
0.105 + 0.002
0.536 + 0.007
0.218 + 0.002
88.0 + 0.2
78.3 + 0.6
0.037 + 0.002
0.342 + 0.003
0.175 + 0.002
that the method scales well and enables reliable uncertainty estimation for large, fine-grained, imbalanced datasets. Refer to Appendix for additional results.
Table 3: Performance the iNat 2017 dataset for all compared methods using three different random seeds. Ensembles have 4 members. Best score for each metric in bold, second-bost underlined.
Method
Accuracy (1)
F1 (t)
ECE (L)
NLL (1)
Brier (1)
Single Network
42.6 0.2
37.8 0.2
0.293 + 0.002
1.054 0.001
0.207 0.001
Single Net w/ LoRA
17.7 0.1
43.1 0.1
0.096 + 0.001
0.662 0.001
0.166 0.000
MC Dropout
47.5 0.1
0.1
0.206 0.002
0.895 0.002
0.172 0.000
Explicit Ensemble
0.2
44.6 0.3
0.199 + 0.002
0.716 0.002
0.165 + 0.000
LoRA-Easemble
49.3 1 0.1
44.1 4 0.2
0.045 1 0.001
0.610 1 0.000
0.160 0.000
3.7 Extension to Additional Modalities: Audio (ESC-50) and Language (SST-2)
As further benchmark from different application domain, we process the ESC-50 envirommental sounds dataset (Piczak) 2015). It consists of 2 000 sound samples, each five seconds long, that represent 50 different semantic classes with 40 samples each. To prepare the raw input waveforms for analysis, they converted into 2-dimensional time/freqqenge spectrograms, see (Gong et al. 2021). These spectrograms form the input for Audio rogctrogram Transformer, a state-of-the-art transformer model for sound classification. The LoRA rank is set to 16 in this experiment.
Morcover, we extend our empirical evaluation to the natural language domain using the SST-2 binary sentiment classification benchmark (Socher et al. 2013). SST-2 consists of short movie review sentences annotated with positive negative sentiment and is part of the GLUE benchmark (Wang al. 2018a). The standard GLUE split contains 67,349 training samples and 872 validation samples. We use BERT base uncased (Devlin et al. 2019) as a backbone, and the LoRA rank is set to 64 in this experiment.
Results for both datascts are provided Appendix A.2 and Appendix A.7 These results suggest that the benefits of LoRA-Ensembly extend beyond the vision domain: while the Explicit Ensemble retains a slight edge in predictive accuracy on both tasks, LoRA-Ensemble remains competitive overall and continues deliver strong calibration, consistent with the trends observed in the image classification experiments.
3.8
Detection & Dataset Shift Robustness
For the out-of-distribution (OOD) experiment, models are trained on CIFAR-100 and evaluated on in- distribution samples from CIFAR-100 as well as OOD samples from CIFAR-10 or SVHN (Netzer et al.
2011), following standard practice (Hendryyks Gimpel 2016). Following Sim et al. 2023) and Chen et al.

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Model performance on the OOD task. CIFAR-100 is used as the in-distribution dataset and CIFAR- 10 and SVHN as the out-of-distribution datasct. Ensembles for all methods consist of 16 members. Results for Split-Ensemmbl are taken from (Chen et al. 2024 The best score for each metric is highlighted in bold, with the second-best score underlined.
OOD Dataset
CIFAR-10
SVHN
Method
AUROC (1)
AUPRO (1)
FPR 95% (1) AUROC (t)
AUPRO (1)
FPR 95% TPR (4)
Split-Ensemble
Chee al. 2024
79.2
81.7
78.5
81.2
69.9
75.0
Single Network
0.3
715
61.+ 1.2
764+18
86.8 1.0
65.9 =
Single Network with LoRAA
80.0+0.11 0.1
78.0 0.3
57.3 + 0.8
85.9 1.0++0.0 0.9
93.1 +.3
49.7 + 3.0
MC Dropout
56.6
92.2
74.5
3.4
ENN (EpiNet)
7.2
=
63.0
0.6
88.0 0.2
50.5 0.1
Explicii Ensemble
79.0 0.1
73.4 = 0.2
0.9
7488
86.6 :: 0.7
61.01 1.8
SoltA-Ensemhil
82.1 0.1
80.4 0.1
54.1 + 0.3
89.9 0.6
95.2 0.3
41.6 4 1.6
2024h, use the maximum softmax probability as the confidence score. Performance is measured using AUROC, AUPRC, and FPR 95% TPR.
Table shows that LoRA-Fnsemble achieves superior performance across both settings and metrics, sur- passing all baselines, including the recently proposed Split-Ensemble approach (Chen et al. 2024), which was specifically designed for OOD detection. Consistent with earlier observations, even a single LoRA model outperforms the Explicit Ensemble, highlighting its robustness in OOD scenarios.
To further assess robustness under distribution shifts, evaluate the CIFAR-10/100-4 benchmarks across varying corruption severilies. As detailed in Appendix A.8 LoRA-Eusemble consistently maintains superior accuracy and calibration under increasing shift intensity.
4 Enhanced Diversity in LoRA-Ensemble
To better understand the behavior JoRA-Ensembol we explore the diversity of its members and compare the Explicit Ensemble. The experiments are run on HAM10000 with 16 ensemble members. Diversity is crucial for effective ensembles, highly correlated members offer little added value (Zhang, (012). If ensemble contains diverse parameter configurations that equally explain observations, then it will more comprehensively capture the epistemic uncerlainly (Kendall & Gal 2017). For empirical evidence, refer to Appendix
Following Fort et al.) 2019b), we first assess function space diversity through the predictions of individ- ual ensemble members. In Fig. 2 we first compute the disagreement rate on the test set, defined as EN I[To, (X,,) & (X..),, where To, (Xa) represents the class label predicted by ensemble member for input Xn, and I is the indicator function. Next, we construct probability distribution for each ensemble member by aggregating their softmax outputs across all test samples, then compute pairwise Jensen-Shannon divergences (JSD). Finally, we use t-SNE (Van der Maaten & Hinton 2008 to visualize their spread in fune- tion space (aggregated softmaxes). The analysis reveals that diversily among ensemble members compared to an Explicit, Ensemble. Lee, LoRA-Ensemble appears to capture a wider range of modes in function space.
cxhibits significantly higher
Explicit
LoRA
Explicit
LoRA
614
Lake misett Marre
12342 DEUSDDO
Costce Merber
Enerbe Nember
EusTcc Nenber
Cremria Mer2a"
Cork LE EEveaA
(s) pairwise disagreement rate
(b) Jensen-Shanan divergence
(c) t-SNE
Figure 2: Function space analysis of Ensembe Explicit Ensemble.
8

---

Published in Transactions on Machine Learning Research (02/2026)
We further inspect the weight spaces of LoRA-Ensemble and Explicit Ensemble with spectral analysis, focusing on the projection matrices within the attention blocks of the ViT (Base-32) model pre-trained on ImageNet. We show the analysis for value projection matrices, given their strong association with learned representations; details for query and key projection matrices can be found in Appendix D We employ Singular Value Decomposition (SVD) to identify the most significant translormations encoded in the weights, following the logic that larger singular values correspond to the most impactful components. As proposed by (Shuttleworth al., 2024), we analyze the similarity between the initial (pre-trained) weights and the final trained weights of ensemble members. LoRA-Ensembll and Explicit Ensemble lead to very different parameter updates. LoRA-Ensemble introduces new high-ranking singular vectors that are near-orthogonal to those in the initial weights, referred to as 'intruder dimensions' (Shuttleworth et al. 2024). In contrast, Explicit Ensemble members tend to adhere closely to the spectral structure of the initial weights (see Fig. 13 in Appendix).
The random initialization of matrices A and in the LoRA module leads to an intriguing phenomenon: the intruder dimensions of different LoRA-Fnsemble members are near-orthogonal, as shown by the cosine similarities between the highest-ranking singular vectors of different members in Fig, 3 (for details see Appendix Dj. The figure shows similarity values averaged over layers and pairs of members, for rank 4. Notably, the highest-ranked singular vectors of distinet members exhibit almost no similarity: in contrast to the Explicit Ensemble, where they are highly correlated. The weight-space cosine similarity provides further evidence of enhanced diversity. LoRA-Ensenen BeBA-ensemble members exhibit greatly increased diversity in weight space. To visualize training trajectories, we apply t-SNE to plot the evolution of the model weights during training. LoRA-Fnsemlle members span larger part of the loss landscape, indicating diverse learning dynamics. In contrast, Explicit Ensemble members remain closer to the initial weights, reflecting reduced diversity. Overall, these results suggest LoRA-EEsennt better explores the weight space, and thus the epistemic uncertainty.
Explicit
LoRA
Explicit
LoRA
Venbo Sinpur veltee 050022
33545 x MULDIII
1 1 S001a0
(a) cos-similarity of high-ranking singular vectors
(b) weight-space cosine similarity
(c) training trajectories
Figure Weight space analysis of CoRA-Ensemble vs. Explicit Ensemble.
5 Discussion
Effectiveness of LoRA-Ensemble Across diverse tasks, our experiments show that LoRA-ensemble matches or surpasses the predictive performance of the slate-of-the-art Explicit Ensemble while offering superior calibration. Note, though, that there are exceptions to this trend: e.g., on ESC-50, the performance of the Explicit Ensemble is comparable, and on SST-2 it retains a small advantage. Adding LoRA to a single model without any ensembling improves calibration in most experiments beyond that of a 16-member Explicit Ensemble. This effect may be linked to the well-documented over-paramcterization of modern neural networks, which often achieve higher predictive accuracy at the cost of poorer calibration (e.g.. Guo et al., 2017). By incorporating LoRA while treating all pre-trained weights as constants, we significantly reduce the trainable parameter space, potentially favoring better calibration. However, limiting trainable parameters alone does not ensure better accuracy calibration, c.g., many forms of regularization or selective training may fall short. We believe that the effectiveness of LoRA-Eusemble stems from its unique learning dynamics, which we explore in Sec. | and Appendix [D] Its members converge to diverse solutions that span a broader area of the loss landscape, cnabling better exploration of the weight space and more effective estimates of epistemic uncertainty. Increasing the number of members in the LoRA-Ensemble enhances predictive power, potentially improving accuracy while maintaining good calibration due to the limited number of trainable
9

---

Published in Transactions on Machine Learning Research (02/2026)
weights. However, if the trainable weights are not limited, c.g. when increasing the LoRA rank too far, calibration can worsen, as shown in Fig Ta Tab 7 This effect aligns with the findings of Shuttleworth et al. (2024), which indicate that excessively increasing the rank of LoRA may cause it to lose its unique learning dynamics. Furthermore, jointly training the backbone and optimizing all parameters simultancously degrades performance, see Appendix [ for details. Conversely, enhancing predictive power by increasing the pre-trained weights (while keeping trainable weights constant) further improves the effectiveness of the JoRA-Ensemble see Appendix B Lastly, the LoRA-Ensemble remains effective even when pre-training is not available; see Appendix I for the experiment where it was pre-trained and fine-tuned on the same target dataset.
Comparison to Bayesian LoRA. On the language modeling task (SST-2 sentiment classification), LoRA- Ensemble consistently outperforms Bayes- LoRA (Yang et al. 2024) across accuracy, F1, NLL, and Brier score, with Bayes-LoRA only achieving a marginally lower ECE. Refer to Tab. g and Appendix [A.7] This result is consistent with previous works, showing that Laplace-basee methods improve calibration at the expense of predictive accuracy (Deng et al. 2022) or often fall short compared to ensembles (Daxberger) et al. 2021 Eschenhagen et al. 2021). Notably, in our experiments,
is more than 10x faster at inference, demonstrating that it combines strong accuracy, reliable calibration, and efficiency. In contrast, the Laplace-based Bayesian method trades accuracy for improved uncertainty estimates while remaining less efficient. Refer to Appendix R.8 for more details and discussion.
Practical Guidance. Our results suggest that the LoRA rank should be kept low unless accuracy saturates, as unnecessarily high ranks may degrade calibration. Calibration behavior also varies with ensemble size and dataset difficulty: on simpler datasets, increasing ensemble size induce under-confidencee, while harder datasets it improves calibration. In cases of under-confidence,, simple post-hoe temperature scaling is effective (Appendix J.. Practical hyperparameter guidelines are summarized in Appendix C
Limitations & Future Work. Despite its memory-effciint design and reduced per-member training and inference overhead, our ensembling approach maintains computational complexity similar to that of conventional ensembles, since each batch still requires separate forward passes for every member. While GoRA-Ensemble is directly applicable transformers with standard linear attention projections, its behavior in the context of more complex variants such as mixture-of-cxperts (MoE) models (Jiang al.) 2024) architectures with non-standard allention parameterizations remains an open question and a promising direction for future work. As discussed by Rahaman & Thiery (2020), ; our work also suggests that in a nigh-parameter regime, deep ensembles may not exhibit the same behavior as they do in a low-parameter regime, where they typically improve calibration properties. This type of phase shift in the bias-variance trade-off, the so-called Double Descent. Phenomenon, has previously been observed for large neural networks (Nakkiran et al. 2021). It would be valuable to conduct an in-depth analysis of how deep ensembles behave in high- pathharrreeerr regimes.
6 Related Work
Estimation of Epistemic Uncertainty. A lot of work has gone into estimating the epistemic uncertainty in Artificial Neural Networks (ANN). As the analytical computation of the posterior in such models is generally intractable, methods for approximate Bayesian inference have been proposed. Such methods rely on imposing an appropriate prior on the weights and using the likelihood of the training data to get an approximate posterior of the weight space.
The main techniques are, on the one hand, Variational Inference Graves 2011 Ranganath et al. 2014), which (Blundell et al.] 2015) have specialized to neural networks as Bayes by Backprop. And on the other hand variants of Markov Chain Monte Carlo (MCMC) NNal 1996 Chen et al. 2014), including Stochastic Gradient Langevin Dynamics (SGLD) (Welling et al. 2011). These, however, are often not able to accurately capture high-dimensional and highly non-convex loss landscapes, like the ones usually encountered in deep learning (Gustafsson et al. 2019). More recently, Bayesian LoRA methods have been explored, with Yang ct al. (2024) using a Laplace approximation for improved calibration and Wang et al. (2024) jointly lcarning mean and covariance during fino-tuning. An alternative line of work focuses on singlo-model uncertainty estimation. Spectral-normalliedd Neural Gaussian Process (SNGP) (Liu et. al. 2020) combines spectral
10

---

Published in Transactions on Machine Learning Research (02/2026)
normalization with a Gaussian Process output layer to achieve distanco-aware uncertainty. However, SNGP's bi-Lipschitz assumptions do not hold for transformers, as dot-product self-attention has an unbounded Lipschitz constant. To address this, Kim et al. (2021) propose L2 Self- Aete-itttntion and Ye et al. (2023) introduce LRFormer, both designed to satisfy Lipschitz constraints in transformer architectures. Epistemic Neural Networks (EpiNet) (Osband et al. 2022) take a different approach by augment a base network with a small auxiliary network that takes learnable epistemic indices as input, producing diverse predictions without maintaining multiple full models.
Ensembles and Implicit Ensembling. Lakshminarayanan et al. (0117 have proposed a method known as deep ensembles. It uses a set of neural networks with identical architecture that are independently and randomly initialized, and (as usual) trained with variants of Stochastic Gradient Descent (SGD). While the latter introduces further stochasticity, Fort et al. (2019a) have shown that the initialization of the weights is more important to explore the admissible weight space. Ensemble members will generally converge to different modes of the loss function, such that they can be considered Monte Carlo samples of the posterior distribution [Wilson & Iamailov] 2020; Izmailov et al.) (201)). While ensembles, in general, yield the best results in terms of accuracy and uncertainty calibration, a straightforwan implementation suffers from high memory and compute requirements, since multiple instances of the full neural network must be trained and stored. This can become prohibitive for modern neural networks with many millions, or even billions, of parameters.
Consequently, researchers have attempted to find ways of mimicking the principle of deep ensembles without creating several full copies of the base model. (Gal & Ghahramani] 2013) have proposed Monte Carlo Dropout, where the posterior is approximated by sampling different dropout patterns at inference time, While this is less expensive in terms of memory, performance is often worse. Masksembles (Durasov et al., 2020) are a variant that attempts to select suitable dropout masks in order to obtain better uncertainty estimates. Snapshot Ensembles (Huang et al.) 2017)) use cyclic learning rates to steer the learning process such that it passes through multiple local minima, which are then stored as ensemble members. This reduces the training effort but does not address memory roquirements inference time.
Particularly relevant for our work are atlempls that employ a shared backbone and modify only selected layers. (Havasi et al., 2020) follow that strategy, in their case only the first and last layer of a neural network are replicated and trained independently to emulato an ensemble. Packed-Ensemble (Laurent et al., 2023 leverage grouped convolutions to train lightweight ensembbee within a single shared backbone. Batch- GoRA-Ensemble in that it also uses low-rank matrices to change the model parameters. More specifically, shared weight matrices are modulated by element-wise multiplication with different rank-1 matrices to achieve the behavior of a deep ensemble while adding only a small number of parameters. Wenzel et al. (2020) take this concept further by also ensembling over different
Ensemble (Wen et al. 2020) is similar
(2022) freeze all weights of the base model and instead vary the feature-wise lincar modulation (FiLM, Li et al. 2018b Takeda et al. 2021). A related concept was recently introducod
settings. Turkoglu et al.
for LLMs: the Mixtral of Experts model (Jiang et al. 2024 averages over a sparse mixture of experts to efficiently generate text.
Low-Rank Adaptation in Transformer Networks. Low-Rank Adaptation was originally conceived as a parameter-efficient way of fine-tuning Large Language Models (LLMs) (Hu et al. 2021). It is based on the observation that, while modern neural networks have huge parameter spaces, the solutions they converge to have much lower intrinsic dimension (Li et al.] 20088b Aghajanyan et al., 2020). LoRA exploits this and Hu et al. (202I) show that even when fine-tuning only low-rank update matrix B- (sometimes with rank as low one or two), the resulting models are competitive with much more expensive fine-tuning schemes. The method quickly became popular and has since also been extended with weiglit-ecoomposition [Liu et al.) 2024). The Low-Rank Adaptation (LoRA) idea has been applied in various fields, notably for denoising diffusion models (Luo et al. 2023 Golnari 2023). As we have shown, the LoRA adaplation mechanism naturally lends itself to parameter-efficien
ensembling, which we investigate in the context of uncertainty calibration, with a primary focus on vision transformers but not limited to them. A similar idea has concurrently been explored for fine-tuning LLMs (Wang et al. 2203). yielding promising results in both predictive performance and uncertainty estimation.
11

---

Published in Transactions on Machine Learning Research (02/2026)
7 Conclusion
We have presented LoRA-Ensemble, a novel, parameter-cffiienn method for probabilistic learning that is tailored to the transformer architecture (and potentially other architectures that make use of the attention mechanism). LoRA-Ensemble uses a simple, but efficient trick to turn a single base model into an implicit ensemble: the weights of the base model are kept frozen, but are modulated with the Low-Rank Adaptation mechanism. By training multiple, stochastically varying instances of the low-rank matrices that define the modulation, one obtains diverse set of ensemble members that share the majority of their weights and introduce only minimal overhead through the coefficients of their individual low-rank matrices. Our extensive experiments demonstrate that the proposed approach excels in both predictive performance and uncertainty calibration. Not only does it surpass other state-of-the-ar implicit ensembling methods, but it also outperforms Explicit Ensembles on many tasks. This challenges the prevailing notion in the literature that Explicit Ensembles represent the upper bound for efficient ensembling methods (Wen et al. 2020).
8 Broader Impact
Belter-calibrated uncertainly estimates allow users to recognize unreliable model predictions and treat them with the necessary caution, thus improving decision-making in high-stakes settings. We point out that good calibration on benchmark datasets does not, by itself, guarantoe safe or reliable behavior real-world deployment: uncertainty may degrade under distribution shift, so external validation for the intended use case is advised before employing any uncertainty estimation method in practice.
More broadly, as models grow in size, running ensembles of multiple model instances further increases computational cost and energy consumption. Efficient ensembling approaches like ours aim to preserve ensemble performance while reducing resource usage and the associated environmental impact, contributing to more sustainable "Green Al".
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness language model fine-tuning. In Annual Meeting of the Association for Computational Linguistics & International Joint Conference on Natural Language Processing, 2020.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International Conference on Machine Learning, 2015.
Glenn W. Brier. Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1950.
Anthony Chon, Huanrui Yang, Yulu Gan, Denis A Gudovskiy, Zhen Dong, Haofan Wang, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. Split-Ensemble: Efficient OOD-aware ensemble via task and model splitting. In International Conference on Machine Learning, 2024.
Tianqi Chen, Emily B. Fox, and Carlos Guestrin. Stochastic gradient hamiltonian Monte Carlo. In Inter- national Conference on Machine Learning, 2014.
Gaurav Chhablani.
case- bert-bass-cause-fnneteee finetumed-sst2. 2023. Fine-tuned BERT model on GLUE SST-2 datasct.
https://huginggfaceccmm http:gginnnnac..ooochhhhaeeeeeee co/gchhablani/
Kevin Clark, Minh-Thang Luong, Quoe V Le, and Christopher D Manning.
Electra: Pro-training text
encoders as discriminators rather than generators. arxiv 2020. ICLR, 2020.
Ben Conrad. Fine-tuming vision transformers, 2023. URL httt:p//ggthubb..oo/oooorra///as Accessed: 2024-05-20.
Yin Cui, Menglin Jia, Tsung Yi Lin, Yang Song, and Serge Belongie. "lass-balanced loss based on effective number of samples. Tn IEFE Conference Computer Vision and Pattern Recognition, 2019.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Jesse Davis and Mark Goadrich. The relationship between precision-recall and ROC curves. In International Conference on Machine Learning, 2006.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace eeforeefs bayesian deep learning. Advances in newral information processing sys- tems, 34:20089-20111 20103, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference Computer Vision and Pattern Recognation, 2009.
Zhijie Deng, Feng Zhou, and Jun Zhu. Accelerated linearized laplace approximation for bayesian deep learning. Advonces in Neural Information Processing Systems, 35:2695 2708, 2022.
Armen Der Kiureghian and Ove Ditlevsen. Aleatory or epistemic? does matter? Structural Safety, 31((), 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
Nikita Durasov, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Masksembles for uncortainty estimation. In IEEE Computer Sociely Conference on Computer Vision and Pallern Recognition, 2020.
Runa Eschenhagen, Erik Daxberger, Philipp Hennig, and Agustimus Kristiadi. Mixtures of laplace approxi- mations for improved post-hoe uncertainty in deep learning. aix preprint (XNx:2111100571 2021.
Vlad Fomenko, Han Yu, Jongho Lee, Stanley Hsieh, and Weizhu Chen. A note on lora. orXiv preprial arXiv:2404.0500 2024.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep Ensembles: A loss landscape perspective. 2019a.
arXiv preprint orXiv:1912.022
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective.
arXiv preprint Xix: 191.0011 2.07577, 2019b.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainly in deep learning. In International Conference on Machine Learning, 2015.
Jort F. Gemmeke, Daniel P.W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and humman-labeles dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing. 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep foodforward neural networks. International Conference on Artificial Intelligence and Statistics, 2010.
Parcesa Amench Golnari. LoRA-cnhanced distillation on guided diffusion models. arXiv preprint
arXiv:2312.0689% 2023.
Yuan Gong, Yu-An Chung, and James Glass. arXiv:210,.01778, 2021.
Ast: Audio spectrogram transformer. ar preprint
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Pro- cessing Systems, 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Fredrik K. Gustafsson, Martin Danelljan, and Thomas B. Schon. Evaluating scalable bayesian deep learning methods for robust computer vision. In IEEE Conference Computer Vision and Pallern Recognation Workshops, 2019.
James A. Hanley and Barbara J. McNeil. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 1982.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snock, Balaji Lakshmi- narayanan; Andrew M. Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In International Conference on Learning Representations, 2020.
Dan Hendrycks and Thomas Dictterich, Benchmarking neural network robustness to common corruptions and perturbations. 2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and outtof-distribution examples in neural networks. arXiv preprint arXiv:1610.02138 2016.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John F. Hopcroft, and Kilian Q. Weinberger. Snapshot Ensembles: Train 1, get m for free. In International Conference on Learning Representations, 2017.
Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, and Andrew Gordon Wilson. What are bayosian neural network posteriors really like? In Proceedings of Machine Learning Research, 2021.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Saymon Anloniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30, 2017.
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In Interna- tional Conference on Machine Learning, pp. 5562 5571. PMLR, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. Tn Advances in Neural Information Processing Systems, 2017.
Olivier Laurent, Adrien Lafage, Enzo Tartaglione, Geoffrey Daniel, Jean-Mare Martinez, Andrei Bursuc, and Gianni Franchi. Packod-ensembles for efficient uncertainty estimation. In International Conference on Learning Representations, 2023.
Hojung Lee and Jong-Seok Lee. Network fission ensembles for low-cost self-ensembles. Pallern Recognation Letters, 2025.
Bonan Li, Yinhan HU, Xuccheng Nie, Congying Han, Xiangjian Jiang, Tiande Guo, and Luoqi Liu. Dropkey for vision transformer. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Regresentations, 2018a.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Yanghao Li, Naiyan Wang, Jianping Shi, Xiaodi Hou, and Jiaying Liu. Adaptive batch normalization for practical domain adaptation. Paltern Recognition, 80, 2018b.
Jeremiah Liu, Zi Lin, Shreyas Padby, Dustin Tran, Tania Bedrax- Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Tn Advances in Newrll Information Processing Systems, volume 33, pp. 7498-7512, 2020.
Jeremiah Liu, Shreyas Padhy, Jie Ren, Zi Lin, Yeming Wen, Alex Rudnicky, Dustin Tran, Balaji Lakshmi- narayanan, and Tania Bedrax- Weiss. A simple approach improve singlo-model deep uncertainty via distance-uwareeess. Journal of Machine Learning Research, 24(42):1-63, 2023.
Bedrax-Weiss.
Shih- Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Mlin-Hung Chen. DoRA: decoppsee low-rank adaptation. arXiv preprint orXiv:2402.09335, 2024.
Nya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Represenlations, 2017.
Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao. LCM-LoRA: A universal stable-diffusion acceleration module. arXiv preprint NNiv:331100555 2023.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for gener- ative adversarial networks. In International Conference on Learning Representations, 2018.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Nya Sutskever. Deep double descent: where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Erperiment, 2021 2021.
Radford M. Neal. Bayesian Learning for Neural Networks. Lecture Notes in Statistics. Springer New York, 1996.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
lan Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. arXiv preprint Stt::210700092, 2022.
Yaniv Ovadia, Emily Fortig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019.
Karol J. Piczak. ESC: Dataset for envirommental sound classification. In ACM Multimedia Conference, 2015.
Barbara Plank, Kristian Norgaard Jensen, and Rob Van Der Goot. Dan+: Danish nested named entities and lexical normalization. In Proceedings of the 28th international conference on computational linguistics, pp. 6649-6662, 2020.
Rahul Rahaman and Alexandre H. Thiery. Uncertainty quantification and deep ensembles. In Advances in Newral Information Processing Systems, 2020.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems, 2007.
Rajesh Ranganath, Sean Gerrish, and David M Blei. Black box variational inference. In International Conference on Artificial Intelligence and Statistics, 2014.
Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. LoRA vs full fine-tuning: An illusion of equivalence. arXiv preprint arXiv:24 10.2221002201 10.21222. 21228, 2024.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Minho Sim, Jongwhoa Lee, and Ho-Jin Choi. Attention masking for improved near out-of-distribution image detection. In IEFE International Conference on Big Dala and Smart Compuling, 2023.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christo- pher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Conference on Empirical Methods in Natural Language Processing, 2013.
Mana Takeda, Gibran Benitez, and Keiji Yanai. Training of multiple and mixed tasks with a single network using feature modulation. In International Conference on Pattern Recognition, 2021.
Arthur Thuy and Dries F. Benoit. Fast and reliable uncertainty quantification with neural network ensembles for industrial image classification. Annals of Operations Research, 2025.
Hugo Touvron, Matthicu Cord, Matthijs Douze; Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In Proceedings of Machine Learning Research, 2020.
Philipp Techandl, Cliff Rosendahl, and Harald Kittler. The HAM10000 dataset, a large collection of multi- source dermatoscopic images of common pigmented skin lesions. Scientific Data, 5, 2018.
Mchmet Ozgur Turkoglu, Alexander Becker, Hüseyin Anil Gündüz, Mina Rezaci, Bernd Bischl, Rodrigo Caye Daudt, Stefano D' Aronco, Jan Dirk Wegner, and Konrad Schindler. FiL.M-Fnsemble: Probabilistic deep learning via feature-wise linear modulation. In Advances in Neural Information Processing Systems, 2022.
Dennis Ulmer, Christian Hardmeier, and Jes Frellsen. Exploring predictive uncertainty and calibration in nlp: A study on the impact of method data scarcity. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2707-2735, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11,, 2008.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 8769-877 87788777, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, Manvel Avetisian, and Lconid Zhukov. Uncertainty estimation of transformer predictions for misclassification detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8237 -2522 Association for Computational Linguistics, 2022.
Alex Wang, Amanprect Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language undorstanding. In Proceedings of the 2018 EMNLP workshop BlackboeNLPP: Analyzing and interpreling newral networks for NLP, pp. 353 355, 2018a.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi- task benchmark and analysis platform for natural language understanding. In BlcckboNI.PGEMNT.P. 2018b.
Xi Wang, Laurence Aitchison, and Maja Rudolph. LoRA ensembles for large language model fine-tuning. arXiv preprint rxXio:2310000035 2023.
Yibin Wang, Haizhou Shi, Ligong Han, Dimitris Metaxas, and Hao Wang. Blob: Bayesian low-rank adapta- tion by backpropagation for large language models. Advances in Neural Information Processing Systems, 37:67758 67794, 2024.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Max Welling, D Bren, and Yee Whye Tch. Bayesian learning via stochastic gradient Langevin dynamics. In International Conference on Machine Learning, 2011.
Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: An alternative approach to efficient ensemble and lifelong learning. In International Conference Learning Representations, 2020.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robust- ness and uncertainty quantification. In Advances in Neural Information Processing Systems, 2020.
Andrew Gordon Wilson and Pavel Tamailov. Bayesian deep learning and probabilistic perspective of generalization. In Advances in Neural Information Processing Systems, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
Adamn X. Yang, Maxime Robeyns, Xi Wang, and Laurence Aitchison. Bayesian low-rank adaptation for large language models. In International Conference on Learning Representations, 2024.
Wenqian Ye, Yunsheng Ma, Xu Cao, and Kun Tang. Mitigating transformer overconfidence via Lipschitz regularization. In Proccedings of the Conference Uncertainty in Artificial Intelligence, 2023.
C Zhang. Ensemble machine learning: methods and applications. Springer Science & Business Media, 2012.
17

---

Published in Transactions on Machine Learning Research (02/2026)
Appendix
Contents
A
Additional Experiments and Results
20
10
20
ESC-50 Environmental Sound Classification
20
CIFAR-100
20
HAM10000 Lesion Classification
21
A.5
Comparison with a Single, High-rank LoRA Network
22
A.6
INaturalist 2017 Large-Scale Fine- Grained Image Classification
................................
22
A.7
SST-2 Language Modcling for Scntiment Classification
23
A
Robustness to Distribution Shifts: CIFAR-TO-C and CIFAR-100-
24
A.9
Computational Cost
24
B OORA-Ensemble" Generalization to Varying Model Sizes
25
C Hyperparameter Sclection and Sensitivity Analysis: LoRA Rank
27
D Weight Space Analysis: LoRRA-Enseblle versus Explicit Ensemble
29
E Correlation Analysis Between Ensemble Diversity and Predictive Performance
30
F Joint Training of Backbone and LoRA-Ensemble Layers on iNaturalist
32
Placement of LoRA-Ensemble Modules and Selection of Trainable Weights
33
LoRA-Ensemile for CNNs
33
Ensemble Tuned on the Same Dataset as the Backbone Model
33
Post-Hoc Temperature Scaling for Model Calibration
34
K Implementation of LoRA-Ensmeble
35
L Training Details of LoRA-Ensemble
36
M Initialization of LoRA-ensemble Parameters
37
N AST Implementation
37
Validation of AST Implementation
38
P Hyper-parameter Tuning for AST Experiment
39
18

---

Published in Transactions on Machine Learning Research (02/2026)
Computational Cost for AST Models
40
R Baselines
40
R.1
Hyperparameter Tuning for MC Dropout
40
R.2
Initialization of Explicit Ensemble Parameters
40
Batch-Ensembie
41
R.4
Snapshot Ensemble Implementation Details
44
R.5
Last Layer Ensemble
45
R.6
Epistemic Neural Networks (EpiNet)
45
R.7
FilM--Ensemble and Other Implicit Basclines: Practical Challenges
47
R.8
Bayesian LoRA
R.9
SNGP Vision Transformers
48
Definitions of Evaluation Metrics
50
S.1
Accuracy
50
S.2
Expected Calibration Error
50
Macro Fl-score
50
$.4
Negative Log- Likelihood (NLL)
51
5
Brier score
51
6
Area Under the Receiver Operating Characteristic Curve (AUROC)
51
7
Area Under the Precision- Recall Curve (AUPRC)
51
S.8
False Positive Rate at 95% True Positive Rate (FPR095TTH
52
19

---

Published in Transactions on Machine Learning Research (02/2026)
A
Additional Experiments and Results
This section presents comprchensive experimental results for the newly introduced CIFAR-10 dataset, En- vironmental Sound Classification on ESC-50. It also includes additional results for the CIFAR-100 and HAM10000 datasets and expanded baseline comparisons for CIFAR-100.
A.1 CIFAR-10
The results for the CIFAR-10 dataset, as shown in Tab. b indicate that JoRA-Ensembil outperforms all other methods. Close behind is a single network enhanced with LoRA. This mirrors the results found in the main paper for CIFAR-100, with the exception of the calibration for a single model. is important to note that although all methods achieve high accuracy and the differences between them are minimal, calibration is nearly perfect for most approaches. This suggests that the CIFAR-10 dataset is relatively easy for modern transformer models, and the results should not be over-inlerpreted. Nevertheless, the consistent, performance across different random seeds suggests that the ranking is likely significant. Given the balanced nature of the CIFAR-10 dataset, the accuracy and F1-score are almost identical.
Table 5: Performance the CIFAR-10 dataset for all compared methods. Ensembles have 16 members. Best score for each metric in bold second-best underlined.
Method
Accuracy (1)
F1 (t)
ECE (L)
NLL (L)
Brier (1)
Single Network
92.8 1 0.1
92.8 0.1
0.051 1 0.001
0.333 0.003
0.120 0.002
Single Net w/ LoRA
94.5 + 0.0
94.5
0.009 + 0.001
0.163 0.002
0.082 + 0.001
MC Dropout
0.2
0.2
0.023 + 0.002
0.260 0.005
0.110 0.003
Snapshot Ensemble
9.11
9.11
0.037 0.002
1.062 0.021
0.510 + 0.008
Match-Ensemble
88.5
0.1
0.048 0.001
0.347 0.001
0.172 0.000
Explicit Ensemble
1.1
9..1
0.031 + 0.001
0.181 0.002
0.087 + 0.001
95.9 11.9000 0.1
95.9 0.1
+
0.128 = 0.001
0.000
A.2 ESC-50 Environmental Sound Classification
Like for the VIT model, we train an Audio Spectrogram Transformer version of LoRA-Ensemble by modifying the attention layers with different sets of LoRA weights. That ensemble is then compared to single instance of AST with and without LoRA, to an Explicit Ensemble of AST-models, and to an MC Dropout variant of AST, similar to (Li et al.
2023 For ESC-50 a LoRA rank of 16 worked best, presumably due to the larger
domain gap between (image- based) pre-training and the actual audio classification task. The experimental evaluation in (Gong et al. 2021 employs the same performance metrics as before, but a slightly different evaluation protocol. Model training (and evaluation) is done a 5-fold cross- valsdvvaii setting, where the epoch with the best average accuracy across all five folds is chosen as the final model. The performance metrics given below are calculated by taking the predictions of all five folds the chosen epoch and evaluating accuracy and calibration metrics jointly. While the accuracy calculated this way is equivalent to the average of all five folds, others are not, so this method results in a more realistic calculation of the calibration metrics.
The results are summarized in Tab. G On this dataset the Explicit Ensemble, but still matches its performance with much lower computational demands, see Appendix Q Accuracy is insiguificantly lower, whereas calibration is slightly better in terms of ECE. We note that, remarkably, the weights used the transformer modules and for creating patch embeddings were pre-trained on images rather than audio streams.
does not significantly outperform
A.3 CIFAR-100
Inereasing the ensemble size of LoRA-Ensemble on CIFAR-100 improves classification accuracy but reduces calibration, as illustrated in Fig. 4 The reliability diagram in Fig. 3 highlights this behavior: networks with LoRA CIFAR-100 generally under-confident, with accuracy exceeding predicted confidence. As
20

---

Published in Transactions on Machine Learning Research (02/2026)
Table 6: Model performance on the ESC-50 dataset for the compared methods. Ensembles have members due memory limitations. Best score for each metric in bold, second-best underlined.
Method
Accuracy (1)
F1 (t)
ECE (1)
NLL (4)
Brier (4)
Single Network
89.6 0.7
89.5 0..50.1 0.7
0.039 = 0.004
0.410 = 0.020
0.164 = 0.009
Single Net w/ LoRA
88.0 + 0.3
87.8 + 0.3
0.043 + 0.004
0.461 = 0.019
0.186 + 0.005
MC Dropout
89.4 0.3
8.4
0.087 = 0.005
0.553 = 0.012
0.176 = 0.005
Explicit Ensemble
91.3 + 0.2
01.2
0.027 = 0.004
0.322 = 0.004
0.133 0.001
LoRA-Ensemble
91.1 + 0.2
90.8 0.2
0.021 + 0.003
0.328 = 0.004
0.138 + 0.001
observed by (Rahaman & Thiery 2020). ensembling under-confident models can exacerbate this discrepancy, leading to poorer calibration metrics.
Accuracy
Expected Calibration Error
82
0.14
LaRA-Ensemble
Explicit Ensemble
81
Expected Calibration Error
Snapshot Ensemble
Accuracy
0.10
0.08
0.06
LoRe-bnsemble
0.04
Expliit Ensemble
MC Dropout
Snapshot Ensemblee
12
14
Number of Members
Number of Members
Figure 4: Accuracy and Expected Calibration Error CIFAR-100, with different ensemble sizes.
Reliability Diagram
Reliability Diagram
1.0
100
1.0
Aug. Confidence 89.7%
Avg. Confidence 79.1%
Accuracy 80.0%
Avg. Accuracy 92.6%
Accuracy
Accuracy
80
0.8
80
% of samples in bin
% of samples in bin
0.7
70
0.7
70
0.6
60
camples
0.6
60
Accurscy
0.5
0.4
40
0.4
40
0.3
30
0.3
30
0.2
20
0.2
20
0.0.0
1.0
0.8.0
1.8
Figure 5: Reliability diagrams for Explicit Ensemble (left) and LoRA-Ensemile (right) with 16 members, CIFAR-100.
A.4 HAM10000 Lesion Classification
Classification accuracy and ECE for HAM10000 dataset are both graphed against ensemble size in Fig. b Again, LoRA-Ensemble outperforms all baselines for larger ensembles. In Fig. the reliability diagrams
21

---

Published in Transactions on Machine Learning Research (02/2026)
for and Explicit Ensemble with 16 members each the HAM10000 dataset are shown. Here, the models are overconfident, further supporting reasoning regarding the surprising behaviour of calibration with growing ensemble size in the case of CIFAR-100.
Accuracy
Expected Calibration Error
88
LoRt-Ensemble
0.14
Explicit Ensemble
Expected Calibration Error
MC
Snapshot Ensemble
0.12
0.08
Explicit Ensemble
MC Drepout
Snapshot Ensemble
12
14
Number of Members
Number of Members
Figure 6: Accuracy and Expected Calibration Error HAM10000, with different ensemble sizes.
Reliability Diagram
Reliability Diagram
1.0
Avg. Confidence
Avg. Confidence 91.5%
0.9
Avg. Accuracy 85.9%
90
0.9
Avg, Accuracy 88.1% Accuracy
90
Accuraty
0.8
Gap
80
0.8
Gap
00
samples
samples
0.7
70
0.7
70
0.6
samples
Accuracy
Accurac
0.5
50
0.5
50
of
0.4
40
0.4
40
30
0.3
30
0.1
10
0.1
10
0.0
0.2
0.3
0.4
0.7
0.0
0.0
0.3
0.4
0.7
0.8
1.0
Figure 7: Reliability diagrams for Explicit. Ensemble (left)) and LoR.A-Ensemble (right) with 16 members, on HAM10000.
A.5 Comparison with Single, High-rank LoRA Network
We compare the proposed LoRA-Ensemble method with an additional baseline: single high-rank LoRA model configured to have the same total number of trainable LoRA parameters as the LoRA-Ensemble. This evaluation is conducted on the CIFAR-100 classification task to examine the relative effectiveness of ensembling versus increasing parameter capacity within a single model.
Notably, shown in Tab. the high-rank LoRA model underperforms compared to the low-rank LoRA model. This result indicates that the performance gains of the LoRA-Fnsemble are not solely due to an increased number of trainable parameters but are instead attributable to the ensembling approach.
A.6 INaturalist 2017 Large-Scar Scale Grained Image Classification
In Fig. reliability diagrams for the iNat 2017 dataset are shown, once for BoRA-Ensemble and once for Explicit Ensemble, both with 4 members. One clearly the over-confidence of the Explicit model,
22

---

Published in Transactions on Machine Learning Research (02/2026)
Model performance on the CIFAR-100 dataset for the compared methods. members. Best score for each metric in bold, second best underlined.
Table 7:
Ensembles have
Method
Rank
Trainable ble params.
Accuracy (1)
F1 (t)
ECE (4)
NLL (4)
Brier (4)
Single Net w/ LoltA
8
666'724
79.6 1. 0.2
79.4 1. 0.2
0.014 0.003
0.671 1. 0.005
0.286 1. 0.003
Siugle Net w/ LoltA
9'514'084
77.0
77.0
0.001
LoRA-Basemble
10'667'584
82.5 0.1
and the much improved uncertainty calibration of LoRA-Fnsemble at almost the same accuracy (49.6% vs. 49.3%, c.f. Tab. 3
Reliability Diagram
Reliability Diagram
1.0
100
1.0
100
Avg. Confidence 69.3%
Avg. Confidence 53.9%
Avg. Accuracy
90
Avg.Accuracy
90
Accuracy
Accuracy
Gap
80
0.8
80
% of samplees in bin
% of samples in bin
0.7
70
0.7
70
0.6
60
AcaJracy
0.6
60
Acalto
50
40
0.3
30
0.3
30
0.2
20
0.2
20
10
10
0.0
1.8
0.0.
1.8
Figure 8: Reliability diagrams for Explicit Ensemble (left) and LoRA-Ensemiie (right) with 4 members, INat2017.
A.7 SST-2 Language Modeling for Sentiment Classification
To further illustrate the generality of our method, we extend the ovaluation to also include languago pro- cessing (NLP). Indeed, we find that LoRA-Ensemble also handles this very different modality and estimates well-calibrated uncertainties for language data.
We chose the Stanford Sentiment Treebank 2 (SST-2) datasct (Socher et al. 2013 for the experiment, widely used binary sentiment classification benchmark, and part of the GLUE benchmark suite (Wang al. 2018b). The model we use is the uncased BERT base model, which we fine-tune for SST-2.
Tab. presents the performance on the SST-2 validation set.. Among the methods, the LoRA-Ensemble shows strong overall performance: it achieves superior calibration in terms negative log- likelihoood (NLL), second-best ECE after Bayes-LoRA (with only negligible difference), and outperforms all basclines including the Explicit Ensemble. The Explicit Ensemble holds only marginal advantage of 0.5 percentage points in accuracy. In contrast, Monte Carlo Dropout improves calibration compared to single models but suffers from a substantial loss in accuracy, pattern consistent with our other experiments and aligned with findings reported in the literature (Li et al. 2023). A single LoRA-augmented model shows better calibration than a single model, but lags in accuracy. Bayes-LoRA achieves competitive uncertainty calibration, obtaining the best ECE, but its NLL, Brier score, and accuracy are worse than those both the Explicit Ensemble and the LnnA--nse and its accuracy even lower than a single model, reflecting tradc-off where improved calibration comes at the expense of predictive performance. Refer to Appendix R.8 for more details and discussion about Bayes-ToRA.
23

---

Published in Transactions on Machine Learning Research (02/2026)
Table 8: Performance on the SST-2 validation dataset, evaluated using five different random seeds per model. Ensembles have 8 members. Best score for each metric in bold, second-best underlined.
Method
Accuracy (1)
F1 (t)
ECE (L)
NLL (1)
Brier (4)
Single Network
92.5 0.2
92.5 0.2
0.064 = 0.003
0.345 = 0.012
0.136 + 0.003
Single Net w/ LoRA
91.6 + 0.5
91.6
0.059 = 0.005
0.292 = 0.016
0.148 + 0.008
MC Dropout
84.9 1.2
1.3
0.061 = 0.004
0.364 = 0.020
0.223 + 0.015
Bayes-LoRA
90.7 - 0.3
0.3
0.036 + 0.003
0.247 = 0.005
0.139 = 0.003
Explicit Ensemble
93.2 + 0.2
93.2 + 0.2
0.047 + 0.002
0.234 + 0.004
0.112 + 0.003
92.7 0.2
92.7 0.2
0.038 + 0.003
0.208 0.007
0.114 + 0.002
Table 9: Parameter counts and computation times for an Explicit Ensemble of 16 ViT models and the corresponding LoRA-Ensemble. Training time is the average duration for one epoch CIFAR-100, with batch size 32. Inference time is the average duration of forward pass, with batch size 1. GPU memory denotes the peak allocated CUDA memory.
Method
Params
Memerain [MiB]
Timerrain [s]
Mem [MiB]
Time [ms)]
Explicit Ensemble
16 87M
2045
114
16 x 1381
16 5.2
LoRA-Ensemble
1.12 x 87M
15821
1832
2335
14.8
A.8 Robustness to Distribution Shifts: CIFAR-10-C and CIFAR-100-C
Despite primarily evaluating the LoRA-Ensemble on in-distribution tasks, we also assess its robustness to out-of-distribution (OOD) imputs. A critical challenge arises when model encounters data at test time that differs from the training distribution. If the model then produces poorly calibrated uncertainty estimates, this can lead to unsafe or unreliable predictions (Hendrycks & Dictterich, 2019
To examine this, we evaluate our method on the CIFAR-10-C and CIFAR-100-C benchmark datasets. These datasets apply 19 distinct corruption types at five severity levels to the original CIFAR- 10 and CIFAR-100 test sets (Hendrycks
Dictterich] 2019)., introducing controlled distribution shifts, similar to prior work Ovadia et. al. 2019). For this evaluation, we use pretrained models trained on the clean datasets with minimal data augmentation (only rotations), and assess both predictive performance and calibration.
Fig. and 10 present the results. It is evident that the maintaining relatively low ECE scores even under high levels of distribution shift.
outperforms the other methods,
A.9 Computational Cost
In addition to evaluating classification performance and calibration, we assess the computational cost in terms of parameters, memory usage, training time, and inference time. The required resources are presented in Tab. g
The total number of parameters is reported for ensemble of 16 members, with LoRA matrices A and B of rank 8, Choosing different rank will slightly alter the parameter count; in many cases lower rank may suffice (Hu et al.) 2021). All measurements were conducted on a single NVIDIA Tesla A100-80GB GPU. Training time is reported as the total wall-clock time per epoch on CIFAR- with batch size 32. For the Explicit Ensemble, this corresponds to training 16 members sequentially; LoRA-ensomble trains all members jointly in a single pass. Inference time the average duration of a forward pass on a single CIFAR-100 example with batch size 1. For the Explicit Ensemble, members are processed sequentially on a single G we report the cumulative time for all 16 members.
Explicit Ensembles can be parallelized by distributing members across multiple GPUs; however, this requires proportionally more hardware and does not reflect fair comparison under fixed compute constraints.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Accuracy
Expected Calibration Error
95
Single Network
0.20
Single Net w/ LoRA
Expected Callbration Error
MC Dropout
Explicit Ensemble
LoRA-Ensemble
Accuracy (%)
0.15
85
80
0.10
Single Network
Single Net w/ LoRA
MC Drepout
Explicit Ensemble
70
Ensemble
0.00
Test
5
1est
Shift Intensity
Shift Intensity
(a) NLL
(b) Brier Score
1.6
Single Network
0.5
Single
Single Net LoRA
Single Net w/ LoRA
Negative Log- Li-Lekellhodd
14
MC Dropout
MC Dropout
Explicit Ensemble
Explicit Ensemble
LoRA Ensemble
LoRA: Ensemblle
Brier Score
1.0
0.8
0.6
0.2
0.4
0.1
0.2
Test
2
Test
5
Shift Intensity
Shift Intensity
(c)
(d)
Figure 9: MoRA-Ennemmbl evaluated under varying levels distribution shift on the CLFAR-10-C dataset. Each ensemble consists of 16 members. "Test" refers to the original CIFAR-10 test set, while the corrupted include test images subjected to 19 different augmentations multiple severity levels, introducing distribution shifts.
The results demonstrate that LoRA-Ensemble achieves substantial efficiency gains: approximately 14 times fewer parameters, times less inference memory, and over times faster inference compared to Explicit Ensembles. Training time comparable between the two methods, as LoRA--nsemble processes all members jointly while the Explicit Ensemble trains members sequentially. The primary advantage of LoRA-Ensemble lies in its memory efficiency, which enables deployment of large ensembles resource-constraimed hardware where Explicit. Ensembles would be infeasible.
B LoRA-Ensembles Generalization to Varying Model Sizes
Building upon our existing experiments with the HAM10000 datasct, we extended our analysis to include different backbone architectures with varying numbers of parameters. Specifically, we utilized various DeiT models pre-trained with distillation, as described by Touvron et al. (2020). The results are presented in Table 10 Notably, the DefT' Base-32 model the same as the VIT Base-32 model.
In the small parameter regime (Tiny-16, Small-16), the addition of single LoRA module did not consistently enhance calibration compared to using a single model. This observation contrasts with our findings in most
25

---

Published in Transactions on Machine Learning Research (02/2026)
Accuracy
Expected Calibration Error
Single Network Single Net LoRA
Single Network
0.30
Single Net w/ LoRA
MC Dropout
Expected Calibration Error
MC Dropout
75-
Explicit Ensemble
0.25
Explicit Ensemble
LoRA-Ensemble
LoRA-Ensemble
Accuracy (%)
70
0.20
65
0.15
0.10
55
50
45
0.00
Test
4
5
Test
Shift Intensity
Shift Intensity
(a)
(b) Brier Score
NLL
Single Network
0.8
Single Network
3.0
Single Net LoRA
Single Net w/ LoRA
Negative Log-Like Likeellhodd
MC Dropout
MC Dropout
Explicit Ensemble
Explicit Ensemble
LoRA Ensemble
LoRA: Ensembble
Brier Score
0.6
0.5
1.5
1.0
0.3
0.5
Test
2
Test
5
Shift Intensity
Shift Intensity
(c)
(d)
Figure 10: LoRA- Ensemble evaluated under varying levels distribution shift on the CIFAR-100-C dataset. Each ensemble consists members. "Test" refers to the original CIFAR- 100 test set, while the corrupted include test images subjected to 19 different augmentations multiple severity levels, introducing distribution shifts.
other experiments. However, in the larger parameter regime (ViT Base-32), incorporating even a single LoRA module significantly improved calibration. Increasing the number of ensembles in the LoRA-Fnsemble not only boosted accuracy but also enhanced calibration, enabling it match the performance of an Explicit Ensemble in both parameter regimes.
Last but not least, as the number of parameters in the backbone architecture increased, the superiority of the LoRA-Ensemble over the Explicit Ensemble in terms of both accuracy and calibration became more pronounced. This trend indicates that as backbone size grows, the advantages of LoRA-Ensemble become increasingly dominant.
Overall, the results demonstrate that the LoRA-Ensenin not only transfers successfully to a different backbone architecture (DeiT versus ViT) but also remains effective across varying parameter regimes.
26

---

Published in Transactions on Machine Learning Research (02/2026)
Table 10: Performance metrics on the HAM10000 dataset for different Vision Transformer architectures. Ensembles have 16 members. The top two results for each metric are highlighted: bold for the best, underlined for the second best.
Arch. Method
# Params,
Accuracy (1)
F1 (t)
ECE (4)
NLL())
Brier (1)
Single Net
89.0 + 0.3
0.4
0.096 0.003
0.909 0.037
0.202 + 0.005
Thy-16
DefI
Single Net w/ LoRA
5M
84.5 = 0.8
716 1.5
0.074
0.542
0.237 0.009
Explicit Ensemble
90.4 + 0.3
81.4
0.069 +
0.340 0.006
0.142 0.002
LoRA-Ensemble
88.9 = 0.4
8.2
0.025 + 0.003
0.325 0.004
0.164 + 0.002
Small-16 16
Single Net
89.6 0.4
0.5
0.093 +
0.876 0.032
0,191 + 0.007
DeiT
Single Net w/ LoRA
22M
86.3 0.5
1.
0.100
0.007
0.731 + 0.053
0.234 + 0.010
Explicit Ensemble
91.5 0.1
0.2
0.061
0.002
0.318 + 0.003
0.130 + 0.001
MoRA-Ensembol
90.4 0.1
82.8 0.4
0.047 = 0.002
0.292 0.002
0.144 + 0.001
Base-32
Single Net
84.1 + 0.3
0.7
0.139 + 0.004
1.138 + 0.040
0,291 + 0.009
DeiT
Single Net w/ LoRA
86M
83.2 = 0.7
1.3
0.085 + 0.004
0.569 + 0.027
0.256 + 0.011
Explicit Ensemble
85.8 + 0.2
7..6+004
0.105
0.002
0.536
+ 0.007
0.218 0.002
LoRA- EeAAmbes
88.0 0.2
78.3 0.6
0.037 + 0.002
0.342 0.003
0.175 0.002
C Hyperparameter Selection and Sensitivity Analysis: LoRA Rank
The main hyper- prrec-parameter introduced by adding LoRA is the rank of the low-rank docomposition (i.c., the common dimension of the matrices A and B). Varying that rank modulates the complexity of the model for the learning task. We have empirically studied the relationship between rank, accuracy, and Expected Calibration Error. Here we show results for HAM10000 and CIFAR-100 dataset.
With increasing rank the classification accuracy increases while the calibration deteriorates, in other words, one can to some degree balance predictive accuracy against uncertainty calibration by choosing the rank. Our focus in this work is on model calibration. We therefore generally choose the rank to favor calibration, even at the cost of slightly lower classification accuracy.
On HAM10000 observe clear trade-off between accuracy and calibration, Fig. 1la
For the CIFAR-100 dataset, our evaluation of LoRA-Ensemble shows both increased accuracy and improved calibration with increasing rank within the studied range. These findings are illustrated in Fig. 11b
This observation aligns with the findings of Rahaman & Thiery 2020 as LoRA-Ensemble contimues to exhibit mnder-confidence even at higher ranks. Increasing model complexity enhances confidence, thereby improving calibration. However, at rank 32, the calibration of a single network augmented with LoRA begins to deteriorate, suggesting that a critical boundary has been reached. Beyond this point, the parameter space becomes insufficiently constrained, leading to effects similar to those observed by Guo et al. (2017).
At higher ranks, accuracy plateaus while memory demand increases linearly with O(d) and ((k) for A E Rexd and B E Rexr respectively, where d and k are the dimensions of the pre-trained weight matrix Wo E Rexd Consequently, we selected rank 8 for our CIFAR-100 experiments.
Overall, the rank serves as the primary control of LoRA's expressive capacity. While larger values tend to improve performance on more complex datascts (c.g., rank 64 for iNaturalist), excessively large choices (e.g. 2 256) suppress the distinctive dimension-learning behavior of LoRA-Ensemble resulting not ouly diminishing returns but in some cases an actual decline in accuracy. In practice, we find that small sweep over (4, 8, 16, 32, 64) on held-out set is typically sufficient to identify near-optimal rank.
Practical guidelines for choosing the LoRA rank. Based on the sensitivity analyses, we propose the following practical recommendations for selecting the LoRA rank in LoRA-Fnsemble:
Default choice. For small- medium-sized datasets and modest domain shifts, low ranks (e.g.. r E (4...8) provide favorable trade-off between accuracy, ensemble diversity, and calibration.
27

---

Published in Transactions on Machine Learning Research (02/2026)
Accuracy
Expected Calibration Error
0.10
88
Expected Calibration Error
0.08
87
Accuracy
86
85
0.04
members
1 members
members
2 members
84
members
4 members
members
0.02
8 members
83
members
members
1
2
4
16
1
4
8
16
LoRA Rank
LoRA Rank
(a) HAM10000.
Accuracy
Expected Calibration Error
members
83
0.06
2 members
Expected Calibration Error
4 members
8 members
82
0.05
16 members
Accuracy
81
0.04
80
1 members members
79
0.02
members
8 members
78
16 members
12
8
16
32
12
4
8
16
32
LoRA Rank
LoRA Rank
(b) CIFAR-100
Figure 11: Impact of LoRA rank accuracy and ECF.
When to increase the rank. Higher ranks may be needed for large or fine-grained datasots (c.g.. NNaturalist), particularly when accuracy saturates at low ranks and calibration remains stable.
Avoid excessive ranks. Increasing the rank too far risk degrading the calibration and tends to reduce the diversity of ensemble members, the characteristic LoRA learning dynamics are gradually lost.
Monitoring calibration. When incrcasing the rank, calibration metrics such ECE should be monitored alongside accuracy, improvements in predictive performance do not necessarily translate into better-calibrated uncertainty estimates.
28

---

Published in Transactions on Machine Learning Research (02/2026)
Explicit
LoltA
Explici
LoRtA
Nember Sighr Ve0000
2 C
Nember Contommat
Nember Wettrn
120 Werter
516
2
Nonoer Seguir Vodars
Nomber Eingu Vitten
(a) cos-similarity high-ranking singular vectors
(b) weight-space cosine similarity
(c) training trajectories
Figure 12: Weight space analysis of
vs. Explicit Ensemble: The first row represents key
matrices, while the second row represents query matrices.
Weight Space Analysis: LoRA-Ensemble versus Explicit Ensemble
This section expands on Sec. which examines the diversity of ensemble members in function and weight space for EeRAmeeseeble and Explicit Ensemble, showing that both spaces. While Sec. focuses on value projection matrices due to their role in learned representations, this section examines query and key projection matrices, too. In Fig. 12 observe that LoRA-Ensemble achieves greater diversity in query and key projection matrices, similar the diversity observed in value projection matrices (Fig.
exhibits greater diversity
Using Singular Value Decomposition (SVD), a weight matrix W E mmxx is decomposed as:
W = UEVT
where U E Rmxm and V E Raxn orthonormal matrices representing rotational components, and E E Raxn is diagonal matrix of singular values capturing the scaling effect. Singular vectors linked to larger singular values highlight key transformations encoded by W.
In Fig. 13 we analyze the differences in weight updates between ensemble methods by computing the Singular Value Decomposition (SVD) of pre-trained and trained weights for ensemble members. Singular vectors corresponding to the lop singular values (16 are shown) are extracted and compared using cosine similarity to evaluate changes in the weight structure. These similarities are averaged across layers and ensemble members. The rosults highlight distinct parameter update patterns between RoR--ensemble and Explicit Ensemble. LoRA- Ensemble introduces new high-ranking singular vectors, referred to as 'intruder dimensions" (Shuttleworth et al., 2024), which are nearly orthogonal to the singular vectors of the pre-trained weights. The number of intruder dimensions depends on the LoRA rank. This effect is particularly pronounced in the value projection matrices, which aligns with their strong association with learned representations. contrast, Explicit Ensemble members tend to preserve structure closely aligned with the spectral properties of the pre-trained weights. This alignment is espocially evident in the key and query projection matrices, which exhibit strong resemblance to the original spectral structure.
We further analyze the B- A matrices learned by different ensemble members. Due to their random initial- ization, these matricos explore diverse directions in weight space. In Fig. 14 we plot the largest eigenvalues of these matrices (with only four non-zero eigenvalues as the LoRA rank is set to 4) and the similarity between the corresponding eigenvectors across ensemble members. The similarities are averaged over lay- ers and member pairs. The results show that while the cigenvalues across members follow a similar trend, the eigenvectors are largely uncorrelated. This indicates that ensemble members explore different regions of weight space while maintaining similar overall transformations. The shared eigenvalue trends suggest
29

---

Published in Transactions on Machine Learning Research (02/2026)
Explicit
LoRA
Singular Values
Ensembie
Explicit Ensemole W. Init
Average Singular Values
lnit Singular Wectorr
VALUE
Singuoo
16
9 10 31 13-36-1533
Explicit Ensemble Singular Vectors
Lora-Fosemble Singular Vertors
Index
10
Ensembie
Explict Ensemale Init
Average Singular Values
Ini Singular Vectors
Singigge
KEY
16
0 30 12 34
Explicit Ensemble Singular Vectors
Lora-Ensembbee Singular Vectors
Index
10
Ensembie Explicit Ensemble
Average Singular Values
nit Enguller Vectors
niguler
QUERY
8 30 12 14 Index
Explicit Ensemble Singular Vectors
Lora-Ensemblee Singular Vectors
Figure Cosine similarity of top singular vectors (and associated singular values) between initial pre- trained and final trained weights, averaged over layers and ensemble members.
consistent semantic contributions across members, while the dissimilar eigenvectors highlight the diversity in their learned representations.
We plot the t-SNE visualizations for different layers in Fig. 15 capturing the evolution of weights during training. The visualizations include the initial pretrained weights, and for each ensemble member, we plot weights from epoch 5 to epoch 65 at 5-epoch intervals. The plots reveal that LoRA-Ensemble members exhibit broader convergence across the loss landscape in various layers, signifying diverse learning dynamics. Converscly, Explicit Ensemble members tend to remain closer their initial weights, indicating reducod diversity throughout the training process.
E
Correlation Analysis Between Ensemble Diversity and Predictive Performance
Prior work has shown that diversity across modes weight space correlates with improved uncertainty estimates (ort al.) 2019b Izmailov et al., 22221). To investigate this phenomenon in the context of LoRA, trained eight-member ensembles the HAM10000 dataset using varying LoRA initialization gains.
30

---

Published in Transactions on Machine Learning Research (02/2026)
VALUE
KEY
QUERY
Member (i<j) Singula Vectors
1.0
Member (i<j) Singular Vectors
1.0
Member (1<]] Singular Vectors
1.0
0.6
0.6
0.6
0,4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
1
2
3
1
2
3
1
2
3
Member Singular Vectors
Member Singular Vectors
Member Singular Vectors
Hember
Pemser3
Menter
Homber
Nember
Average Singular Value
Member
Hember 12
Membor
Member 12
Average Singular Value
Harber
Member
Perbber
6.0
Perbbers
Memter
Nember
Nember
Weekerx
Hember
Average Singular
Wewker
wwdeer
Heettert
5.0
3.0
3.0
2
2
Index
Index
Index
Figure 14: Cosine similarity of top singular vectors from B low-rank matrices (rank set to 4) between LoRA-Ensemble members, averaged over layers and all member pairs (first row), along with corresponding average singular values for different members (second row).
(layer 2. V pro.)
(layer 9, v proj)
5-
t-SNE Dimension 2
t-SNC Dimension 2
2447
-6
2
3
L-SNE Dimension
1
L-SNE Dimension
Figure 15: Training trajectories of ensemble members of KoRA--nsemble and Explicit Ensemble.
We treat the LoRA initialization gain as a simple diversily knob: larger gains induce greater spread among the low-rank adapters. To quantify ensemble diversity, we compute the average pairwise correlation between the LoRA V projection matrices of different ensemble members, averaged across all layers. The diversity score is then defined as
Diversity
average correlation.
31

---

Published in Transactions on Machine Learning Research (02/2026)
This metric captures how dissimilar the ensemble members in terms of their learned parameters, with higher scores indicating less correlated (i.e., more diverse) adapters.
Tab. 11 summariies the results. Both performance and calibration improve as diversity increases, to a point. Beyond a certain threshold, additional diversity provides diminishing returns and eventually plateaus. Moreover, as shown in Fig. 14 excessively large gains degrade performance. From a Bayesian perspective, moderate gains behave like well-chosen prior variance, encouraging the ensemble explore distinct poste- rior modes. Excessively large gains, however, make the initial weights too diffuse, which cause activation saturation, unstable gradients, suboptimal convergence, and ultimately higher variance and miscalibration. A sweep of gain values reveals a U-shaped calibration curve: uncertainty estimates improve with increasing diversity until diversity becomes excessive, at which point performance and calibration degrade.
Gain
Diversity Score
Accuracy
NLL
1
0.006
0.854
0.405
2
0.017
0.856
0.393
4
0.048
0.860
0.374
8
0.117
0.877
0.354
12
0.184
0.881
0.345
20
0.304
0.882
0.345
40
0.511
0.882
0.344
Table 11: Effect of LoRA initialization gain on ensemble diversity, accuracy, and calibration (measured by NLL) HAM10000.
F Joint Training of Backbone and LoRA- Layers on iNaturalist
LoRA-Ensemble keeps the backbone weights frozen and trains only the low-rank matrices. To assess the impact of relaxing this constraint, we evaluated the method in a setting where the full backbone is also trainable. Results on the iNat 2017 dataset, see Tab. 12 show a substantial drop both accuracy and calibration when the entire network is trained, although performance still surpasses that of a single model.
We hypothesize that enabling backbone training washes out the low-rank adjustments introduced by LoRA. With frozen backbone, each model's unique low-rank matrices create intruder dimensions that yield diverse feature spaces. See Appendix D for more details. Once the backbone is trainable, those adjustments merge into the dominant spectral modes, causing all ensemble members to collapse into the same parameter region and behave like single network. Similar behavior was observed for the Batch--Ensemle method, as discussed in Appendix R.3
Table 12: Performance on the iNat 2017 dataset. 'full' indicates that the entire network, including the pre-trained backbone, is trainable. Ensembles consist of members. Best score for each metric in bold, second-best. underlined.
Method
Accuracy (t)
F1 (t)
ECE (1)
NLL (1)
Brier (4)
Single Network
42.6
37.8
0.293
1.054
0.207
Single Net w/ LoRA
47.7
43.1
0.096
0.662
0.166
Single Net w/ LoRA (full)
42.8
38.0
0.271
0.958
0.201
LoRe--nsemble
49.3
44.1
0.045
0.610
0.160
LoRA- T.RA--Eseemle (full)
44.0
39.4
0.249
0.886
0.193
32

---

Published in Transactions on Machine Learning Research (02/2026)
G Placement of GoRA-Ensemble Modules and Selection of Trainable Weights
Typically, LoRA is applied only to the weights in the multi-head attention module (i.e., the query, key, value, and output projections),
demonstrated in the original LoRA paper (Hu et al., 2021]. We acknowledge that, due to the modular nature of transformer architectures, LoRA layers can also be inserted into the feedforward MLP blocks. While this alternative placement has been shown in prior work to improve perfor- mance on certain datasets, it may also lead to reduced robustness and lower overall performance (Fomenko) et al. 2024). Moreover, the projection matrices in the MLP blocks typically have significantly higher dimen- sionality, often four times larger than those in the attention layers. As a result, this placement introduces substantial number of additional parameters, which can increase memory consumption. This effect is es- pecially pronounced when ensemble members are executed in parallel using vectorized mapping rather than sequential execution.
presents the results on the HAM10000 dataset. Notably, adding blocks leads to improved accuracy, but at the cost of poorer calibration performance. Additionally, when the embedding layers of the Vision Transformer (ViT) are also trained alongside the low-rank matrices for the purpose of patch feature extraction, we observe marked drop in classification accuracy. This performance degradation can be attributed the substantial number of additional parameters introduced in the early stages of the model, which are by design order of magnitude larger than in the subsoquent LoRA layers, potentially leading to suboptinal training dynamics.
Table 13
layers to the MLP
Finally, we observe that assigning a separate classification head to each LoRA-Ensenin Ensemble member yields further performance gains. However, we also note that this design choice can be omitted in favor of improved parameter elficiency, depending on the application constraints.
Table 13: Ablation Study. Investigates the placement of LoRA-Ensemble layers and additional trainable components the HAM10000 dataset. Ensembles consist of 8 members. Best score for each metric in bold, second-best underlined.
LaR---Ensemble Config-
Extra Trainable Layers
Trainable Params.
Accuracy (t)
F1
ECE (L)
NLL
Brier (L)
Multi-hes head atsention
Cls. bead
2'364'6799
87.5
77.7
0.041
0.365
0.187
Multi-head atsention
Cls. head tokenizer
4'724'743
84.8
73.5
0.025
0.422
0.217
Multi head atsention
Cla. head
5'313'799
80.9
0.077
Multi-head attention MLP
Cla. hoad 4 tokeniizer Pull backbone
7'673/863
87.4
77-1
0.083
0.438
0.192
Multi-head atsention
90'114'053
85.2
73.3
0.126
1.000
0.264
Multi head atsention
Ensemble cls. head
2'402'380
88.0
78.0
0.036
0.347
0.179
H LoRA-Ensemble for CNNs
We extend LoRA Ensemble to convolutional neural nefworks (CNN) by applying it to a ResNet-18 backbone with an ensemble of four members. We mainly follow the original Batch-Fnsemble (Wen et al. 2020 implementation. For detailed experimental settings, see (Turkoglu et al., 2022).. Table 14 reports the CIFAR- 100 results. LoRA-Ensemble achieves the second-best performance among implicit ensembling methods, behind only FiL.M-Esembll,, but it does not match its efficacy on transformer architectures. As discussed in the main text and in the Appendix R.7, this gap stems from the fundamentally diffcrent computational structures of transformers compared with MLPs and CNNs, which makes direct adaptation of techniques between these domains challenging.
Ensembes Fine- Tuned on the Same Dataset as the Backbone Model
While our study explicitly focused on transfer learning setups, we also explored how LoRA-Eusemble can be applied when the backbone is trained on the same dataset. To this end, we initialized the LoRA ensemble with weights from a single network trained for 65 epochs on HAM10000, and subsequently fino-tuned it for one epoch without learning rate warmup. Fig. 16 presents the results of fine-tuning a LosA-Ensemble with rank 2 for a single epoch. It is evident that even in this scenario, the LoRA-Fasemble improves
33

---

Published in Transactions on Machine Learning Research (02/2026)
Table 14: Performance on the CIFAR-100 dataset for CNN architecture. Ensembles have 4 members and Resnct-18 is used as a backbone. For implicit ensemble methods, the best score for each metric in bold, second-best underlined.
Method
Accuracy (t)
ECE (4)
Single Network
78.0 + 0.4
0.046 + 0.001
Deep Ensemble
81.6 + 0.3
0.041 + 0.002
MC-Dropout
75.5 0.6
0.064 + 0.003
MTMO
48.0
0.083 + 0.017
Masksemble
0.5
0.075 + 0.004
Eeeembbee
0.2
0.038 + 0.000
0.1
0.052 + 0.002
LoRA- eeaeebeee
0.2
0.048 1 0.001
both performance and calibration with minimal computational overhead. We also highlight that alternative methods, such as explicit ensembling, are not directly applicable in this context.
Table 15:
LoRA-Ensemble performance when it is fine-tuned on a pre-trained dataset. The HAM10000
dataset is used, and the ensemble consists of 8 members. The backbone is identical to that of the Single Network, which is fine-tuned for epoch. Best score for each metric in bold.
Method
Accuracy (t)
F1 (t)
ECE (1)
NLL (1)
Brier (1)
Single Network
84.1 + 0.3
71.4 0.7
0.139 +: 0.004
1,138 + 0.040
0.291 + 0.009
Single Net w/ LoRA
83.2 + 0.7
70.7 1.3
0.085 0.004
0.569 0.027
0.256 + 0.011
LoRA-Ensemble (finetuned for 1 epoch)
84.8
72.2
0.059
0.514
0.238
J Post-Hoc Temperature Scaling for Model Calibration
Temperature scaling is simple yet effective post-hoc calibration method used to improve the confidence of probabilistic models (Guo et al. 2017). It rescales the logits of a trained model by a scalar parameter T > 0 (the temperature). Given logits 2, the calibrated probabilities Ps for class i are computed as:
cxp(2./T)
P. Σ, exp(2;/T))
(5)
Here, T corresponds no scaling, and T' > 1 reduces overconfidence by softening the logits.
To assess the impact of temperature scaling calibration, conducted experiments CIFAR-100 with varying temperature values, as shown in Tab. 16 For each method, the model parameters were fixed, and the effect of different temperatures on calibration was evaluated. We observe that calibration can be improved across all methods, with the exception of the single network with LoRA, which does not. require temperature scaling.
As discussed in Section 3.4 Lose-Ense under-confident on CIFAR-100, as evidencod by the optimal lemperature being less than 1 for this method.
When should temperature scaling be applied?
Use it for under-confidencee Temperature scaling a good default post-hoc correction when
predictions are systematically under-confident (accuracy > confidence across bins).
Benefit is datasct-specific. The need for temperature scaling (and the magnitude of improvement) depends on the dataset and training regime; in some cases calibrated, while in others residual miscalibration remains.
already close
34

---

Published in Transactions on Machine Learning Research (02/2026)
Decision rule. Evaluate calibration held-out validation set; apply temperature scaling only if it measurably improves calibration metrics (e.g., ECE/NILI) without harming accuracy.
Table 16: Model performance on the CIFAR-100 dataset with different temperature. Bost score for cach metric and method in bold, second-best underlined.
Method
Temp.
Accuracy (1) F1 (1)
ECE (1)
NLL (1)
Brier (1)
Single Network
1.4
76.8
76.7
0.091
0.969
0.344
Single Network
1.6
0.061
0.928
0.334
Single Network
1.8
0.034
0.920
0.329
Single Network
2.0
0.029
0.939
0.329
Single Network
2.2
0.078
0.982
0.335
Single Net w/ LoRA
0.4
79.2
79.1
0.130
1.020
0.332
Single Net w/ LoRA
0.6
0.088
0.772
0.308
Single Net w/ LoRA
0.8
0.042
0.688
0.294
Single Net w/ LoRA
1.0
0.013
0.680
0.290
Single Net w/ LoRA
1.2
0.073
0.722
0.298
MC Dropout
0.4
76.6
76.6
0.203
1.554
0.372
MC Dropout
0.6
0.174
1.223
0.361
MC Dropout
0.8
0.111
1.114
0.344
MC Dropout
1.0
0.057
1.163
0.342
MC Dropout
1.2
0.175
1.333
0.393
Explicit Ensemble
1.0
79.8
79.9
0.100
0.744
0.285
Explicit Ensemble
1.2
0.072
0.719
0.282
Explicit Ensemble
1.4
0.041
0.718
0.281
Explicit Ensemble
1.6
0.019
0.737
0.284
Explicit Ensemble
1.8
0.046
0.777
0.290
LoRA-Fasemblle
0.4
82.4
82.4
0.103
0.628
0.252
MoRA-Ensemble
0.6
0.063
0.565
0.247
0.8
0.018
0.557
0.247
DORA-Ensemmi
1.0
0.034
0.587
0.253
A-Ensmbble
1.2
0.095
0.650
0.269
K Implementation of LoRA-Ensmeble
In practice, our LoRA-Ensemble is implemented by replacing the respective linear layers (W,, Wk. W., and W.) in the pre-trained model architecture with custom LoRA modules.
As backbone for experiments with image datasets, we employ a Vision Transformer (ViT) model (Doso- vitskiy et al.] 2020). The chosen architecture is the base variant with patch size 32 x 32 as defined in (Dosovitskiy et al. 2020 We load the weights from torchvision, which were trained on ImageNet- 1k DDng et al. 2209). using variant of the training recipe from (Touvron al. 2020 for dctails refer their documentaatin.
The forward pass through the backbone is parallelized by replicating the input along the batch dimension. In each LoRA module, the data is split into separate inputs per member and passed to the respective member with the help of a vectorized map, which allows a parallelized forward pass even through the LoRA modules. The outputs are then again stacked along the batch dimension. In this way, one makes efficient use of the parallclization on GPU, while the same time avoiding loading the pre-trained backbone into memory multiple times. As backbone for audio experiments, we use the Audio Spectrogram Transformer (AST) backbone (Gong al. 2021). That architecture inspired by VIT (more specifically the data-efficient
35

---

Published in Transactions on Machine Learning Research (02/2026)
version of ViT akin to DeiT (Tovvron et al. 2020)) but is designed specifically for audio spectrograms. Following GGong et al. 2021 we initialize the audio model weights by transferring and appropriately in- terpolating them from ImageNet pre-training. See Appendix [N and [ for details. As the AST version of LoRA-Ensemble would run into memory limits, we introduce chunking. While the forward pass through the backbone is still parallelized, the LoRA modules are called sequentially
Finally, the pre-trained model does not have the correct output dimension for our prediction tasks (i.e., it was trained for different mumber of classes). Therefore entirely discard its last layer and add new one with the correct dimensions, which we train from scratch. Obviously, the weights of that last layer are different for every ensemble member. We parallclize it in the same way as the LoRA module described above.
L Training Details of LoRA-Ensemble
The CIFAR-10/100, HAM10000, and iNaturalist 2017 dataset experiments are based the ViT-Base-32 architecture (Dosovitskiy et al., 2020). This model has 12 layers and uses 766-dimennionall patch embeddings, and the multi-head attention modules have 12 heads. All Vision Translormer models for image classification are trained using the AdamW optimizer (Loshchilov & Hutter 2017), except for iNat 2017, which is trained with SGD using a momentum of 0.9. The base learning rate is initially set to 0.0001 with a batch size of 32 for all experiments, except. for iNat 2017, where a learning rate of 0.1 is used with a batch size of 128. Training employs a learning rate warm-up of 500 steps for all experiments, except for iNat 2017, which uses 2500 warm-up steps. During the warm-up phase, the learning rate increases linearly from 0 to the base value, after which it follows cosine decay schedule for the remaining steps. For iNat 2017, an exponential learning rate decay with factor of 0.94 is applied every 4 epochs. During the experiments, the gradients were calculated and then clipped not to exceed a maximum norm of 1. In the case of HAM10000, we used a weighted cross-entropy loss that considered the estimated effective number of samples, which was determined using& a beta parameter of 0.9991 (Cui et al. 2019). , Uniform class weights were used for all other datasets. The maximum number of training epochs varies depending on the datasct. For CIFAR-10/100 1000, the model is trained for 16 epochs (just over 25,000 steps), while for HAM10000 and iNat 2017, it is trained for 65 and 64 epochs, respectively. Overall, the hyperparameters used in this work were loosely based on (Conrad, 2023). The models were trained using pre-trained weights from torchvision 0.17.1 1 an NVIDIA Tesla A100 graphics card. Morcover, the LoRA models are configured with rank of 8 for CIFAR-10/ 100, 4 for HAM10000, and 64 for iNat 2017. For Monte Carlo Dropout the dropout rate was empirically set to be 0.2. Refer to Appendix R.I for details.
The settings used for the ESC-50 dataset training are similar to those used in (Gong et al. 2021 However, we used a batch size of 1 instead of 48 to enable training on single GPU. The base learning rate is set to 0.00001 for the Explicit Ensemble as well as MC Dropout experiments and 0.00005 for MoRR-EEnemblee These learning rates are lower than the ones used in (Gong et al. 2021), which is due to the smaller batch size. Refer to the Appendix P for more details. The LoRA models were implemented with rank of 16. The dropout rate for MC dropout was kept at 0.2.
For language experiments on the SST-2 dataset (Socher et al. 2013) we used the BERT base uncased model (Devlin et al.) 2019), loaded via the HuggingFace Transformers library (Wolf et al. 2020). Training utilizes the AdamW optimizer (Loshchilov & Hutter 2017) with 31 0.9 and .32 = 0.999, a linearly decaying learning rate over three epochs, and a batch size of 16. These settings were informed by prior work that used BERT on SST2 (Chhablani 2023). We conduct a separate hyperparameter tuning for each method and select the learning rate from the candidate set {2 x 10-6, 7 x 10-6, 9 x 10-5, 2 x 10-5, 3 x 10-5, 5 x 10-5, 7 x 10-53 that yiclds the highest accuracy. For MC Dropout, we used a dropout rate of 0.2. For LoRA-Ensemble, we set the rank to 64 and initialize the LoRA layers using Xavier uniform initialization GGlort & Bengio 2010
with a gain factor of 10.
As Fort et al. (2019a) have shown, varying initializations of the weights are most important to getting diverse ensemble members. For this reason, various initialization methods and corresponding parameters were tried,
For the Explicit Ensemble the vectorization could not be used on GPU, due to technical issue with the VIT implementation in Poroh..
36

---

Published in Transactions on Machine Learning Research (02/2026)
with Xavier uniform initialization (Glorot & Bengio, 2010 with gain 10, giving the best combination of accuracy and calibration. For iNat 2017, gain value of is used. For more information, refer to Appendix M This setting is kept for models across all datasets, including the one with an AST backbone.
For the same reason, we investigated whether adding noise to the pre-trained parameters an Explicit Ensemble increases its performance through a higher diversity of members. However, the results did not show any additional benefits beyond what the randomly initialized last layer already provided, hence we did not use that option. For more details, refer to Appendix R.2
M Initialization of LoRA-Ensemble Parameters
Randomness in initialization is key driver of diversity among ensemble members (Fort et al 2019a Therefore, finding the right balance between diversity and overly disrupting parameters is crucial. Hu et al. (2021] propose using a random Gaussian initialization for A while sctting B to zero. This approach results in AW = BA being zero at the start of training. In our experiments, we adopt this pattern by always initializing to zero while varying the parameters and methods for initializing A. Following the method outlined by Hu al. (2021), our initial experiments concentrated on the Gaussian initialization of A, with mean / and varying standard deviations. Additionally, we lested the Xavier uniform initialization conducted the CIFAR-100 dataset and subsequently applied to other experiments. We compared results in terms of accuracy and Expected Calibration Error.
Glorot & Bengio 2010) using different values for the gain. All tests
Table 17: Accuracy and Expected Calibration Error for different initialization methods and varying distri- bution parameters for Eenembeeem
Init. Type
Std.
Gain
Accuracy (t)
ECE (1)
0.02
81.2
0.041
0.05
81.4
0.037
0..
81.7
0.035
Gaussian
0.2
82.1
0.034
0.5
82.6
0.036
1
82.5
0.039
2
81.7
0.046
1
81.5
0.039
5
82.2
0.034
10
82.4
0.034
Xavier Uniform
15
82.6
0.037
20
82.4
0.038
30
82.2
0.043
In Tab. 17 the results are quantitatively presented. is immodiately evident that both techniques and all lested parameters perform similarly. While more specialized models may surpass our results in terms of ac- curacy, our primary focus is calibration, with the goal of maintaining comparable predictive performance. Visual inspection of the results in Fig. 16 confirms the high similarity among all results. Choosing small calibration error while maintaining high accuracy as a decision criterion, both Gaussian initialization with a standard deviation 0.5 and Xavier uniform initialization with a gain of 10 or 15 are viable candidates. Since a gain of combines high accuracy with the lowest Expected Calibration Error, we select Xavier uniform initialization with a gain of 10 for our experiments.
N AST Implementation
A different backbone used for the experiment on the audio dataset. Specifically, we use the Audio Spec- trogram Transformer (AST) following the implementation of Gong al. (202)). with slight modifications
37

---

Published in Transactions on Machine Learning Research (02/2026)
Accuracy
Expected Calibration Error
0.045
Expected Calibration Error
0.040
81
0.035
Accuracy
0.030
0.025
79
members
0.020
1 members
members
2 members 4 members
4 members
0.015
8 members
8 members
16 members
0.010
members
0.02
0.05
0.1
0.2
0.5
1
2
0.02
0.05
0.1
0.2
0.5
Standard deviation
Standard deviation
(a) Gaussian initialization with varying standard deviation.
Accuracy
Expected Calibration Error
0.040
82
Expected Calibration Error
0.035
81
Accuracy
0.030
0.025
80
0.020
1 members
1 members members
79
members
0.015
members
4 members
8 members
8 members
0.010
16 members
members
5
10
15
20
30
15 Gain
20
30
Gain
(b) Xavier uniform initialization with varying gain
Figure 16: Accuracy and Expected Calibration Error for different. initialization methods and varying distri- bution parameters across different ensemble sizes for LoRA-Ensemble.
fit our general architecture. Appendix O demonstrates the equivalence of our implementation. In their experiments, Gong et al. (021)) used two different types of pre-trained weights: pro-trained large image dalasel and the other on audio dataset. For our research, transformer model known dataset (Deng et al. 2009), to the original AST architecture by Gong et al. (2021). The model has 12 layers, uses 76s-dimension patch embeddings, and the multi- head attention modules have 12 heads. This task is considered more challenging than using models pre-trained on audio da datasets.
transfer the weights of vision DeiT (Touvron et al.) 2020).. which has been pre-trained on the ImageNet
O Validation of AST Implementation
The Audio Spectrogram Transformer (AST) model provided by Gong al. (2021 was copied without any changes. However, the training and evaluation pipeline adapted to fit our architecture. Correspondingly,
38

---

Published in Transactions on Machine Learning Research (02/2026)
it essential to validate the equivalence of our implementation by training single AST on the ESC-50 dataset. The results of our model should closely match those provided in Gong et al. (2021).
They offer two sets of pre-trained weights: one where the weights of a Vision Transformer pre-trained on ImageNet (Deng et al. 2009) are transferred to AST, and another where the AST was pre-trained on AudioSet (Gemmeke et al., 2017 To verify our implementation, we ran using the settings provided by Gong et al.) (2021) and compared the results, which are summarized in Tab. 18 The results for both pre praitrining modes fall within the uncertainty range provided by [Gong et al. 2021
This suggests that our pipeline yields
comparable outcomes, validating our implementation for continued use.
Table 18: Comparison of the results obtained for the AST given in Gong et al. (2021) and those obtained by implementation. AST-S refers to the AST pre-trained ImageNet, and AST-P refers to the AudioSet pre-training. Both results fall within the uncertainly range provided by Gong et al. (021).
Model
Accuracy (Gong et al. 2021)
Accuracy (our implementation)
AST-S
88.7 + 0.7
88.0
AST-P
95.6 0.4
95.8
P Hyper-parameter Tuning for AST Experiment
The original training settings of the AST-S model in [Gong et al. 2021 utilize batch size of 48. However, due to the memory constraint of single GPU training on an NVIDIA Tesla A100 with 80 GB memory, replicating a batch size of 48 as the original publication was infeasible for training an Explicit AST-S Ensemble with members. Consequently, we perform minimal lyper-parameter tuning by employing a batch size of for both the explicit AST-S and the LoRA AST-S model, exploring various learning rates. Apart from batch size and learning rate adjustments, all other settings remain consistent with (Gong et al. 2021
The hyper- ppper-parameter tuning results for the explicit model using a batch size of 1, as shown in Tab. 19 demonstrate performance similar to the original implementation with a batch size of 48, allowing for a fair comparison with our method Gong et al. 2021). Additionally, Tab. 20 showcases the outcomes tuning the learning rate for our LoRA AST-S model.
Single model 5-Fold cross-validation results of AST-S on ESC-50 sound dataset with different learning rates and batch size 1. The model settings selected based on accuracy for the experiments are highlighted.
Table 19:
Model
Learning rate Accuracy (t)
ECE (L)
AST-S
0.00001
88.2
0.0553
AST-S
0.00005
81.7
0.0933
Table 20: Single model 5-Fold cross-valides voos-valiootion results for our LoRA AST-S implementation on ESC-50 sound dataset with different learning rales and batch size 1. The model settings selected based on accuracy for the experiments are highlighted.
Model
Learning rate
Accuracy (1)
ECE (L)
LoRA AST-S
0.00001
85.6
0.0447
LoRA AST-S
0.00005
87.9
0.0487
LoRA AST-S
0.0001
84.7
0.0501
LoRA AST-S
0.0005
24.1
0.0291
LoRA AST-S
0.001
11.8
0.0295
39

---

Published in Transactions on Machine Learning Research (02/2026)
Table 21: Parameter counts and computation times for an Explicit Ensemble of AST models and the corresponding LoRm-ensemble. Training time is the average duration for one epoch on ESC-50, with batch size 1. Inference time is the average duration of a forward pass, with batch size 1.
Method
Parameter overhead Training time [s] Inference time [ms]
Explicit Ensemble LoRA-Ennsnnn
8 x 87M
517
8 x 7.3
1.08 87M
348
73.9
Computational Cost for AST Models
Similarly to the way we did for the Vision Transformer models, we estimate the required resources for AST models. The resource needs are presented in Tab. 21 The nunber of parameters is reported for an ensemble of 8 members, with the A and B matrices in models using LoRA having rank of 16. Training and inference times were measured on single NVIDIA Tesla A100-80GB GPU, with batch size of 1. Training time is given as the average wall clock time per training epoch while training on ESC-50, with 8 ensemble members. Inference time is reported as the average time for a single forward pass of an ESC-50 sample with a batch size of 1.
As mentioned Appendix K the Explicit Ensemble processes the members sequentially, while LoRA- Ensemble is parallelized. However, fully parallelizing the training of AST models causes memory issues, chunking was introduced. Thus, in LoRA-Ensemble models, the pass through the backbone runs in parallel, while LoRA modules are called sequentially. This also explains the significantly higher inference time compared to the results in Sec. A.9. Additionally, the one-time delay incurred by PyTorch's umap function causes LoRA-Ensemble to be slightly slower at inference time.
R Baselines
R.1 Hyperparameter Tuning for MC Dropout
We conducted an analysis to determine the impact of dropout probability on the accuracy and calibration of the ViT with Monte Carlo dropout. Fig. 17 displays the accuracy and ECE scores for various dropout probabilities. The experiment carried out on the HAM10000 dataset with members. Our findings show that dropout probability of 0.2 offers good balance between accuracy and calibration.
R.2 Initialization of Explicit Ensemble Parameters
A pre-trained Vision Transformer model is the backbone for our computer vision experiments. Correspond- ingly, the parameters of all members in an Explicit Ensemble are initialized to the same values across members. Initialization a primary driver of diversity in ensemble members (Fort et al., 2019a). Hence, it is crucial to study the effect of noise in the parameter initialization on the calibration of the resulting ensemble. In the case of pre-trained model weights not having been trained on dataset with the same number of classes, the last layer of all models is replaced completely. This means that regardless of the ensemble technique used, the weights of the last layer, which is responsible for classification, will vary across members. This variation in the weights of the classification layer is expected to contribute significantly to the diversity of the members. Nonetheless, we studied the impact of adding noise to the parameters of an Explicit Ensemble. This was done using the following formula:
W Wnew- = W + a- dW,
(6)
where dW ~ (0, ow). Here a is a scale factor to control the amount of noise and aw is the standard deviation of the parameters within a weight matrix. This was applied to all weight matrices separately.
It is expected that the initial layers of neural network will learn basic features, while the later layers will include dataset-specifie properties. Therefore, it is assumed that adding noise to the later layers would
40

---

Published in Transactions on Machine Learning Research (02/2026)
Accuracy
Expected Calibration Error
84.0
0.104
Dropout
Dropout
Expected Calibration Error
83.5
0.100
Accuracy
83.0
0.096
82.5
82.0
0.092
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Dropout Probability
Dropout Probability
Figure 17:
Accuracy and Expected Calibration Error for different dropout probabilities methods for MC Dropout on HAM10000 dataset.
increase diversity while maintaining pre-training. However, adding noise to the earlier layers might disrupt pre-training more significantly, especially with smaller datasets, as these parameters may not converge to meaningful values again. To address this, an experiment was set up where noise was added only the last encoder layers of the model, increasing the number of affected encoder layers gradually. Additionally, tried, ranging from to 0.0001. In the prosented experiment, the last classification layer is initialized using PyTorch's 's default method for linear layers. At the time of writing it is as follows:
several different noise scales
3
3
Winit Unif
-V5.
V5.
(7)
fan in fan in
1
Bint Unif
(8)
fan_ini in' fan in
Here W specifies the weight matrix and B is the bias. Experiments are conducted on the CIFAR- 100 dataset.
R.2.1 Results
The most important metrics for this section are accuracy and Expected Calibration Error. The results for adding noise the last layer up the last five layers are summarized in Fig, [8] Fig. depiccss the results for single model, while Fig. 18b shows the results for ensemble of 16 members.
It is evident that none of the experiments surpass the baseline of not using any additional noise beyond the random initialization of the last classification layer. After the last five layers, the results become uninteresting, as they do not vary significantly from those shown in the plots. Therefore, the presentation is truncated at five layers. Based on the presented results, no additional noise is injected into the Explicit Ensemble, and only the last layer initialization varied.
R.3 Batch-Ensemble
R.3.1
Implementation
Probably the closest method LoRA-Enscmble Batch-Ensemble introduced Wen al. (2020). Batch- Ensemble was originally developed for MLPs, but we extend self-attention networks an implicit ensemble baseline. The methodology draws inspiration from development of LoRA-Fnsemble, the
41

---

Published in Transactions on Machine Learning Research (02/2026)
Accuracy
Expected Calibration Error
0.300
Noise last layer only
0.75
0.275
Noisc in last layers
Expected Calibration Error
Noisc in last layers
0.70
0.250
Noise last layers
Noise last layers
Accuracy
0.65
0.225
Baseline (no noise)
0.60
0.200
Noise last layer only
0.55
Noise in last layers
0.175
Noise in last layers
0.50
Noise in last layers
0.150
Noise last layers
0.45
Baseline (no noise)
0.125
10
10
10-1
100
10
10
10-1
100
Noise level
Noise level
(a) Single model
Accuracy
Expected Calibration Error
0.800
Noise in last layer only
Noise
layers Noise in last 3 layers
0.775
Expected Calibration Error
0.16
Noise in last 4 layers
0.750
Noise in last layers
Accuracy
0.725
0.14
Baseline (no noise)
0.700
0.12
0.675
Noise in last layer only
Noise in last layers
0.650
Noise last layers
0.10
Noise
layers
Noise in last layers
0.600
Baseline (no noise)
0.08
10
10
100
10
10 2
10
Noise level
Noise level
(b) 16 ensemble members
Figure 18: Accuracy and Expocted Calibration Error for different noise levels across varying numbers of layers for the Explicit Ensemble. The baseline with no noise is indicated by a dashed black line.
two implementations share many similarities. The primary difference lies in the parametrization of ensemble members. For each projection matrix (query, key, value, and output), we define one shared full-rank trainable matrix initialized with the pro-trained weights of the base network, along with two additional trainable vectors, and 8, which specific each ensemble member. The projection matrix for ensemble member is defined as:
W W Wshared r;s
(9)
where Wshared is the shared trainable matrix, and o denotes element-wise multiplication. transformer block, unique forward pass is computed for each ensemble member i:
Within each
h4 W.S.
(10)
42

---

Published in Transactions on Machine Learning Research (02/2026)
resulting in N different predictions To, (X) for given input X. The final ensemble prediction is obtained by averaging the individual predictions:
B| Σ (X).
(11)
The forward pass for the Batch-Ensemble layer with shared weights is implemented as shown in Listing Listing 1: Pytorch forward pass for Batch-Enann Ensemble layer
def forward (self, x):
nun
Forward pass for the Batch - Ensemble layer *un
# Step 1: Compute the ensemble member-spectie weights
I self . weight Shape: [1, dim]
Shape: [out_dim, 1]
W_rs s eser
Shape: [out_dim, dim]
# Step 2: Combine with the shared weight combined self shared_w W_rs # Element- multiplication
Step Compute the output for specific ensemble member out x 0 W_combined. # must have shape [batch_size, dim]
return out
The r and 8 vectors are initialized from a Gaussian distribution centered around 1, specifically r, 8 ~ (11, where o² controls the variance. We empirically set 0.02. This initialization ensures that at the beginning of the training, the combined projection matrix for each ensemble member remains close the pre-trained weights of the shared matrix, preventing disruption of learned pre-trained weights. The implementation and training details followed the LoRA-Ensmoble approach; for details, refer to K and L
R.3.2 Why
Outperforms Easch-Ensembbe
Both MoRA-Ensemblle and Batch-Ensee Ensemble leverage shared weights with member-specific low-rank modifica- tions to enable efficient ensembling. The key difference lies in their parameterization: LoRA-Ensemble uses additive low-rank updates, while Batch-Ensembll applies element-wise multiplicative scaling. Despite the conceptual similarity between the two methods, Batch-Ensemble performs significantly worse than LoRA- Ensemble in both accuracy and calibration, demonstrated in Tab. D and Tab. 2 This performance gap persists even when applicd to non-solf-attentinn architectures such as convolutional neural networks, which were the original larget application of Batch-Ensemble, as shown in Tab. 14
To clarify this difference, we examine the gradients of the member-specific parameters. For Ennembe,,,,l,, the layer output is:
h4 W Wehared B,A,,
with gradients:
OC = 8. A
(12)
0B,
0L
8. B,
(13)
0A,
where 8 Dh,
For BateheEusemble, the output is:
h; (W shared ① r:8 Jar
43

---

Published in Transactions on Machine Learning Research (02/2026)
Table 22: Model performance on the CIFAR-10 dataset for the compared methods. members. Best score for each metric bold, second best underlined.
Ensembles have
Method
Accuracy (t)
F1 (t)
ECE (L)
NLL (4)
Brier (4)
Single Network
92.8
92.8
0.051
0.333
0.120
Batch- Ensembe
88.5
88.5
0.046
0.345
0.171
Batch- +
91.7
91.7
0.033
0.260
0.128
LoR AEEnsmbbee
95.6
95.6
0.003
0.133
0.067
with gradients:
ac 5. (W (Wshared Tr)))
(14)
0s;
OC
8. ". ① o x.
(15)
rr:
In Satcc--Ensemble, the gradient updates for and S; are directly scaled by the shared weights W which can constrain the learning dynamics and reduce the independence ensemble members. This scaling introduces sensitivity to the magnitude and sparsity of Wshared; potentially limiting the diversity of the ensemble.
We define Batch- + by modifying the original Batch-Ensemble algorithm, replacing the point- wise multiplication operation with an addition operation as follows:
W; = W shared + r:s."
(16)
In this case, the r and vectors are also initialized from a Gaussian distribution but centered around 0.
We compare the performance of Batch-Ensembal the original Batch-Enseeblee, and LoRA-Ensemble in Tab. 22 Batch-Ensemble+++ significantly oulperforms the original Batch-Ensemble in both accuracy and uncertainty calibration. However, its performance does not reach that of LoRA-Ensemble.
We attribute this performance gap to the following key differences between the methods:
Limited Expressiveness: Batch-Ensemble restricts its ensemble-specifi parameters to rank-1 matrices, inherently limiting the expressive power of individual ensemble members.
Coupled Learning Dynamics: In Batch-Ensemble the shared pre-trained matrix Wahared is not kept frozen. This design choice disrupt the learned pre-trained weights and may restrict the ability of the ensemble-specifie parameters and to learn sufficiently diverse set of features. A similar effect was observed in LoRA- Ensemble when the backbone was also updated during training: see the Appendix F for details.
Initialization Variations: Differences in parameter initialization may also contribute to the per- formance gap.
R.4 Snapshot Ensemble Implementation Details
Snapshot Ensemble (Huang et al.] 2017), in its pure form, consists of training a single model with eycling learning and taking snapshots every few epochs. This can make hard, however, for the model to converge anything meaningful within the low number of epochs available for training per snapshot. Therefore, Snapshot Ensemble was modified slightly, by first letting training run for a number of epochs, without any cycling of the learning rate. After this burn-in period the learning rate is at and first snapshot is taken.
44

---

Published in Transactions on Machine Learning Research (02/2026)
The remaining number of epochs is split evenly. If the remaining number of epochs is not divisible by the desired number of ensemble members, the burn-in period is extended until it is. For the HAM10000 dataset training is left at 65 epochs, with 20 burn-in epochs. For CIFAR-10 and CIFAR-100 using only 16 epochs would only leave 1 epoch per cycle for bigger models. Therefore, training is extended to 30 epochs with a burn-in period of 15 epochs. In our experiments (Tables and 2 snapshot ensembles out perforrmee a single network both calibration and accuracy, but underperformed Explicit Ensemble and LoRA- --Rn-m-e,,e,,, consistent with prior work (Lee & Lee 2025 Thuy & Benoit 2025
R.5 Last Layer Ensemble
Last Layer Ensemble is a simple baseline where multiple independent classification heads are trained on a shared frozen backbone. Diversity arises purely from different random initializations of the heads, making it a minimal-cost approach to implicit ensembling.
Given frozen pre-trained backbone with feature dimension d 768 (ViT-B/32) and M classification heads {hr) the prediction for ensemble member i is f+(x) hx(o(x)). The ensemble prediction is obtained by averaging softmax outputs across members. Each head is initialized using Gaussian distribution W; ~ (000 of I) with iiit = 0.01 and bias initialized to zero. To ensure reproducibility while maintaining diversily, head i uses random seed 42 + i.
The training objective the average cross-entropy loss across all members: L= + En= (cB(h;(o(x)) y). Since the backbonc is frozen, gradients only flow through the classification heads. We use 16 ensemble members trained with AdamW optimizer (learning rate 10-1, weight decay 0.01) for 16 epochs on CIFAR- 100 with batch size 32, 500 warmup steps, cosine learning rate schedule, and gradient clipping at 1.0.
Listing 2: Last-Tayer Ensemble
class LastLayerEnsemble (nn. (nn.Med
def __init__ (self, num sum_classes, num_membera=16, init_atd=0.011 (upprr).._itt....... initt_
self backbone = vit_b_a b___b_a (___b_32(viiiiggee_ (veights= INAGENETIK_ V v1
colf.backbont heads nn Identity
self www.bbaa..ee........ooooee.. () Frozen
7
# Create X heads with different seeds self heads nn Mo.Mouuleliis((
Linear (768, num_classes) for - in range (num_members) ])
for i, head in enumerate (self heads):
torch manual_ (42 + 1000)
18
piinit.norinn (head, weight, nn.init. zeros, (head bias)
weight, std-init_stal
def forlard x):
features = self backbone (x) # [B, 768]
return [head (features) for head self heads]
The main limitation of LLE is that all ensemnble members share identical features, Jundamentally constraining diversity to the final linear transformation. This prevents the method from expressing uncertainty about feature relevance and probably leads to heads converging to similar solutions during training.
R.6 Epistemic Neural Networks (EpiNet)
Epistemic Neural Networks (EpiNets) (Osbnnd al. 2022) estimate uncertainty by augmenting base
network with small auxiliary network (the epinet) conditioned on random epistemic index z ~ (O, Ip.). The prediction is:
So [o(x, 2) (x(x)) (x)))((())))))))))))))))))))))))
(17)
45

---

Published in Transactions on Machine Learning Research (02/2026)
where (x( is the base network output, 6ς(x) R are features from the penultimate layer, sgl-] denotes stop-gradient (preventing gradients from flowing back to the base network through the epinet path), and dy is the epinet. Following the original paper, we decompose the epinet into learnable and prior components: network providing initial diversity and a = 1.0.
o a where of is a frozen randomly-initaalin
The epinet architecture an MLP that takes the concatenation of features ø E R766 and epistemic index RD input. The MLP outputs matrix M E Dx which is then contracted with the epistemic index to produce class logits: 0, z), Σk= Mr.e. 2 = MM z. This design ensures the epinet output is linear in z, enabling smooth interpolation actoss the epistemic index space. The MLP uses two hidden layers of 256 units cach with ReLU activations.
We use D2 = 10 and M = 16 ensemble members, each with its own learnable epinet (weights initialized near zero with std 0.01) and frozen prior network (initialized with seed 42 + x 1000). The base network is a trainable ViT-B/32. Training minimizes: L == M E (cx(fo(x,z)) v) with freshly sampled per member. We use the same training setup as Last Layer Ensemble: AdamW (learning rate 10-2, weight decay 0.01), batch size 32, 16 epochs, 500 warmup steps, cosine decay, and gradient clipping at 1.0.
Listing 3: Py'Torc implementation of EpiNet.
class Epinet (nn Module):
"""Epinet: MLP outputting matrix и, contracted with z to get logits def __init__ feature_dim, num_classes, epistemic_din, hidden 2266) super ().__init__0
self epistemic_din epistemic_dim
self num_classes num_ classes
self : mn. Sequential
nn Linear (feature_dia + epistemic_din, hidden), nn. Re.Ue)
Linear (hidden, hidden), nn. ReLUeL((),
Linear (hidden, epistemic_ num_ classes)
def fsrw,rd features, z):
x = torch. cat ([features, 2], dim=-1)
[B, 768+D_2]
self mlp(x) (-ew self, epistemic dim, self classes)# [B, D_z, c]
return torch. bmm (M. (ranspose (1, 2), unsqueeze (-1)) squeeze (-1) # [B, c]
class EpiNet (nn Module):
20
def .niitt___.... (self, num _um_ classes=100, num sum_member==16 epistemic_dimes (imemm,,_dii=1,, prior_scale-1.0.
22
super (uper()..iin) ().__init__
self backbone = vit_b_32 (e. IMAGENETIK_ V1)
self backbone. heads mn mn. Linear (768, num num_ clmsses) Trainable base
self learnable_epinete = un nn. Modulelliit ([ Epinet (768, num cam_sessss, epistemic_dix) for in range (num_members))) self. prior_netsorks networeetorsss... nn. Moduleliist ([
self www.ffooeeeeiiieee (768, num_classes, epistemic_din
seed=prior_seet sicliorsseed_ttreee start 1000)
for i in range (num_ (bun_nembers))) self self.pria wwfffprior_scct scale prior_scale
33
85
def (self, x, s_samples-None"
features, jss_output self extract extract_features features (x) features_sg = features. detach))) Stop gradient
if z_samples is None:
2 z_samples =
(1en (self learnable (itenabbl__pinets)..
x. aize (0), self epistemic_din, device=x.deviee
46

---

Published in Transactions on Machine Learning Research (02/2026)
outputs for i, (learnable, prior) in enumerate(
cap(self.Learnabee.c seannaabee_eietss.._ self colf.prior_networkks
epinet_out = learnable (fattureeeee
[_)l \
self calf.prror_scaa
prior priatufeeeeee__ee 2_samples [i])
outputs. append @ooase_outpa + epinet_out) return outputs
R.7 FiLM-Ensemble and Other Implicit Baselines: Practical Challenges
Many implicit ensemble methods, such as those proposed in (Wen et al. 2020 Turkoglu et. al. 2022 Durasov et al. 2020 Havasi et al. 2020), are architecture-specin and predominantly designed for MLPs or CNNs. As a result, adapting these techniques to transformer architectures presents significant challenges, since transformers' computation structure is quite different than MLPs and CNNs.
In particular, we attempted implement FiLM-Ensemble (Turkoglu al. 2022) on a self-attention network, given the promising results reported by its authors. However, the authors themselves noted that applying FiLM-Ensemble to transformers is not straightforward, mainly because transformers rely LayerNorm, whereas FiL.M-nssmble was developed with BatchNorm in mind. Our experiments confirmed that directly using BatchNorm in transformers led to notable performance degradation. We explored several approaches to adapt LayerNorm, but the most. effective results were achieved by fixing all affine parameters for each ensemble member. This allowed for slight initial variations to introduce randomness and diversity, while keeping the variation among members minimal. The results, summarized Tab. 23 show that incrcasing the ensemble size slightly improved accuracy, though the Expected Calibration Error (ECE) fluctuated without consistent improvement. In fact, when using larger ensemble sizes, such 8 or 16, both accuracy and calibration worsened across all settings tested.
Table 23: Performance of FiLM-Ensemble for Vision Transformer (VIT) on CIFAR-10. Increasing the ensemble size slightly improves accuracy, but ECE fluctuates without showing consistent improvement.
ensemble members
Accuracy (1)
ECE (1)
1
90.54
0.0286
2
91.18
0.0269
4
91.23
0.0289
R.8
Bayesian LoRA
Bayes-LoRA Yang et al. 2024 introduces Bayesian approach on the LoRA adapter parameters by fitting Gaussian posterior around the maximum a posteriori (MAP) estimate of the fine-tuned model. In practice, this means we first obtain a standard ToRA fine-tuned network and then apply a Laplace approximation over its adapter weights. To make this tractable at scale, Bayes-LoRA relics on Kroneckor-factored ap- proximation of the Hessian, which allows efficient estimation of the posterior covariance. The result is a Bayesian model that can capture uncertainty while remaining computationally efficient compared tradi- tional Bayesian neural networks.
We evaluate Bayes-Lold LoRA on the SST-2 sentiment classification task using BERT base uncased (ocher et al. 2013 Devlin et al.) 2019) backbone. The method is applied in a post-hoc fashion after fine-tuming, The original implementation of (Yang et al., 2024) used For MAP training, we follow the standard LoRA setup with a learning rate of 5 x 10-5, training for epochs with batch size 16. The LoRA rank set 64, identical to the LoRA-Fnsemble. The prior variance is chosen as 10-3, since larger values tend degrade performance. To balance computational efficiency and uncertainty estimation, the number of posterior samples (i.e., ensemble members) is fixed at 512. For the Kroneckor-factore approximation, use Tkkfac 10.
47

---

Published in Transactions on Machine Learning Research (02/2026)
In terms of results, Bayes-LoRA falls short of LoRA-Ensemble and Explicit Ensemble methods in predic- tive performance, measured by accuracy and F1. This observation is in line with previous findings in the literature, such as (Daxberger et al.) 2021). We attribute this limitation to the reliance on a local Gaus- sian approximation around a single MAP solution, in contrast to the diversity gained through sufficiently independent ensemble members. However, the main strength of Bayes-LoRA lies in its ability to caplure pre- dictive uncertainty effectively, reaching level comparable both LoRA-Ensemble and Explicit Ensemble. Detailed results can be found in Tab. 8 in Appendix A.7
From efficiency perspective, Bayes-LoRA requires significantly more computation at inference: evaluating single test example takes roughly 250 ms (512 posterior samples are usod), compared to 22.7 ms (Tab. 9 for LoRA-Ensemble (16 members are used). This overhead makes Bayes-LoRA impractical for real-time applications but potentially valuable in settings where predictive uncertainty crucial and strict latency constraints are less relevant.
R.9 SNGP on Vision Transformers
Spectral-normmalized Neural Gaussian Process (SNGP) Qiu et al. 2020) is an efficient single-model uncer- lainty estimation method that achieves competitive calibration with deep ensembles while requiring only a single forward pass at inference. However, SNGP was originally designed and validated exclusively on residual convolutional architcctures (e.g., Widc-ResNot) (Liu ct al., 2023). and its theoretical guarantees do not extend attention mechanisms.
R.9.1 Method Overview
SNGP combines two architectural modifications to standard neural networks:
Spectral Normalization. Hidden layers are constrained to satisfy the bi-Lipschitz property by applying spectral normalization [Miyato et al. 2018 which bounds the spectral norm of weight matrices:
W
WSN
(W) c,
(18)
where (W) denotes the largest singular value and c < is the spectral norm bound (typically 0.95). This ensures that the network preserves distances in the input space, which is critical for meaningful uncertainty estimation, inputs far from the training distribution should map to representations that are correspondingly distant.
Gaussian Process Output Layer. The classification head is replaced with a Gaussian Process layer approximated via Random Fourier Features (RFF) (Rahimi & Recht 2007). For an RBF kernel, the feature map is:
<(x) V cos( b),
(19)
where W d with entries sampled from VO0, for length scale and b ~ Uniform(O, 2TT). The GP posterior is computed via Laplace approximation, maintaining precision matrix A that is updated during training:
A + A + d())d())
(20)
At inference, predictive uncertainty is obtained through mean-field approximation with factor x/8 or Monte Carlo sampling from the approximate posterior.
Why SNGP Fails on Transformers. The method's theoretical guarantees rely on the bi-Lipschitz property of residual connections in CNNs. However, Kim et al. 2021 prove that dot-product self-attention has an unbounded Lipschitz constant, violating the fundamental assumption underlying SNGP's distance- awareness. They propose an alternative L2 self-attention that satisfies the Lipschitz condition; we include this variant baseline in our HAM10000 experiments (Table
48

---

Published in Transactions on Machine Learning Research (02/2026)
R.9.2 Training Details
We implement SNGP for Vision Transformer (ViT-B/32) pretrained on ImageNet- 1K using PyTorch. Ta- ble 24 summarizes the hyperparamcters.
Table 24: SNGP-ViT hyperparameters for CIFAR-100.
Hyperparameter
Value
Backbone
Architecture
ViT-B/32
Pretrained weights
ImageNet-1K
Hidden dimension
768
Backbone trainable
Spectral Normalization
Spectral norm bound (c)
0.95
Power iterations
1
Layers normalized
Attention MLP (also tested MLP only)
GP Output Layer
Number of RFF (D)
1024
RFF imit (1/E)
0.05
Kernel scale
1.0
Input normalization
LayerNorm
Covariance momentum
1 (exact)
Ridge penalty
10 3
Mean-field factor
//8
Training
Optimizer
AdamW
Learning rate
10-4
Weight decay
0.01
Batch size
Epochs
30
Warmup steps
500
IRR schedule
Cosine decay
Gradient clipping
1.0
We applied spectral normalization to both attention and MLP layers within transformer blocks, replacing the classification head with RFF-GP layer following the original implementation guidelines (Liu et al, 2023). We also experimented with applying spectral normalization to MLP layers only, which yielded similar results.
R.9.3 Results and Discussion
Despite careful implementation, SNGP-ViT exhibited severely degraded performance on CIFAR-100: train- ing accuracy plateaued around 42% while test. accuracy reached only 32.2% with an NLL of 2.7. This is substantially worse than all other basclines, including a deterministic ViT which achieves over 76% test accuracy (Table D..
Our observations align with documented challenges in the literature. Vazhentsev et al. 2022 note that the spectral normalization is thcoretically proven to ensure the bi-Lipschitz constraint on the transformation defined by standard residual connection networks. However, the self-attention blocks in Transformers have a more complicated architecture than ResNet layers, which means the theoretical guarantees do not hold in general. Their experiments with SNGP on ELECTRA (Clark et al., (020)) demonstrate highly unstable performance; on CoNLL-2003, SNGP achieves an RCC-AUC of 56.43 compared to 6.08 for the baseline, nearly order of magnitude worse
49

---

Published in Transactions on Machine Learning Research (02/2026)
Similarly, Ulmer et al. (2022) report that SNGP significantly underperforms sequence tasks: the Danish NER task (Plank et al. 2020). SNGP achieves only 22% accuracy while LSTM achieves 93%. They observe that even with pre-trained BERT models as feature extractors, training is quite unstable. The NLP Uncertainty Zoo library likewise warns that SNGP transformer models exhibit significant training instability
S Definitions of Evaluation Metrics
We primarily evaluate our models on accuracy and Expected Calibration Error (ECE, Guo al., 2017 In addition to accuracy and Expected Calibration Error, have calculated several other scores that have been used in the context of probabilistic deep learning. In the following section, present the formulations used in our implementations.
S.1 Accuracy
Accuracy is defined as the fraction of correctly classified samples:
Acc
Σ 1 yi). N
1
(21)
Here y/ denotes the true label sample i, 9s is the predicted label of sample i, N the total number of samples, and 1(-) the indicator function.
S.2 Expected Calibration Error
The Expected Calibration Error widely used metric for measuring the calibration of neural networks. We use the definition given in (Guo et al., 2017). ECE is defined the expected difference between accuracy and confidence across several bins. In our experiments, use M = 10 equally spaced bins over the confidence interval [0, 1]. We first need to define accuracy and confidence per bin Bm as follows:
Acc(Bm)
Σ
1(0s = y)),
(22)
1ym
Se Bm
Conf(Bm))
Σ D..
(23)
1mm
B
Again, yi and ia denote the true and predicted labels of sample respectively, and P. the predicted confidence of sample i. With this the Expected Calibration Error is given as:
M
Bm Acc((m)) - Conf(Bm))
ECE =
Σ m=1
(24)
S.3 Macro F1-score
1 F1
Σ 2prri
(25)
C Pj +r₃
where Ty represents the Recall of class j, defined as ry TFFFN and pj represents the Precision of class TP j, defined P, TPFFF. and C refers to the number of classes, Here, TP, FP, and FN denote True Positives, False Positives, and False Negatives respectively.
50

---

Published in Transactions on Machine Learning Research (02/2026)
S.4 Negative Log-Likelihoode (NLL)
NLL
(yii log P(.))
N logfs
(26)
N
where denotes the number of datapoints, the number of classes, Yi. if the true label of point is and 0 otherwise and P.j is the predicted probability of sample belonging to class j.
S.5
Brier score
For Brier score we take the definition by (Brier 1950 which as follows:
BS
(P.s - 3n.)
(27)
where denotes the number of datapoints, the number classes, Yi,j if the true label point and zero herwwie and fi,j the predicted probability of sample belonging to class j.
S.6 Area Under the Receiver Operating Characteristic Curve (AUROC)
The AUROC score evaluates the performance of a binary classifier by measuring its ability to distinguish between positive and negative classes, as introduced by (Hanlcy & McNeil 1982). In our out-of-distribution (OOD) delection experiments, the positive class corresponds lo an in-distribution sample, while the negative class corresponds an out-of-distributio sample.
The AUROC is computed as the area under the ROC curve, which plots the true positive rate (TPR) against the false positive rate (FPR) across various decision thresholds. The TPR and FPR are defined as follows:
TP TPR = TP FN
(28)
FP FP TN
FPR
(29)
where TP, FP, FN, and TN represent the true positives, false positives, false negatives, and true negatives, respectively. The AUROC score is given by the following integral:
AUROC ('' (FR(F
(30)
A higher AUROC score indicates better classification performance, with a score of representing a perfect classifier, and score of 0.5 indicating performance equivalent random chance.
S.7 Area Under the Ciciiiion-Recall Curve (AUPRC)
The Area Under the Precision-Recal Curve (AUPRC) assesses the performance a binary classifier by mcasuring its ability to accurately identify positive instances, as described by (Davis & Goadrich 20066) In our cattof-distributiinn (OOD) detection experiments, the positive class corresponds to in-distribution samples, while the negative class corresponds to out-of-distributiin samples.
51

---

Published in Transactions on Machine Learning Research (02/2026)
The AUPRC is calculated as the area under the Precision-Recall (PR) curve, which plots precision against recall at various decision thresholds. Precision and recall are defined as follows:
TP
Precision
(31)
TP + FP
TP
Recall
(32)
TP FN'
where TP, FP, and FN represent true positives, false positives, and false negatives, respectively.
The AUPRC score is the integral of precision with respect to recall, expressed as:
AUPRC - S. Precision| (Recision(Reeal) dRecall.
(33)
A higher AUPRC score indicates better classifier performance in recognizing positive instances, with near 1 representing good classifier, characterized by both high recall and high precision. This metric is especially valuable for evaluating classifiers on imbalanced datasets.
S.8 False Positive Rate at 95% True Positive Rate (FPR095%TPR)
We use the false positive rate at 95% true positive rate (FPR@95TTPR) metric measures the proportion of negative samples that are incorrectly classified as positives when the true positive rate is fixed to 95%. Lower values of FPR@95%TPE indicate better performance, corresponding to fewer false positives at the same, high true positive rate.
evaluation metric. This
Formally, let be decision threshold such that
TP(
TPR(T) (P(+) + FN(T)
0.95.
(34)
The reported score is then
FP(r) FPR095%TPR = FPR(T) FP(7) + TN(?)"
(35)
52


Reviews:
[{"review_id": "yrMeE7rLrs", "reviewer": "Reviewer_GuEP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes “LoRA-Ensemble,” an ensembling method for transformer models that freezes a single pre-trained backbone and trains multiple low-rank LoRA adapters to form a distribution. The paper evaluates on several datasets in different domains/modalities and compares against baselines like MC dropout.", "claims_and_evidence_yn": "No", "claims_explanation": "This is borderline (and I am inclined to flip to yes with some tweaks). There are a few points that would strengthen the paper to ensure that claims are a bit more precise. It can be made more precise either through wording changes and/or by adding additional baselines. Here are some identified areas:\n\n+ The paper is explicitly about efficient uncertainty/ensembling for large pretrained models, but it does not cite or compare against EpiNets from Osband et al (2021), which is a directly comparable baseline trying to make similar claims to this work. This work would be more complete, and accurate, if it baselined against that work, or at the very least added a citation. Additionally, why not compare against a simple ensemble of linear heads, like EpiNets do? Is LoRa really necessary over linear heads? Some additional clarification and baselines would help address some questions that the work raises.\n+ EpiNet is not the only potential missing baseline. Suggest either drop the claim \"outperforms state-of-the-art implicit techniques\" or have more canvasing of baselines. Here are a few others that might be relevant:\n    + https://www.jmlr.org/papers/v24/22-0479.html\n    + https://proceedings.neurips.cc/paper_files/paper/2019/hash/118921efba23fc329e6560b27861f0c2-Abstract.html\n+ \"training about 2x faster and delivering over 3x faster inference\" -> For the sake of accurate discussion, there should be an additional clause here describing under what conditions it is faster (one GPU). While there is information in the appendix this sentence can be misleading. You can, of course, parallelize to be nearly as fast across many GPUs, but it seems that this is indicating speedups under a specific set of compute constraints. Memory overhead is the real benefit here that translates to speedups in low resource settings.\n+ There's a mismatch (or at least some ambiguity) about the number of runs. The main text states \"For each experiment we also show 1·σ error bars, estimated from five independent runs ...\" but then in table 3 it states that there are only 3 random seeds. Adding clarity as to this mismatch is important.\n+ What are the parameters used for ECE (e.g., number of bins)? Was binning done w/equal spacing? Adding these details is important as it can change results.\n+ The snapshot ensemble numbers are surprisingly bad, it might strengthen the paper to attempt to fix this baseline or at least explain the large gap between this and other methods.\n+ Minor overclaiming: \"can be readily combined with most pre-trained transformer networks irrespective of their specific architecture.\" We don't know whether results will hold with more complex architectures like MoE. Perhaps a narrower claim would be more precise.\n+ More complete hyperparameter tuning descriptions for all baselines should be included (e.g., for explicit ensembles). It looks like the appendix only has sections for MC dropout tuning and the main method under test.\n+ It is somewhat surprising that Single Network with LoRA is fairly close to LoRA ensemble. One concern that this raises is that some of the gains are from mostly from lora-based adaptation as opposed to needing to model the ensemble. It may be worth adding some discussion around this.", "audience_yn": "Yes", "audience_explanation": "Yes, ensembling and uncertainty estimation continue to be important methods. LoRA provides a convenient way to do this in transformers. This paper provides significant ablations and empirical analysis to test the feasibility and utility of this approach. This will be useful for future decision-making in uncertainty estimation and is a clear baseline to use against more complicated approaches.", "requested_changes": "+ The paper is explicitly about efficient uncertainty/ensembling for large pretrained models, but it does not cite or compare against EpiNets from Osband et al (2021), which is a directly comparable baseline trying to make similar claims to this work. This work would be more complete, and accurate, if it baselined against that work, or at the very least added a citation. Additionally, why not compare against a simple ensemble of linear heads, like EpiNets do? Is LoRa really necessary over linear heads? Some additional clarification and baselines would help address some questions that the work raises.\n+ EpiNet is not the only potential missing baseline. Suggest either drop the claim \"outperforms state-of-the-art implicit techniques\" or have more canvasing of baselines. Here are a few others that might be relevant:\n    + https://www.jmlr.org/papers/v24/22-0479.html\n    + https://proceedings.neurips.cc/paper_files/paper/2019/hash/118921efba23fc329e6560b27861f0c2-Abstract.html\n+ \"training about 2x faster and delivering over 3x faster inference\" -> For the sake of accurate discussion, there should be an additional clause here describing under what conditions it is faster (one GPU). While there is information in the appendix this sentence can be misleading. You can, of course, parallelize to be nearly as fast across many GPUs, but it seems that this is indicating speedups under a specific set of compute constraints. Memory overhead is the real benefit here that translates to speedups in low resource settings.\n+ There's a mismatch (or at least some ambiguity) about the number of runs. The main text states \"For each experiment we also show 1·σ error bars, estimated from five independent runs ...\" but then in table 3 it states that there are only 3 random seeds. Adding clarity as to this mismatch is important.\n+ What are the parameters used for ECE (e.g., number of bins)? Was binning done w/equal spacing? Adding these details is important as it can change results.\n+ The snapshot ensemble numbers are surprisingly bad, it might strengthen the paper to attempt to fix this baseline or at least explain the large gap between this and other methods.\n+ Minor overclaiming: \"can be readily combined with most pre-trained transformer networks irrespective of their specific architecture.\" We don't know whether results will hold with more complex architectures like MoE. Perhaps a narrower claim would be more precise.\n+ More complete hyperparameter tuning descriptions for all baselines should be included (e.g., for explicit ensembles). It looks like the appendix only has sections for MC dropout tuning and the main method under test.\n+ It is somewhat surprising that Single Network with LoRA is fairly close to LoRA ensemble. One concern that this raises is that some of the gains are from mostly from lora-based adaptation as opposed to needing to model the ensemble. It may be worth adding some discussion around this.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fgrenb4Ckb", "reviewer": "Reviewer_xqsB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes LoRA-Ensemble, a parameter-efficient deep ensemble method for self-attention/transformer models. The method keeps a pre-trained backbone frozen and creates multiple ensemble members by training member-specific LoRA adapters (low-rank updates) in the attention projection layers. The paper evaluates accuracy and uncertainty quality (ECE/NLL/Brier) across vision, audio, NLP, OOD detection, and corruption robustness and argues LoRA-Ensemble yields higher member diversity than explicit ensembles.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- method: Clearly described and easy to follow. LoRA-Ensemble = shared frozen backbone + per-member LoRA adapters in attention projections (Sec. 2).\n\n- On the core transformer benchmarks, LoRA-Ensemble is consistently strong vs MC Dropout / Snapshot / Batch-Ensemble, especially for CIFAR and HAM10000 (Section 3)\n\n- Matches or exceeds Explicit Ensemble accuracy and has superior calibration): Often true in vision benchmarks (Sec. 3), but not universal (e.g., ESC-50 accuracy slightly lower; SST-2 accuracy slightly lower; calibration metrics can be mixed depending on dataset/metric. Partial; the paper should soften “matches or exceeds” / “superior calibration” to be dataset-dependent\n\n- higher diversity than Explicit Ensemble explains gains): Diversity evidence is convincing (Sec. 4; appendices). The causal phrasing (“because diversity”) should be framed more carefully as “evidence consistent with” rather than proven causality. Mostly suppoorted.", "audience_yn": "Yes", "audience_explanation": "Likely interested audience: uncertainty/calibration researchers, transformer practitioners, and people deploying models under memory constraints.", "requested_changes": "### Critical \n\n1. Clarify the HAM10000 experimental protocol: Since this is a medical dataset, please clearly state the split procedure (train/val/test), and whether it is patient-/lesion-level if applicable, and how leakage/near-duplicates are avoided (Sec. 3 + appendix as needed). This is important to trust the strongest “real-world” results.\n\n2. Make calibration/metrics fully checkabl: Please explicitly state the ECE setup (binning choices) and fix/clarify the nonstandard accuracy definition in the metrics appendix so readers can reproduce and interpret results reliably.\n\n3. Qualify the “better than Explicit Ensemble” msg: Please soften “matches or exceeds” / “superior calibration” to reflect the reported exceptions (e.g., ESC-50 and SST-2 accuracy; CIFAR-100 ECE vs single-LoRA). Also clarify exactly which parameters are trained for the Explicit Ensemble baseline so the comparison is unambiguous (Sec. 3; Discussion).\n\n## For stronger contribution\n\n1. Add a more controlled baseline to isolate the confound from freezing: For example: an Explicit Ensemble variant with a frozen backbone (train head only, or head+LoRA) so readers can separate the effect of LoRA-Ensemble vs the effect of freezing/regularization.\n\n2. Standardize efficiency reporting: Along with wall-clock time, report peak GPU memory and clarify best-effort parallelization assumptions; speed results vary by backbone/implementation.\n\n3. Practical guidance: brief recommendations for choosing LoRA rank and when to apply temperature scaling, since calibration behavior can change with ensemble size and dataset (Discussion + appendices).\n\n4. OOD/robustness reporting: consider adding an additional OOD metric (e.g., FPR@95TPR) or additional OOD settings to strengthen the generality claim.", "broader_impact_concerns": "Potential positive impact: better calibrated uncertainty estimates can improve decision-making in high-stakes settings (the paper evaluates a medical classification dataset)\n\nRisks / cautions: uncertainty estimates can be misused or over-trusted; medical results especially require clear dataset split and leakage controls. The paper should include a short caution that benchmark calibration does not guarantee safe deployment under real-world distribution shift, and that external validation is needed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dEihlhKWjK", "reviewer": "Reviewer_kn9D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The article proposes applying low-rank adaptation (LoRA) as an ensembling method for self-attention networks. The main strengths of the method are in my subjective opinion that LoRA-Ensemble appears to be broadly applicable and straightforward to implement whilst also demonstrating good performance on multiple metrics on wide variety of experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I found the experiments performed by the authors to be very convincing, since they contained a wide variety of metrics, experimental settings and datasets, including real-world data. LoRA ensembles attained almost always the best scores among the tested methods and baselines and were at least competitive against them. The briefy presented theory seems accurate, but I lack expertise in related methods to be able to provide a more precise assessment of these (and whether there are any missing baselines or other weaknesses in the experiments).", "audience_yn": "Yes", "audience_explanation": "I'd argue that someone in TMLR's audience will definitely be interested in the article due to its easily applicable nature and strong empirical evidence.", "requested_changes": "In my subjective opinion, the article is already good as it is.", "broader_impact_concerns": "I don't see any immediate ethical implications of the work that would be worth mentioning in a Broader Impact Statement.", "additional_comments": "Please note that I lack expertise in the topics studied in the article and weigh my review accordingly. Still, I wish to emphasize that I find the article convincing and easy to read even with my limited expertise.\n\nFormatting:\n- Citations do not currently have authors' names in parentheses, e.g., \"due to a lack of training data Der Kiureghian & Ditlevsen (2009).\"", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)