LoRA-Ensemble: Efficient Uncertainty Modelling for Self-attention Networks

## Abstract

Abstract Numerous crucial tasks in real-world decision-making rely on machine learning algorithms with calibrated uncertainty estimates. However, modern methods often yield overconfident and uncalibrated predictions. Various approaches involve training an ensemble of separate models to quantify the uncertainty related to the model itself, known as epistemic uncertainty. In an explicit implementation, the ensemble approach has high computational cost and high memory requirements. This particular challenge is evident in state-of-the-art neural networks such as transformers, where even a single network is already demanding in terms of compute and memory. Consequently, efforts are made to emulate the ensemble model without actually instantiating separate ensemble members, referred to as implicit ensembling. We introduce LoRA-Ensemble, a parameter-efficient deep ensemble method for self-attention networks, which is based on Low-Rank Adaptation (LoRA). Initially developed for efficient LLM fine-tuning, we extend LoRA to an implicit ensembling approach. By employing a single pre-trained self-attention network with weights shared across all members, we train member-specific low-rank matrices for the attention projections. Our method exhibits superior calibration compared to explicit ensembles and achieves similar or better accuracy across various prediction tasks and datasets.

## 1 Introduction

Machine learning models are increasingly applied also in fields where incorrect estimates may have severe consequences, e.g., autonomous driving, medical diagnosis, (extreme) weather event prediction, or agricultural management decision support. In such applications well-calibrated predictive uncertainties are crucial to enable self-diagnosis. Uncertainty can be separated into two components. *Aleatoric uncertainty*, a.k.a. irreducible noise, is inherent in the data. *Epistemic uncertainty* on the other hand stems from a lack of knowledge about certain regions of the input space, due to a lack of training data [Der Kiureghian and Ditlevsen, ].

Quantification of epistemic uncertainty in large machine learning models is non-trivial. Analytical computation is usually intractable, thus research has focused on efficient approximations [Graves, , Blundell et al., , Welling et al., ]. To date, probabilistic ensembles remain the best-performing approach [Lakshminarayanan et al., ]. In a naïve implementation, such an ensemble consists of multiple independently trained models. Individual models are interpreted as Monte Carlo samples from the posterior weight space and are used to obtain an unbiased estimator of the posterior distribution. To achieve a low correlation between ensemble members one can capitalize on the stochastic nature of the training process and start from different initial weights, and/or sample different random batches of data. The basic principle is that the predictions of different ensemble members will agree near observed training samples, whereas they may vary far away from the training data. Their spread therefore serves as a measure of epistemic uncertainty. Even small ensembles often capture the uncertainty well (in expectation), i.e., they are well calibrated.

An issue with naïve ensembles is that their computational cost and memory footprint grow proportionally to the number of ensemble members. For smaller models explicit ensembling may still be feasible, albeit with higher financial cost and energy consumption. For modern neural networks with up to several billion parameters, hardware restrictions render the naïve approach intractable, in particular, one can no longer hold the entire ensemble in memory.
Consequently, a lot of research has gone into ways of creating ensembles implicitly, without requiring multiple copies of the full base model [Wen et al., , Wenzel et al., , Huang et al., , Turkoglu et al., ].
Unfortunately, most of these parameter-efficient ensembling techniques are not applicable to the newest generation of neural networks. Transformer networks [Vaswani et al., ] have recently become popular due to their superior ability to capture complex structures in data. However, implicit ensembling schemes tend to underperform for transformers or are entirely incompatible with them, as detailed in Appendix M.

Several studies have shown that modern neural networks are heavily overparametrized and that their results have low intrinsic dimension [Li et al., , Aghajanyan et al., ]. This led Hu et al. [] to propose \glsxtrfullpllora as a way of deploying individually fine-tuned \glsxtrfullplllm to different tasks while avoiding the prohibitively large memory and compute requirements of retraining them. It turns out that the weight matrices in such models can be factorized to have very low rank, with hardly any loss in prediction performance.

This led us to use \glsxtrshortlora as the basis for a novel, parameter-efficient ensemble method that is tailored to the transformer architecture. In line with the trend towards transfer learning, our method uses a pre-trained transformer model, which is expanded into an implicit ensemble by varying the \glsxtrshortlora factorization, while keeping the backbone weights frozen. In this way, our method only requires a small number of additional parameters to turn an existing transformer model into a diverse ensemble whose performance across various tasks is comparable to an Explicit Ensemble.
In summary, our contributions are:

- •
We introduce \glsxtrshortlora-Ensemble, a parameter-efficient probabilistic ensemble method for self-attention networks.
- •
\glsxtrshort
lora-Ensemble can be readily combined with most pre-trained transformer networks, irrespective of their specific architecture and application domain: it simply replaces the linear projection layers in the attention module with \glsxtrshortlora-Ensemble layers.
- •
We apply \glsxtrshortlora-Ensemble to different classification tasks including: conventional image labeling, classification of skin lesions in dermatoscopic images, sound classification from spectrograms, and out-of-distribution (OOD) detection. In these experiments, \glsxtrshortlora-Ensemble not only consistently outperforms other implicit ensemble schemes but also, surprisingly, its classification accuracy and uncertainty calibration are often even better than that of an Explicit Ensemble.

## 2 LoRA-Ensemble

The \glsxtrfulllora technique makes it possible to use a pre-trained model and fine-tune it without having to retrain all its parameters. This is particularly beneficial for modern neural networks with large parameter spaces. The underlying principle is to freeze the pre-trained model weights $W_{0}\in\mathbb{R}^{k\times d}$ and instead constrain the updates to a low-rank decomposition. This can be expressed mathematically as:

$$ $W=W_{0}+\Delta W=W_{0}+B\!\cdot\!A\;.$ (1) $$

Here $B\in\mathbb{R}^{k\times r}$ and $A\in\mathbb{R}^{r\times d}$ are two trainable low-rank matrices, where $r\ll\min(d,k)$. $W$ and $\Delta W$ are then multiplied with the same input $x$, which yields the following modified forward pass:

$$ $h=W_{0}\cdot x+\Delta W\cdot x=W_{0}\cdot x+B\!\cdot\!A\cdot x\;.$ (2) $$

lora applies this low-rank updating scheme only to weights in the self-attention modules of a transformer model while leaving the interleaved MLP modules untouched. I.e., the weight matrices being updated are $W_{q}$, $W_{k}$, and $W_{v}$ for the query, key, and value of the attention mechanism, as well as the $W_{o}$ for merging the multi-head outputs. The former three are each treated as a single matrix, disregarding the fact that they are typically sliced into multiple attention heads. [Hu et al., ]

Although not designed with uncertainty calibration in mind, the \glsxtrshortlora concept fulfills all the requirements of an implicit deep ensemble: By modifying the weights of the highly nonlinear self-attention mechanism one is able to generate a diverse collection of networks with the same architecture and objective. By learning an additive, low-rank update $\Delta W=B\!\cdot\!A$ rather than directly tuning the weight matrices, the expansion into a model ensemble adds only a small number of parameters and is efficient.
In detail, we start from a single, pre-trained model with frozen parameters $W_{0}$ and expand it with a set of trainable low-rank matrices $\Delta W_{i}$, $\forall i=1\ldots N$.
At each transformer block, there now is a different forward pass per ensemble member $i$, as illustrated in Fig. 1:

$$ $h_{i}=W_{0}\cdot x+\Delta W_{i}\cdot x=W_{0}\cdot x+B_{i}\!\cdot\!A_{i}\cdot x\;,$ (3) $$

leading to $N$ different predictions $T_{\theta_{i}}(X)$ for a given input $X$. From those individual predictions, we compute the ensemble estimate by simple averaging:

$$ $\mathbb{E}[Y|X]=\frac{1}{N}\sum_{i=1}^{N}T_{\theta_{i}}(X)\;.$ (4) $$

Figure: Figure 1: A schema of a \glsxtrshortlora-Ensemble. The computation structure of the multi-head self-attention module (right), and \glsxtrshortlora-Ensemble module (bottom left). $X$ denotes the actual input, and $x$ represents the intermediate input representation.
Refer to caption: x1.png

### 2.1 Implementation

In practice, our \glsxtrshortlora-Ensemble is implemented by replacing the respective linear layers ($W_{q}$, $W_{k}$, $W_{v}$, and $W_{o}$) in the pre-trained model architecture with custom \glsxtrshortlora modules.

As a backbone for experiments with image datasets, we employ a \glsxtrfullvit model [Dosovitskiy et al., ]. The chosen architecture is the base variant with patch size $32\times 32$ as defined in Dosovitskiy et al. []. We load the weights from torchvision, which were trained on ImageNet-1k [Deng et al., ], using a variant of the training recipe from Touvron et al. [], for details refer to their documentation.

The forward pass through the backbone is parallelized by replicating the input along the batch dimension. In each \glsxtrshortlora module, the data is split into separate inputs per member and passed to the respective member with the help of a *vectorized map*, which allows a parallelized forward pass even through the \glsxtrshortlora modules. The outputs are then again stacked along the batch dimension. In this way, one makes efficient use of the parallelization on \glsxtrshortgpu, while at the same time avoiding loading the pre-trained backbone into memory multiple times.

As a backbone for audio experiments, we use the \glsxtrfullast backbone [Gong et al., ]. That architecture was inspired by \glsxtrshortvit (more specifically the data-efficient version of \glsxtrshortvit akin to \glsxtrshortdeit [Touvron et al., ]) but is designed specifically for audio spectrograms. Following Gong et al. [], we initialize the audio model weights by transferring and appropriately interpolating them from ImageNet pre-training. See Appendix G and H for details.
As the \glsxtrshortast version of \glsxtrshortlora-Ensemble would run into memory limits, we introduce chunking. While the forward pass through the backbone is still parallelized, the \glsxtrshortlora modules are called sequentially.(^1^11For the Explicit Ensemble the vectorization could not be used on GPU, due to a technical issue with the \glsxtrshortvit implementation in PyTorch.)

Finally, the pre-trained model does not have the correct output dimension for our prediction tasks (i.e., it was trained for a different number of classes). Therefore we entirely discard its last layer and add a new one with the correct dimensions, which we train from scratch. Obviously, the weights of that last layer are different for every ensemble member. We parallelize it in the same way as the \glsxtrshortlora module described above.

A PyTorch implementation of \glsxtrshortlora-Ensemble, as well as pre-trained weights to reproduce the experiments, will be publicly released on GitHub.

## 3 Experiments

In the following section, we evaluate the proposed \glsxtrshortlora-Ensemble on several datasets with regard to its predictive accuracy, uncertainty calibration, and memory usage. For each experiment we also show 1-sigma error bars, estimated from five independent runs with different random initializations.

As a first sandbox experiment, we perform image classification for the popular, widely used CIFAR-100 benchmark [Krizhevsky, ] (see Appendix A for CIFAR-10 experiment). The dataset consists of 100 object classes, each with 600 samples, for a total size of 60 000 images. From that set, 10 000 images are designated test data, with all classes equally distributed between the training and testing portions.

The HAM10000 dataset was proposed for the *Human Against Machine with 10 000 training images* study [Tschandl et al., ]. It consists of 10 015 dermatoscopic images of pigmented skin lesions, collected from different populations. The dataset was initially assembled to compare machine learning methods against medical professionals on the task of classifying common pigmented skin lesions. Compared to CIFAR-100, this is arguably the more relevant test bed for our method: in the medical domain, uncertainty calibration is critical, due to the potentially far-reaching consequences of incorrect diagnoses and treatment planning.

For both datasets, \glsxtrshortlora-Ensemble is compared against several baselines. As a sanity check, we always include results obtained with a single \glsxtrfullvit model, as well as with a single \glsxtrshortvit model with \glsxtrshortlora in the attention modules. These models do not have a dedicated mechanism for uncertainty calibration, instead, the predicted class-conditional likelihoods are used to quantify uncertainty. Furthermore, we compare to an explicit model ensemble, \glsxtrfullmcdropout as implemented in Li et al. [] and a modified version of Snapshot Ensemble [Huang et al., ], detailed in Appendix L.
Snapshot Ensemble is the only well established implicit ensembling technique that is architecture agnostic and can therefore be applied to self-attention networks in a straightforward fashion. For implementation challenges of other implicit methods, please refer to Appendix M. The \glsxtrshortlora rank was empirically set to 8 for CIFAR-100 and 4 for HAM10000.

We evaluate predictive performance and calibration quality for each method using multiple metrics. Predictive accuracy is assessed with classification accuracy (percentage of correctly classified test samples) and the F1-score, which balances precision and recall. Calibration quality is measured using the Expected Calibration Error (ECE), Negative Log-Likelihood (NLL), and Brier score. The ECE quantifies the deviation from a perfectly calibrated model, i.e., one where the estimated uncertainty of the maximum-likelihood class correctly predicts the likelihood of a miss-classification. Definitions of all metrics are provided in Appendix N.

As a further benchmark from a different application domain, we process the ESC-50 environmental sounds dataset [Piczak, ]. It consists of 2000 sound samples, each five seconds long, that represent 50 different semantic classes with 40 samples each. To prepare the raw input waveforms for analysis, they are converted into 2-dimensional time/frequency spectrograms, see Gong et al. []. These spectrograms form the input for the \glsxtrlongast, a state-of-the-art transformer model for sound classification.

As for the \glsxtrshortvit model, we train an \glsxtrlongast version of \glsxtrshortlora-Ensemble by modifying the attention weights with different sets of \glsxtrshortlora weights. That ensemble is then compared to a single instance of \glsxtrshortast with and without \glsxtrshortlora, to an Explicit Ensemble of \glsxtrshortast-models, and to an \glsxtrshortmcdropout variant of \glsxtrshortast, similar to Li et al. []. For ESC-50 a \glsxtrshortlora rank of 16 worked best, presumably due to the larger domain gap between (image-based) pre-training and the actual audio classification task.
The experimental evaluation in Gong et al. [] employs the same performance metrics as before, but a slightly different evaluation protocol. Model training (and evaluation) is done in a 5-fold cross-validation setting, where the epoch with the best average accuracy across all five folds is chosen as the final model. The performance metrics given below are calculated by taking the predictions of all five folds at the chosen epoch and evaluating
accuracy and calibration metrics jointly.
While the accuracy calculated this way is equivalent to the average of all five folds,
others are not, so this method results in a more realistic calculation of the calibration metrics.

For the out-of-distribution (OOD) detection experiment, we trained models on the CIFAR-100 dataset and evaluated their performance using samples from CIFAR-100 (in-distribution) and CIFAR-10 (out-of-distribution), following standard OOD detection practices Hendrycks and Gimpel []. We assessed the performance by calculating the area under the ROC curve (AUROC) and the area under the precision-recall curve (AUPRC).

### 3.1 Computational Cost

In addition to evaluating classification performance and calibration, we assess the computational cost in terms of parameters, training, and inference time. The required resources are presented in Tab. 1.

**Table 1: Parameter counts and computation times for an Explicit Ensemble of 16 ViT models and the corresponding LoRA-Ensemble. Training time is the average duration for one epoch on CIFAR-100, with batch size 32. Inference time is the average duration of a forward pass, with batch size 1.**
| Method | Parameter overhead | Training time [s] | Inference time [ms] |
| --- | --- | --- | --- |
| Explicit Ensemble | $16\times 87\mathrm{M}$ | $16\times 139$ | $16\times 4.6$ |
| \glsxtrshortlora-Ensemble | $1.12\times 87\mathrm{M}$ | 1108 | 22.7 |

The total *number of parameters* is reported for an ensemble of 16 members, and matrices $A$ and $B$ with rank 8 when using \glsxtrshortlora. Choosing a different rank will slightly alter the parameter count. In many cases a lower rank may suffice, cf. Hu et al. []. All times were measured on a single NVIDIA Tesla A100-80GB GPU. *Training time* is given as the average wall clock time per training epoch on CIFAR-100, with 16 ensemble members. *Inference time* is computed as the average time for a single forward pass for a CIFAR-100 example, with batch size 1. As mentioned in Sec. 2.1, the forward pass for the Explicit Ensemble processes the members sequentially.(^2^22Speed comparisons only make sense with the same resources. With sufficiently many GPUs any ensemble method can be parallelized by instantiating explicit copies of different members on separate GPUs.) Hence, we calculate the average time needed for one member and multiply it by 16.
It is evident that the proposed method uses significantly fewer parameters and less memory. \glsxtrshortlora-Ensemble also trains faster, and speeds up inference more than 3 times.

We point out that, with our current implementation, the runtime comparisons are still indicative. It turns out that PyTorch’s vectorized map (vmap) has a large one-time overhead that is only amortized when using large ensembles, while small ensembles are slowed down. Practical ensemble sizes will benefit when implemented in a framework that supports just-in-time compilation, like JAX.

### 3.2 CIFAR-100

Classification accuracy and \glsxtrshortece are both graphed against ensemble size in Fig. 2. Quantitative results for all compared methods are summarized in Tab. 2.

lora-Ensemble consistently reaches higher accuracy than \glsxtrshortmcdropout and Snapshot Ensemble, with a notable edge of approximately 5 percentage points for ensembles of four or more members. Surprisingly, it also consistently surpasses the Explicit Ensemble by about 2 percentage points, apparently a consequence of the fact that already a single \glsxtrshortvit model, and thus every ensemble member, benefits from the addition of \glsxtrshortlora.

Figure: Figure 2: Accuracy and \glsxtrlongece on CIFAR-100, with different ensemble sizes.
Refer to caption: extracted/5917163/Figures/03_Experiments/CIFAR100_results_accuracy.png

With \glsxtrshortlora-Ensemble also the estimates of predictive uncertainty are better calibrated. Interestingly, the calibration is already very good at small ensemble sizes but slightly degrades when adding more members. This effect is not present when looking at NLL and Brier score, though, Tab 2.
The reliability diagram in Fig. 3 somewhat elucidates this unexpected behavior. It turns out that \glsxtrshortlora-Ensemble is generally under-confident, meaning that the classification is more accurate than the model suggests. Rahaman and Thiery [] have found that when ensembling under-confident models, the accuracy grows faster than the confidence. As a result, the difference between accuracy and confidence tends to grow, worsening calibration metrics. Note that in safety-critical applications under-confident models that over-estimate the uncertainty are often preferable to over-confident ones.

Figure: Figure 3: Reliability diagrams for Explicit Ensemble (left) and \glsxtrshortlora-Ensemble (right) with 16 members, on CIFAR-100.
Refer to caption: extracted/5917163/Appendices/Figures/reliability_diagram_Deep_Ensemble_ViT_base_32_16_members_CIFAR100_settings_explicit5.png

mcdropout is not well calibrated for smaller ensembles, but progressively catches up as the ensemble size increases.
Snapshot Ensemble performs similarly to \glsxtrshortmcdropout in terms of accuracy but does not perform competitively for calibration.

**Table 2: Model performance on the CIFAR-100 dataset for the compared methods. Ensembles have 16 members. Best score for each metric in bold, second-best underlined.**
| Method | Accuracy ($\uparrow$) | F1 ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$) | Brier ($\downarrow$) |
| --- | --- | --- | --- | --- | --- |
| Single Network | $76.6\pm 0.3$ | $76.6\pm 0.3$ | $0.145\pm 0.004$ | $1.181\pm 0.019$ | $0.370\pm 0.004$ |
| Single Net w/ LoRA | $79.6\pm 0.2$ | $79.4\pm 0.2$ | $\textbf{0.014}\pm 0.003$ | $\underline{0.671}\pm 0.005$ | $0.286\pm 0.003$ |
| MC Dropout | $77.1\pm 0.5$ | $77.2\pm 0.4$ | $0.055\pm 0.002$ | $1.138\pm 0.014$ | $0.336\pm 0.005$ |
| Snapshot Ensemble | $77.0\pm 0.1$ | $77.2\pm 0.2$ | $0.123\pm 0.002$ | $4.416\pm 0.046$ | $1.614\pm 0.007$ |
| Explicit Ensemble | $\underline{79.8}\pm 0.1$ | $\underline{79.8}\pm 0.2$ | $0.100\pm 0.001$ | $0.745\pm 0.003$ | $\underline{0.284}\pm 0.002$ |
| LoRA-Ensemble | $\textbf{82.5}\pm 0.1$ | $\textbf{82.5}\pm 0.1$ | $\underline{0.035}\pm 0.001$ | $\textbf{0.587}\pm 0.001$ | $\textbf{0.253}\pm 0.000$ |

### 3.3 HAM10000 Lesion Classification

In many medical applications, well-calibrated models are essential. As a test case, we use the classification of pigmented skin lesions and again compare the same group of models in terms of accuracy and calibration. The results are summarized in Tab. 3.

Similar to the CIFAR-100 evaluation, \glsxtrshortlora-Ensemble outperforms all other methods by a clear margin, with respect to both classification accuracy and calibration.
Surprisingly, Snapshot Ensemble performs very well in terms of calibration but is not competitive as far as accuracy is concerned.
The experiments also further support the above discussion of confidence vs. ensemble size (Sec. 3.2). For HAM10000 \glsxtrshortlora-Ensemble is slightly over-confident (just like the Explicit Ensemble) and, indeed, its calibration error decreases with ensemble size in this case, see Appendix A.2.

We conducted further experiments on HAM10000 using different backbone architectures with varying numbers of parameters. The results are shown in Tab. 7 in Appendix B.
In conclusion, LoRA-Ensemble generalizes effectively across different backbones. Moreover, as the number of parameters in the backbone architecture increases, the superiority of LoRA-Ensemble over Explicit Ensemble in both accuracy and calibration becomes more pronounced.

Table: Table 3: Model performance on the HAM10000 dataset for the compared methods. Ensembles have 16 members. Best score for each metric in bold, second-best underlined

### 3.4 ESC-50 Environmental Sound Classification

To go beyond computer vision tasks, \glsxtrshortlora-Ensemble is also applied to an audio dataset, using the \glsxtrlongast as the backbone model.

The results are summarized in Tab. 4.
On this dataset \glsxtrshortlora-Ensemble does not significantly outperform the Explicit Ensemble, but still matches its performance with much lower computational demands, see Appendix J. Accuracy is insignificantly lower, whereas calibration is slightly better in terms of ECE. We note that, remarkably, the weights used in the transformer modules and for creating patch embeddings were pre-trained on images rather than audio streams.

**Table 4: Model performance on the ESC-50 dataset for the compared methods. Ensembles have 8 members due to memory limitations. Best score for each metric in bold, second-best underlined.**
| Method | Accuracy ($\uparrow$) | F1 ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$) | Brier ($\downarrow$) |
| --- | --- | --- | --- | --- | --- |
| Single Network | $89.6\pm 0.7$ | $89.5\pm 0.7$ | $0.039\pm 0.004$ | $0.410\pm 0.020$ | $0.164\pm 0.009$ |
| Single Net w/ LoRA | $88.0\pm 0.3$ | $87.8\pm 0.3$ | $0.043\pm 0.004$ | $0.461\pm 0.019$ | $0.186\pm 0.005$ |
| MC Dropout | $89.4\pm 0.3$ | $89.3\pm 0.4$ | $0.087\pm 0.005$ | $0.553\pm 0.012$ | $0.176\pm 0.005$ |
| Explicit Ensemble | $\textbf{91.3}\pm 0.2$ | $\textbf{91.2}\pm 0.3$ | $\underline{0.027}\pm 0.004$ | $\textbf{0.322}\pm 0.004$ | $\textbf{0.133}\pm 0.001$ |
| LoRA-Ensemble | $\underline{91.1}\pm 0.2$ | $\underline{90.8}\pm 0.2$ | $\textbf{0.021}\pm 0.003$ | $\underline{0.328}\pm 0.004$ | $\underline{0.138}\pm 0.001$ |

### 3.5 Out-of-Distribution (OOD) Detection

To evaluate our method’s effectiveness in OOD detection, a crucial aspect of quantifying uncertainty in deep learning models Hendrycks and Gimpel [], we conducted an experiment where models were trained on CIFAR-100 (in-distribution) and tested on samples from both CIFAR-100 and CIFAR-10 (out-of-distribution). The similar distributions of these datasets make this a near-OOD task, which is more challenging than tasks involving distinctly different distributions [Sim et al., ]. Following Sim et al. [] and Chen et al. [], we used the maximum softmax probability as the confidence score.

Tab. 5 shows that the \glsxtrshortlora-Ensemble significantly outperforms all other methods on both metrics, even surpassing the recently proposed Split-Ensemble method Chen et al. [] designed specifically for OOD tasks. Furthermore, consistent with our earlier observations on LoRA’s effectiveness in improving network calibration, even a single \glsxtrshortlora model achieves performance comparable to the Explicit Ensemble, highlighting its robustness in OOD scenarios.

**Table 5: Model performance on the OOD task. CIFAR-100 is used as the in-distribution dataset and CIFAR-10 as the out-of-distribution dataset. Ensembles for all methods consist of 16 members. Results for Split-Ensemble are taken from Chen et al. []. The best score for each metric is highlighted in bold, with the second-best score underlined.**
| Method | AUROC ($\uparrow$) | AUPRC ($\uparrow$) |
| --- | --- | --- |
| Split-Ensemble Chen et al. [] | $79.2$ | 81.7 |
| Single Network | $75.6\pm 0.3$ | $77.6\pm 0.6$ |
| Single Network with LoRA | $\underline{80.1}\pm 0.5$ | $\underline{82.4}\pm 0.6$ |
| MC Dropout | $75.1\pm 0.5$ | $73.7\pm 0.9$ |
| Explicit Ensemble | $78.9\pm 0.2$ | $80.8\pm 0.2$ |
| LoRA-Ensemble | $\textbf{82.1}\pm 0.1$ | $\textbf{84.1}\pm 0.1$ |

### 3.6 Sensitivity Analysis: LoRA Rank

The main hyper-parameter introduced by adding \glsxtrshortlora is the rank of the low-rank decomposition (i.e., the common dimension of the matrices $A$ and $B$).
Varying that rank modulates the complexity of the model for the learning task. We have empirically studied the relationship between rank, accuracy, and \glsxtrlongece. Here we show results for HAM10000, additional results for CIFAR-100 dataset can be found in Appendix C.

On HAM10000 we observe a clear trade-off between accuracy and calibration, Fig. 4. With increasing rank the classification accuracy increases while the calibration deteriorates, in other words, one can to some degree balance predictive accuracy against uncertainty calibration by choosing the rank.
Our focus in this work is on model calibration. We therefore generally choose the rank to favor calibration, even at the cost of slightly lower classification accuracy.

Figure: Figure 4: Impact of \glsxtrshortlora rank on accuracy and \glsxtrshortece, for HAM10000 dataset.
Refer to caption: extracted/5917163/Figures/03_Experiments/HAM10000_lora_rank_results.png

## 4 Related Work

### 4.1 Estimation of Epistemic Uncertainty

A lot of work has gone into estimating the epistemic uncertainty in \glsxtrfullann. As the analytical computation of the posterior in such models is generally intractable, methods for approximate Bayesian inference have been proposed. Such methods rely on imposing an appropriate prior on the weights and using the likelihood of the training data to get an approximate posterior of the weight space.

The main techniques are, on the one hand, Variational Inference [Graves, , Ranganath et al., ], which Blundell et al. [] have specialized to neural networks as *Bayes by Backprop*. And on the other hand variants of \glsxtrfullmcmc [Neal, , Chen et al., ], including \glsxtrfullsgld [Welling et al., ]. These, however, are often not able to accurately capture high-dimensional and highly non-convex loss landscapes, like the ones usually encountered in deep learning [Gustafsson et al., ].

### 4.2 Ensembles and Implicit Ensembling

Lakshminarayanan et al. [] have proposed a method known as deep ensembles. It uses a set of neural networks with identical architecture that are independently and randomly initialized, and (as usual) trained with variants of \glsxtrfullsgd. While the latter introduces further stochasticity, Fort et al. [] have shown that the initialization of the weights is more important to explore the admissible weight space. Ensemble members will generally converge to different modes of the loss function, such that they can be considered Monte Carlo samples of the posterior distribution [Wilson and Izmailov, , Izmailov et al., ]. While ensembles, in general, yield the best results in terms of accuracy and uncertainty calibration, a straightforward implementation suffers from high memory and compute requirements, since multiple instances of the full neural network must be trained and stored. This can become prohibitive for modern neural networks with many millions, or even billions, of parameters.

Consequently, researchers have attempted to find ways of mimicking the principle of deep ensembles without creating several full copies of the base model. Gal and Ghahramani [] have proposed \glsxtrlongmcdropout, where the posterior is approximated by sampling different dropout patterns at inference time. While this is less expensive in terms of memory, performance is often worse. Masksembles [Durasov et al., ] are a variant that attempts to select suitable dropout masks in order to obtain better uncertainty estimates.
Snapshot Ensembles [Huang et al., ] use cyclic learning rates to steer the learning process such that it passes through multiple local minima, which are then stored as ensemble members. This reduces the training effort but does not address memory requirements or inference time.

Particularly relevant for our work are attempts that employ a shared backbone and modify only selected layers. Havasi et al. [] follow that strategy, in their case only the first and last layer of a neural network are replicated and trained independently to emulate an ensemble. BatchEnsemble [Wen et al., ] is similar to \glsxtrshortlora-Ensemble in that it also uses low-rank matrices to change the model parameters. More specifically, shared weight matrices are modulated by element-wise multiplication with different rank-1 matrices to achieve the behavior of a deep ensemble while adding only a small number of parameters. Wenzel et al. [] take this concept further by also ensembling over different hyper-parameter settings. Turkoglu et al. [] freeze all weights of the base model and instead vary the feature-wise linear modulation [FiLM, Li et al., , Takeda et al., ].

A related concept was recently introduced for \glsxtrshortplllm: the Mixtral of Experts model [Jiang et al., ] averages over a sparse mixture of experts to efficiently generate text.

### 4.3 Low-Rank Adaptation in Transformer Networks

lora was originally conceived as a parameter-efficient way of fine-tuning \glsxtrfullplllm [Hu et al., ]. It is based on the observation that, while modern neural networks have huge parameter spaces, the solutions they converge to have much lower intrinsic dimension [Li et al., , Aghajanyan et al., ]. \glsxtrshortlora exploits this and Hu et al. [] show that even when fine-tuning only a low-rank update matrix $B\!\cdot\!A$ (sometimes with rank as low as one or two), the resulting models are competitive with much more expensive fine-tuning schemes. The method quickly became popular and has since also been extended with weight-decomposition [Liu et al., ]. The \glsxtrfulllora idea has been applied in various fields, notably for denoising diffusion models [Luo et al., , Golnari, ].

As we have shown, \glsxtrshortlora’s adaptation technique naturally lends itself to parameter-efficient ensembling. We study the resulting ensemble for uncertainty calibration, a similar approach has concurrently been explored for the purpose of fine-tuning large language models [Wang et al., ], with promising results.

## 5 Discussion

On Effectiveness of \glsxtrshortlora-Ensemble
Across diverse tasks, our experiments consistently show that LoRA-Ensemble matches or surpasses the predictive performance of the state-of-the-art Explicit Ensemble while offering superior calibration.

Adding \glsxtrshortlora to a single model *without any ensembling* improves calibration in most experiments beyond that of a 16-member Explicit Ensemble. This effect may be linked to the well-documented over-parameterization of modern neural networks, which often achieve higher predictive accuracy at the cost of poorer calibration [e.g., Guo et al., ]. By incorporating LoRA while treating all pre-trained weights as constants, we significantly reduce the trainable parameter space, potentially favoring better calibration.
Increasing the number of ensemble members in the \glsxtrshortlora-Ensemble enhances predictive power, potentially leading to improved accuracy, while still maintaining good calibration due to the limited number of trainable weights. However, if the trainable weights are not kept limited; for instance, by increasing the LoRA rank the calibration can worsen, as demonstrated in Fig. 4.
Conversely, enhancing predictive power by increasing the number of pre-trained weights (without altering the trainable weights) further improves the effectiveness of the \glsxtrshortlora-Ensemble for both accuracy and calibration, see Appendix B.

Limitations
We propose a parameter-efficient ensembling method, which performs well in the conducted experiments.
These results, however, still only heuristically show the power of the method, as there is no theoretical proof, that the members do, in fact, converge to different modes and therefore yield sufficient diversity to fully capture the underlying statistics. The presented work also leaves a number of questions that are yet to be answered. In our experiments, we did not evaluate \glsxtrshortlora-Ensemble on very large datasets, such as those often found in natural language processing. It would be interesting to see, how the method performs on such datasets. Correspondingly, it would also be useful to explore how \glsxtrshortlora-Ensemble performs on \glsxtrlongplllm, especially as these models become ever more popular. Additionally, while our method does address the restrictive memory usage of traditional ensembles, it does not reduce computational complexity. The data still needs to be passed through the model once per batch. Furthermore, it is theoretically possible to perform approximate inference on the parameter distribution of the \glsxtrshortlora matrices. This would enable drawing an infinite number of ensemble members from the approximate posterior.

Future Work
As discussed by Rahaman and Thiery [], our work also suggests that in a high-parameter regime, deep ensembles may not exhibit the same behavior as they do in a low-parameter regime, where they typically improve calibration properties. We have previously witnessed this type of phase shift in bias-variance trade-off for large neural networks akin to Double Descent Phenomena [Nakkiran et al., ]. It would be valuable to conduct an in-depth analysis of deep ensemble behavior in high-parameter regimes, while also considering data size, model size, and compute.

## 6 Conclusion

We have presented \glsxtrshortlora-Ensemble, a novel, parameter-efficient method for probabilistic learning that is tailored to the transformer architecture (and potentially other architectures that make use of the attention mechanism). \glsxtrshortlora-Ensemble uses a simple, but efficient trick to turn a single base model into an implicit ensemble: the weights of the base model are kept frozen, but are modulated with the \glsxtrlonglora mechanism.
By training multiple, stochastically varying instances of the low-rank matrices that define the modulation, one obtains a diverse set of ensemble members that share the majority of their weights (specifically, those of the base model) and introduces only minimal overhead through the coefficients of their individual low-rank matrices.
Our experiments on two different computer vision tasks, a sound classification task, and an OOD detection task show that the proposed approach can outperform other, implicit as well as explicit, ensembling strategies in terms of both classification performance and uncertainty calibration.

#### Broader Impact

In recent years, the size of machine learning models has expanded rapidly. GPT-3 [Brown et al., ] has 175 billion parameters, while its successor, GPT-4, is rumored to contain over 1.7 trillion parameters, with training costs exceeding $100 million. As the trend toward larger models continues, growing computational resources are required.
With this work, \glsxtrshortlora-Ensemble aims to contribute to more efficient ensemble methods, considering the resource usage and environmental impact of AI models. This effort strives for more sustainable practices, advancing the concept of "Green AI."

## Appendix A More Experiments & Results

This section presents comprehensive experimental results for a new dataset: CIFAR-10 and includes additional figures for the HAM10000 dataset.

### A.1 CIFAR-10

The results for the CIFAR-10 dataset, as shown in Tab. 6, indicate that \glsxtrshortlora-Ensemble outperforms all other methods across all metrics. Following closely is a single network enhanced with \glsxtrshortlora. This mirrors the results found in the main paper for CIFAR-100, with the exception of the calibration for a single model. It is important to note that although all methods achieve high accuracy and the differences between them are minimal, calibration is nearly perfect for most approaches. This suggests that the CIFAR-10 dataset is relatively easy for modern transformer models, and the results should not be over-interpreted. Nevertheless, the consistent performance across different random seeds suggests that the ranking is likely significant. Given the balanced nature of the CIFAR-10 dataset, the accuracy and F1-score are almost identical.

**Table 6: Performance on the CIFAR-10 dataset for all compared methods. Ensembles have 16 members. Best score for each metric in bold, second-best underlined.**
| Method | Accuracy ($\uparrow$) | F1 ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$) | Brier ($\downarrow$) |
| --- | --- | --- | --- | --- | --- |
| Single Network | $92.8\pm 0.1$ | $92.8\pm 0.1$ | $0.051\pm 0.001$ | $0.333\pm 0.003$ | $0.120\pm 0.002$ |
| Single Net w/ LoRA | $\underline{94.5}\pm 0.0$ | $\underline{94.5}\pm 0.0$ | $\underline{0.009}\pm 0.001$ | $\underline{0.163}\pm 0.002$ | $\underline{0.082}\pm 0.001$ |
| MC Dropout | $92.9\pm 0.2$ | $92.9\pm 0.2$ | $0.023\pm 0.002$ | $0.260\pm 0.005$ | $0.110\pm 0.003$ |
| Explicit Ensemble | $94.1\pm 0.1$ | $94.1\pm 0.1$ | $0.031\pm 0.001$ | $0.181\pm 0.002$ | $0.087\pm 0.001$ |
| Snapshot Ensemble | $93.1\pm 0.1$ | $93.1\pm 0.1$ | $0.037\pm 0.002$ | $1.062\pm 0.021$ | $0.510\pm 0.008$ |
| LoRA-Ensemble | $\textbf{95.9}\pm 0.1$ | $\textbf{95.9}\pm 0.1$ | $\textbf{0.003}\pm 0.001$ | $\textbf{0.128}\pm 0.001$ | $\textbf{0.064}\pm 0.000$ |

### A.2 HAM10000 Lesion Classification

Classification accuracy and \glsxtrshortece for HAM10000 dataset are both graphed against ensemble size in Fig. 6. Again, \glsxtrshortlora-Ensemble outperforms all baselines for larger ensembles. In Fig. 5 the reliability diagrams for \glsxtrshortlora-Ensemble and an Explicit Ensemble with 16 members each on the HAM10000 dataset are shown. Here, the models are overconfident, further supporting our reasoning regarding the surprising behaviour of calibration with growing ensemble size in the case of CIFAR-100.

Figure: Figure 5: Reliability diagrams for Explicit Ensemble (left) and \glsxtrshortlora-Ensemble (right) with 16 members, on HAM10000.
Refer to caption: extracted/5917163/Appendices/Figures/HAM10000_Explicit_reliability_diagramm.png

Figure: Figure 6: Accuracy and \glsxtrlongece on HAM10000, with different ensemble sizes.
Refer to caption: extracted/5917163/Figures/03_Experiments/HAM10000_results_accuracy.png

## Appendix B Effect of Model Size on Prediction and Calibration Performance

Building upon our existing experiments with the HAM10000 dataset, we extended our analysis to include different backbone architectures with varying numbers of parameters. Specifically, we utilized various DeiT models pre-trained with distillation, as described by Touvron et al. []. The results are presented in Table 7. Notably, the DeiT Base-32 model is the same as the ViT Base-32 model.

In the small parameter regime (Tiny-16, Small-16), the addition of a single LoRA module did not consistently enhance calibration compared to using a single model. This observation contrasts with our findings in most other experiments. However, in the larger parameter regime (ViT Base-32), incorporating even a single LoRA module significantly improved calibration.

Furthermore, increasing the number of ensembles in the LoRA-Ensemble not only boosted accuracy but also enhanced calibration, enabling it to match the performance of an Explicit Ensemble in both parameter regimes. Finally, as the number of parameters in the backbone architecture increased, the superiority of the LoRA-Ensemble over the Explicit Ensemble in terms of both accuracy and calibration became more pronounced.

Table: Table 7: Performance metrics on the HAM10000 dataset for different Vision Transformer architectures. Ensembles have 16 members. Best score for each metric in bold, second-best underlined.

## Appendix C More Sensitivity Analysis: LoRA Rank

As discussed in the paper, varying the rank of the low-rank decomposition in \glsxtrshortlora allows for modulation of the model size. We investigated the effect of rank on predictive accuracy and uncertainty calibration for \glsxtrshortlora-Ensemble. The results for the HAM10000 dataset are presented in the main paper, Sec. 3.6.
For the CIFAR-100 dataset, our evaluation of \glsxtrshortlora-Ensemble shows both increased accuracy and improved calibration with increasing rank within the studied range. These findings are illustrated in Fig. 7.

Figure: Figure 7: Impact of \glsxtrshortlora rank on accuracy and ECE, for the CIFAR-100 dataset.
Refer to caption: x2.png

This observation aligns with the findings of Rahaman and Thiery [], as \glsxtrshortlora-Ensemble continues to exhibit under-confidence even at higher ranks. Increasing model complexity enhances confidence, thereby improving calibration. However, at rank 32, the calibration of a single network augmented with \glsxtrshortlora begins to deteriorate, suggesting that a critical boundary has been reached. Beyond this point, the parameter space becomes insufficiently constrained, leading to effects similar to those observed by Guo et al. [].

At higher ranks, accuracy plateaus while memory demand increases linearly with $\mathcal{O}(d)$ and $\mathcal{O}(k)$ for $A\in\mathbb{R}^{r\times d}$ and $B\in\mathbb{R}^{k\times r}$ respectively, where $d$ and $k$ are the dimensions of the pre-trained weight matrix $W_{0}\in\mathbb{R}^{k\times d}$. Consequently, we selected rank 8 for our CIFAR-100 experiments.

## Appendix D Training Details

The CIFAR-10/100 and HAM10000 dataset experiments are based on the ViT-Base-32 architecture [Dosovitskiy et al., ]. This model has 12 layers and uses 768-dimensional patch embeddings, and the multi-head attention modules have 12 heads. All \glsxtrlongvit models for image classification are trained using the AdamW optimizer [Loshchilov and Hutter, ]. The base learning rate is initially set to 0.0001. The training uses a learning rate warm-up of 500 steps, where the learning rate increases linearly from 0 to the base learning rate before switching to a cosine decline over the rest of the steps. During the experiments, the gradients were calculated and then clipped not to exceed a maximum norm of 1. In the case of HAM10000, we used a weighted cross entropy loss that considered the estimated effective number of samples, which was determined using a beta parameter of 0.9991 [Cui et al., ]. Uniform class weights were used for all other datasets. The maximum number of training epochs varies depending on the dataset. For CIFAR-100, the model is trained for 16 epochs (just over 25000 steps), while on HAM10000, it is trained for 65 epochs. Overall, the hyperparameters used in this work were loosely based on Conrad []. The models were trained using pre-trained weights from torchvision 0.17.1 on an NVIDIA Tesla A100 graphics card. Moreover, the \glsxtrshortlora models were configured with a rank of 8 for both CIFAR-10 and CIFAR-100 and a rank of 4 for HAM10000. For Monte Carlo Dropout the dropout rate was empirically set to be $0.2$. Refer to Appendix K for details.

The settings used for the ESC-50 dataset training are similar to those used in Gong et al. []. However, we used a batch size of 1 instead of 48 to enable training on a single GPU. The base learning rate is set to 0.00001 for the Explicit Ensemble as well as MC Dropout experiments and 0.00005 for \glsxtrshortlora-Ensemble. These learning rates are lower than the ones used in Gong et al. [], which is due to the smaller batch size. Refer to the Appendix I for more details. The \glsxtrshortlora models were implemented with a rank of 16. The dropout rate for MC dropout was kept at $0.2$.

As Fort et al. [] have shown, varying initializations of the weights are most important to getting diverse ensemble members. For this reason, various initialization methods and corresponding parameters were tried, with a Xavier uniform initialization [Glorot and Bengio, ] with gain 10, giving the best combination of accuracy and calibration. For more information, refer to Appendix E. This setting is kept for models across all datasets, including the one with an \glsxtrshortast backbone.

For the same reason, we investigated whether adding noise to the pre-trained parameters of an Explicit Ensemble increases its performance through a higher diversity of members. However, the results did not show any additional benefits beyond what the randomly initialized last layer already provided. Therefore, it was not utilized. For more details, refer to Appendix F.

## Appendix E Initialization of LoRA-Ensemble Parameters

Randomness in initialization is a key driver of diversity among ensemble members [Fort et al., ]. Therefore, finding the right balance between diversity and overly disrupting parameters is crucial. Hu et al. [] propose using a random Gaussian initialization for $A$ while setting $B$ to zero. This approach results in $\Delta W=BA$ being zero at the start of training. In our experiments, we adopt this pattern by always initializing $B$ to zero while varying the parameters and methods for initializing $A$.
Following the method outlined by Hu et al. [], our initial experiments concentrated on the Gaussian initialization of $A$, with a mean $\mu=0$ and varying standard deviations.
Additionally, we tested the Xavier uniform initialization [Glorot and Bengio, ] using different values for the gain.
All tests were conducted on the CIFAR-100 dataset and subsequently applied to other experiments.
We compared results in terms of accuracy and \glsxtrlongece.

**Table 8: Accuracy and \glsxtrlongece for different initialization methods and varying distribution parameters for LoRA-Ensemble.**
| Init. Type | Std. / Gain | Accuracy ($\uparrow$) | ECE ($\downarrow$) |
| --- | --- | --- | --- |
| Gaussian | 0.02 | 81.2 | 0.041 |
| 0.05 | 81.4 | 0.037 |  |
| 0.1 | 81.7 | 0.035 |  |
| 0.2 | 82.1 | 0.034 |  |
| 0.5 | 82.6 | 0.036 |  |
| 1 | 82.5 | 0.039 |  |
| 2 | 81.7 | 0.046 |  |
| Xavier Uniform | 1 | 81.5 | 0.039 |
| 5 | 82.2 | 0.034 |  |
| 10 | 82.4 | 0.034 |  |
| 15 | 82.6 | 0.037 |  |
| 20 | 82.4 | 0.038 |  |
| 30 | 82.2 | 0.043 |  |

In Tab. 8, the results are quantitatively presented. It is immediately evident that both techniques and all tested parameters perform similarly. While more specialized models may surpass our results in terms of accuracy, our primary focus is on calibration, with the goal of maintaining comparable predictive performance. Visual inspection of the results in Fig. 8 confirms the high similarity among all results.
Choosing a small calibration error while maintaining high accuracy as a decision criterion, both Gaussian initialization with a standard deviation of $0.5$ and Xavier uniform initialization with a gain of $10$ or $15$ are viable candidates. Since a gain of $10$ combines high accuracy with the lowest \glsxtrlongece, we select Xavier uniform initialization with a gain of $10$ for our experiments.

Figure: (a)
Refer to caption: extracted/5917163/Appendices/Figures/gaussian_init_results.png

## Appendix F Initialization of Explicit Ensemble Parameters

A pre-trained \glsxtrlongvit model is the backbone for our computer vision experiments. Correspondingly, the parameters of all members in an Explicit Ensemble are initialized to the same values across members. Initialization is a primary driver of diversity in ensemble members [Fort et al., ]. Hence, it is crucial to study the effect of noise in the parameter initialization on the calibration of the resulting ensemble.
In the case of pre-trained model weights not having been trained on a dataset with the same number of classes, the last layer of all models is replaced completely. This means that regardless of the ensemble technique used, the weights of the last layer, which is responsible for classification, will vary across members. This variation in the weights of the classification layer is expected to contribute significantly to the diversity of the members.
Nonetheless, we studied the impact of adding noise to the parameters of an Explicit Ensemble. This was done using the following formula:

$$ $W_{\mathrm{new}}=W+\alpha\cdot dW\,,$ (5) $$

where $dW\sim\mathcal{N}(0,\sigma_{W})$. Here $\alpha$ is a scale factor to control the amount of noise and $\sigma_{W}$ is the standard deviation of the parameters within a weight matrix. This was applied to all weight matrices separately.

It is expected that the initial layers of a neural network will learn basic features, while the later layers will include dataset-specific properties. Therefore, it is assumed that adding noise to the later layers would increase diversity while maintaining pre-training. However, adding noise to the earlier layers might disrupt pre-training more significantly, especially with smaller datasets, as these parameters may not converge to meaningful values again. To address this, an experiment was set up where noise was added only to the last encoder layers of the model, increasing the number of affected encoder layers gradually. Additionally, several different noise scales $\alpha$ were tried, ranging from $1$ to $0.0001$.
In the presented experiment, the last classification layer is initialized using PyTorch’s default method for linear layers. At the time of writing it is as follows:

$$ $\displaystyle W_{\mathrm{init}}$ $\displaystyle=\mathrm{Unif}\left(-\sqrt{5}\cdot\sqrt{\frac{3}{fan_in}},\sqrt{ 5}\cdot\sqrt{\frac{3}{fan_in}}\right)$ (6) $\displaystyle B_{\mathrm{init}}$ $\displaystyle=\mathrm{Unif}\left(-\sqrt{\frac{1}{fan_in}},\sqrt{\frac{1}{fan _in}}\right).$ (7) $$

Here $W$ specifies the weight matrix and $B$ is the bias.
Experiments are conducted on the CIFAR-100 dataset.

### F.1 Results

The most important metrics for this section are accuracy and \glsxtrlongece. The results for adding noise to the last layer up to the last five layers are summarized in Fig. 9. Fig. 9(a) depicts the results for a single model, while Fig. 9(b) shows the results for an ensemble of 16 members.

It is evident that none of the experiments surpass the baseline of not using any additional noise beyond the random initialization of the last classification layer. After the last five layers, the results become uninteresting, as they do not vary significantly from those shown in the plots. Therefore, the presentation is truncated at five layers.
Based on the presented results, no additional noise is injected into the Explicit Ensemble, and only the last layer initialization is varied.

Figure: (a)
Refer to caption: extracted/5917163/Appendices/Figures/noise_study_1_members.png

## Appendix G AST Implementation

A different backbone is used for the experiment on the audio dataset. Specifically, we use the \glsxtrfullast following the implementation of Gong et al. [], with slight modifications to fit our general architecture. Appendix H demonstrates the equivalence of our implementation. In their experiments, Gong et al. [] used two different types of pre-trained weights: one pre-trained on a large image dataset and the other on an audio dataset. For our research, we transfer the weights of a vision transformer model known as \glsxtrshortdeit [Touvron et al., ], which has been pre-trained on the ImageNet dataset [Deng et al., ], to the original \glsxtrshortast architecture by Gong et al. []. The model has 12 layers, uses 768-dimensional patch embeddings, and the multi-head attention modules have 12 heads. This task is considered more challenging than using models pre-trained on audio datasets.

## Appendix H Validation of AST Implementation

The \glsxtrfullast model provided by Gong et al. [] was copied without any changes. However, the training and evaluation pipeline was adapted to fit our architecture. Correspondingly, it was essential to validate the equivalence of our implementation by training a single \glsxtrshortast on the ESC-50 dataset. The results of our model should closely match those provided in Gong et al. [].

They offer two sets of pre-trained weights: one where the weights of a \glsxtrlongvit pre-trained on ImageNet [Deng et al., ] are transferred to \glsxtrshortast, and another where the \glsxtrshortast was pre-trained on AudioSet [Gemmeke et al., ]. To verify our implementation, we ran it using the settings provided by Gong et al. [] and compared the results, which are summarized in Tab. 9.
The results for both pre-training modes fall within the uncertainty range provided by Gong et al. []. This suggests that our pipeline yields comparable outcomes, validating our implementation for continued use.

**Table 9: Comparison of the results obtained for the \glsxtrshortast as given in Gong et al. [] and those obtained by our implementation. AST-S refers to the \glsxtrshortast pre-trained on ImageNet, and AST-P refers to the AudioSet pre-training. Both results fall within the uncertainty range provided by Gong et al. [].**
| Model | Accuracy [Gong et al., ] | Accuracy (our implementation) |
| --- | --- | --- |
| AST-S | $88.7\pm 0.7$ | $88.0$ |
| AST-P | $95.6\pm 0.4$ | $95.8$ |

## Appendix I Hyper-parameter Tuning for AST Experiment

The original training settings of the AST-S model in Gong et al. [] utilize a batch size of 48. However, due to the memory constraint of single GPU training on an NVIDIA Tesla A100 with 80 GB memory, replicating a batch size of 48 as in the original publication was infeasible for training an Explicit AST-S Ensemble with 8 members. Consequently, we perform minimal hyper-parameter tuning by employing a batch size of 1 for both the explicit AST-S and the \glsxtrshortlora AST-S model, exploring various learning rates. Apart from batch size and learning rate adjustments, all other settings remain consistent with Gong et al. [].

The hyper-parameter tuning results for the explicit model using a batch size of 1, as shown in Tab. 10, demonstrate performance similar to the original implementation with a batch size of 48, allowing for a fair comparison with our method [Gong et al., ]. Additionally, Tab. 11 showcases the outcomes of tuning the learning rate for our \glsxtrshortlora AST-S model.

**Table 10: Single model 5-Fold cross-validation results of AST-S on ESC-50 sound dataset with different learning rates and batch size 1. The model settings selected based on accuracy for the experiments are highlighted.**
| Model | Learning rate | Accuracy ($\uparrow$) | ECE ($\downarrow$) |
| --- | --- | --- | --- |
| AST-S | 0.00001 | 88.2 | 0.0553 |
| AST-S | 0.00005 | $81.7$ | $0.0933$ |

**Table 11: Single model 5-Fold cross-validation results for our \glsxtrshortlora AST-S implementation on ESC-50 sound dataset with different learning rates and batch size 1. The model settings selected based on accuracy for the experiments are highlighted.**
| Model | Learning rate | Accuracy ($\uparrow$) | ECE ($\downarrow$) |
| --- | --- | --- | --- |
| LoRA AST-S | 0.00001 | $85.6$ | $0.0447$ |
| LoRA AST-S | 0.00005 | 87.9 | 0.0487 |
| LoRA AST-S | 0.0001 | $84.7$ | $0.0501$ |
| LoRA AST-S | 0.0005 | $24.1$ | $0.0291$ |
| LoRA AST-S | 0.001 | $11.8$ | $0.0295$ |

## Appendix J Computational Cost for AST Models

Similarly to the way we did for the \glsxtrlongvit models, we estimate the required resources for \glsxtrshortast models. The resource needs are presented in Tab. 12.

**Table 12: Parameter counts and computation times for an Explicit Ensemble of 8 \glsxtrshortast models and the corresponding \glsxtrshortlora-Ensemble. Training time is the average duration for one epoch on ESC-50, with batch size 1. Inference time is the average duration of a forward pass, with batch size 1.**
| Method | Parameter overhead | Training time [s] | Inference time [ms] |
| --- | --- | --- | --- |
| Explicit Ensemble | $8\times 87\mathrm{M}$ | $517$ | $8\times 7.3$ |
| \glsxtrshortlora-Ensemble | $1.08\times 87\mathrm{M}$ | $348$ | $73.9$ |

The number of parameters is reported for an ensemble of 8 members, with the $A$ and $B$ matrices in models using \glsxtrshortlora having a rank of 16. Training and inference times were measured on a single NVIDIA Tesla A100-80GB \glsxtrshortgpu, with a batch size of 1. Training time is given as the average wall clock time per training epoch while training on ESC-50, with 8 ensemble members. Inference time is reported as the average time for a single forward pass of an ESC-50 sample with a batch size of 1.

As mentioned in Sec. 2.1, the Explicit Ensemble processes the members sequentially, while \glsxtrshortlora-Ensemble is parallelized. However, fully parallelizing the training of \glsxtrshortast models causes memory issues, so chunking was introduced. Thus, in \glsxtrshortlora-Ensemble models, the pass through the backbone runs in parallel, while \glsxtrshortlora modules are called sequentially. This also explains the significantly higher inference time compared to the results in Sec. 3.1.
Additionally, the one-time delay incurred by PyTorch’s *vmap* function causes \glsxtrshortlora-Ensemble to be slightly slower at inference time.

## Appendix K Hyperparameter Tuning for MC Dropout

We conducted an analysis to determine the impact of dropout probability on the accuracy and calibration of the \glsxtrshortvit with Monte Carlo dropout. Fig. 10 displays the accuracy and \glsxtrshortece scores for various dropout probabilities. The experiment is carried out on the HAM10000 dataset with 16 members. Our findings show that a dropout probability of $0.2$ offers a good balance between accuracy and calibration.

Figure: Figure 10: Accuracy and \glsxtrlongece for different dropout probabilities methods for MC Dropout on HAM10000 dataset.
Refer to caption: extracted/5917163/Appendices/Figures/mc_dropout_prob_16_members.png

## Appendix L Snapshot Ensemble Implementation details

Snapshot Ensemble Huang et al. [], in its pure form, consists of training a single model with cycling learning and taking snapshots every few epochs. This can make it hard, however, for the model to converge to anything meaningful within the low number of epochs available for training per snapshot. Therefore, Snapshot Ensemble was modified slightly, by first letting training run for a number of epochs, without any cycling of the learning rate. After this burn-in period the learning rate is at 0 and a first snapshot is taken. The remaining number of epochs is split evenly. If the remaining number of epochs is not divisible by the desired number of ensemble members, the burn-in period is extended until it is. For the HAM10000 dataset training is left at 65 epochs, with 20 burn-in epochs. For CIFAR-10 and CIFAR-100 using only 16 epochs would only leave 1 epoch per cycle for bigger models. Therefore, training is extended to 30 epochs with a burn-in period of 15 epochs.

## Appendix M Implicit Ensemble Baseline Challenge

Many implicit ensemble methods, such as those proposed in Wen et al. [], Turkoglu et al. [], Durasov et al. [], Havasi et al. [], are architecture-specific and predominantly designed for MLPs or CNNs. As a result, adapting these techniques to transformer architectures presents significant challenges, since transformers’ computation structure is quite different than MLPs and CNNs.

In particular, we attempted to implement FiLM-Ensemble Turkoglu et al. [] on a self-attention network, given the promising results reported by its authors. However, the authors themselves noted that applying FiLM-Ensemble to transformers is not straightforward, mainly because transformers rely on LayerNorm, whereas FiLM-Ensemble was developed with BatchNorm in mind. Our experiments confirmed that directly using BatchNorm in transformers led to notable performance degradation. We explored several approaches to adapt LayerNorm, but the most effective results were achieved by fixing all affine parameters for each ensemble member. This allowed for slight initial variations to introduce randomness and diversity, while keeping the variation among members minimal. The results, summarized in Tab. 13, show that increasing the ensemble size slightly improved accuracy, though the Expected Calibration Error (ECE) fluctuated without consistent improvement. In fact, when using larger ensemble sizes, such as 8 or 16, both accuracy and calibration worsened across all settings we tested.

**Table 13: Performance of FiLM-Ensemble for Vision Transformer (ViT) on CIFAR-10. Increasing the ensemble size slightly improves accuracy, but ECE fluctuates without showing consistent improvement.**
| # ensemble members | Accuracy ($\uparrow$) | ECE ($\downarrow$) |
| --- | --- | --- |
| 1 | 90.54 | 0.0286 |
| 2 | 91.18 | 0.0269 |
| 4 | 91.23 | 0.0289 |

## Appendix N Definitions of Evaluation Metrics

We primarily evaluate our models on accuracy and Expected Calibration Error [ECE, Guo et al., ]. In addition to accuracy and \glsxtrlongece, we have calculated several other scores that have been used in the context of probabilistic deep learning. In the following section, we present the formulations used in our implementations.

### N.1 Accuracy

The accuracy is implemented instance-wise as follows:

$$ $\mathrm{Acc}=\frac{1}{N}\sum_{i=1}^{N}\frac{\lvert\hat{y}_{i}\cap y_{i}\rvert} {\lvert\hat{y}_{i}\cup y_{i}\rvert}$ (8) $$

Here $y_{i}$ denotes the true label of the sample $i$, $\hat{y}_{i}$ is the predicted label of the sample $i$, and $N$ means the total number of samples.

### N.2 Expected Calibration Error

The \glsxtrlongece is a widely used metric for measuring the calibration of neural networks. We use the definition given in Guo et al. []. \glsxtrshortece is defined as the expected difference between accuracy and confidence across several bins. We first need to define accuracy and confidence per bin $B_{m}$ as follows:

$$ $\displaystyle\mathrm{Acc}(B_{m})$ $\displaystyle=\frac{1}{\lvert B_{m}\rvert}\sum_{i\in B_{m}}\mathbf{1}(\hat{y}_ {i}=y_{i}),$ (9) $\displaystyle\mathrm{Conf}(B_{m})$ $\displaystyle=\frac{1}{\lvert B_{m}\rvert}\sum_{i\in B_{m}}\hat{p}_{i}.$ (10) $$

Again, $y_{i}$ and $\hat{y}_{i}$ denote the true and predicted labels of sample $i$ respectively, and $\hat{p}_{i}$ is the predicted confidence of sample $i$.
With this the \glsxtrlongece is given as:

$$ $\mathrm{ECE}=\sum_{m=1}^{M}\frac{\lvert B_{m}\rvert}{n}\lvert\mathrm{Acc}(B_{m })-\mathrm{Conf}(B_{m})\rvert$ (11) $$

### N.3 Macro F1-score

$$ $F1=\frac{1}{C}\sum_{j=1}^{C}\frac{2p_{j}r_{j}}{p_{j}+r_{j}},$ (12) $$

where $r_{j}$ represents the Recall of class $j$, defined as $r_{j}=\frac{TP}{TP+FN}$, and $p_{j}$ represents the Precision of class $j$, defined as $p_{j}=\frac{TP}{TP+FP}$, and $C$ refers to the number of classes, Here, $TP$, $FP$, and $FN$ denote True Positives, False Positives, and False Negatives respectively.

### N.4 Negative Log-Likelihood (NLL)

$$ $NLL=-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{C}\left(y_{i,j}\log\hat{p}_{i,j} \right)=-\frac{1}{N}\sum_{i=1}^{N}\log\hat{p}_{i},$ (13) $$

where $N$ denotes the number of datapoints, $C$ the number of classes, $y_{i,j}$ is 1 if the true label of point $i$ is $j$ and 0 otherwise and $\hat{p}_{i,j}$ is the predicted probability of sample $i$ belonging to class $j$.

### N.5 Brier score

For Brier score we take the definition by Brier [], which is as follows:

$$ $BS=\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{C}(\hat{p}_{i,j}-y_{i,j})^{2},$ (14) $$

where $N$ denotes the number of datapoints, $C$ the number of classes, $y_{i,j}$ is 1 if the true label of point $i$ is $j$ and zero otherwise and $\hat{p}_{i,j}$ is the predicted probability of sample $i$ belonging to class $j$.

### N.6 Area Under the Receiver Operating Characteristic Curve (AUROC)

The AUROC score evaluates the performance of a binary classifier by measuring its ability to distinguish between positive and negative classes, as introduced by Hanley and McNeil []. In our out-of-distribution (OOD) detection experiments, the positive class corresponds to an in-distribution sample, while the negative class corresponds to an out-of-distribution sample.

The AUROC is computed as the area under the ROC curve, which plots the true positive rate (TPR) against the false positive rate (FPR) across various decision thresholds. The TPR and FPR are defined as follows:

$$ $\displaystyle\mathrm{TPR}$ $\displaystyle=\frac{\text{TP}}{\text{TP}+\text{FN}},$ (15) $$

$$ $\displaystyle\mathrm{FPR}$ $\displaystyle=\frac{\text{FP}}{\text{FP}+\text{TN}},$ (16) $$

where TP, FP, FN, and TN represent the true positives, false positives, false negatives, and true negatives, respectively.

The AUROC score is given by the following integral:

$$ $\mathrm{AUROC}=\int_{0}^{1}\mathrm{TPR}(\mathrm{FPR}),d\mathrm{FPR}.$ (17) $$

A higher AUROC score indicates better classification performance, with a score of 1 representing a perfect classifier, and a score of 0.5 indicating performance equivalent to random chance.

### N.7 Area Under the Precision-Recall Curve (AUPRC)

The Area Under the Precision-Recall Curve (AUPRC) assesses the performance of a binary classifier by measuring its ability to accurately identify positive instances, as described by Davis and Goadrich []. In our out-of-distribution (OOD) detection experiments, the positive class corresponds to in-distribution samples, while the negative class corresponds to out-of-distribution samples.

The AUPRC is calculated as the area under the Precision-Recall (PR) curve, which plots precision against recall at various decision thresholds. Precision and recall are defined as follows:

$$ $\mathrm{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}},$ $$

$$ $\mathrm{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}},$ $$

where TP, FP, and FN represent true positives, false positives, and false negatives, respectively.

The AUPRC score is the integral of precision with respect to recall, expressed as:

$$ $\mathrm{AUPRC}=\int_{0}^{1}\mathrm{Precision}(\mathrm{Recall})\,d\mathrm{ Recall}.$ $$

A higher AUPRC score indicates better classifier performance in recognizing positive instances, with a score near 1 representing a good classifier, characterized by both high recall and high precision. This metric is especially valuable for evaluating classifiers on imbalanced datasets.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: Did the authors did any comparisons where they fix the number of parameters, or computational budget (at training or testing) between LoRA ensembles and other baselines? Such comparisons would be very relevant.
soundness: 3
strengths: * S1. Creating lightweight ensembles, requiring fewer resources at training time and test time is a good and relevant direction.

* S2. The approach of using LoRA in ensembles is sound and has good potential.
summary: The paper creates ensembles by starting from a base model and adapting each member independently using LoRA adapters. Each LoRA module adds a small, low-rank learnable matrix to the MLP of Transformers. Experiments on datasets like CIFAR-100, HAM10000, ESC-50 show that these ensembles can give better performance and calibration.
weaknesses: * W1. The approach is simple, ensembling is a classic technique and LoRA is one of the most used ways to fine-tune a pre-trained model in the context of large models. Having a simple method is usually a good aspect, but when the method is straightforward we need a higher bar for results and investigations. In this paper, experiments are mostly made on small datasets and it doesn’t seem like we gain that much from them. Also, combining ensembling and LoRA has been investigated in the context of LLMs [A]. It doesn’t seem like this paper has any additional insights. The authors can also check the reviews of [A] on Openreview for additional related work.

* W2. Small scale experiments. The experiments in this paper are rather on a small scale. Given the simplicity of the method, we need to evaluate it more thoroughly. 

* W3. More baselines are needed. It seems like adding a LoRA module to Single Network gives an important boost, comparable to the ensembling (3% vs 5.9% on Cifar100 classification, 4.8% vs 6.5% on AURPC in OOD detection). It seems that benefits come from adding more learnable parameters, either by LoRA or by ensembling. Thus a single Single Network with a bigger LoRA module (similar number of parameters as the LoRA ensemble) would be a good baseline. Training single, or ensemble method with the same number of parameters would also represent good additional baselines.

[A] Wang et al. LoRA ensembles for large language model fine-tuning. 2023
https://openreview.net/forum?id=X5VElAKt2s

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: While it is reasonable to use CIFAR100 as the in-distribution data and CIFAR-10 as the out-of-distribution data, it would be more comprehensive to include additional out-of-distribution datasets as well. Additionally, it would be beneficial to provide a detailed comparison with existing benchmarks, such as OpenOOD[1], which is widely used for evaluating out-of-distribution detection methods. 
[1] OpenOOD: Benchmarking Generalized Out-of-Distribution Detection. arXiv:2210.07242 [cs.CV]
soundness: 2
strengths: 1.	Improved uncertainty calibration: the authors demonstrate that LoRA-Ensemble achieves superior calibration compared to explicit ensembles. This means that the predicted probabilities of the model align more closely with the true probabilities, resulting in more reliable and accurate predictions. 
2.	Computational and memory efficiency: One strength of the LoRA-Ensemble method is its ability to emulate an ensemble model without the need for separate ensemble members. This makes it computationally and memory efficient compared to explicit ensemble methods, which require training and storing multiple models.
summary: In this paper, the authors address the challenge of overconfident and uncalibrated predictions in machine learning algorithms, specifically in state-of-the-art neural networks like transformers. They propose a parameter-efficient deep ensemble method called LoRA-Ensemble, which is based on Low-Rank Adaptation (LoRA). This method emulates the ensemble model without the need for separate ensemble members, making it computationally and memory efficient. By training member-specific low-rank matrices for attention projections in a single pre-trained self-attention network, LoRA-Ensemble achieves superior calibration compared to explicit ensembles and performs well across various prediction tasks and datasets. The authors demonstrate the effectiveness of LoRA-Ensemble in different classification tasks, including image labeling, skin lesion classification, sound classification, and out-of-distribution detection.
weaknesses: 1.	Theoretical analysis is lacking regarding the relationship between the LoRA ensemble and probability distribution. It would be beneficial to provide a theoretical proof of how the LoRA ensemble can approximate the underlying distribution more effectively.
2.	Additionally, the analysis on why the LoRA ensemble can prevent the degeneration of ensemble members into a point estimate is missing. It would be helpful to include a discussion on how the diversities of ensemble members are maintained through random initialization.
3.	Lack of evaluation on large-scale datasets. LoRA may face overfitting issues, especially when applied to complex tasks. It would be valuable to assess its performance on larger datasets, such as ImageNet-1K, to gain a better understanding of its capabilities in such scenarios.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see and address the comments on the weakness section. Additionally, here are some clarifying questions I have. 
- Considering the relatively simple datasets, how would the approach scale to large datasets such as imagenet?
- For OOD experiment with Cifar100, it would be interesting to see performance on SVHN as the OOD dataset. I think SVHN is more realistic OOD for Cifar100/Cifar10 comapred to using CIfar10 as OOD for Cifar100.
soundness: 3
strengths: - The paper is generally well written and easy to follow. 
- The proposed ensemble method: LoRA-Ensemble is intuitive, and simple. It can easily be extended to different transformer-based classification tasks. 
- LoRA-Ensemble improves both the predictive accuracy and the calibration performance on the considered classification tasks. 
- Propoed method is empirically validated and is parameter/computation efficient.
summary: The paper introduces LoRA-Ensemble, an method for parameter-efficient ensebmling for self-attention networks. Key idea is to have different low-rank matrices for each ensemble component keeping the backbone frozen. The LoRA matrices are applied only to the self-attention layer making the proposed approach parameter efficient. The proposed method is computationally efficient, improves over traditional ensemble methods, and improves both the accuracy and calibration for multiple classification tasks.
weaknesses: - Limited Theoretical insights and empirical validation: The proposed approach is heuristic and is only empirically validated on limited datasets that are relatively simple. Though challenging, the work could be strengthened with some theoretical analysis and guarantees for the proposed technique.  Alternatively, the work could be significantly strengthened by carrying out experiments on more challenging vision datasets of Fine-Grained Visual Classification tasks (see Visual Prompt tuning by Jia et al), Visual Task Adaptation Benchmark/VTAB ( A large-scale study of representation learning with the visual task adaptation benchmark, Zhai et al ), and/or tasks beyond classification (maybe for segmentation/object detection).

- Paper formatting and presentation could be improved: For eg. Table 2, Table 3, Table 7 all overflow.

- For codes, it would have been helpful if codes were included as supplementary material, or at an annonymized github link.

- Post-hoc and other calibration techniques be introduced to further improve the calibration/under-confidence issue, and miscalibration in the baseline models. Discussion and empirical analysis of calibration techniques seems to be missing. 

- Some important works in  epistemic uncertainty estimation are missing: The authors miss literature on Evidential Deep Learning (A Comprehensive Survey on Evidential Deep Learning and Its Applications), and second order based UQ  (Second-Order Uncertainty Quantification: A Distance-Based Approach). These works propose computationally efficient alternatives for UQ, can be extended to DL approaches, and can enable UQ a single forward pass as they do not involve the computational overhead of ensemble. Discussion, and comparison with these works could further strengthen the work.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * How effective is using a mixture-of-experts approach as an “ensemble” for uncertainty evaluation, and how does it compare to this method in terms of efficiency and uncertainty quality?

* Given the method’s focus on attention-based models, how well does it work for MLPs or CNNs?

* What specific advantages does this method offer over applying BatchEnsembles to attention heads?

* The appendix suggests existing efficient ensembling methods don’t apply to Transformers. What specifically prevents methods like BatchEnsembles, Masksembles, or PackedEnsembles from being effective here?
soundness: 3
strengths: * The paper is clearly written and easy to follow. The idea is intuitive and easy to grasp. The related work section provides an adequate discussion of existing approaches to both LoRA and uncertainty estimation for deep learning. The analysis narrative, with the presented drawbacks of existing methods (such deep ensembles), is very clear and easy to understand

* The method is straightforward to apply, requiring only a (potentially pretrained) model and the training of a number of LoRA adapters. During inference, predictions from these adapters are averaged, making implementation efficient and compatible with existing architectures

* The approach is significantly more efficient than traditional (deep-)ensembling, offering reduced computational and memory overhead while maintaining strong performance

* The method demonstrates strong results across a wide range of benchmarks, showing its versatility. This includes performance on image classification tasks (such as CIFAR and HAM10000), as well as on audio classification (ESC-50).
summary: In this paper, the authors propose a novel method for uncertainty estimation in neural networks based on the low-rank adaptation (LoRA) approach. In a nutshell, the method trains several sets of low-rank matrices (commonly used for adaptation/fine-tuning of a pretrained model) and, during inference, runs predictions using all of them, then averages the outputs. Experiments on several benchmarks across various domains demonstrate that the proposed method outperforms several popular ensembling approaches, including (in some cases) deep ensembling. The method improves the final model's performance (e.g., in terms of accuracy), aleatoric uncertainty (measured by calibration metrics), and epistemic uncertainty (in terms of OOD detection quality). At the same time, the method induces significantly lower computational and memory overhead, as well as faster training, compared to conventional ensembles. The proposed approach is relatively simple to use with various attention-based architectures, although it is somewhat limited to them.
weaknesses: * The method is primarily limited to attention-based models, restricting its applicability across model architectures.

* A potential limitation is the method's similarity to existing method BatchEnsembles, as both approaches utilize low-rank adapters to create ensembles. Given this mentioned similarity a question arises: what advantages does the proposed method offer over directly applying BatchEnsembles to attention heads?

* Experiments demonstrates mixed results, making it unclear whether the method consistently outperforms existing approaches. For example, in CIFAR experiments it’s unclear where the significant performance improvement over a single model (without a LoRA adapter) comes from. The calibration performance in terms of ECE may also stem from discrepancies in accuracy. Similarly, ESC-50 experiments show that deep ensembles outperform the proposed method.

* In addition to the previous point, the authors do not compare against other compute-efficient ensembling approaches. In the appendix, they mention that existing methods cannot be applied to Transformer architectures, as these differ significantly from MLP and CNN-based models. However, BatchEnsembles or Masksembles (and not mentioned PackedEnsembles [1]) could be viewed as approaches that apply perturbations to trainable model weights (matrices), therefore the impossibility to apply them to Transformers is unclear.

* An important limitation is the absence of experiments on text data, despite LoRA’s popularity and proven effectiveness within this domain.

[1] Laurent, Olivier, et al. "Packed-ensembles for efficient uncertainty estimation." ICLR (2023).

Meta-review: (a) Summary of Scientific Claims and Findings
This paper introduces LoRA-Ensemble, a parameter-efficient deep ensemble method for self-attention networks. The method leverages Low-Rank Adaptation (LoRA) modules, typically used for fine-tuning, to implicitly create ensemble members by training low-rank matrices while sharing the backbone model’s weights. The method claims to:
Reduce computational and memory costs compared to explicit ensembles.
Achieve superior calibration and accuracy across various classification tasks (e.g., CIFAR-100, HAM10000, ESC-50) and out-of-distribution (OOD) detection.
Exhibit enhanced diversity in ensemble members, contributing to improved epistemic uncertainty modeling.

(b) Strengths of the Paper
Efficiency: The proposed method significantly reduces the computational and memory overhead typically associated with explicit ensembles.
Integration: The use of LoRA modules to create implicit ensembles within self-attention networks is innovative and aligned with trends in efficient fine-tuning.
Broad Applicability within Transformers: Experiments demonstrate the method’s compatibility across multiple Transformer variants, such as ViT and DeiT, and application domains like medical imaging and sound classification.
Improved Calibration: The paper highlights LoRA-Ensemble’s strong performance in epistemic uncertainty modeling and calibration, with empirical results showing improvements over explicit ensembles.

(c) Weaknesses of the Paper
Limited Novelty: The proposed approach closely resembles existing methods, such as BatchEnsemble, which also utilize low-rank adaptations for ensembling. The distinction between LoRA-Ensemble and BatchEnsemble is not convincingly demonstrated, raising concerns about originality.
Claims of novelty are further weakened by a lack of comparisons with key baselines like BatchEnsemble and Masksembles.
Theoretical Gaps: Theoretical justification for why LoRA-Ensemble maintains diversity among ensemble members or improves calibration and uncertainty estimation is lacking.
The absence of analytical insights into the method’s relationship with probability distributions limits its scientific contribution.
Incomplete Evaluation: Experiments are primarily conducted on relatively small-scale datasets, such as CIFAR-100 and HAM10000, which fail to demonstrate the method’s scalability and generalizability to larger, more complex datasets (e.g., ImageNet).
Missing evaluations on text-based tasks, where LoRA is highly relevant, reduce the scope and impact of the work.
Baseline Weaknesses: Critical baselines, such as BatchEnsemble, are omitted, despite their relevance to parameter-efficient ensembling.
The absence of consistent improvements across all benchmarks raises questions about the robustness of the proposed method.

(d) Reasons for Rejection
Limited Novelty and Scope: The similarities with BatchEnsemble, combined with the lack of comparative analysis, significantly undermine the perceived originality of the work. Furthermore, the focus on self-attention networks limits the method’s general applicability.
Weak Theoretical Contribution: The absence of theoretical grounding for the observed performance gains limits the paper’s scientific merit.
Inadequate Evaluation: The reliance on small-scale datasets and the exclusion of key benchmarks (e.g., BatchEnsemble) prevent a comprehensive assessment of the method’s efficacy and scalability.
Unresolved Reviewer Concerns: Despite an extensive rebuttal, key issues—including scalability, baseline omissions, and novelty—remain inadequately addressed.
While LoRA-Ensemble demonstrates potential as a computationally efficient ensembling technique, it lacks the necessary breadth, rigor, and originality to warrant acceptance at ICLR.

Decision: Reject