LoRA-Ensemble: Efficient Uncertainty Modelling for Self-attention Networks

Authors: Michelle Halbheer, Dominik J. Mühlematter, Alexander Becker, Dominik Narnhofer, Helge Aasen, Konrad Schindler, Mehmet Ozgur Turkoglu
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: uncertainty, ensemble, implicit ensemble, calibration, self-attention, transformer

Abstract:
Numerous crucial tasks in real-world decision-making rely on machine learning algorithms with calibrated uncertainty estimates. However, modern methods often yield overconfident and uncalibrated predictions. Various approaches involve training an ensemble of separate models to quantify the uncertainty related to the model itself, known as epistemic uncertainty. In an explicit implementation, the ensemble approach has high computational cost and high memory requirements. This particular challenge is evident in state-of-the-art neural networks such as transformers, where even a single network is already demanding in terms of compute and memory. Consequently, efforts are made to emulate the ensemble model without actually instantiating separate ensemble members, referred to as implicit ensembling. We introduce LoRA-Ensemble, a parameter-efficient deep ensemble method for self-attention networks, which is based on Low-Rank Adaptation (LoRA). Initially developed for efficient LLM fine-tuning, we extend LoRA to an implicit ensembling approach. By employing a single pre-trained self-attention network with weights shared across all members, we train member-specific low-rank matrices for the attention projections. Our method exhibits superior calibration compared to explicit ensembles and achieves similar or better accuracy across various prediction tasks and datasets.

Decision: reject
Reviewer scores: 3, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730396250797

thread_type:
official_review

<thread_object>
creator: reviewer_tsvt

<content>
summary:
In this paper, the authors propose a novel method for uncertainty estimation in neural networks based on the low-rank adaptation (LoRA) approach. In a nutshell, the method trains several sets of low-rank matrices (commonly used for adaptation/fine-tuning of a pretrained model) and, during inference, runs predictions using all of them, then averages the outputs. Experiments on several benchmarks across various domains demonstrate that the proposed method outperforms several popular ensembling approaches, including (in some cases) deep ensembling. The method improves the final model's performance (e.g., in terms of accuracy), aleatoric uncertainty (measured by calibration metrics), and epistemic uncertainty (in terms of OOD detection quality). At the same time, the method induces significantly lower computational and memory overhead, as well as faster training, compared to conventional ensembles. The proposed approach is relatively simple to use with various attention-based architectures, although it is somewhat limited to them.

strengths:
* The paper is clearly written and easy to follow. The idea is intuitive and easy to grasp. The related work section provides an adequate discussion of existing approaches to both LoRA and uncertainty estimation for deep learning. The analysis narrative, with the presented drawbacks of existing methods (such deep ensembles), is very clear and easy to understand

* The method is straightforward to apply, requiring only a (potentially pretrained) model and the training of a number of LoRA adapters. During inference, predictions from these adapters are averaged, making implementation efficient and compatible with existing architectures

* The approach is significantly more efficient than traditional (deep-)ensembling, offering reduced computational and memory overhead while maintaining strong performance

* The method demonstrates strong results across a wide range of benchmarks, showing its versatility. This includes performance on image classification tasks (such as CIFAR and HAM10000), as well as on audio classification (ESC-50).

weaknesses:
* The method is primarily limited to attention-based models, restricting its applicability across model architectures.

* A potential limitation is the method's similarity to existing method BatchEnsembles, as both approaches utilize low-rank adapters to create ensembles. Given this mentioned similarity a question arises: what advantages does the proposed method offer over directly applying BatchEnsembles to attention heads?

* Experiments demonstrates mixed results, making it unclear whether the method consistently outperforms existing approaches. For example, in CIFAR experiments it’s unclear where the significant performance improvement over a single model (without a LoRA adapter) comes from. The calibration performance in terms of ECE may also stem from discrepancies in accuracy. Similarly, ESC-50 experiments show that deep ensembles outperform the proposed method.

* In addition to the previous point, the authors do not compare against other compute-efficient ensembling approaches. In the appendix, they mention that existing methods cannot be applied to Transformer architectures, as these differ significantly from MLP and CNN-based models. However, BatchEnsembles or Masksembles (and not mentioned PackedEnsembles [1]) could be viewed as approaches that apply perturbations to trainable model weights (matrices), therefore the impossibility to apply them to Transformers is unclear.

* An important limitation is the absence of experiments on text data, despite LoRA’s popularity and proven effectiveness within this domain.

[1] Laurent, Olivier, et al. "Packed-ensembles for efficient uncertainty estimation." ICLR (2023).

questions:
* How effective is using a mixture-of-experts approach as an “ensemble” for uncertainty evaluation, and how does it compare to this method in terms of efficiency and uncertainty quality?

* Given the method’s focus on attention-based models, how well does it work for MLPs or CNNs?

* What specific advantages does this method offer over applying BatchEnsembles to attention heads?

* The appendix suggests existing efficient ensembling methods don’t apply to Transformers. What specifically prevents methods like BatchEnsembles, Masksembles, or PackedEnsembles from being effective here?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638319303

thread_type:
official_review

<thread_object>
creator: reviewer_vyv1

<content>
summary:
The paper introduces LoRA-Ensemble, an method for parameter-efficient ensebmling for self-attention networks. Key idea is to have different low-rank matrices for each ensemble component keeping the backbone frozen. The LoRA matrices are applied only to the self-attention layer making the proposed approach parameter efficient. The proposed method is computationally efficient, improves over traditional ensemble methods, and improves both the accuracy and calibration for multiple classification tasks.

strengths:
- The paper is generally well written and easy to follow. 
- The proposed ensemble method: LoRA-Ensemble is intuitive, and simple. It can easily be extended to different transformer-based classification tasks. 
- LoRA-Ensemble improves both the predictive accuracy and the calibration performance on the considered classification tasks. 
- Propoed method is empirically validated and is parameter/computation efficient.

weaknesses:
- Limited Theoretical insights and empirical validation: The proposed approach is heuristic and is only empirically validated on limited datasets that are relatively simple. Though challenging, the work could be strengthened with some theoretical analysis and guarantees for the proposed technique.  Alternatively, the work could be significantly strengthened by carrying out experiments on more challenging vision datasets of Fine-Grained Visual Classification tasks (see Visual Prompt tuning by Jia et al), Visual Task Adaptation Benchmark/VTAB ( A large-scale study of representation learning with the visual task adaptation benchmark, Zhai et al ), and/or tasks beyond classification (maybe for segmentation/object detection).

- Paper formatting and presentation could be improved: For eg. Table 2, Table 3, Table 7 all overflow.

- For codes, it would have been helpful if codes were included as supplementary material, or at an annonymized github link.

- Post-hoc and other calibration techniques be introduced to further improve the calibration/under-confidence issue, and miscalibration in the baseline models. Discussion and empirical analysis of calibration techniques seems to be missing. 

- Some important works in  epistemic uncertainty estimation are missing: The authors miss literature on Evidential Deep Learning (A Comprehensive Survey on Evidential Deep Learning and Its Applications), and second order based UQ  (Second-Order Uncertainty Quantification: A Distance-Based Approach). These works propose computationally efficient alternatives for UQ, can be extended to DL approaches, and can enable UQ a single forward pass as they do not involve the computational overhead of ensemble. Discussion, and comparison with these works could further strengthen the work.

questions:
Please see and address the comments on the weakness section. Additionally, here are some clarifying questions I have. 
- Considering the relatively simple datasets, how would the approach scale to large datasets such as imagenet?
- For OOD experiment with Cifar100, it would be interesting to see performance on SVHN as the OOD dataset. I think SVHN is more realistic OOD for Cifar100/Cifar10 comapred to using CIfar10 as OOD for Cifar100.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691649726

thread_type:
official_review

<thread_object>
creator: reviewer_twzw

<content>
summary:
In this paper, the authors address the challenge of overconfident and uncalibrated predictions in machine learning algorithms, specifically in state-of-the-art neural networks like transformers. They propose a parameter-efficient deep ensemble method called LoRA-Ensemble, which is based on Low-Rank Adaptation (LoRA). This method emulates the ensemble model without the need for separate ensemble members, making it computationally and memory efficient. By training member-specific low-rank matrices for attention projections in a single pre-trained self-attention network, LoRA-Ensemble achieves superior calibration compared to explicit ensembles and performs well across various prediction tasks and datasets. The authors demonstrate the effectiveness of LoRA-Ensemble in different classification tasks, including image labeling, skin lesion classification, sound classification, and out-of-distribution detection.

strengths:
1.	Improved uncertainty calibration: the authors demonstrate that LoRA-Ensemble achieves superior calibration compared to explicit ensembles. This means that the predicted probabilities of the model align more closely with the true probabilities, resulting in more reliable and accurate predictions. 
2.	Computational and memory efficiency: One strength of the LoRA-Ensemble method is its ability to emulate an ensemble model without the need for separate ensemble members. This makes it computationally and memory efficient compared to explicit ensemble methods, which require training and storing multiple models.

weaknesses:
1.	Theoretical analysis is lacking regarding the relationship between the LoRA ensemble and probability distribution. It would be beneficial to provide a theoretical proof of how the LoRA ensemble can approximate the underlying distribution more effectively.
2.	Additionally, the analysis on why the LoRA ensemble can prevent the degeneration of ensemble members into a point estimate is missing. It would be helpful to include a discussion on how the diversities of ensemble members are maintained through random initialization.
3.	Lack of evaluation on large-scale datasets. LoRA may face overfitting issues, especially when applied to complex tasks. It would be valuable to assess its performance on larger datasets, such as ImageNet-1K, to gain a better understanding of its capabilities in such scenarios.

questions:
While it is reasonable to use CIFAR100 as the in-distribution data and CIFAR-10 as the out-of-distribution data, it would be more comprehensive to include additional out-of-distribution datasets as well. Additionally, it would be beneficial to provide a detailed comparison with existing benchmarks, such as OpenOOD[1], which is widely used for evaluating out-of-distribution detection methods. 
[1] OpenOOD: Benchmarking Generalized Out-of-Distribution Detection. arXiv:2210.07242 [cs.CV]
</content>
</thread_object>

</thread>
<thread>
creation date:
1730745815637

thread_type:
official_review

<thread_object>
creator: reviewer_wiq6

<content>
summary:
The paper creates ensembles by starting from a base model and adapting each member independently using LoRA adapters. Each LoRA module adds a small, low-rank learnable matrix to the MLP of Transformers. Experiments on datasets like CIFAR-100, HAM10000, ESC-50 show that these ensembles can give better performance and calibration.

strengths:
* S1. Creating lightweight ensembles, requiring fewer resources at training time and test time is a good and relevant direction.

* S2. The approach of using LoRA in ensembles is sound and has good potential.

weaknesses:
* W1. The approach is simple, ensembling is a classic technique and LoRA is one of the most used ways to fine-tune a pre-trained model in the context of large models. Having a simple method is usually a good aspect, but when the method is straightforward we need a higher bar for results and investigations. In this paper, experiments are mostly made on small datasets and it doesn’t seem like we gain that much from them. Also, combining ensembling and LoRA has been investigated in the context of LLMs [A]. It doesn’t seem like this paper has any additional insights. The authors can also check the reviews of [A] on Openreview for additional related work.

* W2. Small scale experiments. The experiments in this paper are rather on a small scale. Given the simplicity of the method, we need to evaluate it more thoroughly. 

* W3. More baselines are needed. It seems like adding a LoRA module to Single Network gives an important boost, comparable to the ensembling (3% vs 5.9% on Cifar100 classification, 4.8% vs 6.5% on AURPC in OOD detection). It seems that benefits come from adding more learnable parameters, either by LoRA or by ensembling. Thus a single Single Network with a bigger LoRA module (similar number of parameters as the LoRA ensemble) would be a good baseline. Training single, or ensemble method with the same number of parameters would also represent good additional baselines.

[A] Wang et al. LoRA ensembles for large language model fine-tuning. 2023
https://openreview.net/forum?id=X5VElAKt2s

questions:
Did the authors did any comparisons where they fix the number of parameters, or computational budget (at training or testing) between LoRA ensembles and other baselines? Such comparisons would be very relevant.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732610666121

thread_type:
official_comment

<thread_object>
creator: area_chair_rzsf

<content>
title:
Encouragement to Actively Participate in the Discussion Phase

comment:
Dear Reviewers,

Thank you for your valuable contributions to the review process so far. As we enter the discussion phase, I encourage you to actively engage with the authors and your fellow reviewers. This is a critical opportunity to clarify any open questions, address potential misunderstandings, and ensure that all perspectives are thoroughly considered.

Your thoughtful input during this stage is greatly appreciated and is essential for maintaining the rigor and fairness of the review process.

Thank you for your efforts and dedication.
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Summary of Scientific Claims and Findings
This paper introduces LoRA-Ensemble, a parameter-efficient deep ensemble method for self-attention networks. The method leverages Low-Rank Adaptation (LoRA) modules, typically used for fine-tuning, to implicitly create ensemble members by training low-rank matrices while sharing the backbone model’s weights. The method claims to:
Reduce computational and memory costs compared to explicit ensembles.
Achieve superior calibration and accuracy across various classification tasks (e.g., CIFAR-100, HAM10000, ESC-50) and out-of-distribution (OOD) detection.
Exhibit enhanced diversity in ensemble members, contributing to improved epistemic uncertainty modeling.

(b) Strengths of the Paper
Efficiency: The proposed method significantly reduces the computational and memory overhead typically associated with explicit ensembles.
Integration: The use of LoRA modules to create implicit ensembles within self-attention networks is innovative and aligned with trends in efficient fine-tuning.
Broad Applicability within Transformers: Experiments demonstrate the method’s compatibility across multiple Transformer variants, such as ViT and DeiT, and application domains like medical imaging and sound classification.
Improved Calibration: The paper highlights LoRA-Ensemble’s strong performance in epistemic uncertainty modeling and calibration, with empirical results showing improvements over explicit ensembles.

(c) Weaknesses of the Paper
Limited Novelty: The proposed approach closely resembles existing methods, such as BatchEnsemble, which also utilize low-rank adaptations for ensembling. The distinction between LoRA-Ensemble and BatchEnsemble is not convincingly demonstrated, raising concerns about originality.
Claims of novelty are further weakened by a lack of comparisons with key baselines like BatchEnsemble and Masksembles.
Theoretical Gaps: Theoretical justification for why LoRA-Ensemble maintains diversity among ensemble members or improves calibration and uncertainty estimation is lacking.
The absence of analytical insights into the method’s relationship with probability distributions limits its scientific contribution.
Incomplete Evaluation: Experiments are primarily conducted on relatively small-scale datasets, such as CIFAR-100 and HAM10000, which fail to demonstrate the method’s scalability and generalizability to larger, more complex datasets (e.g., ImageNet).
Missing evaluations on text-based tasks, where LoRA is highly relevant, reduce the scope and impact of the work.
Baseline Weaknesses: Critical baselines, such as BatchEnsemble, are omitted, despite their relevance to parameter-efficient ensembling.
The absence of consistent improvements across all benchmarks raises questions about the robustness of the proposed method.

(d) Reasons for Rejection
Limited Novelty and Scope: The similarities with BatchEnsemble, combined with the lack of comparative analysis, significantly undermine the perceived originality of the work. Furthermore, the focus on self-attention networks limits the method’s general applicability.
Weak Theoretical Contribution: The absence of theoretical grounding for the observed performance gains limits the paper’s scientific merit.
Inadequate Evaluation: The reliance on small-scale datasets and the exclusion of key benchmarks (e.g., BatchEnsemble) prevent a comprehensive assessment of the method’s efficacy and scalability.
Unresolved Reviewer Concerns: Despite an extensive rebuttal, key issues—including scalability, baseline omissions, and novelty—remain inadequately addressed.
While LoRA-Ensemble demonstrates potential as a computationally efficient ensembling technique, it lacks the necessary breadth, rigor, and originality to warrant acceptance at ICLR.