Ensembles of Low-Rank Expert Adapters

Authors: Yinghao Li, Vianne R. Gao, Chao Zhang, MohamadAli Torkamani
Primary area: foundation or frontier models, including LLMs
Keywords: Language Model, LoRA, MoE, Ensembles

Abstract:
The training and fine-tuning of large language models (LLMs) often involve diverse textual data from multiple sources, which poses challenges due to conflicting gradient directions, hindering optimization and specialization.  These challenges can undermine model generalization across tasks, resulting in reduced downstream performance.  Recent research suggests that fine-tuning LLMs on carefully selected, task-specific subsets of data can match or even surpass the performance of using the entire dataset.  Building on these insights, we propose the Ensembles of Low-Rank Expert Adapters (ELREA) framework to improve the model's capability to handle diverse tasks.  ELREA clusters the training instructions based on their gradient directions, representing different areas of expertise and thereby reducing conflicts during optimization.  Expert adapters are then trained on these clusters, utilizing the low-rank adaptation (LoRA) technique to ensure training efficiency and model scalability.  During inference, ELREA combines predictions from the most relevant expert adapters based on the input data's gradient similarity to the training clusters, ensuring optimal adapter selection for each task.  Experiments show that our method outperforms baseline LoRA adapters trained on the full dataset and other ensemble approaches with similar training and inference complexity across a range of domain-specific tasks.

Decision: accept (poster)
Reviewer scores: 5, 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730228406855

thread_type:
official_review

<thread_object>
creator: reviewer_5m8l

<content>
summary:
This paper introduces Ensembles of Low-Rank Expert Adapters (ELREA), an ensemble-style LoRA MoE model to improve task adaptability and mitigate optimization conflicts in LLM fine-tuning. ELREA tackles gradient conflicts from diverse data sources by grouping tasks with similar gradient directions, creating specialized experts for each group. During inference, ELREA dynamically assigns weights to these experts based on gradient similarity with the input task instruction, selecting the most relevant experts for each task. Compared to traditional deep ensembles and standard MoE models, ELREA strikes a balance between performance and computational efficiency. Experimental results validate ELREA’s effectiveness.

strengths:
- The paper is well-written and easy to follow.
- The model introduces a novel approach by designing experts based on gradient directions, rather than relying on an explicitly learned gating function, which is innovative in MoE design.
- The experiments demonstrate the model’s competitive performance across diverse tasks.

weaknesses:
Overall, as the authors acknowledge in the limitations, the main weakness lies in efficiency and computational cost:

- In the inference stage, ELREA requires combining predictions from all expert adapters and the base adapter, resulting in inference costs that increase linearly with the number of experts, which can restrict practical applications.
- During inference, the model needs to first compute and store gradients for different task instructions.
- Additionally, clustering gradients during training introduces extra computational overhead.
- Although ELREA shows stronger performance compared to baselines like "MoE Routing", this advantage is somewhat diminished by its high inference costs.

The baseline implementations could be more complete:

- The current implementation of "MoE Routing" bypasses training the MoE gate and instead directly uses weights similar to ELREA’s ensemble approach. However, training the MoE gate separately at each layer would make the baseline comparison more robust.

The marginal benefit of ELREA diminishes as the backbone model scales while the inference cost significantly increases. Additionally, there is limited comparison with a 9B model in the experimental results.

questions:
1. Could ELREA maintain most of its performance by the ensemble of sparse experts, like the top-4 or top-1 experts in addition to the base adapter, to reduce inference costs?

2. Regarding the completeness of baselines, there could be an additional version of "MoE Routing" similar to those implemented in MoLE [1] or LLaVA-MoLE [2]. Further comparisons with other LoRA MoE methods like LoraHub [3] will make the baseline comparisons more robust.

3. Is the clustering process a time-intensive part of the training process?

[1] Wu, Xun, Shaohan Huang, and Furu Wei. "Mixture of LoRA Experts." The Twelfth International Conference on Learning Representations, 2024, https://openreview.net/forum?id=uWvKBCYh4S.

[2] Chen, Shaoxiang, Zequn Jie, and Lin Ma. "Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning mllms." *arXiv preprint arXiv:2401.16160* (2024).

[3] Huang, Chengsong, et al. "Lorahub: Efficient cross-task generalization via dynamic lora composition." *arXiv preprint arXiv:2307.13269* (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730349684686

thread_type:
official_review

<thread_object>
creator: reviewer_ue2l

<content>
summary:
The paper explores the issue of gradient conflict during multi-source data training. It addresses this problem by clustering gradients and training LoRA for each cluster, followed by ensembling to avoid the conflicts. Subsequent experiments on multiple datasets validate the effectiveness of the proposed method.

strengths:
Using gradient features for clustering provides a more direct solution to the gradient conflict issue compared to clustering based on sentence embeddings, resulting in improved performance.

weaknesses:
1. The paper does not sufficiently justify the choice of ensembling over a mixture of experts (MoE) approach. It mentions that ensembling performs better in value prediction and uncertainty estimation; however, this may not hold true for NLP tasks. In fact, MoE can optimize routing at a finer granularity, which may yield better performance compared to ensembling. This should be discussed in the paper. Additionally, a comparison should be made during the second stage of training cluster-specific LoRA, where a router could be trained alongside (with similar computational overhead) instead of using ELREA's weights for routing.
2. The paper needs to address the algorithm's time complexity. The overall process is lengthy and involves additional gradient calculations, leading to higher computational costs compared to other methods. Discussing these costs is essential for the practical application of the proposed method.

questions:
1. Since the training of adapters consists of two stages (base LoRA and cluster LoRA training), my understanding is that the base LoRA's role is to learn general knowledge and assist in subsequent gradient estimation, while the cluster LoRA fine-tunes the model further. Could a higher-rank base LoRA be used, while training lower-rank LoRA for different clusters, to achieve faster adaptation?
2. Regarding clustering by gradients, what patterns exist within the data in each cluster? Are the tasks within each cluster more similar to each other?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730365088330

thread_type:
official_review

<thread_object>
creator: reviewer_pb5a

<content>
summary:
The authors presented a framework aimed at enhancing the fine-tuning of large language models (LLMs) by addressing challenges related to conflicting gradient directions. By clustering training instructions according to their gradient features and independently training expert adapters on these clusters using the Low-Rank Adaptation (LoRA) technique, the authors proposed a method to improve training efficiency and model scalability. The framework dynamically selected the most relevant adapters during inference to optimize performance, which was demonstrated through experiments to outperform baseline methods and other ensemble approaches across various domain-specific tasks.

strengths:
1.	The authors explained the challenges of fine-tuning large models from the perspective of gradient conflicts, which is a novel approach. They then propose clustering training samples based on their gradients and training them separately using LoRA. By integrating multiple LoRA adapters, they enhance the performance of LoRA fine-tuning.

2.	The authors’ writing structure is well-organized and the text is clear and easy to understand. Additionally, the research methodology is rigorous, with detailed experiments conducted.

weaknesses:
1.	The authors did not discuss the complexity of the method. Compared to the vanilla LoRA, this approach introduces a clustering process and the training of multiple LoRAs, which inevitably increases the complexity of training. Additionally, computing different LoRA weight combinations for each sample could also lead to a decline in inference performance. Although the author's method does indeed improve accuracy, whether it is practical in real-world scenarios requires further discussion.

2.	The authors’ experimental comparisons are not sufficiently comprehensive. Currently, there are many methods dedicated to enhancing LoRA's performance on multitasking, such as MixLoRA[1], MultiLoRA[2], and MOELoRA[3]. However, the MOE methods compared by the author appear to be just modified versions of their own approach, which might be more suitable for inclusion in an ablation study. If the authors could compare their method with these more advanced techniques, it would further demonstrate the effectiveness of the proposed method.

[1] Li D, Ma Y, Wang N, et al. Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts[J]. arXiv preprint arXiv:2404.15159, 2024.

[2] Wang Y, Lin Y, Zeng X, et al. Multilora: Democratizing lora for better multi-task learning[J]. arXiv preprint arXiv:2311.11501, 2023.

[3] Liu Q, Wu X, Zhao X, et al. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications[J]. arXiv preprint arXiv:2310.18339, 2023.

3. The authors use the random clustering approach for training, and the performance is only slightly worse than the gradient-based clustering method. This suggests that the gradient-based approach may not be essential, which contradicts the overall motivation of the method.

questions:
1.	Could you have provided more details on the computational overhead associated with the ELREA framework?

2.	You use 5,000 samples for clustering, which is a relatively small portion of the entire training set. If the sample changes, could this lead to different clustering results and significantly alter the final training outcomes? In other words, does this clustering method possess stability?

3.	For the experiments where the performance gains from gradient-based clustering are far less significant than those from LoRA ensembling, can you provide a reasonable explanation? If clustering is not used, and each LoRA is trained using different random seeds and then ensembled, could this also achieve good performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730469348250

thread_type:
official_review

<thread_object>
creator: reviewer_wm5h

<content>
summary:
This paper proposes a novel two-stage fine-tuning method for large language models, named ELREA. In the first stage, ELREA trains a general adapter on the entire fine-tuning dataset. In the second stage, it clusters the training data into multiple groups and trains a specialized adapter for each group. During inference, ELREA dynamically selects the most suitable adapter based on the gradient similarity between the input data and the training clusters. Experimental results confirm the effectiveness of ELREA.

strengths:
1. The paper is well-written and easy to understand.
2. The combination of LoRA with gradient-based clustering for expert adapters to address the conflicting gradient issue is interesting.
3. The framework’s design is well-structured, and the authors provide extensive experimental validation to demonstrate the efficacy of ELREA.

Since I am not very familiar with the latest work in the field of LLM fine-tuning, I may not be in the best position to assess the novelty of this paper. This paper is interesting for me. I am inclined to accept and respect the opinions of other reviewers.

weaknesses:
1. As discussed by the authors in the limitations section, the computational and memory overhead of ELREA is substantial.
2. The paper lacks an in-depth discussion on the robustness of the clusters and the impact of the clustering algorithm (BIRCH) on performance. Since clustering is central to ELREA, variations in cluster formation could significantly affect adapter performance.

questions:
Why is the base adapter still used during the inference phase? Since the cluster adapters are derived from the base adapter, they should already contain the necessary information from it. Could the authors’ inclusion of the base adapter at inference suggest that the cluster adapters may have forgotten some useful information from other clusters during training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730595231789

thread_type:
official_review

<thread_object>
creator: reviewer_vseh

<content>
summary:
This paper presents Ensembles of Low-Rank Expert Adapters (ELREA), a framework that combines efficient adaptation techniques to resolve conflicting gradients in LLM fine-tuning. By leveraging gradient clustering, the study creates expert adapters for diverse tasks, independent of specific data features. Results indicate that ELREA outperforms baseline LoRA adapters and other expert and self-consistency methods across various applications.

strengths:
1.  The presentation is well.
2.  The method section of the paper is clearly articulated, especially Figure 1, which is a clear and self-explanatory flowchart. 
3.  Major differences of MoE and Deep Ensembles are highlighted (P.4). Assumptions (P.5) and limitations (P.10) are clearly stated.
4.  Included detailed appendices to further explain the datasets, experiment setups so the main body is not bulky.

weaknesses:
Missing Paper Structure. The section “Related Work” should be better put in the Appendix.

Limit of Novelty. This paper proposes combine two points together creatively. However, the method for data selection and partitioning (Section 3.2) directly depends on [1]. Moreover, the per-cluster fine-tuning actually has little to do with LoRA; for instance, we could also use full-parameter or soft-prompt tuning for each cluster. Therefore, the motivation for exclusive “LoRA” is not adequate, which needs further illustration.

Lack of comparison with other “LoRA” baselines. It would be more complete if some mixture-of-lora-expert methods[2,3] are included to compare appropriately. 

Some writing typos that will not affect the rating scores:

* A few minor grammatical mistakes are found (e.g. P.4: which is the most adapted for LM fine-tuning; P.6: we the weights do not need to sum to 1).

* Undefined symbols: x_{t} in (1) on P.2; d_{model} on line 2 of P.3; cos’ in (9) on P.6 [they are interpretable but the authors should state clearly].

* Conflicting symbol: sigma^bar_c on the line above (8) and in (8) have different formulae on P.5.

* Figure 1: The paragraph following the section heading “3 Method” (P.4) said “The pipeline, shown in Figure 1, consists of three main steps” but the “steps” are not clearly shown in Figure 1. Simply speaking, the 4 phrases in Figure 1 do not match the 3 steps.

* The reference item “Improving language understanding by generative pre- training” (Radford et al.) on P.15 seems to be incomplete. A possible URL: https://cdn.openai.com/research-covers/language- unsupervised/language_understanding_paper.pdf

[1] LESS: Selecting Influential Data for Targeted Instruction Tuning. 2024 ICML.
[2] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning. 2024
NeurIPS.
[3] Mixture of LoRA Experts. 2024 ICLR.

questions:
Please see the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732250967609

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall response to all reviewers

comment:
Dear Reviewers,

We sincerely thank you for the time and effort you have dedicated to reviewing our manuscript and for providing thoughtful and constructive feedback. Your valuable suggestions have been instrumental in improving the quality of our work.

In this overall response, we summarize the changes we have made to address your general concerns, presented in order of their importance. For your convenience, we have also included a PDF highlighting the updates made in the revised manuscript as **supplementary material**.

1. **Baselines**:  We have incorporated MoLE [1] (trainable gating functions) and LoRA Ensembles [2] as additional baseline methods in our study. The corresponding results and discussions have been updated in the manuscript, primarily in Section 4, Section 5 (Tables 1 & 2), and Appendix E.


2. **Method efficiency**: To address concerns about efficiency, we have added Appendix G, titled "Efficiency Analysis," which includes Table 5. This section provides a comprehensive discussion of efficiency from both theoretical and empirical perspectives. Overall, we find the computational overhead to be reasonable given the performance improvements achieved.


3. **Cluster patterns**: To further clarify data clustering behavior, we have added Appendix H, titled "Further Analysis on Data Clustering," which includes Figure 4 to illustrate data patterns within clusters. Additionally, we have included a sentence in Section 3.3 to confirm that these patterns are robust to variations in random seeds.


4. **Top-k experts**: We have added Figure 3(b) to illustrate the performance when only the Top-1 or Top-3 experts are activated during inference, along with a new paragraph (Lines 502–506) providing a detailed analysis of the results.


In addition to this overall response, we have provided explanations and addressed specific points in our individual responses to each reviewer. We hope that our revisions and responses adequately address your concerns, and we welcome any additional feedback or questions you may have.

Best regards,  
The Authors

[1] Wu, Xun, Shaohan Huang, and Furu Wei. "Mixture of lora experts." arXiv preprint arXiv:2404.13628 (2024).  
[2] Wang, Xi, Laurence Aitchison, and Maja Rudolph. "LoRA ensembles for large language model fine-tuning." arXiv preprint arXiv:2310.00035 (2023).
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this paper, the authors proposed a new framework for fine-tuning LLMs, where training data is first clustered into different groups based on a global adaptor with the LoRA technique, different expert adaptors are then trained for each group, and finally, an ensemble model is applied to generate the final fine-tuning outcomes.

The idea of the proposed framework is interesting, and the design is reasonable. The performance of the proposed framework is promising. This is a borderline paper, As most of the concerns raised by the reviewers about the complexity, novelty, and comparable experiments of the proposed framework have been addressed during the rebuttal, I recommend acceptance of this paper.