Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

Authors: Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Fei Wu
Primary area: foundation or frontier models, including LLMs
Keywords: Parameter Efficient Tuning, LoRA, Model Merging

Abstract:
Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability on platforms like Huggingface. This modularity has sparked interest in combining multiple LoRAs to significantly enhance LLM capabilities. However, existing methods for LoRA composition primarily focus on task-specific adaptations that require additional training, and current model merging techniques often fail to fully leverage LoRA's modular nature, leading to parameter interference and performance degradation.
In this paper, we explore the possibility of disassembling and reassembling multiple LoRAs at a finer granularity, much like assembling LEGO blocks. We introduce the concept of Minimal Semantic Units (MSUs), where the parameters corresponding to each rank in LoRA function as independent units. These MSUs exhibit properties such as permutation invariance and concatenation-summation equivalence, allowing for flexible combinations to form new LoRAs. Building on these insights, we propose the LoRA-LEGO framework. This framework conducts rank-wise parameter clustering by grouping MSUs from different LoRAs into $k$ clusters. The centroid of each cluster serves as a representative MSU, enabling the assembly of a merged LoRA with an adjusted rank of $k$. Additionally, we apply a dual reweighting strategy to optimize the scale of the merged LoRA. Experiments across various benchmarks demonstrate that our method outperforms existing approaches in LoRA merging.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730455622368

thread_type:
official_review

<thread_object>
creator: reviewer_ys49

<content>
summary:
This paper introduces the concept of Minimal Semantic Unit (MSU) for disassembling and reassembling multiple LoRAs at a finer granularity. MSUs have permutation invariance and concatenation summation equivalence properties and thus enable flexible combinations for create new LoRAs. Evaluation results show the MSU-based approach achieves a good performance.

strengths:
+ MSU is an interesting concept with permutation invariance and concatenation summation equivalence properties.

weaknesses:
- While MSU is an interesting concept, it is unclear why the finer granularity merging of LoRAs is the right approach for combing multiple LoRAs to enhance LLM capabilities: the clustering and the use of the centroid of each cluster essentially impact the performance of the merged LoRA. 
- The evaluation used only 7 LoRAs. It is unclear how the proposed approach may work with more LoRAs.

questions:
How can the finer-grained merging of LoRAs preserve the performance of the merged LoRA, particularly in comparison with the approach of selecting individual LoRAs? How many LoRAs can be supported by the MSU-based approach, and what performance it can achieve with more (e.g., tens of) LoRAs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730456112487

thread_type:
official_review

<thread_object>
creator: reviewer_uiyw

<content>
summary:
The paper presents an innovative framework called LoRA-LEGO, which enhances the merging of LoRA for large language models. The approach leverages the modularity of LoRAs by introducing Minimal Semantic Units (MSUs) and employs rank-wise clustering to optimize parameter efficiency and performance. The framework aims to overcome limitations in existing methods by reducing parameter interference and maintaining task-specific knowledge.

strengths:
- The concept of treating LoRA parameters as LEGO blocks, specifically through MSUs, is creative and offers a fresh perspective on modularity in model adaptation.
- The paper provides solid theoretical foundations, including permutation invariance and concatenation-summation equivalence, which support the proposed method.
- Extensive experiments demonstrate the framework's effectiveness across multiple benchmarks, outperforming existing methods.

weaknesses:
- The MSU is not related to some semantics that can be understood by humans, it would be better if we could see some analysis.
- Although the proposed method outperforms existing approaches, a more detailed comparison with a wider range of baseline methods could strengthen the claims.
- It does not report the overhead (e.g., in inference and training time, memory) that may introduced by this approach.

questions:
see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730526800486

thread_type:
official_review

<thread_object>
creator: reviewer_9lla

<content>
summary:
This paper proposed the LoRA-LEGO framework for combining Low-Rank Adaptation (LoRA) modules in large language models (LLMs). LoRA-LEGO introduces Minimal Semantic Units (MSUs), independent components within LoRA, enabling flexible merging through clustering and rank adjustment. This modular approach allows task-specific LoRAs to be disassembled and reassembled, tackling parameter misalignment and knowledge conflicts seen in traditional merging methods. The dual reweighting strategy optimizes the scale of the combined LoRA, resulting in a more efficient and flexible system that performs well across both in-domain and out-of-domain tasks.

strengths:
* Modularity and Flexibility: The MSU-based modular approach enhances flexibility in merging LoRAs, allowing for custom rank adjustments.

* Efficient Scaling: Dual reweighting improves output scale management, maintaining model performance across tasks.

weaknesses:
* Performance may degrade when merging LoRAs designed for highly divergent tasks.

* Merging effectiveness can be highly sensitive to hyperparameter settings.

questions:
1. How does LoRA-LEGO address performance degradation when merging LoRAs for tasks with significant divergence?

2. Why does performance improve for MRPC and RTE tasks in Table 1 following permutation merging?

3. How can the optimal number of clusters $k$ in equation (2) be determined?

4. Why does the weight averaging method in Table 3 achieve identical performance to the task-specific LoRA (upper bound) for the MRPC task?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695269417

thread_type:
official_review

<thread_object>
creator: reviewer_1z1o

<content>
summary:
The paper introduces the LoRA-LEGO framework, which combines multiple LoRAs for multi-task and mixed-task scenarios. It uses clustering to group MSUs from different LoRAs into k clusters before constructing a merged LoRA based on the cluster centroids. Parameter reweighting is then performed to scale the centroid with respect to the average cluster norm. Experiments are conducted against 4 baselines (weighted average, ensemble, task arithmetic and ties merging) for multi-task and mixed-task scenarios, showing improved average performance.

strengths:
After reading the authors' rebuttals, my concerns have been adequately addressed.

weaknesses:
1. The paper is generally well written with good illustrations. However, I found the approach not particularly novel. It relies on a straightforward k-means clustering method for grouping and subsequently aggregating cluster parameters.

2. The approach is not particularly strong, and this simplicity may limit the framework’s performance which is reflected in the empirical results. While the average performance across tasks improves, it does not apply to all tasks. For example, performance remains poor for WNLI, an out of domain task for multi-task learning.

3. The baselines involving LoRA composition methods are basic and are not established methods in existing literature. It lacks citations that would align them with related work. This weakens the comparative analysis and raises questions about the validity of the selected baselines.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1731768215281

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to the Reviewers

comment:
We sincerely appreciate the reviewers for their thoughtful and constructive feedback. We are encouraged by the positive recognition of our contribution, which can be summarized as follows:

1. MSU is highlighted as an innovative and meaningful concept for modular model adaptation (Reviewers `YS49`, `Uiyw`)
2. Our work establishes solid theoretical foundations through properties like permutation invariance and concatenation-summation equivalence (Reviewers `YS49`, `Uiyw`)
3. The LEGO-like approach provides enhanced flexibility and modularity in LoRA merging (Reviewers `Uiyw`, `9LLA`)
4. Our method demonstrates superior empirical performance across multiple benchmarks with efficient scaling (Reviewers `Uiyw`, `9LLA`)

In our revision, we have carefully addressed each of the concerns raised, and below are the main points of the revised manuscript:

1. We have included **two additional baselines**: DARE [1] and DELLA-Merging [2], to provide more comprehensive comparisons with recent model merging methods. Detailed comparisons and analyses are presented in **Table 3 and Appendix H** of the revision. (Reviewers `1z1o`, `Uiyw`)
2. We have conducted experiments with more **diverse tasks**: In **Appendix F**, we evaluate LoRA merging performance using 7 LoRAs from different domains, where our method significantly outperforms baselines with an 11% improvement over the best baseline. (Reviewers `1z1o`, `9LLA`,  `Uiyw`, `YS49`)
3. We have assessed the **scalability** of different methods: In **Appendix G**, we evaluate the merging capabilities of different methods using 20 LoRAs from various tasks and domains. Our method maintains superior performance with a 16% improvement over the best baseline in this challenging scenario.(Reviewers `1z1o`, `9LLA`,  `Uiyw`, `YS49`)
4. We have investigated MSU similarity within LoRAs: In **Appendix I**, we analyze the similarity of MSUs within different task LoRAs, revealing that simpler tasks tend to have more similar internal MSUs, making them more receptive to positive ability transfer from other tasks.(Reviewer `Uiyw`)
5. We have explored MSU relationships across tasks: In **Appendix J**, we investigate whether MSUs reflect semantic structures by projecting MSUs from 48 LoRAs (across different domains and tasks) into a shared space using t-SNE. The results show that MSUs from the same task and task cluster tend to be more closely aligned. (Reviewer `9LLA`)

These changes have been highlighted in blue font in the revision. We have also provided **point-by-point responses** to each reviewer's specific concerns. We believe these detailed revisions further demonstrate our contributions and address the reviewers' concerns. 

Once again, we highly value the reviewers' insightful feedback and are open to any further discussions.

[1] Yu, Le, et al. "Language models are super mario: Absorbing abilities from homologous models as a free lunch." *Forty-first International Conference on Machine Learning*. 2024.

[2] Deep, Pala Tej, Rishabh Bhardwaj, and Soujanya Poria. "DELLA-Merging: Reducing Interference in Model Merging through Magnitude-Based Sampling." *arXiv preprint arXiv:2406.11617* (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces the concept of Minimal Semantic Unit (MSU) for disassembling and reassembling multiple LoRAs at a finer granularity, thereby ameliorating parameter interference and performance degradation. Evaluation results show the proposed MSU-based approach, combined with clustering to control the rank, yields to performance improvements over several benchmarks.

Reviewers generally found the approach interesting, but wanted to see comparisons to more SoTA competitors. The authors addressed this sufficiently during the rebuttal phase, introducing two very recent additional baselines (DARE and DELLA-Merging), to provide more comprehensive comparisons with recent model merging methods. A weakness that remained somewhat was the sensitivity to hyperparameters.