Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering
Authors: Ziyu Zhao
Tao Shen
Didi Zhu
Zexi Li
Jing Su
Xuwu Wang
Fei Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Parameter Efficient Tuning
LoRA
Model Merging
Primary area: foundation or frontier models, including LLMs

Abstract: Low-Rank Adaptation (LoRA) has emerged as a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability on platforms like Huggingface. This modularity has sparked interest in combining multiple LoRAs to significantly enhance LLM capabilities. However, existing methods for LoRA composition primarily focus on task-specific adaptations that require additional training, and current model merging techniques often fail to fully leverage LoRA's modular nature, leading to parameter interference and performance degradation.
In this paper, we explore the possibility of disassembling and reassembling multiple LoRAs at a finer granularity, much like assembling LEGO blocks. We introduce the concept of Minimal Semantic Units (MSUs), where the parameters corresponding to each rank in LoRA function as independent units. These MSUs exhibit properties such as permutation invariance and concatenation-summation equivalence, allowing for flexible combinations to form new LoRAs. Building on these insights, we propose the LoRA-LEGO framework. This framework conducts rank-wise parameter clustering by grouping MSUs from different LoRAs into $k$ clusters. The centroid of each cluster serves as a representative MSU, enabling the assembly of a merged LoRA with an adjusted rank of $k$. Additionally, we apply a dual reweighting strategy to optimize the scale of the merged LoRA. Experiments across various benchmarks demonstrate that our method outperforms existing approaches in LoRA merging.

Published as a conference paper at ICLR 2025
MERGING LORAS LIKE PLAYING LEGO: PUSH- ING THE MODULARITY OF LORA TO EXTREMES THROUGH RANK- WISE CLUSTERING
Ziyu Zhao Tao Shen', Didi Zhu', Zexi Li', Jing Su', Xuwu Wang', , Fei Wu "Zejiangg University, 'Shanghai Innovation Institute. "ByteDance Inc. benzhac. st styx@gmail. com wufei@zjin jufei@zju.edu.ccr edu.
ABSTRACT
Low-Rank Adaptation (LoRA) has emerged a popular technique for fine-tuning large language models (LLMs) to various domains due to its modular design and widespread availability platforms like Huggingface. This modularity has sparked interest in combining multiple LoRAs to enhance LLM capabilities. How- ever, existing methods for LoRA composition primarily focus task-specific adaptations that require additional training, and current model merging techniques often fail to fully leverage LoRA's modular nature, leading parameter interfer- ence and performance degradation. In this paper, we investigate the feasibility of disassembling and reassembling multiple LoRAs finer granularity, analo- gous to assembling LEGO blocks. We introduce the concept of Minimal Semantic Units (MSUs), where the parameters corresponding each rank LoRA fune- tion as independent units. These MSUs demonstrate permutation invariance and concatenation- summmmttonn equivalence properties, enabling flexible combinations to create new LoRAs Building on these insights, we propose the LoRA-LEGO framework. This framework conducts rank-wise parameter clustering by grouping MSUs from different LoRAs into & clusters. The centroid of each cluster serves a representative MSU, enabling the assembly of a merged LoRA with an adjusted rank of k. Additionally, we apply a dual reweighting strategy optimize the scale of the merged LoRA. Experiments across various benchmarks demonstrate that our method outperforms existing approaches in LoRA merging.
INTRODUCTION
Large Language Models (LLMs) like ChatGPT Achiam al. (2023) and LLaMA Touvron et al. (203)) trained on vast amounts general data, demonstrate remarkable performance general tasks. To explore their potential for specialized tasks, adapting LLMs specific domains by fine- tuning model parameters has become a critical area of research. In this context, Low-rank Adapta- tion (LoRA) al. 2222)) as a parameter-effiee efffiietr-effcient fine-tuning approach. has gained widespread recognition, also attributed to its modular design Liu et al.] (223);; Yang et al. (2023b); Hadi et al. (2023). The modular nature of LoRA enables to serve as plug-and-play plugins for L.L.Ms, facil- itating the storage and deployment of large collections of LoRAs on platforms like Hugging Face. The extensive availability of LoRAs has sparked considerable interest in combining multiple Lo-
RAs into a unified adapter to significantly extend the capabilities of LLMs Yadav et al. (024a);
Xiao et al. (2024); Chen et al. (2025); Lyu et al. (2024) Zhu et al. Huang et al. (2023).
Previous methods for composing multiple LoRAs have primarily focused assembling separate
LoRAs tailored to specific downstream tasks, which generally require additional training Wu et al.
2003); Wang et al. (2024); Chronopoulou et al. (2023); Yadav et al. (2024a); Huang et al. (2023).
Model merging /Tang et al. (2024): Yaday et al. 2024b): Iiharco et al. (2022): Yang et al.] (2024a b
2025) offers alternative approach by aggregating the parameters of multiple LoRAs into unified
adapter without extra training, producing a unified LoRA with comprehensive capabilities. However,
these methods typically employ element-wiss parameter fusion, which can neglect and disrupt the
Corresponding authors.

---

Published as a conference paper at ICLR 2025
Vroead
4
Vomsm
40
L
Lekd
LoRA Concolenation
a) Mininal Semantic Unit
b) Permutation Invariance
Concatenotion-Sure Summontionnn Equivolence
a) Each LoRA can be further modularized into mul- tiple Minimal Semantic Units (MSUs), each corresponding to a row in matrix and column in matrix B, differentiated by distinct colors. b) The MSUs within a LoRA display permutation in- variance, implying that any rearrangement of the MSUs does not affect the output generated by the LoRA. c) Multiple LoRAs exhibit Concatenaaion-Summation Equivalence, indicating that the sum- mation of outputs from various LoRAs equivalent to the output of singular LoRA constructed by concatenating their MSUs.
Figure 1: Further Modularization of LoRA:
internal semantic structure within LoRA. This disruption potentially leads to parameter interference (as discussed in (2.3. thereby hindering the performance of merged LoRA. This paper approaches LoRA merging from a novel perspective, focusing on the fine-grained modularization of LoRA by decomposing it into independent units, which enables the flexible reconstruction of unified LoRA with comprehensive capabilities.
As illustrated in Fig I our motivation for further modularizing LoRA stems from the following insights: a) Each rank in LoRA corresponds to a row in the down wwwnpproectioo matrix A and a column in the up-projection matrix B /hhao al. (2025). Since the calculations for each rank are independent, we consider the parameters associated with each rank as a cohesive entity. We define these entities as Minimal Semantic Units (MSUs), which serve as the fundamental building blocks of LoRA. b) Within each LoRA, the MSUs exhibit the property of Permutation Invariance, indicating that any permutation of MSUs within a LoRA does not affect the adapter's output. c) LoRA exhibits the the outputs from multiple LoR As equivalent to the output of single higher-ranked LoRA constructed by concatenating all the MSUs of these LoRAs
Equivalence property, which states that summing
In this paper, we introduce a novel method called LORA-LEGO, which is based on the insight that MSUs act as building blocks that form a LoRA and can be disassembled and reassembled like playing with LEGO. LoRA-LEGO consists of three main steps: (1) Grouping MSUs from candidate LoRAs into MSU pool; (2) Clustering the MSU pool into l clusters, where A: is the target rank of the merged LoRA; (3) Constructing the merged LoRA from the centroids of these clusters, with each centroid representing MSU, thereby setting the merged LoRA's rank to k. LoRA-LEGO enables the flexible combination of LoRAs with arbitrary ranks by clustering similar MSUs, at the same time effectively resolving parameter interference while merging. This approach allows for targeted rank adjustments in the merged LoRA to preserve task-specific knowledge. We also observed that variations in parameter norms and the rank size of the merged LoRA affect the output scale. To address this, we implement a dual reweighting strategy that adjusts both the parameters and the outputs, ensuring optimal scaling for the merged LoRA.
We empirically validate the effectiveness of the proposed LoRA-LEGOO in both masti-task/ Tang et al. (2024) and mixed-tase Zhao et al. (2024b) scenarios. Experimental results show that LEGO consistently outperforms other methods for LoRA merging, demonstrating notable flexibility and ef- ficiency. Additionally, LoRA-LEGO can merge heterogeneous LoRAs varying ranks, surpassing the capabilities of previous model merging methods. Moreover, it can also be applied to individ- ual LoRAs for parameter pruning, revealing that retaining just 50% of the parameters can achieve performance comparable the original model. Our contribution can be summarized as:
We investigate the modularization of LoRA, identifying the MSU as its fundamental building block, which characterized by permutation invariance and sommmmatiiooonmmmmtiiioini equiva- lence properties.
We introduce LoRA-LEGO that merges multiple LoRAs in a LEGO-like fashion by grouping, clustering, and reconstructing MSUs seamlessly combine separate LoRAs
2

---

Published as a conference paper at ICLR 2025
Experimental results show that LoRA-LEGO flexibly disassemble and reassemble LoRAs of any rank, surpassing other model merging methods in performance. Additionally, LoRA-LEGO can be effectively applied to individual LoRAs, enabling parameter pruning and substantial reduction in LoRA parameters while maintaining comparable performance.
PRELIMINARIES
2.1 LOW-RANK ADAPTATION
Directly fine-tuning LLMs with full parameters is computationally intensive and is not feasible in low-resource scenarios. Based on the idea that only a small number of low-rank parameters need to be fine-tuned for sufficient performance in new domains, Hu et al. (2021) proposed the Low-Rank Adaptation, where the LoRA module can be combined with the pre-trained parameters parallel for efficient inference.
Specifically, given pre-trained weights Wo E Rdxk of a sub-module of LLM, the LoRA adds an extra trainable weight matrix Wn + A W Wn BA,, where AW can be decomposed into two smaller matrices B E Rddx and E RRaxd, where stands for the rank of A AW and the rank d. The forward pass for layer y Wo be modified follows:
y W az A AWa
Wo BAA.,
where E Rd is the input and the y E RRd denote the output.
2.2 FURTHER MODULARIZATION OF LORA
Before delving into the issue of LoRA merging, it is imperative to present several pivotal insights and definitions that could serve fundamental components for constructing LoRA module.
Definition 1. Minimum Semantic Unit of LoRA. Let A and B be matrices in LoRA module. For each index define the minimum semantic unit of LoRA as the combined vector 8; [a,, b,). where is the i-th row of A and b, is the i-th row of B (i.e., the transpose of the i-th column of B).
In this context, each row of the down- projection matrix A and its corresponding column in the up- projection matrix are treated as cohesive unit, defined as a Minimum Semantic Unit (MSU). Each MSU contributes to a rank of the LoRA, encapsulating a distinct semantic fragment of the LoRA's capacity. Through this definition, LoRAs exhibit the following properties.
Property 2.1. Permutation Invariance. For a LoRA module parameterized by matrices A and B, if the rows of A are permuted, then by performing a corresponding permutation of the columns of B, the product of these matrices remains unchunged. Formally, let P be permutation matrix that satisfies pT new matrix A' = P.A. and correspondingly permute the columns of B get B' -BP", then, BA A'
I, where I is the identity matrix. If we permute the rows of A to obtain a
The property of permutation invariance indicates that the arrangement of MSUs within LoRA cal- culations can altered without affecting LoRA's output.
Property 2.2. (Az, B2), each of rank r. Specifically, matrices A₁ and A₂ are of size grxd and B and B of size Baxr. Define the concatenated matrices as:
Equivalence. Consider two LoRAs, (A1, B1) and
[A]
A'
2rad
B' B'= B
E Rdx2r
A2
The oulput vector y from the concatenated model equivalent to the sum of the oulputs from each individual LoRA model:
y == B'A' (BiA1 B.Ag)æ.
Based on this property, we can synthesize the knowledge from all LoRAs by constructing a new LoRA through the concatenation of all MSUs from each LoRA. The computational result is equiv- alent to ensembling the outputs all LoRAs. Based on these insights, we can draw the following conclusions:
3

---

Published as a conference paper at ICLR 2025
Merging
Ideal Merging
Element-wise Merging
Merging
An by by
Presérve Task- speci Reoneegge
Aligriment
0g.
Rig
So su na
Paraneter Misaligrment
Knowledge Conflict
Task specttic spectficKroweegee
poramaters should be diigned to allevitte paramater interference.
nerged LoRA need higher rank encompars compreheraive knowledge
Two sources of parameter interference in LoRA merging. The left part illustrates how parameter misalignment can lead interference; the right part demonstrates that knowledge conflict in merged LoRA layers can also result in parameter interference.
Figure 2:
Each LoRA can be modularized into multiple MSUs, with each MSU corresponding to rank within the LoRA. These MSUs can be flexibly permuted and combined to construct unified LoRA.
2.3 PROBLEM FORMULATION AND CHALLENGES
Table 1: Performance degra- dation after merging mis- aligned LoRAs. "Origi- nal" refers to the perfor- mance the unaltered LoRA, while "Misaligned" indicates the performance after merg- ing the LoRA with
Consider a LLM denoted as L and a set of p task-specific LoRAs, represented by Φ {01, 22 ................................ 6p Each LoRA o; is specialized for particular task T. and is crafted by incorporating low-rank ma- trices into different layers of L. thereby tuning the model to better suit T. For simplicity of notation, we denote the parameters of these low-rank matrices at any given layer for each LoRA Φ; A; and B, The goal of merging these LoRAs is to synthesize com- prehensive LoRA ø' that not only excels in all tasks encompassed by & but also generalizes well to unseen tasks. We discuss the dif- ference between the LoRA merging setting and the previous model merging setting in the Appendix A
domly permuted version it- self.
Task
Original
Misaligned
CoLA
61.63
60.96 (1.1%
natural approach to performing LoRA merging involves a sim- ple element wise averaging of the parameters from each LoRA: o' = - -. di. However, parameter interference poses a sig- nificant challenge to effective LoRA merging. We identify two po- tential sources of parameter interference during LoRA merging and demonstrate through experiments that such interference can lead to performance degradation in the merged LoRA.
MNLII
77.46
69.49 (10.3%
MRPC
68.00
68.50
L)
QNLI
77.25
60.44 (21.8%
QQP
75.83
66.94 (11.7% J]
RTF.
52.22
54.44 (.4.%% L)
SST2
75.74
75.52 (0.3% 1)
Overall
69.73
65.18 (5.74%
The first cause of parameter interference stems from parameter misalignment in LoRAs, depicted in the left part of Fig According to Property [2.]] the MSUS of each LoRA can be permuted arbitrar- ily without affecting the functionality of the LoRA module. However, misalignment of MSU parame- ters when merging LoRAs can result in parameter interference. To investigate the impact of param- eter misalignment on model performance, we con- ducted a controlled experiment using the Llama-2- 7b model, training LoRAs different tasks. For the parameters A and B of a task, we randomly generated permutation matrix P and adjusted the parameters A' (A PA)/2 and
Table 2: Parameter interference due to knowledge conflict. "Tuning MSU" indi- cates the performance after tuning the added MSU for each task. "Avg MSU" denotes the performance achieved by directly merging these task-specific MSUs. "Concat MSU" represents the performance after concatenat- ing these task-specific MSUs.
Task Tuning
Avg MSU
Concat MSU 81.36 (3.58%))
MNLI
86.17
46.24 15%
MRPC
87.25
64.75 78%
(6.88% 81.31 (6.23%)
Ovenll
86.71
55.49
(36.006))
B' = (B + )/2")/2. This adjustment simulates the merging of two identical LoRAs with mis- aligned parameters. The results, presented in Tab indicate that parameter misalignment can lead
4

---

Published as a conference paper at ICLR 2025
to decline in model performance, with some tasks experiencing significant performance degrada- tion. Therefore, ideal merging entails aligning MSUs during LoRA merging to mitigate parameter shows more in-depth discussion and analysis.
interference. Appendix I
Another source of parameter interference stems from knowledge conflict during LoRA merging. As depicted the right side of Fig 2 knowledge conflict occurs when the merged LoRA A lacks sufficient parameter space to encapsulate the comprehensive knowledge. This deficiency forces the merging of task-specific MSUs, resulting in parameter interference. To investigate the impact of knowledge conflict during LoRA merging, we conducted an experiment to demonstrate the performance degra- dation resulting from merging task-specific MSUs. With a base LoRA trained on the CoLA task, we adapted this LoRA for two new tasks (MNLI and MRPC) by appending an additional MSU to create two separate task sse-sfpeeffie LoRAs. Throughout the training process for the new tasks, only the newly introduced MSU for each task was trainable. In this way, the only difference between the LoRAs As for MNLI and MRPC was the unique MSU added for each, which encapsulated distinct semantic infor- mation tailored to each task. This setup was designed to create two task- sask-specific LoRAs that differed only in one MSU, allowing us to observe parameter interference when merging these task-specific MSUs. The results, depicted in Tab demonstrated that averaging the task-specific MSUs from the two LoRAs significantly reduced performance on each task. In contrast, maintaining these task- specific MSUs through concatenation preserved the capabilities specific each original task. This suggests that ideal merging should maintain task-specific MSUs during LoRA merging to prevent knowledge conflict and effectively resolve parameter interference.
METHODOLOGY
3.1 LORA-LEGO FRAMEWORK
Based on the motivation that MSUs are the build- ing blocks of LoRA, we can disassemble and re- assemble LoRA like playing with LEGO. Here,
propose flexible and effective method called LoRA-LEGO as shown in Fig 3 This frame- work is structured around three main procedures: MSU Grouping, MSU Clustering, and LoRA Re- construction. These steps collectively facilitate the integration of diverse MSUs into a cohe- sive LoRA, alleviating the parameter interference while LoRA merging.
BGU Groipi
Oate Derrih
ANOO IN
LoRA Rescntmaction
33 IOU/ Chattring
MSU Grouping. The initial stage of merging P LoRAs begins by disassembling each LoRA into various MSUs and grouping all the MSUs from each LoRA together. Let {A., B, represent the LoRA parameters of a layer with rank r,. Each LoRA module A, B, contains rj MSUs, denoted by {8,1,
Figure 3:
The LoRA-LEGO framework merges candidate LoRAs in a manner akin to playing with LEGO by: first disassembling LoRAs into multiple MSUs and grouping them into MSU pool; b) performing MSU cluster- ing to merge similar MSUs; c) reconstructing the merged LoRA from the centroid MSUs to form a cohesive LoRA.
J,r, where 8 [a,t, by] with a₁ A, and byr L The MSU pool Φ, which includes MSUS from all the LoRAs be merged, is constructed
( ,, }.
=
MSU Clustering. After grouping the MSUs from different LoRAs, the next step involves re- grouping these MSUs into clusters based on their similarities. With the MSU pool $ we employed K-means Kanungo et al. (2002) to partition these MSUs into & clusters {C1, C2...., Cx} in which each MSU is assigned to the cluster closest it. This process described by the following opti-
mization problem:
minimize
Σ
Σ sEC,
|| miil?
5

---

Published as a conference paper at ICLR 2025
where μ4 is the centroid of cluster C₁-
LoRA Reconstruction. Following the MSU clustering, we rearrange the MSUs into clusters based their similarity. The centroids of these clusters, denoted by M1, .... Akx, calculated as the average of the MSUs within each cluster. These centroids represent aggregated parameters across the MSUs, encapsulating the generalized semantic information most representative of each cluster. Aggregating within each cluster minimizes information loss compared to directly merging different LoRAs, as the MSUs within a cluster are more similar to each other.
Using these k centroids, we can reconstruct a new LoRA module. Each centroid μn contributes to single rank in the merged model, thus the new LoRA model has a rank k., where 1 S k < S)=1 Ty. The new merged LoRA model is formed by constructing new projection matrices A' and B' from the centroids:
A'
a2
B'
bf
b2
0x.
where a; and b, are extracted from each centroid μ4; = [a., b,] per the MSU definition. The re- constructed LoRA module }'} addresses parameter interference by aligning MSUs based on their similarity before merging, achieving a flexible rank that encapsulates comprehensive knowl- edge across various tasks. An interesting point is that our method sits between model merging, which fuses multiple identical models into a singular model, and model ensemble, which takes the average of outputs from different modules, achieving a balance between performance and computa- tional efficiency. We provide a detailed discussion of how our method relates to model merging and model ensemble in Appendix B
3.2 OPTIMAL SCALE OF MERGED LORA
Given that the rank of the merged LoRA from LoRA-LEGO can range from to Ej= Tj. the scale of LoRA's output could vary significantly, thereby impacting the performance. We identified two key factors that determine the scale of the output.
Norm Decay After LoRA Merging. As shown in Fig we exam- ine the norms of the parameters after merging (i.c., the centroids of each cluster) compared to the average norms of the parameters within each cluster before merging. We observed that after merging, the pa- rameter norms significantly decrease, potentially affecting the output scale of the LoRA module, since the parameter norm influences the magnitude of the output. This phenomenon can be explained by the triangle inequality Klement et al. (2013), which states that for any vectors 8,, |E 8: I > |
Nomis of Clatter Conteet
When computing the centroid
p Ei 8i, its norm satisfies:
Clustr of Diterent Layera
Figure 4: Comparison of cluster center norm to average norm within the cluster.
1
P
P
|+l|
8
s
Therefore, the norm of the centroid is less than or equal to the average of the norms of the original vectors, explaining the observed norm decay after merging. The more diverse the vectors within cluster, the more pronounced this reduction in norm will be. To compensate for the reduced norm after merging, we perform parameter reweighting by scaling the centroid to match the average norm of the cluster: ní E ||=el|
μ. In our implementation, use the infinity norm for reweighting to ensure stability and robustness in the results.
|
Variance Expansion with Increased LoRA Rank. Another factor influencing the scale of the LoRA output is the rank of the merged LoRA. We conducted experiments to investigate how the rank of the LoRA affects the output scale by merging seven LoRAs with rank 8 and varying the rank & of the merged LoRA (which corresponds to the clusters number in LoRA-LEGO), The frequency histograms of outputs from the first layer of the merged LoRA at various ranks, as shown

---

Published as a conference paper at ICLR 2025
in 5 indicate that LoRA outputs approximate normal distribution centered zero. We ob- served that as the rank increases, the variance of the output also increases. To normalize the output variance, similar to the normalization in the self-attention mechanisms Vaswani (2017), we perform output reweighting for the merged LoRA by the factor yf The following theorem ensures that VK"
this rescaling maintains consistent variance in the LoRA output.
Theorem 3.1.
Let A E R and B1 E RR and A2 E - and Bg E R where all elements of these matrices are independently and identically distributed according to the standard normal distribution (00 1). Then, after scaling the product A2B2 by the factor Vr/V/, the variances of the entries of A B, and the scaled B, are equal:
Vr Vk
Var (1 = Var
A2
The proof of Theorem is detailed in the Appendix D Over- all, to ensure that the LoRA output is correctly scaled, we em- ploy two scaling strategies. First, we reweight the parame- ters to match the average norms of the parameters within each cluster. Second, we rescale the output of the merged LoRA for maintaining variance consistency with the original LoRA. These dual scaling strategies enable LoRA-LEGO to deliver enhanced and more robust performance.
Bank Ir
Rank 2r
Bank
Rank
EXPERIMENTS
Given that LoRA merging is essential for many scenarios, have opted for two settings: Multi-task learning Tang et al. (2024) and Mixed-task settings Zhao et al. (2024b). In these settings, we compared various LoRÁ composition methods to assess the performance of the proposed LORA-LEGO ap- proach. We selected Llama2-(7b,1))) trained LoRA for each task with hyperparameters r and a = 12. The evaluation frameworks for multi-task Learning and mixed-task settings are detailed in the subsequent sections, where we provide a comprehensive analysis.
LORA Output
Figure 5: Expansion of variance with increasing rank merged Lo- RAs.
as the base model and
4.1 MULTI-TASK LEARNING
Experiment Setting. Multi-task learning aims to merge individually trained LoRAs into uni- fied model while preserving the performance of each constituent LoRA. Drawing from previous research Tang et al.] (2024); Yadav al. (2024b); []harco et al. (022)) we merged seven LoRA models, each fine-tuned on Llama2- (bb,2--7b,13bb, for in-domain tasks including Cola, Mnli, MRPR, QNLI, BLUE-QQP. RTE, and SST2. We then assessed the performance of the merged LoRA on these in-domain tasks as well on two additional out-of-domain tasks, SNLI WNLI, to evaluate its adaptability and generalization capabilities.
Baseline Methods. We compared the proposed method with 6 post-hoc training-free LoRA com- position methods, including (1) Weight Averaging. (2) Ensemble, (3) Task Arithmetic, (4) Ties- Merging, (5) DARE, and (6) DELLA-Merging. The details of these LoRA composition methods can be found in the Appendix
Main Results. shown Tab. our proposed LoRA-LEGO method significantly outperforms the baseline methods on both IID and OOD tasks. Specifically, the Weight Averaging method suffers from significant performance degradation due to parameter interference during LoRA merging. The Ensemble method encounters issues with parameter redundancy, leading to suboptimal performance and slower inference speeds. Model merging methods such as Task Arithmetic and Ties- Mies-ng perform samenttwwise fusion and fail to adequately address parameter interference in LoRA, result- ing in suboptimal performance during the merging process. Similarly, DARE and DELLA- DELLA-Merging adopt element-wise parameter merging, which ignores the alignment of the parameter semantic
7

---

Published as a conference paper at ICLR 2025
Table 3: Multi-task performance when merging Llama2-(76,13b/ (LoRA fine-tuned) models on
seven seen tasks and two unseen tasks.
HD Tasks
OOD Tasks SNLI
Method
CoLA
MNLI
MRRC
QNLI
QQP
RTE
SST2
WNLI
Average
Lloma2-222
Weight Average
54.42
36.09
68.00
14.31
51.72
45.15
42.95
31.64
47.14
47.17
Ensemble
55.67
45.89
59.25
59.84
67.38
68.89
66.44
36.73
51.43
56
Tocck Arithmeric
55.48
42.15
14.25
58.94
66.41
67.77
99. 54
14.08
54.20
54.77
Mering
18.65
18.81
55.50
61.79
66.75
62.59
70.68
18.45
61.33
58.30
47.20
Merping
36.88
53.25
57.70
LoRA: LEGO
55,48
35.73
66.00
62.29
71.07
71.85
73.23
$1.36
52.56
62:31
I// Lionnz--112 2.4b
LoKA
69.04
88.23
89.25
82.33
76.44
Weight Average
74.75
75.52
54.32
Tosk Arithmetic
63.17
64.41
74,50
71.59
30.84
78.15
75.36
54.16
58.57
69.03
Tics Mering
58.56
64.71
78.75
74.27
80.71
76.67
75. 40
56.02
61.45
69.61
DARE
45,00
46.34
67.75
46.74
47.32
63.33
46.90
42.55
14.29
50.02
DELLA Merging
62.21
62.45
71.25
69.05
30
78.52
75.40
49.86
58.57
67.06
LEGO
05.40
13.50
73.58
CaLA
MIRC:
QUP
MINLI
QNLI
RIK
SST2
25.2%
Averagy
Pruniag
Figure 6: LoRA pruning performance over seven datasets.
space, leading to parameter interference and resulting in suboptimal model merging performance. In contrast, our proposed LoRA-LEGO effectively alleviates parameter misalignment and knowl- edge conflict through flexible MSU clustering, thereby achieving superior performance compared to other methods. In Appendix E we demonstrate that the proposed LORA-LEGO approach can effec- tively merge heterogeneous LoRAs, exceeding the capabilities of previous model merging methods. In Appendix [ we explore the results of model merging on more diverse tasks and find that task di- versity introduces greater challenges for LoRA merging. In such scenarios, our method significantly outperforms other baseline methods.
Performance on LoRA Pruning, Our method also func- tions as LoRA parameter pruning approach. For a single LoRA with rank r, LoRA-LEGO allows for selecting k < r, effectively reducing the rank to i and pruning the model. As illustrated in Fig. 6 evaluate the performance of single LoRA model after retaining various proportions of its param- eters. LoRA-LEGO efficiently compresses model parameters: retaining just 33% of the parameters preserves 79% of the orig- inal model's capabilities while keeping 50% maintains 92% of the performance. This offers new insights into strategies for compressing model parameters, especially those of LoRA.
Avergee Accuracy
Bessight
Rowcightig
Paramater Ourput Heersighting
Number Clusters
Figure strategies.
Ablation on scaling
Ablation of Scaling Strategies. We evaluate the effectiveness of scaling strategies for the merged LoRA by varying the number of clusters for LoRA-LEGO, noting that the cluster number corresponds the rank of the merged LoRA. As illustrated Fig the original computation of LoRA experiences significant performance degradation with increasing rank of the merged LoRA, primarily due to the expansion of variance associated with the increased rank. Additionally, when the rank of the merged LoRA is relatively low, its performance does not reach its optimum due to the degradation of parameter norms. We also present the performance of each scaling strategy and their combination. Applying parameter reweighting can significantly enhance the performance of merged LoRA
the merged LoRA when the rank is relatively low; specifically, the performance at rank 1r improves by 5%. However, the rank increases, eliminating norm decay more severely exposes variance expansion because norm decay can alleviate this phenomenon, leading to greater performance degradation. Stabilizing the variance by output reweighting significantly increases
8

---

Published as a conference paper at ICLR 2025
Table 4: The average performance of each task cluster. The performance of perfectly selected corresponding LoRA for each sample is colored in gray. We have bolded the best performance of each task and underlined the best performance in the "OOD" setting.
LaRA-18G0
Weight Average
Knoemble
The-Mreging
Selection
UOD
110
0OD
UID
OUD
UOD
000
an' a/1lmms2-25
SinazT tn TEst Rouge-
59.1
29.5
5115
38.8
45.2
44.5
41.3
51.2
45.3
45.4
45.4
Sinaet to Text Rouge-2
36.1
25.7
25.6
35.6
25.2
22.6
20.2
26.3
22.4
23.9
27.4
Sinaet to Text Rouge-
45.0
19.5
12.9
46.4
15.3
54.5
51.7
41.11
15.5
39.3
Transkeion BLEU
13.1
12.4
12.8
12.2
12.3
122
13.9
COMMONSENSE TIMINT
62.5
63.5
55.5
51.0
50.0
59.5
900
98.0
91.5
3955
790
505
H1.5
READING Comp.
67.3
54.3
35.7
51.7
40.3
47.3
43.0
51.3
47.3
46.3
56.3
CLOSE-WOOK QA
4500
47.0
45.5
40.0
43.0
41.0
$7.5
45.01
48.5
48.11
13.5
CORERBEENCE
$2.
62.8
613
50.0
46.0
47.0
$3.0
6300
49.0
12.11
43.0
READ. COOMP WYCOM
69.0
66.0
05.0
69.0
30.0
35.0
19.0
46.0
40.0
37.8
64.0
PARAPERASE NLJ
65.5
61.0
58.0
64.3
72.3
70.0
60.6
$1.4
53.8
64.3
Uverall
35.4
51.0
31.2
41.5
Lionn2-130
610
35.0
550
15.4
18:2
13.1
52.9
16.9
50.3
50L9
Sinacl to Texlitrre-
37.7
20.3
24.0
34,9
22.8
36.0
23.5
29.1
24.6
26.2
16.
Struct to Testpage-
50.5
43.9
36.4
47.6
34.8
38.4
35.9
42.9
36.9
41.0
40.9
129
14.7
14.5
12.9
12.7
14.6
14.1
14.0
14.1
11.2
11.5
COMMUNSENSE
695
69.0
05.
19.0
47.5
61.10
56.0
64.11
54.5
50.11
57.5
SENTIMENT
90.0
90.5
$7.0
91.5
91.5
Camp
BOOK COREPERENCE
64.0
62.0
61.0
74.0
77.0
62.0
75.0
55.0
64.0
READ. CNOMP W/OOM
$200
76.0
$4.0
500
150
59.
49.0
76.0
580
680
66.0
PARAPHRASE NLJI Ovcall
77.5
67.5
35.5
68.0
525
55.5
45.5
710
555
610
625
82.4
78.9
78.
78.9
70.2
69.8
65.4
7K1
75.7
657
65.7
62.4
55.2
32.8
37.8
47.
51.6
47.8
37.6
523
50.1
30.3
performance when the rank is high, although it remains suboptimal due the decrease of param- eter norms. Combining these two scaling strategies yields the hest results, demonstrating stable and improved performance across varying ranks of the merged LoRA. Notably, our approach's 's only hyperparameter is the cluster number k. After applying the dual rescaling strategy, our method's 's per- formance becomes highly stable, indicating that it very robust regarding hyperparameter selection; therefore, we use k- = 2r as the default setting.
Merging Different Number of Tasks. We investigated the average performance of the model when merging LoRAs with different numbers of tasks. To better assess the influence of task quantity on our method, normalized the performance of each task by dividing it by the performance of its respective single-task LoRA and then calculated the mean of these normalized scores. From 8 it is evident that the number of merging tasks increases, there general decline in the performance of the merged LoRAs. Specifically, direct averaging experiences a steep performance drop due to parameter interference. The Ensemble method also sees decrease in performance, attributed to parameter redundancy and inconsistencies in the output space. Ties-merging, failing to resolve parameter interference and reliant on hyperparameter selection fully, does not reach optimal performance. LoRA-LEGO, which flexibly addresses parameter interference, experiences lesser decline in performance with an increasing number of tasks, thereby outperforming the baseline model. In Appendix G we discuss the scalability of various model merging methods and showcase their performance in merging LoRAs for 20 diverse tasks. The results clearly highlight the efficiency and effectiveness of our approach.
4.2 MIXED-TASK EVALUATION
Aug Normalies pert
Evaluation Setting, Recent studies Zhao et al. 2024b) have
proposed the creation of LoRA pool from which relevant Lo-
RAs are retrieved for each input to facilitate LoRA composi- tion. We adopt the same setting and construct LoRA pool for 48 tasks from flan-v2, grouped into 10 task clusters. The evaluation set is constructed by randomly choosing 50 samples from each test set. These samples are then mixed and shuffled to form a unified dataset comprising 2400 data points.
Avarge Ensemble
TItS Merging
LoRA Merging
Numberdd Tasks Average performance
Figure 8: varying the number of merged tasks.
Adopting the LoraRetriever approach Zhao et al. (2024b), we initially retrieve the top-3 LoRAs based on the sentence embedding similarities between each input sample and LoRA's few-shot sam- ples. Following this, engage in LoRA composition and evaluate various strategies. This analysis underscores the versatility and superior performance of LoRA-LEGO in handling more complex scenarios.

---

Published as a conference paper at ICLR 2025
Baseline Methods. For all methods, we employ consistent evaluation pipeline. For each instance in the evaluation set, we initially retrieve the top- LoRA, followed by the composition of LoRA. We compared the following LoRA composition methods: (1) Weight Average, (2) Ensemble, (3) Selection (using the top-1 retrieved LoRA), and (4) Ties-Merging.
Main Results. Previous research Zhao et al. 2024b) has shown that using a retriever to identify LoRA tasks tailored to various inputs is more efficient and effective in personalized service settings. Consequently, we concentrate on how multiple LoRAs can be integrated effectively through LoRA merging after retrieving the top-k LoRAs for each input. We assess the performance of LoRA com- position methods in both IID and OOD contexts. "IID" performance refers to scenarios where all LoRAs are accessible to the retriever. "OOD" performance, however, involves masking the LoRA associated with the specific task of each test sample during retrieval, preventing any sample from accessing its ideal LoRA. This approach allows us to evaluate the cross-task generalization capa- bilities of the LoRA composition methods. Tab demonstrates that LoRA-LEGO surpasses other composition methods both IID and OOD scenarios by fully eliminating parameter interference. In contrast, bascline LoRA composition methods experience performance degradation due to their inability to completely mitigate parameter interference. Specifically, in III scenarios, the Selection method excels because the Retriever choose the most appropriate LoRA from closely related tasks for inference. Building on this, LoRA-LEGO further enhances performance by leveraging the transfer capabilities between different tasks, thereby achieving better results. For OOD scenarios, both Ties-Merging and Ensemble show good performance by hamessing knowledge from a wide array of relevant tasks to tackle OOD tasks. LoRA-LEGO, however, outperforms these methods by effectively addressing parameter interference, allowing for more comprehensive utilization of diverse LoRA capabilities and achieving superior results in OOD setting.
5 RELATED WORK
Many works have discussed how to obtain comprehensive model through model merging from various perspectives. Some works discuss how to find a set of low-loss paths in the parameter space for model parameter interpolation from the perspective of linear mode connectivity Ainsworth ett al. (2022); Entezari et al. (202T). From a similar perspective, we further utilized properties of MSUs, employing clustering algorithms provide flexible solution for enhancing the parameter connectivity during LoRA merging. Additionally, many works attempt to coordinate models trained in decentralized and separated manner through model merging, utilizing their knowledge transfer capabilities to obtain model with comprehensive abilities Tang et al. (2024); Don- Yehiya et al. (2022; ; Yadav et al. (2024b) Matena & Raffel (2022): p: Jin et al. (2022); Yang et al. (2023a); Deep et al. a (024)) Yu et al. (2024). Recently, with the rise of large language models, more and more works have focused on how to use model aggregation, especially the aggregation of LoRA Chronopoulou et al. (2023); Huang et al. (2023); Zhao et al. (2024b); WWang et al. (2024), to strategically utilize models adapted to multiple domains. These efforts often overlook the parameter interference that occurs during LoRA merging, and some of them require extensive additional training or adaptation. This leads to suboptimal performance in such scenarios or restricts their applicability. More detailed related work be found in Appendix H
6 CONCLUSION
In this paper, we address the challenge of merging multiple task-specific LoRAs into a unified model. We identify parameter interference, caused by misalignment and knowledge conflict, as the main ob- stacle. Our analysis reveals key LoRA properties: (1) Each rank operates as minimal semantic unit (MSU): (2) MSUs exhibit permutation invariance; (3) MSUs can be concatenated to form a com- prehensive LoRA. Building on these insights, we propose LORA-LEGO, which clusters MSUs from target LoRAs and uses cluster centroids to create a merged LoRA. Extensive experiments validate our approach's 's effecctiveness. Potential future work includes exploring alternative distance metrics for LoRA-LEGO, such as optimal transport, to more effectively capture parameter similarities be- yond the standard Euclidean distance. Additionally, further modularization of LoRA could improve various applications, such as federated learning. We believe these advancements have the potential to significantly benefit broad range of fields and applications.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (62441605, 62441617).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744 2023.
Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint vrXiv:2209.08836 2022.
Dengsheng Chen, Vince Junkai Tan, Zhilin Lu, Enhua Wu, and Jie Hu. Openfed: A comprehen- sive and versatile open-source federated learning framework. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 5018-5026, 2023.
Jiankang Chen, Tianke Zhang, Changyi Liu, Haojie Ding, Yaya Shi, Feng Cheng, Huihui Xiao, Bin Wen, Fan Yang, Tingting Gao, Taskgalaxy: Scaling multi-modal instruction fine-tuning with 2025.
tens of thousands vision task types. arXiv preprint arXiv:2502.
Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained language models. arXiv preprint arXiv:2302.07027, 2023.
Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Poria. Della-merging: Reducing interference in model merging through magnitude-ban based sampling. arXiv preprint arXiv:2406. 1617, 2024.
Shachar Don- Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint arXiv:2212.00178, 2022.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296 2021.
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. A survey on large language models: Applications, challenges, limitations, and practical usage. Authorea Preprints, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09688. 2021.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition, 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849) 2022.
Tapas Kanungo, David M Mount, Nathan S Netanyahu, Christine D Piatko, Ruth Silverman, and Angela Y Wu. An efficient k-means clustering algorithm: Analysis and implementation. JEEE transactions on pattern analysis and machine intelligence, 24(7):881-89 2002.
Erich Peter Klement, Radko Mesiar, and Endre Pap. Triangular norms, volume 8, Springer Science & Business Media, 2013.
11

---

Published as a conference paper at ICLR 2025
Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. Moelora: An mos-based parameter efficient fine-tuning method for multi-task medical applica- tions. arXiv preprint arXiv:2310.1883. 2023.
Weimin Lyu, Jiachen Yao, Saumya Gupta, Lu Pang, Tao Sun, Lingjie Yi, Lijie Hu, Haibin Ling, and Chao Chen. Backdooring vision-language models with duito--distriuution data, arXiv preprint arXiv:2410.01264 2024.
Michael S Matena and Colin A Raffel. Merging models with liser-weighhed averaging. Advances in Neural Information Processing Systems, 35:17703-17716 2022.
Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Do, and Dacheng Tao. Fusionbench: A comprehensive benchmark of deep model fusion. arXiv preprint arXiv:2406.088 03280, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09266 2023.
Vaswani. Attention all you need. Advances in Neural Information Processing Systems, 2017.
Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, and Maosong Sun. Lora-flow: Dynamic lora fusion for large language models in generative tasks. arXiv preprint arXiv:2402. 1455, 2024.
Xun Wu, Shaohan Huang, and Furu Wei. Mole: Mixture of lora experts. In The Twelfth International Conference on Learning Representations, 2023.
Chaojun Xiao, Zhengyan Zhang, Chenyang Song, Dazhi Jiang, Feng Yao, Xu Han, Xiaozhi Wang, Shuo Wang, Yufci Huang. Guanyu Lin, Yingfa Chen, Weilin Zhao, Yuge Tu, Zexuan Zhong, Ao Zhang, Chenglei Si, Khai Hao Moo, Chenyang Zhao, Huimin Chen, Yankai Lin, Zhiyuan Liu, Jingbo Shang, and Maosong Sun. Configurable foundation models: Building Ilms from modular perspective, 2024. URL htt.t:ttp////m //arxiv /tttttts://arxi org/ abs 02877
Prateek Yadav, Colin Raffel, Mohammed Mugeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning. arXiv preprint arXiv:2408.07057, 2024a.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Tees-merging: Re- solving interference when merging models. Advances in Neurul Information Processing Systems, 36, 2024b.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arX1v:2310.02555. 2023a.
Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, milms, and beyond: Methods, theories, applications and opportunities. 2024a.
arXiv preprint arXiv:2408.0706
Hongyang Yang, Xiao Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large language models. arXiv preprint arXiv:2306.06031, 2023b.
Jinluan Yang, Anke Tang, Didi Zhu, Zhengyu Chen, Li Shen, and Fei Wu. Mitigating the backdoor effect for multi-task model merging via saaet subspace. arXiv preprint arXiv:2410.1131 13910, 2024b.
Jinluan Yang. Dingnan Jin, Anke Tang, Li Shen, Didi Zhu, Zhengyu Chen, Daixin Wang, Qing Cui, Zhiqiang Zhang, Jun Zhou, et al. Mix data merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging. arXiv preprint arXiv:2502.0687 2025.
12

---

Published as a conference paper at ICLR 2025
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models super mario: Ab- sorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning. 2024.
Jinnyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang. and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp- 6915 6919. IEEE, 2024.
Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operations. arXiv preprint arXiv:2306. 14870, 2023.
Ziyu Zhao, Leilei Gan, Guoyin Wang, Yuwei Hu, Tao Shen, Hongxia Yang, Kun Kuang, and Fei
Wu. Retrieval-augmmnted mixture of lora experts for uploadable machine learning. arXiv preprint arXiv:2406. 16989, 2024a.
Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild. arXiv preprint arXiv:2402.09997, 2024b.
Ziyu Zhao, Yixiao Zhou, Didi Zhu, Tao Shen, Xuwu Wang, Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu, and Yu Cheng. Each rank could be expert: Single-ranked mixture of experts lora for multi-task learning. arXiv preprint arXiv:2501. 15000,15/0.. 103, 2025.
Didi Zhu, yibing Song, Tao Shen, Ziyu Zhao, Jinluan Yang, Min Zhang, and Chao Wu. Remedy: Recipe merging dynamics in large vision-language models. URL httts:://ppenreviee net Pert/Eorum3 iidiiddiii 1c==17eeHHEE
A DIFFERENCE BETWEEN LOAR MERGING SETTING AND MODEL MERGING SETTING
Previous work on model merging primarily focused on integrating separately trained models to form a comprehensive system. These methods typically involve reloading LoRA parameters into the orig- inal model before merging, which introduces additional overhead by necessitating the reconstruction of a corresponding LLM for each LoRA. In many cases, the goal of LoRA merging to create a new LoRA that consolidates the capabilities of all involved LoRAs for simplified task-specific us- age. In contrast, the LoRA merging setting presented in this paper bypasses the LoRA reload step; directly merges the LoRA parameters to construct a unified LoRA with comprehensive capabilities.
B CONNECTION WITH VANILLA LORA COMPOSITION METHODS
The vanilla LoRA composition can be categorised into two types of training free methods: the model ensembling and model merging Tang et al. (2024). The ensemble strategy involves aggregating the outputs of each submodule within the assembled LoRAs. Let us denote A = {Ai, A2, ..... An) and B B= (B, B2, ..., B,} } as the sets representing submodules within n LoRAs For an input ai the output derived from the ensemble of LoRAs can be expressed as æ 1 Σ j=1 B, A, where af denotes the output. The performance of the ensemble of LoRAs tends to be more stable, but it incurs additional computational overhead. In contrast to the ensemble method, model merging presents an alternative composition strategy. A typical strategy involves employing an glement-wiss. fusion of these parameters, represented as A' = 2 Σ A, and B' = - By. This formulation allows the merged parameters to function similarly to a single LORA. However, directly merging parameters can lead to performance degradation due to parameter interference.
Our proposed LoRA-LEGO method serves as a bridge between the two strategies, ensuring an opti- mal balance between computational efficiency and performance. By selectively aligning and fusing MSUs based on their semantic similarity, LoRA-LEGO effectively condenses the most relevant se- mantic features into fewer clusters. This process allows for the merging of parameters within each cluster, reducing the overall parameter count in a manner similar to the model merging method. By adjusting the number of clusters, LoRA-LEGOO accommodate more parameters for inference,
13

---

Published as a conference paper at ICLR 2025
much like the ensemble method. In this way, method leverages the strengths of both methodolo- gies, ultimately enhancing model performance and inference efficiency.
C DETAILS OF BASELINE METHODS
We compare our method with the following baseline:
1. Weight Averaging. This approach averages the parameters across different instances of LoRA, resulting in new composite LoRA defined A' 1. - A; and B'
B; where A, and B; represent the parameters from the instance of the origi- nal LoRA models, and n is the number of models being averaged.
Ei=1
2. Ensemble. This method averages the outputs from each LoRA, simultaneously activating multiple LoRAs compose combined output. The specific calculation for the mixed output is defined x' = - B,A
3. Task Arithmetic. weights derived from a nyper-parametee search to merge models. The calculations for this composite A' p Ei= A, and B' pE B;, where p represents the hyper- parameter that scales the contributions of each model.
This method is akin to weight averaging, but it differentiates by using
4. Ties-Merging. This method aims to resolve element-win wise parameter interference by ini- tially trimming the redundant parameters, retaining only the top-k% of values based their magnitude. then selects the sign vector for the merged model and finally performs disjoint operation. Tees-Mggging posits that the primary source of parameter inter- ference arises from inconsistencies in the values of merged parameters, while potentially overlooking issues related misalignment and knowledge conflict.
5. DARE. This method randomly drops the parameters and then rescales the remaining values to maintain the model's performance on the target tasks.
6. DELLA-Merging. This method pruning the parameters taking into account their magni- tude, then elects the parameters for merging reduce parameter interference.
D OPTIMAL SCALE OF MERGED LORA
Theorem D.I. Let A₁ E RRxx and Bi E RTXP, and A E Boxk and B2 E REXP, where all elements of these matrices are independently and identically distributed according to the standard normal distribution (0, 1). Then, after scaling the product Ag B2 by the factor /r/ VT, the variances of the entries of A, Bi and the scaled A, are equal:
Var (1 = Var
A₂B2) Vh
Proof. To compute the variance of the entries of the matrices A, Bi and each entry individually. For A₁ Bi. each entry is calculated as:
AgB2, we examine VE
Lf
(A, Σ (a))a(B));
1=1
Since (Ai)a ar and (Bi)i are independent and follow A/ 1), their product has mean zero and variance one:
E (AA)aa =, Var ((Ar)a (B)))) 1. The terms (A))a(B)) are independent for different I, so the variance of (A, is:
Var ((A)
Var ((Ar)a BB) =
14

---

Published as a conference paper at ICLR 2025
Table Multi-task performance when merging heterogeneous LoRAs on seven seen tasks and two unseen tasks.
IID Tasks
OOD Tasks WNLI
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
SNLI
Averuge
w/ Llama2-7h
Task LoRA
61.63
77.46
68.00
82.69
75.83
77.04
77.47
Weight Average
Task Arithmetic
Ties-Mering
Ensemble
56.06
55.84
69.75
64.91
74,85
74.44
70.92
46.19
52.86
62.87
LoRA-LEGO
55.10
60.67
69.25
67.29
65.61
67.04
74.83
57.82
52.86
63.39
Similarly, for A2 each entry is:
(A₂B2)1 Σ (A2)a(Be))
and each term (A2)a B2)15 has variance one. Therefore, the variance of (A₂B2) is:
Var ((AgBe)i))
Σ Var ((A2)a =
After scaling A₂B2 by VF/Vk, the variance becomes:
Var ( ( (Z =
Var ((A2 (()) x k =
Vkx
Thus, the variances of the entries are equal:
VF AB V/
Var (AiBi) = Var
E PERFORMANCE ON MERGING HETEROGENEOUS LORAS
Another advantage of LoRA-LEGO is its ability to merge heterogeneous LoRAs, that is, LoRAs with different ranks. To experimentally verify this feature, we retrained LoRAs for the QNLI, RTE, and SST2 tasks with = 16) to obtain a new LoRA. Since other model merging methods require the merged LoRAs to have the same architecture, we only compared our method with the Ensemble method. As shown in Tab the results demonstrate that our method can effectively merge heterogeneous LoRAs and achieves better overall performance than the Ensemble method.
16 and a 32, and merged them with LoRAs from other tasks (r 8,
PERFORMANCE ON MERGING LORAS FROM DIVERSE TASKS
To comprehensively evaluate different merging methods' performance on diverse tasks, con- ducted experiments with LoRA models trained across broad spectrum of NLP tasks. These tasks include RTE (natural language inference). CoPA (commonsense reasoning), IMDB reviews (senti- ment analysis), MRPC (paraphrase detection), BoolQ (reading comprehension), CosmosQA (read- ing comprehension with commonsense), and ARC (closed-book question answering). The deliber- ate selection of these highly diverse tasks enables a thorough assessment of merging methods across different linguistic capabilities and reasoning requirements.
The performance comparison of different LoRA merging methods on these diverse tasks is presented in Tablo As expected, the significant semantic disparities between tasks lead to increased parameter interference, causing substantial performance degradation in baseline methods such as ties-mer merging and DARE. contrast, BORA-sEGO's approach of parameter alignment before MSU-based merg- ing proves highly effective, achieving 11 % performance improvement over the strongest baseline,
15

---

Published as a conference paper at ICLR 2025
Table 6: Multi-task performance when merging LoRAs from diverse tasks,
Method
RTE
CoPA
IMDB
MRPC
BoolQ
CosmosQA
ARC(Easy)
Average
Weight Average
47.41
51.00
5.90
56.50
60.70
20.80
12.70
36.43
Fasemble
52.59
3.00
33.20
35.00
73.50
0.50
19.10
30.98
Ties-Merging
38.89
23.00
0.00
0.00
10.20
8.90
1.70
11.81
DARE
47.41
53.00
6.40
57.25
60.70
21.20
12.90
36.98
DRLLA- MMergingg
55.56
63.00
58.99
68.25
72.63
18.02
38.90
53.62
LoRA-LEGO
56.30
69.00
65.10
60.00
80.40
35,50
50.40
59.53
DELLA-Merging These results demonstrate that MSU-level alignment and merging strategy successfully mitigates parameter interference, particularly in challenging scenarios where task diver- sity amplifies interference effects. The superior performance validates approach's 's effectiveness in handling heterogeneous task combinations while maintaining model capabilities.
G EXPLORING THE SCALABILITY OF LORA MERGING
To rigorously evaluate the scalability different LoRA merging methods and their capacity to han- dle increasingly diverse tasks, designed comprehensive experiment spanning a broad spectrum of tasks and domains. Our evaluation encompasses the following task categories:
Natural Language Inference: RTE, CB, SNLL, MNLI (matched and mismatched), WNLI, QNLI
Commonsense Reasoning: CoPA, PiQA, StoryCloze
Sentiment Analysis: IMDB, SST-2
Paraphrase Detection and Similarity: MRPC, QQP, STS-B
Reading Comprehension with Commonsense: CosmosQA
Reading Comprehension: BoolQ, RTE
Closed-Book Question Answering: ARC (easy and challenge sets)
We conducted extensive experiments merging 20 LoRAs from these diverse tasks, with results presented in Tabl7] The findings reveal several critical insights about large-scale, multi-domain LoRA merging: Traditional element-wiss wise merging methods exhibit severe performance degrada- tion, with some merged models failing completely due to intensified parameter interference. The ensemble method, in particular, failed entirely due to irreoncciabble conflicts arising from disparate output spaces across multiple LoRA models. While DELLA-Merging achieved the best perfor- mance among baselines through its parameter pruning strategy, still struggled to fully address the fundamental challenges of parameter misalignment and knowledge conflicts inherent large-scale merging. In contrast, LoRA-LEGO demonstrated robust and superior performance in this chal- lenging scenario. By leveraging MSU-based structuring and integration of LoRA knowledge and parameters, our method effectively resolves parameter interference issues, achieving a substantial 16% improvement over the best baseline. These results convincingly demonstrate LoRA-LEGO's effectiveness in large-scale LoRA merging and its ability to maintain performance even under sig- nificant task diversity and scale.
H DETAILED RELATED WORK ON MODEL MERGING
Model Merging Methods. Model Merging has garnered significant attention, as aims to com- bine multiple task-specific models into a unified model capable of handling diverse tasks Tang al. (0024); Yang et al. (2024a). Previous work, such as Weight Averaging Tang et al. (2024), ap- proached model merging by directly averaging all the parameters. Task Arithmetic /Zhang et al. (203)); Iharco et al. (202)) introduced additional scaling factors to determine each model 8 con- tribution to the final output. Subsequently, methods like Fisher Merging Matena Raffel (2022) and RegMean Jin et al. (2022) incorporated additional labeled data merge models based on their relative importance. TeES-Merging Yadav et al. (2024b) tackled element-win wise parameter interfer- ence by trimming low-m magnitude parameters, resolving sign disagreements, and selectively merg- ing parameters with consistent signs. More recent approaches, such as DARE Yu et al. (2024) and DELLA-Merghh Deep al. (2024), further optimized TIES- T'ES-Merging by introducing parameter
16

---

Published as a conference paper at ICLR 2025
Table 7: Multi-task performance when merging LoRAs on 20 tasks.
Weight Average
Ensemble
Tite-Mrggig
DELLA-Merging
LoRA-LEGO
(Kacy)
3,401
ARC (Challeage)
1:30
0.00
2.50
1.30
27.90
35.20
Cal.A
0.00
0.00
0.20
0.00
$4.30
54.60)
QQP
0.00
0.00
0.20
0.00
60.20
57.30
PIQA
1.60
0.00
7.10
1.20
45. 15.10
45.60
SS12
1.38
0.00
7.24
74.02
18.00
0.00
QNLI
14.70
0.00
0.90
15.30
50.30
$2.50
StoryClas
40.00
01000
3.50
40.10
90
64.00
BoolQ
12.80
0.00
5,80
13.00
69.20
81.80
CosmusQA
9.30
0.00
3,90
9.70
16.40
16.50
MNLI
3.50
0.00
39.20
3,20
57.78
STSB
0.20
0.00
0.40
0.20
15.20
17.80
CB
12.00
0.00
4.00
8.00
74,00
74.00
MRPC
0.25
0.00
0.75
0..25
59.00
66.75
SNLI
0.50
0.00
0.70
35.80
55.10
WNLI
0.00
0.00
48.57
Average
8.39
0.00
2,60
48.03
pruning and rescaling techniques. However, these methods mostly rely on element-wise parame- ter merging and fail to address the alignment of parameters, which leads to significant parameter interference in more complex scenarios.
Application of LoRA Merging. LoRA merging can be applied in various scenarios. For instance, in multi-task task learning Tang et al. (2024); Don Yehiya et al. (2022). models adapt to different domains decentralized manner using LoRA, subsequently acquiring multi-task capabilities through merg- ing. In mixed-task scenarios Zhao et al. (2024b a). LoRAs from diverse domain tasks are uploaded centralized service platform, where service retrieves and composes LoRAs to deliver person- alized services based on downstream requests. In federated learning Chen et al. (2023), y /Zhang et al.)
20244,, edge devices train LoRAs on private data and upload them to a central server for merging and distribution, enabling iterative optimization through this process.
EXPLORATION OF AVERAGE MSU DISTANCES WITHIN LORAS
Our experiments in Tab T revealed an interesting phenomenon: while merging LoRA with its per- muted version typically leads to performance degradation due to MSU misalignment and parameter interference, certain tasks like RTE and MRPC showed slight performance improvements. We hy- pothesize that this unexpected benefit stems from these tasks' relative simplieity, where internal MSUs within their LoRA models maintain closer alignment. In such cases, permutation merging may actually serve beneficial regularization mechanism, potentially reducing overfitting.
To validate this hypothesis, we analyzed the average Euclidean distances between MSUs in each layer across all LoRAs, illustrated in Fig The analysis reveals that MRPC and RTE consistently exhibit smaller inter-MSU distances. indicating stronger internal parameter connectivity and poten- tial parameter redundancy. This structural characteristic explains why permutation merging acts as an effective regularization mechanism for these tasks: when MSUs are closely connected, the smoothing effect of permutation merging helps reduce overfitting, leading to performance improve- ments. Conversely, for tasks with larger MSU distances indicating weaker connectivity, permutation merging disrupts the parameter structure, resulting significant performance deterioration.
These findings underscore a crucial insight for LoRA merging: the effectiveness of merging strate- gies strongly depends on MSU connectivity patterns. Our clustering-basee approach capitalizes on this insight by specifically targeting and merging closely aligned MSUs, thereby minimizing param- eter interference and optimizing merger outcomes.
J EXPLORATION OF SEMANTIC RELATIONSHIPS IN MSUS ACROSS TASKS
To investigate whether MSUs capture meaningful structural semantic information, conducted comprehensive analysis using 48 LoRA models, each trained independently on different datasets from Flan-y2 and categorized into 10 distinct task clusters. Our analysis focuses visualizing the
17

---

Published as a conference paper at ICLR 2025
COLA
MAPC
0.80
MNLI
QNU
RTE
Average Distance
0.78
0.72
15
20
25
30
35
40
50
55
60
Layer Number
TSNE LoRA MSUs Layer
440.1 pno
53000 Scuting Com
medet
p
Figure 10: Average Euclidean distance among MSUs of different layers within each LoRA. semantic relationships between MSUs by examining their distributions in the first layer parameters of these LoRAs
To uncover potential patterns in MSU organization, employed t-SNE project MSUs from the initial layers of these models into a unified visualization space. This approach allows us to exam- ine how MSUs distributed across different tasks and domains, potentially revealing underlying semantic relationships and organizational principles.
The visualization results, presented in Fig I0 reveal significant patterns: First, MSUs within individual LoRA tend to form distinct clusters, reflecting their shared training on specific data dis- tributions. Second, and more notably, MSUs from LoRAs within the same domain exhibit spatial proximity in the t-SNE space, suggesting the capture of domain-speciffe semantic features. These findings have important implications for LoRA merging: they demonstrate that MSU similarity is crucial consideration in the merging process, as clustering and merging similar MSUs can minimize parameter interference while preserving semantic integrity. This observation supports our approach of MSU-based alignment in LoRA merging, showing how it helps maintain semantic coherence while reducing conflicts during merging.
18


Reviews:
[{"review_id": "WtlTGte06Z", "reviewer": "Reviewer_1z1o", "summary": "The paper introduces the LoRA-LEGO framework, which combines multiple LoRAs for multi-task and mixed-task scenarios. It uses clustering to group MSUs from different LoRAs into k clusters before constructing a merged LoRA based on the cluster centroids. Parameter reweighting is then performed to scale the centroid with respect to the average cluster norm. Experiments are conducted against 4 baselines (weighted average, ensemble, task arithmetic and ties merging) for multi-task and mixed-task scenarios, showing improved average performance.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "After reading the authors' rebuttals, my concerns have been adequately addressed.", "weaknesses": "1. The paper is generally well written with good illustrations. However, I found the approach not particularly novel. It relies on a straightforward k-means clustering method for grouping and subsequently aggregating cluster parameters.\n\n2. The approach is not particularly strong, and this simplicity may limit the framework’s performance which is reflected in the empirical results. While the average performance across tasks improves, it does not apply to all tasks. For example, performance remains poor for WNLI, an out of domain task for multi-task learning.\n\n3. The baselines involving LoRA composition methods are basic and are not established methods in existing literature. It lacks citations that would align them with related work. This weakens the comparative analysis and raises questions about the validity of the selected baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Syi7qBDF9t", "reviewer": "Reviewer_9LLA", "summary": "This paper proposed the LoRA-LEGO framework for combining Low-Rank Adaptation (LoRA) modules in large language models (LLMs). LoRA-LEGO introduces Minimal Semantic Units (MSUs), independent components within LoRA, enabling flexible merging through clustering and rank adjustment. This modular approach allows task-specific LoRAs to be disassembled and reassembled, tackling parameter misalignment and knowledge conflicts seen in traditional merging methods. The dual reweighting strategy optimizes the scale of the combined LoRA, resulting in a more efficient and flexible system that performs well across both in-domain and out-of-domain tasks.", "questions": "1. How does LoRA-LEGO address performance degradation when merging LoRAs for tasks with significant divergence?\n\n2. Why does performance improve for MRPC and RTE tasks in Table 1 following permutation merging?\n\n3. How can the optimal number of clusters $k$ in equation (2) be determined?\n\n4. Why does the weight averaging method in Table 3 achieve identical performance to the task-specific LoRA (upper bound) for the MRPC task?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* Modularity and Flexibility: The MSU-based modular approach enhances flexibility in merging LoRAs, allowing for custom rank adjustments.\n\n* Efficient Scaling: Dual reweighting improves output scale management, maintaining model performance across tasks.", "weaknesses": "* Performance may degrade when merging LoRAs designed for highly divergent tasks.\n\n* Merging effectiveness can be highly sensitive to hyperparameter settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A8ksNgRESd", "reviewer": "Reviewer_Uiyw", "summary": "The paper presents an innovative framework called LoRA-LEGO, which enhances the merging of LoRA for large language models. The approach leverages the modularity of LoRAs by introducing Minimal Semantic Units (MSUs) and employs rank-wise clustering to optimize parameter efficiency and performance. The framework aims to overcome limitations in existing methods by reducing parameter interference and maintaining task-specific knowledge.", "questions": "see weakness.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The concept of treating LoRA parameters as LEGO blocks, specifically through MSUs, is creative and offers a fresh perspective on modularity in model adaptation.\n- The paper provides solid theoretical foundations, including permutation invariance and concatenation-summation equivalence, which support the proposed method.\n- Extensive experiments demonstrate the framework's effectiveness across multiple benchmarks, outperforming existing methods.", "weaknesses": "- The MSU is not related to some semantics that can be understood by humans, it would be better if we could see some analysis.\n- Although the proposed method outperforms existing approaches, a more detailed comparison with a wider range of baseline methods could strengthen the claims.\n- It does not report the overhead (e.g., in inference and training time, memory) that may introduced by this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ebSBjry0mt", "reviewer": "Reviewer_YS49", "summary": "This paper introduces the concept of Minimal Semantic Unit (MSU) for disassembling and reassembling multiple LoRAs at a finer granularity. MSUs have permutation invariance and concatenation summation equivalence properties and thus enable flexible combinations for create new LoRAs. Evaluation results show the MSU-based approach achieves a good performance.", "questions": "How can the finer-grained merging of LoRAs preserve the performance of the merged LoRA, particularly in comparison with the approach of selecting individual LoRAs? How many LoRAs can be supported by the MSU-based approach, and what performance it can achieve with more (e.g., tens of) LoRAs?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ MSU is an interesting concept with permutation invariance and concatenation summation equivalence properties.", "weaknesses": "- While MSU is an interesting concept, it is unclear why the finer granularity merging of LoRAs is the right approach for combing multiple LoRAs to enhance LLM capabilities: the clustering and the use of the centroid of each cluster essentially impact the performance of the merged LoRA. \n- The evaluation used only 7 LoRAs. It is unclear how the proposed approach may work with more LoRAs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)