SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
Authors: Yong Liu
Di Fu
Shenggan Cheng
Zirui Zhu
Yang Luo
Minhao Cheng
Cho-Jui Hsieh
Yang You
Venue: ICML 2025 poster (2025)
Keywords: Parameter Efficient Fine-Tuning
Model Merging
Large Language Model
Primary area: optimization

Abstract: Despite Low-Rank Adaptation (LoRA)'s popularity for fine-tuning large models, it often exhibits a noticeable performance gap compared to full fine-tuning, particularly in complex tasks such as mathematical reasoning and code generation. Motivated by this discrepancy, we propose a novel fusion approach for LoRA fine-tuned models. Our key insight is that LoRA models trained with different random seeds on the same task often exhibit complementary strengths. In contrast to existing research that typically focuses on fusing models trained on diverse tasks, we explore the potential of combining multiple LoRA models fine-tuned on the same task with different random seeds. This intra-task fusion method aims to leverage the strengths of various fine-tuned models to create a more robust and effective adaptation. To validate our approach, we conducted comprehensive experiments across three key areas: mathematical reasoning, code generation, and general instruction-tuning tasks. The results demonstrate that our fusion method significantly enhances LoRA's performance, outperforming both standalone LoRA models and current fusion methods. Notably, this advancement substantially narrows the gap between LoRA and full fine-tuning, thus offering a more effective approach to model adaptation without the GPU memory burden of full parameter fine-tuning.

SeedLoRA: A Fusion Approach to Efficient LLM Fine- Tine-Tuning
Yong Liu'
Di Fu'
Shenggan Cheng' Zirui Zhu'
Yang Luo
Minhao Cheng? Cho-Jui Hsieh 3 Yang You
Abstract
has gained popularity due to its effectiveness and simplic- ity. Despite its advantages, LoRA often exhibits noticeable performance gap compared to full fine-tuning approaches, limiting its applicability in scenarios where state-of-the-un performance is required.
Despite Low-Rank Adaptation (LoRA)'s popular- ity for fine-tuning large models, it often exhibits a noticeable performance gap compared to full fine-tuning, particularly in complex tasks such as mathematical reasoning and code generation. We propose SeedLoRA, novel fusion approach that bridges this gap by leveraging complemen- tary strengths of multiple LoRA models trained with different random seeds on the same task. Un- like existing model merging methods that focus on combining knowledge from different tasks, SeedLoRA introduces a two-stage fusion strat- egy specifically designed for single-task scenar- ios: first identifying and preserving strong shared patterns across models, then performing princi- pled subspace fusion in unified representation space. Comprchensive experiments on LLaMA2- 7B and Mistral-7B demonstrate that SeedLoRA significantly improves performance over individ- ual LoRA models by 4.9% on GSMSK and 6.6% on HumanEval, effectively matching or exceed- ing full fine-tuning performance while maintain- ing the efficiency benefits of LoRA. Our analysis reveals that this improvement stems from Seed- LoRA's ability to effectively combine comple- mentary strengths Iearned by different seeds in a common representation space.
Researchers have proposed various approaches to narrow the performance gap between LoRA and full fine-tuning in LLMs. These methods typically fall into three cate- gories: increasing LoRA's capacity, optimizing LoRA's structure, and combining multiple LoRA adaptations. For instance, ReLoRA (Lialin et al., 2024) proposes periodically increasing the rank during training, while DoRA (yang Liu et al., 2024) and MiLoRA (Wang et al., 2024a) suggest alternative low-rank structures and initialization strategies. Techniques such as MultiLoRA (Wang al., 2023b) and MoLoRA (Zadouri et al., 2024) attempt to leverage multi- ple LoRA modules, inspired by Mixture of Experts models. While these approaches have shown improvements, they of- ten come at the cost of increased computational complexity or fail to fully close the gap with full fine-tuning, particu- larly in challenging domains like mathematical reasoning and code generation.
In our investigation of these limitations, we made key observation: models trained on identical tasks with dif- ferent random seeds exhibit similar overall performance, yet demonstrate varying proficiency across different subdo- mains of the task. This opens the opportunity to combine their strengths into more robust model.
Inspired by this insight, naturally turn model merging techniques, which have gained significant attention in the field of LLMs as means to combine knowledge from mul- tiple models without increasing inference costs. However, we find that applying existing merging methods to our sce- nario presents unique challenges. Specifically, most existing work on model merging focuses on multi-task scenarios, aiming to integrate capabilities from models trained on dif- ferent tasks (Wortsman et al., 2022; Ilharco et al., 2022). In contrast, our experiments reveal that the challenges faced in single-task model merging- our focus-diffee substan- tially from those in multi-task scenarios. To elucidate this distinction, our analysis of cosine similarities reveals a cru- cial difference: models trained on different tasks exhibit near-zero similarity, indicating orthogonality, which leads
1. Introduction
Parameter-Effiiiees Fine-Tuning (PEFT) methods have emerged as promising training schemes in fine-tuning large language models (LLMs), offering a balance between perfor- mance and efficiency. Among these, LoRA (Hu al., 2022)
DDeparmentt Computer Science, National University of Singapore "College of Information Sciences and Technol- ogy, Pennsylvania State University 'Department of Computer Science, University of California, Los Angeles. Correspon- dence to: Yong Liu <liuyon@@comp.nu edu.sg>, Yang You www.:/oommu........mo
our
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
to interference issues in multi-task merging. Conversely, models trained on the same task with different seeds show high cosine similarity, suggesting a high degree of shared information. This fundamental difference shifts the primary challenge in single-task merging from interference mitiga- tion to effective information combination and redundancy elimination, necessitating a new approach tailored specifi- cally to single-task model merging.
2. Preliminaries and Related Work
2.1. LoRA
LoRA enables efficient LLM fine-tuning by introducing trainable low-rank matrices while keeping original weights frozen. For a pre-trained weight matrix W E Rdxk LoRA introduces the update W = W+ BA, where B E Rdxr and A E g are low-rank matrices with rank << min k). Recent research has expanded upon the LoRA framework, exploring various enhancement. These include novel initial- ization method of A and B matrices (MiLoRA (Wang et al., 2024a), Pissa (Meng et al., 2024a), LoRA-GA (Wang et al., 2024b)), higher-rank approaches (MoRA (Jiang et al., 2024), PeriodicLoRA (Meng et al., 2024b), ReLoRA (Lialin et al., 2024)), innovative structural modification(DoRA (yang Liu et al., 2024)), advanced training (LoRA+ (Hayou et al., 2024)).
Building on these insights, propose SeedLoRA, novel approach to address the unique challenges of single-task model merging. Our approach capitalizes on the high cosine similarity and shared information between models trained on the same task with different seeds, focusing on effective information combination and redundancy elimination.
At the core of SeedLoRA is a two-stage merging strategy that addresses the distinct aspects of model fusion. The first stage focuses on identifying and handling extreme parameter values- - both consistently large magnitudes and conflicting directions- across different seed-specific models. This stage ensures robust integration of strongly learned patterns while preventing destructive interference. The second stage em- ploys sophisticated SVD-based fusion approach for the remaining parameters, operating in a shared subspace to effectively combine complementary information while elim- inating redundancy. By decomposing the merging process into these distinct stages, our method can better preserve the unique strengths of individual models while creating a more robust combined representation.
Drawing inspiration from the Mixture of Experts (MoE) paradigm, this approach dynamically combines multiple Lo- RAs, each potentially specialized for different tasks or do- mains. Examples include MultiLoRA (Wang et al., 2023b), MoLLRR (Zadouri et al., 2024), LoRAHub (Huang et al., 2023), and HydraLoRAA (Tian et al., 2024). By leveraging both LoRA's parameter efficiency and the adaptive capacity of expert models, Mixture of LoRA aims to create more versatile models that can perform effectively across a board range of tasks than single-adaptation LoRA implementa- tions.
Our experimental results demonstrate the effectiveness of ScedLoRA. By merging multiple LoRA models with a rank of 8, we achieve performance comparable to full fine-tuning in challenging tasks such as mathematical reasoming and code generation. This approach not only narrows the per- formance gap between LoRA and full fine-tuning but also preserves the efficiency advantages of PEFT methods.
2.2. Model Merge
Model merging aims combine the knowledge encoded in multiple trained models into a single, enhanced model. Cur- rent research in model merging focuses on two main areas: Multi-task Merging and Same/Similar-task Merging. Multi- task merging combines models trained on different tasks into a single model capable of performing multiple tasks, leveraging task-specific knowledge, and maintaining effi- ciency. Same/Similar-tass merging, though less explored, focuses on combining models trained on identical or closely related tasks to enhance robustness and generalization, with studies showing improved performance on shifted data dis- tributions. Most work in this area has been conducted in computer vision, leaving significant opportunities for appli- cation in fields like natural language processing. Although a range of methods (Ilharco et al., 2022: Lu et al., 2024; Verma & Elbayad, 2024; Huang et al., 2024; Salamanca et al.; Tam et al.; Deep al., 2024; Lu et al., 2024) for model merging have been proposed, this paper focuses methods that provide distinct ways of combining model pa- rameters. Model soup (Wortsman et al., 2022) improves the accuracy of fine-tuned models by averaging the weights of multiple models fine-tuned with different hyperparameters, rather than selecting only the best individual model.
The main contributions of this paper are:
A comprehensive analysis of the performance character- istics of LoRA models trained with different seeds the same task, revealing their complementary strengths in various subdomains.
Insights into the fundamental differences hetween single- task and multi-task model merging, highlighting the need for specialized approaches in each scenario.
The introduction of SeedLoRA, a novel intra-task model merging method that effectively combines information from multiple models while eliminating redundancy.
selected set of
Extensive empirical evidence demonstrating the effective- ness of SeedLoRA in narrowing the performance gap be- tween LoRA and full fine-tuning, particularly in complex tasks like mathematical reasoning and code generation.
2

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
3. Proposed Method
different LoRA models, each trained with unique random seeds, tend excel in distinct subdomains. Building on this insight, we formulated hypothesis: by strategically merging multiple LoRA models, each is trained with a distinct seed and with its own specialized strengths, we could potentially achieve performance comparable to full fine-tuning.
key observation:
3.1. Motivation
Performance Gap between LoRA and Full Fine- -unnng
Prior research (Biderman al., 2024) shows LoRA's per- formance gap persists even with increased rank extended training. In Table 1, we conducted the experiments fine- tuning LLaMA2-7B with LoRA from rank-8 to rank-64 on math (MetaMathQA) and code generation (CodeFeedback) tasks (Yu et al., 2023). The results reveal that although increasing the rank value of LoRA from 8 to 64 improves performance on GSM8K and MATH from 39.64 to 41.05, significant gap remains when compared with Full Fine-
LoRA (seed 11)
46
LoRA (seed 42)
LoRA (seed 202) Full FT
Performance
42
H
40-
Tuning. A similar trend observed in code generation tasks, where increasing the rank initially improves perfor- mance, but at extremely high ranks, such as 64, performance begins to decline. This indicates that while higher ranks can lead to gains, LoRA still struggles to fully match the perfor- mance of full fine-tuning, particularly in complex domains such as mathematical reasoning and code generation. These observations motivate the need for a novel training strategy and optimization method to further narrow the performance gap, enhancing LoRA's applicability across a wider range of tasks.
38
36
32-
30
pro_accoooo vro_accounting
pro_aaw
college_cs
college_math
Figure I. Performance comparison of LoRA and Full across MMLU subdomains.
The Definition of Single- Task model Merging. To explore our hypothesis of combining LoRA models with diverse strengths, we turn to the concept of model merging, which the process of combining multiple models to enhance overall performance. While model merging is typically applied to integrate models trained on different tasks, we propose novel application: merging models trained on a single task with different random seeds to achieve superior performance within that task. The proposed method defined as follows:
Task
r=8
r=16
r=24
r=32
r=
FT
GSM8K
64.0
65.6
64.9
64.7
65.6
66.5
MATH
15.3
15.3
16.3
16.6
16.5
19.8
Average
39.6
40.4
40.6
40.7
41.1
43.2
Table 1. Fine-Tuning L.L.MA--7BB model with LoRA on Meta-
Let pre be the pre-trained base model, and {01, 02. ... be set of n models men-tuned on the same task using LoRA, each with different random seed 8;. Each fine-tuned model 0, can be represented as 0; = tTr where is the LoRA delta model for the i-th fine-tuned model, obtained using seed s;. We then aim to merge these seed-specific delta models:
.e......0. 0,,
MathQA (seed=1l).
3.2. Narrowing the Performance Gap via Model Merging
Analyzing LoRA and Full Fine- TuninT Performance. We conducted comprehensive analysis to better understand the performance discrepancy between LoRA and full fine- tuning across various subdomains. Our approach involves visualizing the performance of multiple models trained with different random seeds using both LoRA and full fine-tuning techniques. Specifically, we leverage the Massive Multitask Language Understanding (MMLU) benchmark, which cov- ers wide range of subjects and allows for fine- iineggriiedd per- formance analysis. Figure 1 illustrates the performance of LoRA and full fine-tuning models across different MMLU subdomains. The results reveal an interesting pattern: while LoRA models generally underperform compared to full fine- tuning, they exhibit competitive performance in specific subdomains. This nuanced performance distribution led to a
Tm Merge(TI, T2...., Tn).
(I)
The merging is performed layer-wise for each LoRA adapter:
(8) F Merge (8) .T2
(9).
(2)
where 7. represents the ê-th layer of the i-th delta model trained with seed s;. The final merged model is obtained by:
4. = 8 + Tm = Opre + Merge( (T1, T2...., T.).
(3)
This approach leverages the diverse strengths of multiple LoRA models, each potentially excelling in different sub- domains, to create a merged model that approaches or even
3

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
Matth
Math
Mathh
Instruet
Instruct 2
Instruct
These fundamental differences explain why current main- stream multi-task merging techniques perform suboptimally in single-task scenarios. Methods like T'ES-Mggging and DARE primarily address interference from unrelated knowl- edge in orthogonal multi-task models. However, for highly similar and redundant single-task models, these strategies often fail to capture and integrate the subtle complementar- ity embedded within shared information. Their core design wasn't t for extracting insights from high similarity, making them inefficient or ineffective here.
1.
Math
1.00
0.28
0.30
0.00
0.00
Math
0:28
1.00
0.29
0.00
0.00
Math
0.30
0.29
1.00
0.00
0.00
0.00
0.6
Instract 1- 0.00
0.00
0.00
1.00
0.24
0.20
0.4
Instruct 2- 0.00
0.00
0.00
0.24
1.00
0.22
These limitations underscore the urgent need for novel method tailored to single-task model merging. An effective strategy must leverage high knowledge sharing and manage redundancy, but also precisely extract and integrate com- plementary information from training randomness while mitigating minor interference.
Instruct 0.00
0.00
0.00
0.20
0.22
-.0
Figure 2. Cosine similarity comparison between LoRA delta mod- els trained on the same and different tasks.
3.4. SeedLoRA: A Two-Stage Approach for Model Merging
surpasses the performance of full fine-tuning models. By focusing on models trained on single task with different seeds, we capture a broader spectrum of task-specific knowl- edge while maintaining LoRA's efficiency advantages.
To tackle the redundancy, complementarity, and potential interference observed in single-task LoRA model merging, we propose SeedLoRA, a novel two-stage fusion strategy. This approach is designed to precisely integrate multiple LoRA adapters fine-tuned on the same task but initialized with different random seeds.
3.3. Difference between Multi-Tass Task and Single-Task Model Merging
Before delving into our proposed single-task model merge method, we conducted a cosine similarity analysis to high- light its crucial differences from traditional multi-task ap- proaches. To illustrate this, we trained LoRA models: 3 for a mathematical reasoning task and 3 for instruction following task, each with distinct random seeds (seed=1, 2, 3). Figure 2 shows our findings, highlighting a significant contrast in LoRA delta model (r = BA) cosine similarity between multi-task and single-task scenarios, profoundly impacting merging strategies:
The first stage effectively manages inter-model redundancy and resolves potential interference by identifying and han- dling extreme parameter dimensions-thor with consis- tently large magnitudes (robust) or opposing signs (conflict- ing). The second stage then targets the residual dimen- sions, which encompass all other parameters that survived the initial filtering. These residual dimensions contain sub- te variations and complementary information that cannot be effectively merged through simple averaging, necessitat- ing sophisticated subspace fusion techniques to extract and integrate their contributions within unified low-rank repre- sentation space. Through this divide-and-conquer strategy, we can comprehensively leverage shared knowledge, un- cover unique complementarities, and mitigate interference, thereby narrowing the performance gap between LoRA and full fine-tuning.
Delta models from different tasks cxhibit near-zero cosine similarity, indicating orthogonal learned knowledge. For multi-task merging, the main chal- lenge is aggregating this diverse knowledge and resolving interference from unrelated information, aiming to retain unique contributions without compromising other tasks.
Multi-task Scenarios:
Stage 1: Identifying Robust and Conflicting Dimensions.
Single-task Scenarios: In contrast, Delta models trained on the same task with different random seeds consistently show significantly positive cosine similarity. This reveals strong inherent correlation and high knowledge sharing (redundancy). Consequently, for single-task merging, the core challenge shifts to identifying and integrating subtle complementary information from training randomness. While interference risk exists, precisely extracting truly beneficial and unique contributions from this abundant, similar, and redundant information becomes critical.
Given n LoRA adapters, each adapter т; introduces low- rank parameter increments on top of pre-trained model Opre- Let 7, (j) represent the value in dimension j of the i-th adapter. We begin by defining threshold to mark "large" magnitudes, and we sean through every dimension j across all n adapters:
Robust Dimensions. If a sizeable subset of adapters exhibit [T()) 2 and share the same sign, we mark these entries
4

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
as robust. Their consistent sign and large magnitude suggest that most adapters converge on similar direction for that parameter. These robust dimensions represent the highly consistent shared core knowledge among models. illustrat- ing their inherent redundancy. In practice, we average them to retain their collective strength, thereby effectively pro- cessing and leveraging this redundancy, and emphasizing the core shared patterns among models.
where n is the total number of adapters. This step collapses moderate differences into single, consolidated update.
Step 2: SVD-Based Low-Rank Decomposition. While the averaged matrix Mavg provides a consolidated represen- tation, directly merging individual adapters in the original parameter space may not effectively capture their comple- mentary strengths. SVD decomposition identifies the princi- pal directions of variation across models and establishes a common coordinate system where model differences can be systematically compared and merged, enabling more precise fusion of complementary information.
Trabust (j)
1
Σ I
T(U),
(4)
where I, is the set of adapters whose value in dimension j exceeds T and matches the dominant sign.
We perform truncated singular value decomposition on Mag.
Confficting Dimensions. If multiple distinet groups of adapters produce equally large but opposite-sign values in the same dimension, that dimension is regarded as a conflict- ing dimension. This indicates strong learning opposition in that specific parameter direction. If left unaddressed, this opposition will lead to negative interference at the model level, significantly undermining the effectiveness of straight- forward averaging. Following related work on multi-task model merging (e.g.. TIES (Yadav et al., 2023)), we resolve these conflicts by assigning them to one of the dominant sign groups (e.g., retaining only the majority sign). This step effectively prevents interference caused by parameter con- flicts from contaminating subsequent processing, ensuring the robustness of the fusion.
M U( V(OT.
(6)
The rank r is chosen to capture the principal directions while discarding less important components. The matrices U(()) gaxr and ((0) e R (for layer of shape d x k) define shared basis in the row and column spaces, respectively, whereas ( indicates the importance of each direction.
Step 3: Re-Projection of Individual Adapters. Each adapter's parameters 7,() can then be projected onto this common subspace. In this way, multiple single models can be aligned and easily edited or merged. Specifically, we compute a coordinate matrix:
z, = (V'1) -(0) V())
(7)
Stage 2: Subspace Fusion for Residual Dimensions
which projects adapter i the common subspace. By this way, we can further edit fuse these projected matrices ( } to merge the knowledge from different models in the next step.
After addressing parameter redundancy and interference in Stage 1, our second stage focuses on integrating com- plementary information from the remaining dimensions. These Residual Dimensions hold distinct, valuable contribu- tions from different models. Since directly combining these unique patterss in their original high-dimensional space is challenging, our Subspace Fusion approach leverages SVD to align and combine their underlying complementary insights within a common subspace. This enables more precise identification and combination of complementary strengths, while also filtering out potential noise.
Step 4: Fusion and Reconstruction. To form single merged model for these dimensions in stage 2, we merge
{ (z(²)) } into a fused set of coordinates 700 Specifically, we reuse existing fusion methods like TIES, DARE, and Weighted Averaging. Finally, we reconstruct:
(
( 2 VOO
Fused
(8)
The specific process consists of the following steps:
This yields single low-rank representation of the parame- ters in stage 2.
Step 1: Dimensionwise Averaging.
Final Assembly of the Fused Adapter. We assemble the fi- nal fused LoRA adapter Thinll by combining these subspace- fused residual parameters with the robust and conflict out- comes from Stage 1. In particular, for each layer E:
For each layer É, consider the original updates 7. from the i-th adapter. For each dimension j that was not classified as robust conflict in Stage 1 (i.e., a residual dimension), we build a single averaged value M by averaging across all adapters i:
(f)
(t)
(E) Trused
final =
Trobust +
Toonflict
+
(9)
avg (f) = Σ
(2)
(5) where Trbuutt includes dimensions identified as consistently
T
large and same-sign, Fused is the outcome of the SVD-
(1)
5

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
Method
seed 11
seed 42
seed 202
SeedLoRA
Model Soup
TIES
DARE
Evaluating LLaMA2-7B on GSM8K. The performance Full Fine-Tuning 66.5.
LoRA (r=8)
64.0
63.8
64.1
68.9
66.6
65.7
65.7
LLaMA2-7B
LoRA+ (r=8)
64.4
64.7
65.4
69.8
67.0
63.2
56.7
DoRA (r=8)
64.6
64.7
64.7
68.9
66.3
66.0
67.3
Evaluating LLaMA2-7B MATHL The performance of Full Fine- Fine-Tuming is 19.8.
LoRA (r=8)
15.3
15.3
14.9
17.8
15.7
16.0
15.5
LLaMA2-7B
LoRA+ (r=8)
15.4
15.5
16.0
18.2
16.1
16.7
16.4
DoRA (r=8)
15.4
15.4
14.9
17.8
16.0
15.8
15.6
Evaluating Mistral-7B on GSM8K. The performance of Full Fine-Tuning is 78.6.
LoRA (r=8)
75.4
75.7
76.3
80.7
79.1
75.1
75.1
Mistral-7B
LoRÅ+ (r=8)
76.5
73.5
75.9
80.3
79.7
79.4
78.7
DoRA (r=8)
77.0
75.7
76.5
81.0
77.0
79.1
78.5
Evaluating Mistral-7B on MATH. The performance Full Fine- Toning 28.5.
LoRA (r=8)
25.9
24.8
25.4
28.8
28.5
24.8
25.0
Mistral-7B
LoRA+ (r=8)
25.1
25.2
25.4
28.0
27.9
25.9
24.3
DoRA (r=8)
25.9
25.3
25.8
29.0
28.3
26.5
25.7
Table 2. Fine-Tuning LLaMA-2-7B and Mistral-7B with LoRA MetaMathQA.
based merging residual entries, and confflitt reflects any decisions made about sign-divergent dimensions.
(2)
utilized in training on the LLaMA2-7B and Mistral-7B-v0l
Following the setting of Open-Instruct (Ivison et al., 2023), we evaluate model on MMLU (Hendrycks et al., 2020), GSM8K, BBH (Suzgun et al., 2022), TyDiQA (Clark et al., 2020), TruthfulQA (Lin al., 2021) and HumanEval.
Once aggregated across all layers, the result is a single low- rank update Trinal that retains the efficiency of LoRA while exploiting each adapter's advantages. Robust dimensions preserved in their collectively strong direction, conflicts neutralized to avoid destructive interference, and resid- ual parameters are reconciled through shared subspace.
4.2. Mathematical Reasoning
validate the efficacy of our proposed merge method, we first evaluate the LoRA models with 3 different seeds on GSM8K and MATH, followed by assessment of merged model. The experimental results, shown in Table 2, demonstrate that the merged model substantially improve the performance of cach independent model. Notably, for LoRA fine-tuning on LLaMA2-7B, SeedLoRA can improve the performance vanilla LoRA from 64.1 to 68.9 on GSMSK and from 15.3 17.8 on MATH. Furthermore. to evaluate the generalizability, we extend our evaluation to LoRA variants (such LoRA+ and DoRA) and more advanced pre-trained LLM (such Mistral-7B). These ad- ditional experiments consistently demonstrate performance improvement when using our model merging approach. Fi- nally, we also conduct experiments to compare SeedLoRA with current popular model merge methods, such as Model Soup (Wortsman al., 2022), TIES (Yadav et al., 2023) and DARE (Yu et al., 2024). The experimental results Table 2 illustrate these methods can also improve the per- formance of vanilla LoRA, but SeedLoRA can obtain more performance gain.
4. Experimental Results
4.1. Experimental Setting
Training and Evaluation: For code generation, we use Code-Feedback (Zheng al., 2024) as training data, LLaMA2-7B (Touvron al., 2023) and Mistral-7B-v0.11 (Jiang et al., 2023) serve base models. We evalu- ate using HumanEval (Chen et al., 2021), an established benchmark for Python text-to-code generation. For com- prehensive assessment, we incorporate HumanEval+ from EvalPlus (Liu et al., 2024). For math reasoning, the Meta- MathQA (Yu al., 2023) dataset employed fine-tune the ILAMMA-7B and Mistral-7B models. The evalua- tion is conducted using the GSM8k (Cobbe al., 2021) and MATH (Hendrycks et al., 2021) benchmarks, which are specifically constructed to test the model's capacity for mathematical reasoning and problem-solving. For the gen- eral domain, the TÜLU V2 (Wang et al., 2023a) dataset is
6

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
LoRA (rm8, seed=11)
1oTA [pm14] 1o (am02)
LoRA Ir= epoch=
LoRA (1-8, anod=42)
LoRA I6-16, epoch
Singolar Valum (log)
SeedLoRA (sood=11 and 42)
0.25
ir=8, opoch-6)
10P
Eval Performance
Traieing Loss
Training Loss
0.20
10-1
0:35
fo
10
0.30
10
Index
15
25
30
Training Loss
Performance
29
500
1000
(*100)
2500
2000 3500
(a) Singular Value Analysis
(b) Generalization Analysis
(c) Training Loss Comparison
Figure 3. (a) Singular Value Analysis. (b) Generalization Analysis (c) Training Loss Analysis.
4.3. Code Generation
uation to general domain instruction tuning tasks. The ex- perimental results, shown in the Table 4, demonstrate that our proposed method continues to improve upon the per- formance of the best individual model. However, the mag- nitude of improvement in this domain is less pronounced than observed in math reasoning and code generation. We believe this discrepancy arises from the nature of general do- main tasks, where models are required to follow instructions rather than acquire new knowledge, as is often necessary for mathematical and coding tasks. Moreover, this observation underscores the efficacy of our method while also highlight- ing the challenges of achieving substantial gains areas where LoRA already performs close full fine-tuning.
Building upon our findings in mathematical reasoning, we further evaluate the performance gain of our merged model on the Code Generation task. Table 3 presents the exper- imental results of individual training models and merged models on CodeFeedback benchmark. The data demon- strates that merged model consistently outperforms in- dividual models in the HumanEval and HumanEval+ tasks. Particularly. ScedLoRA exhibits exceptional performance on HumanEval (+) benchmark, surpassing the best individ- ual LoRA by both 6.1% on LLaMA2-7B and Mistral-7B.
To contextualize our method's performance within a broader range of model merging approaches, we conducted com- parative analysis with popular approaches such as Model Soup, TIES and DARE. Our finding indicates our method achieves superior performance compared to these existing merge methods. For instance, our method enhances the per- formance of model soup from 34.1% to 40.2% for LoRA.
4.5. Further Discussions
Why Merging Models from the Same Task Can Improve the Performance? To understand the performance improve- ments achieved by merging models from the same task (but different seeds), we conduct two key analyses: knowledge fusion and generalization ability.
LoRA
SeedLoRA
MS
TIES
DARE
Firstly, we evaluate whether the merged model can effec- tively fuse the knowledge from two individual models. We employ Singular Value Decomposition (SVD) to analyze the knowledge representation in each model. Figure 3a il- lustrates the singular value distribution of individual LoRA models (each with rank 8) and the merged SeedLoRA model. Notably, SeedL.RR, LoR exhibits a broader range of non-zero singular values compared to the individual LoRA, suggest- ing successful knowledge fusion from multiple sources.
Full FT LLaMA2-7B on Humaneval: 40.3
LoRA LoRA+
34.1
40.7
34.1
38.4
36.0
36.6
39.2
39.0
32.3
30.5
DoRA
34.1
37.3
32.3
33.5
34.8
Full FT LLaMA2-7B on Humaneval4: 37.1
LoRA LoRA+
30.5
36.6
29.9
34.1
30.5
34.1
36.6
34.1
28.7
27.4
DoRA
32.3
32.3
29.3
29.9
31.7
Inspired by SWA (Izmailov et al., 2018), which claims that averaging weights can lead to wider optima and better gener- alization, we investigate whether our model exhibits similar benefits. We analyze the training loss and the performance on downstream evaluation tasks, as shown in Figure 3b and Figure 3c. Interestingly, SeedLORA demonstrates a slightly higher training loss but achieves superior evalua- tion performance on downstream evaluation tasks which takes different distributions from the training data. This
Table LLaMA2-7B model with LoRA (Delta) on CodeFeedback (Humaneval and Humanevalt). MS represents Model Soup.
4.4. Instruction Following
Having examined the effectiveness of our proposed method SeedLoRA in specialized domains, we now extend our eval-
7

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine- Fine-Tuning
LOM LM
(Soed 421 LOTK (Seed 2621
Soutla TuOFT
70.0
(m224))
310
67.3
325
(a) MetaMathQA
(b) Code- Bede-Fee
(c) LoRA vs SeedLoRA
Figure 4. The comparison between vanilla LoRA training with different seeds, with higher rank, with more epochs and Full Fine-Tuning
(Full FT). (a) Comparison on MetaMathQA benchmark. (b) Comparison Codc- Code-Feedback benchmark. (c) The Performance Comparison
between LoRA and Seed LoRA under the similar training cost constraint.
Method
MMLU
GSM8K
BBH
TyDiQA
HumanEval
Average
LoRA (r=8)
49,2
22.5
43.3
51.8
14.9
36.4
SeA
49.9
22.0
46.3
52.7
15.5
37.3
LoRA
Model Soup
50.3
20.5
45.5
50.9
15.2
36.5
TIES
49.3
19.5
43.3
53.6
15.1
36.2
DARE
50.2
22.5
44.3
53.4
15.3
37.0
LoRA+ (r=8)
49.7
25.0
46.5
53.1
16.0
38.1
SeedLoRA
51.0
25.5
46.9
52.8
18.0
38.8
LoRA+
Model Soup
51.2
24.5
45.7
52.7
17.4
38.3
TIES
50.2
23.0
42.9
52.8
17.5
37.3
DARE
50.1
22.0
43.1
52.9
17.4
37.1
DoRA (r=8)
49.4
25.5
46.3
50.4
16.0
37.5
SeedLoRA
50.0
28.5
46.6
52.2
15.2
38.5
DoRA
Model Soup
50.4
23.0
47.7
51.0
15.2
37.5
TIES
49.8
22.5
45,6
53.1
14.9
37.2
DARE
49.7
23.5
45.3
53.3
14.7
37.3
Table LLaMA-2-7B model with LoRA Tulu-v2, For the results of LoRA and its variants, we report the best performance LoRA models, which is trained with different seeds.
pattern indicates improved generalization ability, suggest- ing ScedLoRA learns more robust, task agnostic features rather than overfitting the training data. These findings knowledge fusion and generalization provide insight into the mechanisms underlying LoR''s improved performance across various tasks.
the advantage SeedLoRA effectively combining the strengths of multiple lower-rank models, achieving better performance than simply increasing the rank of single model.
Training SeedLoRA with Similar Cost as Formal LoRA. Our method requires obtaining several LoRA models trained on the same tasks. While some suitable models can be found on platforms like Huggingface, it often necessary to train multiple models by ourselves, potentially incurring additional training time. To address this, investigate whether can achieve better performance with comparable training cost using SeedLoRA.
Comparing with Higher Rank LoRA. To further validate the effectiveness of our approach, we compare LoRA with higher rank LoRA models. The main reason is that compare with training
training models, LoRA with rank 8+3, Since experiments focus merg- ing three LoRA models with rank 8, we perform an ablation study comparing our merged model with a single LoRA model with rank 24. As shown in Figure 4, ScedLoRA outperforms the higher-rank LoRA model. This highlights
Specifically, we propose an alternative to the standard 3- epoch LoRA fine-tuning LLaMA2-7B: training 3 indi- vidual models with epoch each, then merging these
8

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
partially trained models. We conduct this experiment on MetaMathQA benchmark, with the results shown in Figure 4(c). Remarkably, this approach outperforms the standard 3-epoch training while maintaining the same overall training time. This finding suggests a potentially new, more efficient training paradigm for PEFT.
fuse knowledge from individual models that specialize in different sub-domains, leading improved generalization. By bridging the performance gap between PEFT methods and full fine-tune, our work highlights the potential to en- able broader adoption of state-of-the-art L.LMs in resource- constrained environments.
SeedLolt 25
LoRA (Seed 111
Impact Statement
LoRA (Seed 421
Model Soup
(Seed
76-
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
-3
GSMBK
74-
-22
-20
Acknowledgements
Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 1 grant, ByteDance grant, ARCTIC grant, SMI grant and Alibaba grant. This work is also supported by NSF 2048280, 2325121, 2244760, 2331966 and ONR N00014- 23-1- 2300:00001
18
Figure 5. Scaling Results of LLaMA2-13B on MctaMathQA.
Training with More Epochs. The Training paradigm of SeedLoRA can be regarded as training LoRA with more epochs. To rigorously validate the superior performance of ReedLoRRA we train vanilla LoRA with more epochs and compare our merged model with it. We conduct the and the
References
Biderman, D., Portes, J., Ortiz, J. G., Paul, M., Greengard, P., Jennings, C., King, D., Havens, S., Chiley, V., Frankle, J., Blakeney, C.,, and Cunningham, J. P. LoRA learns less and forgets less. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8856. URL https: openreview net forumm wweeiidddallldueeee Fea- tured Certification.
experiment MetaMathQA and comparison result is shown in the Figure 4 and Figure 3c. The results illustrate that SeedLoRA can outperform LoRA training with more epochs on both math reasoning and code generation tasks, although training more epochs can slightly improve its performance.
Scaling Results. To verify the scalability of SeedLoRA, we conduct experiments on pre-trained models with larger num- ber of parameters. Specially, we evaluate the performance of SeedLoRA on the LLaMA2-13B 13B model, The results are pre- sented in Figure 5. SeedL .oRA A achieves approximately 2.3% performance gain compared to the best individual LoRA model. This demonstrates that SeedLoRA can effectively improve the performance even on larger models, highlight- ing its scalability and potential for enhancing models across different sizes.
Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. D. O., Kaplan, J., Edwards, H.,, Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
Clark, J. H., Choi, E., Collins, M., Garrette, D., K Kwiatkowski, T., Nikolaev, V., and Palomaki,, J. Tydi qa: A benchmark for information seekonn question answering in ty pologically di verse languages. Transuctions of the Association for Computational Linguistics, 8:454-470. 2020.
5. Conclusions
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint 2r 2021.
In this paper, we introduce SeedLoRA, a novel single-task model merging approach designed to enhance LoRA fine- tuning. Our method effectively narrows the performance gap between LoRA and full fine-tuning in complex tasks like mathematical reasoning and code generation by com- bining complementary strengths of models trained with different seeds. Notably, SeedLoRA consistently outper- forms existing merging techniques in single-task scenarios. The effectiveness of SeedL.ok oRA stems from its ability to
Deep, P. T., Bhardwaj. R., and Poria, S. Della-merginge Re- ducing interference in model merging through magnitude- based sampling. arXiv preprint arXiv:240.. 11617, 2024.
Hayou, S., Ghosh, N., and Yu, Lorat: Efficient

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
low rank adaptation of large models. arXiv preprint arXiv:2402. 120002..2334. 12354, 2024.
Lialin, V., Muckatira, S., Shivagunde, N., and Rumshisky, A. ReloRA: High-rank training through low-rank updates. In The Twelfth International Conference Learning Representations, 2024. URL /ttttpps///penzevierr net / forum? d=DIJIZSS6633.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M.. Song, D., and Steinhardt, J. Mcasuring mas- sive multitask language understanding. arXiv preprint 0tt1:200000330000 2020.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falschoods. arXiv preprint arXiv: 000::21000000 209.02889,,758, 2021.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv preprint wrXiv:2103.03877 2021.
Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang. S., Wang, L.., and Chen, W. LoRA: Low-rank adap- tation of large language models. In International Confer- ence on Learning Reprexentationss 2022. URL https: //openreview net forum www.rmmiidddeeeeeeeeeeee
Lu, Z., Fan, C.,, Wei, W., Qu, X., Chen, D., and Cheng, Twin-merging Dynamic integration of modular expertise in model merging. arXiv preprint vrXiv:2406.1549, 2024.
Huang, C.,, Liu, Q. Lin, B. Y., Pang, T., Du, C., and Lin, M. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint 1rXiv23307113269 2023.
Meng, F., Wang, Z.,, and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.0000
2024a.
Huang, C., Ye, P., Chen, T., He, T., Yue, X., and Ouyang, W. Emr-merging: Tuning-free free high-performance model merging. arXiv preprint arXiv:2405. 17461,
Meng, X., Dai, D., Luo, W., Yang. Z., Wu, S., Wang, X.,
Wang, P., Dong, Q., Chen, L.,, and Sui, Z. Periodiclora: Breaking the low-rank bottleneck in lora optimization. arXiv preprint arXiv:2402. 16141, 2024b.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Gururangan, S., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing mod- els with task arithmetic. arXiv preprint arXiv:2212.04005 2022.
Salamanca, A. R., Üstün, A., Detlefsen, N. S., and Dettmers, T. Seeded lora: Collaborative fine-tuning through seed ini- tialization of adapters. In Workshop on Efficient Systems for Foundation Models @ ICM1.2024.
Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M., Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I., et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:2311.10702. 2023.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q. V., Chi, H., Zhou, D.,, et al. Challenging big- tasks and whether chain-of-thouge can solve them. arXiv preprint arXiv:2210.099 2022.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: 1803.05407, 2018
Tam, D., Li, M., Yadav, P., Gabriclsson, R. B., Zhu, J., Greenewald, K., Yurochkin, M., Bansal, M., Raffel, C., and Choshen, L. Lim merging: Building llms efficiently through merging. In NeurIPS 2024 Competition Track.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b. arXiv preprint arXiv:2310.06822 2023.
Tian, C., Shi, Z., Guo, Z., Li, L.,, and Xu, C. Hydralora: An asymmetric lora architecture for efficient fine-tuning. arXiv preprint arXiv:2404. 19245, 2024.
Jiang T. Huang, S. Luo, S., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., Zhang, Q., Wang, D., et al. Mora: High-rank updating for parameter-efficient fine-tuning. arXiv preprint 2rX.v:2405.12//1 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Kwon, W., Li, Z., Zhuang, S., Sheng. Y., Zheng, L., Yu, C. H.. Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Verma, N. and Elbayad, M. Merging text transformer models from different initializations. arXiv preprint arXiv:2403.009m 2024.
10

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
Wang, H., Xiao, Z., Li, Y., Wang. S., Chen, G., and Chen, Y. Milora: Harnessing minor singular components for parameter-efficient I1m finetuning, arXiv preprint arXiv:2406.09044, 2024a.
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Opencodeintepprtter: Integrating code generation with execution and refinement. arXiv preprint arXiv:2402. 14658, 2024.
Wang, S., Yu, L., and Li, J. Lora-ga: Low-rank adap- tation with gradient approximation. arXiv preprint arXiv:2407.05000, 2024b.
Wang, Y., Ivison, H., Dasigi, P., Hessel, Khot, T., Chandu, K., Wadden, D., MacMillan. K., Smith, N. A., Beltagy, I,, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neu- ral Information Processing Systems, 36:74764-74786, 2023a.
Wang, Y., Lin, Y., Zeng, X., and Zhang, G. Multilora: Democratizing lora for better multi-task learning. arXiv preprint ar arXiv:2311.11501 2023b.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-opes,, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965- 23998. PMLR, 2022.
Yadav, P., Tam, D., Choshen, L., Raffel, C., and Bansal, M. TIES :ee-mngg:ng::: Resolving interference when merg- ing models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview. net wwr.m//rmmmmmmmmm http://oommmmixxdxaaaaaaa- widmmiixxxxxxxixi wtw.://wwww.t//ouummammaaaa..aaa
yang Liu, S., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.-C. F,, Cheng, K.-T., and Chen, M.-H. DoRA:
Weight-decomposed low-rank adaptation. In Forty- first International Conference on Machine Learning, 2024. URL https::////....... http:///pperveiieeeee n forum?
3d=dGCCIRG1n2.
Yu, I.., Jiang, W., Shi, H., Yu, J., Liu, 7., Zhang, Y., Kwok, T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Yu, L., Yu, B., Yu, H., Huang, F., and Li, Language models are super mario: Absorbing abilities from homol- ogous models free lunch. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreviens net forum ? fd=fqNaaiU88F
Zadouri, T., Üstün, A., Ahmadian, A., Ermis, B., Locatelli, A., and Hooker, S. Pushing mixture of experts to the limit: Extremely parameter efficient moe for instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. URL https: h //tp://epeeeee//ee// net/ forum? d=RvDei Lv7gc.

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine- Tine-nunin
A. Appendix
A.1. Implementation Details
Training is conducted on Nvidia A100 and H100 GPUs using B precision. We set weight decay to and employ a
cosine learning rate scheduler with a 0.03 ratio linear warmup. For evaluation, we utilize vLLM (Kwon al., 2023) to conduct tests, ensuring efficient and scalable inference.
Model
Dataset
Method
Qt
LR
Scheduler
Warmup
Epochs
Batch Size
LLaMA2-7B
MetaMathQA
LoRA
8
16
3e-5
cosine
300
3
128
median
LLaMA2-7B
Tulu-v2
LoRA
8
16
3e-5
cosine
500
2
128
median
LLaMA2-7B
Code-Feedhack
LoRA
8
16
3c-5
cosine
300
3
32
median
Mistral-7B
MetaMathQA
LoRA
8
16
3e-5
cosine
300
3
128
median
Mistral-7B
Tulu-v2
LoRA
8
16
3e-5
cosine
500
2
128
median
Mistral-7B
Code-Foedhack
LoRA
8
16
5e-5
cosine
300
3
32
median
Table 5. LLaMA-2-7B model with LoRA on Tulu-v2, For the results of LoRA and its variants, report the best performance of LoRA models, which is trained with different seeds.
A.2. The Performance Analysis about LoRA and Full Fine-Tuning on MBPP
Table 6, we conducted the experiments fine-tuning LLaMA2-7B with LoRA from rank=8 to rank-64 on code generation (CodeFeedhack) tasks (Yu et al., 2023). The results illustrate that although increasing the rank value of LoRA from 64 improves performance on HumanEval from 34.1 to 35.4, significant gap remains when compared with Full Fine-Tuning.
Task
rank=8
rank=16
rank=24
rank=32
rank=64
Full FT
HumanEval Eval
34.1
34.1
34.8
34.8
35.4
40.3
HumanEval+
28.0
32.3
31.7
31.7
31.7
37.1
MBPP (+)
45.8 (38.6)
43.7 (36.0)
44.2 (36.2)
46.6 (39.7)
42.1 (36.2)
53.1
Average
40.0 (33.3)
38.9 (34.2)
39.5 (34.0)
40.7 (35.7)
38.8 (34.0)
46.7
Table LLaMA-2-7B model with LoRA (Delta) on Code-Feedback (seed=11).
A.3. The experimental Results for Fine-Tuning Mistral-7B on Code-Feedback.
Table 7 presents the experimental results of Mistral-7B model with LoRA on CodeFeedback. The results demonstrate that merged model consistently outperforms individual models in the HumanEval and HumanEval+ tasks.
Method
seed
seed 42
seed 202
SeedLoRA
Model Soup
TIES
DARE
LoRA (r=8)
53.0
51.8
48.2
58.0
53.7
55.5
56.7
HvmanEval
LoRA+ (r=8)
54.3
48.8
47.6
56,9
54.3
51.8
54.3
DoRA (r=8)
54.3
55.5
45.1
57.6
54.3
56.7
55.5
LoRA (r=8)
49.4
47.6
40.9
51.2
50.6
49.4
50.0
HumanEval+
LoRA+ (r=8)
48.2
43.9
40.2
49.4
48.2
47.6
48.8
DoRA (r=8)
47.6
49.4
42.1
49.6
48.8
51.8
50.2
Table Mistral-7B model LoRA (Delta) on CodeFeedback (HumanEval and HumanEvalt)..
12

---

SeedLoRA: A Fusion Approach to Efficient LLM Fine-Tuning
A.4. The experimental Results for Fine-Tuning Mistral-7B Tulu-v2
Table demonstrates that our proposed method on Tulu-v2 can continue improve upon the performance of the best individual model with fine-tuning Mistral-7B.
MMLU-0
GSM8K
BBH
TyDiQA
HumanFval
Average
46.0
55.0
59.9
33.8
LoRA (r=8)
58.8
44.0
56.6
59.5
35.5
50.9
58.2
50.5
58.7
59.0
31.4
51.6
SeedLoRA
61.0
52.0
59.6
62.0
33.2
53.6
TIES
42.5
53.8
60.4
33.7
DARE
42.0
60.5
35.4
60.8
45.0
59.4
58.2
34.1
51.5
LoRA+ (r=8)
61.2
45.5
59.7
59.7
32.2
51.7
60.5
47.0
58.6
59.1
32.0
51.4
SeedLoRA
61.6
47.0
60.8
60.2
35.2
53.0
TIES
60.6
46.0
57.9
58.8
34.1
51.5
DARE
60.4
41.0
56.3
59.2
34.2
50.2
61.1
46.0
58.8
58.9
34.3
51.8
DoRA (r=8)
60.3
52.0
58.8
60.1
33.1
52,9
60.3
52.0
58.5
59.9
32.9
52.7
ScedLoRA
62.0
61.0
61.2
34.2
54.0
TIES
58.2
58.2
35.3
DARE
60.5
44.0
57.4
59.3
35.6
51.4
LoRA (r=24)
60.4
46.5
57.4
59.6
31.8
51.1
LoRA (epoch=6)
56.5
47.0
54.0
55.5
32.5
49.1
Table 8, Mistral-7B model with LoRA (Delta) on Tulu-v2.
A.5. The Number of Models
To investigate scaling behavior when merging additional models, we conducted experiments merging up different seed models in Table 9. We should also mention that these results might vary slightly from previous findings due to differences in experimental environments. These results show significant initial gains when merging 2 models (+3.7% on GSM8K), with continued improvements up to models (+5.7% on GSM8K). Beyond 4 models, we observe diminishing returns without performance degradation.
Task
N=1
N=2
N=3
N=4
N=5
N=6
GSMSK
64.0
67.7
68.4
69.7
69.6
69.8
MATH
15.3
16.6
17.3
16.6
17.1
Average
39.7
42.2
43.0
43.4
43.1
43.5
Table 9. LLaMA2-7B on MetaMathQA with More Seed Models.
13


Reviews:
[{"review_id": "K8qAADxRaf", "reviewer": "Reviewer_CWvH", "summary": "The paper introduces SeedLoRA, an approach to improving LoRA fine-tuning for LLMs. SeedLoRA is based on the observation that multiple LoRA models trained on the same task with different random seeds can have complementary performance. It uses a two-stage approach to merge different LoRA adapters, first identifying conflicting/robust parameters, and then performing subspace fusion via SVD to merge the remaining parameters. Experiments on Llama-2-7B/13B and Mistral-7B demonstrate improvements in math reasoning and code generation, achieving near full fine-tuning performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive and inspiring feedback, we carefully address your concerns below.\n\n >W1: Effectiveness across different ranks.} \n\nTo directly address this concern about the scalability of SeedLoRA across different rank settings, we have conducted comprehensive additional experiments with higher-rank LoRA adapters (r=16, 32, 64 and 96) on the MetaMathQA benchmark. \n\n- When merging 3 models (LLaMA2-7b): \n\n|  | GSM8K |  |   |  | MATH |  |  |  | \n|---------|------|-------|---------|-------|---------|------|---------|------|\n|Rank |seed=11 |seed=42 | seed=202 | SeedLoRA | seed=11 | seed=42 | seed=202 | SeedLoRA |\n|LoRA (R=8)| 64.0 | 63.8 | 64.1 | **68.4** | 15.3 | 15.3 | 14.9 | **17.3** \n|LoRA (R=32)| 65.4 | 66.0 | 66.9 | **70.3** | 16.0 | 16.7 | 16.4 | **17.8**  \n|LoRA (R=64)| 65.7 | 65.5 | 66.6 | **69.2** | 16.2 | 16.5 | 16.5 | **17.4**     \n|LoRA (R=96)| 66.7 | 66.0 | 66.4 | **70.0** | 16.7 | 17.2 | 16.9 | **17.1**  \n\nThese results clearly demonstrate that SeedLoRA consistently improves performance across all tested rank settings. This confirms that SeedLoRA's effectiveness is not limited to low-rank settings but extends to the higher-rank adapters. \n\n>W2: Merging adapters to longer training (9 epochs) lacks granularity. \n\nTo address this concern, we have conducted additional experiments comparing SeedLoRA with equivalent or higher-rank single LoRA adapters: \n\n|Model |Task |LoRA (R=24) | SeedLoRA (3*(R=8)) | LoRA (R=96) | SeedLoRA (3*(R=32)) | \n|---------|------|-------|---------|-------|---------|\n|LLaMA2-7b| GSM8K | 64.9 | **68.4**  | 66.7  | **70.0** \n|LLaMA2-7b| MATH | 16.3  |  **17.3** |  16.7 |  **17.1** \n\nThese results demonstrate that SeedLoRA offers superior performance compared to single higher-rank alternatives. SeedLoRA (merging three r=8 adapters) achieves +3.5\\% better performance on GSM8K compared to a single r=24 LoRA adapter.  Moreover, the improvement scales to very high ranks, with SeedLoRA (merging three r=32 adapters) outperforming a single r=96 LoRA adapter by +3.3\\% on GSM8K. \n\n>W3: The proportion of parameters classified as robust/conflicting during stage 1, nor does it justify the threshold . \n\nTo address these concerns, we've analyzed the proportion of parameters classified in each stage across different layers.  \n\n(1) Layer-wise parameter distribution analysis\n\nWe measured the proportion of parameters classified across different layers and tasks: \n\n- LLaMA2-7b on MetaMathQA: \n\n|Layer |robust (stage 1) | conflicting (stage 1) | stage 2 | \n|---------|------|-------|---------|\n Attention Q  | 9.0 \\%  | 6.4 \\%  |  84.6 \\%                  \n Attention K   | 9.3 \\%  | 6.7 \\%  |  84.0 \\%                 \n Attention V  | 5.2 \\%  | 8.5 \\%  |  85.6 \\%                \n MLP up\\_proj  | 6.9 \\%  | 7.5 \\%  |  86.0 \\%             \n MLP down\\_proj  | 6.2 \\%  | 7.8 \\%  |  86.3 \\%       \n\nOverall, approximately 13-16\\% of parameters are classified as robust or conflicting in Stage 1, with variations across layer types. \n\n(2) Threshold selection and ablation study\n\nThe threshold was determined using the top 50\\% magnitude of weights across adapters in each layer. This approach provides an adaptive threshold that scales appropriately with the distribution of weight magnitudes in different layers and models. \n\nTo verify the importance of Stage 1, we conducted an ablation experiment: \n\n- LLaMA2-7b and Mistral-7b on MetaMathQA \n\n|Model | stage 1 | stage 2 |  GSM8K  |  MATH  | \n|---------|------|-------|---------|-------|\n LLaMA2-7b  | $\\checkmark$  |    |   67.3  | 16.7                  \n LLaMA2-7b  |  | $\\checkmark$  |  67.9  | 16.8               \n LLaMA2-7b  | $\\checkmark$ | $\\checkmark$  |  69.1 | 17.1             \n Mistral-7b  | $\\checkmark$  |   |  79.4 | 28.3            \n Mistral-7b  |  | $\\checkmark$  |  79.7  |  28.5 \n Mistral-7b  | $\\checkmark$ | $\\checkmark$  |  80.7   |  28.8\n\nThese results show both stages are valuable, with their combination yielding optimal performance. \n\n- Merging all parameters through subspace fusion: we conducted additional experiments about SeedLoRA (rank=8) with only subspace fusion for all parameters, which resulted in 67.5\\% on GSM8K and 16.4\\% on MATH (SeedLoRA: 69.1\\% on GSM8K and 17.1\\% on MATH). \n\n>W4: If performance continues to improve with more seed models. \n\nTo investigate scaling behavior when merging additional models, we conducted experiments merging up to 6 different seed models: \n\n|Task | Rank | SeedLoRA (2)  | SeedLoRA (3)  | SeedLoRA (4)  | SeedLoRA (5)  | SeedLoRA (6)  | \n|---------|------|-------|---------|-------|---------|-------|\n GSM8K  | LoRA (R=8)  | 67.7 | 68.4 | 69.7 | 69.6 | 69.8                 \n MATH  | LoRA (r=8) | 16.6  | 17.3 | 17.1 | 16.6 | 17.1            \n\nThese results show significant initial gains when merging 2 models (+3.7\\% on GSM8K), with continued improvements up to 4 models (+5.7\\% on GSM8K). Beyond 4 models, we observe diminishing returns without performance degradation.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The key contributions of SeedLoRA build on several lines of research in PEFT and model merging. SeedLoRA directly extends the LoRA framework, which freezes pre-trained weights and injects trainable low-rank matrices. Some more recent works, like ReLoRA/DoRA/MoRA, improve LoRA's performance and can be plugged directly into SeedLoRA as this work focuses on post-training fusion rather than specific architectural changes. SeedLoRA also takes inspiration from Model Soup and methods like TIES/DARE/SWA and developed a two-stage adapter merging approach while showing superior performance over these methods.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths\n1. SeedLoRA achieves comparable performance to full fine-tuning and shows significant improvement in performance over vanilla LoRA over a wide range of tasks.\n2. The idea of handling robust/contradicting parameters first and then merging is interesting and works well in the examples.\n\n\nWeaknesses\n1. The experiments mainly focus on merging models with rank 8, with only a single comparison against a rank 24 model. This raises questions about the method's effectiveness across different ranks. Given that higher-rank (rank 32/64) LoRA adapters are commonly used (e.g., various models on Huggingface), it remains unclear whether SeedLoRA's improvements are consistent across higher-rank LoRAs or are limited to low-rank settings. \n2. The ablation experiment comparing merging adapters to longer training (9 epochs) lacks granularity. Training high-rank models for fewer epochs might offer better compute-performance tradeoffs than training a low-rank model for 9 epochs, especially if overfitting occurs with extra training. \n3. The paper does not quantify the proportion of parameters classified as robust/conflicting during stage 1, nor does it justify the threshold $\\sigma $. What percentage of parameters fall into robust/conflicting categories, and how does this vary across tasks/layers? How is $\\sigma$ chosen here, is it task/layer dependent, and could adaptive thresholds improve results? If the proportion of robust/conflicting parameters is very small, it would be insightful to include an ablation on whether stage 1 is needed or not. For example, does merging all parameters only through subspace fusion degrade performance?\n4. The experiments are limited to merging 3 seed models. Based on the assumption that different adapters have different comparative strengths, it would be interesting to see if performance continues to improve with more seed models. A follow-up question is when does the performance stop to scale or even drop with more seeds added, or is 3 seeds the maximum number of seeds that works with this method?", "other_comments_or_suggestions": "NA", "questions_for_authors": "See strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QTXCuKyYHt", "reviewer": "Reviewer_1ENu", "summary": "The authors propose a method for combining multiple LoRA adapters trained for the same task with different seeds and show it improves performance. The authors’ method consists of (1) identifying and preserving large consistent dimensions or “robust” directions, (2) using a principal-component-like decomposition (SVD / PCA) for ``moderate'' dimensions, and (3) merging these decomposed updates via a thresholded weighted average.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive and inspiring feedback, we carefully address your concerns below. \n\n>Clarification on seed reporting and cosine similarity calculations\n\nThank you for your feedback on the experimental design and reproducibility. We agree that additional clarity on seed variation would strengthen our presentation.\n\n(1) Reporting of LoRA results in Tables 2 and 3\n\nIn Tables 2 and 3, we report individual results for each seed (seed=11, seed=42, seed=202) rather than averages. Each column represents the performance of a single LoRA model trained with that specific seed. We chose to present the individual seed results rather than averages to highlight the performance variation across seeds, which is a key motivation for our SeedLoRA approach.\n\n(2) Clarification on cosine similarity in Figure 2\n\nRegarding Figure 2, the cosine similarity values shown are computed as averages across all layers of the models. Specifically, for each adapter pair, we first computed the cosine similarity between corresponding parameters in each layer, then averaged these similarities across all layers to obtain the single value shown in the figure. \n\n>Possibly more thorough discussion of SWA-like arguments (which they partially reference) could reinforce the “wider optimum” viewpoint in merging. \n\nThank you for your insightful comment regarding the potential connection between SeedLoRA and SWA-style techniques.\n\nWhile both SWA and SeedLoRA involve combining model weights, their underlying assumptions and practical implications differ substantially. SWA averages model weights sampled along the same optimization trajectory, typically assuming that these checkpoints lie within the same basin of attraction in the loss landscape. This allows SWA to converge to flatter optima, often associated with better generalization and robustness.\n\nIn contrast, SeedLoRA merges multiple LoRA adapters trained from different random seeds, which often leads to models residing in different regions of the parameter space—potentially even in distinct basins. As a result, direct weight averaging, as in SWA, can be ineffective or even detrimental in this context.", "claims_and_evidence": "* Single- vs. Multi-Task Model Differences: They assert that they have examined differences between single-task and multi-task model merging (Section 3.3). However, the presented evidence mostly focuses on single-task settings (and the multi-task data come only from a brief cosine similarity comparison). This makes the claim of “understanding differences” (their stated contribution (i)) less thoroughly supported. The study would benefit from more extensive experiments or references to truly confirm those differences.\n\n* Performance Gains: Their experiments do convincingly demonstrate that their adapter merging method yields improvements. This is generally well-supported by results on math and coding tasks.", "methods_and_evaluation_criteria": "The experimental settings and metrics used (GSM8K, MATH, HumanEval, MMLU, etc.) are standard, and the results are generally comprehensive.", "theoretical_claims": "There is no deep theoretical result presented in the paper. Most claims about the “differences” between single-task and multi-task models, or about why merging addresses complementary subdomains, remain at an intuitive or empirical level. This is okay since this is as an applied paper, though ensuring clarity of definitions (e.g., the rank constraints, the threshold) would strengthen the argument. The paper could also benefit from a more clear notation (e.g. tau denotes both the threshold and the weight updates)", "experimental_designs_or_analyses": "The experimental design is straightforward, benchmarks are well-known, and baselines reasonable. The results seem reproducible. Since across seed variation is one of the fundamental claims of the papers, it would be helpful if the authors added e.g. in table 2 and 3 how when reporting lora if that is the average across seeds and how meny where used (and perhaps stds). Also, in Figure 2, it is unclear whether the “cosine similarity” reported is computed for a specific layer or is an average across all layers/adapters—further clarity in the text would help.", "relation_to_broader_scientific_literature": "The idea of ensembling models across runs to leverage the sources of randomness during training is not new, but there are few works that combine it with PEFT of LLMs. Most existing works (Multi LoRA, IterIS) focus on the multi-task, while this paper focuses on single tasks.", "essential_references_not_discussed": "This is of course related to MoE and model merging, which I am not that familiar with. Possibly more thorough discussion of SWA-like arguments (which they partially reference) could reinforce the “wider optimum” viewpoint in merging. But otherwise, no major omissions stand out.", "other_strengths_and_weaknesses": "Although their proposed method is more general and is not restricted LoRA adapters, they demonstrate its  utility in this setting which is relevant in practice.", "other_comments_or_suggestions": "Cite TIES in 3.3. Make figure and table captions more comprehensive.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nFjGEBsszb", "reviewer": "Reviewer_7HoA", "summary": "The paper presents a method that leverages multiple LoRA models trained with different random seeds on the same task and merges their trained weights using a two-stage merging strategy. In the first stage, the algorithm detects robust and conflicting dimensions from the multiple trained weights using thresholding and counting opposite-sign values, respectively. In the second stage, the algorithm calculates an average space using SVD and projects the weight matrices onto this space. The projected coordinates are then fused and used to reconstruct the fused weight matrix. The final adapter is formed by combining the robust dimensions, conflicting dimensions, and the reconstructed fused matrix. The experimental results demonstrate significant improvements in mathematical reasoning and code generation tasks over individual LoRA models, often matching or exceeding full fine-tuning performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments, we carefully address your concerns below. \n\n>W1: However, a downside of the paper is that the training cost is not mentioned or discussed. \n\nWhile SeedLoRA maintains the same inference memory footprint as vanilla LoRA, we acknowledge that the training computation increases proportionally with the number of seed models. However, since each training process is independent, we can leverage parallelization techniques to accelerate the overall training. \n\nAdditionally, we conducted experiments to evaluate SeedLoRA under comparable computational budgets as vanilla LoRA (Line 429). For instance, we compared vanilla LoRA trained for 3 epochs against SeedLoRA with 3 merged models (each trained for only 1 epoch). Our results demonstrate that SeedLoRA achieves superior performance. \n\n>Q1: How do the merging methods affect the final performance? \n\nWe try to provide the results of ablation study on LLaMA2-7b fine-tuning on MetaMathQA. We can obtain that these results show both stages are valuable, with their combination yielding optimal performance. Stage 2 alone performs slightly better than Stage 1 alone, and the combined approach consistently improves results across different model architectures. \n\n|Model | stage 1 | stage 2 |  GSM8K  |  MATH  | \n|---------|------|-------|---------|-------|\n LLaMA2-7b  | $\\checkmark$  |    |   67.3  | 16.7                  \n LLaMA2-7b  |  | $\\checkmark$  |  67.9  | 16.8               \n LLaMA2-7b  | $\\checkmark$ | $\\checkmark$  |  69.1 | 17.1             \n Mistral-7b  | $\\checkmark$  |   |  79.4 | 28.3            \n Mistral-7b  |  | $\\checkmark$  |  79.7  |  28.5 \n Mistral-7b  | $\\checkmark$ | $\\checkmark$  |  80.7   |  28.8 \n\n>Q2: Would the authors explicitly state the values of $\\sigma$ used for each dataset and model? \n\nFor all models and datasets, we determined the threshold $\\sigma$ using the top 50\\% magnitude of weights across adapters in each layer. This approach provides an adaptive threshold that scales appropriately with the distribution of weight magnitudes in different layers and models.\n\n>Q3: Could the authors define what “multiple distinct groups” means and explain how to calculate and determine the conflicting dimensions? \n\nWe determine conflicting dimensions through the following specific process:\n\nFor each parameter dimension $j$, we examine the values $\\tau_i(j)$ across all n adapters.\nWe classify adapters into sign groups based on the sign of their values in dimension $j$.\nA dimension is considered \"conflicting\" when it meets both of the following criteria:\n\n(1) It contains at least two sign groups (positive and negative). \n\n(2) Each sign group contains at least  $\\lfloor n/3 \\rfloor$ adapters with magnitude $|\\tau_{i}(j)| \\geq \\sigma$. \n\nThis means that for dimension j to be conflicting, there must be a substantial number of adapters (at least one-third of the total) strongly pulling in opposite directions. For example, with 3 adapters, a dimension is conflicting if at least 1 adapter has a large positive value and at least 1 has a large negative value. \n\n>Q4: The algorithm for fusing the set of coordinates $\\widetilde{Z}$ is also unclear. \n\nThe calculation of the fused coordinate matrix $\\widetilde{Z}^{(l)}$ proceeds as follows:\n\n  - After projecting each adapter's moderate parameters $\\tau_i^{(l)}$ onto the common subspace to obtain coordinate matrices $Z_i^{(l)}$, we compute the element-wise average of these coordinate matrices: $Z_{\\text{avg}}^{(l)} = \\frac{1}{n}\\sum_{i=1}^{n} Z_i^{(l)}$\n   \n- We then examine each element $(m,k)$ in these coordinate matrices across all adapters:\n   \\begin{enumerate}\n\t- If the element values $Z_i^{(l)}(m,k)$ have the same sign across all adapters, we retain the average value: $\\widetilde{Z}^{(l)}(m,k) = Z_{\\text{avg}}^{(l)}(m,k)$\n\t- If there are sign conflicts among adapters, we follow a modified TIES approach:\n$$\\widetilde{Z}^{(l)}(m,k) = \\frac{1}{|I^+|}\\sum_{i \\in I^+} Z_i^{(l)}(m,k), \\quad \\text{if } |I^+| \\geq |I^-|$$\n$$\\widetilde{Z}^{(l)}(m,k) = \\frac{1}{|I^-|}\\sum_{i \\in I^-} Z_i^{(l)}(m,k), \\quad \\text{otherwise}$$\n       where $I^+$ and $I^-$ are the sets of adapter indices with positive and negative values for element $(m,k)$, respectively. \n\nThis approach preserves agreement between adapters while resolving conflicts by favoring the dominant sign group. \n\n>Q5: Would you compare the training time? What if we fix a seed and train it for n $\\times$ longer so that the total training time matches the entire SeedLoRA process? \n\nWe have conducted the experiments comparing SeedLoRA with longer training of a single model. In the \"Training with More Epochs\" section (line 408) and Figure 4(a)-(b), we compare merging 3 LoRA models (each trained for 3 epochs) against a single LoRA model trained for 9 epochs, ensuring the training computation cost is equivalent. Our results show that SeedLoRA outperforms the longer training approach on both mathematical reasoning and code generation tasks.", "claims_and_evidence": "The key claims made in the paper are well-supported by empirical evidence:\n- The claim that SeedLoRA improves performance over individual LoRA models is substantiated through rigorous benchmarking on LLaMA2-7B and Mistral-7B models, showing improvements of up to 4.9% on GSM8K and 6.6% on HumanEval.\n- The paper convincingly argues that models trained with different seeds exhibit complementary strengths, and merging them can lead to a more robust final model.", "methods_and_evaluation_criteria": "The methods and evaluation criteria align well with the problem at hand. The authors utilize well-established benchmarks (GSM8K, MATH, HumanEval, etc.) and compare SeedLoRA against strong baselines, including vanilla LoRA, full fine-tuning, and alternative model merging methods. The evaluation setup is comprehensive, covering multiple datasets and architectures, and the results are consistently analyzed across different configurations.", "theoretical_claims": "The paper does not provide theoretical analysis.", "experimental_designs_or_analyses": "The experimental design is generally strong, incorporating thorough performance comparisons on different sizes of model. The authors evaluate SeedLoRA on multiple datasets, different model sizes (LLaMA2-7B, Mistral-7B, and LLaMA2-13B), and various LoRA configurations.", "relation_to_broader_scientific_literature": "The paper resides within the existing literature on parameter-efficient fine-tuning (PEFT), model merging, and low-rank adaptation techniques. The discussion effectively differentiates SeedLoRA from related work, such as Model Soup, MoE-based LoRA approaches, and traditional multi-task model merging methods. The paper acknowledges key related works, such as ReLoRA, DoRA, and MultiLoRA, and articulates how SeedLoRA improves upon them.", "essential_references_not_discussed": "The paper provides a well-rounded discussion of prior work. However, it would be beneficial to mention and discuss more about recent advancements in merging techniques for PEFT.", "other_strengths_and_weaknesses": "I think the idea and observations presented in the paper are novel, as they address the performance gap by merging LoRA modules trained with different random seeds. The authors provide strong empirical validation with diverse datasets and models, as well as practical relevance for improving PEFT methods. However, a downside of the paper is that the training cost is not mentioned or discussed. Although the authors present strong evidence supporting their proposed method, the theoretical justification could be more rigorous beyond cosine similarity analysis. Additionally, an ablation study on the fusion methods is expected.", "other_comments_or_suggestions": "Minor suggestions:\n- TIES at line 217 should be cited first.", "questions_for_authors": "- How do the merging methods affect the final performance? For example, what is the performance of $\\tau_{\\text{fused}}$​ alone, $\\tau_{\\text{fused}} + \\tau_{\\text{conflict}}$​, and $\\tau_{\\text{fused}} + \\tau_{\\text{robust}}​$?\n- Would the authors explicitly state the values of σ\\sigmaσ used for each dataset and model?\n- The algorithm for selecting conflicting dimensions is vaguely described. Could the authors define what “multiple distinct groups” means and explain how to calculate and determine the conflicting dimensions?\n- The algorithm for fusing the set of coordinates $\\tilde{Z}$ is also unclear. Could the authors explicitly show how $\\tilde{Z}$ is calculated??\n- Would you compare the training time? What if we fix a seed and train it for n \\times longer so that the total training time matches the entire SeedLoRA process?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces SeedLoRA, a novel post-training method for improving LoRA performance by merging adapters trained with different random seeds. The core insight is that different initializations lead LoRA to capture complementary aspects of a task, and their combination yields a stronger model. All reviewers found the idea interesting and the empirical results, showing significant gains over vanilla LoRA and often matching full fine-tuning on challenging benchmarks, compelling.

Initial reviews were unanimously Weak Accept, raising concerns about scalability (rank, number of seeds), algorithmic clarity/justification, comparison fairness/cost, and theoretical grounding. The authors provided a thorough rebuttal with substantial new experiments and clarifications that addressed the majority of these points effectively. Reviewers CWvH and 7HoA explicitly acknowledged that their concerns were addressed, with CWvH indicating an intent to raise their score. While theoretical backing remains light, the strong empirical validation and successful rebuttal addressing key experimental and methodological questions support acceptance.
Decision: Accept (poster)