LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
Authors: Juzheng Zhang
Jiacheng You
Ashwinee Panda
Tom Goldstein
Venue: COLM 2025 (2025)
Keywords: Large Language Models
Parameter-Efficient Fine-Tuning
Model Merging
Sparsity
TL;DR: LoRI is a simple yet effective method for parameter-efficient fine-tuning that reduces cross-task interference by freezing projection matrices $A$ and sparsifying $B$.

Abstract: Low-Rank Adaptation (LoRA) has emerged as a popular parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), yet it still incurs notable overhead and suffers from parameter interference in multi-task scenarios. We propose LoRA with Reduced Interference (LoRI), a simple yet effective approach that freezes the projection matrices $A$ as random projections and sparsifies the matrices $B$ using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. 
Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95\% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference.
Code is available at: https://github.com/juzhengz/LoRI.

Published as a conference paper at COLM 2025
LoRI: Reducing Cross-Task Interference in Multi-Task Low- Rank Adaptation
Juzheng Zhang', Jiacheng You?, University of Maryland
Ashwinee Panda',
Tom Goldstein
2Tsinghua University
Abstract
Low-Rank Adaptation (LoRA) has emerged as a popular parameter- efficient fine-tuning (PEFT) method for Large Language Models (LLMs), yet it still incurs notable overhead and suffers from parameter interfer- ence in multi-task scenarios. We propose LoRA with Reduced Interference (LoRI), a simple yet effective approach that freezes the projection matrices A as random projections and sparsifies the matrices using task-specific masks. This design substantially reduces the number of trainable param- eters while maintaining strong task performance. Moreover, LoRi mini- mizes cross-task interference in adapter merging by leveraging the orthog- onality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full tine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference. Code is available at: https: /tap:://itttiee com/ juzhengz/LoRT.
1 Introduction
Large language models (LLMs) (Brown et al., 2020; Touvron et al., 2023; Chowdhery et al., 2023) have transformed deep learning, showcasing remarkable capabilities across vari- ous domains. However, their deployment remains computationally demanding, partic- ularly when fine-tuning is required to adapt to downstream tasks or align with human preferences. To mitigate the high resource costs, researchers have developed range of parameter-efficient fine-tuning (PEFT) techniques. Among these techniques, LoRA (Hu et 2021) has gained widespread adoption due to its compelling balance of performance and efficiency. Nevertheless, LoRA still introduces notable memory overhead, particu- larly in large-scale models. Consequently, recent research has focused on further optimiz- ing LoRA by reducing the number of trainable parameters without compromising perfor- mance (Kopiczko al., 2023; Ding al., 2023; Zhang et al., 2023b).
Recent studies (Yu et al., 2024; Panda et al., 2024) have shown that delta parameters - the differences between fine-tuned and pretrained model weights exhibit significant redun- dancy. Furthermore, previous works (Zhang et al., 2023b; Zhu et al., 2024) have observed that freezing matrices A in LoRA often achieves comparable performance to training them. Motivated by these findings, we propose LoRA with Reduced Interference (LoRI). LoRI keeps matrices fixed as random projections, while training matrices using task-specific sparse masks. To retain the most critical elements of B, LoRI performs calibration process to extract sparse masks by selecting the projections. As shown in Figure (()), LoRI maintains performance even with 90% sparsity in B while keeping A frozen. This demonstrates that adaptation does not require updating A, and that B has considerable redundancy. By applying more constrained updates than LoRA, LoRl significantly reduces the number of trainable parameters while better preserv- ing the pretrained model's knowledge during adaptation.
elements across all layers and
Correspondence to: juzheng@umd.c edu.

---

Published as a conference paper at COLM 2025
Math
Code
Single-Task LoRA
Multi-Task LoRA
Multi-Task LoRi
100
Performance (%)
60
80
80
55
60
34.3
61.5
60
48.4
50
40
40-
45
21
20
40
0.00
0.25
0.50
0.75
1.00
NLU
Math
Code
Safety (c) Continual learning.
NLU
(a) Varying sparsity ratios of B.
(b) Adapter merging.
Figure 1: (a) Varying sparsity ratios in matrices B while freezing A. Performance remains stable even at 90% sparsity in matrices B. (b) Merging three adapters via weighted averaging. LoRA suffers degradation due to parameter interference, while LoRI preserves task performance. (c) Continual learning from Safety to NLU. LoRA suffers from catastrophic forgetting, while LoRI retains safety alignment. Results for NLU are averaged over eight tasks. GSMHK accuracy (Math), HumanEval pass@10 (Code), and HEx-PHI refusal rate (Safety) reported individually, Base model: 3-ama-3- 8B, rank = 32.
Multi-task learning is essential for enabling versatile models with multi-task capabili- ties, which is traditionally performed via joint training on a combination of task-specific datasets (Caruana, 1997; Sener & Koltun, 2018). However, training large models on this data mixture is prohibitively expensive in terms of time and compute. Model merging is training-free frre alternative for building powerful models by combining existing ones (II- harco al., 2022; Yadav al., 2023; Yu al., 2024). This approach is well-suited for merg- ing LoRA adapters, enabling multi-task capabilities within a single model during infer- ence (Wang et al., 2024a; Prabhakar et al., 2024; Stoica et al., 2024). However, as shown in Figure 1(b), directly merging heterogeneous LoRAs often results in parameter interference, leading to degraded performance compared to single-task LoRAs. Additionally, many ex- isting merging methods require trial-and-error to identify the optimal method for spe- cific combination of tasks. LoRI addresses these challenges by using fixed, randomly ini- tialized projection A, which maps task-specific adapters into approximately orthogonal subspaces. This reduces interference when merging multiple adapters. In addition, LoRI enables adapter merging without manual selection of merging methods.
Beyond multi- tallii-tasking, safety-critical scenarios require that each newly introduced adapter enhances model capabilities while preserving the safety alignment of the pretrained base model (Qi et al., 2023). LoRI provides lightweight continual learning approach for adapting models while preserving safety, where training is performed sequentially across tasks (Lopez-Paz & Ranzato, 2017; Wu et al., 2022; Ouyang et al., 2022). The strategy in- volves first fine-tuning an adapter on safety data to establish alignment, followed by sepa- rate adaptation to each downstream task. However, learning often leads to catastrophic forgetting (Li & Hoiem, 2017; Dong et al., 2023; Luo et al., 2023), wherein the adaptation to new tasks substantially compromises previously acquired knowledge. LoRI mitigates forgetting by leveraging the sparsity of projection B through task-specific masks. This isolation of parameter updates across tasks facilitates continual learning with minimal interference, preserving both safety and task effectiveness.
illustrated in Figure 1(c), continual
To evaluate the effectiveness of LoRI, we conduct extensive experiments across a diverse suite of benchmarks spanning natural language understanding (NLU), mathematical rea- soning, code generation, and safety alignment tasks. Using Llama-3-8B and Mistral-7B as base models, our results show that LoRI achieves performance comparable to or better full fine-tuning (FFT), LoRA, and other PEFT methods, while using up 95% fewer trainable parameters than LoRA. Notably, LoRI with 90% sparsity in B surpasses LoRA by 17.3% on HumanEval with Llama-3. Beyond single-task adaptation, we evaluate LoRI in multi-task settings, including adapter merging and continual learning scenarios. Concate- nated merging of LoRI adapters consistently outperforms LoRA adapters overall, closely matching the performance of single-task LoRA baseline. In continual learning, LoRI signif- icantly outperforms LoRA in mitigating catastrophic forgetting of safety alignment, while maintaining strong performance on downstream tasks.
2

---

Published as a conference paper at COLM 2025
Orthogonal
Orthagoral
Safe
Safe NLU
NLU
Pretrained Weight
e, O.M,
Sparse
Merge
Safety
Math
Mathy
Safety Adapters
Merge
Merge
A
Dense
Safety
Safe Math
NUU Math Code
Code
Safe Corc
NLU Math Code
Safe Codo
Trainable Params
Params
(a) LoRl method.
(b) LoRT merging.
(c) LoRl continual learning.
Figure 2: Overview of the proposed LoRI method. (a) LoRT freezes the projection matrices A and sparsely updates B, using task-specific masks M, (b) LoRI enables adapter merging of multiple task- specific adapters with reduced parameter interference. (c) LoRI builds safety adapters by continual learning with reduced catastrophic forgetting.
Method
2.1 Freezing Low-Rank Projections with Sparse Masking
Freezing Projection A. LoRA (Hu et al., 2021) fine-tunes a weight update matrix as a product of two low-rank matrices to adapt LLMs to new tasks. Formally, for specific task t, given pretrained weight matrix Wo R Xdeot, the weight update A; E R4 is constrained to low-rank decomposition:
h = xw + xA, xWy + xAAB
where At E Rdm B; E Rexdout and r < min (din dout]. We denote A, as the LoRA adapter for task f. In practice, LoRA adapters are typically applied to multiple projection matrices (e.g,, Wg, We) within each transformer layer.
Typically, the low-rank projection matrices A, and the low-rank expansion matrices Br are updated via gradient descent. Matrices A, are usually initialized with Kaiming Uniform distribution (He et al., 2015), while matrices B, are initialized to zero, ensuring that A 0 at the start of training. However, in LoRI, we fix A, as random projections, meaning that the model only learns how to combine the fixed subspace via B,. By freezing A,, we eliminate the need to store their gradients and optimizer states, thereby reducing memory consumption. During inference, similar to LoRA, LoRI merges the low-rank updates by adding A, to Wo, ensuring additional inference latency compared to full fine-tuning.
Sparse Masking for Projection B. LoRI freezes matrices A, and selectively updates only the most relevant parameters in B, for each task, as illustrated in Figure 2(a). For task f, first extracts sparse masks M, through a calibration process, then applies the masks to constrain training to limited subset of parameters in B,. During mask calibration, LoRI updates B, without masking using calibration dataset DG, sampled from the adaptation dataset D,. After this phase, LoRI collects all B, matrices from the model across layers and projections. Then it computes global threshold T,, defined the s% quantile of the absolute values of all elements from these matrices, where s is the sparsity ratio. For each matrix B,, the corresponding sparse mask M; computed as:
M, = I ( 2 TH), where
= Quantile; (UIB.)).
(2)
Here, I(.) denotes the indicator function applied element-wise. This ensures that only the top-(1 5)% of parameters (by magnitude) across all layers and projections are retained. The masks can also be derived using gradient-based measures such as the Fisher informa- tion matrix (Guo et al., 2023; lurada et al., 2025) or SNIP score (Lee et al., 2018). However, these methods capture local sensitivity at a specific training step, whereas magnitude re- flects cumulative importance over the entire fine-tuning process.
3

---

Published as a conference paper at COLM 2025
It is well established that the importance of projection matrices varies significantly across different layers and projections (Zhang et al., 2023a;d; Kopiczko et al., 2023). Our masking strategy enables global comparison of parameters and facilitates effective allocation of the parameter budget determined by the sparsity ratio. Notably, the masks for each task are calibrated only once and can be reused as needed.
After mask calibration, LoRI resets B, to zero and trains on the adaptation dataset D with updates restricted to the masked parameters. The LoRI adapter expressed as A (B @ M2). The algorithm of LoRI is detailed in Appendix B. In practice, the sparsity ratio can reach up to 90%, meaning that only a small fraction of parameters in matrices B, are updated, while the majority remain unchanged. This selective adaptation enables the model to focus on modifying the most critical parameters needed for specific tasks, while preserving the foundational knowledge encoded in the pretrained base model. In the limiting case of single task and zero sparsity, our method reduces to LoRA-FA (Zhang et al., 2023b), which has been shown to perform competitively with standard LoRA.
2.2 Reducing Interference in Adapter Merging via Orthogonality
Orthogonality of LoRI Adapters. A central challenge in adapter merging is parameter in- terference, where combining multiple adapters leads degraded performance due to con- flicting parameter updates. Given a set of trained LoRI adapters {A, .2,,..,AA)) the goal to construct unified model that combines knowledge from all tasks with minimal inter- ference, as illustrated in Figure 2(b). Formally, we define the excess loss due to parameter interference for a specific task l as:
I₁ = Li (W (WW + a;
(3)
where Wmerge is the merged model, Wo is the pretrained weight matrix, A is the LoRT adapter for task t, a4 E R is a scalar weight, and L, is the loss function for task t. A high I indicates significant interference.
LoRI mitigates this interference by leveraging approximate orthogonality, achieved by freez- ing the projection matrices A, as independent random matrices. This design leads to the following property, whose proof is provided in Appendix C:
Property 1. Let As E Rdaxr be independent random matrices with i.i.d. entries drawn from a Kaiming Uniform distribution for distinct tasks s + f. Let their corresponding LoRI adapters be A, As (B. M3) and A A₂( (B, @ Mi), where the trained matrices (Bg Ms) and (B, + M,) have finite Frobenius norms. Under the condition that r < dm as the input dimension din -> 60, the adapters are approximately orthogonal:
(As, A:) - in probability.
(4)
We describe two merging methods: concatenated merging (weighted averaging) and linear merging (Task Arithmetic) (Ilharco et al., 2022), both of which exploit the approximate
orthogonality of LoRIs
Concatenated Merging (Weighted Averaging). This method constructs the merged model by creating weighted sum of individual task adapters. This is achieved by con- catenating the weighted A and masked B matrices:
A' [aiA] a2A2 ... AT),), B' = [[BIO (B.. OBT
where 0; E IR are scalar weights (e.g., uniform or task posk-prroritizzed. The final merged model then formed by adding their product to the base model weights:
Wmerge = Wo + A'B' = Wo + Σ M. M,) = Wo + Σ G.
(6)
By summing approximately orthogonal adapters, ensure that the updates for each task occupy largely disjoint subspaces, thereby reducing interference (Ilharco et al., 2022; Ortiz- Jimenez et al., 2023; Xiong et al., 2024).
4

---

Published as a conference paper at COLM 2025
The reduction in interference can be explained by theoretical sketch based on two key sumptions. The first is the local linearity of the loss landscape (Li et al., 2018), which allows for first-order Taylor approximation. The second is the gradient alignment assumption, formally expressed as (Wo + a;Ai) x A, This posits that at task's solution, the di- rection of steepest descent is primarily aligned with the adapter updates already made for that task. Under these assumptions, the excess loss Z, is approximately the inner product of the gradient and the updates from the other tasks:
I ~ (W + arAr), E as 5f1
a L
Since Property establishes that (At, Az) 0 for # the total interference loss becomes negligible: I = O. This heuristic argument provides strong intuition for why concatenated
merging is effective, which then validated by our empirical results.
Linear Merging (Task Arithmetic). Alternatively, the merged model can be formed by summing the A and masked B₁ matrices independently before multiplication:
)
T (
T
Wmerge- = Wo +
Σ Ai
a
(B.
Wo +
L s=1==1
Σ
a,a A, O M.). (8)
While concatenated merging directly sums approximately orthogonal adapters, this linear merging approach introduces problematic cross-terms Rs @ M,) for s * t. These terms cause interference because components like {As(B, (Br @ M,) for a fixed s are gen- erally not mutually orthogonal. As result, concatenated merging offers cleaner and empirically more effective strategy for combining LoRI adapters.
2.3 Reducing Interference in Continual Learning via Sparsity
Adapters. For safety-critical applications, ensuring that new task adaptations do not compromise established safety behaviors is crucial. Therefore, each newly introduced adapter must preserve the base model's safety alignment. A straightfor- ward approach achieve this is to merge a safety LoRI adapter into the deployed model during every inference. However, sufficient for scenarios that demand strong safety guarantees. In such cases, as illustrated in Figure 2(c), more reliable solution is to adopt two-phase continual learning process for each LoRI adapter to reinforce safety:
we will show in Section 3.4, this method may be in-
Train a LoRI adapter on curated safety dataset Dsafety. yielding saaeety = A(Bsafety @ Mgafety).
Safety Alignment Phase:
2. Task Adaptation Phase: Fine-tune Asafety on each task adaptation dataset Di, = 1, reusing the calibrated task-specific masks Mr resulting in safety- preserving adapters A = A(B @ Mt).
This method does not require recalibrating masks for each task or performing multi- ple rounds of continual learning. Notably, we do not enforce non-overlapping masks M, Msaeety = Ø. Enforcing such a constraint would require recalibrating masks after the safety alignment phase due to the reduced parameter space, and could potentially de- grade performance on downstream tasks. The expected overlap between sparse masks with 90% sparsity is theoretically 1%. Empirically, we find that this expectation holds: the average overlap between task-specific masks is indeed ~ 1%, without explicitly enforcing non-overlap. This slight overlap allows important parameters to be shared across tasks, potentially enabling positive knowledge transfer.
Catastrophic Forgetting. Continual learning models vulnerable to catastrophic forget- ting (Li & Hoiem, 2017; Dong et al., 2023; Luo et al., 2023), where updates for new tasks can overwrite and degrade previously learned knowledge. Despite the slight overlap between
5

---

Published as a conference paper at COLM 2025
task-specific masks, the sparsity in B, induced by M, enables LoRI to facilitate isolated pa- rameter updates for safety alignment and task adaptation. As a result, LoRI minimizes cross-task interference and mitigates catastrophic forgetting in safety alignment.
3 Experiments
3.1 Experimental Setup
Datasets. We conduct series of experiments to evaluate LoRI's effectiveness on single- task and multi-task settings, including adapter merging and continual learning. We focus on four capabilities: (i) Natural Language Understanding (NLU): LoRI trained on the aggregation of eight NLU datasets (Hu et al., 2023), including BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SocialIQA (Sap et al., 2019), ARC-Challenge (Clark et al., 2018), ARC-Easy (Clark et al., 2018), OpenBookQA (Mihaylov et al., 2018), HellaSwag (Zellers et al., 2019), and Winogrande (Sakaguchi et al., 2021). We evaluate accuracy on the indi- vidual test split for each dataset. (ii) Mathematical Reasoning (Math): LoRi is trained on the GSM8K (Cobbe et al., 2021) training split and evaluated on the GSMSK test split. (iii) Code Generation (Code): LoRI is trained on CodeAlpaca (Chaudhary, 2023) and evaluated using pass@1, pass@5, and pass@10 @10 on HumanEval (Chen et al., 2021). (iv) Safety Align- ment (Safety): LoRT trained on Saferpaca (Bianchi al., 2023), which extends Alpaca- Cleaned (Taori et al., 2023) with 2,000 safety instructions. Safety performance is assessed by measuring the refusal rate on harmful queries from HEx-PHI (Qi et al., 2023).
Baselines. In single-task experiments, we compare LoRI with full fine-tuning (FFT), LoRA (Hu et al., 2021), and DoRA (Liu et al., 2024). Results for additional PEFT base- lines, including VeRA (Kopiczko et al., 2023), IA3 (Liu et al., 2022), LoRA-FA (Zhang et al., 2023b), AdaLoRA (Zhang et al., 2023d), rsLoRA (Kalajdzievski, 2023), PiSSA (Meng et al., 2024), and LoRA+ (Hayou et al., 2024), are available in Appendix E.1. In merging experi- ments, we compare LoRI merging with several LoRA merging methods, including concate- nated merging, linear merging (Ilharco et al., 2022), magnitude pruning, TIES-Merging (Ya- dav al., 2023), and DARE (Yu et al., 2024). Magnitude pruning, TIES, and DARE are pruning-based approaches that apply sparsification to the A and B matrices before merg- ing, based on specified density. Magnitude pruning removes low-magnitude parameters; TTES-Merging further merges weights with consistent signs; and DARE performs random pruning followed by rescaling. For fair comparison, all baseline results are reproduced using a consistent experimental setup.
Implementation Details. We use Llama-3-8B (Grattafiori et al., 2024) and Mistral- 7B (Jiang et al., 2023) as base models. We conduct all experiments on NVIDIA A5000 GPUs. To explore the impact of sparsity, we provide two variants of LoRI: LoRI-D, which uses dense matrices, and LoRi-s, which applies 90% sparsity to Sparsity is imple- mented by masking the gradients of B during backpropagation. For optimal performance, we use the entire adaptation dataset as the calibration dataset for each task. Ablation re- sults for calibration are presented in Section 3.5. For consistency, we use the same hyperpa- rameters for PEFT baselines as for LoRI-D, For all adapter merging experiments, uniform weights at are employed across all adapters. The weights a; are treated hyperparame- ters, and their ablation study is detailed in Section 3.5. Detailed hyperparameter settings are provided in Appendix D.
3.2 Single-Task Performance
Table 1 presents single-task performance on eight NLU benchmarks, while Table 2 reports single-task performance on the math, code, and safety benchmarks. Results for additional
PEFT baselines are available in Appendix E.1. The rank for our experiments is set to 32. We observed stable performance across different ranks, with additional results for = 64 provided in Appendix F.2.

---

Published as a conference paper at COLM 2025
Table 1: Performance comparison of different adaptation methods on eight NLU bench- marks using Llama-3 and Mistral with r 32. Bold indicates the best-performing method, and underline indicates the second-best.
Method
# Params (%)
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Avg.
Llama-3 3-8B 8B
FFT
8.03G (100%)
73.8
86.8
77.6
76.7
87.6
84.1
93.2
85.1
83.1
LoRA
84M (1.03%)
76.3
89.8
82.7
83.4
91.7
88.4
95.8
88.7
87.1
DoRA
85M (1.05%)
75.9
89.8
82.7
83.5
93.2
87.9
95.3
88.2
87.1
LoRI-D
(0.54%)
76.4
82.7
84.2
93.6
88.5
95.9
87.9
87.3
LoRI-S
4.4M (0.05%)
75.2
89.2
82.8
83.8
92.6
88.4
95.2
87.5
86.8
Mistral-7B
FFT
7.24G (100%)
74.1
84.6
78.0
79.3
90.5
88.4
94.4
83.5
84.1
LoRA
84M (1.15%)
75.2
90.1
82.9
82.9
92.0
88.7
95.1
88.1
86.9
DoRA
85M (1.16%)
75.8
90.4
82,9
83.3
92.6
90.6
96.3
87.9
87.5
LoRI-D
44M (0.60%)
75.9
90.6
83.0
83.6
91.9
88.4
95.9
87.4
87.1
LoRI-S
4.4M (0.06%)
74.0
90.1
82.6
82.6
91.5
90.8
95.5
87.5
86.8
Table 2: Performance comparison of different adaptation methods on GSM8K (math), Hu- manEval (code), and HEX-PHI (safety) benchmarks using Llama-3 and Mistral with = 32. Bold indicates the best-performing method, and underline indicates the second-best.
HumanEval Pass@5
Method
# Params (%)
GSM8K
HEx-PHI
Pass@1
Pass@10
Llama-3-a
FFT
8.03G (100%)
58.8
30.5
39.3
41.7
94.8
LoRA
84M (1.03%)
64.4
34.7
46.4
50.8
91.6
DoRA
85M (1.05%)
65.4
33.1
44.0
48.6
93.6
LORI-D
44M (0.54%)
63.2
43.2
57.6
63.2
92.8
LORI-S
4.4M (0.05%)
62.7
41.3
54.4
59.6
93.8
Mistral-7B
FFT
7.24G (100%)
55.5
29.1
38.5
40.4
94.1
LoRA
84M (1.15%)
57.8
33.8
42.4
45.3
91.9
DoRA
85M (1.16%)
57.5
33.7
42.6
46.8
95.3
LoRI-D
44M (0.60%)
58.0
33.8
42.0
45.1
94.7
LoRI-S
4.4M (0.06%)
57.1
33.7
43.6
48.1
95.9
While full fine-tuning (FFT) updates all model parameters, LoRA and DoRA reduce the number of trainable parameters to approximately 1%. LoRI-D further reduces this to about 0.5% by freezing matrices A, and LoRI-S pushes this reduction to 0.05% by applying 90% sparsity to matrices B, achieving 95% reduction in trainable parameters compared to LoRA. Despite tuning fewer parameters, LoRI-D and LoRI-S achieve performance com- parable to and even better than LoRA and DoRA NLU, math, code, and safety tasks. LoRI-D generally outperforms LoRI-S slightly, due to the extremely limited parame- ter budget in LoRI-S, Remarkably, LoRI-D and LoRi-S consistently outperform FFT, LoRA, and DoRA on code generation tasks. On HumanEval with Llama-3, LoRI-D achieves pass@10 score of 63.2%, outperforming LoRA by 24.4%. LoRI-S achieves 59.6% pass@10, exceeding LoRA by 17.3%.
The strong performance LoRI-D suggests that effective adaptation can be achieved with- out updating A, while the strong performance of LoRI-S indicates that B contains substan- tial parameter redundancy. LoRI's performance gains are attributed to the principled use of sparsity, which serves as strong regularizer during adaptation. Additionally, LoRI pre- serves latent task-specific knowledge embedded in the pretrained model. This supports the view that supervised fine-tuning (SFT) primarily unlocks capabilities already present in pretrained models, rather than introducing new ones, which is consistent with findings from Liu et al. (2024); Yu et al. (2024).

---

Published as a conference paper at COLM 2025
Table 3: Comparison of merging methods for combining four adapters, evaluated on their respective benchmarks. The best- pers-reerfoming single-task adapter, LoRI-D, is used as the single-task baseline. Results for NLU are averaged over eight tasks. Base model: Llama-3- 8B, rank r = 32. Bold indicates the best-performing method, and underline indicates the second-best.
HumanEval Pass@5
Merging
Adaptation
NLU
GSM8K
HEx-PHI
Pass@1
Pass@10
Single-Task
LoRI-D
87.3
63.2
43.2
57.6
63.2
92.8
Concat
LoRA
85.0
57.8
13.0
20.0
22.3
84.4
Linear
LoRA
84.8
54.1
14.2
20.8
23.3
79.4
Magnitude
LoRA
81.9
50.3
24.1
36.7
42.4
74.4
TIES
LoRA
72.6
24.0
32.5
46.3
51.7
77.8
DARE
LoRA
79.1
48.9
34.1
48.7
53.5
74.1
Concat
LoRI-D
83.2
55.8
40.5
56.9
86.6
Linear
LoRI-D
82.5
53.8
40.9
54.9
60.3
85,9
Concat
LORI-S
81.2
45.2
34.3
48.7
54.0
84.7
Linear
LORI-S
79.1
41.3
23.2
36.6
42.3
78.8
3.3 Adapter Merging
We consider four heterogeneous tasks for LoRA and LoRI merging: NLU, math, code, and safety. This setting is generally more challenging than merging homogeneous adapters, such merging multiple NLU adapters. Table presents results for merging LoRAs and LoRIs on these four tasks. For LoRI, we apply concatenated and linear merging to the LoRI-D and LoRI-S variants. Pruning-based methods such magnitude pruning, TIES, and DARE are not applied to LoRI, since these methods will prune the A matrices as LoRI already sparsifies B, resulting in an inconsistent pruning scheme across A and B. Additional results, including experiments on merging three adapters and evaluations of pruning-based methods on LoRI, are provided in Appendix E.4 and E.5.
As shown in Table 3, directly merging LoRAs results in substantial performance degra- dation, particularly for code generation and safety alignment. Although pruning-base methods (e.g., DARE, TIES) improve code performance, they often compromise accuracy on other tasks. In contrast, LoRi achieves consistently strong performance across all tasks.
Concatenated merging with LoRI-D achieves the best overall performance, closely match- ing the single-task baseline, which indicates minimal interference between LoRI adapters. For instance, it achieves 62.2% pass@10 10 on HumanEval and an 86.6% refusal rate HEx- PHI. Despite using only 5% of the parameters of LoRA, LoRI-S retains competitive perfor- mance. Notably, on code and safety tasks, concatenated merging with LoRI-S outperforms all LoRA merging methods.
Linear merging with LoRT also performs competitively, though it lags slightly behind con- catenated merging due to cross-term interactions that introduce some interference. LoRT eliminates the need for manual selection of merging methods: simple concatenated merg- ing yields strong results. The choice between LoRI-D and LoRI-S can then be guided by the desired trade-off between performance and parameter efficiency. We also note an im- portant trade-off between code generation performance and other domains during adapter merging, a phenomenon further explored in Section 3.5.
3.4 Continual Learning
While merging adapters enables multi-task capabilities, falls short of providing robust safety alignment in scenarios that demand strong safety guarantees. As shown in Table 3, the highest refusal rate on HEx-PHI achieved through LoRA or LoRI merging is 86.6%. To address this limitation, trained on the safety alignment dataset Saferpaca; then, it is individually adapted to each downstream task, including NLU,, math, and code.
adopt two-phase training process: first, safety adapter is
8

---

Published as a conference paper at COLM 2025
Single-Task LoRI-D
Multi-Task LoRA
Multi-Task LoRI-D
Multi-Task LoRI-S
100
21.4
347 22334720444
100
Performance (%)
37.11.1 57.2 05.
80
80
80
60
60
632
10.4
60
03.2
66.1
535
48.4
40
40
40
37.3
20
20
20-
Safety
NLU
Safety
GSMBK
Safety (c) Safety
HumanEval Code.
(a) Safety NLU.
(b) Safety
Math.
Figure 3: Continual learning results from safety to NLU, math, and code domains. Results for NLU averaged over eight tasks. GSM8K accuracy, HumanEval pass@10, and HEx-PHI refusal rate reported individually. Base model: Llama-3-8B, rank 32.
62.75
98
up
62.50
96
Accuracy (%6)
57.5
Sparsity Ratio
62.25
94
62.00
55.0
92
61.75
52.5
61.50
50.0
Code
88
61.25
10
20
30
Calibration Steps
Layer Index
(a) Effect of calibration steps.
Sparsity ratios across layers and projections.
Model-wise
62.1
80
Weight 0.3 Weight
Weight 0.2
822
Performance (96)
Module-wise
62.2
60
Projection-wise
61.6
40
37.0
Layerwise
21.
Matrix-wise
60.5
20
GSMBK Accuracy (%)
NLU
Math
Code
Safety
(c) Effect of mask granularities.
(d) Effect of merging weights.
Figure 4: Ablation studies across different settings. Base model: Llama-3-8B, rank r = 32. Additional ablation studies are provided in Appendix F.
Figure presents results from these continual learning experiments. LoRA exhibits se- vere catastrophic forgetting on safety alignment particularly in the safety - NLU exper- mment-likely due the large size of the NLU training split (~110k examples). Among all methods, LoRI-S achieves the best preservation of safety alignment, even outperforming single-task LoRI-D. This due to its 90% sparsity the B matrices, which enables isolated parameter updates between the initial safety alignment and subsequent task adaptations. LoRI-D also shows some resistance to forgetting, benefiting from frozen matrices. For task adaptation, LoRI-D generally outperforms LoRI-S, the latter's 's aggressive sparsity limits its adaptation capacity. Overall, LoRI offers a lightweight and effective approach to building safety adapters that preserve alignment while supporting adaptation to down- stream tasks.
3.5 Ablation Studies
Calibration Steps. Calibration steps refer to the number of update steps used to generate sparse masks for each task. Figure 4(a) shows how performance of LoRI-S changes with

---

Published as a conference paper at COLM 2025
different numbers of calibration steps math and code tasks. We observe that perfor- mance generally improves as the number of calibration steps increases. Since the masks only need to be calibrated once per task and can be reused, use the entire adaptation dataset as the calibration dataset to achieve the best performance.
Sparsity Ratio. We use model-wise masks in our experiments that retain the highest- magnitude parameters across all layers and projections. Figure 4(b) presents the sparsity ratios of different projection types (e.g, up, down, key, value) across layers under 90% sparsity GSM8K. We observe that feedforward (FFN) projections tend to retain more parameters (i.e,, lower sparsity) than self-attention projections, indicating they are more critical for adaptation. Additionally, the top layers are less sparse than lower layers, sug- gesting that the top layers play a more important role in adaptation.
Mask Granularity. We compare five levels of mask granularity under 90% sparsity on GSM8K, as shown in Figure 4(c). We compare module-wise, projection-wise, layer-wise, and matrix-wise masking against our model-wise masking, where parameters are selected within progressively smaller scopes. We find that coarse-grained masking (e.g,, model- wise) yields the best performance, while fine-grained masking (e.g., matrix-wise) results in degradation. This suggests that global magnitude-based selection enables better parameter allocation, as the importance of projection matrices varies across the model.
Merging Weights. We adopt uniform weights across all adapters for adapter merging, rather than task-specific weights, as we do not wish to prioritize any individual task. Fig- ure 4(d) shows the effect of different merging weights (0.2, 0.3, 0.4) for concatenated merg- ing with LoRI-S. We observe that LoRI is moderately sensitive to merging weights, with noticeable trade-off between performance on code tasks and other domains. We adopt 0.3 for all adapters in LoRI-S merging, as it offers balanced performance across domains.
Conclusion
In this work, we introduced LoRI, a simple yet effective approach to parameter-efficien fine-tuning (PEFT) that substantially reduces trainable parameters while minimizing cross- task interference. By freezing the projection matrices as random projections and spar- sifying B using task-specific masks, LoRI achieves strong single-task performance across diverse domains including natural language understanding, mathematical reasoning, code generation, and safety alignment- - while reducing trainable parameters by up to 95% compared to LoRA. Furthermore, LoRT enables training-free adapter merging with mini- mal performance degradation, and supports continual learning with significantly reduced catastrophic forgetting. also provides a lightweight approach to building safety adapters that preserve the safety alignment of the base model.
Future Work. We identify several promising avenues for extending this work. While LoRT currently leverages unstructured magnitude-based sparsity, future research can ex- plore structured sparsity patterns such as block sparsity, head pruning, or group-wise masking - which may offer better hardware compatibility. Additionally, although this study focuses on LLMs, the core design of LoRI is modality-agnostic Extending LoRI to diffusion and vision-language models for multi-modal generation a promising direction.
Acknowledgements
This material is based upon work partially supported by the NSF Grant No. 2229885 (NSF Institute for Trustworthy Al in Law and Society, TRAILS), Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
10

---

Published as a conference paper at COLM 2025
References
Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned Ilamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint arXio:2309.07875 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 7432-7493 2229,, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, el. Lan- guage models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Rich Caruana. Multitask learning. Machine learning, 28:41-75, 1997.
Sahil Chaudhary. Code alpaca: An instruction-folloiing llama model for code generation, 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Eval- uating large language models trained code. ar Xiv preprint ar arXiv:2107.0337 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Alexandra Chronopoulou, Matthew E Peters, Alexander Fraser, and Jesse Dodge. Adapter- soup: Weight averaging to improve generalization of pretrained language models. ar Xiv preprint ar Xii::2302.07027, 2023.
Christopher Clark, Kenton Lee, Ming-Wes Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no no ques- tions. arxiv preprint ar 1rXio:1905.10044, 20109
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint ar Xxx::1803.05453 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Train- ing verifiers to solve math word problems. arXiv preprint arXiv:21 14.16 2021.
Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation pre-trained language models. arXio preprint ar Xix:22311.16666 2023.
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large lan- guage models are affected by supervised fine-tuning data composition. arXiv preprint ar arXit:2370.05492, 2023.
Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficient fine-tuning with discrete fourier transform. arXiv preprint ar drXiv:2005.03003, 2024.
Aaron Gratta Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint Xix::22407 .27883, 2024.
Han Guo, Philip Greengard, Eric P Xing, and Yoon Kim. Lq-lora: Low-rank plus quan- tized matrix decomposition for efficient language model finetuning. arXiv preprint arXit:2311.120233 2023.
11

---

Published as a conference paper at COLM 2025
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. Xrv preprint arXio:2402.12354, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Larous- silhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient trans- fer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. Xiv preprint xxXiv:2006.09685, 2021.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Lim-adapters: An adapter family for parameter- efficient fine-tuning of large language models. arXiv preprint ar arii:23044009333 2023.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXit:2307.13269, 2023.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint ar Xit:2212.00408 2022.
Leonardo Iurada, Marco Ciccone, and Tatiana Tommasi. Efficient model editing with task- localized sparse fine-tuning. ar Xiv preprint ar erXiv:2504.02620, 2025.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, et al. Mistral 7b. Xiv preprint arXiv:2310.06825 2023.
Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. Xiv preprint ar Xri::23312.0373,, 2023.
Tatsuya Konishi, Mori Kurokawa, Chihiro Ono, Zixuan Ke, Gyuhak Kim, and Bing Liu. Parameter-level soft-masking for continual learning. In International Conference on Ma- chine Learning, pp. 17492-17505. PMLR, 2023.
Dawid Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. ar preprint ar Xrx::22310.114 2023.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. ar preprint arXt::1810.02340, 2018.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arxio preprint ar xxXi::2104.08611 2021.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the
loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for genera-
tion. Xrv preprint ar arXiv:2101.00190, 2021.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern
analysis and machine intelligence, 40(12):2935-22 2947, 2017.
Zujie Liang, Feng Wei, VinJie, Yuxi Qian, Zhenghong Hao, and Bing Han. Prompts can play lottery tickets well: Achieving lifelong information extraction via lottery prompt tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 277-292, 2023.
12

---

Published as a conference paper at COLM 2025
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficiet fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
Shin-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adapta- tion. In Forty-first International Conference on Machine Learning, 2024.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint ar arX:::2110.072 0700600, 2021.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical
study of catastrophic forgetting in large language models during continual fine-tuning. arviv preprint ar XXXiv:2008.08777, 2023.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging.
Advances in Neural Information Processing Systems, 35:17703-17716, 2022.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37:121038-121077 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint Kii:1800.00288, 2018.
Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances Neural Information Processing Systems, 06666727-667544 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information pro- cessing systems, 01277730-27744 2022.
Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Pra- teek Mittal. Lottery ticket adaptation: Mitigating destructive interference in lims. arv preprint ar xrxi:22066.1699) 2024.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arx preprint ar arXit:2005.00247 2020.
Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks. arviv preprint ar Xiv:2410. 13025, 2024.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Hen- derson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXio:2310.03699, 2023.
13

---

Published as a conference paper at COLM 2025
Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. Effect of scale catas- trophic forgetting in neural networks. In International Conference on Learning Representa- fions, 2021.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Ex- perience replay for continual learning. Advancess in neural information processing systems, 32, 2019.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirk- patrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint ar Xie:1606..4661, 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9): 99-106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXii:1904.09728, 2019.
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objectivve optimization. Ad- vances in neural information processing systems, 31, 2018.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots. ar preprint drXiv:2410.19735, 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-folloiing llama model, 2023.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li and Cheng-Zhong Xu. Hydralora: An asym- metric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 37:9565-9584 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint ar a/Xtt:2302.13971, 2023.
Hanqing Wang, Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, and Maosong Sun. Lora-flow: Dynamic lora fusion for large language models in generative tasks. arXiv preprint ar X124:2240..11455, 2024a.
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of contin- ual learning: theory, method and application. IEEF Transactions on Pattern Analysis and Machine Intelligence, 2024b.
Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learn- ing. arXiv preprint arXiv:2310.14112,, 2023.
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan Fang Li, Guilin Qi, and Gholamreza Haf- fari. Pretrained language model in continual learning: A comparative study. In Interna- tional Conference on Learning Representations 2022. OpenReview, 2022.
Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. arXiv preprint arXio:2404.13628, 2024.
Feng Xiong, Runxi Cheng, Wang Chen, Zhanqiu Zhang, Yiwen Guo, Chun Yuan, and Ruifeng Xu. Multi-task model merging via adaptive weight disentanglement. Xiv preprint ar Xix:222411.188 2024.
14

---

Published as a conference paper at COLM 2025
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties- merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36,0931115,, 2023.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. In Forty-first Inter- national Conferencee on Machine Learning, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXio preprint XxXi:19905.0783, 2019.
Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. Increlora: Incremental parameter allocation method for parfmeter-efficient fine-tuning. aiv preprint ar Xiv:22308 12043, 2023a.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory- efficient low-rank adaptation for large language models fine-tuning, arXiv preprint arXiv:2308.03300 0333033 2023b.
Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bo- han Zhuang. Loraprune: Pruning meets low-rank pfrfieter-efficient fine-tuning, arv preprint Xix::2305.18403, 2023c.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, arviv preprint ar Xrir:2303.10512, 2023d.
Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang. Lora-drop: Efficient lora parameter pruning based on output evaluation. arXiv preprint arXim:2402.07721, 2024.
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez De Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters of foundation models. arXiv preprint ar Xiv:2402. 16842, 2024.
15

---

Published as a conference paper at COLM 2025
A Related Works
Parameter-Efficient Fine-Tuning. Parameter-efficient fine-tuning (PEFT) methods for LLMs (Houlsby et al., 2019; Pfeiffer et al., 2020; Li & Liang, 2021; Lester et al., 2021; Liu et al., 2021; Hu et al., 2021) have received increasing attention in recent years. Among them, LoRA (Hu et al., 2021), which introduces trainable low-rank matrices, has become one of the most widely adopted PEFT methods due to its strong performance and effi- ciency. LoRI is motivated by reducing parameter redundancy in LoRA through an asym- metric design: we freeze the projection matrices A and enforce sparsity on the matrices B. Our work is closely related to several lines of research. In terms of parameter efficiency, our goal is shared by methods such as IA3 (Liu et al., 2022), VeRA (Kopiczko et al., 2023), and FourierFT (Gao et al., 2024). More specifically, our approach builds on the concept of asymmetric LoRA variants, which has been explored in works like LoRA-FA (Zhang et al., 2023b), AsymmetryLoRA (Zhu et al., 2024), and HydraLoRA (Tian et al., 2024). How- ever, LoRI is distinct from these works by uniquely combining frozen A with sparsely up- dated B. This targeted, asymmetric pruning of only the B matrices also differentiates our method from general LoRA pruning techniques like Loraprune (Zhang et al., 2023c), LoRA- drop (Zhou al., 2024), and SoRA (Ding et al., 2023), as well as SVD-based approaches such as AdaLoRA (Zhang et al., 2023d) and PiSSA (Meng et al., 2024).
Model Merging. Achieving multi-task capabilities typically involves training on a mix- ture of diverse task datasets (Caruana, 1997; Sener & Koltun, 2018), which is often pro- hibitively expensive in time and compute. As an alternative, model merging has gained attention for combining multiple task-specific models into single model (Matena & Raffel, 2022; Ilharco al., 2022; Yadav al., 2023; Yu al., 2024). Fisher Merging (Matena Raf- fel, 2022) uses weights from the Fisher information matrix to combine parameters, while Task Arithmetic (Ilharco et al., 2022) employs predefined scaling factors. TIES-Merging (Ya- dav et al., 2023) prunes low-magnitude parameters and merges those with consistent signs, and DARE (Yu et al., 2024) applies random pruning with rescaling. However, identify- ing the optimal merging method often requires trial and error. More recently, there has been growing interest in merging task-specific LoRA adapters (Chronopoulou et al., 2023; Huang et al., 2023; Wu et al., 2024; Wang et al., 2024a; Prabhakar et al., 2024; Stoica et al., (MoE) architectures. Nonetheless, these methods
2024), often utilizing typically require additional training to coordinate the adapters effectively. In contrast, LoRI eliminates the need for manual selection of merging methods or additional training. By ensuring approximate orthogonality between adapters, LoRI minimizes interference and preserves task-specific performance.
cexture-of-Expeer
Catastrophic Forgetting. Catastrophic forgetting is fundamental challenge in continual learning (McCloskey & Cohen, 1989; Ramasesh et al., 2021; Liang et al., 2023; Wang et al., 2024b), where neural networks struggle to retain previously learned knowledge when adapting to new tasks. Wu et al. (2022) analyzed this phenomenon using layer-wise and task-wise probing to assess knowledge retention across tasks. Several studies (Dong et al., 2023; Luo et al., 2023) have empirically examined catastrophic forgetting in the continual fine-tuning of LLMs. To mitigate catastrophic forgetting, various approaches have been proposed. Rehearsal-based methods (Rolnick et al., 2019; Shin et al., 2017) store or generate past data to reinforce prior knowledge during training. Parameter isolation methods (Rusu et al., 2016; Mallya & Lazebnik, 2018; Konishi et al., 2023; Panda et al., 2024) allocate sep- arate subnetworks or sparsely mask parameters for different tasks to prevent interference. Additionally, O-LoRA (Wang et al., 2023) learns tasks in distinct low-rank subspaces while ensuring orthogonality between them. LoRI falls under the category of parameter isolation methods, leveraging sparse task-specific masks mitigate catastrophic forgetting during continual learning.
Forggtttt
B Algorithm of LoRI
The full procedure of LoRT is summarized in Algorithm 1.
16

---

Published as a conference paper at COLM 2025
Algorithm 1: LoRA with Reduced Interference (LoRI)
Require:
Task mask calibration dataset D,, adaptation dataset D,, sparsity ratio model f, loss function L,, learning rate
1: for each layer 1 1,...., do
2: for each projection m = 1,..., do
3:
Initialize: A!(.m) E Rd Xr + U1 u(-v Ain V 2,), B. B!(
E RRrxdeut t-
4: end for
5: end for
6: for each batch y) sampled from D. do
Calibration steps
7: for each (1,m) m) do
B! (l) + B (1))) V B. al( fi(f(x, B, )
8:
9: end for
10: end for
11: To <- Quantile, (Uim (B(()mi))
D Compute global threshold тę
12: for each m) do
13: M
[ (B(0m)) (1.m) 2 Ti
D Generate mask for top-(1 s)% entries
14:
B
B((m)
+
Reset to zero before adaptation
15: end for
16: for each batch (x, y) sampled from D; do
D. Adaptation steps
17:
for each ml) do
B((m) +- B((m) 1/: (v (l))) (x, @ M1
18:
19: end for
20: end for
C Proof of Property
Proof. Our goal to show that the Frobenius inner product At) converges zero in probability. Let B3 B3 O M2 and B, B, o M, The inner product given by:
(I) Tr A.) Tr(B, A, A,B.).
(9)
We will prove this by showing that the random matrix X A3 A: converges to the zero matrix in probability as din + co.
Let at at Rd be the k-th and 1-th columns of A₃ and A, respectively. The entries of these vectors are ii.d. from Kaiming Uniform distribution U[-a,a] al where /3/dmn This implies mean of 0 and variance of x2 a² /3 1 1/dmn- An entry of X is the inner product XR = ) d. = Imm (As)ae(A)iii
Let Z; = (As) (A:) The terms Z; are i.d.d with E|Z] = ({((s) [(()aa) = Each term is bounded: |Z,| a² 3/dm- We apply Hoeffding's inequality to the sum I Zi, where each term lies in 1/3/dm. 3/ /dd:::
d
212
-12
P([xM 2 1) = P
Σ
7
21
< 2xxp
2 exp
(10)
Fm (mm(6// (6/din)²
(6/
18
We now bound the probability that any of the 2 entries of X exceeds a threshold f using the union bound:
(mmax| Xul 2 f) - kd
[1xll ( <
P f) 2F exp
(-fe
(fddm)
(11)
18
E1=1
17

---

Published as a conference paper at COLM 2025
Table 4: Hyperparameter settings for LoRI on NLU datasets.
Method
LoRI-D
LoRI-S
LoRI-D
LORI-S
LoRI-D
LoRI-S
LoRI-D
LoRI-S
Base Model
Llama-3
Llama-3
Llama-3
Llama-3
Mistral
Mistral
Mistral
Mistral
Rank
32
32
64
64
32
32
64
&
64
64
128
128
64
64
128
128
Sparsity Ratio
0
0.9
0
0,9
0
0.9
0
0.9
Learning Rate
5e-5
5e-4
5e-5
1e-4
1e-5
1e-4
1e-5
1e-4
Dropout
0.05
Optimizer
AdamW
Batch size
Warmup Steps
Epochs
Where
k, v, o, gate, up, down
We can now show that ||
/ is small with high probability. Let the failure probability be 8. By setting the bound from the previous step to ò, we can solve for t:
(-Pdim
18 log (2rg(222/0)
5 = =222 exp
(12)
18
dm
With probability at least 1 - 8, we have maxi, [xu| S t. This allows us to bound the Frobenius norm of X:
|X||
Xurl S 22 (max (xa)) < 22
(13)
kj
k,==1
Thus, with probability at least 1 - 6:
-0
=0(r)
18 18 (2r² 18)
logr
| sr-t=ry
(14)
din
din
Since & din, the term |||||= as dm matrix in probability.
This shows that X converges to the
Finally, we bound the magnitude of the original inner product using the Cauchy-Schwarz inequality for the Frobenius inner product and the sub-multiplicative property of the Frobenius norm:
(A Ar) = [Tr(B, XB:) = ((Bs, XB:)
< | XB, xBr
(15)
< B X
|| B.
The norms || and ||B are finite, as determined by the trained adapters. Since we have shown that |||||= - 0 in probability, the entire expression must also converge to 0 in probability.
D Hyperparameter Settings
We summarize the hyperparameter settings used for LoRI in Tables 4, 5, 6, and 7. These include settings for different tasks (NLU, math, code, safety), adapter variants (LoRI-D, LoRI-S), base models (Llama-3-8B and Mistral-7B), and ranks (32 and 64).
For the merging experiments, the hyperparameter settings for merging four adapters are provided in Tables 8 and 9, while those for merging three adapters are provided in Table 10.
18

---

Published as a conference paper at COLM 2025
Table 5: Hyperparameter settings for LoRI the math dataset GSM8K.
Method
LoRI-D
LoRI-S
LoRI-D
LoRI-S
LoRI-D
LoRI-S
LoRI-D
LoRI-S
Base Model
Llama-3
Llama-3
Llama-3
Llama-3
Mistral
Mistral
Mistral
Mistral
Rank
32
64
32
32
64
64
128
128
64
64
Sparsity Ratio
0
0.9
0.9
0.9
0.9
Learning Rate
5e-5
5e-4
5e-5
le-3
5e-5
5e-4
le-4
5e-4
Dropout
0.05
Optimizer
AdamW
Batch size
32
Warmup Steps
Epochs
Where
k, v, o, gate, up, down
Table 6: Hyperparameter settings for LoRI on the code dataset CodeAlpaca.
Method
LoRI-D
LoRI-S
LoRI-D
LoRI-S
LoRI-D
LoRI-S
LoRI-D
LoRI-S
Base Model
Llama-3
Llama-3
Llama-3
Llama-3
Mistral
Mistral
Mistral
Mistral
Rank
32
32
64
64
32
32
64
64
64
128
128
64
128
Sparsity Ratio
0.9
0.9
0.9
0.9
Learning Rate
5e-5
5e-4
1e-5
le-4
5e-5
5e-4
1e-5
1e4
Dropout
0.05
Optimizer
AdamW
Batch size
32
Warmup Steps
Epochs
Where
9. k, v, o, gate, up, down
Table 7: Hyperparameter settings for LoRI on the safety dataset Saferpaca.
Method
LoRI-D
LoRI-S
LoRI-D
LoRI-S
LoRI-D
LoRI-S
LoRI-D
LoRI-S
Base Model
Llama-3
Llama-3
Llama-3
Llama-3
Mistral
Mistral
Mistral
Mistral
Rank
32
32
64
64
32
32
64
64
64
128
128
64
64
128
128
Sparsity Ratio
0.9
0.9
0.9
Learning Rate
5e-5
5e-4
le-5
le-4
5e-5
5e-4
le-3
le-4
Dropout
0.05
Optimizer
AdamW
Batch size
32
Warmup Steps
0
Epochs
Where
9, v, o, gate, up, down
Table 8: Hyperparameter settings for merging four adapters using Llama-3-8B.
Adaptation LoRA Concat
LoRA Linear
LoRA Magnitude
LoRA TIES
LoRA DARE
LoRI-D Concat
LoRI-D Linear
LoRI-S Concat
LoRI-S
Merging
Linear
Base Model
Llama-3 0.4
Llama-3 0.4
Llama-3
Llama-3
Llama-3 0.4
Llama-3
Llama-3
Llama-3
Llama-3
Weights
0.4
0.4
0.3
Density
0.7
0.7
Table 9: Hyperparameter settings for merging four adapters using Mistral-7B.
Adaptation
LoRA
LoRA Linear
LoRA Magnitude
LoRA TIES
LoRA DARE
LoRI-D Concat
LoRI-D Linear
LoRI-S Concat
LoRI-S Linear
Merging
Concat
Base Model
Mistral
Mistral 0.4
Mistral 0.4
Mistral 0.4
Mistral 0.4
Mistral 0.4
Mistral 0.4
Mistral 0.3
Mistral 0.3
Weights
0.4
Density
0.3
0.7
0.7
19

---

Published as a conference paper at COLM 2025
Table 10: Hyperparameter settings for merging three adapters using Llama-3-8B.
Adaptation
LoRA
LoRA Linear
LoRA Magnitude
LoRA TIFS
LoRA DARE
LoRI-D
LoRI-D
LoRI-S
LoRI-S
Merging
Concat
Concat
Linear
Linear
Base Model
Llama-3 0.5
Llama-3 0.5
Llama-3 0.5
Llama-3 0.5
Llama-3
Llama-3 0.5
Llama-3
Llama-3 0.4
Llama-3
Weights
0.5
0.5
0.4
Density
0.3
0.7
0.7
Table 11: Performance comparison of different adaptation methods on eight NLU bench- marks using Llama-3 with r = 32. Bold indicates the best-performing method, and underline indicates the second-best.
Method
Params (%)
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Avg.
FFT
8.03G (100%)
73.8
86.8
77.6
76.7
87.6
84.1
93.2
85.1
83.1
LoRA
84M (103%)
76.3
89.8
82.7
83.4
91.7
88.4
95.8
88.7
87.1
VeRA
1.38M (0.02%)
64.4
62.6
67.3
85.7
60.9
78.5
56.9
69.8
AA3
L.70M (0.02%)
68.6
84,8
74.5
77.6
89.4
75.7
90.6
75.0
79.5
LoRA-FA
44M (0.54%)
74.0
89.6
83.3
83.8
93.4
88.6
96.1
87.4
87.0
AdaLoRA
84M (1.03%)
75.6
89.2
82.4
83.1
91.0
87.8
94.4
87.6
86.4
LsLoRA
84M (1.03%)
72.8
84.8
78.8
76.0
87.0
85.0
91.0
82.8
82.3
PISSA
84M (1.03%)
68.1
84.4
78.2
75.1
85.1
82.8
89.3
82.8
80.7
LoRA+
84M (1.03%)
67.0
80.3
78.5
70.1
82.3
81.5
88.9
79.7
78.5
DoRA
85M (1.05%)
75,9
89.8
82.7
83.5
93.2
87.9
95.3
88.2
87.1
LoRI-D
44M (0.54%)
76.4
89.0
82.7
84.2
93.6
88.5
95.9
87.9
87.3
LORI-S
4.4M (0.05%)
75.2
89.2
$2.8
83.8
92.6
88.4
95.2
87.5
86.8
E Additional Experimental Results
E.1
Comparison with Additional PEFT Methods
To provide comprehensive benchmark, we evaluate LoRI against several widely adopted parameter-efficient fine-tuning (PEFT) methods, including VeRA (Kopiczko et al., 2023), IA3 (Liu al., 2022), LoRA-FA (Zhang al., 2023b), AdALoRA (Zhang et al., 2023d), rsLoRA (Kalajdzievski, 2023), PiSSA (Meng et al., 2024), LoRA+ (Hayou et al., 2024), and DoRA (Liu et al., 2024). The results, presented in Tables 11 and 12, demonstrate that our proposed methods are highly effective.
LoRI-D, which uses 44M trainable parameters (0.54% the full model and half of LoRA's), consistently achieves state-of-the-art performance, particularly NLU and code genera- tion benchmarks. LoRI-S, despite its aggressive sparsity (0.05% of the full model and 5% of LoRA's), remains highly competitive and often surpasses other PEFT methods. While VeRA and IA3 are more parameter-efficient their performance is substantially lower than LoRI-S. Despite this efficiency, LoRI-D and LoRI-S deliver comparable and often superior performance across NLU, math, code, and safety domains. These results underscore two key insights: (1) effective adaptation does not require updating the projection matrices A, as demonstrated by LoRI-D; and (2) the matrices contains significant redundancy that can be effectively pruned, as shown by LoRI-S.
E.2 Results with Rank 64
We evaluate several adaptation methods using a higher adapter rank of - 64 across diverse set tasks. This allows for more expressive adapter representations while still maintaining efficiency compared to full fine-tuning. Table 13 presents performance eight natural language understanding (NLU) benchmarks, while Table 14 includes results GSMBK (math), HumanEval (code), and HEx-PHI (safety). Across Llama-3 and Mistral models, LoRI-D and LoRI-S consistently perform competitively, often outperforming larger adapter methods like LoRA and DoRA, while using fewer parameters.
20

---

Published as a conference paper at COLM 2025
Table 12: Performance comparison of different adaptation methods on GSM8K (math),
HumanBval (code), and HEx-I'HI (safety) benchmarks using Llama-3 with 32. Bold indicates the best-performing method, and underline indicates the second-best.
HvmanEval Pass@5
Method
# Params (%)
GSM8K
HEx-PHI
Pass@1
Pass@10
FFT
8.03G (100%)
58.8
30.5
39.3
41.7
94.8
LoRA
84M (1.03%)
64.4
34.7
50.8
91.6
VeRA
1.38M (0.02%)
30.6
32.4
45.1
50,9
74.7
IA3
1.70M (0.02%)
48.0
32.7
45.6
51.5
85.4
LoRA-FA
44M (0.54%)
64.8
42.9
57.5
64.2
94.1
AdaLoRA
84M (1.03%)
63.3
33.5
45.0
49.4
91.9
rsLoRA
84M (1.03%)
61.3
28.4
35.5
38.3
98.1
PiSSA
84M (1.03%)
61.3
32.0
40.3
43.3
97.8
LoRA+
84M (1.03%)
61.7
33.0
42.7
46.0
98.8
DoRA
85M (1.05%)
65.4
33.1
44.0
48.6
93.6
LoRI-
44M (0.54%)
63.2
43.2
57.6
63.2
92.8
LoRI-S
4.4M (0.05%)
62.7
41.3
54.4
59.6
93.8
Table 13: Performance comparison of different adaptation methods on eight natural lan- guage understanding (NLU) benchmarks using Llama-3 and Mistral with 64. Bold indicates the best-performing method, and underline indicates the second-best.
Method # Params (%)
BoolQ
PIQA
SIQA
ARC-
ARC-e
OBQA
HellaS
WinoG
Avg.
Llama-3-88
FFT
8.03G (100%)
73.8
86.8
77.6
76.7
87.6
84.1
85.1
83.1
LoRA
168M (2.05%)
75.2
89.0
81.2
82.3
92.4
89.1
95.3
88.2
86.6
DoRA
169M (2.06%)
76.4
89.0
82.0
82.6
92.3
87.5
95.1
87.3
86.5
LORI-D
88M (1.07%)
75.8
90.4
82.7
83.3
92.6
88.6
95.9
87.4
87.1
LoRI-S
8.8M (0.11%)
76.5
90.2
81.9
83.5
93.8
87.5
96.2
87.2
87.1
Mistral 7B
FFT
7.24G (100%)
74.1
84.6
78.0
79.3
90.5
88.4
94.4
83.5
84.1
LoRA
168M (2.26%)
77.4
90.2
83.5
84.0
93.0
89.3
95.6
89.4
87.8
DoRA
169M (2.28%)
76.0
90.6
83.5
83.3
92.8
89.6
95.7
87.6
87.4
LoRI-D
(1.18%)
75.9
90.7
83.7
82.0
92.1
90.0
96.4
87.8
87.3
LoRI-S
(0.12%)
74.2
83.0
92.6
89.5
87.3
Table 14: Performance comparison of different adaptation methods on GSM8K (math), HumanEval (code), and HEx-PHI (safety) benchmarks using Llama-3 and Mistral with r = 64. Bold indicates the best-pertee cost-performing method, and underline indicates the second- best.
HumanEval
Method
# Params (%)
GSMSK
HEx-PHI
Pass@1
Pass@5
Pass@10
Llama-3-8B
FFT
8.03G (100%)
58,8
30.5
39.3
41.7
94.8
LoRA
168M (2.05%)
63.9
38.6
52.9
59.2
94.1
DoRA
169M (2.06%)
63.8
39.4
53.6
59.7
93.4
LoRI-D
88M (1.07%)
63.8
41.9
55.4
60.3
LoRI-S
8.8M (0.11%)
61.8
44.1
57.4
62.4
96.3
Mistral-7B
7.24G (100%)
55.5
30.5
39.3
94.1
LoRA
168M (2.26%)
56.7
33,9
43.1
46.9
95.9
DoRA
169M (2.28%)
57.8
32.9
43.3
47.2
96.6
LORI-D
88M (1.18%)
58.2
33.3
43.6
47.3
90.9
LORI-S
8.8M (0.12%)
58.4
32.1
42.2
46.3
93.4
21

---

Published as a conference paper at COLM 2025
Table 15: Comparison of merging methods for combining four adapters, evaluated on their respective benchmarks. The best rest-performing single-task adapter, LoRI-D, is used as the single-task baseline. Results for NLU are averaged over eight tasks. Base model: Mistral- 7B, rank r = 32. Bold indicates the best-performing method, and underline indicates the second-best.
HumanEval Pass@5
Merging
Adaptation
NLU
GSM8K
HEx-PHI
Pass@1
Pass@10
Single-Task
LoRI-D
87.1
58.0
33.8
42.0
45.1
94.7
Concat
LoRA
82.5
52.4
32.3
40.8
44.1
75.6
Linear
LoRA
81.4
48.0
33.1
41.6
43.9
76.6
Magnitude
LoRA
77.5
42.7
32.7
41.8
45.6
80.9
TIES
LoRA
31.3
23.5
32.0
40.2
43.5
81.9
DARE
LoRA
76.1
43.0
32.0
41.0
44.6
83.4
Concat
LoRI-D
79.3
52.4
34.4
42.8
83.8
Linear
LoRI-D
78.1
50.5
35.2
42.7
45.5
79.7
Concat
LORI-S
79.2
46.1
33.3
41.6
45,9
79.4
Linear
LoRI-S
75.5
40.3
28.8
36.0
39.6
83.1
Table 16: Comparison of merging methods for combining four adapters eight NLU benchmarks. The best-performing single-task adapter, LoRI-D, is used as the single-task baseline. Base model: Llama-3-8B, rank 32. Bold indicates the best-performing method, and underline indicates the second-best.
Merging
Adaptation
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Avg.
Single-Task
LoRI-D
76.4
89.0
82.7
84.2
93.6
88.5
95.9
87.9
87.3
Concat
LoRA
73.9
89.1
81.1
92.4
83.0
94.4
84.5
85.0
Linear
LoRA
73.7
88.8
81.1
80.7
91.6
84.4
93.9
84.1
84.8
Magnitude
LoRA
72.0
87.1
76.8
79.4
91.7
81.5
90.4
76.4
81.9
TIES
LoRA
68.2
83.8
67.3
69.5
87.8
69.2
73.3
61.4
72.6
DARE
LoRA
70.7
85.0
74.1
77.5
90.7
76.6
86.8
71.0
79.1
Concat
LoRI-D
74.0
87.7
77.8
81.0
81.0
92.7
78.9
83.2
Linear
LoRI-D
73.7
76.7
92.0
77.7
82.5
Concat
LoRI-S
71.8
86.2
76.1
79.2
91.5
78.6
89.8
76.3
81.2
Linear
LoRI-S
70.7
85.3
75.1
78.0
90.8
75.0
86.5
71.3
79.1
E.3 Merging Four Adapters
To support multi-task learning within a unified model, we study the merging of four task- specific adapters using various strategies. Table 15 reports results using Mistral-7B across range of tasks. Additionally, Tables 16 and 17 break down the performance of NLU on indi- vidual benchmarks using Llama-3 and Mistral, respectively. We compare merging methods such as concatenated merging, linear merging, magnitude pruning, TIES, and DARE. LoRI- based approaches demonstrate strong performance and stability when merging multiple adapters.
E.4 Merging Three Adapters
We further evaluate the merging of three adapters to understand performance when adapt- ing a smaller set of tasks. Tables 18 and 19 summarize the results for Llama-3 across different benchmarks. Similar to the four-task setting, LoRI-D remains strong performer, often exceeding the performance of LoRA. These results highlight that LoRI-based meth- ods are effective with varying levels of task diversity.
E.5 Pruning-Based Merging Methods
Finally, explore pruning-based merging methods, which aim to compress and combine multiple adapters by selectively retaining important weights. We focus on three methods: magnitude pruning, TIES, and DARE. Results are reported for merging both four-adapter
22

---

Published as a conference paper at COLM 2025
Table 17: Comparison of merging methods for combining four adapters eight NLU benchmarks. The best-performing single-task adapter, LoRI-D, is used as the single-task baseline. Base model: Mistral-7B, rank 32. Bold indicates the best-performing method, and underline indicates the second-best.
Merging
Adapiation
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Avg.
Single-Task
LoRI-D
75.9
90.6
83.0
83.6
91.9
88.4
95.9
87.4
87.1
Concat
LoRA
69.0
78.1
84.2
92.4
77.8
82.5
Linear
LoRA
69.2
86.9
77.9
78.5
90.2
82.1
91.5
75.1
814
Magnitude
LoRA
68.7
84.9
74.4
75.9
89.1
77.5
85.6
64.1
77.5
TIES
LoRA
184
69.8
40.7
14.0
21.9
20.1
14.6
50.9
31.3
DARE
LoRA
69.4
84.3
73.1
74.2
88.9
74.3
82.6
61.8
76.1
Concat
LoRI-D
68.4
75.6
76.6
71.1
79.3
Linear
LoRI-D
66.3
68.0
78.1
Concat
LoRL-S
72.6
85.4
74.6
76.5
89.7
80.1
86.0
68.9
79.2
Linear
LoRI-S
67.6
83.8
72.0
73.0
88.3
74.6
80.9
64.3
75.5
Table 18: Comparison of merging methods for combining three adapters, evaluated their respective benchmarks. The best-performing single-task adapter, LoRI-D, is used as the single-task baseline. Results for NLU averaged over eight tasks. Base model: Llama-3-8B, rank = 32. Bold indicates the best-performing method, and underline indi- cates the second-best.
HumanEval Pass@5
Merging
Adaptation
NLU
GSMSK
Pass@1
Pass@10
Single-Task
LoRI-D
87.3
63.2
43.2
57.6
63.2
Concat
LoRA
86.4
54.5
13.0
19.8
21.8
Linear
LoRA
51.9
8.8
16.7
Magnitude
LoRA
83.8
52.0
23.3
37.4
43.0
TIES
LoRA
79.4
26.9
36.3
48.7
53.7
DARE
LoRA
81.1
53.3
36.0
49.5
53.9
Concat
LoRI-D
84.8
59.6
41.5
56.4
61.6
Linear
LoRI-D
84.6
57.6
38.3
56.8
Concat
LORI-S
83.3
51.8
31.2
44.6
49.8
Linear LoRI-S
81.0
41.7
26.6
40.0
44.6
Table 19: Comparison of merging methods for combining three adapters on eight NLU benchmarks. The best-performing single-task adapter, LoRI-D, is used as the single-task baseline. Base model: Llama-3-8B, rank 32. Bold indicates the best-performing method, and underline indicates the second-best.
Merging
Adaptation
BoolQ
PIQA
SIQA
ARC-c
ARC-e
OBQA
HellaS
WinoG
Avg.
Single-Task
LoRI-D
76.4
89.0
82.7
84.2
93.6
88.5
95.9
87.9
87.3
Concat
LoRA
74.7
81.8
82.9
86.2
95.8
86.8
86.4
Linear
LoRA
73.9
89.6
81.4
81.9
93.5
85.5
95.6
87.1
86.1
Magnitude
LoRA
72.2
87.2
78.9
81.2
92.2
83.2
93.0
82.4
83.8
TIES
LoRA
69.5
84.8
74.0
78.4
91.2
77.4
88.8
71.4
79.4
DARE
LoRA
71.0
85.6
75.8
79.5
91.0
78.8
90.7
76.2
81.1
Concat
LoRI-D
73.8
89.0
79.8
$1.0
94.6
84.0
84.8
Linear
LoRI-D
74.1
88.4
80.2
81.3
82.1
94.1
83.6
84.6
Concat
LoRL-S
70.3
87.2
79.1
80.8
92.4
82.1
93.2
81.3
83.3
Linear
LoRI-S
61.5
86.4
78.0
79.5
91.7
80.8
91.3
78.5
81.0
23

---

Published as a conference paper at COLM 2025
Table 20: Comparison of magnitude pruning, TIES, and DARE for combining four adapters, evaluated on their respective benchmarks. The best-performing single-task adapter, LoRI-D, is used as the single-task baseline. Results for NLU are averaged over eight tasks. Base model: Llama-3-8B, rank 32. Bold indicates the best-performing method within each group.
HumanEval Pass@5
Merging
Adaptation
NLU
GSM8K
HEx-PHI
Pass@1
Pass@10
Single-Task
LoRI-D
87.3
63.2
43.2
57.6
63.2
92.8
Magnitude
LoRA
81.9
50.3
24.1
36.7
42.4
74.4
Magnitude
LoRI-D
84.3
50.5
33.3
45.2
51.4
85.9
Magnitude
LoRI-S
76.4
35.2
25.2
36.5
41.0
68.4
TIES
LoRA
72.6
24.0
32.5
46.3
51.7
77.8
TIES
LoRI-D
79.1
38.0
40.3
54.6
59.8
85.3
TIES
LoRI-S
70.4
25.9
34.6
48.4
53.2
77.8
DARE
LoRA
79.1
48.9
34.1
48.7
53.5
74.1
DARE
LoRI-D
83.4
52.0
35.4
51.3
57.8
81.9
DARE
LORI-S
73.4
27.2
34.8
48.1
53.5
75.3
Table 21: Comparison of magnitude pruning, TIES, and DARE for combining four adapters, evaluated on their respective benchmarks. The best-performing single-task adapter, LoRI-D, is used the single-task baseline. Results for NLU averaged over eight tasks. Base model: Mistral-7B, rank r = 32. Bold indicates the best-performing method within each group.
HvmanEval
Merging
Adaptation
NLU
GSM8K
HEx-PHI
Pass@1
Pass@5
Pass@10 10
Single-Task
LoRI-D
87.1
58.0
33.8
42.0
45.1
94.7
Magnitude
LoRA
77.5
42.7
32.7
41.8
45.6
80.9
Magnitude
LoRI-D
76.0
41.5
29.0
36.0
38.7
79.4
Magnitude
LoRI-S
70.5
32.4
28.1
36.1
39.3
77.5
TIES
LoRA
31.3
23.5
32.0
40.2
43.5
81.9
TIES
LoRI-D
65.0
45.4
35.3
44.5
47.8
TIES
LoRI-S
67.8
32.9
28.6
37.2
40.8
78.4
DARE
LoRA
76.1
43.0
32.0
41.0
44.6
83.4
DARE
LoRI-D
76.2
42.3
29.2
37.1
40.7
89.1
DARE
LoRI-S
71.9
34.3
29.2
40.5
44.9
85.0
(Tables 20 and 21) and three-adapter (Table 22) settings, using Llama-3 and Mistral as base models. LoRI-D consistently achieves strong performance across pruning- merg- ing methods. However, the performance of LoRI-S is somewhat lower in these settings. This is because pruning-based methods operate on the dense A matrices but not on the sparse B matrices. This mismatch leads to inconsistent pruning scheme, which can sult in a loss of effectiveness.
F Additional Ablation Studies
Figure 5 presents GSM8K accuracy across a grid of sparsity ratios and learning rates using Mistral-7B with rank 64. We observe that sparse adapters require larger learning rates to train effectively. In particular, models with high sparsity (e.g., above 70%) perform best with learning rate of 14 or higher. This suggests that stronger optimization is necessary to compensate for limited capacity in sparse adapters.
In Figure 6, we analyze how sparsity is distributed across layers and projections when enforcing 90% global sparsity on GSMSK. We find that feedforward (FFN) projections tend to retain more parameters i.e., they exhibit lower sparsity than self-attention projections.
24

---

Published as a conference paper at COLM 2025
Comparison of magnitude pruning, TIES, and DARE for combining three adapters, evaluated on their respective benchmarks. The best-performing single-task adapter, LoRI-D, is used as the single-task baseline. Results for NLU are averaged over eight tasks. Base model: Llama-3-8B, rank 32. Bold indicates the best-performing method within each group.
Table 22:
HumanEval
Merging
Adaptation
NLU
GSM8K
Pass@1
Pass@5
Pass@10
Single-Task
LoRI-D
87.3
63.2
43.2
57.6
63.2
Magnitude
LoRA
83.8
52.0
23.3
37.4
43.0
Magnitude
LoRI-D
84.6
53.7
34.8
48.9
54.7
Magnitude
LoRI-S
77.8
36.6
25.5
38.8
43.8
TIES
LoRA
79.4
26.9
36.3
48.7
53.7
TIES
LoRI-D
82.1
42.2
39.2
52.7
57.7
TIES
LoRI-S
73.8
35.2
34.8
47.9
52.5
DARE
LoRA
81.1
53.3
36.0
49.5
53.9
DARE
LoRI-D
84.0
55.2
33.8
45.8
51.8
DARE
LORI-S
75.3
36.6
36.2
48.9
53.4
1e-3
2.8
1.5
3.6
47.5
54.2
54.6
56.5
56.9
45.8
50
GSM8K Accuracy (%)
5e-4
Learning Rate
51.3
49.9
54.0
54.7
56.1
55.9
58.4
56.9
58.4
42.8
40
1e-4
5e-5
57.2
57.3
56.0
56.9
57.7
56.0
53.3
50.6
47.1
30.8
1e-5
10
20
30
40
50
60
70
80
90
99
Sparsity Ratio of (%)
Figure 5: GSM8K accuracy under different sparsity ratios and learning rates. Base model: Mistral-7B, rank 64.
This indicates that FFN components are more critical for effective adaptation. Additionally, sparsity decreases toward the top of the network, suggesting that higher layers are more important for task-specific specialization.
Lastly, Figure/ 7 explores the effect of merging weights when combining three LoRI-S adapters using concatenated and linear merging. We find noticeable trade-off between performance on code tasks and other domains (e.g., NLU and math). Higher merging weights can improve NLU performance but tend to degrade performance code, high- lighting the challenge of balancing generalization and specialization in multi-task settings.
25

---

Published as a conference paper at COLM 2025
up
down
gate
96
Sparsity Ratio (%6
90
86
0
5
10
15
20
25
30
Layer Index
Figure 6: Sparsity ratios across layers and projections under a 90% sparsity on GSM8K. Base model: Llama-3-8B, rank 32.
53
Weight
Weight
Weight
Weight
Performance (%)
Performance (%))
Weight
Weight
60
Weight
Weight
45.9
3P.13P2
35.335:1
31.2
30.
20
20
NLU
Math
Code
NLU
Math
Code
(a) Concatnated merging with LoRI-S.
(b) Linear merging with LoRI-S.
Figure 7: Ablation study on the effect merging weights when combining three adapters.
Base model: Llama-3-8B, rank 32.
26


Reviews:
[{"review_id": "5f6snK7TuC", "reviewer": "Reviewer_nmP5", "summary": "This paper introduces LoRI, which is based on LoRA but 1) leaves the A matrix unadapted and 2) learns elementwise sparse masks for B. By sharing the Gaussian-initialized A matrix the LoRA updates in a multi adaptor setting will be approximately orthogonal. And the learned sparsity further reduces inteference. Empirically LoRI has good single task performance, and also performs better under adapter merging and continual learning.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Extensive results show that LoRI is competitive.", "reasons_to_reject": "The sparsity matrix is created using the B matrix magnitudes. Justification is needed. For example, sensitivity analysis might be a more principled approach (relevant work in the area include [LQ-LoRA](https://openreview.net/forum?id=xw29VvOMmU) and [TaLoS](https://arxiv.org/abs/2504.02620). See also my questions to authors.", "questions_to_authors": "Did the authors consider sensitivity analysis as a criterion for pruning B?", "extra_scores": {}}, {"review_id": "KWTIuN6jUB", "reviewer": "Reviewer_wCSw", "summary": "The authors propose LoRI, an adapter-based approach that freezes a shared random down-projection and sparsifies the up-projection. Across various tasks, including NLU, mathematical reasoning, code generation, and safety alignment tasks, it shows ~95 % parameter savings and effortless multi-task fusion compared to LoRA.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) It achieves extreme parameter and memory efficiency (>95 % smaller than LoRA) by freezing the down-projection and sparsifying the up-projection while maintaining competitive accuracy.\n2) It minimises cross-task interference due to a shared random subspace, enabling straightforward adapter merging without re-training or complex heuristics.\n3) It offers a simple hyperparameter-light pipeline (single mask-calibration step) that scales smoothly to hundreds of tasks across diverse domains.", "reasons_to_reject": "1) The authors mainly repackage prior PEFT ideas via frozen random A (as in VeRA/O-LoRA) and sparsified B (as in AdaLoRA) rather than introducing a fundamentally new algorithmic insight.\n2) The adapter’s updates are unstructured-sparse, which saves parameters on paper but often slows training and inference in practice because today’s GPU libraries are tuned for dense (or very structured-sparse) matrix multiplies.", "questions_to_authors": "1) Please clarify why the analysis doesn't include strong modern PEFT methods such as QLoRA, AdaLoRA or HyperLoRA?\n2) In Table 1, the authors give LoRA almost twice the trainable parameters of LoRI-D (84 M vs 44 M), yet the average NLU score is separated by just 0.2 points (87.1 vs 87.3). With a fairer test, e.g. LoRA at rank 16 to match LoRI-D’s budget, there's every chance LoRA would equal or surpass it, but that experiment is missing.\n3) For tasks that require a richer, learned subspace or very low-rank settings, LoRI can hit a ceiling that LoRA does not have because LoRA is allowed to tune both A and B.", "extra_scores": {}}, {"review_id": "BfgSsBfzPh", "reviewer": "Reviewer_cZht", "summary": "This paper proposes LoRI, a method that enhances parameter-efficient fine-tuning by freezing the projection matrices A and introducing sparsity into the matrices B. To mitigate cross-task interference during adapter merging, LoRI exploits the orthogonality of adapter subspaces. Additionally, it supports continual learning by leveraging sparsity to reduce catastrophic forgetting. Experimental results across a variety of tasks demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The study of methods to prevent parameter interference during LLM adaptation is an important topic and could benefit the community.\n\n- The paper is generally easy to follow.", "reasons_to_reject": "- The ideas presented in the paper are conceptually similar to existing work.\n\n- Related works are not in the main body of the paper, but they are primarily included in the appendix.", "questions_to_authors": "- Before adapter merging and continual learning can be performed, the task-specific masks must first be obtained, which might introduce additional training overhead?\n\n- Are there any discernible patterns in the structure or distribution of the sparse masks? Do they exhibit significant overlap across different tasks, or are they largely distinct?", "extra_scores": {}}, {"review_id": "VPaN49C4Xf", "reviewer": "Reviewer_5RGh", "summary": "The paper introduces Low-Rank Adaptation with Reduced Interference (LoRI) a parameter-efficient training method suited for training multiple tasks and merging the resulting parameters for a combined inference. LoRI follows the architecture of LoRA with two low-rank matrices A and B, but instead of training both, LoRI keeps A freezer (randomly initialized) and imposes sparsity to B. The work further proposes combining the per-task trained delta parameters through concatenation and scaler weight learning. the authors show that this feature can also be leveraged for preserving the effectively of safety adapter in the inference.\n\nThe method is compared against the common baseline LoRA methods as well as merging methods, showing an overall good performance with much less trainable parameters.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and clear.\n- The method provides a practical solution for learning much less parameters (in comparison with standard LoRA methods) for a specific task.\n- The", "reasons_to_reject": "My main concern is the novelty of the method especially when considering the matureness and breadth of the studies on LoRA and PEFT. I believe the paper must provide a better analysis of the existing methods and shows - both in terms of method and experimental results - how the introduced approach different/better/preferred in comparison with other methods. The work in compared solely with \"basic\" baselines and the Related Work (which is appeared in Appendix) lacks relevant studies.\n\nRelated to the parameter-efficiency and simplicity in multi-task learning, this paper comes in my mind:\nFrohmann, Markus, et al. \"ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale.\" Findings of the Association for Computational Linguistics ACL 2024. 2024.\n\nThis work also seems relevant regarding safety preservation and bias mitigation:\nKumar, Deepak, et al. \"Parameter-efficient Modularised Bias Mitigation via AdapterFusion.\" Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 2023.\n\nI strongly encourage a deeper study in relevant studies and adding further baselines.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ue4qLIDED3", "reviewer": "Reviewer_nvBE", "summary": "This paper proposed LoRA with Reduced Interference (LoRI), a simple yet effective approach that freezes the projection matrices A as random projections and sparsifies the matrices B using task specific masks. I like this kinds of work which can be benefit for many downstream tasks.\n\nSuggestions:\n\nHuggingface has its public source code for load multiple adapters (https://huggingface.co/docs/peft/main/developer_guides/lora#load-adapters). Do you think it will be a good way to compare and what will be the advantages of your methods like adapter merging in Figure 1 (b) in this regards?\nCan your method been applied to vision language model scenario? Since currently all tasks are been evaluating on text-based tasks.\nThere are some existing methods working on optimizing LoRA such as [1] and [2]. Do you think those methods can be considered as a baseline method or related work in your future work?\nIn the implementation details, the paper mentioned that the experiments are done using 8 Nvidia A5000 GPUs. My question is since this is an lora training. I am more interested in what will be the performance if model can be trained using a single A5000 GPU. In many real-world cases, the main reason we used lora is we did not have many GPUs so it will be helpful if the authors can provide an additional experiment to show if the performance will not degrade when we use less amount of GPUs.\n\n[1] CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models, https://arxiv.org/abs/2409.02119\n\n[2] LoRA-Pro: Are Low-Rank Adapters Properly Optimized? https://arxiv.org/abs/2407.18242", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper has performed a large scale experiments which arcoss multiple tasks and demonstrated the effectiveness of the proposed method.", "reasons_to_reject": "The github repo is not public and I believe it will be much help if the paper been accepted and public its code.\nReferences:", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers mostly believe the paper is worthy of publication at the conference. There were some additional experiments by the authors which strengthened their work and the response is satisfying.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept