Gated Integration of Low-Rank Adaptation for Continual Learning  of Large Language Models
Authors: Yan-Shuo Liang
Jiarui Chen
Wu-Jun Li
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Continual Learning
low-rank adaptation
Primary area: deep_learning
TL;DR: We propose a novel continual learning method for language models, built upon the low-rank adaptation framework.

Abstract: Continual learning (CL), which requires the model to learn multiple tasks sequentially, is crucial for large language models (LLMs). Recently, low-rank adaptation (LoRA), one of the most representative parameter-efficient fine-tuning (PEFT) methods, has gained increasing attention in CL of LLMs. However, most existing CL methods based on LoRA typically expand a new LoRA branch to learn each new task and force the new and old LoRA branches to influence old tasks equally, potentially leading to forgetting. In this work, we propose a new method, called gated integration of low-rank adaptation (GainLoRA), for CL of LLMs. GainLoRA expands a new LoRA branch for each new task and introduces gating modules to integrate the new and old LoRA branches. Furthermore, GainLoRA leverages the new gating module to minimize the influence from the new LoRA branch to old tasks, effectively mitigating forgetting and improving the model's overall performance. Experimental results on CL benchmarks demonstrate that GainLoRA outperforms existing state-of-the-art methods.

Gated Integration of Low-Rank Adaptation for Continual Learning of Large Language Models
Yan-Shuo Liang, Jia-Rui Chen and Wu-Jun Li" National Key Laboratory for Novel Software Technology, School of Computer Science, Nanjing University, R. China (liangys, ( www.nnjuuuuuuu...e.. edu. en, liwujunEnju edu.
Abstract
Continual learning (CL), which requires the model to learn multiple tasks sequen- tially, crucial for large language models (LLMs). Recently, low-rank adapta- tion (LoRA), one of the most representative parameter-efficient fine-tuning (PEFT) methods, has gained increasing attention in CL of LLMs. However, most exist- ing CL methods based on LoRA typically expand new LoRA branch to learn each new task and force the new and old LoRA branches to influence old tasks equally, potentially leading to forgetting. In this work, propose called gated integration of low-rank adaptation (GainLoRA). for CL of LLMs. LoRA branch for each task and introduces gating modules to integrate the new and old LoRA branches. Furthermore, GainL.RA leverages the new gating module to minimize the influence from the new LoRA branch to old tasks, effectively mitigating forgetting and improving the model's overall performance. Experimental results CL benchmarks demonstrate that GainLoRA outperforms existing oftte-of-the-art methods. Code is available at http///hhhho.cao/aaaaa////////// gainoora
method,
GainLoRA expands
Introduction
Continual learning (CL), which requires the model to learn multiple tasks sequentially, is crucial for large language models (LLMMs) [88. Specifically, although existing L.MSs have demonstrated strong performance for wide range of tasks [4] g (53 (4 [3 with extensive pre-trained know ledge and further fine-tuning strategies, they may lose knowledge acquired from old tasks when learning multiple tasks sequentially. This phenomenon, known as catastrophic forgetting [36] 59 61]. highlights the need for developing effective CL methods for LLMs. Existing CL methods can be categorized into two main categories. The first category 45] assumes that task identities are available
during inference, while the second category [32 75 tackles more difficult and practical setting where task identities are unavailable during inference.
Recently, low-rank adaptation (LoRA) [21], of the most representative parameter-efficient fine- tuning (PEFT) methods, has gained increasing attention the CL of LLMS K 611 Specifically, by reparameterizing pre-trained weights in low-rank form, LoRA updates only limited number of parameters to adapt I.LMs to a downstream task, making the fine-tuning process much more efficient than updating all parameters of LLMs [17] This efficiency also benefits CL, making LoRA increasingly popular in CL of LLMs.
typically expand new LoRA branch for learning each new task while freezing all old LoRA branches. In this way, they avoid forgetting caused by directly updating the LoRA parameters of old tasks. However, to handle the practical CL scenario
Most existing CL methods based LoRA 32 75
Wuu-Jun Li is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

where task identities are unavailable at inference time, existing methods [32 50 61 based on LoRA integrate new and old LoRA branches through a simple addition. Consequently, they force the new and old LoRA branches to influence old tasks equally, which means that the new LoRA branch may cause relatively large change the model's ss output on old tasks. This leads to forgetting and degrades the model's 's overall performance in CL..
In this work, we propose a new method, called gated integration of low-rank adaptation (GainLoRA), for CL of LLMs. The contributions of GainLoRA are listed as follows:
GainLoRA expands new LoRA branch to learn each new task and introduces gating modules to integrate the new and old LoRA branches.
GainLoRA leverages the new gating module to minimize the influence from the new LoRA branch to old tasks, effectively mitigating forgetting and improving the model's overall performance.
Experimental results on CL benchmarks show that GainLoRA outperforms existing state-of- the-art CL methods.
2 Related Work and Preliminaries
2.1 Related Work
Ceiamntter-fffciiee Fine- Fine-n Parameter-efficient fine-tuning (PEFT) methods tune limited number of parameters adapt a pre-trained model for downstream tasks, showing much higher effi- ciency than tuning all the parameters of the pre-trained model, especially for LLMS (72.. For example, Adapter (20 modifies the model architecture by introducing trainable modules into Transformer lay- ers and tunes these modules for downstream tasks. Prompt-tuning [26] and Prefix-tuning (27 insert learnable tokens into the input and tune them for downstream tasks. Low-rank adaptation (LoRA) [21] reparameterizes the original model parameters with low-rank matrices and tunes these matrices for downstream tasks. Although tuning significantly fewer parameters than full fine-tuning, PEFT achieve comparable performance to full fine-tuning across wide range of natural language processing (NLP) tasks [16, 21 37 70].
Continual Learning There are three main types of CL methods, categorized as regularization based methods, memory-based methods, and expansion- based methods. Regularization-bass based meth- methods incorporate regularization term to mitigate catastrophic forgetting. Memory-based
ods 23 24 29
75 utilize memory mechanisms to preserve knowledge from old tasks. 30 46) mitigate catastrophic forgetting by introducing new parameters for learning new tasks while typically freezing old parameters.
Expansion- methods [22 28
Many CL methods II
30 are designed to train models from scratch. Recent studies 32 45
56 61 67 have shown that leveraging PEFT strategies to fine-tune pre-trained models enables CL methods to achieve superior performance across tasks. For example, some methods [42 45
67 75
use prompt-tuning for continual learning. They either maintain independent prompts for each task or maintain prompt pool and perform query-ke matching to learn new tasks. There are also many methods 32
adopting LoRA for continual learning. Most of these methods expand a new LoRA branch to handle each new task while freezing old LoRA branches to mitigate catastrophic forgetting. However, they force the new and old LoRA branches to influence old tasks equally, potentially leading to forgetting.
49
61 75]
2.2 Preliminaries
Problem Definition We follow existing CL works 75) to formalize the problem definition for CL of LLMs. Specifically, in CL, sequence of tasks {Ti T2, ..., Tr is presented to the model sequentially, where T denotes the total number of tasks. The f-th task Tz consists of a training dataset D, For any given sample (æ,, y1) D,, x, denotes input sentence and y, denotes the corresponding output. When learning the t-th new task, the model is required to mitigate catastrophic forgetting of the / previously learned tasks.
Similar to existing CL. works for L.MSs 75), we consider more challenging CL setting with three key challenges: (1) the model is presented with sequence of tasks spanning various types, such

---

+ Addition
Multipication
Frozen
Intaliaaioon constraints
Usdacing constrants
Imposing
Language Models
Integrating old
Expardinn LofA branch
3. Updwtiing new branch learn
B,
Sonieone who hod fouhflght might optan diaa S0praptiion Krssusable Ellans Eboutiiinn
Some same who hed hut right reight Ourien vo abed Ripes oas. Compuratie iStone Shutfoiee
x
Old Branches
New Branch
(a) Expandable LoRA Architecture in GainLoRA
(b) Gating Modules GainLoRA
Figure 1: (a) shows the expandable LoRA architecture of our GainLoRA for learning the t-th new task. (b) shows that for each task T, GainLoRA uses an independent gating module 9:(-) to generate integration coefficient a,.
as dialogue generation, information extraction and so on; (2) the model is not provided with task identities at inference time; (3) the model must learn without access to real or synthetic samples from previously learned tasks.
Low-Rank Adaptation LoRA [1 is a widely adopted PEFT method used for fine-tuning various pre-trained models, particularly LLMs. Specifically, let W E IRdoxt represent pre-trained weight in LLMs, where d,n and dout are the input and output dimensions, respectively. LoRA introduces an additional branch consisting of matrices, A iR x and B E grxdd,, where r < main(din dout). LoRA then modifies the forward propagation of this layer as
( + AB)h.
Here, and e denote the input and output, respectively. is initialized to O, and B is initialized with Gaussian distribution. During fine-tuning for downstream tasks, the pre-trained weight W remains frozen, and only the parameters A and B are fine-tuned.
3 Methodology
Our GainLoRA employs an expandable LoRA architecture, which is illustrated in Figure (a). Specifically, before learning the t-th task (1 < < 7), GainLoRA first expands the LoRA architecture by introducing the t-th new branch with matrices A, Xd and B, E grxd:.. The new and old LoRA branches are then integrated as
W, = Wi a. A, = Σ a. A.A.
(I)
where a; is integration coefficient that determines the influence of the i-th LoRA branch to the input h. Note that W is zero matrix when 1. As a result, the forward propagation this layer is modified
e= (W + W.)h.
(2)
Finally, only the new LoRA branch (i.e. the t-th LoRA branch) is updated for the t-th new task, while all the old LoRA branches are frozen. After learning the f-th task, (2) is also used for inference across all test samples, thereby ensuring compatibility with the scenario where task identities are unavailable during inference.
share similar architecture to our method, as illustrated in Figure (a). However, these methods fix all coefficients (a:)== II) to 1, forcing the new and old LoRA branches influence old tasks equally. As result, the new LoRA branch introduces change of B, to output for inputs h associated with old tasks, potentially leading to forgetting. Although some methods attempt to mitigate this forgetting by
Many existing CL methods based on LoRA 32
50
61 75

---

imposing regularization [50] or orthogonality constraints [32 on the new LoRA branch, the fixed integration coefficients {a, J still limit their performance, as demonstrated by the experimental results presented in Section 4 The method in [75 does not force the new and old LoRA branches to influence old tasks equally but relies on replaying synthetic old samples to mitigate forgetting.
making it unsuitable for the scenario considered in this work.
Different from existing methods, GainLoRA introduces an independent gating module gi(-) for each task T. to generate the integration coefficients (1 < i S T". To mitigate the forgetting caused by the new task, GainLoRA leverages the gating module minimize the influence from the new LoRA branch to the old tasks. The details will be introduced in the following subsections.
3.1 Architecture of Gating Modules
As illustrated in Figure 1 (b), given an input sample æ, the gating module g:(+) generates the integration coefficient for the i-th LoRA branch, denoted as a; = g)()). The computation of gi(-) is defined as
Pto = Po = (oke(n)))
Pit o (G.,,P...)), le (1,
((()) = f(G,,++PP,,))
(3)
Here, Token(-) represents the tokenizer used in LLMs to extract token embeddings from the input Pool(-) denotes an average pooling operation applied to the token embeddings to produce a fixed-size vector. (-) denotes the non-linear activation function. G. denotes the weight matrix for the 1-th layer of (:(-) (1 < L < L + 1). In the final layer, G.L++ is a vector that maps the input vector P., scalar. Following existing works with gating mechanisms 7 [9], the function f(-) is designed to map scalar to value within [0, 1J. that is, ʃ(-) IR- + [0, 1].
Note that the imput to gating modules is the same that of LLMs, denoted æ, which differs from the input to LoRA in a specific layer, denoted as h. During the learning of the t-th new task, only the new gating module (+).) is updated, while all the old gating modules (+)())] remain frozen.
3.2 Minimizing the Influence from the New LoRA Branch to Old Tasks
GainLoRA minimizes the influence from the new LoRA branch to old tasks by making a, = 9 (x) as close to 0 as possible for any input æ from old tasks {T.) However, since we focus on the scenario where no real or synthetic samples from old tasks are accessible. directly optimizing gr (x) to 0 is impractical. To overcome this challenge, GainLoRA imposes constraints on the new gating module ge (-), implicitly guiding 9r (x) to close 0 and reduce the influence of the new LoRA branch to old tasks.
In the following two subsections, we first describe the constraints imposed on the new gating module ge(-) and explain how these constraints guide ge (ar) close to 0 for any x from the old tasks. Then, we detail the implementation of these constraints during training.
3.2.1 Constraints New Gating Module
To formalize the constraints imposed on the new gating module 9x(+), we define the subspace spanned by the inputs to Gt (1 S < T. + 1) from the previous f. tasks as:
Me = span Pt,-1 is defined in 3 (x, y) E D.).
(4)
L+1 cannot be obtained directly due to the unavailability of samples from old tasks. However, by introducing additional constraints, [M, I++ can be solved iteratively, which will be discussed in Section 3.2.2
Note that subspaces [M
Initialization Constraints Before learning the 1-th task, the following constraints imposed on the initialization of the new gating module (.):)):
f(0) 0,
(5)
where Init (+1)+1) denotes the initialization of G.,L+1. These constraints ensure that for any sample from the old tasks, the integration coefficient satisfies
a, = (x)
= 0.
(6)

---

where Pt,L is defined in (3) The second equality holds since Gr,L+1 = Init (+,)+)) before learning
the t-th new task. The third equality holds because f(0) 0 and Pt, E M,,++1 for any æ from previous 1 tasks.
Updating Constraints During the learning of the i-th task, the following constraints are imposed on the updates to the new gating module ge(-):
L., for 1
where AG denotes the update to G..-- Based existing studies [60,
30), the constraints in
ensure that Sr (x) remains unchanged for inputs æ from the old tasks during the learning of the f-th task. Formally, the following proposition holds:
Proposition 3.1. If the constraints in 7 are satisfied, subspaces {Mai Lbtl remain unchanged during the learning of the t-th task. Furthermore, for any input x from the previous t tasks, gt (a)
remains unchanged during the learning of the t-th task.
The proof of this proposition provided in Appendix AA.3) Since the initialization constraints in (5)
ensure St (a) 0 before learning the (-th new task, (x) = 0 is preserved throughout the learning process if the updating constraints
7 are satisfied.
The fact that subspaces (f remain unchanged, as stated in Proposition 3.1 is essential for implementing the orthogonal constraints in (7). Specifically, as will be detailed in Section 3.2.2
orthonormal bases for the subspaces )Mr,))) [-1 are learned to enforce the orthogonal constraints L+1
in 5) and 7). Since the subspaces {Me,r /=1 remain unchanged during the learning of the 1-th task, their orthonormal bases also remain unchanged, allowing them to be pre-computed before learning the f-th task, thus facilitating the implementation of orthogonal constraints in (5) and (7) throughout the learning process.
3.2.2 Implementation of Constraints
There exist many functions f(-) R + [0, 1] satisfying f(0) = 0. In this work, we define f(-) as s(b) = |2 sigmoid(() 1/.
(8)
where sigmoid(-) denotes the sigmoid function. Other functions f(-) IR -> [0. that satisfy [(0) = are also applicable, and experiments with different choices of f()) are provided in Appendix C Better model performance be expected by designing more effective f(-), but this is not the focus of this paper.
Implementing the orthogonal constraints in and is challenging due to the lack of samples from 1++1 To address this issue, we further
previous / - 1 tasks to approximate the subspaces [M impose the following constraints on the initialization G.,, (I
Init(G,,) + G.-1.--
This strategy initializes the first L layers of ge(-) using the corresponding layers from the previous gating module (--1(-). As result, the first layers of gc(-) can be viewed as being initialized and starting their training at the beginning of the first task, continuing until the t-th task. Simultancously, the first I layers in g: serve as checkpoints, preserving the state of 9c(-) after learning the i-th task (1 S i t). At this time, we can use existing method gradient projection memory (GPM) [47 to iteratively learn a set of matrices (M,)) L++1 where the columns of M.. contribute to set of orthonormal bases of subspace Me, Details of GPM are provided Appendix (A.I Then, before
leamning the t-th task, the following operation can be performed on Init (t,
Init(Gr,Lt1) + Init(Gr.L41) - MM, M Init(Gr.,44))
(10)
At this time, we have
M M (Init(Gr,I++)) M Init(,,,,
-M
=(I TM, Mf,++1M,,++ M,,+,
(11)
Since the columns of Mt form an orthonormal basis, we have M M = I, which means I - M Mr,
-MT
O. Therefore, the equation in [1] is equal to zero matrix O. Note
5

---

Algorithm GainLoRA for Continual Learning
Input:
The data of different tasks (Di) Output: Learned LoRA parameters {(A,, B.) and gating modules (9:(-)) for t 1: do
Expand the (-th LoRA branch with A, and B;; Impose initialization constraints on the new gating module ge(-) by and 10 Integrate and old LoRA branches by !
for B, S Dr do
Compute the loss in (3
Perform backward propagation to compute the update of the new LoRA branch and the new gating module;
Impose updating constraints on the update of the new gating module by
end for end for
that Mr,4+1 is spanned by the columns of Mt,41 Init Init(Gr,L+1) satisfies the constraints in 5 after the operation in (10).
Similarly, during the learning of the f-th task, the following operation can be performed on {AGa)h=
14+1,
AG AG M M
(12)
With the same proving process in 11I (AGr,) L+1 to satisfy the constraints in
can show that the update in 12 allows the update
3.3 Updating the New LoRA Branch
Our GainLoRA aims to effectively integrate new and old LoRA branches while mitigating forgetting caused by the new LoRA branch on old tasks. Since GainLoRA does not impose specific update
strategies for the new LoRA branch, is inherently compatible with various existing CL methods that adopt similar LoRA architecture as our method and can update the new LoRA branch [32] [50] 61].
Since these existing methods fix all integration coefficients {a,) to 1, combining our method with these existing methods can enhance their performance, as demonstrated in Section 4
3.4 Whole Process of GainLoRA
Algorithm outlines the whole process of our GainLoRA. Before learning the 1-th new task T. GainLoRA first expands the LoRA architecture by introducing the t-th new branch with matrices A, and B, Simultameously, new gating module go( (-) is initialized through the operations specified in (8). (IO) and (9) to ensure that the initialization constraints in (5) are satisfied. The new and old
LoRA branches are then integrated using (T) and the forward propagation is modified as (2)
During the learning of the t-th task Te with the corresponding dataset D,, our method follows existing methods 61 75 and computes the loss for the new task through
L.
1
Σ
Σ
log [P( Vt.1..... v-l-
(13)
Dee
(DD
where y/: [t, Jt. Ve.. ]- Each time, GainL samples a mini-batch B, to minimize
the loss in 13) by updating the new LoRA branch and the new gating module g-(-). During this process, the projections defined in (1) are applied to the parameters of g, (-), ensuring that the update constraints are satisfied.
GainLoRA introduces a new gating module for each new task, which incurs additional parameters and computational cost when combined with other methods. Section will demonstrate that the trainable parameters added by GainLoRRA are limited, making the number of trainable parameters in GainLoRA A comparable other methods. Additionally, Appendix C.T and C.2 will demonstrate that the computational cost introduced by GainLoRA is minimal compared to the original LLMs.

---

Experiments
4.1 Experimental Settings
we evaluate different methods on SuperNI and Long Sequence (45) benchmarks. SuperNI benchmark includes various types of NLPP tasks, including dialogue generation, information extraction, question answering. summarization, and sentiment analysis. Following the protocols of existing method (75, three tasks are selected from each type, resulting in 15 tasks, These tasks are arranged into two different task sequences with different orders, referred to Order and Order 2. Long Sequence benchmark consists of 15 diverse classification tasks, which are similarly arranged into two task sequences with different orders, referred to as Order 3 and Order More details about the benchmarks and task sequences are provided in Appendix B
Datasets Following existing CL methods 45 61 75
64
Evaluation Metric We use AJ.: to denote the model's performance the i-th task once the model learns the j-th task. Specifically, AJ.s represents accuracy for classification tasks and Rouge-L. [33 for other types of tasks. Following traditional CL works [5] () employ average performance (AP) and forgetting (FT) to evaluate the model's performance. The formulas for these two metrics are defined as
T AP
E AT&
FT FT FT-T-E
(1...... AT,,),
(14)
T
where T denotes the total number of tasks in the task sequence. AP evaluates the model's final performance, and FT' quantifies the forgetting.
Baselines We compare our method with ofate-of-the-art CL methods, including LFPT5 (2), EPI 65
MIGU M. EWC [24), TASL [13. KIFLORA [4]. IncLoRA [6], C-LoRA (50, O-LoRA [1]. and InfLoRA B22 Additionally, we introduce simple baseline called SeqLoRA, which does not expand new LoRA branches but sequentially updates old LoRA parameters for new tasks and lacks mechanism to mitigate forgetting. Note that many CL methods based on pre-traime trained models in CV focus on classification tasks, relying either on carefully designed classifiers 38 7 741 or the [CLS]
token in VIT 25 51 57 58
66 67]. In contrast, we follow existing works in NLP 61 75 and
adopt next-token prediction to handle both classification and generation tasks, where models 43 54
lack [CLS] token. Consequently, these CV methods incompatible with our setting and cannot be directly compared. For completeness, we adapt some of them to our setup and report results in Appendix C.7 C.7
68], all methods are implemented To ensure fair comparisons, for all the methods based on LoRA, we follow existing CL methods [21] 75] by incorporating the LoRA architecture into the query and value components of the multi-head attention mechanism in each Transformer block. Similar to the existing CL methods for LLMS 61 75), we use T5 (43), Llama-2 [54] and Llama-3 [12] the base architectures. Each experiment is repeated three times with different seeds, and the average result is reported. More details, such as the learning rate, batch size, and architecture of the gating modules GainLoRA, are provided in Appendix Bn2 and Appendix B.3
Implementation Details Following existing CL works 40
61
with instruction tuning [40] and optimized using AdamW
35
4.2 Experimental Results
Compare with Existing Methods We first follow existing works 11] 75 and evaluate different CL methods using T5-Large. Since our method does not impose specific update strategies for the new LoRA branch, we adopt the same update strategies as the two state-of-the-an methods, O-LoRA 611 and IniLoRA [2) Note that these two methods leverage LoRA architecture similar to our method but fix all integration coefficients )T to 1. Details of these two methods are provided in Appendix A.2 We use GainLoRA (O-LoRA) and GainLoRA (InfLoRA) to respectively denote methods adopting O-LoRA and InfLoRA to update the new LoRA branch. GainLoRA is also compatible with other methods that leverage expandable LoRA architecture shown in Figure (a), and we give some results in Appendix C.5
The results are shown in Table T As we can see, our methods GainL (O-LoRA) and Gain- LoRA (InfLoRA) outperform O-LoRA and InfLoRA in both AP and FT, respectively. This im-
7

---

Table 1: Results different task sequences with T5-large model. Results of methods with copied from existing paper [75].
Order
Order
Order AP1
Order 4
Method
APt
FT
APt
FT
FT4
AP1
FT4
LFPT5 42
39.03
10.87
29.70
20.72
66.62
14.57
67.40
13.20
EPI 65
75.19
0.77
75.10
2.44
MIGU+FT
71.30
11.39
69.05
14.06
EWC
15.32
26.78
18.19
30.28
43.24
23.66
46.25
32.90
TaSL. 15
27.51
18.53
28.05
17.39
71.37
6.20
73.11
6.52
KIFLoRA
14
28.33
16.44
30.31
16.27
72.19
3.10
73.72
4.75
SeqLoRA
7.30
47.60
7.03
47.97
49.46
27.60
33.81
45.53
IncLoRA 661
12.33
41.93
16.65
36.56
61.19
13.63
62.46
15.92
C-LoRA
50
22.69
24.25
32.81
11.60
66.83
8.64
61.86
14.18
O-LoRA
61
26.37
19.15
32.83
11.99
70.98
3.69
71.21
4.03
GainLoRA (O-LoRA)
47.84
2.26
46.84
2.91
73.37
3.02
76.01
2.49
InfLoRA
32
39.78
7.64
39.57
8.93
75.15
4.19
75.79
3,47
GainLoRA (InfLoRA)
46.21
2.40
46.44
2.61
78.01
0.77
77.54
1.25
O-LORA
GainLoRA (O-1oRA)
ORA)
Averaged Performance
Averaged Performance
Averaged Performance
Averaged Performance
Task (a) Order 1
Task (b) Order 2
Task ID (c) Order
Task ID
(d) Order
Figure 2: The variation of performance across different CL methods during training on different task sequences.
provement demonstrates that fixing all coefficients {a to leads to forgetting old tasks, thereby limiting the performance of O-LoRA and InfLoRA. By effectively mitigating this forgetting. GainLoRA (O-LoRA) and GainLoRA (InfLoRA) achieve superior performance. Furthermore, our methods consistently achieve the best performance across all task sequences.
Figure 2 illustrates the variation in the average performance across all learned tasks for different methods throughout the CL process. As shown, GainLoRA consistently outperforms the performance of O-LoRA and InfLoRA throughout the whole training process.
Scaling to Larger Model Architectures To evaluate the effectiveness of our method on larger model architectures, we scale different LoRA-baseed CL methods to larger models, including T5-XL, 1-ama-2-7B. Llama-2-113 13B and Llama-3-88 Table 2 and Table 3 present the results of different methods. As shown, across models of varying sizes, GainLoRA (O-LoRA) and GainLoRA (InfLoRA) consistently outperform O-LoRA and InfLoRA in terms of AP and FT, respectively. This demonstrates that GainLoRA effectively mitigates forgetting in the new LoRA branch across different model architectures.
Trainable Parameters We compare the number of trainable parameters across different methods for training different task sequences. The results for T5, Llama-2 and Llama-3 are shown in Figure 3 and the detailed computation trainable parameters is provided Appendix [.4] As shown, GainLoRA (O-LoRA) and GainLoRA (InfLoRA) have more trainable parameters than O-LoRA and InfLoRA, respectively. This increase arises from the introduction of the trainable gating module in GainLoRA. However, the additional trainable parameters introduced by GainLoRA much fewer than those in LoRA. Therefore, the total number of trainable parameters in GainLoRA (O-LoRA) and GainLoRA (InfLoRA) comparable that of O-LoRA and InfLoRA, respectively.
Distribution of Outputs New Gating Module To demonstrate that our GainL.RR effectively minimizes the influence from the new LoRA branches to old tasks, analyze the output distributions

---

Table 2: The overall results different task sequences with T3-XL model.
Order APT
Order 2
Order 3
Order
Method
FT
APT
FT
APT
FT
APt
FT
O-LoRA 61
36.50
11.42
40.64
6.37
73.77
2.70
76.19
3.56
GainLoRA (O-LoRA)
50.10
3.21
49.86
3.04
78.41
2.59
77.21
3.30
InfLoRA 32
45.61
5.60
45.85
5.10
80.22
2.09
79.43
1.71
GainLoRA (InfLoRA)
50.06
1.86
50.26
2.64
81.22
0.58
80.30
0.75
Table 3: The overall results on different task sequences with Llama-2-7B. Llama-2-13B and Llama-3-
8B.
Order
Order
Models
Methods
APt
FT
APt
FT
O-LoRA [1]
39.37
15.84
37.55
20.23
GainLoRA (O-LoRA)
51.10
4.96
51.14
5.57
Llama-2-7B
InfL.RA 32
42.93
11,23
39.94
15.00
GainLoRA (InfLoRA)
51.27
2.84
50.17
4.71
O-LoRA 61
43.92
14.15
40.05
19.53
GainLoRA (O-LoRA)
52.47
4.78
51.68
5.86
Llama-2-13B
InfLoRA 32
43.64
14.85
45.74
10.61
GainLoRA (InfLoRA)
53.64
2.87
52.46
4.90
O-LoRA [6]
42.49
8.85
38.67
19.28
GainLoRA (O-LoRA)
53.39
3.56
51.69
6.20
Llama-3-8B
InfLoRA 32
43.27
6.02
48.77
5.88
Gainl.oRA (InfLoRA)
52.18
1.40
52.48
4.21
OLRA GEUUAAOAAAA
IELARA
Trainale Paramnteess (M)
Trainable Paramndeen
Trainable
XL
36
138
Model Size
Model Size (b) Llama-2
Model Size
(a) TS
(3) Llama-3
Figure 3: (a), (b) and (c) show the number of trainable parameters for different CL methods and model backbones under task sequences Order and Order 2.
of the new gating modules. Specifically, after training on the final task (i.e., the 15-th task) in the task sequences, the 15-th task corresponds to the new task, and its associated gating module 91s(-) serves as the new gating module.
We obtain the outputs of the new gating module (-5(-) on the samples from old and new tasks, respectively. Then, we analyze their distributions Figure As shown, the outputs gis(.) for the samples from old tasks are concentrated around 0, effectively minimizing the influence from the new LoRA branch to old tasks. Furthermore, GainLoRA does not constrain the outputs of g1s(.) (-) for the samples from the new task. As a result, the outputs of 915(.) for the samples from the new task distributed near enabling the model to effectively learn the new task.
Ablation Study To verify the necessity of both the initialization and updating constraints introduced in Section 3.2.1 we define several variants of GainLoRA. The first variant, referred "No Initialization Constraints". removes the initialization constraints defined in (5). Specifically, it replaces π(.) defined in g) with function sigmoid()) and eliminates the operation (10) while keeping other components unchanged. The second variant, referred to as "No Updating Constraints" removes the updating constraints defined in (7) by eliminating the operations in (12) while preserving all other components of GainLoRA. The third variant, referred to as "No Constraints" follows "No

---

Ousput of New Gating Module
Quiputs New Caating Module
03
DM Tade
OWTTaix New
Order
Order
Order
(a) GaA (O-LoRA)
(b) GainLoRRA (InAL)RR)
(c) GainLoRA (O-LoRA)
(d) Gainl .RRA (Inf).RRA)
Figure (a) and (b) show outputs of new gating module in our GainLoRA on different task sequences with T5-Large. (c) and (d) show outputs of new gating module in our GainLoRA on different task sequences with Llama-2-7B.
Table 4: Ablation study GoinLoRA with T3-Large and Llama-2-7B.
T5- L
Llama-2-7B
Method
Order
Order
Order
Order
APt
FT
AP1
FT
AP1
FT
AP1
FTY
Gainl LoRA (O-LoRA)
47.84
2.26
46.84
2.91
51.10
4.96
51.14
5.57
No Initialization Constraints
35.30
17.19
39.82
12.90
44.02
11.71
42.89
14.77
No Updating Constraints
23.01
30.32
24.96
28.14
33.74
23.06
34.71
22.36
No Constraints
26.32
26.00
30.63
22.37
34.48
23.46
36.87
21.24
GainLoRA (InfLoRA)
46.21
2.40
46.44
2.61
51.27
2.84
50.17
4.71
No Initialization Constraints
45.38
3.40
43.05
5.15
50.48
3.48
48.17
6.45
No Updating Constraints
37.69
10.94
38,85
9.31
48.52
5.68
47.85
7.00
No Constraints
36.75
12.18
41.00
6.66
49.10
6.07
45.77
8.70
Initialization Constraints" and "No Updating Constraints" remove both the initialization and updating constraints. Table presents the experimental results of these variants. As shown, none of these variants perform as well as our GainLoRA, indicating the critical role of both the initialization constraints and updating constraints in our GainLoRA.
5 Conclusion
In this work, propose new method, called GainLoRA, for CL of language models. GainLoRA expands a new LoRA branch for each new task and introduces gating modules to integrate the new and old LoRA branches. Furthermore, GainLoRA leverages the new gating module minimize the influence the new LoRA branch to old tasks, effectively mitigating forgetting and improving the model's overall performance. Experimental results on CL benchmarks demonstrate that GainLoRA outperforms existing state-of-the-art methods.
Limitations Similar many CL methods for LLMs [1] 61], method imposes constraints on the model to mitigate forgetting. While effective, these constraints may accumulate with increasing tasks, potentially hindering the learning of new tasks. Furthermore, consistent with existing works [24] 32 61]. our method primarily targets at catastrophic forgetting with non-overlapping tasks, and further complex scenarios, such scenarios where there
investigation is needed assess its effect
overlap between tasks [39 2)
6 Broader Impacts
Continual learning (CL) offers promising direction for improving the efficiency and scalability of language models, particularly in settings with continuously arriving tasks. By enabling incremental updates without retraining from scratch, it significantly reduces computational overhead and resource demands. However, CL often introduces additional components (e.g. memory or gating mechanisms), increasing complexity and requiring effort for maintenance deployment.
10

---

Acknowledgment
This work is supported by NSFC (No.62192783).
References
[1] Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general con- tinual learning method based on complementary learning system. In International Conference on Learning Representations, 2022.
[2] Jihwan Bang, Hyunseo Koh, Seulki Park, Hwanjun Song, Jung-Wor Woo Ha, and Jonghyun Choi. Online continual learning a contaminated data stream with blurry task boundaries. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9275-9284, 2022.
[3] PENG Bohao, Zhuotao Tian, Shu Liu, Ming-Chama Yang. and Jiaya Jia. Scalable language model with generalized continual learning. In International Conference Learning Represen- tations, 2024.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in Neural Information Processing Systems, pages 1877-1901, 2020.
[5] Arslan Chaudhry, Naeemullah Khan, Puneet Dokania, and Philip Torr. Continual learning in low-rank orthogonal subspaces. Advances in Neural Information Processing Systems, pages 9900-9911, 11 2020.
[6] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning, arXiv preprint 19Xiv:9902.14444 2019.
[7] Kyunghyun Cho. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv: 1409. 1259, 2014.
[8] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, pages 1-113, 2023.
[9] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, 2024.
[10] Cyprien de Masson d' Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems, pages 13122-13131, 2019.
[11] Wenyu Du, Shuang Cheng, Tongxu Luo, Zihan Qiu, Zeyu Huang, Ka Chun Cheung, Reynold Cheng, and Jie Fu. Unlocking continual learning abilities in language models. Findings of the Conference on Empirical Methods in Natural Language Processing, pages 6503-6522, 2024.
[12] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pages arXiv-2407, 2024.
[13] Stefan Elfwing. Fiji Uchibe, and Kenji Doya. Sigmoid-weighteds linear units for neural network function approximation in reinforcement learning. Newral Networks, pages 3-11, 2018.
[14] Yujie Feng, Xu Chu, Yongxin Xu, Zexin Lu, Bo Liu, Philip $ Yu, and Xiao-Ming Wu. Kif: Knowledge identification and fusion for language model continual learning. arXiv preprint arXiv:2408.05200 2024.
11

---

[15] Yujie Feng, Xu Chu, Yongxin Xu, Guangyuan Shi, Bo Liu, and Xiao-Ming Wu. Tasl: Continual dialog state tracking via task skill localization and consolidation. Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1266-1167777 1279, 2024.
[16] Chin-Lun Fu, Zih-Ching Chen, Yum-Ru Lee, and Hung- Yi Lee. Adapterbias: serameter-efficient token-dependent representation shift for adapters in nip tasks. Findings of the Conference of the North American Chapler of the Association for Computational Linguistics, pages 2608-26211 2022.
[17] Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
[18) Jinghan He, Haiyun Guo, Kuan Zhu, Zihan Zhao, Ming Tang, and Jinqiao Wang. Seekr: Selective attention attention-guidee guidedd knowledge retention for continual learning of large language models. In Proceedings of the Conference Empirical Methods in Natural Language Processing, pages 3254-3266, 2024.
[19] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation. pages 1735-1780, 1997.
[20] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In Proceedings of the International Conference on Machine Learning, pages 2790-2799, 2019.
[21] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[22] Steven C. Y. Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. In Advances in Neural Information Processing Systems, pages 13647-13657, 2019.
[23] Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node-importance based adaptive group sparse regularization. Advances in Neural Information Processing Systems, pages 3647-3658, 2020.
[24] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Strabska-Barwinss et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, pages 3521-3526. 2017.
[25] Minh Le. Huy Nguyen, Trang Nguyen, Trang Pham, Linh Ngo, Nhat Ho, et al. Mixture of experts meets prompt-based continual learning. Advances in Neural Information Processing Systems, pages 119025-119000 11962, 2024.
[26] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficice prompt tuning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2045-3059, 2021.
[27] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 4582-4597, 2021.
[28] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In Proceedings of the International Conference on Machine Leurning, pages 3925-3934, 2019.
[29] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, pages 2935-29002 2947, 2017.
[30] Yan-Shuo Liang and Wu-Jun Li. Adaptive plasticity improvement for continual learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2023.
12

---

[31] Yan-Shuo Liang and Wu-Jun Li. Loss decoupling for task-agnostic continual learning. In Advances Neural Information Processing Systems, 2023.
[32] Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 23638-23647, 2024.
[33] Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries, In Text Summarization Branches Out, pages 74-81, 2004.
[34] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pages 6470-6479, 2017.
[35] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019.
[36] Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning, arXiv preprint arXiv:2308.087 2023.
[37] Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems, pages 1022-1035, 2021.
[38] Mark D McDonnell, Dong Gong, Amin Parvanch, Ehsan Abbasnejad, and Anton Van den Hengel. Ranpac: Random projections and pre-trained models for continual learning, Advances in Neural Information Processing Systems, pages 12022-12053, 2023.
[39] Jun- Jun-Yeong Yeong Moon, Keon-Hee Park, Jung Uk Kim, and Gyeong-Moon Park. Online class incremental learning stochastic blurry task boundary via mask and visual prompt tuning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11731-11741, 2023.
[40] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, pages 27730-27744, 2022.
[41] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter, Continual lifelong learning with neural networks: A review. Neural Networks, pages 54-71, 2019.
[42] Chengwei Qin and Shafiq R. Joty. LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T5. In International Conference on Learning Representa- lions, 2022.
[43] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, pages 1-67, 2020.
[44] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the International Conference on Knowledge Discovery & Data Mining, pages 3505-3506, 2020.
[45] Anastasia Razdaibiedina, Yuning Mao. Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. In International Conference on Learning Representations, 2023.
[46] Andrei Rusu, Neil Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 1606 04671, 2016.
[47] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In International Conference on Learning Representations, 2021.
13

---

[48] Haizhou Shi. Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, and Hao Wang. Continual learning of large language models: A comprehensive survey. arXiv preprint arXiv:2404. 16789, 2024.
[49] James Seale Smith, Paola Cascante- Bonillaa, Assaf Arbelle, Donghyun Kim, Rameswar Panda, David Cox, Diyi Yang, Zsolt Kira, Rogerio Feris, and Leonid Karlinsky. Construct-vl: Data-free continual structured vl concepts learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14994-15004, 2023.
[50] James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin, Continual diffusion: Continual customization of text-to-image diffusion with c-lora. Transactions on Machine Learning Research, 2024.
[51] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Gascante-Bonilla Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-basel prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909-11919, 2023.
[52] Fan- Fan-Keng Sun, Cheng-Hao Ho, and Hung- Yi Lee. Lamol: Language modeling for lifelong language learning. In International Conference on Learning Representations, 2020.
[53] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
[54] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09280 2023.
[55] Lloyd N Trefethen and David Bau. Numerical Linear Algebra. SIAM, 2022.
[56] Huiyi Wang, Haodong Lu, Lina Yao, and Dong Gong. Self-expansion of pre-trained models with mixture of adapters for continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10087-10092 2025.
[57] Liyuan Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchi- cal decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. Advances Neural Information Processing Systems, pages 69054-69076, 2023.
[58] Liyuan Wang. Jingyi Xic, Xingxing Zhang, Hang Su, and Jun Zhu. Hide-pet: Continual learning via hierarchical decomposition of parameter-efficieet tuning. arXiv e-prints, 2024.
[59] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. comprehensive survey of continual learning: Theory. method and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 5362-5383 2024.
[60] Shipeng Wang, Xiaorong Li, Jian Sun, and Zongben Xu. Training networks in null space of feature covariance for continual learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 184-193, 2021.
[61] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang. Tao Gui, and Xuan-Jing Huang. Orthogonal subspace learning for language model continual learning. In Findings of the Conference on Empirical Methods in Natural Language Processing, pages 2023.
[62] Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, al. Trace: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.06762, 2023.
[63] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained trans- formers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, pages 5682-5695, 2022.
14

---

[64] Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabshi,, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super--natraalisstruttioss Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 5085-5109 2022.
[65] Zhicheng Wang, Yufang Liu, Tao Ji, Xiaoling Wang, Yuanbin Wu, Congcong Jiang, Ye Chao, Zhencong Han, Ling Wang. Xu Shao, et al. Rehearsal-free continual language learning via efficient parameter isolation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 10933-10946, 2023.
[66] Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal free continual learning. In Proceedings of the European Conference on Computer Vision, pages 631-648, 2022.
[67] Zifeng Wang. Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[68] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[69] X Ye. calflops: flops and params calculate tool for neural networks in pytorch framework, 2023.
[70] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1-9, 2022.
[71] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. Siea: Slow learner with classifier alignment for continual learning pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19148-19158, 2023,
[72] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng. Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter--fffiient fine-tuning. In International Conference on Learning Representations, 2023.
[73] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint wwXiv:2205.01066 2022.
[74] Linglan Zhao, Xuerui Zhang, Ke Yan, Shouhong Ding, and Weiran Huang. SAFE: slow and fast parfmeter-efffcientt tuning for continual learning with pre-trained models. In Advances in Neural Information Processing Systems, 2024.
[75] Weixiang Zhao, Shilong Wang, Yulin Hu, Yanyan Zhao, Bing Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. Sapt: A shared attention framework for parfmeter-efficient continual learning of large language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 11641-11661, 2024.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
15

---

Answer: [Yes]
Justification: Yes, the main claims presented in the abstract and introduction accurately reflect the paper's core contributions and scope. Specifically, the paper clearly positions itself as an improvement over existing LoRA-based methods for CL of language models. The abstract and introduction appropriately emphasize the targeted application domain, namely natural language processing. This alignment ensures that the reader is given faithful overview of the paper's goals, proposed methods, and experimental focus.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Section discusses the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well- sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
16

---

Justification: Yes, the paper provides a complete and correct proof for the theoretical result stated Proposition 3.1 All necessary assumptions are clearly stated, and the full proof is presented in Section A.3 of the Appendix in the supplemental material. The logical steps are detailed and consistent with the claim, ensuring transparency and rigor in the theoretical justification.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Ys]]
Justification: Section 3 provides a complete description of our method, and Algorithm ! clearly outlines its step-by-step implementation. Furthermore, Section 4. in the main text and Section B of the Appendix present all essential experimental settings, including datasets, evaluation protocols, hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky some cases, in which case authors are welcome describe the particular way they provide for reproducibility.
17

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: are committed to ensuring reproducibility and transparency. We have released the full source code along with detailed instructions for reproducing all main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4.1 in the main text outlines the overall experimental setup, including datasets, evaluation metrics, baseline methods, and training protocols. In addition, Section B of the Appendix provides further details such as data splits, hyperparameter settings (e.g.. learning rates, batch sizes), the choice and type of optimizer, and the rationale behind key parameter selections. Together, these sections ensure that the experiments can be accurately reproduced and fairly compared.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
18

---

Justification: While the main text follows the reporting conventions of existing works on LLMs 77 T3 6T], presenting averaged performance across methods and datasets, we provide more detailed statistical analysis in the Appendix. Specifically, Section of the supplemental material includes properly defined error bars (e.g., standard deviation across multiple runs), offering insight into the variability and robustness of the reported results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8, Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section of the Appendix specifies the types of compute resources used, including CPU and GPUs. Additionally, Section B.44 reports the extra computational overhead (in FLOPs) introduced by our method during training and inference.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:///wwpppb.ceeciciciiiieeeee
Answer: [Yes]
Justification: Our research conducted in the paper conform, in every respect, with NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
19

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Section discusses both positive societal impacts and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymenn of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
20

---

Justification: All external assets used in this paper, such as T5, Llama-2 and DeepSpeed, are properly credited. We ensure that all assets are used in accordance with their respective licenses and terms of use. Specific citations are provided in the paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papesswippeeeeeeeee thcode coo/dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
21

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM for what should or should not be described.
22

---

A More Details of Methods
A.1 Gradient Projection Memory
We initialize the first I. layers of (+()) using the corresponding layers from the previous gating module 0t-1 (-). Therefore, the first I. layers of ge(-) can be viewed as being initialized at the beginning of the first task and continue their training until the f-th task. Additionally, the first L layers in g-(-) serve as checkpoints, preserving the state of ge(-) after learning the i-th task (1 S i S t). At this time, existing method gradient projection memory (GPM) [47] can be used to learn matrices (M..) 1+1 where the columns of M, approximate the orthonormal bases of the subspace M Specifically, when 1, since there is no old task, MM1 is a null space and M1 is a zero matrix. After learning the t-th new task, GPM expands Mr, to M by first computing the input matrix Hi, where each column of H1 represents an input to the 1-th layer. Then, the component of H already in Me, is removed by
i=1
H. Hor M,,(M.))
(15)
Next, singular value decomposition (SVD) is performed on Hr, t,t. which is decomposed as U., Then, u new orthonormal bases U1 ..., Uy are chosen from the columns of Ü,,, where is the minimum number satisfying the following criteria for given threshold cyh:
H Hu + (M,.) H.I.P 2 Eth He,lle.
(16)
Here, (H denotes the components of H corresponding to the top-u singular values. Then, the orthonormal bases of subspace M are obtained by augmenting the orthonormal bases of subspace Me. with the new orthogonal vectors U,,... Uu resulting in M,+1. [Mrt Uı...., Uy).
A.2 More Details of O-LoRA and InfL.RA
O-LoRA O-LoRA [6]] ensures that the new LoRA branch remains orthogonal to all the old LoRA branches. Specifically, during the learning of the t-th new task with the t-th LoRA branch (A,, B,), O-LoRA computes the inner product between the new and old LoRA branches
O B, B, for 1 sist-1
(17)
Then, the loss function of O-LoRA defined as
1
yel
log [P [2, It, Ut.j -1) + A
t-
Σ
(18)
D (DD.
1=1 J.K
For further details on O-LoRA, we refer readers to the original paper 61].
InfLoRA InfLoRA 33 ensures orthogonality between the new LoRA branch and the gradients of old tasks. Specifically, it shows that only fine-tuning the down- www-jproeection matrix A, in the new LoRA branch is equivalent to directly fine-tuning the pre-trained weights within a subspace spanned by the rows of B, Therefore, before learning the t-th task, InfLoRA designs B, to be orthogonal to the gradients of the old tasks. During the learning of the t-th task, InfLoRA only tunes A, in the new LoRA branch while freezing B, and the old LoRA branches. For further details on InfLoRA, we refer readers to the original paper [32]
A.3 Proof of Proposition 3.1
Proposition A.1. If the constraints in 7 are satisfied, subspaces (Mct) L+1 remain unchanged during the learning of the t-th task. Furthermore, for any input from the previous t - 1 tasks, 9r (x) remains unchanged during the learning of the t-th task.
Proof. For any from previous t tasks, we rewrite (x) as
(x) f(G,LL
Pt. = o (G..IP...) ), le (1, L},
Pr, Po = Pool (Tkken(z)..
(19)
23

---

Since Pt.o = Pool Token is unrelated to the parameters of the new gating module 9e(-), Pt.o does not change with the update of ge(-). Since M,, is spanned by Pt.o- Mr, remains unchanged during the learning of the 1-th task.
Suppose that have proven that Pt,-1 does not change with the update of the gating module gr(-) (1 S I S L). Since M1,1 is spanned by Pt,i-1, Me, remains unchanged during the learning of the 1-th task. At this point, Pt. can be expressed as
Pt. = ((Iit((GG.) + AGu))(.-1) = (Init (G.
(20)
Here, the second equality holds since Pt,t- MMa and A AG,,LM,.. Therefore, Pt, does not change with the update of the new gating module (c(-) (1 < / S L). Since Ma,41 is spanned by Pt,, Ma,4+1 remains unchanged during the learning of the f-th task.
Furthermore, during the learning of the t-th task, gc(x) can be expressed as
gr (x)))= = f((nnit(Gr.L))) + AGI,++))P.L) = f(Int(Gr,L+))P..) Here, the second equality holds since Pt,L E M,,4+1 and AG
(21)
B More Details of Experimental Settings
B.1 More Details of Datasets
Table and Table 6 show the details of Long Sequence Benchmark and SuperNI Benchmark, respectively. Long Sequence Benchmark consists of 15 classification tasks while SuperNI Benchmark consists of various NLP tasks, including dialogue generation, information extraction, question answering, summarization, and sentiment analysis.
Table 5: Details of different tasks in Long Benchmark.
Dataset name
Category
Domain
Task Type
Metric
Yelp
CL Benchmark
sentiment analysis
Yelp reviews
Accuracy
Amazon
CL Benchmark
sentiment analysis
Amazon reviews
Accuracy
DBpedia
CL Benchmark
topic classification
Wikipedia
Accuracy
Yahoo
CL Benchmark
topic classification
Yahoo Q&A
Accuracy
AG News
CL Benchmark
topic classification
news
Accuracy
MNLI
GLUE
natural language inference
various
Accuracy
QQP
GLUE
paraphrase detection
Quora
Accuracy
RTE
GLUE
natural language inference sentiment analysis
news, Wikipedia
Accuracy
SST-2
GLUE
movie reviews
Accuracy
WiC
SuperGLUE
word sense disambiguation
lexical databases
Accuracy
CB
SuperGLUU
natural language inference
various
Accuracy
COPA
SuperGLUE
question and answering
blogs, encyclopedia
Accuracy
BooIQA
SuperGLUE
boolean question and answering
Wikipedia
Accuracy
MultRC
SuperGLUE
question and answering
various
Accuracy
IMDB
SuperGLUE
sentiment analysis
movie reviews
Accuracy
The task sequences are constructed using Long Sequence Benchmark and SuperNI Benchmark. The details of different task sequences are presented in Table
B.2 More Implementation Details
68 all methods are implemented using instruction tuning 40 Experiments are conducted on 5 NVIDIA RTX A6000 GPUs with AdamW (35) the optimizer. The type of CPU Intel(R) Xeon(R) Gold 6240R CPU 2.40GHz. For T5-Large and TS-XL, their relatively smaller model sizes allow experiments to be performed on single A6000 GPU with
Following existing CL works 44
24

---

Table 6: Details of different tasks in SuperNI Benchmark.
Dataset name
Task Type
Metric
Task639 1_00_i_ml_ii___ woz_user wttuuuttaneeeeneeeeeee
summarization
Rouge-L
159k _ecpoodddpllmay___mm _eeettt______ttttttteee
text_generation
summarization
Rouge-L
summarization
Rouge-L
Task181 Faskl81_ottcome ott.outtomeeeettcttttioo
information extraction
Rouge-L
information extraction
Rouge-L
information extraction
Rouge-L.
dialogue generation
Rouge-L
Task073 http::/o_oooooosseennaaaseeeeeee
dialogue generation
Rouge-L.
Task591_seat 1a1999l__si___sseeeeee sunnsrrrgeneration
dialogue generation
Rouge-L
Task511 Bei_st1_reddltttifuuheiee setti__ooo__ee__sumrriiiiiiiiin
question answering
Rouge-I.
Task 2tt.:/90_ssummmmmmrittttiii summarization
question answering
Rouge-L
Task1572_samsung -ummmarr
question answering
Rouge-L
Task363 Bask363_sst2__ppp 1ust2_polaritty pttppooltyllassiictalltet cllassificatiinn
sentiment analysis
Accuracy
1e4k875_emotion__.. classification
sentiment analysis
Accuracy
101111877_eenmmmnii classification
sentiment analysis
Accuracy
Table 7: The order of different task sequences for experiments.
Benchmark
Order
Task Sequence
task 1572 + task363 - task + task181
181-1111111 - task002 -
task 1510 + task639 - t7sk task073 > task1590
SuperNI Benchmark
task748 + task511 1 - task591 - task1687 - task875
task748 >> task073 + task > task639 - task -
task task591 - task363 - task1510 task1729 +
task181 + task511 - task002 - task 1290 + task875
MNLI - CB + WiC + COPA - QQP
BooIQA - RTE IMDB + Yelp Amazon
CL Benchmark
SST-2 - DBpedia + AG News - MultiRC + Yahoo
Yelp > Amazon MNLI CB COPA
4
QQP - RTE + IMDB + SST-2 - DBpedia +
AG News + Yahoo - MultiRC BoolQA + WiC
gradient accumulation. For Llama-2-7B and 1lama-2-13BB data parallclism with DeepSpeed ZeRO- 2 [44) is prioritized across multiple A6000 GPUs. FlashAttention-2 9 is employed to reduce memory usage during training, ensuring sufficient GPU memory enable DeepSpeed ZeRO-2 whenever possible. However, the sequence lengths of certain tasks are too long to enable DeepSpeed ZeRO-2 with FlashAttention DeepSpeed ZeRO-3 is utilized handle these tasks.
To ensure fair comparisons, for the methods based on LoRA, we follow existing CL methods 21 75 by integrating the LoRA A architecture into the query and value components of the multi-head attention mechanism in each Transformer block. Following cxisting works 161 75), for all the methods based LoRA, the rank of single LoRA branch is set to 4 for Order I and Order 2, and 8 for Order 3 and Order 4. We also vary the rank in LoRA branches and show the results in Appendix C.4
For our methods, the global batch size is set 32 across all model backbones. The learning rate is set to 3e-4 for T5 backbones and 5e-5 for Llama backbones. Each task is trained for 100 epochs with T5 backbones and 50 epochs with Llama backbones. For baselines, we follow their official implementations to set the hyperparameters, making the comparison as fair possible. If this does not achieve the expected performance, we perform hyperparameter search for the learning rate and batch size.
25

---

B.3 More Details about the Architecture of the Gating Module
The architecture of the gating module gi(-) can be represented
gi(x) - f(G,,L+PP.L)
Pi,1 a (G.,,P..-); Ee (1, D),
Pio Po = Poooe(Tke)(())).
(22)
Non-line Non-lineer activation function a-)) is set to SiLU [1] For all experiments, unless otherwise stated, is set 2. In other words, the gating module g((-) has three layers. For Ta-Large and T5-XL, the parameters in the i-th gating module 9:(-) are G1,1 E 10000d, G.,2 E 12d and Gr,3 E R For Llama-2-7B and Llama-2- 1-BB, the parameters in the i-th gating module g-(-) are G R0 2. do and G. a IRidd Here, denotes the dimension of the embeddings. For different models, d is 1024 for Ta-Large and TS-XL, 4096 for Llama-2-7B, and 5120 for Llama-2-13BB
Additionally, investigate the influence of the architecture of the gating module the performance of our method. Results are provided in Appendix C.3
B.4 Computation of Trainable Parameters
To ensure fair comparisons, we set the same rank for each LoRA branch across all CL methods based on the expandable LoRA architectures shown in Figure (a). Additionally, for all the methods based on LoRA, the LoRA modules are incorporated into the query and value components of the multi-head attention mechanism within each Transformer block.
B.4.1 Computation of Trainable Parameters in T5-Large
In T5-Large, the projection weights for the query and value components have shapes Wg. We R 1024xx1021 The model consists of 24 self-attention modules in the encoder, 24 self-attention modules in the decoder, and 24 cross-attention modules in the decoder, resulting in a total of 24) 2 144 pre-trained weights that incorporate the LoRA architecture.
(24 + 24
During the learning of the 1-th new task, O-LoRA updates the parameters A, E 102224x and B, E Rrx1024 resulting in 1024 * r 144 + 1024 144 294912r trainable parameters. When r 4, the number of trainable parameters in O-LoRA is 294912 * 4 = 1179648 1.18M. InfLoRA only updates the parameters A, R 1024xr, resulting in 1024
144 = 1474567 trainable parameters. When r 4, the number of trainable parameters in InfLoRA is 147456r = 589824 = 0.59M.
GainLoRA A introduces an additional new gating module (+(-) with parameters Gt,1 E TR 100 1024 Gt.2 E 11024x100 and Gr.3 E 10121024 Therefore, the number of trainable parameters in Gain- LoRA (O-LoRA) is 1179648- 1024 100 + 1024 100 + 1024 1385472 = 1.39M. The number of trainable parameters in GainLoRA (InfLoRA) is 589824 + 1024 * 100 + 1024 * 100 + 1024 795648 0.80M.
B.4.2 Computation of Trainable Parameters in T5-XL
In T5-XL, the projection weights for the query and value components have shapes W;, W, e 4006x 1024 The model architecture similar to Ta-Large, with 144 pre-trained weights incorporating
LoRA.
During the learning of the 1-th new task, O-LoRA updates the parameters A, E R 1096xr and B, E r021 resulting in is 4096 * * 144 + * 1024 * 144 = 737280r trainable parameters. When O-LoRA has 737280 4 2049120 2.95M trainable parameters. InfLoRA only updates A, E IR 4096xr resulting in 4096 r * 144 = 589824r trainable parameters. When r = 4, 2359296 2.36M trainable parameters.
InfLoRA has 589824
GoinLoRA introduces the same new gating module 9-(-) as in T5-Large, with parameters Gi,1 E 100011021 G. E fg1024x100 100 and G.,a 101210 Thus, the total number of trainable parameters for GainL (O-LoRA) is 2949120 + 1024 100 1024 100 + 1024 3154944 = 3.15M. The total number of trainable parameters GainL.RRA (Infl is 2359296 1024 100 + 1024
100 + 1024- 2565120 2.57M.
26

---

B.4.3 Computation of Trainable Parameters in Llama-2-7B
In 2-7ma-2-7B, the projection weights for the query and value components have shapes Wa We R4096x 2096 The model contains 32 self-attention modules, resulting in 32 2 64 pre-trained weights that incorporate the LoRA architecture.
During the learning of the t-th new task, O-LoRA updates the parameters A. E R 4096xr and B, E x resulting in 4096 r 64 + r * 4096 64 524288r trainable parameters. When r = 4, the number of trainable parameters in O-LoRA is 524288 * 4 = 2097152 = 2.10M. InfLoRA
only updates the parameters A, E R4400xxr, resulting in 4096 64 262144r trainable parameters.
When r = 4, the number of trainable parameters in InfLoRA is 262144 4 = 1048576 1.05M.
GainLoRA introduces new gating module (+(-)) with parameters Gr, E R 1096 .. E IR 4096x30
and .r,3 3 E I' 2 4096 Therefore, the number of trainable parameters in GainLoRA (O-LoRA) is 2097152 4096 * 50 + 4096 + 50 + 4096 2510848 2.5 The number of trainable parameters in GainL (InfLoRA) is 1048576 + 4096 50 + 4096 50 + 4096 = 1462272 = 1.46M.
B.4.4 Computation of Trainable Parameters in Llama-2-13B
In Llama-2-13B, the projection weights for the query and value components have shapes Wq. W, R5120x5120 The model contains 40 self-attention modules, resulting in 40 2 80 pre-trained weights that incorporate the LoRA architecture.
During the learning of the (-th new task, O-LoRA updates the parameters A, E R and B, E Rrx5120 resulting in 5120 * r * 80 + r * 5120 * 80 = 819200r trainable parameters. When r= 4, the number of trainable parameters in O-LoRA is 819200 * 4 3276800 = 3.28M. InfLoRA only updates the parameters A; E R5120xr resulting in 5 + 80 = 409600r trainable parameters.
When 4, the number of trainable parameters in InfLoRA is 409600 4 1638400 1.64M.
GainLoRA introduces new gating module ge with parameters Gt,1 E R R.50x6120 G.22 E 51200x50 and Gr.3 | Therefore, the number of trainable parameters in GainLoRA (O-LoRA) is 3276800 + 5120 * 50 + 5120 # 50 + 5120 3793920 = 3.79M. The number of trainable parameters in GainLoRA (InfLoRA) 1638400 + 5120 50 5120 50 + 5120- 2155520 2.16M.
B.4.5 Computation of Trainable Parameters in Llama-3-8B
In Llama-3--88 3-8B, the projection weights for the query and value components have shapes W E 1090x 4096
W. E 1-906x10244 The model contains 40 self-attention modules, resulting in 32 * = 64 pre-trained weights that incorporate the LoRA architecture.
During the learning the 1-th new task, O-LoRA updates the parameters A. E R 4096xr and B, E R 4096 for query and A, RR 1024xr and B, E X 4096 for value, resulting in 1024 32 4096 r 32 + y * 4096 64 = 425984r trainable parameters. When r 4, the number of trainable parameters in O-LoRA is 425984 4 = 1703936 1.70M. InfLoRA only updates the parameters A, E IR x0 for query and A, E R 1024xx for value, resulting in 4096 r * 32 + 1024 * r 32
163840r trainable parameters. When 4, the number of trainable parameters in InfLoRA is 163840 4 = 655360 = 0.66M.
GainLoRA introduces new gating module ge(-) with parameters G1,1 E R 500x 4096 and G. E IR 1x 4096 Therefore, the number of trainable parameters in GainLoRA (O-LoRA) is 1703936 + 4096 * 50- 4096 # 50 + 4096 2117632 2. 12M The number of trainable parameters in GainLoRA (InfLoRA) is 655360 + 4096 * 50 + 4096 50 + 4096 = 1069056 = 1.07M.
G. E T T=4096x50
C More Experimental Results
C.1 Discussing Computational Costs Introduced by GainLoRA
Existing methods, such as O-LoRA and InfLoRA, adopt the expandable LoRA architecture shown in Figure (a) and fix the integration coefficients { to 1, allowing the model to merge the expanded LoRA branches into the pre-trained matrix at inference time, thereby avoiding additional computational costs. However, when using our gating module to integrate different LoRA branches, the LoRA branches cannot be merged into the pre- pre-trained matrix at inference time, which introduces
27

---

Table 8: FLOPs and MACs for different models.
Method
Input Shape (batch,length)
FLOPs (G)
MACs (G)
Original
(1,128)
194.25
97.1
GainLoRA (O-LoRA)
(1,128)
198.79
99.37
TS-Large
GainLoRA (InfLoRA)
(1,128)
198.79
99.37
Original
(1,128)
751.7
375.78
GainLoRA (O-LoRA)
(1,128)
763.03
381.45
T5-XL
GainLoRA (IniLoRA)
(1,128)
763.03
381.45
Original
(1,128)
1701.07
850.5
Goi (O-LoRA)
(1,128)
1709.14
854.53
Llama-2-7B
Gai (InfloRAA
(1,128)
1709.14
854.53
Original
(1,128)
3291.66
1645,79
GainLoRA (O-LORA)
(1,128)
3304.26
1652.09
Llama-2-132
GainLoRA (InfLooRA)
(1,128)
3304.26
1652.09
Original
(1,128)
1929.86
964.89
GainLoRA (O-LoRA)
(1,128)
1930.49
965.21
Llama-3-8B
GainLoRA (InfLoRA)
(1,128)
1930.49
965.21
additional computational costs. Nevertheless, we demonstrate that these computational costs are minimal compared to the computational cost of the original large language models (LLMs).
Table compares the Seating-point operations (FLOPs) and multiply-add operations (MACs) during inference for different models with and without GainLoRA. The computation of FLOPs and MACs follows the existing project calflops [69]. Here, "Original" denotes the original LLMs without any LoRA adaptation. Methods such as O-LoRA and InfLoRA avoid additional computational costs by merging their LoRA branches into the original weights during inference, resulting in FLOPs and MACs identical to the original LLMs. Despite introducing additional FLOPs and MACs compared to the original LLMs, GainLoRA maintains minimal computational overhead relative to the original LLMs.
C.2 Additional Computation Introduced by Subspace Construction
The memory and computational overhead of subspace construction in GainLoRA is minimal due to the small size of the gating module (only 3 layers, see B.3). We provide detailed analyses below.
Memory The number of orthogonal bases stored for each subspace does exceed its dimension. For T5-Large, the dimensions of the three subspaces are 1024, 100, and 1024, respectively. This results in worst-case memory of less than 0.3% of the total model parameters ((2 10242 1002)/CTS Large's (orrams)0.3%% Similar estimates yield 0.07%, 0.5%, and 0.4% for T5-XL, Llama-2-7BB Llama-2-131 and Llama-3-8B. respectively. Since this calculation represents a rough upper bound, the actual memory is even lower.
Computational Overhead The computational overhead for subspace construction requires single forward pass over the task dataset and SVD on the feature matrices of the gating module.
Assuming single forward pass over the task dataset requires A FLOPs. For T5-Large, training a task for 100 epochs needs 100 forward and backward passes. Since single backward pass has roughly 2/ FLOPs, the total FL.OPs are 500 Thus, single forward pass for subspace construction accounts for only 1/500=0.295 2% of total computation. Similar estimates yield 0.2%, 0.4%, and 0.4% for X5,XL, -lama-2-7B, and Alama-2-13BB respectively.
GPM requires performing SVD on matrix HiHI Rdxx, where Hi the feature matrix in the
1-th layer of gating module. Based existing conclusions (55), the FLOPS for SVD H HT is less than 10d3 For Ta-Large (d, = d3 1024 and d2 100), this results 10 (2 * 10243 + 1003) <
30GFLOPs, which is negligible compared a single forward pass with sequence length 128 (see Table 8). Similar calculations give the same conclusion for T5-XL, Llama-2-7B, Llama-2-13B and Llama-3-8B.
C,3 Varying the Architecture of Gating Module
28

---

Table 9:
Varying the function f(.) in GainLoRA on different task sequences with T5-large model.
Order 1 APM
Order
Method
FT
APt
FT
GainLoRA (InfLoRA) (f(b) ( 11)
46.21
2.40
46,44
2.61
GainLoRA (InfLoRA) (f(b) min [lb] 1
45.05
2.07
45.00
1.74
GainLoRA (InfLoRA) (f(b) sin 2
47.48
1.21
45.03
2.37
InfLoRA
39.78
7.64
39.57
8.93
GainLoRA (O-LoRA) (f(b) [2sigmoid 11)
47.84
2.26
46.84
2.91
GainLoRA (O-LoRA) (f(b) = min{(b (b) 1))
49.62
2.83
48.62
3.74
GoinLoRA (O-LoRA) (f(b) =ssin( mh
1)
48.49
3.84
47.20
4.69
2
O-LoRA
26.37
19.15
32.83
11.99
Averaged Performance
Averaged Performance
Averaged Perf Berformance
Averuged Performance
45.0
42.5
47-
OMotA GazLakA (0-LaBA)
17.54
O.l.RA
O.LoRA
GainLalA (D-LAAA)
GasLaRA (6-UdA))
GasLakk (DLaRA)
38
15.0
10
Inflatx
IvE:RA
GasLaRA delLaA
32.5
ds
da
Order
Order
(c) Order
(d) Order
Figure 5: (a) and (b) show the variation of our methods' performance with the shapes of the weights the gating module. (c) and (d) show the variation our methods' performance with the Layers of the gating module.
C.3.1 Varying Function f(.) in Gating Module
To implement our method, we define function f(-) as 8 as the following two functions:
Here, we vary the formula of function f(-)
xb 2 )
min [b), 1}, sin
(23)
Clearly, these two functions map real values among [0, 1] and satisfy f(0) Table 9 shows the results. As we can see, when changing the formula of f()), GainL also improves the performance of O-LoRA and Inf..
C.3.2 Varying the Shapes of Weights in Gating Module
In this section, we vary the shapes of the weights in the gating modules with T5- Specifically, we set the weights G. 1 E Ed, x1024 and G, IR 1024xd, in each gating module g.(-) and vary da over {50, 100, 200}. Figure 5 (a) and Figure (b) show the results, As we can see, when increasing dy, the performance of GainLoRA remains relatively stable, indicating that our method is robust to the shape of the weights in the gating module. Note that the number of trainable parameters increases
as d, increases.
C.3.3 Varying the Layers of Gating Module
In this section, we vary the layers of the gating modules with T5-Large. Specifically, we vary across
{0, 2, 4). when L = 0, there only one layer with G. R x 1024 1024 in each gating module (-).
When I. 2, there three layers with G R 100x10021
G E R1021x100 and G E R1x1021
When L = 4, there are 5 layers with G. E R 100x G. 1024x100 G E R 1024
G E R 1024x100 100 and G E R 1024 in each gating module. Figure (c) and Figure 5 (d) show the results. As we can see, when increasing the layers of gating modules, the performance of GainLoRA remains relatively stable, indicating that our method is robust to the layers of the gating module. Note that the number of trainable parameters increases as the number of layers in gating modules increases.
29

---

Averged Performance
Avengeel Performunncs
GunLat (AUORA)
GanuatA (0-U0AA) UaaaatA (bstoR0)
GURLaRA (MILORA)
Rank
Rank
(a) Order
(b) Order 2
Figure 6: The variation of our methods' performance with the Layers of the gating module.
Table 10: The overall results on different task sequences with Tarlerge model.
Order
Order
Method
APM
FT
APT
FT
IncLoRA
12.33
41.93
16.65
36.56
GainLoRA (IncLoRA)
47.82
3.73
45.42
5.83
C-LoRA
22.69
24.25
32.81
11.60
GainLoRA (C-LoRA)
49.24
2.94
46.23
6.05
C.4 Varying Ranks in LoRA Branches
In this section, we vary the rank of LoRA branches across {2, 4, 8} with T5-Large. Figure shows the results. As shown, when the rank of LoRA branches increases, the performance of GainLoRA remains relatively stable. Note that the number of trainable parameters increases as the rank LoRA branches increases.
C.5 Adopting Other Update Strategies for the New LoRA Branch
Our GainLoRA does not impose specific update strategies for the new LoRA branches. In this work, we adopt the same update strategies the existing two methods, O-LoRA [61 and InfLoRA (32). Related methods, such as IncLoRA (1 and C-LoRA (50), also adopt the expandable LoRA ar- chitecture illustrated in Figure (a) and fix all integration coefficients {at) 1 to 1. Our method
GainLoRA can also adopt their update strategies for the new LoRA branch. Table I0 presents the results, demonstrating that GainLoRA further improves the performance of these two methods.
C.6 Performance on the TRACE Benchmark
To further demonstrate our method's effectiveness, we follow existing CL methods for LLMs 18 62 and conduct experiments on the TRACE dataset [62] with -laam--2-7B-Che The dataset comprises a diverse set challenging instruction-tune tasks, spanning multilingual comprehension, domain- specific knowledge, arithmetic reasoning, and coding. An overview of the tasks in TRACE is provided
in Table
Table 11: The order TRACE benchmark for experiments.
Benchmark
Task Sequence
C-STANCE -+ FOMC MeetingBank - Py150 ScienceQA - NumGLUE-cm - NUMGLUE-ds - 20Minuten
TRACE Benchmark
Table 12 reports the average performance the TRACE benchmark after sequentially learning all tasks. The results demonstrate that our method effectively mitigates catastrophic forgetting and outperforms existing baselines. This capability is crucial for real-world applications.
Retention General Capabilities We also follow existing work explicitly evaluate the preservation of general abilities, such foatuuction-followingg after continual learning the TRACE
30

---

Table 12: TRACE benchmark performance using LLama-2-77 2-7B-Chaa
APT
FTV
O-LoRA 66
41.04
8.05
GainLoRA (O-LoRA)
48.10
0.99
InfLoRA R
47.67
2.25
GainLoRA (InfLoRA)
49.15
0.89
Table 13: Comparison of general ability scores across six diverse evaluation tasks between the base LLaMA-2-7B chat model and different methods.
PIQA
MMLU
GSM8K
BBH
BooIQA
TydiQA
O-LoRA [6]
72.85
32.87
13.42
35.10
56.88
19.48
GainLoRA (O-LoRA)
73.61
33.33
18.57
36.47
59.69
25.00
InfLoRA 32
74.86
40.86
15.69
35.87
65.29
27.25
GainLoRA (InfLoRA)
75.24
44.25
21.30
37.44
68.81
27.84
Hamma-2-7B-Cie
75.35
46.13
26.54
40.09
70.46
23.45
benchmark using different methods. Table 13 indicates that continual learning with different methods often leads to a degradation of general abilities. However, GainLoRA demonstrates a stronger ability mitigate forgetting compared other LoRA-based methods, including O-LoRA and InfLoRA.
C.7 Compared with More CL Methods in CV
Following many existing continual learning methods in NLP 75 models based on next-token prediction, such T5 43 and LLaMA [54] which lack the [CLS] token used VIT. Although many continual learning methods based on pre-trained models in computer 58, 31] cannot be directly applied to setting, adapt several of them to the T5 architecture to ensure a comprehensive comparison. Specifically, We implement these methods by injecting prompts into both the keys and values in T5, and introduce vero-padding in the position bias tensor to ensure shape compatibility. Note that we do not add positional information to the prompts, which is consistent with DualPrompt and CODA-Promy in VIT. The results on order with T5 are reported in Table [4] showing that these methods perform significantly worse than our GainLoRA A and exhibit noticeable forgetting.
61 18 this paper focuses on
vision 66 67 63
57
Table 14: Compare with different methods order with T5 architecture.
AP1
FT
L2P [67
15.23
11.34
DualPrompt 66
17.40
10.63
CODA-Prompt 51
19.28
14.62
Lai (O-LoRA)
47.84
2.26
GainLoRA (InfLoRA)
46.21
2.40
31


Reviews:
[{"review_id": "G9nJQbHDYd", "reviewer": "Reviewer_KRVe", "summary": "This paper addresses the challenge of parameter-efficient fine-tuning in multi-task settings, where naively combining task-specific LoRA adapters often leads to performance degradation. It introduces a gated integration framework called GLoRA, which enables dynamic weighting over pre-trained LoRA experts using a lightweight gating module. GLoRA can be plugged into existing PEFT workflows and improves both multi-task and continual learning performance, without requiring retraining or task-specific annotations.", "questions": "none", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper tackles the pain point that naïve fusion of task-specific LoRA adapters hurts multi-task performance, and puts forward GLoRA—a gated integration scheme that \n(1) keeps the LoRA parameter budget tiny, \n(2) lets a lightweight gate dynamically weight experts at run-time, and \n(3) plugs straight into existing PEFT pipelines without re-training the backbone. The authors back the idea with broad experiments on ten diverse NLP tasks, three foundation models (T5-Base, OPT-1.3B, LLaMA2-7B), plus ablations on gate order and continual-learning settings, demonstrating consistent gains and limited forgetting. Implementation is simple (just an extra MLP and softmax), code-path changes are minimal, and the paper provides clear analyses of parameter/compute overhead.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive and encouraging feedback. We appreciate your recognition of the motivation and effectiveness of our method."}, {"review_id": "INq2HSkijy", "reviewer": "Reviewer_5sqb", "summary": "In this paper, the authors introduce a novel method to mitigate catastrophic forgetting in continual learning (CL) using language models (LMs). Specifically, the authors introduce gated integration of LoRA (GainLoRA), which will adaptively control the output of a set of LoRA modules corresponding to different tasks such that they do not interfere with each other. GainLoRA achieves this by imposing constraints on the initialization and the update of the gating module.  Thus, this approach does not require samples from prior tasks or task labellings to mitigate the interference of the LoRA adapters that are irrelevant to a given task at the inference time. The empirical results provided in the paper suggest that incorporating the proposed gating mechanism with the existing LoRA-based CL methods can improve their ability to mitigate catastrophic forgetting and improve the average accuracy across tasks.", "questions": "* Can the authors provide more discussion/explanation on why the updates (9) and (10) allow the initialization and the update to satisfy the constraints (5) and (6), respectively?\n\n* How can the gating model assign the correct weight to the LoRA output when the input distribution is similar accross the tasks, whie the output is different?", "limitations": "Please refer to the questions above.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n* The paper discusses the problem of catastrophic forgetting in LMs in the CL setting, which is a critical and pertinent problem in the field of machine learning.\n* The flow of the paper is easy to follow, and the proposed method is introduced intuitively.\n* The ablation studies provided in the paper validate the importance of the different components of the proposed method\n* The empirical results provided in the paper suggest that the performance of existing methods designed for CL can be significantly improved by the proposed method. \n\n**Weaknesses**\n\n* Some key information needed to follow the paper seems to be missing in the main text. For example, some more explanation on why the operation in (9) makes the initialization to satisfy the constraint (5), in addition to providing a reference to prior work.\n\n* It is unclear how only input can be used to determine the identity of the task. For example, the same input can be used for a text generation task (e.g., summarization) and a classification task. How can the proposed method mitigate the task interference in a situation like this?\n\n**Minor Weaknesses**\n\n* Line 127: $p_{L+1}$ -> $p_{L}$?\n* Although the main text points to the supplementary for some key definitions and metrics, the supplementary is not provided adjacent to the main text, which makes the paper less self-contained.", "was_revised": true, "final_justification": "The authors were able to address my concerns in the rebuttal period, and agreed to revise the final version based on these concerns. Thus, I recommend this paper for acceptence.", "rebuttal": "**Q1: Some key information needed to follow the paper seems to be missing in the main text. For example, some more explanation on why the operation in (9) makes the initialization to satisfy the constraint (5), in addition to providing a reference to prior work.**\n\n**A1** Thank you for pointing this out. Here, we provide a more detailed explanation to clarify this point.\n\nThe operation in (9) is designed to project the initialization ${\\rm Init}(\\mathbf{G}\\_{t,L+1})$ onto the orthogonal complement of the subspace $\\mathcal{M}\\_{t,L+1}$, which is spanned by the columns of the orthonormal basis matrix $\\mathbf{M}\\_{t,L+1}$. Specifically, the projection matrix in (9) is given by\n$${\\rm Init}(\\mathbf{G}\\_{t,L+1})-\\mathbf{M}\\_{t,L+1}\\mathbf{M}\\_{t,L+1}^{T}{\\rm Init}(\\mathbf{G}\\_{t,L+1})=(\\mathbf{I}-\\mathbf{M}\\_{t,L+1}\\mathbf{M}\\_{t,L+1}^{T}){\\rm Init}(\\mathbf{G}\\_{t,L+1}).$$ \nFor any vector $\\mathbf{m}\\in\\mathcal{M}\\_{t,L+1}$, since $\\mathcal{M}\\_{t,L+1}$ is spanned by the columns of the orthonormal basis matrix $\\mathbf{M}\\_{t,L+1}$, we have $\\mathbf{m}=\\mathbf{M}\\_{t,L+1}\\mathbf{a}$ for some coefficient vector $\\mathbf{a}$. Then\n$$\\mathbf{m}^{T}({\\rm Init}(\\mathbf{G}\\_{t,L+1})-\\mathbf{M}\\_{t,L+1}\\mathbf{M}\\_{t,L+1}^{T}{\\rm Init}(\\mathbf{G}\\_{t,L+1}))=\\mathbf{a}^{T}\\mathbf{M}\\_{t,L+1}^{T}(\\mathbf{I}-\\mathbf{M}\\_{t,L+1}\\mathbf{M}\\_{t,L+1}^{T}){\\rm Init}(\\mathbf{G}\\_{t,L+1})=\\mathbf{a}^{T}(\\mathbf{I}-\\mathbf{M}\\_{t,L+1}^{T}\\mathbf{M}\\_{t,L+1})\\mathbf{M}\\_{t,L+1}^{T}{\\rm Init}(\\mathbf{G}\\_{t,L+1}).$$\nSince the columns of $\\mathbf{M}\\_{t,L+1}$ form an orthonormal basis, we have  $\\mathbf{M}\\_{t,L+1}^{T}\\mathbf{M}\\_{t,L+1}=\\mathbf{I}$, which means $\\mathbf{I}-\\mathbf{M}\\_{t,L+1}^{T}\\mathbf{M}\\_{t,L+1}=\\mathbf{O}$. Therefore, the above equation is equal to zero matrix $\\mathbf{O}$, which means the projected initialization in (9) is orthogonal to the entire space $\\mathcal{M}\\_{t,L+1}$, and hence the constraint (5) is satisfied. We will add the above explanation in the final version.\n\n**Q2: It is unclear how only input can be used to determine the identity of the task. For example, the same input can be used for a text generation task (e.g., summarization) and a classification task. How can the proposed method mitigate the task interference in a situation like this?**\n\n**A2** We follow the instruction tuning paradigm as adopted in existing works [1, 2]. In particular, even if the same input can be used for a summarization task and a classification task, the corresponding instructions are inherently different. For instance, the instruction for summarization may be: \"Summarize the following passage,\" whereas the instruction for classification task may be: \"Classify the following text into one of the predefined categories: (A)..., (B)..., (C)...\". In this case, our mechanism can identify different tasks based on the differences in the input instructions.\n\nWithout the instruction, neither our method nor other methods can handle both summarization and classification tasks for the same input. Specifically, in the absence of instructions, methods such as OLoRA and TaSL will get the same output for a given input. Hence, in real applications, we adopt different instructions for different tasks. \n\n\n**Q3: Line 127: p_{L+1} -> p_{L}?**\n\n**A3** Yes, this is a typo. We will fix it in the final version.\n\n\n**Q4: Although the main text points to the supplementary for some key definitions and metrics, the supplementary is not provided adjacent to the main text, which makes the paper less self-contained.**\n\n**A4** We sincerely apologize for any inconvenience to the reviewers. To provide more comprehensive support for our work, we have added lots of clarifications and additional experiments in the supplementary materials. We will include the supplementary adjacent to the main text in the final version to ensure the paper is more self-contained and easier to follow.\n\n\n**Q5: Can the authors provide more discussion/explanation on why the updates (9) and (10) allow the initialization and the update to satisfy the constraints (5) and (6), respectively?**\n\n**A5** The discussion in **A1** has shown that the update in (9) allows the initialization to satisfy the constraint in (5). With the same proving process, we can show that the update in (10) allows the update to satisfy the constraints in (6).\n\n\n**Q6: How can the gating model assign the correct weight to the LoRA output when the input distribution is similar across the tasks, while the output is different?**\n\n**A6** As mentioned in **Q2** and **A2**, when the same input is used for different types of tasks (e.g., summarization vs. classification), our gating model leverages the instructions to assign the correct weight to the LoRA output.\n\nFor tasks within the same type (e.g., classification tasks), if the input is exactly the same, they can essentially be regarded as the same task. In practice, tasks within the same type typically differ in domain and style. With these differences in input distribution, our gating model can assign the correct weight to the LoRA output.\n\n[1] Wang X, Chen T, Ge Q, et al. Orthogonal Subspace Learning for Language Model Continual Learning, EMNLP 2023.\n\n[2] Zhao W, Wang S, Hu Y, et al. SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models, ACL 2024."}, {"review_id": "WU9iJjrapV", "reviewer": "Reviewer_uWep", "summary": "This paper introduces GainLoRA, a parameter‐efficient continual‐learning method for language models that builds on the popular LoRA framework. For each new task, GainLoRA adds a new low‐rank adaptation branch and integrates old and new branches via lightweight, task‐specific gating modules.  By imposing orthogonality constraints at initialization and during updates, GainLoRA drives the new branch’s contribution toward zero on previously learned tasks, thereby mitigating catastrophic forgetting while allowing full expressivity on the current task.  Extensive experiments on SuperNI and Long Sequence benchmarks, across multiple task orders and model scales are conducted.", "questions": "See weakness", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The presentation is straightforward and easy to follow.\n2. The experiments and ablations are comprehensive, addressing key concerns.\n3. The proposed GainLoRA can be plugged into other existing LoRA-based methods.\n\nWeaknesses\n1. The gating module itself is not inherently novel, as similar mechanisms using query-key matching ([1,2,3]) or moe ([4,5]) have been explored in PEFT and CL methods.\n2. The proposed gating module is a collection of task-wise 3-layer MLPs with SiLU activation, operates in parallelly to the LLM backbone, which acts as a task-id predictor. Fig. 3(c,d) reports gating activations only for the most recent (last) task. To fully support the paper’s claims, the authors should also report gating outputs (or task‐prediction accuracy) for **each** of the previous tasks.\n3. Weird performance in Table 18, two stronger variants of L2P ( DualPrompt and Coda-prompt) both underperform the weaker baseline L2P. The low AP and low FT mean the model fails to learn new knowledge as new tasks are introduced, which suggests that there may exist some errors/bugs in the reimplementation of these methods in the LM domain. As the authors noted, there is no [CLS] token, whereas a similar operation as in Eq.3 should be used for these methods without using the proposed constraints.\n\n[1] Learning to Prompt for Continual Learning https://arxiv.org/abs/2112.08654\n\n[2] DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning https://arxiv.org/abs/2204.04799\n\n[3] S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning https://arxiv.org/abs/2207.12819\n\n[4] CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning https://arxiv.org/abs/2211.13218\n\n[5] Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning https://arxiv.org/abs/2403.18886", "was_revised": true, "final_justification": "Thank the authors for providing justifications and additional experiments. Most of my concerns are addressed. I will raise my final rating.", "rebuttal": "**Q1: The gating module itself is not inherently novel, as similar mechanisms using query-key matching ([1,2,3]) or moe ([4,5]) have been explored in PEFT and CL methods.**\n\n**A1** We acknowledge that the gating module itself is not new, and we do not claim novelty in the structure of the gating module itself. However, what differentiates our approach is the application scenarios, motivation and technical design, particularly within the setting of continual learning (CL) for NLP using LoRA.\n\nFirst, our work focuses on NLP scenarios, but the methods mentioned by the reviewer primarily focus on CV tasks. Due to differences in model architectures and datasets between NLP and CV, prior works [1, 2] show that CL methods being effective in CV may not perform well in NLP. This supports the “no free lunch”principle.\n\nSecond, our motivation is different from existing methods. We show that existing NLP continual learning methods using multiple LoRA branches typically sum them in a naïve and static manner, potentially leading to forgetting. Our method addresses this issue by introducing gating mechanism to dynamically weigh different LoRA branches. This is a different motivation compared to query-key matching methods or MoE methods.\n\nThird, our technical design introduces orthogonal projection for both the initialization and update of the gating module. This ensures that newly learned gating parameters do not interfere with those from previous tasks. To the best of our knowledge, no methods have attempted to impose orthogonal projections as constraints on both the initialization and updating of the gating module.\n\nWe have already cited query-key matching methods and will include additional citations to MoE-based methods mentioned by the reviewer in the final version. We will also clarify these differences more explicitly in the paper, particularly in terms of application scenarios, motivation, and technical design. Thanks for the valuable suggestions.\n\n\n**Q2: The proposed gating module is a collection of task-wise 3-layer MLPs with SiLU activation, operates in parallelly to the LLM backbone, which acts as a task-id predictor. Fig. 3(c,d) reports gating activations only for the most recent (last) task. To fully support the paper’s claims, the authors should also report gating outputs (or task‐prediction accuracy) for each of the previous tasks.**\n\n**A2** We would like to clarify that the goal of the proposed gating module is not to predict the task identity. As shown in Figure 1 of the main text, each LoRA branch has its own gating module, and these gating modules are responsible for determining the weights of their corresponding LoRA branches based on the input samples. The objective of the gating module is to ensure that the output for the current task $t$ is close to 1, and for previous tasks $s$ ($1\\leq s< t$), the output is close to 0. For the input samples from future tasks $u$ ($u>t$), no specific constraint is imposed on the $t$ gating module, since the gating module for task $t$ will not cause any forgetting for future task $u$. This design further distinguishes the gating module from a task-id predictor.\n\nFig. 3(c,d) reports the outputs of the $15$-th gating module across both the 15-th tasks and all previous tasks (task ID<15). In the table below, we report (mean, std) for the outputs of the $t$-th gating modules across old tasks (task ID$<t$) and the current task $t$ on order 1, where $1<t<15$. As shown, gating modules consistently output values close to 1 for samples of the current task and values close to 0 for samples of all previous tasks. We will incorporate these results into the final version in a similar graphical form as Fig. 3(c,d). Thanks for the suggestions.\n\n|GainLoRA (OLoRA)|2|3|4|5|6|7|8|9|10|11|12|13|14\n:-|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nold tasks (task ID$<t$) |(0.14,0.08)|(0.26,0.05)|(0.20,0.10)|(0.22,0.11)|(0.13,0.07)|(0.16,0.09)|(0.12,0.09)|(0.10,0.05)|(0.19,0.09)|(0.12,0.07)|(0.15,0.10)|(0.06,0.04)|(0.15,0.07)\ncurrent task $t$ |(0.94,0.00)|(0.94,0.03)|(0.95,0.01)|(0.93,0.01)|(0.96,0.00)|(0.94,0.00)|(0.95,0.00)|(0.94,0.01)|(0.95,0.00)|(0.92,0.02)|(0.95,0.01)|(0.95,0.01)|(0.92,0.01)\n\n\n**Q3: Weird performance in Table 18, two stronger variants of L2P (DualPrompt and Coda-prompt) both underperform the weaker baseline L2P. The low AP and low FT mean the model fails to learn new knowledge as new tasks are introduced, which suggests that there may exist some errors/bugs in the reimplementation of these methods in the LM domain. As the authors noted, there is no [CLS] token, whereas a similar operation as in Eq.3 should be used for these methods without using the proposed constraints.**\n\n**A3** We would like to clarify that the performance in Table 18 is not due to any bugs in our reimplementation. Instead, it is primarily due to architectural differences between ViT and T5, which make it non-trivial to directly apply prompt-based methods originally designed for ViT (vision Transformer) to language models such as T5.\n\nSpecifically, in our initial implementation of prompt-based methods, we injected prompts into the input embeddings rather than into the keys and values of the attention layers as official implementation of many prompt-based methods in CV. This design choice was made because T5 uses relative position encoding by adding a position bias to the attention scores, while ViT adds position embeddings directly into the input. Injecting prompts to the keys of some Transformer layers increases the attention score length in these layers, causing a shape mismatching with the position bias in T5.\n\nWe re-implemented these methods by injecting prompts into both the keys and values, and introduced zero-padding in the position bias tensor to ensure shape compatibility. Note that we did not add positional information to the prompts, which is consistent with DualPrompt and CODA-Prompt in ViT. Furthermore, as suggested by the reviewer, we generate the query using a 3-layer MLP. The updated results are reported below, showing that both DualPrompt and CODA-Prompt now outperform L2P.\n\n||AP|FT\n:-|:-:|:-:\nL2P|15.23|11.34|\nDualPrompt|17.40|10.63\nCODA-prompt|19.28|14.62\n\nIn addition, we would like to emphasize that the low AP and FT are consistent with prior reproduction efforts for L2P in the LM domain [1] and are not unexpected. Specifically, a recent study [2] has shown that prompt-based methods converge more slowly than LoRA and typically require larger learning rates to be effective. However, in continual learning for LMs, where methods like L2P, DualPrompt, and CODA-Prompt struggle with catastrophic forgetting, using large learning rates can introduce instability. Hence, the learning rate cannot be set to be too large, which leads to relatively low AP and FT. If the learning rate is set to be too large, it will result in a lower AP and a larger FT.\n\nWe will add these results and discuss the key techniques and difficulties for adapting methods from CV to NLP in the final version. Thanks a lot for the insightful comment.\n\n[1] Zhao W, Wang S, Hu Y, et al. SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models, ACL 2024.\n\n[2] Gao Q, Zhao C, Sun Y, et al. A unified continual learning framework with general parameter-efficient tuning. ICCV, 2023."}, {"review_id": "57PKjBgkQH", "reviewer": "Reviewer_fYm8", "summary": "This article proposes a method called GainLoRA for continuous learning of language models. This method aims to solve the problem that existing LoRA-based CL methods easily forget old task knowledge when learning new tasks. By extending a new LoRA branch for each new task and introducing a gating module to integrate the new and old LoRA branches, the new gating module is used to minimize the contribution of the new LoRA branch to the old task, thereby effectively alleviating forgetting and improving the overall performance of the model.", "questions": "1、The GainLoRA method proposed in the article was only experimented on T5 and Llama2. Can more comprehensive experiments be conducted on the latest models such as Qwen3 to verify the generalization of the method?\n2、Have the authors considered verifying the effectiveness of GainLoRA in more complex scenarios, such as scenarios where there is overlap between tasks, rather than just sequentially learning non-overlapping tasks?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1、The article proposes an innovative gating module for integrating new and old LoRA branches. This gating mechanism can dynamically adjust the contribution of the new LoRA branch to the old task, and by imposing specific initialization and update constraints, the impact of the new LoRA branch on the old task when learning the new task is minimized.\n2、The GainLoRA method has good compatibility and can be combined with a variety of existing LoRA-based CL methods, and can be extended to language model architectures of different sizes.\n\nWeaknesses：\n1、The GainLoRA method proposed in the article was only experimented on T5 and Llama2, and lacked experiments on the latest models such as Qwen3.\n2、In GainLoRA, the initialization constraint of the new gating module requires it to be orthogonal to the subspace of the previous t-1 tasks. However, this may cause the model to fail to fully utilize the useful information related to the old tasks in the initial stage of the new task, thereby affecting the learning of the new task or making the model unable to transfer the knowledge of the old task to the new task.", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The GainLoRA method proposed in the article was only experimented on T5 and Llama2, and lacked experiments on the latest models such as Qwen3.**\n\n**A1** In the table below, we present the experimental results of our method on Order 1 using Qwen-3-8B and LLaMA-3-8B. As shown, all methods perform better with Qwen-3-8B and LLaMA-3-8B compared to LLaMA-2-7B. Moreover, our method, GainLoRA, achieves the best performance on both Qwen-3-8B and LLaMA-3-8B. We will include these results in the final version of the paper. Thank you for the suggestion.\n\n|Qwen-3-8B|AP|FT\n:-|:-:|:-:\nOLoRA|44.11|8.85\nGainLoRA (OLoRA)|**54.28**|2.05\nInfLoRA|45.69|9.43\nGainLoRA (InfLoRA)|53.59|**1.80**\n\n|Llama-3-8B|AP|FT\n:-|:-:|:-:\nOLoRA|42.49|12.85\nGainLoRA (OLoRA)|**53.39**|3.56\nInfLoRA|43.27|6.02\nGainLoRA (InfLoRA)|52.18|**1.40**\n\n\n**Q2: In GainLoRA, the initialization constraint of the new gating module requires it to be orthogonal to the subspace of the previous t-1 tasks. However, this may cause the model to fail to fully utilize the useful information related to the old tasks in the initial stage of the new task, thereby affecting the learning of the new task or making the model unable to transfer the knowledge of the old task to the new task.**\n\n**A2** We selected three sentiment analysis tasks (Task363, Task1687, Task875) from the 15 tasks in Order1 and Order2. Intuitively, these tasks belong to the same type and are therefore more similar to each other than to tasks from a different type. We evaluated GainLoRA on these tasks (Task363 → Task1687 → Task875). The results in the table show that while GainLoRA remains effective, its improvement is smaller than that in the 15-task setting with dissimilar tasks. Furthermore, GainLoRA underperforms InfLoRA and O-LoRA on the new task (Task875) but outperforms them on the old tasks (Task363 and Task1687). This suggests that orthogonality constraints might hinder forward transfer. This is a common trade-off in continual learning (CL): constraints help mitigate forgetting but may restrict transfer, particularly for similar tasks. Conversely, weak or no constraints increase the risk of forgetting in dissimilar tasks. Future work will explore adaptive strategies: stronger constraints for dissimilar tasks and weaker ones for similar tasks. Anyway, our GainLoRA is effective in terms of overall (average) accuracy. These discussions and results will be included in the final version of the paper. Thanks for this insightful comment.\n\n||Task363|Task1687|Task875|Avg\n:-|:-:|:-:|:-:|:-:\nOLoRA|85.0|75.3|**79.7**|80.0\nGainLoRA (OLoRA)|90.7|79.3|77.3|**82.4**\nInfLoRA|88.3|75.7|**79.3**|81.1\nGainLoRA (InfLoRA)|**92.0**|78.7|76.3|82.3\n\n\n**Q3: Have the authors considered verifying the effectiveness of GainLoRA in more complex scenarios, such as scenarios where there is overlap between tasks, rather than just sequentially learning non-overlapping tasks?**\n\n**A3** While we consider some alternative scenarios such as the similar tasks discussed in **A2**, we do not include experiments under explicit task-overlapping settings. This is because the form of task overlapping in NLP needs further investigation and clarification. Specifically, we follow existing NLP continual learning setups [1, 2] and assume a single task sequence may involve diverse task types such as summarization and classification. In this case, it remains unclear whether task overlapping should only occur between tasks of the same type, or if it can also happen across different types of tasks.\n\nWe agree that evaluating continual learning methods in overlapping task settings is an important and challenging direction for future research. It requires not only task design but also methodological advances that go beyond catastrophic forgetting mitigation. GainLoRA, as currently designed, focuses on mitigating forgetting and may not be directly effective under overlapping scenarios, which is consistent with the no-free-lunch principle.\n\nIn the final version, we will explicitly discuss this limitation and highlight overlapping tasks as a promising direction for future work. Thanks for the thoughtful suggestion.\n\n[1] Wang X, Chen T, Ge Q, et al. Orthogonal Subspace Learning for Language Model Continual Learning, EMNLP 2023.\n\n[2] Zhao W, Wang S, Hu Y, et al. SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models, ACL 2024."}]

Author rebuttal:

Meta-review: **Summary**

GainLoRA presents a parameter-efficient continual learning framework that addresses catastrophic forgetting in LoRA-based multi-task settings. The method introduces a gated integration mechanism where each new task receives a dedicated LoRA branch, and lightweight gating modules dynamically weight the contributions of different LoRA branches. The core technical innovation lies in orthogonal constraints applied to both initialization and updates of gating modules, ensuring that new task learning minimally interferes with previously acquired knowledge.

**Reason to accept**
- The paper tackles the important challenge of catastrophic forgetting in LoRA-based continual learning, where naive fusion of task-specific adapters degrades multi-task performance—a recognized pain point in the field.

- The orthogonal constraint mechanism for both initialization and updates of gating modules represents a novel approach to ensuring minimal interference between tasks, going beyond existing gating mechanisms in the literature.

- Multiple reviewers praised the paper's clarity and intuitive explanation of the methodology, making it accessible and easy to follow.

**Summarize the discussion**
Author provide good rebuttal:
- Reviewer **5sqb** initially raised concerns about mathematical explanations and task identification mechanisms. The authors provided detailed mathematical proofs for orthogonal projection operations and clarified their instruction-tuning approach for task identification. The reviewer acknowledged these responses and explicitly confirmed a score increase.

- Reviewer **uWep** questioned novelty, gating outputs analysis, and baseline reimplementation issues. The authors clarified their technical novelty through orthogonal constraints, provided comprehensive gating output statistics across all tasks, and corrected implementation issues with prompt-based baselines. The reviewer stated most concerns were resolved and committed to raising their score.

- Reviewer **fYm8** requested evaluation on newer models and raised concerns about orthogonal constraints potentially limiting knowledge transfer. The authors added Qwen-3-8B and LLaMA-3-8B results and provided insightful analysis of the forward transfer trade-off using similar tasks, acknowledging this as a fundamental continual learning challenge.

- Reviewer **KRVe** provided consistently positive feedback throughout, recognizing the method's effectiveness and practical value.
The authors' responses demonstrated deep understanding of their method's limitations while providing substantial additional evidence and clarifications that strengthened the submission.
Decision: Accept (poster)