Title: Gated Integration of Low-Rank Adaptation for Continual Learning  of Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Continual Learning

low-rank adaptation
Abstract: Continual learning (CL), which requires the model to learn multiple tasks sequentially, is crucial for large language models (LLMs). Recently, low-rank adaptation (LoRA), one of the most representative parameter-efficient fine-tuning (PEFT) methods, has gained increasing attention in CL of LLMs. However, most existing CL methods based on LoRA typically expand a new LoRA branch to learn each new task and force the new and old LoRA branches to influence old tasks equally, potentially leading to forgetting. In this work, we propose a new method, called gated integration of low-rank adaptation (GainLoRA), for CL of LLMs. GainLoRA expands a new LoRA branch for each new task and introduces gating modules to integrate the new and old LoRA branches. Furthermore, GainLoRA leverages the new gating module to minimize the influence from the new LoRA branch to old tasks, effectively mitigating forgetting and improving the model's overall performance. Experimental results on CL benchmarks demonstrate that GainLoRA outperforms existing state-of-the-art methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes a method called GainLoRA for continuous learning of language models. This method aims to solve the problem that existing LoRA-based CL methods easily forget old task knowledge when learning new tasks. By extending a new LoRA branch for each new task and introducing a gating module to integrate the new and old LoRA branches, the new gating module is used to minimize the contribution of the new LoRA branch to the old task, thereby effectively alleviating forgetting and improving the overall performance of the model.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1、The article proposes an innovative gating module for integrating new and old LoRA branches. This gating mechanism can dynamically adjust the contribution of the new LoRA branch to the old task, and by imposing specific initialization and update constraints, the impact of the new LoRA branch on the old task when learning the new task is minimized.\n2、The GainLoRA method has good compatibility and can be combined with a variety of existing LoRA-based CL methods, and can be extended to language model architectures of different sizes.\n\nWeaknesses：\n1、The GainLoRA method proposed in the article was only experimented on T5 and Llama2, and lacked experiments on the latest models such as Qwen3.\n2、In GainLoRA, the initialization constraint of the new gating module requires it to be orthogonal to the subspace of the previous t-1 tasks. However, this may cause the model to fail to fully utilize the useful information related to the old tasks in the initial stage of the new task, thereby affecting the learning of the new task or making the model unable to transfer the knowledge of the old task to the new task.\n\nQUESTIONS:\n1、The GainLoRA method proposed in the article was only experimented on T5 and Llama2. Can more comprehensive experiments be conducted on the latest models such as Qwen3 to verify the generalization of the method?\n2、Have the authors considered verifying the effectiveness of GainLoRA in more complex scenarios, such as scenarios where there is overlap between tasks, rather than just sequentially learning non-overlapping tasks?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GainLoRA, a parameter‐efficient continual‐learning method for language models that builds on the popular LoRA framework. For each new task, GainLoRA adds a new low‐rank adaptation branch and integrates old and new branches via lightweight, task‐specific gating modules.  By imposing orthogonality constraints at initialization and during updates, GainLoRA drives the new branch’s contribution toward zero on previously learned tasks, thereby mitigating catastrophic forgetting while allowing full expressivity on the current task.  Extensive experiments on SuperNI and Long Sequence benchmarks, across multiple task orders and model scales are conducted.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The presentation is straightforward and easy to follow.\n2. The experiments and ablations are comprehensive, addressing key concerns.\n3. The proposed GainLoRA can be plugged into other existing LoRA-based methods.\n\nWeaknesses\n1. The gating module itself is not inherently novel, as similar mechanisms using query-key matching ([1,2,3]) or moe ([4,5]) have been explored in PEFT and CL methods.\n2. The proposed gating module is a collection of task-wise 3-layer MLPs with SiLU activation, operates in parallelly to the LLM backbone, which acts as a task-id predictor. Fig. 3(c,d) reports gating activations only for the most recent (last) task. To fully support the paper’s claims, the authors should also report gating outputs (or task‐prediction accuracy) for **each** of the previous tasks.\n3. Weird performance in Table 18, two stronger variants of L2P ( DualPrompt and Coda-prompt) both underperform the weaker baseline L2P. The low AP and low FT mean the model fails to learn new knowledge as new tasks are introduced, which suggests that there may exist some errors/bugs in the reimplementation of these methods in the LM domain. As the authors noted, there is no [CLS] token, whereas a similar operation as in Eq.3 should be used for these methods without using the proposed constraints.\n\n[1] Learning to Prompt for Continual Learning https://arxiv.org/abs/2112.08654\n\n[2] DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning https://arxiv.org/abs/2204.04799\n\n[3] S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning https://arxiv.org/abs/2207.12819\n\n[4] CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning https://arxiv.org/abs/2211.13218\n\n[5] Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning https://arxiv.org/abs/2403.18886\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a novel method to mitigate catastrophic forgetting in continual learning (CL) using language models (LMs). Specifically, the authors introduce gated integration of LoRA (GainLoRA), which will adaptively control the output of a set of LoRA modules corresponding to different tasks such that they do not interfere with each other. GainLoRA achieves this by imposing constraints on the initialization and the update of the gating module.  Thus, this approach does not require samples from prior tasks or task labellings to mitigate the interference of the LoRA adapters that are irrelevant to a given task at the inference time. The empirical results provided in the paper suggest that incorporating the proposed gating mechanism with the existing LoRA-based CL methods can improve their ability to mitigate catastrophic forgetting and improve the average accuracy across tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The paper discusses the problem of catastrophic forgetting in LMs in the CL setting, which is a critical and pertinent problem in the field of machine learning.\n* The flow of the paper is easy to follow, and the proposed method is introduced intuitively.\n* The ablation studies provided in the paper validate the importance of the different components of the proposed method\n* The empirical results provided in the paper suggest that the performance of existing methods designed for CL can be significantly improved by the proposed method. \n\n**Weaknesses**\n\n* Some key information needed to follow the paper seems to be missing in the main text. For example, some more explanation on why the operation in (9) makes the initialization to satisfy the constraint (5), in addition to providing a reference to prior work.\n\n* It is unclear how only input can be used to determine the identity of the task. For example, the same input can be used for a text generation task (e.g., summarization) and a classification task. How can the proposed method mitigate the task interference in a situation like this?\n\n**Minor Weaknesses**\n\n* Line 127: $p_{L+1}$ -> $p_{L}$?\n* Although the main text points to the supplementary for some key definitions and metrics, the supplementary is not provided adjacent to the main text, which makes the paper less self-contained.\n\nQUESTIONS:\n* Can the authors provide more discussion/explanation on why the updates (9) and (10) allow the initialization and the update to satisfy the constraints (5) and (6), respectively?\n\n* How can the gating model assign the correct weight to the LoRA output when the input distribution is similar accross the tasks, whie the output is different?\n\nLIMITATIONS:\nPlease refer to the questions above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

comments: SUMMARY:
This paper addresses the challenge of parameter-efficient fine-tuning in multi-task settings, where naively combining task-specific LoRA adapters often leads to performance degradation. It introduces a gated integration framework called GLoRA, which enables dynamic weighting over pre-trained LoRA experts using a lightweight gating module. GLoRA can be plugged into existing PEFT workflows and improves both multi-task and continual learning performance, without requiring retraining or task-specific annotations.

STRENGTHS AND WEAKNESSES:
This paper tackles the pain point that naïve fusion of task-specific LoRA adapters hurts multi-task performance, and puts forward GLoRA—a gated integration scheme that 
(1) keeps the LoRA parameter budget tiny, 
(2) lets a lightweight gate dynamically weight experts at run-time, and 
(3) plugs straight into existing PEFT pipelines without re-training the backbone. The authors back the idea with broad experiments on ten diverse NLP tasks, three foundation models (T5-Base, OPT-1.3B, LLaMA2-7B), plus ablations on gate order and continual-learning settings, demonstrating consistent gains and limited forgetting. Implementation is simple (just an extra MLP and softmax), code-path changes are minimal, and the paper provides clear analyses of parameter/compute overhead.

QUESTIONS:
none

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05