Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning
Authors: Xiao Han
ZIMO ZHAO
Wanyu Wang
Maolin Wang
Zitao Liu
Yi Chang
Xiangyu Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Continual Learning
Parameter-Efficient Fine-Tuning
Large Language Model
Low-Rank Adaptation
Lifelong Learning
Primary area: deep_learning
TL;DR: We propose DEAL, a continual low-rank fine-tuning framework that enables efficient and privacy-preserving adaptation of large language models.

Abstract: Recent advancements in Large Language Models (LLMs) have emphasized the critical role of fine-tuning (FT) techniques in adapting LLMs to specific tasks, especially when retraining from scratch is computationally infeasible. Fine-tuning enables LLMs to leverage task- or domain-specific data, producing models that more effectively meet the requirements of targeted applications. However, conventional FT approaches often suffer from catastrophic forgetting and suboptimal data efficiency, limiting their real-world applicability. To address these challenges, this paper proposes DEAL, a novel framework that integrates Low-Rank Adaptation (LoRA) with a continuous fine-tuning strategy. By incorporating knowledge retention and adaptive parameter update modules, the framework mitigates the limitations of existing FT methods while maintaining efficiency. Experiments on 15 diverse datasets show that DEAL consistently outperforms baseline methods, yielding substantial gains in task accuracy and resource efficiency. These findings demonstrate the potential of our approach to advance continual adaptation in LLMs by enhancing task performance while improving resource efficiency. The source code is publicly available at https://github.com/Applied-Machine-Learning-Lab/DEAL.

Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning
Xiao Han"
Zimo Zhao"
Zhejiang University of Technology, Zhejiang Key Laboratory of
City University of Hong Kong
Hong Kong, China zmzhao6-cQmy cityu edu hk
Visual Information Intelligent Processing Hangzhou, China hahahenha@gmail. com
Wanyu Wang* City University of Hong Kong Hong Kong, China
Maolin Wang
City University of Hong Kong
Hong Kong, China edt.yuuudu..ee edu. hk Morin. wang@my. cityu.
wanyuwang4-comy cityu. edu. hk
Zitao Liu
Yi Chang
Xiangyu Zhao
Jinan University
Jilin University
City University of Hong Kong
Jinan, China
Jilin, China
Hong Kong, China
Liuzitao@jnu. edu. cn
yichangejlu. edu.
xianzhao@cityy edu. hk
Abstract
Recent advancements in Large Language Models (LLMs)) have emphasized the critical role of fine-tuning (FT) techniques in adapting LLMs to specific tasks, especially when retraining from scratch computationally infeasible. Fine-tuning enables LLMs to leverage task- or domain-specific data, producing models that more effectively meet the requirements of targeted applications. However, con- ventional FT approaches often suffer from catastrophic forgetting and suboptimal data efficiency, limiting their real-workd applicability. To address these challenges, this paper proposes DEAL, a novel framework that integrates Now-Rank Adapta- tion (LoRA) with a continuous fine-tuning strategy. By incorporating knowledge retention and adaptive parameter update modules, the framework mitigates the limitations of existing FT methods while maintaining efficiency. Experiments on 15 diverse datasets show that DEAL consistently outperforms bascline methods, yielding substantial gains in task accuracy and resource efficiency. These find- ings demonstrate the potential of our approach to advance continual adaptation in LLMs by enhancing task performance while improving resource efficiency. The source code is publicly available at https://gitubb.com///piiccc Mwwpeiiiddd-aai---eeeeeeeee Learning.
Lab/DEAL
1 Introduction
The advent of Large Language Models (LLMs) has catalyzed transformative advances in Natural Language Processing (NLP), enabling breakthroughs across healthcare, education, web technologies. and other domains [15].. However, training and utilizing these models to evolve to real-world
"Equal contribution.
t "oorrsppnniigg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

demands remains a critical challenge. Direct finettuning of billion-scale seaeonssslll--aaaaaaea parameter models incurs prohibitive computational costs, creating accessibility barriers for esouure-constrainee researchers and institutions. Even for small and medium-sized enterprises, it is also difficult to independently implement pre-training of these models. In such scenarios, Aerameter-Fffiiien Fine-Tuning (PEFT) methods-partionn Low-Rank Adaptation (LoRA)- have emerged as practical solutions that leverage pre-trained models by "standing on the shoulders of giants" [6 7 By applying low-rank matrix decompositions, LoRA reduces the number of trainable parameters by over 90% while maintaining baseline performance. By selectively updating task-specific subspaces, LoRA enables targeted knowledge integration in non-stationary environments a capability aligned with the core objectives of continual learning.
As LLMs require ongoing updates to ensure their knowledge remains current and relevant over time, continual learning of LoRA is required to integrate new information while preserving existing capabilities [8] This approach mitigates catastrophic forgetting by freezing most parameters and restricting updates to a low-rank matrix, allowing LoRA to differentially activate specific knowledge within the model. For instance, a LoRA-based model trained on Wiki-QA [9] can be further refined on TruthfulQA [I0] to enhance performance. However, while continual learning of the LoRA module only unlocks up-to-date task-specific outcomes, it may compromise cross-domain performance Α. particularly when smaller, specialized datasets lack the breadth of the original pre- training data [12 Therefore, could we design a high-level fine-tuning method that maintains excellent performance across all tasks while allowing continuous fine-tuning with small-scale datasets?
Several existing studies have explored this problem. In particular, continuous learning can be achieved through two main strategies: (1) direct model editing, and (2) introducing additional adapters. On the one hand, studies [1318] have shown that fey-value-lite Mikuee-kkeeea structures in the Transformer layers can be directly edited. For example, ROME [33] and MEMIT [14] directly update the key-value-like structures in Transformer layers via causal weight interventions. However, these approaches require massive additional experiments to pinpoint which neurons to edit, making it both inefficient and costly, On the other hand, for A-RA-based LLMs, the process of locating and modifying parameters in the low-rank matrix remains largely opaque [19]. To reduce the complexity of targeting specific parameters, many studies turn to stack additional adapter modules (3) Yet, these modules inevitably impose extra computational overhead. Consequently, interpretability and efficiency remain the two major challenges in applying continuous learning LoRA-tuned L.Mss.
To address these limitations, we introduce Eata-Efficient Adaptation via continuous Low-rank fine-tuning (DEAL). a method that facilitates efficient knowledge acquisition while preserving the interpretability of model updates. Specifically, we design a wavelet kernel to adaptively preserve core features of historical knowledge in the filtered low-rank matrix while seamlessly incorporating new information. By focusing on core aspects of historical knowledge, DEAL prevents catastrophic forgetting, thus maintaining model performance across multiple tasks. This innovative approach provides a robust framework for continuous learning, making it an effective solution for dynamic data environments. Our contributions of the paper summarized as follows:
We introduce DEAL, an innovative continual learning framework that efficiently utilizes small amounts of new data for continuous learning, thereby avoiding the need for relearning and signifi- cantly conserving computing resources.
2. We leverage a wavelet kernel to preserve historical knowledge and deploy differentiated regulariza- tion terms to control the knowledge updating process, improving both transparency and efficiency. Additionally, by simply replacing the original low-rank matrices with their fine-tuned counterparts, DEAL ensures that inference time remains unchanged.
3. Comprehensive experiments on 15 multi-task open-source datasets validate the effectiveness and efficiency of our framework. These experiments demonstrate its ability to maintain high performance across different tasks while efficiently managing computational resources.
2 Preliminaries
In this section, we give the definitions of I LoRA-based L.M at first and then introduce the continual learning for LoRA fine-tuning. Finally, state the problem that we solve in this paper.

---

Answer
A+1
A Task-specific Dataset
B
Pre-trained LLM
DEAL Model
Wavelet Kermel-based
Knowledge Retention
Controlled Knowledge
Query
Updating
LoRA-based LLM
B
Frozen
Trained
Continual Learning Procedure
Figure 1: The framework overview.
LoRA-based LLM. LoRA enhances LLM by introducing low-rank matrices to weight updates during fine-tuning. Given pre-trained weight matrix W E mmxx LoRA decomposes the weight update AW into two smaller matrices A E |R and B E IR*x,, where is a user-defined rank (r << min m, n n.}). The update is then expressed as AW = A x B allowing the model to adapt new tasks with significantly fewer parameters compared to full fine-tuning, This decomposition reduces the number of trainable parameters from m to m + n X r, leading to substantial computational savings.
Continual LoRA Fine-Tuning. The parameter-efficien fine-tuning approaches of LoRA enable LLMs learn new tasks while retaining performance previously learned tasks. By updating only the low-rank matrices A and B, the LoRA-based LLM mitigates catastrophic forgetting, a common challenge in continual learning, by enabling the model to learn tasks while retaining performance on prior tasks.
In this paper, aim effectively learn tasks with acceptable training cost, while retaining performance on previously learned tasks. Mathematicallyy this involves adjusting the weight matrix W of the model Aw by introducing low-rank updates AW = A x B where A and B are low-rank matrices. The learning target is to minimize the loss function L over the new task data Deew, subject to a regularization term that penalizes changes to the original parameters to prevent catastrophic forgetting. This can be formulated
min L (AW+AxB", Drew) + A/A BT,, A.B
where À is a hyperparameter controlling the trade-off between learning new information and retaining prior knowledge. The goal is find the optimal low-rank matrices A and B that allow the model to adapt new tasks while preserving its performance previous tasks.
3 Methodology
In this section, we first provide a framework overview of DEAL and then we introduce each part of DEAL in detail. Finally, we outline the training procedure, which fine-tunes the low-rank matrices to ensure the model adapts to new tasks without sacrificing performance previously learned tasks.
3.1 Framework Overview
Figure I shows the overall framework for continuous learning on Nomain--ppecine dataset, which consists of wavelet kernel-based knowledge retention module and controlled knowledge updating module. (a) Wavelet Kernel-based Knowledge Retention Module extracts and filters singular values from the low-rank matrices to preserve the core representations of historical knowledge, which should be maintained throughout continual learning. (b) Controlled Knowledge Updating Module

---

applies higher-ordet regularization to constrain parameter updates in LoRA, thereby regulating the integration of new knowledge while minimizing disruption to previously learned representations. In LoRA, the small-parameter knowledge representation embedded in the low-rank matrix could activate the corresponding understanding and reasoning abilities in the original LLM,, facilitating the learning of new tasks. These two modules allow the model to effectively learn and incorporate new information without disrupting previously acquired knowledge. As for the inference phase, the updated low-rank matrix, with its new knowledge representation, will directly replace the corresponding part in the original LoRA module, ensuring that the inference delay of DEAL remains unaffected.
3.2 Wavelet Kernel-based Knowledge Retention
Due to limited hardware resources, continual learning with LoRA-based LLM typically involves fine- tuning only the LoRA module, while the pre-trained LLM parameters remain unchanged. However, updates to the low-rank matrix can increasingly hinder the model's ability to retain the original features crucial for activating the corresponding capabilities within the LLM. To address this, we introduce a wavelet kernel filter and preserve the key features of LoRA during continual learning.
In the LoRA module, the matrices A and B are singular, meaning they are not full-rank. This presents challenge for effective feature extraction. We assume that the singular matrix Y A or B can be decomposed into a task-relevant component X and a redundant or noisy component D, ie., Y X D.
Here, X denotes the core feature matrix, capturing the intrinsic low-rank structure that encodes task-relevant semantics. According the Sckart-Young--We approximation of a matrix the Frobenius norm sense achieved via truncation of its singular value decomposition (SVD). This motivates our use of truncated SVD to estimate X from the observed matrix Y, with the goal of recovering task-relevant features from its singular representation.
theorem (24), the best low-rank
We further assume that both Y and X lie in R**", sharing the same dimensions. Their corresponding singular value decompositions (SVD) can given by:
Y Y= PI P2 (s.
S
0
Q
m1
(2)
S
#2
X UVEVV
Σn
) 0
0
Vzl
(3)
UU1 U
V.2
where Un1 E X== Un E R2 (m. Val E Rr Xr Va2 E (" (Tr
The following theorem states that we cannot directly compute the core features of X from Y:
Theorem Let Y be the observed data matrix and X the underlying core feature matrix. Then, without additional constraints, there does not exist a pair of matrices P1 and Un such that U .. See Appendix A.2
Therefore, we aim to recover the core feature matrix X by representing it as linear combination of the columns of the observed matrix Y. To this end, we introduce a coefficient matrix H and formulate the following least-squares objective:
min YH X P:
where ||- denotes the Frobenius norm. In this formulation, X is treated as the target (e.g., s., the ideal low-rank component), and H is the optimization variable that linearly combines the basis vectors in Y to approximate X. The optimal H can be derived as:
H ( Y
(5)
Then, X, the minimum variance estimate of X, can be presented as:
(YTY) -1 TTX,
x=
Y=YH YH
(6)

---

Obviously, cannot calculate X directly by Eq. 6. Note that Py Y (xTx) -1
is the orthogonal projection operator onto the column space of Y. Importantly, X is denoised when the target X lies entirely within that subspace. To proceed with estimation, we assume that the redundant
Y
features in the matrices A and B behave as white noise, i.c., D D'D IL. D 0, where ob the variance of the noise. Then we can simply X:
of O
X
4
where uy and Vy is the left and right singular vectors of Y, JF is the k-th largest singular value of Y (01 > d2 > ... > o,). In traditional signal analysis algorithms, large singular values represent low-frequency data distribution and macro trends, and small singular values represent frequeeyy disturbances [25]. However, since o² in the above formula is unknown, we will define a series of the heat kernel low-pass
wavelet functions at different scales for feature filtering. Here, filter:
1 X cg113 c 203
(,,,,(()) = exp
(8)
where cj is the center of the j-th kernel, of represents the width of the kernel. By setting series of different heat kernel widths o² [oî o.. ... J. defining a series of learnable diagonal matrices
91.92 -.. and learnable centers C = [c1. C2 ... I.TT. we can define a wavelet neural network to extract the features of X from Y:
H H
{
Σ
H 03.o 100.5
where 6(-) is the activation function, HD := and X = Hh Here K is the total number of layers
and & {0, 1,... KK- 1}. As ó the heat kernel, i.c. o o=((x) d(-x)), we could simply Eq. ( to be:
H (
oz H
(10)
Eq.
10 avoids directly calculating the inverse of the function and improves calculation efficiency.
3.3
Controlled Knowledge Updating
In the previous section, derive the core features of the original knowledge. In this section, our goal is to integrate the new knowledge into the LoRA module. The new knowledge can be categorized into two parts: one that does not overlap with the original knowledge, and the other that requires updates due to the outdated nature of the original knowledge. The latter is the primary focus, as it may lead to slight changes in the core features. However, these changes should not alter the overall impact of the original knowledge.
To achieve this goal, constrain the update of parameters while learning new knowledge through an MLP:
O++l g (H++1)
(11)
= witt H + +b+t1).
where OK is the updated matrix A' or B¹, 5'()) is the activation function, w E Ω and bt E B are learnable paramters. Defining LoRA-based LLM with pre-trained parameters W and LoRA parameters AW as Aw.aw("), then the loss function is defined as follows:
Coss MSE (Aw,aw AW (Q), G)
(12)
+11110 - + A21102118
where AW' A' x B' is parameters of the updated LoRA, G is the ground truth in the new dataset, Q the input query correspondingly, X and Xa are two regularization hyperpa- rameters, 0 = {g, C}. O2 == 1St B} are the learnable parameters in the model, and b
5

---

are tworegularization orders which satisfies 2 b. By ensuring that the regularization order for the knowledge retention module's parameters is at least as high as that used for the knowl- edge update module's s regularization term, we can minimize the adjustments made to the reten- tion model's parameters. This, in turn, enhances the overall generalization ability of the model.
Algorithm 1
The DEAL framework
3.4 Training Procedure
Input: Pre-trained LoRA-based LLM Aw.aw.
In this subsection, we introduce the training pro- cess of DEAL, as shown in Algorithm I We
training samples 2>
Output: Fine-tuned model Aw,aw'
begin by locating and obtaining the low-rank ma- trices A and B in LoRA (lines 1-3). Next, we apply wavelet kernel-based neural networks to extract the core features of historical knowledge from A and B (line 5). Then, another neural net- work is introduced to superimpose features from new knowledge onto these core features, result-
AW' +- AW
2: for each batch (Q, G) E E do
3: Extract A, B from AW'
4:
for each Y E AA B) do
5:
X < Eq. 10 0 < Eq.
6:
if Y
A
then
7:
A'
O
ing in a newly constructed low-rank matrix A' or B'' (lines 6-11). Using this updated low-rank matrix, we recalculate the LoRA parameters to obtain the fine-tuned model. By calculating the loss with regularization terms and performing pack-propagation, ensure that the model can learn new knowledge while preserving historical features in controlled manner (lines 13-15).
else 9:
8:
B' B' 0
10:
end if
11:
end for
12:
AW" +- A' B'
13: Compute loss Loss using Eq. 12
14: Update parameters via back-propagation
15:
end for
16:
return Aw,aW'
Experiments
Datasets. We evaluate DEAL on three continual learning (CL) benchmarks in sequential task setup, where data from previous tasks is unavailable during training on subsequent ones. These benchmarks are designed to evaluate key challenges in continual learning, including: (i) catastrophic forgetting in semantically related tasks, (ii) inefficient data utilization resulting from restricted access to prior samples, and (iii) scalability across long and diverse task sequences.
(i) Same-domain tasks: We use a three-task benchmark consisting of AG News (news classification), DBpedia (entity typing), and Yahoo Answers (question topic prediction). This setup evaluates DEAL's ability to mitigate catastrophic forgetting by retaining transferable knowledge across semantically similar tasks.
(ii) Domain-shift tasks: To introduce domain variability, we augment the benchmark with Amazon Reviews (26) for binary sentiment classification. This domain-shift setting assesses generalization under limited data access and distributional shifts, reflecting practical constraints in real-world continual learning.
(iii) Heterogeneous multi-task learning: We evaluate on the 15-task benchmark proposed by [27]. which spans text classification (AG News, DBpedia, Yahoo, Amazon, Yelp), GLUE tasks (MNLI,
QQP, RTE, SST-2) (28). SuperGLUE tasks (WiC, CB, COPA, MultiRC, BoolQ) [29, and IMDB [30]. This benchmark tests DEAL's 's scalability and robustness across heterogeneous tasks and long task sequences. Full details on dataset preprocessing and prompt construction are provided in Appen- dices B. l andd]
Baselines.
We compare DEAL against three LoRA-compatio continual learning baselines:
SeqL.RR: A naive baseline that sequentially updates a single fixed-size LoRA adapter across tasks, without any mechanism to mitigate forgetting.
A recent method that allocates task-specific adapters to orthogonal subspaces, reducing parameter interference.
O-LoRA 31:
PerTaskFT: An oracle baseline in which each task is fine-tuned using separate LoRA adapter without sharing. Although impractical for deployment, serves as an upper bound on task-specific retention.

---

All baselines are evaluated under identical settings, including a fixed adapter architecture, optimization protocol, and tokenization, to ensure fair comparison. Replay-based and non-LoRA methods are excluded to prevent confounding effects from architectural or memory differences. Implementation details are provided Appendix [.2 and comparisons with broader classes methods are reported in Appendix G
Implementation Details. We use two pretrained models: LLaMA 3.1-8B 32 a decoder-only model fine-tuned on dnstruction-follwwee corpora, and 15-Large-Instrucc [33. an encoder-decoder model adapted for general-purpost instruction following. All experiments are conducted on a single NVIDIA A10 GPU. LoRA-based continual fine-tuning applied both models using a fixed adapter rank, dropout, learning rate, and batch size across tasks. Gradients are masked as needed to enforce parameter sparsity. Unless stated otherwise, hyperparameters remain consistent across DEAL and all baselines. Full training configurations, hardware details, and random seed settings are provided in Appendix B.2
Evaluation Metrics. We report Average Accuracy (AA), the mean test accuracy across all tasks after training concludes, and ROUGE-1 (R-1), which measures unigram Fl overlap between gener- ated outputs and ground-truth labels for free-form generation tasks. Formal metric definitions and application contexts are detailed in Appendix [ We evaluate the training and inference efficiency of DEAL in Appendix
4.1
Main Results
Table D summarizes the continual learning performance of all methods on the three benchmark suites: the 3-task Text Classification (TC), the 4-task Standard CL benchmark, and the 15-task Large-Scale benchmark.
Across all tasks and model backbones, our proposed method, DEAL, consistently outperforms both SeqLoRA and O-LoRA, and achieves performance comparable to the oracle upper bound (PERTASKFT) in terms of average accuracy (AA) and ROUGE-L Fl (R-1). On the 4-task benchmark with T5-Large, DEAL achieves 78.5% Average Accuracy (AA) and 82.5% ROUGE-I (R-1), com- pared 44.6%/44.66% for SeqLoRA and 71.2% for O-LoRA. These improvements stem from DEAL's ability to balance knowledge retention and transfer more effectively. SeqLoRA lacks mecha- nisms for preserving prior knowledge, leading to severe forgetting. O-LoRA mitigates forgetting via orthogonal subspace constraints but limits beneficial cross-task transfer. In contrast, DEAL integrates shared LoRA modules with dual-branch adapters and stability-award regularization, enabling robust adaptation while preserving task-specific information.
Table 1: Continual learning performance across three benchmarks.
Method
3-Task (TC)
4-Task (Standard)
15-Task (Large)
AA
R-1
AA
R-1
AA
R-1
+ SegLoRA
$2.4
52.8
44.6
44.6
42.1
44.0
T5 + O-LoRA
85.2
87.1
71.2
73.3
70.8
80.3
T5 + PerTaskFT
90.3
91.7
70.0
73.0
76.5
78.2
T5 DEAL(ours)
87.7
89.3
78.5
82.5
73.9
79.1
LLaMA + SeqLoRA
54.1
55,9
47.6
54.8
45.2
53.2
LLaMA + O-LoRA
86.4
88.1
75.3
80.8
73.2
77.4
LLaMA + PerTaskFT
88.2
90.0
77.5
79.4
77.1
82.5
LLaMA + (EAL()urs)
88.9
90.2
78.9
81.3
74.6
78.9
Note. Bold indicates statistically significant improvements
(i.e., two-sided t-test with p < 0.05) over the best baseline.
Notably, DEAL approaches the performance of the oracle baseline, PerTaskFT, which fine-tunes each task independently without parameter sharing. On the 3-task benchmark with T5-Large, DEAL achieves 87.7% AA, closely matching the 90.3% attained by PerTaskFT. This near-oracle performance is achieved with significantly lower computational overhead. By leveraging modular adapters and regularized updates, DEAL retains -isediiiriiintevve signals while benefiting from shared representations, offering scalable and efficient alternative to task-isolater fine-tuning.
7

---

75.6
75.6
AA(%)
74
72.44
74
72
(7
65.2
724
70
$4.8
68
70
Onty Adspter A
Oaly Adupter B
Burhh A
Order
Order 1
Order 3
LoRA Rank
(4) Adapter Upd ate Strategy
Task Order Sunsifivity
(c) Effect of LoRA Rank
Figure 2: DEAL ablations: (a) adapter update strategy, (b) task-order robustness the 4-task benchmark, (c) LoRA-rank sensitivity on the 3-task benchmark.
The advantages of DEAL become increasingly prominent task complexity grows. On the 15-task benchmark with LLaMA-3.1 1-B. DEAL achieves 74.6% AA, outperforming SeqLoRA by more than 29 percentage points and surpassing O-LoRA well. In long-horizon continual learning settings, catastrophic forgetting compounds across tasks. DEAL mitigates this degradation through regularization- -uided updates and flexible routing, demonstrating strong scalability and robustness to extended task sequences.
Ablation Studies
We conduct ablation experiments to assess the contribution of five core components DEAL: (i)
adapter update strategy, (ii) task-order robustness, (iii) LoRA rank Figure (iv) the kernel function (Table (3), and (v) the regularization strength (Table Unless otherwise specified, all experiments the T5-Large backbone with fixed random seeds and data orderings.
Adapter Update Strategy. We evaluate three adapter update strategies: updating only Adapter A, updating only Adapter B, and updating both jointly. Joint updates achieve the highest post-tase bast-task Average Accuracy (AA) of 75.6%, exceeding single-branch updates by 2.8 5.4 percentage points. Across multiple runs, updating only Adapter A consistently outperforms updating only Adapter B. This result reflects the continual learning setting: Adapter A governs global projection directions that capture generalizable semantic patterns and reasoning structures, whereas Adapter B primarily encodes task-specific features. Joint training of both adapters therefore enables the most effective integration of broad knowledge transfer with task specialization.
Task-Order Robustness. To simulate non-stationary task arrivals, we evaluate three random permutations of the 4-task sequence, following the setup in O-LoRA [31]. This experiment aims to examine how sensitive DEAL is to variations task order a crucial factor in realistic continual learning, where the order of tasks is typically unpredictable. Across the three permutations. the average accuracy (AA) ranges narrowly from 73.1% % to 75.6%, reflecting a fluctuation of less than three percentage points. This low variance demonstrates that DEAL maintains stable performance regardless of task order, indicating strong robustness and adaptability to dynamic, non-stationary learning environments. The specific task permutations used detailed in Table 2
Table 2: Task sequences used for continual learning evaluations.
Order
Task Sequence
DBpedia Amazon Yahoo + AG News
DBpedia - Amazon - AG News - Yahoo
Yahoo - Amazon
AG News DBpedia
Using the 3-task benchmark, we vary the LoRA rank across 4, 8, 16, 32. Accuracy improves significantly from rank to rank 8 (71.5% to 84.3%), then saturates (84.5% at rank 16.
LoRA Rank.

---

84.6% at rank 32). These results suggest that a compact rank-8 configuration captures most of the task-specific variation while offering strong efficiency in both memory and computation.
Kernel Functions. To further support the selection of the heat kernel, compared it against two alternatives- f() = xe- and quadratic splines- results summarized in Table 3 All three kernels achieve comparable accuracy, indicating that the model's representational capacity is relatively insensitive to the specific kernel choice. The heat kernel's 's primary advantage lies in its computational efficiency: it consistently yields sndees-of-maggituee reductions in runtime.
This improvement stems from its inverse-free update rule (Eq. (T0 which avoids expensive matrix inversions while maintaining numerical stability. These results demonstrate that the heat kemnll offers a favorable trade-off between simplicity and efficiency, supporting its adoption as the default kernel throughout our experiments.
Table 3: Comparison of different kernel functions.
Kernel Function
Average Accuracy (%)
Training Time (ms/sample)
(a) = xe
78.4
1759
Quadratic Splines Heat Kernel (Ours)
78.3
1291
78.5
56
Regularization. We conduct grid search over reg- ularization weights (a, b), where regulates the reten- tion branch (wavelet-based) and governs the adap- tation branch (MLP-basede The search ranges over
Table Grid search over asymmetric regular-
ization weights (a, b).
b
AA (%)
a E 1, 10, 20 and E 1,2,5, capturing a vari- ety of retention-to-adappiee This ablation aims to identify the optimal trade-off between preserving prior knowledge and facilitating new task adaptation.
1
74.8
penalty combinations.
1
83.9
10
2
85.5
10
5
84.1
20
2
82.7
As summarized in Table 4 the best performance is obtained with (a 10, 2). reaching an after- task accuracy (AA) of 85.5%. This configuration outperforms both low-penalty settings (e.g.. (1)
Note. Bold indicates statistically significant im- provements (i.e. two-sided t-test with < 0.05) over the best baseline.
and heavier regularization schemes (e.g., (10, 5) or (20, 2)). These results suggest that appropriately tuning the balance between the retention and adaptation branches is critical: a moderately stronger emphasis on knowledge retention improves stability, while maintaining flexibility in the adapta- tion pathway supports effective learning of new tasks. Notably, this balance mitigates forgetting without hindering new knowledge acquisition, highlighting the importance of carefully calibrated regularization in continual learning.
6
Related Work
Continual Learning for Large Language Models Continual learning (CL) with large language models (LLMs) presents a fundamental challenge: acquiring new knowledge without catastrophic forgetting, while ensuring efficiency and adaptability. Existing approaches largely fall into three categories: memory-based regularization-
and subspace isolation-based methods.
Memory-based approaches, such as experience replay (FR) 34 mitigate forgetting by revisiting buffered past data. However, these methods raise concerns about scalability and data privacy, limiting their practicality in real-world LLM deployments. Regularization-basee methods constrain parameter updates to preserve previously learned knowledge. For instance, CLoRA [8] introduces angular regu- larization between task ssss--pcifiic LoRA adapters. While computationally lightweight, such approaches often under perform on dissimilar tasks due to overly restrictive adaptation dynamics. Subspace-basse techniques offer memory free alternative by decoupling task representations. O-LoRA [31], for example, updates LoRA parameters within orthogonal subspaces of prior tasks, effectively reducing representational interference. However, it faces two key limitations: (i) orthogonality is enforced only in first-order gradient space, overlooking higher-order interactions; and (ii) subspace directions are treated uniformly, with prioritization of semantically meaningful components. The TRACE

---

benchmark (35] further highlights trade-offs in CL for LLMs: high per-task accuracy, suffers from severe forgetting; in contrast, naive LoRA tuning preserves general capabilities but degrades instruction-following performance.
while full- tull-parameter fine-tuning yields
In contrast prior work, our method introduces structural regularization and selective routing, aligning task-specific updates with semantically salient directions while preserving cross-task gener- alization. This design promotes transferability and robustness without relying on external memory or imposing rigid parameter constraints.
Parameter-Efficien Tuning Parameter efficient tuning (PET) approaches, such as adapters [36-- 38), prompt tuning 39 40), and LoRA [6 4T] reduce training cost by updating only a small subset of model parameters. Several extensions of LoRA have been proposed to improve adaptability and representational capacity. ReLoRA [22 enables high-rank representations by scheduling low- rank updates across training phases. FLORA[4] employs stochastic resampling to approximate richer adaptation structures with low memory overhead. To support knowledge transfer across tasks, modular PET strategies have also been developed. LoraHub [4] assembles reusable task-specific adapters that can be composed dynamically. MOLE treats multiple LoRA modules as experts and selects among them using a learned gating mechanism.
In contrast to prior work that focuses on isolated improvements in capacity or flexibility, we propose wavelet regularized continual tuning framework that jointly addresses knowledge retention and adaptive transfer. Our method sustains performance across evolving tasks without relying on data replay, while incurring minimal computational overhead.
Conclusion
We present DEAL, continual learning framework that integrates instruction-guided fine-tuning with lightweight adapter updates and structured regularization. Built on a LoRA-style architecture, DEAL enables scalable, interference-resistaal learning across sequential tasks while maintaining strong performance on diverse benchmarks. Extensive experiments highlight three core advantages:
Reduced Forgetting. DEAL employs regularized low-rank updates to preserve task-relevant subspaces, mitigating catastrophic forgetting without relying on explicit memory buffers.
Efficient Adaptation. Shared LoRA components serve as inductive priors, accelerating conver- gence and supporting efficient generalization in low-resource scenarios.
Scalability, By introducing only a small number of task-specific parameters, DEAL scales effec- tively to long task sequences and large LLM backbones such LLaMA-3.1-8B.
Limitations and Future Directions. While DEAL demonstrates strong performance in diverse continual learning scenarios, it currently assumes fixed task order and static model capacity. Promising directions include lightweight rehearsal mechanisms, dynamic capacity allocation, and enhancing forward transfer through meta-regularization. Addressing robustness under ambiguous task boundaries remains an open challenge.
Acknowledgements
This work was partially supported by the National Natural Science Foundation of China (No. 62502404), the Research Impact Fund of the Hong Kong Research Grants Council (No. R1015-23). the Collaborative Research Fund of the Research Grants Council (No. C1043-24GFF 244F-, the General Research Fund of the Research Grants Council (No. 1 11218325), and the Institute of Digital Medicine at City University of Hong Kong (No. 9229503). Additional support was provided by Huawei (Huawei Innovation Research Program), Tencent (CCF-Teenent Open Fund; Tencent Rhino-Bird Focused Research Program), Alibaba (CCF-Alimama Tech Kangaroo Fund No. 2024002), Didi (CCF-Didi Gaia Scholars Research Fund), Kuaishou, and ByteDance.
10

---

References
[1] Renqian Luo, Liai Sun, Yingee Xia, Tao Qin, Sheng Zhang. Hoifung Poon, and Tie- Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in bioinformatics, 23(6):bbac409, 2022.
[2] Hongyang Yang, Xiao- Yang Liu, and Christina Dan Wang. Fingpt: Open-source financial large language models. arXiv preprint urXiv:2306.06013 2023.
[3] Yuqian Fu, Runze Wang, Yanwei Fu, Danda Pani Paudel, Xuanjing Huang, and Luc Van Gool. Objectrelator: Enabling cross-view object relation understanding ego-centric and exo-centric videos. ICCV, 2025.
[4] Qidong Liu, Xian Wu, Yejing Wang, Zijian Zhang, Feng Tian, Yefeng Zheng, and Xiangyu Zhao. Lim-esr: Large language models enhancement for long-tailed sequential recommendation. Advances in Neural Information Processing Systems, 37:26701-26721 2024.
[5] Zichuan Fu, Xiangyang Li, Chuhan Wu, Yichao Wang. Kuicai Dong. Xiangyu Zhao, Mengchen Zhao, Huifeng Guo, and Ruiming Tang. unified framework for multi-domain ctr prediction via large language models. ACM Transactions on Information Systems, 43(5):1-33, 2025.
[6] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhur Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint mxXiv:2106.09653 2021.
[7] Maolin Wang, Jun Chu, Sicong Xie, Xiaoling Zang, Yao Zhao Wenliang Zhong, and Xiangyu Zhao. Put teacher in student's shoes: dross-disiillation for cltra-compact model compression framework. In Proceedings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. pages 4975 4985, 2025.
[8] James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization of rext-to-imape diffusion with c-lora. Xrxiv preprint arXhv:2304.0602: 2023.
[9] Chen. Reading wikipedia to answer open-domain questions. arXiv preprint urXiv:1704.0055) 2017.
[10] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint 1rXiv:2109.0777 07958, 2021.
[11] Xuanwen Ding, Jie Zhou, Liang Dou, Qin Chen, Yuanbin Wu, Chengcai Chen, and Liang He. Boosting large language models with continual learning for aspect-based sentiment analysis. arXiv preprini arXiv:2405.0544 2024.
[12] Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 37:9565-9588 2024.
[13] Kevin Meng. David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 35:17399-17372 2022.
[14] Kevin Meng, Amab Sen Sharma, Alex Andonian. Yonatan Belinkov, and David Bau. Mass- editing memory in a transformer. arXiv preprint arXiv:2210.07229, 2022.
[15] Xiaopeng Li. Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. Pmet: Precise model editing in a transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18564-18572, 2024.
[16] Shuaiyi Li, Yang Deng, Deng Cai, Hongyuan Lu, Liang Chen, and Wai Lam. Consecutive batch model editing with hook layers. arXiv preprint arXiv:2403.05330, 2024.
[17] Yujie Feng, Liming Zhan, Zexin Lu, Yongxin Xu, Xu Chu, Yasha Wang, Jiannong Cao, Philip S Yu, and Xiao-Ming Wu. Geoedit: Geometric knowledge editing for large language models. arXiv preprint 2502 2025.

---

[18] Zichuan Fu, Wentao Song, Yejing Wang, Xian Wu, Yefeng Zheng, Yingying Zhang, Derong Xu, Xuetao Wei, Tong Xu, and Xiangyu Zhao. Sliding window attention training for efficient large language models. arXiv preprint arXiv:2502. 18845, 2025.
[19] Minsoo Kim, Sihwa Lee, Wonyong Sung, and Jungwook Choi. RA-LoRA: Rank adaptive parameter-efficien fine-tuning for accurate 2-bit quantized large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 15773-15785 15786, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10.18653/v 1/2224 h200:/.inn..2....ss4oeea. acl.nns--acl... 133.. URL https://
acllattbology.org/2000 htt.a/ooogg..rr//2..........gng. windiinnnidnnnnneee acl. 9933/
[20] Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. Melora: mini-ensemble low-rank adapters for parameter- efficient fine-tuning, arXiv preprint arXiv:2402.1726, 2024.
[21] Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint arXiv:2409.05976 2024.
[22] Yimin Tian, Bolin Zhang. Zhiying Tu, and Dianhui Chu. Adapters selector: Cross-domains and multi-tasks LoRA modules integration usage method. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend AI-Khalifa, Barbara Di Eugenio, and Steven Schockaert, editors, Proceedings of the 31st International Conference on Computational Linguistics, pages 593-605, Abu Dhabi, UAE, January 2025, Association for Computational Linguistics. URL https
7 //aclanthology org. 2025. col.nn...onn.o..... 40/
[23] Zijian Zhang, Shuchang Liu, Jiaao Yu, Qingpeng Cai, Xiangyu Zhao, Chunxu Zhang, Ziru Liu, Qidong Liu, Hongwei Zhao, Lantao Hu, et al. Mßoe: dulti-domain multi-task mixture-oo of experts recommendation framework. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 893-902, 2024.
[24] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):211-118, 1936.
[25] Kai Liao and Yan Xu. A robust load frequency control scheme for power systems based on second-order sliding mode and extended disturbance observer. IEEE Transactions on industrial informatics, 14(7):307--386, 2017.
[26] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[27] Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. Buffet: Benchmarking large language models for few-shot cross-lingual transfer. arXiv preprint arXiv:2305. 14857, 2023.
[28] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.07461, 2018
[29] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
[30] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011 1. Association for Computational Linguistics. URL httpps:/acclantholooyy owg/oogg//// 1015/
[31] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310.1413 14152, 2023.
12

---

[32] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXivv2407.2178.. 2024.
[33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of muchine learning research, 21(140):1-6) 2020.
[34] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne. Expe- rience replay for continual learning. In Neural Information Processing Systems, 2018. URL org/ Serg/CorpusID:538800
[35] Xiao Wang, Yuansen Zhang, Tianze Chen, Songyang Gao, Senjie Jin, Xianjun Yang, Zhiheng Xi, Rui Zheng, Yicheng Zou, Tao Gui, et al. Trace: A comprehensive benchmark for continual learning in large language models. arXiv preprint arXiv:2310.06762, 2023.
[36] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang, Parameter-efffcient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 2024.
[37] Maolin Wang. Tianshuo Wei, Sheng Zhang, Ruocheng Guo, Wanyu Wang, Shanshan Ye, Lixin Zou, Xuetao Wei, and Xiangyu Zhao. Dance: Resource-efficient neural architecture search with data-aware and continuous adaptation. arXiv preprint arXiv:2.077 2.500.047777 2025.
[38] Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. When moe meets 1mss: Parameter efficient fine-tuning for multi-task medical applications. In Proceedings of the 47th International ACM SIGIR Conference on Reseurch and Development in Information Retrieval, pages 1 1104-1114, 2024.
[39] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, arXiv preprint (xXiv:2104.0869/ 2021.
[40] Wenxin Luo, Weirui Wang, Xiaopeng Li, Weibo Zhou, Pengyue Jia, and Xiangyu Zhao. Tapo: Task-referenced adaptation for prompt optimization. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2025.
[41] Maolin Wang, Xiangyu Zhao, Ruocheng Guo, and Junhui Wang. Metalora: Tensor-enhanced adaptive low-rank fine-tuning, In 2025 IEEE 41st International Conference on Data Engineering (ICDE), pages 4680-4684. IEEE, 2025.
[42] Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High- rank training through low-rank updates. arXiv preprint arXiv:2307.05669, 2023.
[43] Chi-Chih Chang, Yuan- Yao Sung, Shixing Yu, Ning-Chi Huang, Diana Marculescu, and Kai- Chiang Wu. Flora: Fine-grained low-rank architecture search for vision transformer. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 2492,2491, 2024.
[44] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307. 2023.
[45] Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. arXiv preprint arXiv:2404.3322 2024.
[46] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[47] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 114(13):35213335 2017.
[48] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2955-2947, 2017.
13

---

[49] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 139-149, 2022.
[50] Chengwei Qin and Shafiq Joty. Lfpt5: A unified framework for lifelong few-shot language learning based prompt tuning of t5. arXiv preprint arXtv:2110.07298 2021.
[51] Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, and Amjad Almahairi. Progressive prompts: Continual learning for language models. arXiv preprint 1rXiv:2301.23314 2023.
[52] Fuli Qiao and Mehrdad Mahdavi. Learn more, but bother less: parameter efficient continual learning. Advances in Neural Information Processing Systems, 57:97476-97499 2024.
14

---

Appendix
A Supplementary Algorithmic Analysis
A.1 Notation
Let X E 1 be matrix of rank Its singular value decomposition (SVD) is:
X = V
Ex1 0
0 0
V
(13)
m1 U n2
V
Here, U21 E RT and 21 grxr, correspond to the principal subspace, while U #2 and V x2
span the orthogonal complement.
A.2
Proof of Theorem 1
Consider perturbation E R and define:
x=X+D.
(14)
We decompose D via projection onto the column spaces of Val and V
Y= V Y == x
V (XV DV .1) V + (DV V
(15)
Denoting SVDs of each term:
XV 21 + DV #1 P
(16)
DV #2 P2S.Q2
(17)
with P.P.2 we can express Y
S
Q Q₂
Y Y= P1 Pa]
(18)
S2
V z2
This constitutes the SVD of Y, and the basis P1 generally differs from U 21 due to perturbation. This completes the proof.
B
Experimental Setup
B.1 Datasets
Table summarizes all datasets used across the continual learning benchmarks. These datasets were also employed in O-LoRA (3I), where each is framed classification task using unified instruction-based text-to-text format.
15

---

Table 5: Overview of datasets used in experiments.
Dataset
Source
Task Type
Evaluation Metric
AG News
CL Benchmark
Topic Classification
Accuracy
DBpedia
CL Benchmark
Entity Typing
Accuracy
Yahoo
CL Benchmark
Topic Classification
Accuracy
Amazon
CL Benchmark
Sentiment Analysis
Accuracy
MNLI
GLUE
Natural Language Inference
Accuracy
6
QQP
GLUE
Paraphrase Detection
Accuracy
RTE
GLUE
Natural Language Inference
Accuracy
SST-2
GLUE
Sentiment Analysis
Accuracy
WiC
SuperGLUE
Word Sense Disambiguation
Accuracy
10
CB
SuperGLUE
Natural Language Inference
Accuracy
COPA
SuperGLUE
Causal Reasoning
Accuracy
12
BoolQ
SuperGLUE
Boolean QA
Accuracy
13
MultiRC
SuperGLUE
Multi-hop QA
Accuracy
14
IMDB
External
Sentiment Analysis
Accuracy
B.2
Implementation Details
We implement DEAL using the Hugging Face Transformers library and perform training with FP16 mixed precision on single NVIDIA A A100 GPU. Unless otherwise stated, we adopt consistent experimental setup across both LLaMA-3.1 and TS-large backbones, tailored for low-resource, instruction-driven continual learning. All models are trained with adapter- based fine-tuning using LoRA, where we set the rank 32 for LLaMA and 16 for T5, selected based on backbone capacity. Optimization is performed using AdamW with a constant learning rate scheduler. Regu- larization is enforced via f--orrm constraints adapter weights and MLP modules, with details provided in Table
Table 6: Hyperparameter settings for DEAL on LLaMA-3.1 and T5-large.
Hyperparameter
LLaMA-3.1
T5-large
LoRA Rank r
32
16
Learning Rate
1e-5
le-5
Batch Size
8
4
Gradient Accum. Steps
2
Epochs
1
I
Max Source Length
512
512
Max Target Length
50
50
Generation Max Length
50
50
Warmup Steps
0
0
Dropout
Optimizer
AdamW constant 0.01 0.001
AdamW
Scheduler Type
constant
Regularization A1
0.01
Regularization Az
0.001
|0]|s Norm Reg.
|MLP||2 Norm Reg.
LoRA Modules
p_proj v_proj CausalLM
q V
Task Type
Seq2SeqLM
set 0.01 to constrain the fs-norm of adapter parameters 0, and A2 0.001 to regularize the f2-norm of task-specific MIP weights, when applicable. All experiments are conducted for one epoch over instruction-bant task mixtures, without early stopping or checkpointing unless explicitly mentioned.
16

---

C
Case Study Examples
Case DBpedia Classification Task
Prompt:
You are smart Al evaluator. Given an input paragraph and three model outputs (Base Model, Adapter after DBpedia, Adapter after
judge which output(s) correctly
classify the paragraph according to the true label.
Question:
What is the topic the following paragraph? Choose one from the options below. ["Company", "Educational Institution", "Artist", "Athlete", "Office Holder", "Mean of Transportation", "Building". "Natural Place". "Village", "Animal", "Plant", "Album", "Fil"", "Written Work"]
Input (DBpedia sample)
Label: Office Holder
Text: Raimundas Palaitis (born October 1957) is a Lithuanian politician. He was Minister of the Interior from 2008 to 2012.
Base Model: Building
Adapter after DBpedia: Office Holder
Adapter after DBpedia-+Amazon Amazon: Office Holder
Expected Answer:
The base model incorrectly predicted "Building," which unrelated to the paragraph. In contrast, both adapters correctly predicted "Office Holder," indicating that they effectively learned the DBpedia task and that the second adapter retained this knowledge even after subsequent training on the Amazon task
Case 2:
Amazon Sentiment Task
Prompt:
Given an input review and three model outputs, decide whether the sentiment is classified correctly and briefly explain why.
Question:
What is the sentiment the following review? Choose one from ("very negative", "negative", "neutral". "positive", "very positive"]
Input (Amazon sample)
Label: very negative
Text: I don't 't understand how they can advertise that this humidifier work up to 12 hours. runs out of water so fast. have to get up in the middle of the night and refill would say works well for about hours. So, while it inexpensive, you get what you pay for.
Base Model: neutral
Adapter after DBpedia: neutral
Adapter after DBpedia- Amazon: very negative
Expected Answer:
Both the base model and the first adapter misclassified the review as "neutral". Only the adapter further fine-tuned on Amazon data predicted the correct label, "very negative," " demonstrating its ability to acquire new task knowledge that the earlier models failed to capture.
Figure 3: Two case studies demonstrating continual learning and knowledge retention across classifi- cation tasks.
17

---

D Evaluation Metrics
We adopt the following standard metrics for continual learning evaluation:
Average Accuracy (AA): Measures the average test accuracy across all tasks after the final task is learned:
AA
Σ O.T
T
where a, is the test accuracy on task after training on task T.
ROUGE-1 (R-1): Used for generative label decoding, computed as the unigram Fj between model output and reference:
2. J.P.R.
lon'll
by 1y
R R-1 P+R
P=
R
M
18*1
E
Instruction Prompts
We adopt the task-specific instruction prompts introduced in O-LoRA 31 as summarized in Table 7
Table 7: Instruction prompts provided to the model for each task.
Task
Prompt
NLI
What is the logical relationship between "sentence I" and "sentence 2"? Choose one from the options.
QQP
Do "sentence 1" and "sentence 2" express the same meaning? Choose one from the options.
SC
What is the sentiment of the following passage? Choose one from the options. What the topic of the following passage? Choose one from the options. According to the passage. is the statement true or false? Choose one from the options.
TC
BooIQA
MultiRC
Based on the passage and question, is the candidate's 's answer correct? Choose one from the options.
WiC
Given word and two sentences, is the word used with the same sense in both? Choose one from the options.
F
Training and Inference Efficiency
Training Efficiency. We measure the training throughput and GPU memory usage on the DBpedia dataset with the T5-large backbone (Table 8
Table 8: Training efficiency on DBpedia with T5-large.
Method Training Throughput (samples/sec)
GPU Mem. Train (GB)
LoRA DEAL
31.62
20.41
17.88
22.93
Inference Efficiency.
We evaluate inference latency and GPU memory usage under the same setting
(Table 9)
Table Inference efficiency on DBpedia with Ta-ger
Method Inference Latency (ms/sample) GPU Mem. Infer (GB)
LoRA DEAL
71.89
3.15
73.32
3.16
DEAL incurs a 4x3% drop in training throughput and small increase in training GPU memory, while inference-time performance remains nearly identical to LoRA in both latency and memory usage. Since the wavelet module is only active during training and enables consistent performance gains (see Table this overhead represents a worthwhile trade-off for better generalization and robustness.
Discussion. As shown in Tables g and 9
18

---

G Comparison with Additional Continual Learning Methods
To provide comprehensive evaluation, expand our comparisons include a wide range of strong non-LoRA continual learning (CL) baselines. Table ro]rrts the average accuracy (AA) on standard CL benchmark using the Ta-le backbone.
Our method, DEAL, achieves the highest average accuracy while maintaining parameter efficiency, It outperforms both schearsal-basedd approaches (Replay) and regularization-base techniques (EWC, LwF). Additionally, DEAL surpasses prompt-based strategies (1.2,, ProgPrompt) and recent com- petitive baselines (LFPT5, LB-CL., highlighting its effectiveness continual learning.
general-purposs solution for
Table 10: Comparison with Additional Continual Learning Methods.
Method
Average Accuracy (AA)
SeqSVD
63.3
Replay
52.0
EWC
45.3
LwF
52.9
L2P
60.5
LFPT5
71.2
ProgPrompt
76.0
LB-CL
76.5
DEAL (Ours)
78.5
SeqSVD: Learns fixed-size SVD parameter space across sequential tasks without regular-
ization or replay.
Replay: Rehearses past samples to prevent forgetting by storing and replaying them during training [46].
EWC: Mitigates forgetting by penalizing changes to important parameters estimated via Fisher information [47].
LwF: Preserves knowledge of past tasks through knowledge distillation without storing old
data [48).
L2P: Introduces learnable prompts for continual learning to guide pre-trained models effectively [49].
LFPT5: Proposes unified framework for lifelong few-shot learning based on prompt tuning T5 [50].
ProgPrompt: Designs progressive prompts adapt large language models for continual learning [51].
LB-CL: Achieves parameter-efficient continual learning by learning more but disturbing
less 52
19

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and the "Main Results" paragraph in Section ] Jccurately reflect the key contributions and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

Justification: A summary of the key limitations of this work is provided in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Appendix A lists all the assumptions used and provides the complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: A comprehensive description of the experimental setup and hyperparameters is included in Appendix B
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include detailed instructions for running the experiments, and the code is publicly released as open source.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nisscc/public//uust http://iccggiiee/eeeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
22

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The dataset splits, hyperparameter configurations, and detailed experimental setup are thoroughly documented in Appendix B
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We evaluate performance in Section 4.1 Jusing three continual learning bench- marks across different models.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section provides clear report of the computational resources used in our experiments.
23

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics bttb httpppblii/tiiicciieeiiieene.e
Answer: [Yes]
Justification:
The paper follows the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts of our approach are discussed in detail in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
24

---

Answer:
[NA]
Justification:
No datasets or models are released in this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
is completely open source. In addition, the method mentioned in the paper brand new and is also the innovation this article.
Justification: The dataset
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pwpprswithhaeecoc thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pan se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release our code as open source and include detailed instructions to facilitate use.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
25

---

14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The usage of LLMs illustrated in detail in the Method and Experiments section.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy auteee///baerrrpeeccc////ffeeerp for what should or should not be described.
26


Reviews:
[{"review_id": "rMVvOF2eim", "reviewer": "Reviewer_pDXr", "summary": "This paper proposes DEAL, a framework designed to address the challenges of data efficiency and continual learning in large language models. By integrating LoRA with a continuous fine-tuning strategy and introducing modules for knowledge retention and adaptive parameter updates, DEAL enables effective adaptation while mitigating catastrophic forgetting. Experiments on 15 diverse datasets demonstrate its superior accuracy and resource efficiency compared to existing approaches.", "questions": "1. Could the authors provide a detailed analysis on how DEAL would perform on larger models?\n2. Could the authors include comparisons with other strong baseline methods to better demonstrate the advantages of DEAL?\n3. Could the authors provide more quantitative data on the computational and memory efficiency of DEAL?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes a practical and well-motivated framework for data-efficient continual learning in LLMs.\n2. The method delivers solid empirical results across 15 diverse benchmarks.\n3. The authors provide open-source code for reproducibility.\n\nWeaknesse:\n\n1. The evaluation is conducted on 8B and T5-Large backbones and it remains unclear how DEAL would perform on larger-scale or commercially deployed models (e.g., 70B+ parameters).\n2. The evaluation is limited to LoRA-based methods, without including a wider range of strong baselines.", "was_revised": true, "final_justification": "Thank you for your replies. The rebuttal has addressed my concerns, so I will keep my positive score.", "rebuttal": "We thank the reviewer for your appreciation of the valuable contribution of the model and the quality of the paper.\n\n---\n\n**W1&Q1: Could the authors provide a detailed analysis on how DEAL would perform on larger models?**\n\nA1: Thank you for your question. We will include additional experimental results on larger-scale models in the final version of the paper. Due to the limited rebuttal period, these results are not yet available, but we will release them in the next few days as soon as they are ready.\n\n---\n\n**W2&Q2: Could the authors include comparisons with other strong baseline methods to better demonstrate the advantages of DEAL?**\n\nA2: Thanks for your suggestion. We have expanded our experimental comparisons to include a broader range of strong baselines. The following table reports the average accuracy (AA) on the standard Continual Learning (CL) benchmark using the T5-large backbone:\n\n| Method      | Average Accuracy (AA) |\n|-------------|------------------------|\n| IncLoRA     | 62.2                   |\n| SeqSVD      | 63.3                   |\n| Replay      | 52.0                   |\n| EWC         | 45.3                   |\n| LwF         | 52.9                   |\n| L2P         | 60.5                   |\n| LFPT5       | 71.2                   |\n| ProgPrompt  | 76.0                   |\n| LB-CL       | 76.5                   |\n| **DEAL**    | **78.5**               |\n\n**Method Descriptions:**\n\n- **IncLoRA**: Incremental learning of new LoRA parameters over a sequence of tasks, without any regularization or replay of prior data.\n- **SeqSVD**: Learning a fixed-size SVD parameter space across sequential tasks, without regularization or replay.\n- **Replay**: _D. Lopez-Paz and M. Ranzato, \"Gradient episodic memory for continual learning,\" NeurIPS 2017._\n- **EWC**: _J. Kirkpatrick et al., \"Overcoming catastrophic forgetting in neural networks,\" PNAS 2017._\n- **LwF**:  _Z. Li and D. Hoiem, \"Learning without forgetting,\" ECCV 2016._\n- **L2P**: _X. Wang et al., \"Learning to prompt for continual learning,\" CVPR 2022._\n- **LFPT5**: _H. Qin et al., \"LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T5,\" NeurIPS 2022._\n- **ProgPrompt**:  _L. Wang et al., \"ProgPrompt: Continual learning for language models,\" ICLR 2023._\n- **LB-CL**: _F. Qiao and M. Mahdavi, \"Learn more, but bother less: Parameter-efficient continual learning,\" *NeurIPS*, 2024._\n\n---\n\n**Q3: Could the authors provide more quantitative data on the computational and memory efficiency of DEAL?**\n\nA3: Thank you for the question. We will provide more efficiency study in the revised paper. We summarize the timing efficiency and overhead of our method compared to LoRA as follows:\n\n- **Training Efficiency**: Our method introduces a modest increase in training time due to the wavelet module, which processes input features in the frequency domain. However, this additional overhead remains limited and does not hinder scalability.\n- **Inference Efficiency**: The wavelet module is **entirely discarded at inference time**, resulting in **no additional latency or memory usage** compared to the base model or LoRA.\n\nTo concretely quantify this, we evaluated our method on the DBpedia dataset using the T5-large backbone. The results are as follows:\n\n| Method | Training Throughput (samples/sec) | GPU Memory during Training (GB) | Inference Latency (ms/sample) | GPU Memory during Inference (GB) |\n|--------|-----------------------------------|----------------------------------|-------------------------------|----------------------------------|\n| LoRA   | 31.62                           | 20.41                             | 71.89                         | 3.15                             |\n| DEAL   | 17.88                           | 22.93                             | 73.32                         | 3.16                             |\n\nAs shown, **DEAL introduces a ~43% drop in training throughput** and a *slight increase in GPU memory during training*, but the **inference-time performance remains nearly identical to that of LoRA** in both latency and memory consumption.\n\nGiven that the wavelet module is only active during training and enables consistent performance improvements(as shown in Table1), we believe this overhead is a worthwhile trade-off for enhanced model generalization and robustness."}, {"review_id": "Km8DuGP170", "reviewer": "Reviewer_8PR6", "summary": "This paper introduces DEAL, a novel framework for continual learning in Large Language Models (LLMs) that enhances Low-Rank Adaptation (LoRA) by integrating a wavelet kernel-based knowledge retention module and a controlled knowledge updating module. The method aims to address catastrophic forgetting and data inefficiency in parameter-efficient fine-tuning scenarios, especially when dealing with privacy-sensitive, small-scale datasets. DEAL adaptively preserves the core features of historical knowledge while incorporating new information, maintaining performance across evolving tasks without relying on data replay. The proposed method is evaluated on 15 diverse tasks spanning standard text classification, domain-shift, and multi-task settings. Experimental results demonstrate consistent performance improvements over strong baselines (e.g., SeqLoRA, O-LoRA), approaching the oracle upper bound (PerTaskFT) while remaining highly efficient and scalable. The framework is rigorously analyzed through ablation studies and supported by theoretical foundations for feature extraction in low-rank matrices. Overall, this is a strong and timely contribution to the community, addressing critical challenges in continual learning for LLMs with a novel and effective approach.", "questions": "1) Could the authors clarify why the white noise assumption for matrix noise is valid in the context of LoRA fine-tuning and whether this impacts generality?\n\n2) Why were T5 and LLaMA-3.1 selected as the representative backbones\n\n3) Could the authors briefly summarize the three task order permutations used in the main paper to support the task-order robustness claim?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1) proposes a unique integration of wavelet-based knowledge retention with a controlled updating mechanism, targeting both catastrophic forgetting and interpretability in LoRA-tuned LLMs. \n\n2) The paper provides solid theoretical foundations for the core feature extraction process, including derivations based on singular value decomposition (SVD) and minimum variance estimation. The proof of Theorem 1 in the appendix and the structured use of orthogonal projections strengthen the paper's rigor. \n\n3) The authors conduct extensive experiments on 15 tasks with different characteristics (same-domain, domain-shift, and heterogeneous multi-task). \n\n4) The framework scales well with large backbones (e.g., LLaMA-3.1-8B) and long task sequences, without increasing inference time. The low-rank factor analysis and regularization ablations demonstrate DEAL’s ability to maintain performance with minimal computational overhead.\n\nWeaknesses\n\n1) The assumption that noise in the low-rank matrices is white noise is reasonable but not clearly justified in the main text. Clarifying why this assumption is acceptable in practice would help support the theoretical derivation.\n\n2) While DEAL achieves strong empirical results, the paper briefly mentions only two model architectures (T5 and LLaMA-3.1). A short explanation of why these are representative would strengthen the broader applicability claims.\n\n3) The ablation study confirms DEAL's low sensitivity to task order, but the specific task permutations used are relegated to the appendix. Including a brief summary in the main text would enhance transparency and reader confidence in the robustness claim.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, so I keep my positive scores.", "rebuttal": "We appreciate the valuable suggestions and your recognition of the solid techniques and the novelty of the paper, which is encouraging.\n\n---\n\n**W1&Q1: Could the authors clarify why the white noise assumption for matrix noise is valid in the context of LoRA fine-tuning and whether this impacts generality?**\n\n**A1:** Thanks for your comments. The white noise assumption for matrix noise in our formulation is motivated by both empirical evidence and theoretical considerations.  \nFirst, white noise is a widely acknowledged approximation for residual noise in real-world stochastic systems, including deep learning training processes. In the context of LoRA fine-tuning, the residual difference between the full-model gradient space and the low-rank update directions can be treated as a high-frequency, zero-mean perturbation. This aligns with how many prior works in optimization and signal processing treat unmodeled components—as approximately white noise (e.g., i.i.d. Gaussian or uniform).  \nSecond, our empirical studies (see Section 4.3 and Appendix C.1) demonstrate that DEAL remains robust under this assumption across a range of LoRA configurations and task permutations. Importantly, our model does not assume exact white noise structure in implementation, but rather uses it as a theoretical proxy to motivate orthogonal decomposition of adaptation subspaces, which improves continual learning stability.  \nFinally, this abstraction does not harm generality. As also supported by prior continual learning work such as O-LoRA (Wang et al., 2023a), orthogonal subspace methods remain effective even when the residual is not strictly white, provided it is not heavily task-aligned.  \nWe appreciate the opportunity to clarify this point and will consider elaborating it in the main text for completeness.\n\n- **O-LoRA**: C. Wang et al., \"Orthogonal Subspace Learning for Language Model Continual Learning,\" *Findings of EMNLP*, 2023.\n\n**W2&Q2: Why were T5 and LLaMA-3.1 selected as the representative backbones**\n\n---\n\n**A2:** Thank you for the question regarding our choice of backbone models. We selected T5-Large and LLaMA-3.1-8B as representative backbones based on several criteria aligned with the core goals of our study:\n\n1. **Architectural diversity:** T5 is an encoder-decoder model, whereas LLaMA is a decoder-only model. This dual choice enables us to evaluate the generality and adaptability of DEAL across different model architectures, as discussed in Section 4 and Appendix B.\n2. **Instruction-following capabilities:** Both models have instruction-tuned variants (T5-Large-Instruct and LLaMA-3.1-Instruct), making them well-suited for our instruction-based continual learning setting. This ensures consistent and fair prompt-based evaluation across tasks (refer to Appendix D).\n3. **Popularity and Community Usage:**  The T5 and LLaMA model families are among the most widely adopted open-source language models for fine-tuning and evaluation. Specifically, T5-large has received 6.4k GitHub stars and was downloaded over 540,000 times from Hugging Face in the past month. Similarly, LLaMA 3.1 has accumulated 28.9k GitHub stars and over 850,000 downloads in the same period. These metrics highlight their widespread adoption and establish them as leading open-source foundation models in current NLP research.\n4. **Scalability and efficiency trade-off:** LLaMA-3.1-8B serves as a modern large-scale model to test scalability, while T5-Large offers a more lightweight alternative that is computationally efficient. Together, these models span a broad range of use cases in continual LoRA fine-tuning.\n\nWe clarify these points in Appendix B.2 to support our experimental design rationale.\n\n---\n\n**W3&Q3: Could the authors briefly summarize the three task order permutations used in the main paper to support the task-order robustness claim?**\n\n**A3:** Thank you for the question. To assess task-order robustness, we evaluated our method under three task permutations from the standard CL benchmark (Zhang et al., 2015), as detailed in Appendix C (Table 5):\n\n- **Order 1:** DBpedia → Amazon → Yahoo → AG News  \n- **Order 2:** DBpedia → Amazon → AG News → Yahoo  \n- **Order 3:** Yahoo → Amazon → AG News → DBpedia  \n\nAs shown in Figure 2(b), the fluctuation in Average Accuracy (AA) across these permutations is less than 3 percentage points, demonstrating that DEAL remains stable across varied task sequences. This supports our claim of robustness to task ordering.\n\nWe also have made sure that the task order settings align with those used in prior work (e.g. O-LoRA [Wang et al., 2023a]) to ensure fair comparison.\n\n- X. Zhang, J. Zhao, and Y. LeCun, \"Character-level convolutional networks for text classification,\" *NeurIPS*, 2015."}, {"review_id": "fu9y1peGqR", "reviewer": "Reviewer_A3Fs", "summary": "The work introduces a continual learning framework that uses small amounts of new data for continuous learning, thereby avoiding the need for relearning. The method uses a wavelet kernel to preserve historical knowledge and deploy differentiated regularization terms to control the knowledge updating process. Experimental results shows the effectiveness of the proposed framework to maintain high performance across different tasks.", "questions": "I have some questions about different parts of the paper, which are listed in the following:\n\n1. In line 3 and 14, what is meant by \"privacy-sensitive scenarios\" and \"data privacy\"? In the main body of the paper, the authors have not specifically discussed data privacy. Does it mean the data privacy of the fine-tuning data? If so, why is it important in the context of continual learning?\n\n2. Line 43: there is a typo: One -> On\n\n3. Line 123: What is meant by \"core feature\"? and why can it be found by SVD? This part and the few lines after that really need to be clarified.\n\n4. Line 125: The dimensions of Y and X should be the same, right?\n\n5. In Theorem 1: Why should $P_1$ be equal to $U_{x_1}$? As mentioned earlier, this whole section needs to be re-written and clarified.\n\n6. Line 133: The optimization problem (4) assumes $X$ is given and the minimization is over $Z$, right? What is $f$?\n\n7. Line 134: Is it a typo? $H$ should change to $Z$?\n\n8. Line 135: $\\hat{X}$ is the estimate of $X$, but $X$ appears in equation (6)! How is it possible?\n\n9. Line 136: The expression for $YZ$ is different from that in equation (6). \n\n10. The notations in eq. 8 and 9 for the function $\\phi$ are different. \n\n11. Line 16 of algorithm 1: What is the \"$=0$\" at the end?\n\n12. In Table 1, and \"4-Task (Standard)\" column, why is DEAL better than the oracle PerTaskFT? The oracle one should perform better right? because it uses a separate LoRA adapter for each task and there is zero sharing.\n\n13. Line 251: This line refers to Fig. 2 a. Why does updating only the \"task-specific adapter B\" results in a better average accuracy (AA) than when updating only \"the shared adapter A\"? Shouldn't the task-speciifc one be better? Updating B adapts the model to the new upcoming tasks, right?", "limitations": "The writing of the paper really needs to be improved to make it more understandable and easier to follow. The related works section can be pushed to the appendix and can be replaced with some background about wavelet filtering and other details related to the proposed idea. Also, the baselines mentioned above need to be discussed and compared with if they are related. It seems that they are all closely related. Overall, the paper and the results need to be improved.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The considered topic of continual learning of LLMs is important and interesting. \n2. Experimental results have been reported for three different benchmarks.\n\nWeaknesses:\n1. The paper is not well-written and is hard to follow, especially the whole section 3. It seems that there are multiple typos in there as well. See below. The writing of the paper needs to be improved to clarify further about the proposed idea and make it understandable. \n\n2. Some directly related baselines are not studied. See the following works, which are very closely related. Comparing with them or at least discussing them and their potential shortcomings is essential. \n\n[1] M. Wistuba, et. al., \"Continual Learning with Low Rank Adaptation\", Neurips 2023.\n\n[2] Y. Liang, et. al., \"InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning\", CVPR 2024.\n\n[3] F. Qiao, et. al., \"Learn more, but bother less: parameter efficient continual learning\", Neurips 2024.\n\n[4] X. Wei, et. al., \"Online-LoRA: Task-free Online Continual Learning via Low Rank Adaptation\", WACV 2024.\n\n3. The section 6 (related work) can be put at the beginning of the paper (e.g. after section 1) to provide a better view of the content to the reader. \n\n4. The experimental results are limited due to not comparing to multiple baselines. Currently, from the three considered baselines, one is a naive one and one is an oracle method.", "was_revised": true, "final_justification": "The authors have acknowledged that **\"the work (F. Qiao et al., Learn more, but bother less: Parameter Efficient Continual Learning, NeurIPS 2024) serves as a strong and directly relevant baseline\"**, but have not used it as a baseline. This work is a strong baseline from Neurips 2024 that should have been used as a baseline. Without complete comparison with this strong and recent work, I strongly doubt about the contribution of this work in terms of improving the SoTA. I strongly believe that, despite being interesting, this work should be improved (in its writing and experiments) and include the above baseline in its experiments to prove its contribution. Only then, it can be resubmitted to the next venues. \n\nTherefore, I keep my score and also encourage the authors to improve their draft as explained.", "rebuttal": "Thanks for your comments, and we would like to clarify your major concerns:\n\n---\n\n**Q1: In line 3 and 14, what is meant by \"privacy-sensitive scenarios\" and \"data privacy\"? In the main body of the paper, the authors have not specifically discussed data privacy. Does it mean the data privacy of the fine-tuning data? If so, why is it important in the context of continual learning?**\n\n**A1:** Thank you for the question. We will clarify it in the revised manuscript to avoid misunderstandings. The private information in the data samples is not helpful for the target learning task in continuous learning and is considered as noise data. We use this example to highlight that continual learning methods are often prone to catastrophic forgetting and poor model performance in such settings, as illustrated in lines 5-6. We apologize for the misunderstanding and will delete the potentially misleading description in the revised manuscript.\n\n---\n\n**Q2-Q9: Paper clarity issues in lines 121-149.**\n\nThanks for your questions. In the revised version, we will improve the clarity of this part and correct any typos. There do have some misunderstandings: For example, the Question 5 is \"Why should $P_1$ be equal to $U_{x_1}$?\", while in lines 129-131, Theorem 1, we said \"... there ***does not*** exist a pair of matrices $P_1$ and $U_{x_1}$ such that $P_1=U_{x_1}$\". In addition, as for Question 6, $X$ is the variable that we need to estimate rather than it is given, as we illustrated in line 132: \"... to find the optimal estimate of $X$ ...\".\n\nWe apologize once again for the misunderstanding. We have reorganized lines 121-149 and corrected the typos you pointed out to address your concerns, as detailed below:\n\nIn the LoRA module, the matrices $A$ and $B$ are singular, meaning they are not full-rank. This presents a challenge for effective feature extraction. We assume that the singular matrix $Y := A$ or $B$ can be decomposed into a task-relevant component $X$ and a redundant or noisy component $D$, i.e., $Y = X + D$.\n\nHere, $X$ denotes the core feature matrix, capturing the intrinsic low-rank structure that encodes task-relevant semantics. According to the Eckart–Young–Mirsky theorem (Eckart and Young, 1936), the best low-rank approximation of a matrix in the Frobenius norm sense is achieved via truncation of its singular value decomposition (SVD). This motivates our use of truncated SVD to estimate $X$ from the observed matrix $Y$, with the goal of recovering task-relevant features from its singular representation.\n\nWe further assume that both $Y$ and $X$ lie in $\\mathbb{R}^{n \\times r}$, sharing the same dimensions, since $Y$ is formed by the additive composition of $X$ and the residual term $D$. Their corresponding singular value decompositions (SVD) are given by:\n\n$$\nY = \n\\begin{pmatrix}\nP_1 & P_2\n\\end{pmatrix}\n\\begin{pmatrix}\nS_1 & 0 \\\\\\\\\n0 & S_2\n\\end{pmatrix}\n\\begin{pmatrix}\nQ_1^\\top V_{x1}^H \\\\\\\\\nQ_2^H V_{x2}^\\top\n\\end{pmatrix},\n\\tag{2}\n$$\n\n$$\n\\begin{align}\nX &= U_x \\Sigma_x V_x \\\\\\\\\n&= \n\\begin{pmatrix}\nU_{x1} & U_{x2}\n\\end{pmatrix}\n\\begin{pmatrix}\n\\Sigma_{x1} & 0 \\\\\\\\\n0 & 0\n\\end{pmatrix}\n\\begin{pmatrix}\nV_{x1} \\\\\\\\\nV_{x2}\n\\end{pmatrix},\n\\tag{3}\n\\end{align}\n$$\n\nwhere $U _{x1} \\in \\mathbb{R}^{n _x \\times r _x}$, $U _{x1} \\in \\mathbb{R}^{n _x \\times (n _x-r _x)}$, $V _{x 1} \\in \\mathbb{R}^{r _x \\times r}$, $V _{x 2} \\in \\mathbb{R}^{(n _x-r _x) \\times r}$.\n\nThe following theorem states that we cannot directly compute the core features of $X$ from $Y$:\n\n**Theorem 1.**\nLet $Y$ be the observed data matrix and $X$ the underlying core feature matrix. Then, without additional constraints, there does not exist a pair of matrices $P_1$ and $U_{x1}$ such that $P_1 = U_{x1}$. (See Appendix A.2)\n\nTherefore, we aim to recover the core feature matrix $X$ by representing it as a linear combination of the columns of the observed matrix $Y$. To this end, we introduce a coefficient matrix $H$ and formulate the following least-squares objective:\n\n$$\n\\min_{H} \\| Y H - X \\|_F^2\n\\tag{4}\n$$\n\nwhere $|| \\cdot ||_F$ denotes the Frobenius norm. In this formulation, $X$ is treated as the target (e.g., the ideal low-rank component), and $H$ is the optimization variable that linearly combines the basis vectors in $Y$ to approximate $X$. The optimal $H$ can be derived as:\n\n$$\nH = \\left( Y^\\top Y \\right)^{-1} Y^\\top X.\n\\tag{5}\n$$\n\nThen, $\\hat{X}$, the minimum variance estimate of $X$, can be presented as:\n\n$$\n\\hat{X} = Y H = Y \\left( Y^\\top Y \\right)^{-1} Y^\\top,\n\\tag{6}\n$$\n\nwhere $Y H = Y \\left( Y^\\top Y \\right)^{-1} Y^\\top$ is an orthogonal projection operator. Assume that the redundant features in the $A$ and $B$ matrix are white noise, that is, $D^\\top D = \\sigma^2_{D} I$, $X^\\top D = 0$, where $\\sigma^2_{D}$ is the variance of the noise. Then we can simplify $\\hat{X}$:\n\n$$\n\\hat{X} = \\sum_{k=1}^{r_x} \\frac{\\sigma_k^2 - \\sigma^2_{D}}{\\sigma_k} u_k v_k^\\top,\n\\tag{7}\n$$\n\nwhere $u_k$ and $v_k$ is the left and right singular vectors of $Y$, $\\sigma_k$ is the $k$-th largest singular value of $Y$ ($\\sigma_1 > \\sigma_2 > \\cdots > \\sigma_r$). In traditional signal analysis algorithms, large singular values represent low-frequency data distribution and macro trends, and small singular values represent high-frequency disturbances (Liao and Xu, 2017). However, since $\\sigma^2$ in the above formula is unknown, we will define a series of wavelet functions at different scales for feature filtering. Here, we use the heat kernel as a low-pass filter:\n\n$$\n\\phi_{\\sigma_j^2, c_j}(X) = \\exp\\left( - \\frac{1}{2 \\sigma_j^2} \\\\| X - c_j \\\\|^2 \\right),\n\\tag{8}\n$$\n\nwhere $c_j$ is the center of the $j$-th kernel, $\\sigma_j^2$ represents the width of the kernel. By setting a series of different heat kernel widths $\\sigma^2 = [\\sigma_1^2, \\sigma_2^2, \\cdots]^\\top$, defining a series of learnable diagonal matrices $g =[g_1, g_2, \\cdots]^\\top$ and learnable centers $C = [c_1, c_2, \\cdots]^\\top$, we can define a wavelet neural network to extract the features of $\\hat{X}$ from $Y$:\n\n$$\nH _{:,i}^{k+1} = \\delta\\left( \\sum _j \\phi _{\\sigma _j^2, c _j} \\ g _j \\ \\phi _{\\sigma _j^2, c _j}^{-1} H _{:,j}^k \\right),\n\\tag{9}\n$$\n\nwhere $b$ is also a trainable scalar, and $\\delta(\\cdot)$ is the activation function, $H^0 := Y$, and $\\hat{X} = H^K$. Here $K$ is the total number of layers and $k \\in \\{0,1,\\cdots,K-1\\}$. As $\\phi$ is the heat kernel, we could simply Eq.(9) to be:\n\n$$\nH _{:,i}^{k+1} = \\delta\\left( \\sum _j \\phi _{\\sigma _j^2, c _j} \\ g _j \\ \\phi _{-\\sigma _j^2, c _j} H _{:,j}^k \\right).\n\\tag{10}\n$$\n\n- C. Eckart and G. Young, \"The approximation of one matrix by another of lower rank,\" *Psychometrika*, vol. 1, no. 3, pp. 211–218, 1936.\n- K. Liao and Y. Xu, \"A robust load frequency control scheme for power systems based on second-order sliding mode and extended disturbance observer,\" *IEEE Trans. Ind. Informat.*, vol. 14, no. 7, pp. 3076–3086, 2017.\n\n---\n\n**Q11: 1. Line 16 of algorithm 1: What is the \"=0\" at the end?**\n\nSorry for the typo we caused. In the revised version, we will delete \"=0\" at the end of Line 16 in Algorithm 1. At the same time, we re-checked the full paper and carefully revised the minor errors in the paper to ensure that similar problems would not appear in the revised document. Thank you again for pointing out the problems.\n\n---\n\n**Q12: 1. In Table 1, and \"4-Task (Standard)\" column, why is DEAL better than the oracle PerTaskFT? The oracle one should perform better right? because it uses a separate LoRA adapter for each task and there is zero sharing.**\n\nThank you for your insightful questions. DEAL achieves superior performance over PerTaskFT in the 4-Task (Standard) setting by exploiting shared representations that enable beneficial cross-task transfer—a capability unattainable by the fully isolated PerTaskFT. While PerTaskFT trains each task independently and thus precludes any knowledge sharing, DEAL’s shared parameter space helps mitigate catastrophic forgetting while promoting transfer across related tasks. As stated in the paper, \"we demonstrate that our method performs comparably to multi-task learning and even outperforms the oracle PerTaskFT in some settings,\" underscoring DEAL's ability to exploit inter-task synergies when tasks are related.\n\nWe also clarify that for same-domain or large-scale task setups, PerTaskFT generally outperforms continual learners—consistent with our observation that continual learning with many heterogeneous tasks remains a challenging open problem.\n\n---\n\n**Q13: Line 251: This line refers to Fig. 2 a. Why does updating only the \"task-specific adapter B\" results in a better average accuracy (AA) than when updating only \"the shared adapter A\"? Shouldn't the task-speciifc one be better? Updating B adapts the model to the new upcoming tasks, right?**\n\nThank you for your question and close reading of our results. We would like to clarify a possible misunderstanding regarding Fig. 2(a). As shown in Fig. 2(a), the setting where only the shared adapter A being updated achieves higher average accuracy (AA) than updating only the task-specific adapter B, and updating both adapter A \\& B achieves the best performance. We hope this clears up any misunderstandings you may have."}, {"review_id": "o3coomdIM7", "reviewer": "Reviewer_naB7", "summary": "This paper presents DEAL, a continual learning framework for LoRA-based large language models that addresses catastrophic forgetting through two key components: a wavelet kernel-based module that preserves core historical knowledge features, and a controlled updating module that integrates new information via asymmetric regularization. Experiments show DEAL consistently outperforms existing baselines while maintaining computational efficiency and achieving near-oracle performance without additional inference overhead.", "questions": "1. Why did you choose heat kernels specifically? Have you experimented with other types of wavelets\n2. Can you provide detailed timing comparisons during training? The wavelet neural network introduces computation overhead - at what point does this overhead outweigh the benefits?", "limitations": "1. Limited baseline scope: The authors only compare against LoRA-compatible methods but fail to acknowledge or discuss the absence of comparisons with established continual learning approaches such as experience replay, elastic weight consolidation. \n2. The paper does not address how the approach handles very long task sequences or situations where new knowledge fundamentally conflicts with historical knowledge. This is particularly concerning for real-world deployments where models may encounter hundreds of tasks over time or face contradictory information.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Provides mathematical proof showing the infeasibility of direct feature extraction from singular LoRA matrices, which motivates the wavelet kernel approach.\n2. Comprehensive evaluation: Extensive experiments spanning multiple domains (text classification, GLUE, SuperGLUE) demonstrate consistent improvements.\n3. No additional inference overhead since updated LoRA matrices directly replace original ones, making it deployment-friendly.\n\nWeaknesses\n\n1. Only compares against LoRA-compatible methods, missing comparisons with other continual learning approaches.\n2. Unclear how the approach scales to very long task sequences or when dealing with conflicting knowledge updates over time.", "was_revised": true, "final_justification": "The responses from the authors of this paper addressed most of my concerns. Considering the contribution and application potential of this paper, I decided to keep my positive score.", "rebuttal": "We appreciate the valuable suggestions and your recognition of the solid techniques and the novelty of the paper, which is encouraging.\n\n---\n\n**Q1: Why did you choose heat kernels specifically? Have you experimented with other types of wavelets?**\n\nA1: Thank you for the question. The purpose of selecting heat kernels is to reduce the computational complexity of the proposed model. The concern arises primarily due to the inverse function $\\phi^{-1}$ appearing in Equation (10) of our method (line 151-152, page 5). In general, computing the inverse of a nonlinear function can be computationally expensive. However, for the specific case where we use the heat kernel $h(x) = e^{-x}$, its inverse satisfies $h^{-1}(x) = h(-x)$, which is a simple negation. This property eliminates the need for explicit inverse computation, significantly improving computational efficiency in practice.\n\nFor reference, our update step is as follows:\n\n$$\n\\boldsymbol{H} _{:,i}^{k+1} = \\delta\\left( \\sum _j \\phi _{\\sigma _j^2, c _j} \\ \\boldsymbol{g} _j \\ \\phi _{-\\sigma _j^2, c _j} \\boldsymbol{H} _{:,j}^k \\right).\n$$\n\nThis **inverse-free formulation** is the main reason we adopt the exponential (heat kernel) form for \\(\\phi\\), as it enables both conceptual simplicity and computational efficiency. We appreciate the reviewer’s insightful comment.\n\nIn addition, we have also compared two different types of functions as kernels: $f(x) = x e^{-x}$ and quadratic splines. The experimental results are shown in the table below. As observed, when using the heat kernel, the model incurs less computational overhead while achieving similar accuracy.\n\n|Method|Average Accuracy|Traning Time (ms/per sample)|\n|:--|:--|:--|\n|$f(x) = xe^{-x}$|78.4|1,759|\n|Quadratic Splines|78.3|1,291|\n|**Ours**|**78.5**|**56**|\n\n---\n\n**Q2: Can you provide detailed timing comparisons during training? The wavelet neural network introduces computation overhead – at what point does this overhead outweigh the benefits?**\n\nA2: Thank you for the question. We summarize the timing efficiency and overhead of our method compared to LoRA as follows:\n\n- **Training Efficiency**: Our method introduces a modest increase in training time due to the wavelet module, which processes input features in the frequency domain. However, this additional overhead remains limited and does not hinder scalability.\n- **Inference Efficiency**: The wavelet module is **completely discarded during inference**. Instead, the trained output of the module (LoRA A or B) is directly substituted with the parameters in the meta-model. As a result, there is **no additional** latency or memory usage during inference compared to the base model or LoRA.\n\nTo concretely quantify this, we evaluated our method on the DBpedia dataset using the T5-large backbone. The results are as follows:\n\n| Method | Training Throughput (samples/sec) | GPU Memory during Training (GB) | Inference Latency (ms/sample) | GPU Memory during Inference (GB) |\n|--------|-----------------------------------|----------------------------------|-------------------------------|----------------------------------|\n| LoRA   | 31.62                           | 20.41                             | 71.89                         | 3.15                             |\n| DEAL   | 17.88                           | 22.93                             | 73.32                         | 3.16                             |\n\nAs shown, **DEAL introduces a ~43% drop in training throughput** and a *slight increase in GPU memory during training*, but the **inference-time performance remains nearly identical to that of LoRA** in both latency and memory consumption.\n\nGiven that the wavelet module is only active during training and enables consistent performance improvements(as shown in Table1), we believe this overhead is a worthwhile trade-off for enhanced model generalization and robustness.\n\n---\n\n**W1: Only compares against LoRA-compatible methods, missing comparisons with other continual learning approaches.**\n\nA3: Thanks for your suggestion. We have expanded our experimental comparisons to include a broader range of strong baselines. The following table reports the average accuracy (AA) on the standard Continual Learning (CL) benchmark using the T5-large backbone:\n\n| Method      | Average Accuracy (AA) |\n|-------------|------------------------|\n| IncLoRA     | 62.2                   |\n| SeqSVD      | 63.3                   |\n| Replay      | 52.0                   |\n| EWC         | 45.3                   |\n| LwF         | 52.9                   |\n| L2P         | 60.5                   |\n| LFPT5       | 71.2                   |\n| ProgPrompt  | 76.0                   |\n| LB-CL       | 76.5                   |\n| **DEAL**    | **78.5**               |\n\n**Method Descriptions:**\n\n- **IncLoRA**: Incremental learning of new LoRA parameters over a sequence of tasks, without any regularization or replay of prior data.\n- **SeqSVD**: Learning a fixed-size SVD parameter space across sequential tasks, without regularization or replay.\n- **Replay**: _D. Lopez-Paz and M. Ranzato, \"Gradient episodic memory for continual learning,\" NeurIPS 2017._\n- **EWC**: _J. Kirkpatrick et al., \"Overcoming catastrophic forgetting in neural networks,\" PNAS 2017._\n- **LwF**:  _Z. Li and D. Hoiem, \"Learning without forgetting,\" ECCV 2016._\n- **L2P**: _X. Wang et al., \"Learning to prompt for continual learning,\" CVPR 2022._\n- **LFPT5**: _H. Qin et al., \"LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of T5,\" NeurIPS 2022._\n- **ProgPrompt**:  _L. Wang et al., \"ProgPrompt: Continual learning for language models,\" ICLR 2023._\n- **LB-CL**: _F. Qiao and M. Mahdavi, \"Learn more, but bother less: Parameter-efficient continual learning,\" *NeurIPS*, 2024._\n\n---\n\n**W2: Unclear how the approach scales to very long task sequences or when dealing with conflicting knowledge updates over time.**\n\n**A4:** Thank you for the insightful comment. We agree that scaling to very long task sequences and handling conflicting updates over time is a central challenge in continual learning. In our work, we have already evaluated the method on a relatively demanding benchmark with **15 sequential tasks** (lines 187–192, page 6), as shown in **Table 1**.\n\nIt is worth noting that, as shown in Table 1, **almost all continual learning methods perform significantly worse than PerTaskFT**, suggesting that scaling to a large number of tasks remains inherently difficult for the entire field—not just our method.\n\nNevertheless, our approach maintains competitive performance across all task counts and shows less degradation than other baselines as the number of tasks increases. We believe this demonstrates the scalability potential of our method, and we plan to explore even longer task sequences in future work."}]

Author rebuttal:

Meta-review: This paper proposes DEAL, a novel framework that integrates Low-Rank Adaptation (LoRA) with a continuous fine-tuning strategy that mitigates the limitations of existing FT methods while maintaining efficiency in privacy-preserving settings. 

Strengths: Wavelet-based knowledge retention is innovative. The proposed method is principled and derived from the theoretical analysis. 

Weaknesses: The claim of data privacy is not justified. Comparisons to many baseline methods are missing. 

Questions regarding scaling to larger models has been addressed the rebuttal. Stronger baselines have been added in the rebuttal. This is the main reason for my decision.
Decision: Accept (poster)