Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning
Authors: Haomiao Qiu
Miao Zhang
Ziyue Qiao
Liqiang Nie
Venue: NeurIPS 2025 poster ()
Keywords: Continual Learning
Model Merging
Primary area: applications
TL;DR: We propose P&M, a continual learning framework that mitigates forgetting via optimal post-training model merging and task-vector perturbations, achieving sota performance.

Abstract: Continual Learning (CL) aims to enable models to continuously acquire new knowledge from a sequence of tasks with avoiding the forgetting of learned information. However, existing CL methods only rely on the parameters of the most recent task for inference, which makes them susceptible to catastrophic forgetting. Inspired by the recent success of model merging techniques, we propose Perturb-and-Merge (P\&M), a novel continual learning framework that integrates model merging into the CL paradigm to mitigate forgetting. 
Specifically, after training on each task, P\&M constructs a new model by forming a convex combination of the previous model and the newly trained task-specific model. Through theoretical analysis, We minimize the total loss increase across all tasks 
and derive a closed-form solution for the merging coefficient under mild assumptions. To further improve the performance of the merged model, we observe that the degradation introduced during merging can be alleviated by a regularization term composed of the task vector and the Hessian matrix of the loss function. Interestingly, we show that this term can be efficiently approximated using second-order symmetric finite differences, and a stochastic perturbation strategy along the task vector direction is accordingly devised which incurs no additional forward or backward passes while providing an effective approximation of the regularization term. 
Finally, we combine P\&M with LoRA, a parameter-efficient fine-tuning method, to reduce memory overhead. Our proposed approach achieves state-of-the-art performance on several continual learning benchmark datasets. The code is available at \url{https://github.com/qhmiao/P-M-for-Continual-Learning}.

Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning
Haomiao Qiu!?, Miao Zhang Ziyue Qiao²', Liqiang Nie
Harbin Institute of Technology (Shenzhen)
2 Great Bay University
24B951058@stu. edu cn, zhangmiao@hit. weeeehhti..uu.... edu.cn, en, zyqiaolgbu.edie edu nieligiang@gmail. coo
Abstract
Continual Learning (CL) aims enable models to continuously acquire new knowledge from a sequence of tasks with avoiding the forgetting of learned infor- mation. However, existing CL methods only rely the parameters of the most recent task for inference, which makes them susceptible to catastrophic forgetting. Inspired by the recent success of model merging techniques, we propose Perturb- and-Merge (P&M), a novel continual learning framework that integrates model merging into the CL paradigm to mitigate forgetting. Specifically, after training on each task, P&M constructs a new model by forming a convex combination of the previous model and the newly trained task-specific model. Through theo- retical analysis, We minimize the total loss increase across all tasks and derive a closed-form solution for the merging coefficient under mild assumptions. To further improve the performance of the merged model, we observe that the degradation introduced during merging can be alleviated by a regularization term composed of the task vector and the Hessian matrix of the loss function. Interestingly, we show that this term can be efficiently approximated using second-order symmet- ric finite differences, and stochastic perturbation strategy along the task vector direction accordingly devised which incurs no additional forward or backward passes while providing an effective approximation of the regularization term. Fi- nally, we combine P&M with LoRA, a parameter-efficient fine-tuning method, to reduce memory overhead. Our proposed approach achieves state-of-the-art perfor- mance on several continual learning benchmark datasets. The code is available at httts:///ithhu: com hamm//hmiao.../ a for Montinual-LLarniin
1 Introduction
Continual Learning (CL) aims to enable models to continuously acquire new knowledge from sequence of tasks while retaining previously learned information M In many real-world applications, data arrives in streaming fashion, and due to constraints such privacy, storage, or computational resources, models are typically unable to retain or revisit earlier task data. This setting poses a model continually adapt to new tasks while maintaining good
fundamental challenge: how performance on previous ones? Although considerable progress has been made through methods
such parameter isolation 2 3 4 regularization [5] 6 g), and experience replay 9 10 12
catastrophic forgetting remains a core issue in CL.
In parallel, the rise of furge-scale pretrained models has spurred interest in model merging simple yet effective strategy for post-training enhancement. Prior studies have shown that independently
"Co-corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

trained models, initialized from the same pretrained weights and optimized on different tasks, can be effectively merged- through parameter interpolation or more structured methods-into a single unified model that performs well across tasks [13
14 15 16
17
19 20 Model merging offers 18
a practical mechanism for consolidating task-specific models without requiring shared training data.
Although CL and model merging differ significantly in their procedures, they share a common goal: learn a single model that performs well across multiple tasks. The key distinction lies in the assumptions and timing of parameter updates: CL follows a sequential paradigm, where only current-task data accessible and previous task parameters are reused for initialization; whereas model merging assumees that tasks are trained independently from the same pretrained model, without any data sharing. Both paradigms operate under task-isolated settings but differ in how and when task integration occurs.
A major advantage of CL is that all tasks trained along shared optimization trajectory, which increases the likelihood that the resulting parameters reside near a joint optimum [21 [22]. In contrast, model merging offers a stable post-training integration mechanism, particularly attractive in scenarios where previous data cannot be revisited. Notably, most CL methods use the model parameters obtained after task to perform inference on all tasks to 6, even though these parameters are mainly optimized for the current task (23
24
Motivated by this observation, we propose novel method, Perturb-and-Mergg (P&M, which unifies the training dynamics of CL with the inference principle of model merging.
Infer after Merging. Specifically, for task t, we initialize training with the inference parameters 0,-1 from previous tasks, and after training, obtain the task-specific optimum 0r. We then merge the two using the convex combination: 0, (1 a + a0.- This process can be viewed as scaling the task vector A6; = = 8:_ to reduce interference with previously learned knowledge. To determine the optimal merging coefficient a. we analyze its influence on the total loss across all tasks and derive closed-form solution that minimizes overall performance degradation. This solution depends on the Hessian matrix at the optimum of each task, which we approximate using the empirical Fisher information matrix.
Train with Perturbation. flicts flicts-causing the merged model to underperform compared task-specific models find that this degradation can be mitigated by introducing regularization loss term composed of the task vector and the Hessian matrix during training. However, computing the Hessian matrix on a per-batch basis is computationally expensive. Interestingly, we show that this regularization term can be efficiently approximated using second-order symmetric finite differences. More importantly. injecting vask-vector-aligned parameter perturbations during training provides a stochastic approximation of the regularizer, requiring no additional forward or backward passes. Our experiments demonstrate that such perturbations can effectively reduce parameter conflicts during model merging and enhance the performance of the merged model.
Furthermore, because model merging can introduce parameter con-
Finally, we combine P&M with LoRA (Low-Rank Adaptation) 25 parameter-efficient fine-tuning strategy, to reduce memory overhead for model storage. Our overall approach achieves tate-of-thee-art performance on multiple CL benchmark datasets.
2 Related Work
2.1 Continual Learning
CL aims to enable models to learn new tasks sequentially without forgetting previously acquired knowledge. Regularization-bart based methods preserve important parameters by applying constraints to prevent forgetting 6 6 7 8) while memory- based approaches use external buffers to store historical Architecture-bats methods dynamically expand model Another direction focuses on constraining gradient directions to reduce task interference. Examples include Orthogonal Weight Modulation (OWM)266, Orthogonal Gradient Descent (OG and Gradient Projection Memory (GPM) [28 which project gradients into task-specific subspaces retain prior knowledge.
data for rehearsal or sampling [9
10
capacity to accommodate new tasks 2 3
With the rise of large-scale pre-trained models 229 increasingly popular. However, full fine-tuning is computationally expensive, and more parameter-
continual fine-tuning has become

---

efficient tuning strategies- suchh methods -35-have been proposed to improve the scalability and practicality of CL. this work, we integrate our method with LoRA to reduce both training and storage costs.
such as prompt baspt--bsedddee learning [32 33 34], LoRA, and modular tuning
2.2 Model Merging
Model merging has gained traction in both multi-task and CL scenarios. Generally, it can be categorized into two directions: one line of work merges models fine-tuned on the same task to the other merges models trained on different tasks to form single, unified model that can handle all constituent tasks. Model merging aims to integrate knowledge
improve generalization 36 37
38 39
from different tasks without retraining from scratch 40 4T 42 Simple strategies use fixed merging weights based on the number of tasks 43 while more ad advaneed methods such as IMM and CoMA [2 empirically tune coefficients to improve performance.
Some approaches [45 39 go further by assigning parameter-wise merging coefficients to reflect the heterogeneous impact of different weights, but such methods often require extensive hyperparameter tuning. In contrast, our method provides a closed-form, theoretically optimal merging strategy without the need for manual tuning.
3 Method
3.1 Infer after Merging
In continual learning, there T tasks Ti.... Tr that each task includes data: D; {(x), yf) 9f))." where x E I2 is the input, vf IR is the label, and (-) is pretrain model with parameters Og-
When receiving the t-th task, the goal of CL is to achieve an overall optimal performance across all T tasks. Since we currently only have access to data from task t, can only update O, by training on D.. Given an input x with ground-truth label from D. and a model prediction p softmax(foxx)) E R,, the cross-entropy loss is defined as:
L (0) I(y - c) log P. - logggy
where Py is the predicted probability for the correct class y and II() is the indicator function. At this point, we have the model Ú,-1 used for inference on tasks to 1-1, which we take as the starting point for training. After training with Eq. [ we obtain the optimal parameters 02 on the dataset D,;
02 - Ú,- A07- = Ō. argmin Co A0.).
Ao.
Traditional CL methods directly use OR the inference parameter for the 1-th task, i.e., 0. 07- However, since of- is trained on D., may not fully guarantee the performance on Dı,..., D,-1-
A natural idea is to combine models trained on different tasks. Specifically, Ô.-1, which be used as the inference parameter for these tasks, performs well on tasks 1 to 1, while 87, which is optimized for task t, serves as the inference parameter for task t. Therefore, models. Here we use a simple convex combination:
merge the two
0, = (1 (1-0))0-1+aa +
where S a; S 1 is the merging coefficient used to balance the weights of the two parameters, and is used for inference on tasks I to f. Since 07 is trained based on 0.-1, 0ž = Ô,-1 + A0; then the Eq. 3 can be rewritten as:
- + A,
4
which means that taking the weighted average of 0,-1 and 02 is equivalent to scaling A0,. This scaling strategy means that it will not harm the parameters 0-1 for tasks 1 to / 1, but only by adjusting the task vector for task / to reduce forgetting old tasks. our experiments 4.3 that this scaling has slight impact on the knowledge of the new task.
we found
3

---

Table 1: Training and Inference Strategies
Task t
Training: 0 argmino, (1))
Inference: O..
Continual Learning
0i 0-- + A0;
0, 0²
Model Merging
0i 00 + A0i
0, Dim1 a;O;
Ours (P&M)
0r = 01-1 + A0I
0. 0.-1 + a4
Infer after Merging unifies the benefits of model merging and continual learning, as demonstrated in Tab. In contrast to conventional CL, it mitigates interference with previously acquired knowledge while encouraging more compact task-specific optima through continued addatatton-uultimateel promoting better generalization [2 46). Unlike momentum updates exponential moving averages (EMA) [47] [48] that perform step-wise smoothing, our method merges the full task vector after completing each task. Instead of relying on historical gradients or parameter trajectories, it directly combines the current task's 's optimal solution with the previous model, better preserving task semantics and overall structure.
3.2 A Closed-form Solution for Optimal Merge Coefficient
Next, we aim to obtain an optimal a₃ We are concerned with the performance degradation of the merged model compared to each task-specific optimal model 07, where i S For task we define the performance drop 8; as:
6, C,(,)) -
- (.))))
(5)
6, evaluates the impact of the merged model O, on the loss of task i compared to its optimal parameter 0% We expand into Taylor series, then we have
6 = (L,) - 0,) + 2 (o. 03) T H.
03) + O(I0,, - 0, 11s,
(6)
where H;,0;) represents the corresponding Hessian matrix. We can consider that for task i, 07 the optimal parameter for task i, so VL, tends to O. And can assume that 1|0, 0g l|s tends to
so we only need to consider the second-order term 1 (0;- - -0)) (O;))( -,)) Then we consider the change in the loss for all tasks, i.c..
a- = argmin
Σ 8, ~ argmin
1 (0, 01) H())) 0z),
Cxe
1=1
a1
which has the closed-form solution:
Ei (0-1 0:
H. (0;))00;
(8)
Ei=1 H,
Please refer to the Appendix A for the detailed derivation. To avoid the prohibitive cost of computing the full Hessian, we approximate it with the diagonal of the empirical Fisher information matrix Given a model with parameters 0, the empirical Fisher matrix is defined as:
49 50
F(0)
E (xx))> [Vo log po
x) V) log po(y x)
(9)
In practice, we approximate F(0) using the average over the whole training dataset:
N
1 N
F(0) =
Vo log x, log x:)
(10)
and use only the diagonal entries of F(O), After completing the training of task t, we first compute the diagonal empirical Fisher matrix F, ((;)) z H,(()), then obtain of using Eq. g and finally compute the merged model 0, 0.1 + aj A0; the inference parameters for tasks 1 to t. Note that for each previous task < 1, the corresponding Fisher matrix F((0?) computed and stored at the time of training task i.

---

3.3 Train with Perturbation
Further, we can also reduce Σi=1 5;(a;) make the merged model optimal on all tasks by optimizing 0r. Next, we abbreviate H as Hi, then we have
EE (0--1 0: T H.AoF) )²
2
L Σ (((,,) = Σ (0-1- -6:) oo)] T H; (0--1 -8-1 8:)
(11)
2 1=1
2 E1 10; H,A0
00-1- - 0:) $ Σ 1=1
T H. (0--1 - 07)
(12)
For the details, please refer to the Appendix A.2
To enhance the performance of the merged
model, we aim to minimize the upper bound of Σ ci= 5;(a;)) Among all the terms in this ex-
pression, only the last term (z)) involving 07 -f-ppccifiallly, (0.-1 1 0: H, (o-- 1 02
A0; H1 on the current task and can be optimized during training.
During training, this term can be added as a regularizer to reduce ó,(a)), but it requires real-time Hessian computation. While the Fisher Information Matrix is commonly used surrogate, it only approximates the Hessian near the optimum under certain conditions (e.g.. the number of data samples N - oo). Since training parameters are far from optimal and the Fisher can only be estimated from a single batch during training. it fails to capture the true curvature and is thus unreliable in this setting. Fortunately, can approximate the quadratic form using symmetric finite differences as follows:
Ad, T H,A0,
(cT cL0.) + 2 cL0.) 2C (0c)).
(13)
where < 1 is a small constant, A0, denotes the task vector during training and 02 - Ō2- A0, This approximation is derived via a second-order Taylor expansion of the loss around 9. We incorporate the right-hand side of Eq. 13 as regularization term during training. and define the total training loss as:
(0,) (0:) + AC (0t)
X = 1 L. (0,) + 2 (C, (0, + ca0c)) 27, -laf))
2A
(14)
where À control the strength of the regularization term. Eq. 14 requires two additional forward passes of the cross-entropy loss with parameter perturbations along the task vector direction in each training step. While this avoids computing the full Hessian, it results in a threefold increase in memory and computation cost. To improve efficiency, we propose a stochastic approximation.
We observe that Eq. 14 is effectively equivalent to applying a random perturbation to the model parameters during trainin,, where the perturbation direction aligns with the task vector of task t. Specifically, the loss is evaluated under one of three perturbations: eAit, -eAtt, or O. Specifically, each training step, we sample one of the three versions of the loss:
(0.)
with probability po:
(,()))
(0, + eA0;)
with probability P++
(15)
(7, - cd07)
with probability p-.
We define the sampling probabilities for the three perturbation terms as follows: the original point is sampled with probability Po 24, and the two perturbed points are each sampled with probability
p+ =p- 2. In the experiments, we treat Po as a hyperparameter, and set p+ = p- = { - (1)))- These probabilities are designed to ensure that the expected loss remains consistent with the original definition, resulting in an unbiased estimate of the total loss:
E [[2. = CT (0) + CP + <a0;) + Pp- - <l0;) = ()).
(16)
This sampling strategy reduces the forward cost per batch from 3x to 1x, without introducing bias into the gradient estimation. While variance may increase slightly, this technique enables us to scale P&&M to large models and datasets efficiently.
5

---

Algorithm LORA-P&M
Input: Datasets Dj (x 12 for T tasks T1, Ti...., Tr, pre-trained model fo(-) with
parameters Op. fo = fo, hyper-parameters Po and p+ =p- = {1 -P))
2: Output: 07 for all tasks.
3: for 1 to do
4: Fix 0 and initialize new task-specific LoRA set for task t: LoRA,
5: repeat
6:
for each example (x vf) Dj do
7:
> Train with Perturbation:
8:
Sample perturbation é E {-e, 0, +e} according to probabilities (p, p*}
9:
Compute loss LT (A,-1 + (1 + L x;, v/) Update LoRA, via A
10:
11:
end for
12:
until convergence
13:
Infer after Merging:
14:
0r = 0 1 + LoRAt
15:
Estimate the Fisher Information Matrix (0]) task Tr using Eq. and Alg.
16:
0, - 0. aoLoRA, where of is from Eq, 8
end for
3.4 LoRA-P&M
Note that computing Eq. 8 requires storing the optimal parameters 0j of all previous tasks. As the number of tasks grows, this results in increased memory demands. To address this issue, we integrate LoRA (25], a low-rank parameter-efficient fine-tuning method, to reduce storage overhead.
For task t. the linear layer becomes W, W,-1 + A,B,, where only A, and B, are updated. LoRA reduces trainable parameters by decomposing the weight update into low-rank matrices: AW = A.. In CL. new LoRA module will be added for new task t, while other modules are kept fixed and participate jointly in the forward pass 23 51
Let PLORA = (A(p)) (P), B(7) | E I} denote the set of LoRA modules inserted at various locations in the network, where I indexes the parameter subsets affected. The overall model parameters 9r can then be written as:
0: = 0,-1 +
(A B!())
(17)
PEP
denotes location- Docation-wise module insertion, not element wise addition.
In our experiments, we apply LoRA modules to the key and value projections. so that only the Fisher Information Matrices of these parameters need to be stored. The experiments comparing the memory and time overhead of our method with LoRA are provided in the Appendix B.3 The overall procedure of the algorithm is presented in Algorithm
Experiments
In this section, we first present the experimental setups, and then compare P&M with state-of-the-art CL methods and model merging methods across multiple benchmarks.
4.1 Experimental Setups
Evaluation Benchmarks and Metrics. Following the evaluation protocols in 35 23
we assess
LoRA-P&M on five standard CL. benchmarks: ImageNet-R [52]. ImageNet-A (53), DomainNet (54).
CIFARI00 [55). and CUB200 [56). As in prior work [23 21, split ImageNet-R into 5, 10, and

---

Table 2: Performance comparison with CL methods on ImageNet-R across different task lengths.
ImageNet-R R 5 tasks
ImageNet-R 10 tasks
ImageNet-R R 20 tasks
Method
Acc 1
AAA 1
Acc T
AAA T
Acc 1
AAA 1
Full Tun-Tuning
64.92+ 0.87
75.57+ 0.50
60.57+
1.06
72.31+ 1.09
49.95+ 1.31
65.32 0.84
L2P
88
73.04 0.71
76.94= 0.41
71.26
0.44
76.13 0.46
68.974
0.51
74.16= 0.32
DualPrompt [58
69.99.1 0.57
72.241
0.41
68.22.1
0.20
73.81 0.39
65.231
0.45
71,30 0.16
CODA-Prompt
76.631 0.27
80.30= 0.28
74.05:
0.41
78.141 0.39
69.384: 0.33
73.95= 0.63
HiDe-Prompt
62
74.77+ 0.25
78. 0.24
74.65+ 0.14
78.46+ 0.18
73.59+ 0.19
77.93 0.19
InfLoRA 23
76.95 0.23
81811
74.75 0.64
80.67 0.55
69.89+ 0.56
76.68= 0.57
SD-LoRA 21
79.15.1. 0.20
83.01 0,42
77.34
82.04 0.24
75.26 0.37
80.22 0.72
LoRA
71.224 1.47
78.15= 1.08
65.72+
76.144 01.96
56.35+ 0.80
71.08 1.04
LoRA-P&M
81.47 = 0.56
85.96 0.52
79.95 + 0.18
85.29 = 0.93
76.37 = 0.09
82.77 + 0.71
Table 3: Performance comparison with model merging methods on 5 datasets.
INR-10
INR-20
INA-10
DN
C100-10
CUB-10
Method
Acc
Acc f
Acc f
Acc 1
Acc t
Acc f
LoRA
65.72
56.35
44.41
71.81
72.58
64.82
w/ Model Averaging
76.90
74.64
54.54
81.84
87.52
74.87
w/ DARE 39
75.09
66.03
55.87
80.58
87.28
76.57
w/ CoMA
63
79.34
75.60
53.24
83.98
86.95
74.65
w/ COFIMA
63
79.06
75.09
54.09
83.85
86.58
74,43
w/ P&M
79.95
76.37
56.57
84.71
88.45
78.29
20 tasks; ImageNet-A into 10 tasks; DomainNet into 5 tasks; and both CIFAR100 and CUB200 into 10 tasks. Specifically, DomainNer refers the full version with all 345 classes, while DN* denotes variant where we select the top 200 most populous classes, following [57
Following (21), we report two widely used CL metrics: average accuracy (Acc) and average anytime accuracy (AAA). Acc computes the mean accuracy over all N tasks after all tasks is completed. AAA further captures learning dynamics by averaging accuracy on all seen tasks after training on each new task.
Competing Methods and Implementation Details. We compare LoRA-P&M against state-of- the-art ViT-based CL. methods, including I 1.2P (33, DualPrompt (58) CODA-Prompt 32, HiDe- Prompt [59]. InfL.oRA
form of performance lower bound. We also compare against model merging methods, including model averaging, DARE (39]
2 and SD-LoRA 21. while full fine-tuning
CoMA 22 and CoFIMA [22]. Following prior work [35), employ ViT-B/16 [30], pre-trained on ImageNet-21K and fine-tuned on ImageNet-IK as the foundation model for classification. Fol- lowing 23 insert LoRA (rank=10) modules into the key and value projections in multi-head attention. We set c
0.5 in Eq. 15 with uniform sampling Pu p+ = p 3. Our method is optimized using AdamW 600 with an initial learning rate of 1e-3 for LoRA and 1e-2 for the classification head following [61. We use batch size of 256 across all datasets. Each task is trained for 10 epochs, except for DomainNet, which is trained for 5 epochs. We report the mean and standard deviation over three runs to reflect the stability of the results. All results are obtained by running single NVIDIA L40s GPU.
4.2 Main Results
Comparison with State-of-the-Art CL Methods. As shown in Tab. LoRA-P&M consistently outperforms both the original LoRA baseline and recent CL methods, including SD-LoRA (21). across all evaluated settings. On ImageNet-R with 5, 10, and 20 tasks, method yields significant gains over LoRA (c.g., + +14.23% in the 10-task setting) and surpasses SD-LoRA by up to +2.61% %. For additional results NetainNet, ImageNet-A, CIFARI00. and CUB200, please refer to Appendix B
Comparison with Model Merging Methods. Tab. 3 compares LoRA-P&M with LoRA and several representative model merging methods across six benchmarks. P&M delivers substantial the
improvements over LoRA LoRA-up to +13.6% on INR-20 and +1 +11.1% 1% on 1100--0-highhlighe benefits of post-training merging in preserving task knowledge. While methods such as DARE and CoFIMA offer moderate improvements over LoRA, they consistently underperform compared to P&M. Notably, P&M exceeds CoFIMA by +3.86% on CUB-10 and +2.48% on INA-10.

---

IMAGENET 20 tasks
CIFAR10D tasks
DN
CUB2000 tasks
100
LORA
LoRA LORA-PM
LoRA LoRA PGM
LoRA
LoRA: RAM
LoRA-PAM
60
60
40
20
Plsticity
Forgeting
ACC
Plasticit
Forgetting
ACC 1
Plasticity
Forgetting 4
ACC
Plastcity
Forgeting
ACC
Figure 1: P&M reduces forgetting with minimal impact on plasticity, Comparison of LoRA and LoRA-P&M across four benchmarks. P&M achieves similar plasticity (current task performance) while significantly mitigating forgetting (average performance drop on previous tasks), resulting in higher overall ACC.
Table 4: Performance comparison on ImageNet-A and DomainNet.
ImageNet-A 10 tasks
DomainNet 5 tasks
Method
Acc f
AAA f
Acc
AÁA f
Full Fine-Tuning
16.31 + 7.89
30.04 = 13.18
51.46 = 0.47
67.08 + 1,13
L2P 133
4
51.40 1.95
70.26 0.25
75.83 0.98
DualPrompt
58
0.96
54.68 1.24
68.26 = 0.900
+ 73.84
CODA-Prompt
32
0.78
$7.03:0..9
70.58 =0.53
76.68
0.44
HiDe-Prompt
[59
+
56.32 +
72.20 0.08
77.01 0.04
Infl.oRA [23
49.20 1.12
60.92
71.59 0.23
+.550
SDLoRA [21]
55.96 0.73
64.95
72.82 0.37
78.89 0.50
SD-LoRA RR 21
55.59 + 1.08
14.59:10
72.58 = 0.40
78.79
+ 0.78
SD-LoRA-KI KD 21
54.24 + 1.12
63.89
72.15 = 0.50
78.44 0.66
LoRA
44.41 + 0.57
3.57
64.82 2.30
75.28 = 1.56
LoRA-P&M
56.57 + 0.78
65.35 1.81
76.91 = 0.53
85.27 = 0.07
More results on DomainNet, ImageNet-A, CIFARI00, and CUB200. As shown in Tab. and 5 Across all four benchmarks, LoRA-P&M consistently outperforms both standard LoRA and the recent SD-LoRA. On challenging datasets such as ImageNet-A and DomainNet, it achieves the highest accuracy and AAA, surpassing SD-LoRA by up to +0.61 % and +6.38% respectively. Notably, on CIFAR 100 and CUB200, LoRA-P&M matches or exceeds SD-LoRA's accuracy while maintaining lower variance. These results highlight the robustness and generalization advantage of post-training task vector merging over both naive LoRA and structured dynamic variants like SD-LoRA.
Ablation on the Merging Coefficient Strategy. A potential concern that using uniform merging coefficient a across all parameters might be overly restrictive. To evaluate this, we compared our global-a strategy with a more -neiggri variant that learns separate values for each LoRA module. The results on ImageNet-R are summarized in Table [ The performance differences are negligible. We further observed that the learned per- per-module a values were highly similar to each other, suggesting that single global coefficient is sufficient to capture the merging dynamics. Therefore, we adopt the global-or strategy for simplicity, efficiency, and theoretical tractability, This finding aligns with prior research [13] where a unified coefficient was shown to preserve the structural integrity the task vector A0,, which represents coherent direction in parameter space. In contrast, a parameter-wis may introduce inconsistent scaling and distort the internal correlations of A thereby complicating the derivation of closed-form solutions.
4.3 Analysis
This subsection investigates how P&M enhances CL performance.
Observation @: Scaling task vectors reduces forgetting with minimal impact on plasticity. We define plasticity as the average accuracy on the new task and forgetting as the average accuracy drop on previously learned tasks. As shown in Fig. I scaling the task vector during inference (P&M)
8

---

Table 5: Performance comparison CIFARIO0 and CUB200.
CIFARI00 10 tasks
CUB200 10 tasks
Method
Acc f
AAA
Acc f
AAA
Full Fine-Tuning Tuning
69,49 + 0.50
80.35 + 0.87
51.43 1.41
69.74 = 0.93
L2P (3
83.18 + 1.20
87.69 1.05
65.18 + 2.49
76.12 + 1.27
DualPrompt 58
81.48 - 0.86
86.41 #
79.40 = 0.88
CODA-Prompt 332
86.31 = 0.12
90.67 71.92 0.22
78.76 = 0.65
InfLoRA
23
86.75 = 0.35
91.72
70.22
81.39 = 0.14
SD-LoRA 21
88.01 + 0.31
92.54
0.18
77.48 + 0.20
85.59 + 0.44
LoRA
72.58 + 1.57
30.64 2.31
64.82 2.30
75.28 1.56
LoRA-P&M
88.45 + 0.35
92.89 + 1.13
78.29 + 0.50
83.39 = 0.61
Table 6: Acc comparison between global and per-module strategies ImageNet-R.
Method
5 tasks
10 tasks
20 tasks
Per-modulc
80.53
76.82
74.68
Global a (ours)
80.88
78.48
74.13
maintains plasticity comparable to that of the standard task vector, as also observed in prior work (64, while significantly reducing forgetting and thus improving overall performance.
Then we analyze its behavior through loss landscape visualizations and directional perturbation experiments. Specifically, the merged model is computed as: 0₂ = 3 0 1 + a 0f, where a is derived from Eq. and B 1 a.
Figure visualizes the average loss landscape ImageNet-R (left) and CUB (right). we visualize merging after Tasks and 7. Each subplot spans 2D convex space formed by Q and f7, where the horizontal axis indicates the task-specific model weight B, and the vertical axis indicates the previous model weight a. Each point corresponds to a merged model, colored by its average loss across all learned tasks. M refers to only Infer after Merging without applying Train with perturbation.
Observation @: Convex Combination lies in Low-Loss Regions. Our method determines coefficient a and computes the merged parameter as a convex combination. This combination represents direct interpolation between the starting and ending points of task 4's training. As the loss contours in all eight plots reveal, the convex paths between 0g and f1. 0g and 8 consistently lie in low-loss region. This supports our design choice of convex combination over other merging methods, such as task arithmetic (65), which may move the merged model into unstable regions.
We next evaluate the effectiveness of the coefficients computed by our method. In each left-hand plot (without parameter perturbation), the merged point obtained using our optimal consistently lies in a lower-loss region compared to either endpoint f or 07 This indicates that our closed-fon solution reliably identifies favorable interpolation points on the loss surface, providing empirical support for the theoretical foundation of our merging formulation.
Observation @: Optimal Coefficients Locate Low-Loss Interpolation Points.
Observation @: Task- Vector Perturbation Encourages Flat Minima and Better Generalization. In each dataset's visualization, the left plot corresponds to training with standard cross-entropy (CE) loss, while the right plot depicts training with CE loss combined with parameter perturbation. The loss contours indicate that the additional perturbation enlarges the flatness and width of the low-loss basin around the merged model, making it more likely for the model to fall near an optimal region. This suggests that parameter perturbation helps avoid sharp minima and reduces parameter interference during model merging, thereby enhancing the generalization ability of the merged model. To further evaluate the importance of perturbation direction, we compare task-vector-basee random Gaussian noise that has the same Frobenius norm. As shown Tab. 7 models perturbed along the task vector consistently outperform those using random noise. This demonstrates that applying perturbation in the direction of the task vector provides a reliable approximation Eq. thereby improving the performance of model merging.
perturbation with

---

ImageNet-R 10 tasks
CUB 10 tasks
Lass
Loss
8
8
10
8
84
8=
300
1.00
0.5
230
0
0
0.0
0.0
1400
0.5
0.0
0.5
1.0
0.5
6.0
0.5
1.5
0.5
0.0
0.5
9.5
1.0
(a) Tasks 1-4 (M)
(b) Tasks 1-4 (P&M)
(c) Tasks 1-4 (M)
(f) Tasks 1-4 (P&M)
Lass
Losss
8
2.5
0.5
6
8,
8
3.
0.0
(c) Tasks 1-7 (M)
(d) Tasks 1-7 (P&M)
(g) Tasks 1-7 (M)
(b) Tasks 1-7 (P&M)
Loss landscape visualization on ImageNet-R and CUB. M denotes using only Infer after Merging (no perturbation during training). Each subplot shows the average loss surface after merging at Task 4 and Task 7, with axes representing weights a and 3 in the convex combination 0, = -8;-11 + ad,, The convex path lies in a low-loss region (Obs. @)), and our optimal a consistently locates near the minimum (Obs. @). Task-vector perturbation further enlarges the flat region (Obs. +).
Figure 2:
Table 7: The ablation study of the proposed P&M.
INR-10
INR-20
INA-10
DN
C100-10
CUB-10
Method
Acc 1
Acc f
Acc 1
Acc
Acc 1
Acc
LoRA LoRA-M
65.72
56.35
44.41
71.81
72.58
64.82
78.35
74.26
56.16
81.28
86.57
74.98
LoRA-M w/ gauss noise
78.48
74.13
49.51
83.00
85.83
74.09
LoRA-P&M
79.95
76.37
56.57
84.71
88.45
78.29
In summary, P&M improves CL through three key aspects: (1) Post-training scaling mitigates forgetting; (2) The convex combination of models, along with a theoretically grounded optimal coefficient, ensures that the merged model lies near an optimal region; (3) Task-vector perturbation perturbations enhance generalization and reduce parameter interference.
Conclusion
We propose P&M, a novel CL framework that incorporates post-training model merging into the learning paradigm. By combining theoretically grounded merging strategy with task-vector-aligned perturbations, P&M effectively mitigates catastrophic forgetting. Our approach merges models via convex combination of the current task optimum and the previous model, with the optimal coefficient derived from a closed-form, loss-based objective. To further enhance robustness, we introduce a lightweight regularization mechanism during training that applies stochastic perturbations along the task vector direction to improve the performance the merged model. Integrated with LoRA, P&M offers memory-efficient solution and achieves strong performance across various CL benchmarks. Experimental results show that unifying training-time dynamics with post-training merging provides simple yet effective strategy for building continual learners with strong generalization and stability.
Limitation. Our method estimates the optimal merging coefficient using an analytical form based on the diagonal empirical Fisher Information Matrix. However, this diagonal approximation may not fully capture the true curvature of the loss landscape, and thus does not always guarantee optimality. Exploring more accurate yet efficient curvature approximations, is direction for future work.
10

---

Acknowledgment
Miao Zhang was partially sponsored by the National Natural Science Foundation of China under Grant 62306084 and U23B2051, Shenzhen College Stability Support Plan under Grant GXWD2023 128102243003, and Shenzhen Science and Technology Program under Grant DDSYS2202306226012000000 and KIZD202309231155 13326 Ziyue Qiao, School of Computing and Information Technology. Great Bay University, was supported by the National Natural Science Foundation of China (Grant No. 62406056) and the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515140111
References
[1] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter, Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
[2] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins. Hubert Soyer. James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXiv:
2016.
[3] Steven C. Y. Hung, Che Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu-Song Chen. Compacting, picking and growing for unforgetting continual learning. In Advances Neural Information Processing Systems, pages 13647-13657, 2019.
[4] Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting. Proceedings of the International Conference on Machine Learning, pages 9925-3934, 2019.
[5] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In ICML, pages 3987-3995, 2017.
[6] Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node-importance based adaptive group sparse regularization. Advances in Neural Information Processing Systems, pages 3647-3658, 2020.
[7] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In ECCV, pages 139-154, 2018.
[8] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka al. Overcoming catastrophic forgetting in neural networks. PNAS, 114(13)3521-3356, 2017.
[9] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lueas Page-Caccis Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pages 1189-11866) 2019.
[10] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, pages 11816-11822 1825, 2019.
[11] Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan. Exploring example influence in continual learning. Advances in Neural Information Processing Systems, pages 27075-27086, 2022.
[12] Yan-Shuo Liang and Wu-Jun Li. Loss decoupling for task-agnostic continual learning. Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances Neural Information Processing Systems, 2023.
[13] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference Learning Representations, 2023.

---

[14] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Ties- meegingg Resolving interference when merging models. In NeurIPS, 2023.
[15] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In ICLR, 2024.
[16] Michael Matena and Colin Raffel. Merging models with weight- averaging. In NeurIPS, 2022.
[17] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietre and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In ICLR, 2023.
[18] Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Moham- mad Emtiyaz Khan. Model merging by uncertainty-based gradient matching. In ICLR, 2024.
[19] Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309. 1598,, 2023.
[20] Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in Ilms, milms, and beyond: Methods, theories, applications and opportu- nities. arXiv preprint (4X.:::408.07666 2024.
[21] Yichen Wu, Hongming Piao, Long- Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. Sd-lora: Scalable decoupled low-rank adaptation for class incremental learning.
[22] Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. Weighted ensemble models strong continual learners. In ECCV, 2024.
[23] Yan-Shuo Liang and Wu-Jun Li. Inflora: Interference-free low-rank adaptation for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 23638-23647,
[24] Hao Yu, Xin Yang, Xin Gao, Yan Kang, Hao Wang, Junbo Zhang, and Tianrui Li. Personalized federated continual learning via multi-granularity prompt. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4023 4034, 2024.
[25] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[26] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 1(8)33643722 2019.
[27] Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li, Orthogonal gradient descent for continual learning. In AISTATS, pages 3762-3773, 2020.
[28] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In ICLR, 2021.
[29] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[30] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference Learning Representations, 2021.
[31] Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171-4186. 2019.
12

---

[32] James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante Bonilla, Donghyum Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11909-11919, 2023.
[33] Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[34] Muhammad Gul Zain Ali Khan, Muhammad Ferjad Naeem, Luc Van Gool, Didier Stricker, Federico Tombari, and Muhammad Zeshan Afzal. Introducing language guidance in prompt- based continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 11463-11473, 2023.
[35] Qiankun Gao, Chen Zhao, Yifan Sun, Teng Xi, Gang Zhang, Bernard Ghanem, and Jian Zhang. A unified continual learning framework with general parameter-efficien tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1
2023.
[36] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In NeurIPS, volume 34, pages 22405-22418, 2021.
[37] Mitchell Wortsman, Gabriel Ilharco. Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes. Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICMI., pages 23965-23998 PMLR, 2022.
[38] Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renormalizing permuted activations for interpolation repair. In ICLR, 2023.
[39] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yonghin Li. Language models super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference Machine Learning. 2024.
[40] Zafir Stojanovski, Karsten Roth, and Zeynep Akata. Momentum-basee weight interpolation of strong zero-shot models for continual learning. In NeurIPS Workshop, 2022.
[41] Christian Simon, Masoud Faraki, Yi-Hsuan Tsai, Xiang Yu, Samuel Schulter, Yumin Suh, Mehrtash Harandi, and Manmohan Chandraker. On generalizing beyond domains in cross- domain continual learning. CVPR, pages 9265-9274, 2022.
[42] Janghyeon Lee, Donggyu Joo, Hycong Gwon Hong, and Junmo Kim. Residual continual learning. In AAAI, volume 34, pages 4553-4560, 2020.
[43] Guoliang Lin, Hanlu Chu, and Hanjiang Lai. Towards better fasticitt-sttbbiliy trade-off in incremental learning: A simple linear connector. In CVPR, pages 89-98, 2022.
[44] Sang-Woo Lee, Jin-Hwa Kim, Jachyun Jun, Jung-Woo Ha, and Byoung-Tak Tak Zhang. Overcoming catastrophic forgetting by incremental moment matching. In NeurIPS, volume 30, 2017.
[45] Daniel Marczak, Bartlomiej Twardowski, Tomasz Trzciński, and Sebastian Cygert. Magmax: Leveraging model merging for seamless continual learning. In ECCV, 2024.
[46] Yeoreum Lee, Jinwook Jung, and Sungyong Baik. Mitigating parameter interference in model merging via sharpness-aware fine-tuning. arXiv preprini arXiv:2504. 21t2::2200..4622.. 2025.
[47] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
[48] Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7201-7211, 2022. 2022.
13

---

[49] James C Spall. Monte carlo computation of the fisher information matrix in nonstandard settings. volume 14, pages 889-909. Taylor & Francis. 2005.
[50] James C Spall. Improved methods for monte carlo estimation of the fisher information matrix. In 2008 American Control Conference, pages
IEEE, 2008.
[51] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng. Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning, arXiv preprint 1rXiv:2310.141555 2023.
[52] Matteo Boschini, Lorenzo Bonicelli, Angelo Porrello, Giovanni Bellitto, Matteo Pennisi, Si- mone Palazzo, Concetto Spampinato, and Simone Calderara. Transfer without forgetting. In Proceedings of the European Conference on Computer Vision, pages 692-709, 2022.
[53] Dan Hendrycks, Steven Basart, Mantas Mazeika, Andy Zou, Christina Kwon, and Jacob Steinhardt. Natural adversarial examples. In JEEE/CVF Conference Computer Vision and Puttern Recognition, pages 15262-15271, 2021.
[54] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1406-1415, 2019.
[55] A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
[56] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie, The Caltech- UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-200 1-001, California Institute of Technology, 2011.
[57] Yue Lu, Shizhou Zhang, De Cheng, Yinghui Xing, Nannan Wang, Peng Wang, and Yan- ning Zhang. Visual prompt tuning in null space for continual learning. arXiv preprint 2xXiv:4066006558 2024.
(58) Zifeng Wang, Zizhao Zhang. Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen- Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In ECCV, pages 631-648, 2022.
[59] Liyuan Wang. Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. In Advances in Neural Information Processing Systems, pages 69054-69076, 2024.
[60] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018.
[61] Gengwei Zhang, Liyuan Wang, Guoliang Kang, Ling Chen, and Yunchao Wei. SIea: Slow learner with classifier alignment for continual learning on a pre-trained model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19148 19148-19158, 2023.
[62] Liyua Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical decomposition of prompt-based continual learning: Rethinking obscured sub-optimality. arXiv preprint arXiv:23 2023.
[63] Imad Eddine Marouf, Subhankar Roy, Enzo Tartaglione, and Stéphane Lathuilière. Weighted ensemble models are strong continual learners. In European Conference Computer Vision, pages 306-324. Springer, 2024.
[64] Ke Wang, Nikolaos Dimitriadis, Alessandro Favero, Guillermo Ortiz-Jimenez, Francois Fleuret, and Pascal Frossard. Lines: Post-training layer scaling prevents forgetting and enhances model merging. 2024.
[65] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannanch Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023.
[66] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 248-255, 2009.
14

---

[67] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning, arXiv preprini arXiv:2310. 14152, 2023.
[68] Jingyang Qiao, Xin Tan, Chengwei Chen, Yanyun Qu, Yong Peng, Yuan Xie, et al. Prompt gradient projection for continual learning. In ICLR, 2024.
[69] Zhanxin Gao, Jun Cen, and Xiaobin Chang. Consistent prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28463-28473, 2024.
[70] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-335 2017.
et al.
15

---

A Proof and Theoretical Details
A.1 Proof of Eq. 8
To determine the optimal at, we consider the following optimization problem:
04 = arg min Σ
de Fs arg min
Σ 1 2 (0.11-0)) 10:-00 03 H;(0)) (0--1 - 0;)
(18)
a2
where 0 denotes the inference-time model parameters after learning on tasks through 1, and Or represents the optimal parameters for task i. The matrix (,,),) denotes the Hessian of the loss
function for task evaluated its optimum 07. Substituting Ôz Ô.-1 + or a, into the objective, we obtain:
(0-1+ a, 0:) (,,) (0-- + a,50; -0:)
(19)
Expanding the quadratic form inside the summation yields:
(0-1 - H,,00)00--111111 0,) + 2ax -07)) -00 ( H,
(20)
Define the objective function J(ae)) as:
[(ar) = Σ1 100,1-0000,,,,000------- 1=1
(21)
+ 20, 03) (0,))))00 + T
HH,
(22)
Taking the derivative of J(ar) with respect to ar, get:
dJ
Σ (0-1--77 (H) + aa(A)))) (0))))00
(23)
da.
dJ Setting dar
0 leads to the first-order optimality condition:
(b-1 87) (0)) a (A07) H( O.
(24)
Solving for a,, we obtain the optimal a,:
E 11=1 (0,-1 0,) (0)))
a
(25)
Σ (A0;) TH, (0;)
A.2 Details of Eq.
Further, we want reduce Ei= to make the combined model better. is evident that Σ 2
Σis 5,(a;) holds. We also have
Ei (0x-1 8:) H, )²
T
{(())) = Σ
(0-11
10-1-07 (0-1-00))
H; (0--1 (0-1--0;) -0;)
(26)
2 Σ 107 THA00
1=1
<
Σ
(0--1 - 07) H; (0--1 - 07)
(27)
2
16

---

The inequality holds because the second term is negative. We next explain why it is reasonable to ignore the second term and focus only on the first. The first term on the right-hand side dominates, is typically larger in magnitude and easier to compute, while the second term involves nested quadratic forms and is costly to evaluate. Therefore, we retain only the leading term as an approximation and obtain the upper bound:
(B 5a())) & (0.11 (0:11-0:) 0:) H, ( - 8:)
(28)
1=1
We next show why the first term is the dominant one. We aim show that for the decomposition:
(Ei (0.11- 0:) H,
Σ ((a))) - 20011110 H: 00)
2E:11 D0G H2
the second term is always less than equal to the first, i.e.,
Ei=1 a. H₂v
a H,a;-
(29)
Σ i==1 T H1C
Proof. Let Ei=1 a. H.a, and B -i= T HAv, and define:
C= Σ H2 (a, U)H.
(30)
1=1
where ( denotes generalized inner product defined via weighted Hessians.
Then by the Cauchy-Schwarz inequality in inner product spaces:
C
Σ
Hiv
Σ 0 1=1
Hia:
r THo Σ 1=1
Hav
= A B.
(31)
1=1
Therefore:
C
SA
= Second term < First term.
(32)
This proves that, in the decomposition where the objective is the first term minus the second, the second term is always less than or equal to the first. result, the first term dominates. Therefore, we focus on optimizing the first term through regularization, which is likely to reduce the overall objective as well.
B
More Experimental Results and Details
B.1 Details of Datasets
ImageNet-R contains 200 ImageNet [66] classes rendered various artistic styles. ImageNet-A comprises 200 classes featuring natural adversarial examples that are typically misclassified by standard ImageNet-trained models. DomainNet covers 345 object categories across six distinct visual domains. CIFAR 100 is classic image classification dataset with 60, 000 images equally distributed across 100 classes. CUB200 fine-grained bird classification dataset containing 11, 788 images over 200 categories.
17

---

Effect of Po (E 0.5)
Effect of E (Po 0.33)
57.4
Accuracy (%)
56.4
55.4
54.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Po
Figure 3: Hyperparameter ablation on ImageNet-A. Left: performance under varying Po values (probability zero perturbation). Right: performance under different c values (magnitude of perturbation).
B.2 Hyperparameter ablation
We conduct ablation studies the hyperparameters Po (probability of zero perturbation) and (perturbation magnitude) on the ImageNet-A dataset. As shown Figure we observe that setting Po 0.33 and 0.5 yields consistently good performance, though necessarily optimal. To reduce tuning overhead and maintain consistency, we use Po = 0.33 and = 0.5 as default values for all experiments. Interestingly, when po 0.33, po p+ p-
B.3 Memory and Training Time Analysis
We compare the memory footprint and training time of LoRA and our proposed LORA-P&M on ImageNet-R and CIFAR 100. As shown Tab. LORA-P&M introduces only a modest increase in memory aaag--approimmately 1.5 GB compared to standard LoRA- due to compute and store the diagonal empirical Fisher Information Matrix, while the cost introduced by stochastic perturbation is negligible. Similarly, the training time increases by 0.2 to 0.4 hours dataset. Despite this overhead.
achieves substantial accuracy gains: + +14.23% on ImageNet-R and +15.87% on CIFARI00 These results demonstrate that our method offers favorable trade-off between performance and resource efficiency.
Table 8: Comparison of memory usage, training time, and accuracy between model merging methods on ImageNet-R and CIFAR100.
ImageNet-R 10 tasks
CIFAR100 10 tasks
Method
Memory
Time 4 Acc
Memory 1
Time |
Acc 1
LoRA
22.80GB
1.6h
65.72
22.80GB
3.1h
72.58
LoRA-P&M
24.29GB
1.8h
79.95
24.29GB
3.5h
88.45
B.4 Comparison with Recent Continual Learning Methods
To further contextualize our approach, we compare against several recent and competitive continual learning methods, including O-LoRA (67), Prompt Gradient Projection (PGP) (68], and Consistent Prompting (C-Prompt) [69].
Our method consistently outperforms prior approaches across all datasets and task splits, demonstrat- ing strong adaptability and effective knowledge retention under the rehearsal free continual learning setting.
18

---

Table 9: Comparison with recent continual learning methods. Results reported Ace three
standard benchmarks.
Method
ImageNet-R 10 tasks
CIFAR100 tasks
CIFAR100 20 tasks
O-LoRA
79.15
85.69
83,22
DualP-PGP
69.34
86.92
83.74
C-Prompt
77.14
87.82
83,97
LoRA-P&M
81.47
88.45
85.45
B.5
Memory Efficiency and Online Fisher Approximation
One potential concern of our approach that storing one Fisher information matrix per task leads to linearly increasing memory usage with the number of tasks. We further introduce an online variant of our method by maintaining running average of the Fisher information across past tasks, resulting in a fixed-size Fisher representation. We denote this variant as LoRA-online P&M, and compare it with our standard LoRA-P&M framework.
Table 10: Comparison between LoRA-P&M and online variant on ImageNet-R with different task splits. Values denote Acc (%).
Method
Image Net- tasks
ImageNettRR 10 tasks
ImageNet-RR R 20 tasks
LoRA-P&M
81.47
79.95
76.37
LoRA-online P&M
79.97
78.12
71.20
As shown in Table 10 the online variant performs comparably to the original LoRA-P&M in the 5-task and 10-task settings. However, the task sequence becomes longer (e.g., 20 tasks), the performance gradually degrades due to accumulated approximation error. Mitigating this trade-off between memory efficiency and performance stability an important direction for our future work.
B.6 Comparison with EWC
To further validate our approach, we additionally include experimental comparisons with Elastic Weight Consolidation (EWC) (70). Since method also leverages Fisher-hased curvature approx- imations, EWC serves as a meaningful baseline. As shown in Table [I] our LoRA-P&M framework significantly outperforms LoRA-EWC across all benchmark settings.
Table 11: Comparison between LoRA-EWC and our proposed LoRA-P&M on ImageNet-R under different task splits. Results are reported as Acc (%).
Method
ImageNet-R 5 tasks
ImageNet-R tasks
ImageNet-R tasks
LoRA-EWC
71.47
65.42
55.67
LoRA-P&M
81.47
79.95
76.37
Although both EWC and our framework utilize the Fisher Information Matrix, the conceptual foundations and objectives of the two methods differ substantially. Our approach introduces model merging-base -aased inference paradigm, where closed-form merging coefficient of is derived minimize the total loss increase across all tasks. In contrast, EWC introduces Fisher-weighted regularization term during training to penalize deviations from parameters that were important in previous tasks.
While EWC uses the Fisher matrix to estimate the importance of each parameter during sequential training, our method treats the Fisher matrix as tractable approximation of the Hessian, which facilitates computing an optimal merging direction in the parameter space. Therefore, although we share the of Fisher-based eurvature information, our formulation, objective, and application context are fundamentally different.
19

---

Algorithm/ 2 Diagonal Empirical Fisher
Require: Dataset D {x4, y: } model fa, loss E(., -,, parameter-selection mask m E {0, 1]a
(e.g., LoRA modules), optional mini-batch size B.
Ensure: Diagonal empirical Fisher F TRd
1: Initialize +- 0₂
2: for each mini-batch D of size [B] B do
3: LB (E,
Σ
Ky. fx))
4:
Compute gradient f VoLs
Dg Rd
5:
+
> keep only selected parameters (e.g.. LoRA), zero otherwise }> element-wise square-and square-and-sum
6:
F F F+g+
7:
end for
8:
F+ N
F
F
normalize by the number of samples
9: return F
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The introduction and abstract are aligned with the paper's 's overall scope and contributions.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses potential limitations and assumptions that may affect generalization or performance.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification:
All theoretical claims are supported with assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper includes sufficient details for others to reproduce the key experi- mental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code are shared openly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttp:ccppppcccceeeeeeeeee..c.ett httpa//gguudee//eeeeabiisssssss /Cc////oddiiisssseeeeeee ssinsoliccyyyyycyccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: The paper provides necessary training and evaluation settings such as hyperpa- rameters and data splits.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars or variance measures are included for the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies the compute resources used, including hardware type and training time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps://eeurps.... wwtp://c/wwc.ccccseiiiiiiiee
Answer: [Yes]
Justification: The research adheres to ethical standards outlined in the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is just a basic study on continual learning, and does directly address societal impacts to the best of our knowledge.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work does not involve high-risk models or datasets that require usage safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All external assets used are properly cited and comply with relevant licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new datasets models are released in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The study does not involve experiments with human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No research involving human subjects was conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used in the development of methods or results in this work.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //appp:///eurps..... /0tpss//coooeeeeeeee/22///////// for what should or should be described.
27


Reviews:
[{"review_id": "6u60wuzdRE", "reviewer": "Reviewer_Fr4E", "summary": "This paper introduces the concept of model merging into continual learning. Unlike traditional continual learning approaches, the proposed P&M, after acquiring parameters for the new task, employs an parameter interpolation method to balance the model performance across both old and new tasks. The interpolation weights are theoretically derived. Furthermore, the authors incorporate regularization term during training to enhance the effectiveness of model merging.", "questions": "(1) Please provide more/additional justification for the necessity of the two-stage approach. For example, in what scenarios does this method demonstrate superiority in terms of performance, efficiency, or practicality? Including concrete examples would also help readers build an intuitive understanding of its benefits.\n\n(2) Is it possible to add **α** during the derivation of training with perturbation?\n\n(3) Can more parameter statistics be added to demonstrate the validity of the hypothesis proposed in the paper?", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nS1: This paper proposes a novel continual learning paradigm. The authors have conducted detailed theoretical derivations tailored to this new paradigm, along with reasonable training perturbation and model merging methods. The work shows high originality\n\nS2: The derivations are rigorous and appear with no obvious major weaknesses. The conclusion seems sound and well-supported.\n\nS3:  The writing is clear and well-structured. Experiments are conducted on multiple datasets, enhancing the reliability and credibility of the work.\n\n*********\n*********\n\nWeaknesses\n\nW1:​​ This paper proposes a novel continual learning paradigm with detailed theoretical derivations. However, I have doubts whether this pipeline approach, which first trains the model based on the new task and then solves the closed form solution to find the optimal **α**, can outperform traditional one-stage incremental training method. Intuitively, more steps lead to error propagation, which may not guarantee improved performance.\n\nW2: Another concern is that the proposed linear interpolation method is overly rigid. Specifically, assigning a same “α” across all network parameters ignores the fact that different parameters contribute unequally to task performance.\n\n​​W3:​​ The derivation in the second part (train with perturbation) seems to be a relatively simple deduction, which lacks explicit linkage to the optimal **α** in the first step. The final conclusion is also somewhat simple, merely introducing some perturbations without clearly establishing a connection to the eventual interpolation method.\n\n​​W4:​​ This paper provides extensive derivations and makes some assumptions. Are these assumptions supported by empirical evidence, such as statistics of parameters from training datasets?", "was_revised": true, "final_justification": "The main reason why I modified the score is that the authors explained the rationale for incorporating model merging into continuous learning. In addition, the authors conducted experiments to demonstrate that the performance of parameter-independent “α” (i.e. global “α”) is as competitive as learning separate “α” values for each LoRA module.", "rebuttal": "We sincerely thank the reviewer for their valuable time and insightful feedback. Below we address the raised concerns point-by-point.\n\n---\n\n> W1 & Q1: Whether our two-stage pipeline offers clear advantages over traditional one-stage training and  further justification of its necessity and effectiveness.\n\n\nWe would like to clarify that our two-stage framework aims at introducing the *model merging* inference paradigm into continual learning (CL) to better preserve prior task knowledge.\n\nOur motivation stems from a key observation: in traditional CL methods, the model trained on the current task is directly used to perform inference on both new and old tasks. This often leads to severe forgetting of previously acquired knowledge. In contrast, our method introduces a **theoretically grounded post-training merging step**. Specifically, we compute a convex combination of the previous inference model $\\theta_{t-1}$ and the current task-specific optimum, where the merging coefficient $\\alpha_t^*$ is derived in closed form (see Eq. (8)) to explicitly minimize the **total loss increase across all seen tasks**. Importantly, traditional CL approaches are equivalent to setting $\\alpha = 1$ in our formulation—thus our method **generalizes and improves upon one-stage training** under the merging paradigm.\n\nRegarding the reviewer’s concern on potential *error propagation* due to a multi-step process, we emphasize that our framework does **not** introduce additional backward passes. Structurally, our \"Train + Merge Inference\" pipeline is equivalent to standard \"Train + Inference\" approaches. The only difference is that we apply a **theory-informed model merging step before inference** (i.e., LoRA-M), which empirically **reduces interference from new tasks** and thereby *mitigates* error propagation. As shown in **Figure 1** and **Table 4**, this leads to significant reduction in forgetting while preserving new task plasticity.\n\n---\n\n> W2: The use of a uniform merging coefficient α across all parameters may be too restrictive.\n\nUsing a more fine-grained merging strategy for model parameters may potentially lead to a higher performance upper bound. However, such strategies would introduce additional computational overhead and theoretical complexity. Our experimental results also demonstrate that a unified merging coefficient already achieves strong and competitive performance.\n\nWe ran experiments comparing the global α with a more granular strategy—learning separate α values for each LoRA module. The results are as follows:\n\n| ImageNet-R         | 5 tasks | 10 tasks | 20 tasks |\n|--------------------|---------|----------|----------|\n| Per-module α       | 80.53   | 76.82    | 74.68    |\n| Global α (ours)    | 80.88   | 78.48    | 74.13    |\n\nThe performance differences are negligible. Moreover, we observed that the learned per-module α values were very similar to each other, suggesting that a global α is sufficient to capture the merging dynamics. Therefore, we adopt a single-α approach for its simplicity, efficiency, and theoretical tractability.\n\nBased on prior research[1][2], a global α preserves the structural integrity of the task vector $\\Delta \\theta_t$ , which represents a coherent direction in parameter space. A parameter-wise α may introduce inconsistent scaling, distorting the internal correlations of $\\Delta \\theta^*_t$ and complicating the derivation of a closed-form solution.\n\n\n---\n\n> W3 & Q2: The theoretical connection between the two stages of our framework.\n\nThere is a direct theoretical link: after deriving the optimal $\\alpha_t$, we observed that even optimal merging leads to a total performance drop across tasks, quantified as $\\sum_{i=1}^{t} \\delta_i(\\alpha_t)$ in Eq. 11. The **Train with Perturbation** step is specifically designed to reduce this upper bound during training.\n\nThe perturbation is not simply random noise; rather, it is a directional regularization based on $\\Delta \\theta_t$, approximating the Hessian-based quadratic penalty. This encourages convergence to flatter minima that are more robust to parameter interpolation, thus improving the final merged model’s generalization. This effect is clearly visualized in Figure 2, where the merged models from P&M exhibit significantly flatter and wider low-loss regions compared to the baseline.\n\n---\n\n> W4 & Q3: Whether our theoretical assumptions are empirically grounded and suggests including parameter statistics to support their validity.\n\n\nWe would like to clarify that the assumptions we adopt—such as approximating the Hessian with the empirical Fisher Information Matrix, applying a second-order Taylor expansion of the loss, and neglecting higher-order terms—are not unique to our work. These are widely used and well-accepted approximations in the continual learning [3][4][5].\n\nSuch approximations have been extensively validated and empirically supported in recent deep learning theory and algorithms. Our use of them follows established practices rather than introducing new or unverified assumptions. \n\n\n---\n\nWe hope that our clarifications and empirical justifications effectively address the reviewer’s concerns.\n\n\n---\n\n[1] Gabriel Ilharco, et al. \"Editing Models with Task Arithmetic.\" ICLR23. \n\n[2] Hongkang Li, et al. \"When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers.\" ICLR25.\n\n[3] James Kirkpatrick, et al. \"Overcoming Catastrophic Forgetting in Neural Networks.\" 2017.\n\n[4] Zhenyi Wang, et al. \"A UNIFIED AND GENERAL FRAMEWORK FOR CONTINUAL LEARNING.\" ICLR24.\n\n[5] Mei Li, et al. \"BECAME: BayEsian Continual Learning with Adaptive Model MErging.\" ICML25."}, {"review_id": "BFGYSUOeA1", "reviewer": "Reviewer_AEtr", "summary": "This paper presents LoRA-P&M, a continual learning (CL) method that applies model merging to LoRA components across tasks. To enable effective merging and improve generalization, two techniques are introduced: (1) a theoretically grounded merging coefficient, $\\alpha$, computed using Fisher matrices; and (2) perturbations along task vectors introduced during training. Experimental results show that LoRA-P&M outperforms existing methods on multiple CL benchmarks. Loss landscape visualizations further suggest that it promotes flatter minima for the merged models.", "questions": "Besides the weakness shown in the above section, please also see the following questions: \n\nQ1: Since perturbation training aims to optimize a more complex loss function (i.e., Equation 14), does it lead to slower convergence? \n\nQ2: Given that perturbation training leads to flatter minima, as shown in Figure 2, could it also benefit other LoRA-based continual learning methods, such as SD-LoRA? \n\nQ3: LoRA is introduced in Section 3.4 mainly to address memory consumption. If memory were not a limiting factor, would the proposed method still be effective—or perhaps perform even better?", "limitations": "The paper acknowledges a limitation that the Fisher matrix may not fully capture the true curvature of the loss landscape. Another potential limitation is that storing a Fisher matrix for each task results in memory usage that grows linearly with the number of tasks, which may hinder scalability to large task sequences.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength: \n- The core idea adopts a simple yet effective model merging strategy for continual learning, supported by additional techniques that enhance its performance. \n\n- Tables 2, 3, 5, and 6 show promising results compared to LoRA-based and prompt-based continual learning methods across diverse datasets.\n\n- The analysis in Figure 2 is insightful and highlights the benefits of perturbation training. \n\nWeakness: \n- LoRA-P&M requires storing a Fisher matrix $F_i$ for each task, leading to memory growth that is linear in the number of tasks. \n\n- Although the paper claims to achieve state-of-the-art performance (Lines 20-21 in the abstract), several relevant continual learning methods are missing from the comparison. For example, recent LoRA-based approaches such as I-LoRA [1] and O-LoRA [2] are not included. Moreover, several newer prompt-based methods are also omitted [3][4][5][6].\n\n- Equation 17 in Section 3.4 is somewhat unclear. Since LoRA parameters are typically inserted at different locations within the network, it is not immediately clear why they are summed in this equation. \n\n- The idea of weighted model parameter combination (i.e., model merging) in continual learning has also been explored in CoFiMA [7]. However, when comparing the ablated version of LoRA-P&M (i.e., LoRA-M in Table 4) with CoFiMA (Table 3), LoRA-M appears to perform worse. This raises the question of whether the proposed weighted model merging strategy—specifically, the use of the optimal merging coefficient $\\alpha$—is less effective than CoFiMA's approach.\n\n\n[1] 2025 ICASSP Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning. \n\n[2] 2023 EMNLP Orthogonal Subspace Learning for Language Model Continual Learning. \n\n[3] 2024 ECCV RCS-prompt: Learning Prompt to Rearrange Class Space for Prompt-based Continual Learning. \n\n[4] 2024 ICLR Prompt Gradient Projection for Continual Learning. \n\n[5] 2024 ECCV PromptFusion: Decoupling Stability and Plasticity for Continual Learning. \n\n[6] 2024 CVPR Consistent Prompting for Rehearsal-Free Continual Learning. \n\n[7] 2024 ECCV Weighted Ensemble Models Are Strong Continual Learners", "was_revised": true, "final_justification": "The rebuttal and follow-up discussion with the authors have addressed the main concerns I raised in the initial review:\n\n- Additional costs: The authors clarified the storage overhead associated with the Fisher matrix and presented experimental results demonstrating the method’s convergence speed, which alleviated my concerns regarding storage cost and training efficiency. \n\n- Comparison with recent methods: The authors provided additional comparisons with recent approaches, including O-LoRA, PGP, and C-Prompt, and showed that the proposed method achieves better performance. \n\n- Effectiveness relative to CoFiMA: The authors clarified the experimental results and included further comparisons to better demonstrate the effectiveness of the proposed method compared to CoFiMA. \n\nGiven that these key concerns have been addressed, I have updated my assessment. I intend to raise my score from 3 to 4 accordingly.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and constructive feedback. Below we address each point in detail and clarify the concerns.\n\n---\n\n> W1: LoRA-P&M requires storing a Fisher matrix for each task, leading to memory growth that is linear in the number of tasks.\n\nIn our setup, the actual overhead is modest for the following reasons:\n\nWe only compute and store the diagonal Fisher matrix for modules equipped with LoRA, which, in our implementation, are limited to the key and value projections of each attention module. These parameters account for approximately one-sixth of the total model parameters. Moreover, in practice, since the diagonal Fisher matrix is only used at the end of training for each task, it can be stored on the CPU or disk, preventing linear growth in GPU memory usage.\n\nAs shown in Table 7, the overall memory increase from LoRA(22.80GB) to LoRA-P&M(24.29GB) is approximately 1.5 GB for a dataset with 10 tasks.\n\n---\n\n> W2: Comparison with more continual learning methods\n\nWe appreciate the reviewer pointing out recent methods. Due to time constraints, we were able to compare against **O-LoRA**, **Prompt Gradient Projection(PGP)**, and **Consistent Prompting(C-Prompt)**. These represent strong and recent methods across both LoRA-based and prompt-based paradigms.\n\nRegarding I-LoRA and PromptFusion: these methods rely on data replay, which fundamentally differs from our task setting (rehearsal-free continual learning). As such, we respectfully argue that they are not directly comparable to our approach.\n\n| Method       | ImageNet-R 10tasks | CIFAR100 10tasks | CIFAR100 20tasks | \n|--------------|----------|--------|-------------|\n| OLoRA      | 79.15  | 85.69 | 83.22     | \n| DualP-PGP     | 69.34    | 86.92  |83.74       | \n| CPrompt      | 77.14   | 87.82  | 83.97       | \n| LoRA-P&M     | 81.47   | 88.45  | 85.45      | \n\nWe will add the comparisons and discussion with these methods to the revised version of our paper.\n\n---\n\n\n> W3: Clarification of Equation 17\n\nThank you for the helpful comment. We agree that the current form of Equation 17 may lead to confusion. Specifically, it may be misinterpreted as a global parameter-wise summation, whereas in practice, each LoRA update is applied **locally** and independently at specific network locations (e.g., attention key or value projections).\n\nTo clarify this, we will revise Equation 17 as:\n\n$\\theta_t = \\theta_{t-1} + \\bigoplus_{p \\in l} A_t^{(p)} B_t^{(p)}$\n\nwhere $\\bigoplus$ denotes *location-wise module insertion*, not element-wise addition. Each low-rank update $A_t^{(p)} B_t^{(p)}$ is injected into the pre-trained weight at its corresponding module $p$, and does not affect other components of the model.\n\n---\n\n> W4: Detailed Comparison with CoFIMA\n\n\nWe thank the reviewer for raising this important comparison. In table  below, we report the full LoRA-M versus CoFiMA across multiple benchmarks, showing that the performances are very close:\n\n| Method       | average FAA on 5 datasets|\n|--------------|----------|\n| LoRA-CoFiMA       | 75.52    | \n| LoRA-M     |  75.27   | \n\nTo further analyze, we conducted a **CoFiMA + Perturbation** experiment (i.e., applying our task-vector perturbation during training and using CoFiMA merging). The results show improved performance over plain CoFiMA, but still underperform our full LoRA-P&M method. This indicates that:\n- Our second-stage theoretical derivation is based on the model merging perspective, suggesting that perturbation is generally beneficial for merging-based strategies. As shown in Figure 2, such perturbation indeed improves the flatness of the loss landscape around the merged model;\n- Our closed-form optimal merging coefficient and the perturbation are mutually reinforcing, yielding the best result.\n  \n| Method       | ImageNet-R 5tasks | ImageNet-R 10tasks  | ImageNet-R 20tasks  | \n|--------------|----------|--------|-------------|\n|LoRA-CoFiMA       | 80.26   | 79.06 | 75.09    | \n| LoRA-CoFiMA-P     | **81.03**    | **79.70**  |**76.17**       | \n| LoRA-M       | 80.86     | 78.35  | 74.26        |\n| LoRA-P&M     | **81.47**   |**79.95** | **76.37**     | \n\n---\n\n> Q1: Whether the perturbation may slow down the convergence speed\n\nWe measured convergence speed in terms of loss decay per training step. As shown in the  table below as a example, perturbation training (LoRA-P&M) converges at a comparable rate to standard LoRA:\n\n| Train Loss of Task 8(ImageNet-R 10tasks)     | 1 % Steps  | 2 % Steps| 10 % Steps  | \n|--------------|-----------------------------|---------------|------\n| LoRA-M        | 1.90                       | 0.70         |0.26\n| LoRA-P&M     | 1.92                        |0.71         | 0.25\n\nThe added regularization term leads to slightly more stable but not slower convergence, confirming the efficiency of the stochastic perturbation.\n\n---\n\n> Q2: Whether the proposed perturbation strategy remains effective when integrated with other continual learning methods, such as SD-LoRA.\n\n\nWe explored this question by applying our perturbation scheme to **SD-LoRA**, a state-of-the-art LoRA-based continual learning method. Interestingly, we observed **performance degradation** when adding perturbation to SD-LoRA:\n\n| Method       | ImageNet-R 5tasks | ImageNet-R 10tasks  | ImageNet-R 20tasks  | \n|--------------|----------|--------|-------------|\n| SD-LoRA        | 79.15   | 77.34 | 75.26    | \n| SD-LoRA + Perturbation    | 76.9    | 75.6  |72.22       | \n\n\nThis suggests that our perturbation training is particularly effective for **merging-based strategies** like P&M or CoFiMA, where parameter interpolation plays a central role in preserving task knowledge.\n\nIn contrast, SD-LoRA involves **dynamically optimizing task-specific vectors** during training. Our perturbations, which are aligned with a fixed task vector direction, may interfere with SD-LoRA’s learning dynamics and thus reduce performance.\n\nThis further highlights that the benefit of perturbation is **targeted toward model merging**, and not necessarily applicable to all continual learning strategies.\n\n---\n\n> Q3: Whether the proposed method would be more effective when applied on top of non-PEFT methods.\n\n\nWhile Section 3.4 introduces LoRA primarily as a means to reduce memory overhead, we emphasize that LoRA is also better aligned with the goals of continual learning (CL) beyond efficiency. LoRA tends to cause less interference with previously learned tasks[1], allows for easier task-specific adaptation, and requires fewer training examples and parameters to achieve strong performance. In contrast, full fine-tuning often leads to greater forgetting and lacks modularity. Therefore, even when memory is not a limiting factor, LoRA remains a more suitable and effective paradigm for continual learning.\n\n\n\n---\n\nWe again thank the reviewer for their helpful feedback. We will incorporate the clarifications, additional comparisons, and ablations in the revised version.\n\n\n[1] Dan Biderman, et al. \"LoRA Learns Less and Forgets Less.\" TMLR24."}, {"review_id": "gF3ld5rky1", "reviewer": "Reviewer_47Vz", "summary": "The paper proposes a model merging idea for continual learning. The idea to continually train from a base model for a sequence of tasks and take a snapshot of the weights after each task is trained. And at inference, the weight combination of the weight snapshots is used. A simple idea that seems effective according to the experiments.", "questions": "see above", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The idea of merging task snapshots makes sense, and the EWC-inspired regularisation is intuitive.\n2. The merging coefficient is obtained using a Fisher-based curvature approximation -- previously used in continual learning.\n3. Results show consistent improvements with the proposed method, compared to other fine-tuning or prompt-based methods.\n\n## Weaknesses\n1. Previous continual learning methods are not compared. These methods were developed before the introduction of foundational models; however, they can be easily incorporated into the pretraining+finetuning framework. Many notable continual methods include EWC, GEM [a] and their variants. Given the similarity to EWC, these methods need to be compared.\n2. Related to the above point, the perturbation-based method and the merging coefficient derivation have similarities with EWC and should be credited, in my opinion.\n3. The weight coefficient is mentioned as \"optimal\" in the abstract; however, it is not optimal, and it is derived for a Fisher-based second-order approximation of the loss.\n4. How does the order of tasks affect the performance? If the weights are sequentially trained, it seems that the task order would matter, and it can have an impact on the final performance. \n- Why not train for all the tasks from the base model and do the merging? How would this compare to the proposed method?\n5. The memory grows linearly with the no of tasks, anyway to reduce this?\n6. The experiment setup is not clear. Is the task ID known at inference (multi-head or single-head evaluation) [b]?\n\n## References\n- [a] Lopez-Paz, David, and Marc'Aurelio Ranzato. \"Gradient episodic memory for continual learning.\" Advances in neural information processing systems 30 (2017).\n- [b] Chaudhry, Arslan, et al. \"Riemannian walk for incremental learning: Understanding forgetting and intransigence.\" Proceedings of the European conference on computer vision (ECCV). 2018.", "was_revised": true, "final_justification": "Authors' response has addressed all of my concerns, and I'm increasing the score to 4.", "rebuttal": "We sincerely thank the reviewer for their insightful and detailed feedback. Below we address each point individually.\n\n---\n\n> W1: Comparison with methods include EWC and GEM.\n\n\nWe have now included experimental comparisons with **EWC** in our updated evaluations. Since our method also leverages Fisher-based curvature approximation, we believe this is a meaningful baseline. The results show that LoRA-P&M significantly outperforms EWC across all benchmarks.\n\n| Method       | ImageNet-R 5tasks | ImageNet-R 10tasks  | ImageNet-R 20tasks  | \n|--------------|----------|--------|-------------|\n|LoRA-EWC      | 71.47  | 65.42 | 55.67  | \n| LoRA-P&M     | **81.47**   |**79.95** | **76.37**     | \n\nAs for **GEM**, we respectfully did not include it because it is based on **experience replay**, which violates the **rehearsal-free continual learning assumption** adopted in our work. Our method, like SD-LoRA and InfLoRA, operates under a constraint that previous data cannot be revisited. Thus, GEM is not directly comparable in our setting.\n\n---\n\n> W2: A detailed comparison with EWC.\n\n\nWe would like to clarify that our two-stage framework is conceptually distinct from EWC. Our approach introduces a *model merging*-based inference paradigm, where a closed-form merging coefficient $\\alpha_t^*$ is derived to minimize total loss increase across all tasks.  \nIn contrast, EWC adds a Fisher-weighted regularization term during training to penalize deviations from previously important parameters. In EWC, the Fisher matrix serves to estimate the *importance of each parameter* for past tasks. In our case, the Fisher matrix is used solely as a tractable approximation of the Hessian in order to compute an optimal merging direction. Thus, while we share a technical approximation with EWC, our theoretical formulation, implementation, and application context are entirely different. \n\nWe appreciate the reviewer’s observation regarding the connection to EWC. While both our method and EWC utilize the Fisher Information Matrix, the motivation and role of this approximation are fundamentally different. \nWe will clarify this distinction in the revised version and cite EWC appropriately. \n\n---\n\n> W3: The expression of \"optimal\" in the paper is not sufficiently rigorous.\n\n\nThank you for highlighting this. Our merging coefficient $\\alpha_t^*$ is **theoretically optimal under the Fisher-based second-order loss approximation**, but we agree that this does not imply global optimality with respect to the original loss surface.\n\nTo prevent misunderstanding, we will revise our language throughout the paper (including the abstract) to refer to α as **“analytically derived under a curvature approximation”**, and clarify its theoretical basis and limitations.\n\n---\n\n> W4: To what extent does the task order influence the performance of our method? How would the results change if all sub-models were initialized from the same base model rather than sequentially?\n\n\n1. Indeed, task order can influence performance in continual learning. In all of our experiments, **we train three runs under different random seeds**, each corresponding to **a different task order**. We report mean and standard deviation to capture this variability. We will clarify this in the experimental setup.\n\n2. We agree this is an important baseline. In fact, Table 3 in our paper already includes such methods:\n- **Model Averaging**: directly merges task-specific models trained from the same base model.\n- **DARE**: a recent data-free merging method.\n\n---\n\n> W5: The memory grows linearly with the no of tasks, anyway to reduce this?\n\nAs you noted, storing one Fisher matrix per task results in linear memory growth. However:\n- We only store **diagonal Fisher approximations** for low-rank **LoRA modules**, making the per-task cost minimal (Table 7 shows ~1.5GB/24.29GB across 10 tasks). Moreover, in practice, since the diagonal Fisher matrix is only used at the end of training for each task, it can be stored on the CPU or disk, preventing linear growth in GPU memory usage.\n- Inspired by **Online EWC** [1], we plan to replace per-task Fisher storage with a simple **running average** over past tasks, resulting in a **fixed-size** Fisher representation. We denote this variant as *Online P&M*, and provide experimental comparisons against our original P&M framework.\n\n\n\n| Method       | ImageNet-R 5tasks | ImageNet-R 10tasks  | ImageNet-R 20tasks  | \n|--------------|----------|--------|-------------|\n|LoRA-P&M     | **81.47**   |**79.95** | **76.37**     | \n|LoRA-online P&M     | 79.97  | 78.12 | 71.2  | \n\nIn the 5-task and 10-task settings, the approach of aggregating Fisher information from all previous tasks performs comparably to our LoRA-P&M method. However, when the task sequence becomes longer (e.g., 20 tasks), its performance degrades significantly. Addressing this limitation is one of our directions for future research.\n\n\n---\n\n> W6: The experiment setup is not clear. Is the task ID known at inference (multi-head or single-head evaluation).\n\nWe follow the **class-incremental learning (CIL)** setting, where the task ID is **not provided at inference time**, and every tasks has a classifier head. Each task expands the output space (e.g., from 20 classes to 200 classes across 10 tasks in CIFAR100), and the model must distinguish among **all seen classes**.\n\nThis is considered the **most challenging and realistic** continual learning setting, and aligns with the evaluation protocols used in SD-LoRA, CODA-Prompt, and other recent works. We will clarify this explicitly in the revised experimental setup.\n\n---\n\nWe again thank the reviewer for their constructive suggestions. We will revise the manuscript to reflect the above clarifications, add missing citations, and expand our baseline coverage accordingly.\n\n[1] Ferenc Huszar, et al. \"On Quadratic Penalties in Elastic Weight Consolidation.\" 2017."}, {"review_id": "fYfbUmybHk", "reviewer": "Reviewer_VKzv", "summary": "The authors integrate continual learning with model merging to develop a single model capable of performing well across multiple tasks while mitigating catastrophic forgetting. They propose a method called *Infer after Merging*, which involves scaling the task vector ∆θ\\*\\_t = θ\\**t − θ̂*{t−1} to reduce interference with previously acquired knowledge, and derive a closed-form solution to minimize overall performance degradation. Additionally, they introduce *Train with Perturbation*, which incorporates a regularization loss term based on the task vector and the Hessian matrix during training. Finally, they combine the P\\&M approach with LoRA to reduce memory overhead.", "questions": "1. In line 149, under the maximum likelihood estimation setting, shouldn't the expected Fisher information matrix be equal to the negative of the Hessian matrix?\n2. In Figure 2, the average loss landscape is visualized. It is unclear how this figure was constructed—what exactly does the \"average\" on the loss axis represent?\n3. In line 139, should θ*_t be written as θ*_i to maintain consistency with the notation?", "limitations": "Please provide the conditions under which the approximations or include error bounds to justify their use in the derivation.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper provides a clear and well-structured introduction, effectively outlining the differences and connections between continual learning and model merging, along with a clear motivation for the proposed approach.\n2. Experimental results consistently outperform the baseline methods, demonstrating the effectiveness of the proposed techniques.\n3. The use of loss landscape visualizations offers intuitive insights into model behavior and optimization dynamics.\n\n**Weaknesses**\n\n1. The methodological derivation relies on several approximations—such as Taylor expansion, Fisher matrix estimation, Hessian matrix approximation, and symmetric finite differences—without sufficient justification for their validity. The potential cumulative error raises concerns. Can the actual benefit of minimizing the upper bound be empirically verified?\n2. The related work section is brief. Some early studies on continual learning and model merging are neither discussed nor compared experimentally, making it difficult to assess the true advantages of the proposed method.\n3. Although the method reduces memory overhead using LoRA, the computation and storage of the Fisher matrix still require significant resources. Is the approach inherently limited to LoRA-based fine-tuning?", "was_revised": true, "final_justification": "The main issue of this paper was the writing quality, which the authors have improved based on my suggestions.", "rebuttal": "We sincerely thank the reviewer for their thoughtful comments and careful reading. Below we respond to each concern in detail.\n\n---\n\n> W1:  Whether the theoretical approximations—such as Taylor expansion, Fisher approximation, and symmetric finite differences—are valid, and whether minimizing the upper bound brings empirical benefits.\n\n\n1. We would like to clarify that the assumptions we adopt—such as approximating the Hessian with the empirical Fisher Information Matrix, applying a second-order Taylor expansion of the loss, and neglecting higher-order terms—are not unique to our work. These are widely used and well-accepted approximations in the continual learning [1][2][3].\n\nSuch approximations have been extensively validated and empirically supported in recent deep learning theory and algorithms. Our use of them follows established practices rather than introducing new or unverified assumptions. \n\n2. To empirically evaluate the benefit of minimizing the upper bound (Eq. 11), we conducted ablation experiments:\n\n- LoRA-M (with merging but no perturbation) vs LoRA-P&M (with perturbation): The latter consistently improves performance, as shown in Table 4 and Figure 1.\n- Additionally, in Figure 2, we visualize the loss surface across merged models. The optimal α derived from the upper-bound formulation consistently lands in low-loss regions.\n- We also show in table below that adding our perturbation to external merging methods (e.g., CoFiMA(CoFiMA-P)) improves performance—indicating that minimizing the derived upper bound brings empirical benefits beyond our method.\n  \n| Method       | ImageNet-R 5tasks | ImageNet-R 10tasks  | ImageNet-R 20tasks  | \n|--------------|----------|--------|-------------|\n|LoRA-CoFiMA       | 80.26   | 79.06 | 75.09    | \n| LoRA-CoFiMA-P     | **81.03**    | **79.70**  |**76.17**       | \n\nWhile the approximations are imperfect, the **combined analytical formulation and empirical outcomes** strongly suggest that they are effective and valid in practice. \n\n---\n\n> W2: The related work section is brief. Some early studies on continual learning and model merging are neither discussed nor compared experimentally.\n\n\nThank you for pointing this out. We agree that the related work section can be expanded. In our revised version, we will add discussion of classical continual learning methods such as EWC, GEM, and  cite early model merging works then discuss distinctions from our coefficient-guided merging.\n\nFurthermore, we have included experimental comparisons with **EWC** in our updated results. The results show that LoRA-P&M substantially outperforms EWC.\n\n| Method       | ImageNet-R 5tasks | ImageNet-R 10tasks  | ImageNet-R 20tasks  | \n|--------------|----------|--------|-------------|\n|LoRA-EWC      | 71.46  | 65.42 | 55.67  | \n| LoRA-P&M     | **81.47**   |**79.95** | **76.37**     | \n\nWe also conducted additional comparisons with several recent continual learning methods:\n| Method       | ImageNet-R 10tasks | CIFAR100 10tasks | CIFAR100 20tasks | \n|--------------|----------|--------|-------------|\n| OLoRA[4]      | 79.15  | 85.69 | 83.22     | \n| DualP-PGP[5]     | 69.34    | 86.92  |83.74       | \n| CPrompt[6]      | 77.14   | 87.82  | 83.97       | \n| LoRA-P&M     | 81.47   | 88.45  | 85.45      | \n\n\n---\n\n\n> W3: The computation and storage of the Fisher matrix require significant resources. Is the approach inherently limited to LoRA-based fine-tuning?\n\nWe appreciate the reviewer’s concern and would like to clarify that while our method is currently implemented with LoRA-based fine-tuning, it is not inherently restricted to LoRA. The core of our approach—the task vector merging strategy and curvature-guided perturbation—can be applied to more PEFT methods, including but not limited to adapters, prompt tuning. \n\nIn our setup, the actual overhead is modest for the following reasons:\n\nWe only compute and store the diagonal Fisher matrix for modules equipped with LoRA, which, in our implementation, are limited to the key and value projections of each attention module. These parameters account for approximately one-sixth of the total model parameters. Moreover, in practice, since the diagonal Fisher matrix is only used at the end of training for each task, it can be stored on the CPU or disk, preventing linear growth in GPU memory usage.\n\nAs shown in Table 7, the overall memory increase from LoRA(22.80GB) to LoRA-P&M(24.29GB) is approximately 1.5 GB for a dataset with 10 tasks.\n\nWhile Section 3.4 introduces LoRA primarily as a means to reduce memory overhead, we emphasize that LoRA is also better aligned with the goals of continual learning (CL) beyond efficiency. LoRA tends to cause less interference with previously learned tasks[1], allows for easier task-specific adaptation, and requires fewer training examples and parameters to achieve strong performance. In contrast, full fine-tuning often leads to greater forgetting and lacks modularity. Therefore, even when memory is not a limiting factor, LoRA remains a more suitable and effective paradigm for continual learning.\n\n\n\n\n---\n\n> Q1&Q3: Typos\n\nThank you for this sharp observation.  We will correct this in the revision.\n\n---\n\n> Q2: In Figure 2, the average loss landscape is visualized. It is unclear how this figure was constructed—what exactly does the \"average\" on the loss axis represent?\n\nThank you for pointing this out. In Figure 2:\n\n- The “average loss” refers to the **mean task-wise cross-entropy loss** computed on validation data from all tasks seen up to that point.\n- Each point on the 2D plane corresponds to a convex combination of $\\hat{\\theta}_{t-1}$ and $\\theta^*_t$, i.e., $\\theta = \\beta \\hat{\\theta}_{t-1} + \\alpha \\theta^*_t$.\n- We evaluate the merged model at that point and compute its average loss across all past tasks.\n\nWe will clarify this in the figure caption and the main text.\n\n---\n\nWe appreciate the reviewer’s close reading and constructive feedback. The points raised will be addressed thoroughly in the revised version, and we thank you again for helping us improve the clarity and rigor of the work.\n\n\n[1] James Kirkpatrick, et al. \"Overcoming Catastrophic Forgetting in Neural Networks.\" 2017.\n\n[2] Zhenyi Wang, et al. \"A UNIFIED AND GENERAL FRAMEWORK FOR CONTINUAL LEARNING.\" ICLR24.\n\n[3] Mei Li, et al. \"BECAME: BayEsian Continual Learning with Adaptive Model MErging.\" ICML25.\n\n[4] Xiao Wang, et al. \"Orthogonal Subspace Learning for Language Model Continual Learning.\"  EMNLP23.\n\n[5] Jingyang Qiao, et al. \"Prompt Gradient Projection for Continual Learning.\"  ICLR24.\n\n[6] Zhanxin Gao, et al. \"Consistent Prompting for Rehearsal-Free Continual Learning.\" CVPR24\n\n[7] Dan Biderman, et al. \"LoRA Learns Less and Forgets Less.\" TMLR24."}]

Author rebuttal:

Meta-review: The paper received all positive reviews, leading to a final acceptance recommendation.
Decision: Accept (poster)