Title: Train with Perturbation, Infer after Merging: A Two-Stage Framework for Continual Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Continual Learning

Model Merging
Abstract: Continual Learning (CL) aims to enable models to continuously acquire new knowledge from a sequence of tasks with avoiding the forgetting of learned information. However, existing CL methods only rely on the parameters of the most recent task for inference, which makes them susceptible to catastrophic forgetting. Inspired by the recent success of model merging techniques, we propose Perturb-and-Merge (P\&M), a novel continual learning framework that integrates model merging into the CL paradigm to mitigate forgetting. 
Specifically, after training on each task, P\&M constructs a new model by forming a convex combination of the previous model and the newly trained task-specific model. Through theoretical analysis, We minimize the total loss increase across all tasks 
and derive a closed-form solution for the merging coefficient under mild assumptions. To further improve the performance of the merged model, we observe that the degradation introduced during merging can be alleviated by a regularization term composed of the task vector and the Hessian matrix of the loss function. Interestingly, we show that this term can be efficiently approximated using second-order symmetric finite differences, and a stochastic perturbation strategy along the task vector direction is accordingly devised which incurs no additional forward or backward passes while providing an effective approximation of the regularization term. 
Finally, we combine P\&M with LoRA, a parameter-efficient fine-tuning method, to reduce memory overhead. Our proposed approach achieves state-of-the-art performance on several continual learning benchmark datasets. The code is available at \url{https://github.com/qhmiao/P-M-for-Continual-Learning}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors integrate continual learning with model merging to develop a single model capable of performing well across multiple tasks while mitigating catastrophic forgetting. They propose a method called *Infer after Merging*, which involves scaling the task vector ∆θ\\*\\_t = θ\\**t − θ̂*{t−1} to reduce interference with previously acquired knowledge, and derive a closed-form solution to minimize overall performance degradation. Additionally, they introduce *Train with Perturbation*, which incorporates a regularization loss term based on the task vector and the Hessian matrix during training. Finally, they combine the P\\&M approach with LoRA to reduce memory overhead.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper provides a clear and well-structured introduction, effectively outlining the differences and connections between continual learning and model merging, along with a clear motivation for the proposed approach.\n2. Experimental results consistently outperform the baseline methods, demonstrating the effectiveness of the proposed techniques.\n3. The use of loss landscape visualizations offers intuitive insights into model behavior and optimization dynamics.\n\n**Weaknesses**\n\n1. The methodological derivation relies on several approximations—such as Taylor expansion, Fisher matrix estimation, Hessian matrix approximation, and symmetric finite differences—without sufficient justification for their validity. The potential cumulative error raises concerns. Can the actual benefit of minimizing the upper bound be empirically verified?\n2. The related work section is brief. Some early studies on continual learning and model merging are neither discussed nor compared experimentally, making it difficult to assess the true advantages of the proposed method.\n3. Although the method reduces memory overhead using LoRA, the computation and storage of the Fisher matrix still require significant resources. Is the approach inherently limited to LoRA-based fine-tuning?\n\nQUESTIONS:\n1. In line 149, under the maximum likelihood estimation setting, shouldn't the expected Fisher information matrix be equal to the negative of the Hessian matrix?\n2. In Figure 2, the average loss landscape is visualized. It is unclear how this figure was constructed—what exactly does the \"average\" on the loss axis represent?\n3. In line 139, should θ*_t be written as θ*_i to maintain consistency with the notation?\n\nLIMITATIONS:\nPlease provide the conditions under which the approximations or include error bounds to justify their use in the derivation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a model merging idea for continual learning. The idea to continually train from a base model for a sequence of tasks and take a snapshot of the weights after each task is trained. And at inference, the weight combination of the weight snapshots is used. A simple idea that seems effective according to the experiments.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The idea of merging task snapshots makes sense, and the EWC-inspired regularisation is intuitive.\n2. The merging coefficient is obtained using a Fisher-based curvature approximation -- previously used in continual learning.\n3. Results show consistent improvements with the proposed method, compared to other fine-tuning or prompt-based methods.\n\n## Weaknesses\n1. Previous continual learning methods are not compared. These methods were developed before the introduction of foundational models; however, they can be easily incorporated into the pretraining+finetuning framework. Many notable continual methods include EWC, GEM [a] and their variants. Given the similarity to EWC, these methods need to be compared.\n2. Related to the above point, the perturbation-based method and the merging coefficient derivation have similarities with EWC and should be credited, in my opinion.\n3. The weight coefficient is mentioned as \"optimal\" in the abstract; however, it is not optimal, and it is derived for a Fisher-based second-order approximation of the loss.\n4. How does the order of tasks affect the performance? If the weights are sequentially trained, it seems that the task order would matter, and it can have an impact on the final performance. \n- Why not train for all the tasks from the base model and do the merging? How would this compare to the proposed method?\n5. The memory grows linearly with the no of tasks, anyway to reduce this?\n6. The experiment setup is not clear. Is the task ID known at inference (multi-head or single-head evaluation) [b]?\n\n## References\n- [a] Lopez-Paz, David, and Marc'Aurelio Ranzato. \"Gradient episodic memory for continual learning.\" Advances in neural information processing systems 30 (2017).\n- [b] Chaudhry, Arslan, et al. \"Riemannian walk for incremental learning: Understanding forgetting and intransigence.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents LoRA-P&M, a continual learning (CL) method that applies model merging to LoRA components across tasks. To enable effective merging and improve generalization, two techniques are introduced: (1) a theoretically grounded merging coefficient, $\\alpha$, computed using Fisher matrices; and (2) perturbations along task vectors introduced during training. Experimental results show that LoRA-P&M outperforms existing methods on multiple CL benchmarks. Loss landscape visualizations further suggest that it promotes flatter minima for the merged models.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n- The core idea adopts a simple yet effective model merging strategy for continual learning, supported by additional techniques that enhance its performance. \n\n- Tables 2, 3, 5, and 6 show promising results compared to LoRA-based and prompt-based continual learning methods across diverse datasets.\n\n- The analysis in Figure 2 is insightful and highlights the benefits of perturbation training. \n\nWeakness: \n- LoRA-P&M requires storing a Fisher matrix $F_i$ for each task, leading to memory growth that is linear in the number of tasks. \n\n- Although the paper claims to achieve state-of-the-art performance (Lines 20-21 in the abstract), several relevant continual learning methods are missing from the comparison. For example, recent LoRA-based approaches such as I-LoRA [1] and O-LoRA [2] are not included. Moreover, several newer prompt-based methods are also omitted [3][4][5][6].\n\n- Equation 17 in Section 3.4 is somewhat unclear. Since LoRA parameters are typically inserted at different locations within the network, it is not immediately clear why they are summed in this equation. \n\n- The idea of weighted model parameter combination (i.e., model merging) in continual learning has also been explored in CoFiMA [7]. However, when comparing the ablated version of LoRA-P&M (i.e., LoRA-M in Table 4) with CoFiMA (Table 3), LoRA-M appears to perform worse. This raises the question of whether the proposed weighted model merging strategy—specifically, the use of the optimal merging coefficient $\\alpha$—is less effective than CoFiMA's approach.\n\n\n[1] 2025 ICASSP Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning. \n\n[2] 2023 EMNLP Orthogonal Subspace Learning for Language Model Continual Learning. \n\n[3] 2024 ECCV RCS-prompt: Learning Prompt to Rearrange Class Space for Prompt-based Continual Learning. \n\n[4] 2024 ICLR Prompt Gradient Projection for Continual Learning. \n\n[5] 2024 ECCV PromptFusion: Decoupling Stability and Plasticity for Continual Learning. \n\n[6] 2024 CVPR Consistent Prompting for Rehearsal-Free Continual Learning. \n\n[7] 2024 ECCV Weighted Ensemble Models Are Strong Continual Learners\n\nQUESTIONS:\nBesides the weakness shown in the above section, please also see the following questions: \n\nQ1: Since perturbation training aims to optimize a more complex loss function (i.e., Equation 14), does it lead to slower convergence? \n\nQ2: Given that perturbation training leads to flatter minima, as shown in Figure 2, could it also benefit other LoRA-based continual learning methods, such as SD-LoRA? \n\nQ3: LoRA is introduced in Section 3.4 mainly to address memory consumption. If memory were not a limiting factor, would the proposed method still be effective—or perhaps perform even better?\n\nLIMITATIONS:\nThe paper acknowledges a limitation that the Fisher matrix may not fully capture the true curvature of the loss landscape. Another potential limitation is that storing a Fisher matrix for each task results in memory usage that grows linearly with the number of tasks, which may hinder scalability to large task sequences.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the concept of model merging into continual learning. Unlike traditional continual learning approaches, the proposed P&M, after acquiring parameters for the new task, employs an parameter interpolation method to balance the model performance across both old and new tasks. The interpolation weights are theoretically derived. Furthermore, the authors incorporate regularization term during training to enhance the effectiveness of model merging.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nS1: This paper proposes a novel continual learning paradigm. The authors have conducted detailed theoretical derivations tailored to this new paradigm, along with reasonable training perturbation and model merging methods. The work shows high originality\n\nS2: The derivations are rigorous and appear with no obvious major weaknesses. The conclusion seems sound and well-supported.\n\nS3:  The writing is clear and well-structured. Experiments are conducted on multiple datasets, enhancing the reliability and credibility of the work.\n\n*********\n*********\n\nWeaknesses\n\nW1:​​ This paper proposes a novel continual learning paradigm with detailed theoretical derivations. However, I have doubts whether this pipeline approach, which first trains the model based on the new task and then solves the closed form solution to find the optimal **α**, can outperform traditional one-stage incremental training method. Intuitively, more steps lead to error propagation, which may not guarantee improved performance.\n\nW2: Another concern is that the proposed linear interpolation method is overly rigid. Specifically, assigning a same “α” across all network parameters ignores the fact that different parameters contribute unequally to task performance.\n\n​​W3:​​ The derivation in the second part (train with perturbation) seems to be a relatively simple deduction, which lacks explicit linkage to the optimal **α** in the first step. The final conclusion is also somewhat simple, merely introducing some perturbations without clearly establishing a connection to the eventual interpolation method.\n\n​​W4:​​ This paper provides extensive derivations and makes some assumptions. Are these assumptions supported by empirical evidence, such as statistics of parameters from training datasets?\n\nQUESTIONS:\n(1) Please provide more/additional justification for the necessity of the two-stage approach. For example, in what scenarios does this method demonstrate superiority in terms of performance, efficiency, or practicality? Including concrete examples would also help readers build an intuitive understanding of its benefits.\n\n(2) Is it possible to add **α** during the derivation of training with perturbation?\n\n(3) Can more parameter statistics be added to demonstrate the validity of the hypothesis proposed in the paper?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}