Title: Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging
Venue: NeurIPS 2025
Accepted: yes
Keywords: model merging

continual learning

knowledge transfer
Abstract: Deep model merging represents an emerging research direction that combines multiple fine-tuned models to harness their specialized capabilities across different tasks and domains. Current model merging techniques focus on merging all available models simultaneously, with weight interpolation-based methods being the predominant approach. However, these conventional approaches are not well-suited for scenarios where models become available sequentially, and they often suffer from high memory requirements and potential interference between tasks. In this study, we propose a training-free projection-based continual merging method that processes models sequentially through orthogonal projections of weight matrices and adaptive scaling mechanisms. Our method operates by projecting new parameter updates onto subspaces orthogonal to existing merged parameter updates while using an adaptive scaling mechanism to maintain stable parameter distances, enabling efficient sequential integration of task-specific knowledge. Our approach maintains constant memory complexity to the number of models, minimizes interference between tasks through orthogonal projections, and retains the performance of previously merged models through adaptive task vector scaling. Extensive experiments on CLIP-ViT models demonstrate that our method achieves a 5-8% average accuracy improvement while maintaining robust performance in different task orderings. Code is publicly available at https://github.com/tanganke/opcm .


Peer reviews:
comments: SUMMARY:
This paper proposes a novel continuous model merging method, named OPCM. Unlike traditional model merging approaches that require access to all models simultaneously, OPCM processes models one-by-one using orthogonal projections and adaptive scaling. The method maintains constant memory complexity and mitigates task interference. Extensive empirical evaluations using CLIP-ViT architectures on up to 20 image classification tasks show that OPCM outperforms strong baselines in both average accuracy and backward transfer (BWT), while remaining robust to task orderings.

STRENGTHS AND WEAKNESSES:
Strengths.
+ This paper is clearly written and easy to follow.
+ This paper introduces a new setting for model merging by incorporating the idea of continual learning.
+ Extensive experiments on various architectures and tasks show the effectiveness of the proposed method.

Weaknesses.
+ The paper proposes to solve the continuous model merging problem, where old task vectors cannot be stored. However, in practice, storing task vectors is usually not expensive, making this assumption somewhat questionable. So, I think this problem may not be a significant setting for model merging.
+ This paper uses the assumption that projecting the task vectors in the orthogonal space can cause less conflict with old tasks. Does this assumption always hold? Maybe some important dimension of old task vectors is zero just simply because the pretrained weights are good enough.

QUESTIONS:
See weakness.

LIMITATIONS:
No. This paper uses the assumption that projecting the task vectors in the orthogonal space can cause less conflict to old tasks, which may not be valid for some scenarios.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for continual model merging, where new models arrive over time and the old models are not available anymore other than via the current merged checkpoint.\nThe method relies on projections that keep the incoming models' task vector and the current merged model orthogonal as well as adaptive scaling.\nThe authors show that the method outperforms other baselines for continual model merging on a variety of tasks and provide theoretical analyses of their method.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper is easy to read and follow.\n\n- The method proposed by the authors outperforms other baseline techniques for continual model merging on a variety of datasets.\n\n## Weaknesses\n\n- The motivation of memory complexity increasing with the number of models is of course true but this can also be simple disk storage which is easy to get by or even cloud storage which makes the task setting overall a bit artificial.\n\n- I'm not sure if claiming \"we present a *novel* perspective on multi-task model merging through the lens of \ncontinual learning\" (L96-97) is fair, as the formalization shares many similarities with Dziadzio et al. 2024, which is nevertheless also discussed and cited. \n\n- The adaptation of Task Arithmetic in Eq. 5 is not consistent with theoretical derivations of Task Arithmetic, e.g., from Daheim et al. 2024, as the task vector is calculated wrt. a different model than the one it is added too, which could be discussed more.\n\n- The work should discuss other papers that use SVD to reduce Task interference more, such as Gargiulo et al. 2025.  Perhaps a comparison could also be added.\n\n- The fact that the method is outperformed by other merging techniques like RegMean if the restriction on continually merging and not keeping all checkpoints is lifted is hidden in Appendix E which is not referenced in the main paper. It would be more appropriate to discuss and reference it there.\n\n## References\n\nDaheim, Nico, et al. \"Model merging by uncertainty-based gradient matching.\" ICLR, 2024.\n\nGargiulo, Antonio Andrea, et al. \"Task singular vectors: Reducing task interference in model merging.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\nQUESTIONS:\n- Could you please explain how task information is exactly guaranteed to be preserved in Corollary 5.3? Initially it seems like a trivial rewriting of the merging equation that does not provide any guarantees.\n\n- Could you please expand a bit on the motivation for continual model merging? What are the uses cases where checkpoints could not be kept? I think clarifying this also in the paper could help motivating the method.\n\n- How did you choose the scaling for e.g. Task Arithmetic in Table 6?\n\nLIMITATIONS:\nThe paper discusses some limitations, such as, that the order matters in the method and that SVD adds additional overhead.\nHowever, the limitations could be extended by discussing the results in Appendix E that show that conventional model merging techniques (that are not continual in this works' sense) can outperform the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-16"}

comments: SUMMARY:
The paper introduces OPCM, a training‑free, continual model merging method. It sequentially integrates new fine‑tuned models into a merged model via orthogonal projections and adaptive scaling, reducing interference and maintaining constant memory per merge.

STRENGTHS AND WEAKNESSES:
Strengths: 
- 1. The paper is clearly written and addresses a novel and timely problem in model merging.

- 2. The authors conduct comprehensive experiments to demonstrate the effectiveness of their method.

Weaknesses:
- 1. In Table 1, the results for C. finetune lack variance/error bars.
- 2. The extremely low variance of accuracy when using SWA in Table 1 suggests the tasks are highly similar, and possibly only a few models dominate the merged performance. To strengthen the claim of order-robustness, the authors should consider testing on tasks with greater diversity.

QUESTIONS:
- 1. In classification settings, the method does not consider the classification head, and the tasks used appear to be relatively similar. As a result, for Continual Model Merging, there is a possibility that only a few models dominate the merged result, leading to low variance and insensitivity to task order.
   - Does the use of simple classification tasks limit the persuasiveness of the method for Continual Model Merging?
   - Would it be more meaningful to evaluate the method on more diverse tasks with shared objectives, such as language generation, where the differences between tasks are more pronounced?
- 2. Why is the accuracy of the single EuroSAT task in Figure 12(a) significantly lower than the accuracy when MNIST and EuroSAT are merged?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the problem of merging fine-tuned models in a sequential setting where models arrive over time without retraining or revisiting data. Unlike prior approaches that rely on simultaneous access to all models and simple weight averaging, the authors propose a projection-based method that merges each new model by projecting its updates orthogonally to the subspace of previously merged updates, with an adaptive scaling mechanism to control drift. This setup helps mitigate task interference and maintain performance across tasks. The method is simple, memory-efficient (constant in the number of models), and shows strong empirical results on CLIP-ViT, outperforming continual baselines by 5–8% on average\n\nSTRENGTHS AND WEAKNESSES:\nWhat stands out in this paper is how it reframes model merging as a continual process where models arrive one at a time rather than being merged all at once. That feels like a much more realistic and practical setting, especially for real-world systems where tasks evolve over time. The idea is clean and well-motivated. The method itself is simple, does not require retraining, and keeps memory use low, which makes it easy to apply. The results are strong and consistent across model sizes and task counts, and it's good to see that performance holds up even when task order changes.\n\n\nThe technical novelty is on the lighter side. Using orthogonal projections to reduce interference has been done before in the continual learning literature. Adaptive scaling makes sense in this context but is also a pretty natural move. The method is well-executed, but it mostly repurposes existing ideas for the merging setting rather than introducing something fundamentally new. It would help to better understand what is specific to merging here that is not already covered in related projection-based continual learning approaches.\n\nhttps://arxiv.org/abs/2301.12131\nhttps://openreview.net/pdf?id=hac6DzbMa7\n\nQUESTIONS:\nHow is this different than the works I listed about from the orthogonality contribution. \n\nIn addition, what about this paper:\n\nhttps://arxiv.org/html/2505.12082v1\n\nLIMITATIONS:\nDoes having the weights be projected orthogonal to each other cause limitations - lack of sharing useful information from different tasks?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}