Continual Model Merging without Data: Dual Projections for Balancing Stability and Plasticity
Authors: Enneng Yang
Anke Tang
Li Shen
Guibing Guo
Xingwei Wang
Xiaochun Cao
Jie Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Model Merging
Continual Learning
Multi-task Learning
Primary area: general_machine_learning
TL;DR: This paper proposes a data-free dual orthogonal projection framework to perform continual model merging.

Abstract: Model merging integrates multiple expert models with diverse capabilities into a unified framework, facilitating collaborative learning. However, most existing methods assume simultaneous access to all models, which is often impractical in real-world scenarios where models are received sequentially. While some studies have investigated continual model merging (CMM)--which involves sequentially merging multiple models--the challenge of balancing prior knowledge (stability) and incorporating new tasks (plasticity) remains unresolved. This paper, for the first time, formally defines the stability and plasticity of CMM from the perspective of orthogonal projection. Subsequently, we analyze the relationships among the spaces spanned by task data, historical gradients, and accumulated gradients. Building on this, we propose a data-free \textbf{D}ual \textbf{O}rthogonal \textbf{P}rojection (DOP) method, which eliminates data dependence and mitigates interference between the merged model and models for old and new tasks by projecting their parameter differences onto their respective approximate data spaces. Finally, to solve potential conflicts between stability and plasticity, we reformulate DOP as a multi-objective optimization problem and employ a multi-gradient descent algorithm to obtain a Pareto-optimal solution.  Extensive experiments across multiple architectures and task configurations validate that our approach significantly outperforms state-of-the-art CMM methods.

Continual Model Merging without Data: Dual Projections for Balancing Stability and Plasticity
Enneng Yang 1a.5..2.6,, Anke Tang', Li Shen Guibing Guo",, Xingwei Wang", Xiaochun Cao", Jie Zhang" Shenzhen Campus of Sun Yat-sen University, China Northeastern University, China
Shenzhen Loop Area Institute, China Wuhan University, China Nanyang Technological University, Singapore ennengyang @ gmail.com; anketang @ edu.eed shenli6@ mail sysu...ss..c... edu.en; guogb @ www.neu.cdu.cr wangxw swc.neu. edu.en; caoxiaochun maall.llseeeee www.ssysuuuu... edu zhangj@nti ent..du.ug
Abstract
Model merging integrates multiple expert models with diverse capabilities into a unified framework, facilitating collaborative learning. However, most existing methods assume simultaneous access all models, which is often impractical in real-world scenarios where models are received sequentially. While some stud- ies have investigated continual model merging (ChM) involves sequen- tially merging multiple models-the challenge of balancing prior knowledge (sta- bility) and incorporating new tasks (plasticity) remains unresolved. This paper, for the first time, formally defines the stability and plasticity of CMM from the perspective of orthogonal projection. Subsequently, we analyze the relationships among the spaces spanned by task data, historical gradients, and accumulated gradients. Building on this, we propose a data-free Dual Orthogonal Projection (DOP) method, which eliminates data dependence and mitigates interference be- tween the merged model and models for old and new tasks by projecting their parameter differences onto their respective approximate data spaces. Finally, solve potential conflicts between stability and plasticity, we reformulate DOP as multi-objective optimization problem and employ multi-gradient descent algo- rithm to obtain a Pareto-optimal solution. Extensive experiments across multiple architectures and task configurations validate that our approach significantly out- performs state-of-the-ar CMM methods.
1 Introduction
Fine-tuning pre-trained foundation models for task- or domain-speeinf applications has become dominant paradigm in machine learning [50, 102]. The open-source culture in the machine learn- ing community has further accelerated this trend, driving the rapid expansion of fine-tuned expert models. For instance, Huggingface [84] currently provides access to over 1.6 million models. A key challenge how to effectively leverage these diverse expert models to address novel or more complex tasks. Recently, model merging has gained prominence tegrating multiple expert models with identical architectures at the parameter level. As a general technique, it has diverse practical applications, including multi-task learning [35, 98, 2], continual pre- tet-triiinggfiie-tummeee -uinnngg-tnnnggggg [64, 86, 106, 52], zero/few-shot learning [92, 58, 34, 32], model attack/pro- tection [101, 6] and others [30, 9, 32].
promising solution by in-
Traditional model merging settings primarily aim to resolve parameter conflicts during the merging process [93, 103]. To address this challenge, researchers have introduced various innovative strate- gies, including weighted- basee merging methods [53, 37, 97], subspace-based approaches [98, 91],
"This work was completed during Enneng Yang's study and employment at institutions 1, 2,5. Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and dynamic merging techniques [90, 51]. However, these approaches are built on the fundamental assumption that all models are simultaneously availlble for merging. In practical scenarios, tasks or domains typically emerge incrementally, and preserving past data or models indefinitely is often infeasible [22, 81, 104, 79]. Furthermore, loading all expert models into memory simultaneously dur- ing merging becomes infeasible when the number of tasks is large or the models are of substantial size, resulting in excessive memory consumption. To address these challenges, two recent studies have introduced the concept of continual/temporal model merging (CMM) [24, 74], which aims to merge a sequence of models arriving incrementally. Among them, TIME [24] is an analytical study that highlights key conclusion: Due to the greater complexity of CMM setting. traditional merging methods struggle to generalize effectively to this setting. OPCM [74] introduced a method tailored for the CMM setting, employing parameter decomposition to mitigate catastrophic forgetting [63] of previous models when merging new ones. However, OPCM fails to account for the interference that prior tasks may introduce to newly learned tasks, ultimately limiting overall performance.
In this paper, we are the first to formally define two key properties that CMM should satisfy, namely stability and plasticity, from the perspective of orthogonal projection [100, 67], thereby aligning them with the objectives of traditional continual learning [79, 104, 81]. Stability [63] ensures that the continually merged model retains knowledge of previously merged tasks, while plasticity [22] en- ables the model to effectively integrate knowledge from new tasks. However, as the original task data is unavailable in the CMM setting, existing orthogonal projection techniques [67, 49, 95] in contin- ual learning cannot be employed. In addition, we demonstrate across vision and language tasks that existing CMM methods struggle to effectively balance these two objectives (in 83.2.1). To address this limitation, we analyze the relationships among the subspaces spanned by data, gradients, and accumulated gradients (in 54.1). Our analysis reveals that in the linear model, the accumulated gra- dient subspace is contained within the gradient subspace, which in turn lies within the data subspace. Therefore, the space spanned by the accumulated gradients can be viewed an approximation of the space spanned by the data.
Building on these findings, we propose data-free Dual Orthogonal Projection (DOP) method that minimizes the projections of parameter differences between the merged model and both the old and new models in the upproximate data subspace (in 84.2). Furthermore, to achieve better trade-off between stability and plasticity in CMM, we reformulate DOP as a multi-objective optimization problem and employ the multi-gradient descent algorithm (MGDA) [20] to obtain Pareto-optimal solution. Additionally, we incorporate an exponential moving average strategy to reduce fluctuations during the MGDA optimization process. It is worth noting that our approach is both memory- and time-efficient, requiring at most only three linear layers to be loaded into memory during the model merging optimization process; furthermore, the average optimization time for a single linear layer is only 1 to 3 seconds. Finally, we conducted extensive experiments across three vision architectures, one language architecture, and three different task quantities (i.e.. 8/14/20). The results demonstrate that our method achieves the best performance under the CMM setting by better balancing stability and plasticity. The contributions can be summarized into the following:
We formalize the two key optimization objectives of CMM, namely stability and plasticity, within the framework of orthogonal projection theory, and we further examine the relationships among the subspaces spanned by task data, gradients, and accumulated gradients.
We propose data-free Dual Orthogonal Projection (DOP) method for CMM, recasting it multi-objective optimization problem to obtain a Pareto-optimal solution, ensuring an effective trade-off between stability and plasticity.
We conduct extensive CMM experiments on four architectures (covering vision and language models) with varying task numbers, and show that our method consistently outperforms state-of- the-art (SOTA) CMM approaches on both old and new tasks.
2
Related Work
This paper is related to model merging [68, 35, 92, 2] and continual learning [104, 79, 81]. Due to space limitations, we provide a brief overview in this section and discuss the details in Appendix B.
Traditional Model Merging: Based on the merging strategies, existing methods can be roughly di- vided into three main categories [93, 103]: (1) Weight- Methods: These methods employ grid search, hcuristic strategies, or learnable approaches to determine the optimal merging coefficients,
2

---

thereby mitigating task interference [85, 35, 105, 97, 53, 37]. (2) Subspace-but basspa based Methods: These approaches project the full parameter space into an appropriate subspace and then merge the param- eters within this subspace to reduce task conflicts [91, 29, 98, 78, 33, 106, 25, 83]. (3) Router-based Methods: These methods dynamically adjust model parameters for each task or instance during in- ference, rather than using shared parameter set across all tasks or instances [90, 73, 51J. However, the aforementioned methods have a fundamental limitation: they necessitate simultaneous access to all models (i.e.. "offline" merging), thereby limiting their applicability in streaming scenarios.
Temporal and Continual Model Merging: To the best of our knowledge, only two studies have explored the CMM setting [24. 74], in which sequentially arriving models are merged into a unified model. TIME [24] introduced the concept of temporal model merging and examined the impact of various merging techniques, initialization strategies, and deployment approaches. It concluded that standard offline merging techniques generalize poorly to CMM, underscoring the challenges of this setting. However, TIME primarily functions as an analytical pipeline for evaluating existing methods, without introducing a tailored approach for CMM. The study most closely related this work is OPCM [74], which mitigates interference with prior tasks by decomposing the parameters of new models in order to address catastrophic forgetting. However, OPCM primarily addresses the forgetting induced by the new model on the old model, while overlooking the influence of the old model on the new model, resulting in suboptimal performance.
3 Preliminaries
3.1 Notations and Problem Definitions
Consider a scenario with T expert models, denoted as { O2,..., Or). Each model O, is in- dependently fine-tuned from a common pre-trained model O, (e.g., ViT [23, 61 on a domain- or task-specific dataset D {xp", ) where X and Y denote the input data and correspond- ing labels. The aim of model merging is to merge expert models at the parameter level to construct unified model O, that preserves the capabilities of all experts without requiring access their re- spective training data. Formally: O_ = (orege((O, O.. Or), where merge() is merging oper- ation [85, 53, 37]. A straightforward approach is weight averaging [85]: θ + Ei= O. Recent studies indicate that task vector-based model merging [35, 57, 91, 97] frequently surpasses simple weight averaging in performance. Specifically, the task vector [35] for task t is defined as the differ- ence between the fine-tuned parameters O and the pre-trained parameters Og. i.e., Te = O, - Oo- Based on task vectors, the model merging problem is redefined Definition 3.1.
Definition 3.1 (Task Vector-based Model Merging). Given a pre-trained model O, and multiple expert task vectors {TI, ......,... .r.. the goal is to aggregate the pre-trained model and expert models to produce unified model θ,, defined as: O, (er, T1, T2..... ....... TT).
However, traditional model merging faces a key limitation: it requires all expert models to be avail- able simultaneously. which is often unrealistic. To overcome this challenge, we aim to adapt model merging an incremental setting, referred to Continual Model Merging (CMM) [74). The defi- nition of CMM is provided in Definition 3.2.
Definition 3.2 (Continual Model Merging). Given pre-trained model O, and sequence of models {O), .,,..... } arriving sequentially, the goal to incrementally merge these models at the parameter level. At each time step 6, the merged model e.!) integrates information from the previously merged model (!--1) and the newly arrived task model O₂ (or task vector Tt). ensuring high performance across all tasks (i.e., 1, 1...... f) without requiring access to the orig- inal training data or previously accessed expert models. The process of CMM is formalized as:
( merge merge(Og, (!1)) Te), where 2 2.
As task-specific expert models cannot be accessed simultaneously, the CMM setting poses a greater challenge than the traditional model merging setting. Prior studies have demonstrated that existing model merging methods have failed to generalize effectively in the CMM setting [24, 74), and our results in the next section further support this conclusion.
3

---

OPCP
22
(a) Stability VIT-B/32 (b) Stability Flan- T5
(c) Plasticity ViT-B/32 (d) Plasticity Flan-TS
Figure 1: Analysis of different CMM methods for stability of old tasks (a-b) and plasticity of new
tasks (c-d). This analysis covers the vision model ViT-B/32 and the language model Flan-T5 T5-Base.
3.2 Rethinking Stability and Plasticity in CMM
Unlike traditional model merging, which primarily aims to mitigate task interference 191, 93], we argue that CMM should simultaneously satisfy two key properties-stabiliyy and plasticity-in align- ment with the principles of traditional continual learning [81, 104, 79]: (1) Stability: Preventing catastrophic forgetting of previously merged tasks when incorporating new models [63]. (2) Plastic- ity: Enabling the model to effectively assimilate knowledge from new models [22].
3.2.1 The Challenge of Stability and Plasticity CMM
This section first analyzes the performance of existing CMM methods in terms of stability and plas- ticity. Without loss of generality, we take the most classic method, SWA [36], and the SOTA method, OPCM [74], as representative methods. Details of both methods are in Appendix D. Experiments conducted on two architectures: ViT-B/32 [23, 61] and Flan-T5-Base [14].
The results for stability are shown in I
(a) and Fig. We evaluate the performance of the merged model on the first task after sequentially merging up to the (-th model (L E (1, 8)) according to task order. We observed clear performance degradation for both SWA and OPCM, indicating poor stability. For instance, after merging eight models using SWA and OPCM, the performance of VIT on the first task (i.e., EuroSAT) dropped from 99.1 to 75.7 and 88.8, respectively. Similarly, on Flan-T5, the performance on the first task (i.e., MRPC) drops from 87.5 to 79.4 and 79.7. This suggests that current CMM methods exhibit limited stability.
The results for plasticity are shown in Fig.1(c) and Fig. Idd). We evaluate the models performance on the newly merged task at step A score closer to the expert model (i.e., a model fine-tuned independently on that task) indicates better plasticity. We find that on the 8-th task (i.e., Cars), the expert model on the VIT architecture achieved score of 78.5, while SWA and OPCM only reached 62.4 and 63.1, respectively. On Flan-T5, for the 8th task (i.e., STSB), the expert model achieved 88.7, while SWA and OPCM achieved only 73.2 and 76.2, respectively. These results demonstrate that current CMM methods also perform poorly regarding plasticity.
3.2.2 Reinterpreting Stability and Plasticity from the Perspective of Orthogonal Projection
In this section, we analyze how an ideal CMM satisfies stability and plasticity from the perspective of orthogonal projection [67]. Recall Definition 3.2 of CMM: at time step f, the available models in- clude the pre-trained model Oo. the previously merged model (!(-1) and the newly arrived model O (or task vector T²). In particular, from the perspective of network architecture and parameter distribution, Transformer-based architectures [75. 23] (Fig. 4(a) in the Appendix) allocate most of their parameters to linear transformations, including the Q/K/V projection layers in multi-head at- tention, the two linear transformations in the feed-forward network, and the embedding layers. Our analysis in Fig. 4(b) in the Appendix indicates that linear layer parameters constitute 90- - 99% of the entire model (i.e., Og). Therefore, effective linear layer merging is crucial for successful model merging [37]. Next, we illustrate how to decompose the model merging problem into sequence of linear layer merging sub- problems, incorporating stability and plasticity.
Consider two linear layers, f(X.) W.T X, and f(X,) = W., X.. where W., W,, E Rdd xd2 denote the weight matrices of a linear layer in old (---)) and new O models, respectively. X., X,, E IRd represent the input data for the old and new tasks, while dj and d2 denote the input and output dimensions of the linear layer. To achieve model merging, the goal is to in- tegrate the lincar layer parameters W, and W,, to produce new linear layer with parameters

---

W, = =:mrgee W.). Alternatiely, this can be expressed in the form of task basedr-based merg- ing: W. m.. p. Tas Tn). where T, W. -Wo and T, W,, W represent the task vectors, i.e., parameter shifts from a shared pre-trained model Wo, for the old and new models, respectively.
To minimize the loss of the merged linear layer W. across both the old and new tasks, seek to simuitancously satisfy the following stability and plasticity objectives:
min (Cx: Lp} s.L. L. = d(w. X. - Wo X.), Cp d(W. X, WT, X.),
(1)
where d(-,-) denotes an arbitrary distance function, such as Lg loss. To further refine the formulation, we introduce the delta parameters of the merged model relative to the old model W. and the new model W,, as follows: AW" = W. W.. AW; = W. - Wn, Thus, L. and Lp can be equivalently reformulated as L, - X.) and Ep = d( d(AW
Furthermore, based on the properties of orthogonal projection theory [67, 49, 95] in continual learn- ing, if AW" is orthogonal the subspace spanned by the old task's data X., i.e., AW; 1 span(X.), then AW" X, O, allowing us to deduce that: W.X, = (W, W.AAW") Xx, = W.X,- In other words, the merged linear model W. produces exactly the same output as the expert model W. for the old task, thereby preserving the capabilitiess of the old task. Similarly, if the projection of the delta parameters AW* onto the input space spanned by the new task's data X,, is 0 (i.e., lies in the or- thogonal complement: AW; . span(X..)). then W.X, = (Wn + AW;)
W,,,, That is, the merged W. effectively incorporates the knowledge of the new task, meaning that produces the same response as W,, to the X,,
Building upon the definition of the delta parameters and the theory of orthogonal projection, the two objectives of CMM-stability and plasticity-can be reformulated as minimizing the projections of the delta parameters onto their respective inpul spaces. However, in the CMM setting, the data X. and Xn for the previous and new tasks are unavailable, and the spaces spanned by these data cannot be directly obtained. Instead, only the expert models trained on these tasks are available, posing a significant challenge to solving CMM.
4 Methodology
This section first analyzes the correlation among the subspaces spanned by data, gradients, and task vectors $4.1. Leveraging this correlation, we introduce a data-free DOP method for CMM in 54.2.
4.1 Subspace Correlation Analysis
Property 4.1 (The Gradient Space within the Input Space). For linear layer f(X) W x. the space spanned by the gradients Vw C with respect to its weight matrix W lies within the space spanned by the input data X.
Proof. Consider a linear layer with input X, ground truth Y. weight matrix W Rdd and prediction Z = f(X) = o (I),,), where (-)) is a nonlinear activation function, and dj and d2 represent the input and output dimensions of the linear layer, respectively. Let the loss function be L(Z, Y), which is differentiable with respect to f(X) and can take various forms (e.g., cross- entropy, L2 loss, or other differentiable functions). By applying the chain rule, we obtain: ac = ac az (2C 87 aw 88 @ oo) xT = [a, X, azX,.... adg X]. where @ denotes element- multiplication, and [a, ag..... adz] refers to the vector formed by 82 Co'. It clearly shows that each column of the gradient Vw is a scalar multiple of the input X, i.e., Or (k E {1, 21, .d2)). This result confirms that the gradient Vw L resides within the subspace spanned by the data X [67, 49, 95, 48, 12].
aw
.......d2))
Property 4.2 (The Task Vector Space within the Gradient Space). Within the gradient- gradient-based it- erative update framework, given the initial parameters Wo. the learning rates (n,,)) and the gradients C) from k updates, the task vector is directly associated with the accumu- lated gradients. Specifically, the task vector is computed as the weighted sum of the gradients, where the weights correspond to the learning rates across all training steps. Consequently, the task vector Tr resides within the subspace spanned by these historical gradients.
Proof. Consider a task undergoing & iterative updates during fine-tuning (e.g., via SGD or its vari- ants [65]). According to the standard gradient descent update rule, the parameter updates are given
5

---

by: WA, = 11, nt.k- W.,, C This mt.VV... Vw.,,, Consequently, the difference between the parameters Wr& at the step k and the initial Wn corresponds the cumulative sum of the up- dates at each step [89, 83, 12]. This is consistent with the definition of the task vector [35], which is 7: ==
Σ "), Vw,,f.
4.2 Data-free Dual Orthogonal Projection Method
In the optimization objective of CMM (in $3.2.2), the unavailability of original data X, and Xn makes it infeasible to construct imput subspaces span(Xo) and span(X,)). Motivated by Properties 4.1 and 4.2, we propose upproximating the input subspaces using the subspaces spanned by the task vectors To and Tr. i.e., span(T.) and (Tan(T.)- Notably, for the newly arrived model, two proper- ties hold strictly. For the old merged model, the task vector, gradient, and data can be interpreted as the combined (or approximated) result of multiple tasks. The CMM objective thus reformulated as minimizing the projections of the delta parameters AW" W. W. and AW" -W W,, onto the task vector subspaces: minw. {Ca, Cp where Ls = Proj (AW or span (To)) Lp = Proj (A W span (Tn)) Although To and 7n are commonly referred to "task vectors" [35], they are in fact matrices of the same shape as Wo and Wn, ie., Id xda
n
To construct the subspaces spanned by task vectors To and Tn. we apply singular value decomposition (SVD) [41] to each as follows: U. Σ. V.T sTDD U, Σ, V, x (Tn) where the columns of U, and V. form an orthonormal basis for the row space and column space of To. respectively. In practice, we can retain only the top min(d1, .d) singular vectors and singular values, rather than employing the full decomposition. To capture both row-wise and column-wise directions, we leverage the left and right singular matrices obtained from the SVD decomposition. Additionally, to penalize projections along principal directions, we retain the singular values E, or Σn as weighting factors to regulate projection strength along each singular direction. By minimizing the projections of AW,"" and AW;" onto the subspaces spanned by To and Tu. respectively, such that their Frobenius norms reduced, ensure that W. retains prior task knowledge, therehy maintaining stability. Simultaneously, this approach enables W. to effectively incorporate knowledge from new tasks, thereby preserving plasticity. Since this optimization does not require access the original data, introduce the data-free Dual Orthogonal Projection (DOP) objective:
min Cos [p} s.t. C. = IIU. + Volli, C, = |0, F., I. V.ll.... (2) W.
straightforward approach is to combine the two loss terms via weighted summation and optimize W, iteratively (c.g., using SGD Adam [65]): LCMM aC, + (1 Cp. where serves as a trade-off parameter between stability and plasticity. However, grid search for a is computation- ally expensive, and a fixed weighting scheme often fails to achieve an optimal trade-off due to the inherent conflict between stability and plasticity.
We formulate DOP as a multi-objective optimization problem to balance these competing objectives and seek the Pareto-optimal solution, thereby ensuring both stability and plasticity. Without losing generality, we employ the multi-gradies descent algorithm (MGDA) [20] to iteratively solve this multi-ohjective optimization problem, that is minw. (L., Cpp in $4.2. The update rule of the merged linear model W. at each step & is given by: W,,++1 = W. 77 Jk. where W..0 (W. + Wn)/2, 7) is a global learning rate like SGD or adaptive learning rate as Adam [39] at step k. and ga is gradient update direction. W.. is initialized using simple weight averaging [85], which can be easily replaced by existing model merging methods. More specifically, MGDA seeks to determine an updated direction 9k at each iteration is to simultancously minimize all objective functions. This is formulated as the following quadratic programming problem, and the one-oonennte dieeennnniionl optimization problem for ag admits an analytical solution:
-all--a))-........ Co s.t. a+= =1=plamp (VW. W Go Vw.,a.) VW,
min ||ox [
VV C.- -V Cpll
(3)
where the function cl (1) max(min(r,) 1), 0) ensures ag remains within [0, 1). thereby main- taining a valid convex combination of the two gradients.
To prevent abrupt changes in < at each update step & and to incorporate preference 7 E (0, 1) (e.g., prioritizing stability or plasticity), we adopt an exponential moving average (EMA) strategy, commonly used approach in adaptive learning rate optimizers [65]. Specifically, given ag computed

---

Table 1: Performance comparison of continual (C.) merging methods on 8/14/20 visual tasks, report- ing ACC, and BWT over ten randomly shuffled task orders. Ref. the abbreviation of reference.
ViT-R/32
VTT-R/16
VIT-1114
Method
tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
ReL.(1)
Pre-Trained
48.1
56.9
55.6
55.4
62.0
59.8
64.9
69.1
65.6
Single Tuned
90.4
89.3
89.8
92.4
91.3
91.6
94.3
93.4
93.5
Fine-TTnnd
79.8
67.4
62.6
82.9
72.2
68.2
90.0
70.9
77.7
Continual
Model Merging Methods
Averaging (SWA)
66.3:10.0
65.4
61.1:0.00
72.3=0.0
69.7=0.0
64.8:10.0
80.010.0
77.540.00
71.10:0.0
ACC (1)
Task Arithmetic
67.5++0.0
66.5+0.0
60.6+0.0
77.1=000
70.9=0.0
64.2-+0.00
82.1+0.0
77,9+000
70.3++0.0
C. Ties-Merging
49.0=10.2
66.2+00.
59.9=0.77
66.8=3.7
70.5=0.8
63.021166
64.3+7.0
78.0=10.6
68.3++0.9
OPCM
75.5++0.5
71.9+0.3
65.7420.2
8188=0.3
7..1=0.5
70.3+00.2
87.0+0044
83.5++0.2
76.0++022
DOP (Ours)
78.3::166
73.01:.2
68.5::1.
8..5=0.8
76.2=1.2
72.24:1.8
0..3
85.5:0.88
80.71009
Averaging (SWA)
-11.5=222
-..0+1.3
-7.122.1
1.77
17.114
-7341.77
-7,311,4
-5.+10
16.1.
BWT (f)
C. Task Arithmetic
-9.61:1.55
-1,3
3.440.4
4.2++..0
-13340
-36404
-7.10088
-1840.3
3.34-033
Ties-Merging
-15.3-8.0
1.94066
-15407
554044
14407
-15412
-13045577
1.104
-29110
OPCM
-6.31:1.1
6.0.1.0
7.881.
4.8107
5.1114
6.3422
2.6110
4.300.7
6.51L8
DOP (Ours)
-6.142.3
-88442.1
-12.5488
1.94..
-75422
-11.4428
-.221.
4.61.
-84114
step k; we apply the following smoothing strategy: Os, = Ba,,x
- ==ppp=p000 Bap,
Cpp,k
(1-B)(1-00) - where as,0 7, 0p,0 = =1-% and the smoothing factor / E [0,1] controls the degree of historical information incorporated into the current update, while 7 determining the initial preference for stability or plasticity. Finally, the optimal update direction Jb at iteration & is given by: 9 a Vw. pppk Vw., Cp. This update process repeated iteratively until convergence. We give the pseudocode of the DOP algorithm in Appendix E.
Experiment
5.1 Experimental Setup
For visual tasks, our experimental setup follows OPCM [74]. We adopt three ViT models of varying scales from CLIP [61] as backbones: ViT-B/32, ViT-B/16, ViT-L/14. Their statistical details are presented in Fig. 4. We fine-tune three architectures 20 datasets to obtain task-specific expert models for continual merging. For each architecture, we consider three task number settings: 8 tasks, 14 tasks, and 20 tasks. For language tasks, following FusionBench [72], we primarily use the Flan-T5-Base [14] architecture and fine-tune it on the eight GLUE [77] tasks to obtain eight expert models for continual merging. Dataset details are provided in Appendix C.
Models and Datasets:
Baselines: We primarily compare four state-of-the-art model merging methods: Averaging (SWA) [30, 85]. C. Task Arithmetic [35]. C. Ties-Merging [91], and OPCM [74]. Detailed de- scriptions of these baselines are provided in Appendix D. We also include the pre-trained model, individual expert models, and continually fine-tuned models as reference baselines.
Evaluation Metries:
We mainly measured the final merged model's average task accuracy (ACC) across all evaluation tasks, where a higher ACC indicates better retention of old and new task capa- bilities. We also evaluate Backward Transfer (BWT), where a higher BWT, meaningful only under comparable accuracy, indicates less forgetting.
5.2 Evaluating the Performance of Continuously Merged Visual Models
Accuracy: As shown in Tab. make the following key observations: (1) The pre-trained model lacks task-specific knowledge, resulting in poor performance. Single fine-tuned models achieve the best performance but necessitate maintaining and deploying separate model parameters for each task. Continual fine-tuning allows single model to sequentially learn downstream tasks; however, due to catastrophic forgetting, its performance falls between that of the pre-trained and fine-tuned models. As the number of tasks increases, the performance of continual fine-tuning deteriorates significantly. approaching that of the pre-trained model. (2) Among all CMM methods, OPCM exhibits superior performance compared to SWA, C. Task Arithmetic, and C. Ties-Merging. This is attributed to OPCM being specifically designed for the CMM setting, leveraging parameter decomposition to mitigate forgetting of old tasks. In contrast, the other methods were originally designed for settings where all models are available simultaneously, making them less effective the CMM [24]. (3) Our
DOP method consistently demonstrates superior performance across nearly all scenarios, owing
7

---

Corctert
Strettat
-90
ALAT
HEST
trep
mooe
rrom ALl
22
Can
(a) SWA
(b) C. Task Arithmetic
(c) OPCM
(d) Ours
Figure 2: Performance during merging of 8 tasks with ViT-B/32. Row shows accuracy on previous
tasks (1 -) after merging task & (stability); diagonal shows accuracy on task / itself (plasticity).
Table 2: Experimental results of continually merging Flan-T5-base models on all eight tasks.
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
ACC(t) BWT())
(Ref.) Single Fine-Tune
69.1
82.7
85.5
90.9
84.0
84.4
92.9
87.4
84.6:10.0
Averaging (SWA)
69.1
62.6
79.4
89.8
83,9
81.2
91.7
73.2
78.8=00.0
1.1
C. Task Arithmetic
70.5
57.8
78.4
90.2
83.6
80.5
92.3
77.8
78.9+0.0
118
C. C.Ties-Merging
68.8
50.5
79.8
89.9
83.1
79.2
91.9
79.3
1.68+1.6
10,5+199
OPCM
69.7
72.9
78.8
90.2
83.8
82.2
92.3
74.7
80.6-0.3
-2.120.6
DOP (Ouns)
72.6
80.9
81.0
90.4
83.4
81.0
92.9
86.1
83.610.11
-1.340.3
to its effective balance of plasticity for new tasks and stability for old tasks via multi-objective optimization. For example, when merging 20 tasks using the ViT-L/144 model, our method attains an accuracy of 80.7, substantially surpassing the strongest baseline OPCM, which achieves 76.0.
Stability and Plasticity: While the BWT metric in Tab. partially quantifies the degree of forget- ting, only meaningful when comparing methods with comparable performance. When com- paring DOP and OPCM, their BWT values exhibit slight differences but remain generally close. Moreover, in the CMM setting, the performance differences between methods are more pronounced, and task order can also influence the final results [74]. To more precisely evaluate the stability and plasticity of each method, we fix the task order and assess their performance on each task different stages. As illustrated in Fig. 2, under fixed task order, DOP exhibits significantly superior plas- ticity on new tasks. For example, when learning GTSRB (6-th row), DOP achieves an accuracy of 93.9, whereas OPCM, C. Task Arithmetic, and SWA achieve only 78.1, 73.4, and 61.3, respectively. Additionally, after merging all eight tasks (8-th row), DOP maintains performance of 93.8 on the first task (EuroSAT), while the other three methods achieve only 88.8, 73.3, and 75.7. This suggests that DOP provides superior stability, effectively mitigating forgetting of old tasks. In addition, the results in Figs. I(a,c) also indicate that our method has better stability and plasticity.
5.3 Evaluating the Performance of Continuously Merged Language Models
Accuracy: As shown in Tab. 2, we evaluate the performance of five CMM methods when merging eight expert models trained on the GLUE dataset. We observe that: (1) SWA, C. Task Arithmetic, and C. Ties-Merging achieve comparable performance, with scores of 78.8 and 78.9, respectively. Among them, Tiee-nggging performs the worst, likely due to significant knowledge loss caused by its sparsification of task vectors during the merging process. (2) OPCM, the only method among the baselines specifically designed for CMM, achieves the second-best performance, with an average score of 80.6 across the eight tasks. (3) Most notably, our proposed DOP method significantly outperforms OPCM, reaching a score of 83.6, remarkably close to the performance of independently fine-tuned expert models, which achieve 84.6. While expert models deliver the best results, they require maintaining a separate model for each task, which is costly.
Stability and Plasticity: As shown in the BWT of Tab. 2, our DOP method demonstrates a small degree of forgetting at higher average performance. In addition, as shown in Figs. 1(b,d), after merging all eight tasks, our method achieves performance of 82.1 on the first task (i.e., MRPC), significantly outperforming OPCM and SWA, which achieve 79.7 and 79.4, respectively, demonstrat- ing the superior stability of our approach. Furthermore, when merging the 8-th task (i.e., STSB), our method achieves an accuracy of 86.9, substantially higher than OPCMs 76.2 and SWAs 73.2, demon- strating greater plasticity of our DOP method.
8

---

30
ACC 050
A4 060
B1-01
AG- (50
Scar
2
03
159
250 350
26-4
93
94
0.5
07
opollon of svd
Stop
alpha
(a) SVD threshold
(b) Optimization steps
(c) Learning rate
(d) Coefficient Alpha
Figure 3: Hyperparameter effects on ViT-B/32 task merging (8 tasks).
5.4 Efficiency and Ablation Analysis
Efficiency: DOP employs the MGDA to iteratively refine the merged model. Tab. 3 presents the time required op- timize a single linear layer across four architectures. Our method demonstrates high computational efficiency, com- pleting each layers optimization in just 1 - 3 seconds, with- out requiring access to any original task data. As outlined in the methodology, DOP primarily targets linear layers and optimizes stability and plasticity losses the layer level. During merging only three linear layers need to be loaded: the pre-trained, the previously merged, and the new. The above results indicate that our method achieves high efficiency in both computation and memory usage.
Table 3: Running time statistics
Models Time (s)
VIT-B/32 1.74=1.18
VIT-B/16
1.55++0.47
Models
VIT-L/14
Flan-T5-Base
Time (s)
3.11=1.73
1.30+1133
Multi-objective Optimization: As shown in Tab. 4, to assess the robustness of multi-objective optimization, we
Table 4: Analysis of MOO strategy
replaced MGDA with naïve approach that directly sums the two losses in Eq. As a result, performance declined from 78.2 to 75.9, confirming the effectiveness of iden- tifying the Pareto solution. The results in Fig. 3(d) fur- ther illustrate that even with an extensive grid search to balance stability and plasticity losses, the performance remains inferior to the multi-objective optimization approach proposed in this paper. Moreover, removing the EMA strategy for loss weights in MGDA resulted in a further performance decline, underscoring the benefits of stable optimization introduced by EMA in Eq. 3.
(f)
Std (1)
weo MGDA w/o EMA
X
75.9
3.88
76.3
3.66
DOP
78.2
1.63
5.5
Hyper-parameters Analysis
the SVD-based subspace construction (in $4.2), only a subset principal directions is retained for projection. As shown in Fig. 3(a), we experimented with different subspace retention ratios and observed that when the retention ratio is low (e.g.. 0.2 or 0.3), the approximated subspace undergoes information loss. resulting in degraded performance. As the ratio increases, performance improves significantly. When the ratio exceeds 0.7, the overall performance stabilizes.
SVD Threshold:
Optimization Step: In DOP optimization, we iteratively update the merged model parameters using MGDA (in $4.2). We evaluated the effect of varying iteration counts on performance. As depicted in Fig. 3(b), performance improves as the number of iterations increases. When the iteration count sur- passes 50, performance stabilizes, demonstrating the high efficiency of our optimization approach.
Learning Rate: We analyzed the effect of different learning rates on performance (in 84.2). As shown in Fig. 3(c), when the learning rate is too small (e.g., 1e-6), fails effectively optimize the stability and plasticity objectives, thereby limiting DOP's performance improvement. However, with appropriate learning rates (e.g.. 1e-5 to 1e-3), 3), DOP exhibits stable and consistent performance.
6 Conclusion
In this paper, we discuss the challenging CMM setting. We first formally define two key objectives for CMM JMMM-platticty and stability- from the perspective of orthogonal projection. Next, we analyze the relationships among the subspaces spanned by task data, gradients, and task vectors. Building on these insights, we propose a data-free Dual Orthogonal Projection (DOP) method for CMM. By reformulating DOP multi-objective optimization problem, our method aims to find Pareto- optimal solutions that effectively balance stability and plasticity. Extensive experiments across three architectures and diverse task configurations validate the effectiveness and efficiency of our DOP.

---

Acknowledgements
This research is supported by A*STAR under its RIE2025 Manufacturing. Trade and Connectiv- ity (MTC) Industry Aligmment Fund- Pre- Pre-Positinning (IAF-PP) (Award M23L4a0001), and by the National Natural Science Foundation of China under Grant (No. 62576083, 62032013, 62025604, 62576364), Shenzhen Basic Research Project (Natural Science Foundation) Basic Research Key Project (NO. JCYJ20241 2AHH122222244334 124430041), the Open Research Fund from Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) (No. GML-KF-24-23).
References
[1] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. re-basn::: Merging models modulo permutation symmetries. In ICLR, 2023.
[2] Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization model merging recipes. arXiv preprint arXiv:2403. 13187, 2024.
[3] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) to forget. In ECCV, volume 11207, pages 144 161, 2018.
[4] Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccias Online continual learning with maximal interfered retrieval. NeurlPS, 32, 2019.
[5] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with network of experts. In CVPR, 2017.
[6] Ansh Arora, Xuanli He, Maximilian Mozes, Srinibas Swain, Mark Dras, and Qiongkai Xu. Here's 's a free lunch: Sanitizing backdoored models with model merge. ACL, 2024.
[7] Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Won Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In CVPR, pages 8218-8227. Computer Vision Foundation / IEEE, 2021.
[8] Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Foo--n01-mining discriminative components with random forests. In ECCV, pages 446-461. Springer, 2014.
[9] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. NewrIPS, 34:22405-224111 2021.
[10] Arslan Chaudhry, Puneet Kumar Dokania, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In ECCV, volume 11215, pages 556-572. Springer, 2018.
[11] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10): 1865-1833, 2017.
[12] Runxi Cheng, Feng Xiong, Yongxian Wei, Wanyun Zhu, and Chun Yuan. Whoever started the interference should end Guiding data-free model merging via task vectors. arXiv preprint arXiv:2503.08099 2025.
[13] WU Chenshen, L Herranz, LIU Xialei, al. Memory replay gans: Learning to generate images from new categories without forgetting, In NeurlPS, 2018.
[14] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yumxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-fimetuneed language models. Journal of Machine Learning Research, 25(7))1 2024.
[15] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, pages 3606-3613, 2014.
10

---

[16] Tarin Clanuwat, Mikel Nober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint drXiv:1812.01118. 2018.
[17] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pages 215-223 JMLR Workshop and Conference Proceedings, 2011.
Andrew
[18] Gregory Cohen, Saced Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist handwritten letters. IJCN,, pages 2921-2926. IEEE, 2017.
[19] Danruo Deng, Guangyong Chen, Jianye Hao, Qiong Wang, and Pheng-Ann Heng. Flattening sharpness for dynamic gradient projection memory benefits continual learning. Advances in Neural Information Processing Systems, 34:18710-153 18700-188111111111111 2021.
[20] Jean-Antoine Désidéri. Multiple-gradient descent algorithm (mgda) for multiobjective opti- mization. Comptes Rendus Mathematique, 350(5-6) 2012.
[21] Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng. Ziyan Wu, and Rama Chellappa. Learning without memorizing. In CVPR, pages 5138-5146. Computer Vision Foundation IEEE, 2019.
[22] Shibhansh Dohare, J Fernando Mermandez-Garrii Qingfeng Lan, Parash Rahman, A Rupam Mahmood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026):768-71)) 2024.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[24] Sebastian Dziadzio, Vishaal Udandarao, Karsten Roth, Ameya Prabhu, Zeynep Akata, Samuel Albanie, and Matthias Bethge. How to merge your multimodal models over time? arXiv preprint arXiv:2412.06712. 2024.
[25] Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà. Task singular vectors: Reducing task interference in model merging, arXiv preprint arXiv:2412.00081, 2024.
[26] Alessandro Giuseppi, Sabato Manfredi, and Antonio Pietrabissa. A weighted average consen- sus approach for decentralized federated learning. Machine Intelligence Research, 19(4)3319- 330, 2022.
[27] Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vladimir Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcees mergekit: A toolkit for merging large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 477-485, 2024.
[28] Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in representation learning: A report on three machine learning contests. In Neural information processing: 20th international conference, ICONIP 2013, daegu, korea, november 3-7, 2013. Proceedings, Part III 20, pages 117-124. Springer, 2013.
[29] DU Guodong, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, al. Parameter competition balancing for model merging. In NeurlPS, 2024.
[30] Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. In ICLR. OpenReview.net, 2020.

---

[31] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land and land cover classification. JSTARS, 12(7):2217-2226, 2019.
[32] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.132699 2023.
[33] Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr- merging: Tuning-free nigh-performance model merging. NewrIPS, 2024.
[34] Shih-Cheng Huang, Pin-Zu Li, Yu-chi Hsu, Kuang-Minn Chen, Yu Tung Lin, Shih- Kai Hsiao, Richard Tsai, and Hung-yl Lee. Chat vector: A simple approach to equip LLMs with instruc- tion following and model alignment in new languages. ACL, 2024.
[35] Gabriel Ilharco. Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Ha- jishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023.
[36] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads wider optima and better generalization. UAI, 2018.
[37] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In ICLR, 2023.
[38] Haeyong Kang, Rusty John Lloyd Mina, Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D Yoo. Forget-free continual learning with winning subnetworks. In ICML, pages 10734-10750. PMLR, 2022.
[39] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.600 2014.
[40] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka maabsk-BBarwinske et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526 2017.
[41] Virginia Klema and Alan Laub. The singular value decomposition: Its computation and some applications. IEEE Transactions on automatic control, 2542):164-176, 1980.
[42] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, pages 554-561, 2013.
[43] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny im- ages. 2009.
[44] Yann LeCun. The mnist database of handwritten digits. http://yannn. lecun. com/exdb/mnist/, 1998.
[45] Chanhyuk Lee, Jiho Choi, Chanryeol Lee, Donggyun Kim, and Seunghoon Hong. Adarank: Adaptive rank pruning for enhanced model merging. arXiv preprint arXiv:2503.22/78, 2025.
[46] Yu-Ang Lee, Ching- Yun Ko. Tejaswini Pedapati, Chung, Mi-Yen Yeh, Pin-Yu Yu Chen, al. Star: Spectral truncation and rescale for model merging, arXiv preprint arXiv:2502.10339, 2025.
[47] Zhizhong Li and Derek Hoiem. Learning without forgetting. TPAMI, 40(12):9335-2947, 2017.
[48] Yan-Shuo Liang and Wu-Jun Li. Adaptive plasticity improvement for continual learning. In CVPR, pages 7816-7825, 2023.
[49] Sen Lin, Yang, Deliang Fan, and Junshan Zhang. Trgp: Trust region gradient projection for continual learning. ICLR, 2022.
12

---

[50] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[51] Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin- merging: Dynamic integration of modular expertise in model merging. NeurlPS, 2024.
[52] Daniel Marczak, Barttomiej Twardowski, Tomasz Trzciński, and Sebastian Cygert. Mag- max: Leveraging model merging for seamless continual learning. ECCV, pages 379-395. Springer, 2025.
[53] Michael S Matena and Colin Raffel. Merging models with fisher-weighted averaging. NeurIPS, 15:17703-17716, 2022.
[54] Mohammed Muqeeth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning route among specialized experts for zero-shot generalization. In Proceedings of the 41st International Conference on Machine Learning, pages 36829-36846, 2024.
[55] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008.
[56] Changdae Oh, Yixuan Li, Kyungwoo Song, Sangdoo Yun, and Dongyoon Han. Dawin: Training-free free dynamic weight interpolation for robust adaptation. arXiv preprint arXiv:2410.03782, 2024.
[57] Guillermo Ortiz-iimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. NeurIPS, 2023.
[58] Oleksiy Ostapenko, Zhan Su, Edoardo Maria Ponti, Laurent Charlin, Nicolas Le Roux, Matheus Pereira, Lucas Caccia, and Alessandro Sordoni. Towards modular Ilms by build- ing and reusing library of loras. ICML, 2024.
[59] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, pages
IEEE, 2012.
[60] Haiquan Qiu, You Wu, and Quanming Yao. Superpose singular features for model merging. arXiv preprint 2rXx::2502.10698, 2025.
[61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763. PMLR, 2021.
[62] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In CVPR, pages 5533-5542. IEEE Computer Society, 2017.
[63] Anthony Robins. Catastrophic forgetting, rehearsal and pscudorchearsal. Connection Science, 7(2):123-146, 1995.
[64] Karsten Roth, Vishaal Udandarao, Sebastian Dziadzio, Ameya Prabhu, Mehdi Cherti, Oriol Vinyals, Olivier Hénaff, Samuel Albanie, Matthias Bethge, and Zeynep Akata. A practi- tioner's guide to continual multimodal pretraining. NewrlPS, 2024.
[65] Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint S0VV:16609.04447,, 2016.
[66] Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick. Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, abs/1606.00671. 2016.
[67] Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learn- ing. ICLR, 2021.
13

---

[68] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. NeurlPS, 33:22045 22055, 2020.
[69] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631-1642, 2013.
[70] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: multi-class classification competition. In IJNN,, pages 1453-1460.
JEEE, 2011.
[71] Derek Tam, Yash Kant, Brian Lester, Igor Gilitschenski, and Colin Raffel. Realistic evalu- ation of model merging for compositional generalization. arXiv preprint arXiv:2409.18314 2024.
[72] Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Fusionbench: A compre- hensive benchmark of deep model fusion. arXiv preprint 24X.v:2406.032880 2024.
[73] Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight-ensembling mixture of experts. ICML, 2024.
[74] Anke Tang, Enneng Yang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Merg- ing models on the fly without retraining: A sequential approach to scalable continual model merging, arXiv preprint arXiv:2501.09522 2025.
[75] A Vaswani. Attention is all you need. NeurIPS, 2017.
[76] Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. MICCAI, pages 210-218. Springer, 2018.
[77] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bow- man. GLUE: A multi-task benehmark and analysis platform for natural language understand- ing. In ICLR. OpenReview.net, 2019.
[78] Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-limenez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. ICML, 2024.
[79] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: theory, method and application. TPAMI, 2024.
[80] Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. arXiv preprint arXiv:2310.14152. 2023.
[81] Zhenyi Wang, Enneng Yang, Li Shen, and Heng Huang. A comprehensive survey of forget- ting in deep learning beyond continual learning. TPAMI, 2024.
[82] Qi Wei, Shuo He, Enneng Yang, Tingcong Liu, Haobo Wang, Lei Feng, and Bo An. Repre- sentation surgery in model merging with probabilistic modeling. In ICML, 2025.
[83] Yongxian Wei, Anke Tang, Li Shen, Feng Xiong, Chun Yuan, and Xiaochun Cao. Mod- eling multi-task model merging as adaptive projective gradient descent. arXiv preprint arXiv:2501.01230. 2025.
[84] T Wolf. Huggingface's transformers: State-of-the the-thh-n natural language processing. arXiv preprint 1100v:191, 2019.
[85] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Korbblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, pages 23965-23998 PMLR, 2022.
14

---

[86] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong. et al. Robust fine-tuning of zero-shot models. In CVPR, pages 7959-7971, 2022.
[87] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mniss a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 07747, 2017.
[88] Jianxiong Xiao, Krista Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring large collection of scene categories. IJV,, 119:3-22, 2016.
[89] Feng Xiong, Runxi Cheng, Wang Chen, Zhanqiu Zhang, Yiwen Guo, Chun Yuan, and Ruifeng Xu. Multi-task model merging via adaptive weight disentanglement. arXiv preprint arXiv:2411.1872) 2024.
[90] Prateek Yadav, Colin Raffel, Mohammed Mugeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model mo- erging: Recycling and routing among specialized experts for collaborative learning. arXiv preprint arXiv:2408.070557 2024.
[91] Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving interference when merging models. NeurlPS, 2023.
[92] Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale? arXiv preprint
2024.
[93] Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang. Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in lims, milms, and beyond: Methods, theories, applications and oppor- tunities. arXiv preprint arXivv2408.07666, 2024.
[94] Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. ICMI., 2024.
[95] Enneng Yang, Li Shen, Zhenyi Wang, Shiwei Liu, Guibing Guo, and Xingwei Wang. Data augmented flatness-aware gradient projection for continual learning. In /CCV, 2023.
[96] Enneng Yang, Li Shen, Zhenyi Wang, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Revisiting flatness-aware optimization in continual learning with orthogonal gradient projection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
[97] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. ICIR, 2024.
[98] Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. ICML, 2024.
[99] Netzer Yuval. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop, 2011.
[100] Guanxiong Zeng, Yang Chen, Bo Cui, and Shan Yu. Continual learning of context-dependent processing in neural networks. Nature Machine Intelligence, 188):364-31) 337,, 2019.
[101] Jinghuai Zhang, Jianfeng Chi, Zheng Li, Kunlin Cai, Yang Zhang, and Yuan Tian. Badmerg- ing: Backdoor attacks against model merging. In CCS, 2024.
[102] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint 23Xi::2303.18223 2023.
[103] Hongling Zheng, Li Shen, Anke Tang, Yong Luo, Han Hu, Bo Du, Yonggang Wen, and Dacheng Tao. Learning from models beyond fine-tuning, Nature Machine Intelligence, pages 1-12, 2025.
[104] Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Tlas-inncrmmentll learning: A survey. TPAMI, 2024.
15

---

[105] Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. Metagpt: Merging large language models using model exclusive task arithmetic. EMNLP. 2024.
[106] Didi Zhu, Zhongyi Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Kun Kuang, and Chao Wu. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. ICML, 2024.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract we present the contributions, and in the introduction we also clearly summarize the three contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer:[Yes)
Justification:
We discuss the limitations existing in this paper in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations their approach ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
16

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We provide proofs of the main theory or hypothesis of this paper in Section
4.1.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the of the paper should be comple- mented by formal proof's provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: We provide the main implementation details in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, deseribing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
17

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All our experimental datasets and weight files are sourced from HuggingFace, which can be downloaded by everyone.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https: //ni ps..... ce/ cesioPoolccy for more details.
public/ /wwidggusssseeeeeieeeee / CoddeSuum
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //n //nips.oo.. cc/
ctappscccpubllii public gu i des (CodSSubmi as i conPolccy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the experimental setting in Section 5, and the main implementa- tion details in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question:
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer:[Yes
Justification: Our main experimental results include the average and standard deviation of ten experiments.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the computational resources in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eerippp.eee
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and our work complies with all stated guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
19

---

Justification:
We discuss the broader impacts of this paper in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release any models or data with known risks of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We provide references all models and data.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
20

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswitheoe com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
No new assets are released in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects or crowdsourcing experiments are involved.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No research involving human subjects is performed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMss if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our research does not use L.MSs as component of our core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //nurrisss.pp///// ctt /acc/ofeeeeeecs///// /2055/ LLM) for what should or should not described.
22

---

Appendix Overview.
The content of this appendix is organized as follows:
In Appendix A, we discuss the limitations and broader impact of this work.
In Appendix B, we discuss the research progress on model merging and continual learning in detail.
In Appendix C, we provide details on the dataset configurations and implementation of our exper- iment.
In Appendix D, we present the core formulas used by existing baselines when performing contin- ual model merging.
In Appendix E, we provide the complete pseudocode of the proposed data-free dual orthogonal projection (DOP) method.
In Appendix F, we present additional experimental results and analyses that could not be included in the main text due to space constraints.
A Limitations and Broader Impact
Limitations. Although this work demonstrates promising stability and plasticity under the contin- ual model merging setting, there remain several limitations and potential directions for future ex- ploration. First, this paper focuses the most extreme setting of continual model merging, where no data is available. Future work could investigate the impact of incorporating a small amount of data during continual merging. Second, primarily evaluate the proposed DOP method on vision and language models, lacking validation on larger-scale models and data of other modalities. In the future, aim to extend continual model merging to large-scale multimodal settings.
Broader Impact. In this paper, we propose a novel approach for continual model merging. holds the potential to enable continual accumulation of new knowledge without forgetting previous knowledge-an ability that serves cornerstone toward more advanced artificial intelligence, even in the absence of data. Moreover, this continual model merging setup naturally supports decentral- ized development: each user can locally train their own expert model, and these models can then be sequentially merged into unified model without sharing data. As such, continual model merging offers an orthogonal and complementary alternative to traditional continual learning.
B Related Work
In this work, research objective is to explore how merge sequence of expert models without forgetting the knowledge accumulated from previously merged models. This topic is related to both traditional model merging and continual learning, yet it is fundamentally distinct from both. this section, we provide a detailed discussion on these distinctions.
B.1 Model Merging
Model merging aims to integrate multiple models with diverse capabilities into unified model at the "parameter level" [103, 93, 71, 26, 27, 85, 35, 2]. Based on the merging strategy, existing methods can be broadly categorized into static model merging and dynamic model merging. The former can be further divided into weight-basse based merging and subspace-bart based merging approaches.
Weight-based merging methods aim to assign appropriate merging coefficients to balance the contri- butions of different expert models to the final merged model [105, 97]. Using average weighting is the most straightforward and simple approach [30], but its performance is limited, especially when the expert models differ significantly. Task Arithmetic [35] adopts grid search, while Evolutionary model merge [2] and LoRAHub [32] employ evolutionary algorithms to search for optimal merg- ing coefficients. However, such approaches become inefficient when the number models is large. Some studies have proposed assigning merging coefficients based on statistical information leamn able strategies. For example, Mereing [53] uses the Fisher Information Matrix to measure parameter importance and assigns weights accordingly. RegMean [37] derives closed-form solu- tion for merging linear layers based on training data. AdaMerging [97] introduces an unsupervised loss based on entropy minimization optimize layer-wise merging coefficients.
Subspace-based merging methods project models into sparse or low-rank subspace to alleviate parameter interference during merging. Ties-Merging [91] removes the less significant components
23

---

of task vectors and aligns parameter signs before merging. DARE [98] randomly drops proportion of elements in the task vectors and rescales the remaining parameters. TALL-masks [78] maintains a sparse mask matrix for each task to isolate conflicting parameters. Model Tailor [106] measures parameter importance based on loss sensitivity and removes unimportant parameters, though it relies on data availability. PCB- PCB-Merging [29] carefully evaluates the importance of parameters both within and across tasks to prune unimportant ones. TSV-M [25], STAR [46], SFTM [60], and AdaRank [45] perform singular value decomposition (SVD) on the weight matrices and merge expert models only within the low-rank subspace spanned by the most significant singular directions.
Dynamic merging methods differ from the approaches above in that they construct different merged models for each individual sample or task, thereby increasing the flexibility of model merging. For example, WEMoE [73] dynamically combines the multilayer perceptron modules of expert models through a learnable routing mechanism. Twin-Merging [51] and PHATGOOSE [54] dynamically compose series of low-rank expert modules. DAWIN [56] merges the parameters of two models based on their predicted probabilities. However, due to the need to perform merging before each inference, such dynamic approaches typically incur significant latency during prediction.
Some other model merging methods operate during the fine-tuning stage by disentangling weights [57], aligning parameters before merging [1], or correcting representation bias after merg- ing [94, 82]. In particular, our work shares some conceptual relevance with WUDI-Merging [12], as both leverage the correlation between task vectors and data in linear layers. However, there are key differences in the problem setting. WUDI-Merging assumes that all models are available simultane- ously and focuses on resolving task conficts. In contrast, our method assumes sequential arrival of models and emphasizes balancing the sability-plasticii trade-off between new and old tasks. We also propose multi-objeccivv optimization strategy to dynamically adjust the combination coeffi cients for the two objectives.
Continual Model Merging (CMM). In summary, the aforementioned model merging methods all assume that expert models are available simultaneously, overlooking the practical scenario where expert models may arrive sequentially. While TIME [24] and OPCM [74] consider this setting, TIME primarily an vvaluation-focusee work, and OPCM does not explicitly address or tailor its design for the stability-plastiiity trade-off, which limits its performance in continual merging scenarios. contrast, this paper is the first to formulate stability and plasticity as dual objectives in the continual model merging setting from the perspective of orthogonal projection, and proposes a targeted data-free optimization scheme, effectively filling this gap in the literature.
Application Scenarios of CMM. The practical scenarios for continual model merging include, but are not limited to: (i) cemmor--effciien model merging: Traditional model merging requires loading all candidate models simultaneously, which is often infeasible in resource-constrained environments. CMM provides a lower-cost and more scalable alternative. (ii) Continual model reuse: When users wish to further enhance a composite model published by others (as in the HuggingFace example) by merging their own private models, CMM enables continual capability enhancement without re- quiring access to the original private models of previous users. (iii) Improved model generalization: Standard training often leads to convergence in sharp regions of the loss landscape, resulting poor generalization. Continual merging of checkpoints can guide models toward flatter minima, thereby improving generalization [9, 30].
B.2 Continual Learning
The goal of continual learning is to enable deep neural network to continuously acquire knowledge from new tasks without forgetting the knowledge learned from previous tasks, thereby achieving a human-like ability for lifelong learning [104, 79, 81]. To mitigate catastrophic forgetting of old tasks, existing approaches can be broadly categorized into replay-based methods, architecture-based methods, regularization- bbased methods, and orthogonal projection methods.
based methods alleviate forgetting by replaying small buffer of old task data while learning new tasks. The buffer can be filled by randomly sampling from previous data or by using various heuristic strategies. For example, iCaRL [62] selects samples closest to the class centers for replay, while RWalk [10] and MIR [4] select samples near the decision boundaries. Rainbow [7] emphasizes maximizing the diversity of replayed samples. Some other approaches use generative models to synthesize data from previous tasks for replay [13].
24

---

Architecture- beaheeecture-bbsedd methods learn new tasks in sparse subspaces or progressively expand the network by adding new parameter modules accommodate incoming tasks. For example, WSN [38] selects a winning subnetwork for each specific task based on the lottery ticket hypothesis. PNN [66] creates a separate branch for each task, while ExpertGate [5] adds a new expert module for every new task.
Regularization-ban based methods adopt the idea of knowledge distillation, treating the model trained on new tasks as the student and the model from previous tasks as the teacher, supervise the student and prevent forgetting. LwF [47] uses the teacher models output as an additional training signal for the student model, while LwM [21] leverages attention maps from previous tasks as supervision. EWC [40] and MAS [3], on the other hand, constrain the parameter changes between the new model and the previous model preserve prior knowledge.
Orthogonal projection- methods update gradients in directions orthogonal to the subspace oc- cupied by previous tasks, thereby preserving prior knowledge during new task learning [80]. For example, OWM [100] constructs the subspace of previous tasks based on historical gradients, while GPM [67] and DFGP [95, 96] build the old task subspace using feature representations from histori- cal data across spaces. FS-DGPM [19] and TRGP [49] further refine this idea by scaling each basis vector in the old task subspace according its importance.
However, all the aforementioned continual learning methods are data-driven, meaning they rely on continuously training the model on data from each task to obtain unified model. As such, they are not applicable to our setting. In this work, we obtain a unified model by sequentially merging series of expert models [24, 74], without directly accessing the original training data of each task. This novel approach (moeel-driven) offers an orthogonal perspective to traditional continual learning paradigms (data-driven).
C Experiment Details
Vision Tasks: For image classification tasks, we fine-tuned three architectures on 20 datasets, con- sistent with OPCM [74]. to obtain task-specific expert models for merging. For each architecture (Clip- BBT--V ViT-B/16 or Clip-ViT-L/I [61, 23I). we consider three different task con- figurations: (1) 8 Tasks: SUN397 [88], Cars [42], RESISC45 [11], EuroSAT [31]. SVHN [99].
GTSRB [70], MNIST [44], DTD [15]. (2) 14 Tasks: In addition to the 8 tasks above, the fol- lowing tasks are included: Flowers [55], PCAM [76], FER2013 [28], OxfordlITPPet [59], STLI0 [17]. CIFAR100 [43]. (3) 20 Tasks: Building on the 14 tasks above, the following 6 tasks are added: CIFARIO [43], Food101 [8], FashionMNIST [87], EMNIST [18]. KMNIST [16], and RenderedSST2 [69, 61J. These datasets cover diverse range of task types, including digit recog- nition, satellite image classification, and natural image classification, among others. We fine-tuned using cross-entropy loss and Adam with cosine annealing (max learning rate: 1e-5, batch size: 128) for 4,000 steps as Fusionbench [72].
Language Tasks: For natural language processing tasks, we select 8 text-to-text generation tasks from the GLUE benchmark, including CoLA, MNLI, MRPC, QNLI. QQP, RTE, SST-2, and STSB. Following the prompt settings in FusionBench [72], we fine-tune the Flan-T5-Base [14] model on each task build expert models. During evaluation, we report Spearmans p for the STSB task, and exact match accuracy for the other seven tasks. Specifically, for the STSB task, if the textual output cannot be parsed into a numerical value, we set p to O.
Implementation Details: For all baseline methods, we employed the same hyperparameter search strategy as OPCM [74]. For our proposed method, we utilized the Adam optimizer with an initial learning rate of 1e-4 and set the number of iterations to 200 to optimize the DOP objectives. The initial value of the weight in MGDA is set to 0.8. For the EMA smoothing strategy, we set / to 0.999, All vision model experiments were conducted on single NVIDIA RTX A6000 GPU (48 GB), while all language model experiments were conducted on single NVIDIA V100 GPU (32 GB). All results are averaged over ten runs, and both the mean and standard deviation are reported.
D Existing CMM Methods
To the best of our knowledge, nearly all existing model merging methods are designed for multi- task model merging (i.c., assuming simultancous availability of all models), with only few studies
25

---

addressing CMM [24, 74]. Among the comparison methods, TIME [24] functions as an evaluation pipeline, whereas OPCM constitutes a state-of-the-an CMM algorithm [74]. To facilitate a compre- hensive comparison with our approach, we further adapted Stochastic Weight Averaging (SWA) [36],
Task Arithmetic [35], and Ties-Merging [91] to the continual model merging setting as additional baselines for comparison. The core operations of these methods are summarized as follows.
Stochastic Weight Averaging (SWA) [36, 85]:
o - 1)
2 ,0,
(4)
2
Continual Task Arithmetic [35]:
θ (t) = Oo + (T + +1)12 22,,
(5)
where "o(t-1> = (((1))
O. that is, the task vector constructed by the merged model of step (-1 and the pre-trained model. À is hyperparameter control the importance of the task vectors when merging them with the pre-trained model.
Continual Ties-Merging [91]:
1 = Oo A- ft), 2 2,
(6)
where fett-iy ,y and ft represent the task vectors (g((-1) and after resolving sign conflicts using the "Trim" and "Elect Sign" strategies from original Ties-Megging [91].
Orthogonal Projection-based Continual Merging (OPCM) [74]:
(
() - (p((-1) (Te)
O = Oo
.122
(7)
X(9
where p(t-1)(TE) is a projection operation that projects T to the orthogonal directions of the task vector spanning the subspace at step 1-
E Pseudo-code of the DOP Algorithm
As shown in Alg. I, we provide the complete pseudocode for the DOP method. The merging process in our algorithm consists of two main components: DOP optimization for linear layers (lines 3-19) and simple parameter averaging strategy for non-linear layers (lines 21-23). This step option- ally be replaced with any existing model merging strategy. It is worth noting that the optimization of linear layers can be executed in parallel across all layers.
More specifically, the core steps of DOP are as follows: (1) Lines 3-7: Perform SVD on the task vectors to construct the core subspaces.
(2) Line 8: Use simple parameter averaging to initialize the merged model. (3) Lines 10-19: Optimize the merged linear layers, which involves computing the stability and plasticity losses (lines 12-13). calculating the gradient descent direction based on multi-objective optimization (lines 14-17), and updating the parameters via gradient descent (line 18). Specifically, getW () and setW () denote the operations for retrieving and updating the layer weights, respectively.
F
Experimental Results and Analysis
Architecture and Parameter Distribution Analysis: The Transformer [75] architecture has emerged as one of the most widely adopted frameworks in machine learning, demonstrating remark- able performance in both vision [23] and natural language processing tasks [75]. Transformer-baser models are constructed by stacking multiple Transformer blocks. Fig. 4 (a) illustrates the standard architecture of a Transformer block, comprising multi-head attention layers, normalization layers, and -orward layers. Notably, the fundamental components of the multi-head attention and feed- forward layers are linear layers. As shown in Fig. 4 (b), linear layers constitute the majority of the models parameters. For example, in Flan-T5-Baad Base, ViT-B/32, ViT-B/16, and ViT-L/14, linear layers account for 90.01% % 97.21 %, 99.09 99.09% and 99.68% of the total parameters, respectively. Thus, the key to merging Transformer-bast based models lies in effectively integrating their linear layers.
26

---

Algorithm 1: Dual Orthogonal Projection for Continual Model Merging
Input: pre-trained model Ou previously merged model Oota, model nnww hlpper-aaametenst %, K, r, Output: merged model
for layer 1 in O or Ooid or neww do
1/ Linear layer optimi zation can be performed in parallel across all layers.
if layer l Linear layer then
11 Construct subspace using SVD
Wo, Wotd, Wnew f getw(Oo, Sata Onew,!); Told + Wold Wo:
Tnew Wnew Wo;
U.E.VO + (Taaa)
U.E.V, V.T (Tdr(Tnew):
1/ Initialize the nerged parameters and iterate the optimization
W. +- (Wold + Wnew) /2; for step & in (1, K do
/1 optimize stability and plasticity
AW old f W. Wotsi
AW new W.
W newws
12
L. 6 Lstabiiity Wa Uo Fo. Vo);
13
Cp + Eplasticity newr Un. Fm, V.));
// Calculate the multii-objetthee
balance coefficient
(Fw,, fw-V,,ww L.) V.,
a + clip
VW., VV- as + 8. + 8) .az
15
Op, & 3.a
B)
(1 al);
// Calculate the gradient descent direction and perform the update
17
9i f aad Vw. C. ap. Cp. W. W., 1) 99:
19
setw(O.,1) - W..Ki
20
// Perform simple parameter averaging for nonl inear layers Wold, Wnew f- gett(OOdd Onew, W., +- (Wold + /2; setW(O.,1) W.,o;
21
24 return O.;
Stability and Plasticity Analysis: In this section, we analyze how the performance of different CMM methods evolved over time under fixed model arrival orders. Fig. 5 and Fig, 6 illustrate the results of merging vision tasks under two specific task orders. Specifically, the task order DED-CCSA--DD--NN-N--C Car.--r--G..RB.... while the order in Fig. 6 is OTD-SVHN-GTSRE SEEHENGGTSR--MIISS-SSUNNNE MNIST-SUN3977 AT RESTEESSSS proposed DOP method consistently demonstrates stronger plasticity and stability under both task orders. For example, in the first task order, when merging the eighth model, DOP achieved a per- formance of 68.0, whereas OPCM, Task Arithmetic, and SWA achieved only 63.9, 55.7, and 62.4, respectively-indica.. a performance of 94.1 on the second task, compared to 84.0, 77.9, and 64.5 for OPCM. C. Task Arithmetic, and SWA, respectivelydemonstrattte we also evaluate two random task arrival orders for the language tasks. Fig. 7 corresponds to the order: WWWTE-SSTT-COLL--NLLL-EEEEE
in Fig. 5 is MNIST-SUN397 SESS--SUU377---USAT----
We observe that the
superior plasticity of our method. At the same time, DOP maintains
better stability (i.c., less forgetting). Similarly,
STSBB, and Fig. 8 corresponds to: STSB- Consistently, we observe that our proposed method demonstrates superior plasticity and stability under both task orders.
Single-task Performance Analysis: For visual classification tasks, Fig. and Fig. 10 depict performance of the merged model on individual tasks when sequentially merging 8, 14, and 20 tasks using the VIT-B/32 and ViT-I/14 architectures. The performance each task is normalized relative to its single fine-tuned model, with values closer to I indicating better performance. These
27

---

(a) Traansoomeer Encoder
(b) Number and ratio statistics of parameters
Linear
FC2
Architecture
VIT-B-32
VTT-B/16
x
Add & Norm
Activation
Total Parameters
87,455,232
85,798,656
Feed Forward (Linear)
Linear
FC1
Lincar Parameters
85,017,600
85,017,600
Linear Ratio
97.21%
99.09%
Linear
Out proj
Add & Norm
Concat
Architecture
ViT-L/14
Flun-TS-Buse
Multi-Heed
Sealed dot-product attention
Total Parameters
303,178,752
247,577.856
Attention (Linear)
Lincar Paramoters
302,211,072
222,855,168
Lincar
Linear
Linear
Input Patchs
to
f
Lincar Ratio
99.68%
90.01%
Figure 4: The total number of parameters and the statistics of linear layer parameters for three ViT architectures. (a) is a general Transformer block [75], including multi-head attention, feed- forward layers, layer normalization, etc. The primary components within the Transformer block are linear modules. (b) presents the statistics on the total parameters, linear layer parameters, and the parameters of linear layer weights (excluding bias terms) for the three vision architectures (i.e., ViT-B/32, ViT-B/16, ViT-L/14,, and Flan-T5- ---ase.
Ceerer
Circury Carcole
Criig
HMCT-
MU5T
14477
H4.07
$43007-
12:00/3
102 0000
S/R
70
70
Can
12
(a) SWA
(b) C. Task Arithmetic
(c) OPCM
(d) Ours
Figure 5: Performance changes during the merging of 8 tasks with the ViT-B/32 architecture. Row
t shows test accuracy on all previous tasks t) after merging the t-th model (stability), while
diagonal values indicate the model's plasticity, reflecting its performance on the newly merged task.
figures clearly show that the proposed DOP method consistently appears on the outermost ring. indicating superior performance on individual tasks compared to other CMM methods. For natural language processing tasks, Table presents the performance of various model merging methods as well as independently fine-tuned expert models. We observe that the proposed DOP method achieves performance on individual tasks that is closer to the expert models compared to other continual model merging approaches.
Projection Space Analysis: As detailed in $4.2, we ap- proximate the input subspace using the subspace spanned by task vectors. In practice, this subspace obtained via SVD decomposition of the task vectors. We analyze the performance variations when employing only the row
Table 5: Analysis of subspace projec- tion strategy
Methods
Space
Space
ACC (f)
Sid (1)
w/o V
3
63,3
3.06
U
75.9
space (U), only the column space (V), or both spaces for projection. As presented in Tab. 5, utilizing both sub-
DOP
/
78.2
1.63
spaces preserves more spatial information and yields the best performance. In contrast, employing only the row or column space leads to suboptimal performance.
Linear Module Analysis: In the main text, our model merges all linear modules using the proposed DOP scheme. In this section, we validate the effectiveness of applying DOP to only subset of the linear layers. We pri- marily compare against the state-of-the-an art CMM method, OPCM [74]. To this end, we conducted the following ab- lation experiments: (i) None: Neither OPCM nor DOP is
Table 6: Performance impact analysis of merging different linear layers
Methods
OPCM
DOP (Ours)
(i) None
66.3+0.0
66.3:00.0
(ii) Q/K/V Projection
66.6.0.11
72.911..8
(iii) FEN
75.2=0.4
75.8:2.00
(iv) ALL
75.5:0..
78.3:11.6
applied to any linear layer. (ii) Q/K/V Projection: OPCM
and DOP are applied only to the Q/K/V projection layers. (iii) FFN: OPCM and DOP are applied
only to the FFN layers. (iv) ALL: OPCM and DOP are applied to all linear layers. As shown in Tab. 6, we observe that DOP consistently outperforms OPCM. Notably, the Q/K/V Projection
28

---

vectol
crbee
2013
1955
10
samon
a05000
(a) SWA
(b) C. Task Arithmetic
(c) OPCM
(d) Ours
Figure 6: Performance changes during the merging of 8 tasks with the ViT-B/32 architecture. Row t shows test accuracy on all previous tasks (1 ()- after merging the 1-th model (stability), while
diagonal values indicate the model's plasticity, reflecting its performance the newly merged task.
500 -133
70
and
50
C00
(a) SWA
(b) C. Task Arithmetic
(c) OPCM
(d) Ours
architecture. Row shows test accuracy on all previous tasks (1-t) after merging the t-th model (stability), while diagonal values indicate the model's s plasticity, reflecting its performance on the newly merged task.
Figure 7: Performance changes during the merging of 8 tasks with the
layers, DOP achieves an accuracy of 72.9, compared to 66.6 for OPCM. Furthermore, when opti- mizing all linear layers, DOP achieves an accuracy of 78.3, surpassing OPCM's 75.5. These results demonstrate that DOP consistently outperforms OPCM across different linear layers.
Nonlinear Module Analysis: the main text, we adopt simple weight averaging [85] the default strategy for handling nonlinear layers. In this part, we further ex- plored additional strategies for merging nonlinear layers, such as Task Arithmetic [35]. As shown in the Tab. 7, we observe that the performance differences between these two approaches are minor, and both significantly outper- form baselines such as SWA and OPCM. This is mainly because the parameters of nonlinear layers constitute only a small proportion of the entire model.
Table 7:
Analysis of strategies for merg- ing nonlinear layers
Methods
ACC
Averaging (SWA)
66.310.00
OPCM
75.5=0.5
Our DOP (Weight Averaging) Our DOP (Task Arithmetic)
78.3+11.6
78.9+11.3
As shown in Fig. 11, we visualize the optimization trajectory of the stability loss (blue line) and plasticity loss (red line) in the DOP objective across different model merging stages (e.g., stage denotes merging the 8-th incoming model with the previously merged 7 models) and various linear modules (Q/K/V/Out _roj layers in the attention module and FC1/FC2 layers in the feed-forward module). Consistently, both losses exhibit a steady downward trend across all layers and stages, ultimately converging smoothly. This highlights the robust optimization stability of the DOP method.
Convergence Analysis:
29

---

100
100
100
100
-00
-00
100
-00
240
rea -I7A 020
NJ
164
554 HEV
(a) SWA
(b) C. Task Arithmetic
(c) OPCM
(d) Ours
Figure 8: Performance changes during the merging of 8 tasks with the Flan-T5-base architecture. Row i shows test accuracy on all previous tasks (1-f) after merging the 1-th model (stability), while diagonal values indicate the model's plasticity, reflecting its performance the newly merged task.
SWNV
EUTSAT
OTO
EXTSKT
IGM
0.2
DT3
SUN197
R100
PCAM
57110
(a) V/T-B/32: Tasks
(b) VIT-B/32: Tasks
VIT-B/32: Tasks
Figure 9: Performance of the final model on all tasks when continually merging different numbers of tasks (8/14/20) using the ViT-B/32 architecture. Each task's ' accuracy is normalized by its expert model. All results are averaged over ten randomly shuffled task orders.
RESISC4S
SVHN
EuroSAT
NI
SAMI
EUrOSAT
GTSRB
RES
BANACIN
MNIST
GUR397
WHOSE
Dobed mi
STL10
CD8230
CNESTT
MNIT
201 560
Tax Artfeett
DPC
0m
5th
Tex Moveak
DPdP
Daw. SWN
0NCH
(a) VIT-L/14: 8 Tasks
(b) VIT-L/14: 14 Tasks
VIT-L/14: 20 Tasks
Figure 10: Performance of the final model all tasks when continually merging different numbers of tasks (8/14/20) using the ViT-L/14 architecture. Each task't 's accuracy is normalized by its expert model. All results averaged over ten randomly shuffled task orders.
30

---

Sage
Stage3
Sage-
Zuged
Suga
Stage
Stage
0.0010-
Laa
Suailly Lnn
Los
Sen by low
By
0.0005
5
0.0000
200
200
1:00
200
100
100
100
Step
Sten
Sxe
(a) "vision_ crooeel.enoodr.layerss. S.self a.tp....prrj"""
Maye
Map-1
Wapp-
Stage
Moor
Loss
sualliyy Lass
Stuality by Loke
dy Lous
saas By Lass
da Lass
Srakdiyy Loss
0.0000
Skep
(b) "vision modell.nccodrr.lyem O.self attn.k__ttttt proj
Mage
Mapp-1
Shage
Moor
0.0002
Loas
suaility Lass
Siuaiity Los
dy Loss
By Less
Bi
Stakliyy Less
0.0001
100
1000
100
(c) "vision_ "vision_modell.co wondel.encoder.layrrre selef www.efattttttn attm v v_proj"
Stage
Stage
Stac By Los Plasccyy Los
Subii to loar "aaticyy Loaa
Start Bl Lan Fistticly Loua
Suaiiiy Join Neatiiity Lasa
Stubi by Lose eatt
Stabilty das Flaitdly Las
Stabiity Lom
flarr
2
100
200
0
100 step
200
100 sop
200
100 Strp
200
1000 stop
200
120 Soop
200
1.00
200
Sax
(d) "vision www.ee..ncoddr.laerr.s..o auttnoout_t Baciioooto__rr___ proj"
Maic?
Magn
Winee
Stage
0.002
Naalcity Laye
Peatlcts Las
Pralblyy Lave
Nistchh Lav
Picatiitty
Pest
5
0.000
100
100
100
200
200
100
260
100
Strp
S0op
Step
Shep
(c) "vision_ "vision_model.cncco www.mmeeeeccddrrllrrr..oo.mommm
Mige
Magel
Wago-1
Shoge
Moor
0.0010
Loas
suailty Las
tonilt--une Loka
By Lous
Bo
Staliiyy Lau
0.0003
200
120
100
100
Simp
1kmp
9.x
See
(1) "vision mooeel.encodderllaers.......
Figure 11: Visualization of the stability and plasticity losses during the optimization process across different linear modules (as shown Fig. 4(a)) within the first Transformer block of the ViT-B/32 architecture. (a) Q in the Multi-Head Attention module, (b) K in the Multi-Head Attention module, (c) V in the Multi-Head Attention module, (d) Out Proj in the Multi-Head Attention module, (e) FCI in the Feed-Forward module, (f) FC2 in the Feed-Forward module. Here, Stage t denotes merging the old model with the t-th new task model. We observe that both stability and plasticity losses are low (check the Y-axis scale for reference).
31


Reviews:
[{"review_id": "qYs4vGaPR6", "reviewer": "Reviewer_munj", "summary": "This paper addresses the problem of continual model merging (CMM), where models are merged incrementally without access to original training data. The authors propose Dual Orthogonal Projection (DOP), a data-free method that balances stability (retaining old task knowledge) and plasticity (adapting to new tasks) through multi-objective optimization. Key contributions include formalizing stability and plasticity through orthogonal projection theory, analyzing subspace relationships, and introducing DOP with MGDA optimization.", "questions": "Questions\n1. OPCM is also a data-free method based on orthogonal projections. Are there additional advantages of DOP beyond the use of MGDA for adaptively balancing the weights between old and new tasks?\n2. In the Tab.4 of ablation studies for projection space, it seems that the orthogonal constraints in row space and column space have huge differences (63.3 v.s. 75.9). Could authors explain this phenomenon?\n3. Fig.3(d) highlights the importance of the coefficient alpha. Could the authors visualize the solved alpha during optimization and give more insights into the model merging process?\n4. Linear components mainly reside in embedding layers, Q/K/V projection layers, and FFN layers. Does DOP demonstrate varying degrees of improvement in these three parts compared to SWA and OPCM?", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. This work proposes a novel perspective of the relationships among the subspaces spanned by task data, gradients, and task vectors.\n2. DOP is a data-free method, which is both memory- and time-efficient.\n3. Extensive experiments on vision and language tasks demonstrate superior performance over SOTA methods.\nWeaknesses\n1. Non-linear components are handled via simple averaging, while different linear layers are not differentiated, which may still leave room for further optimization.", "was_revised": true, "final_justification": "The authors' response sufficiently answered my questions. I will keep my positive rating.", "rebuttal": "Thank you very much for your review and affirmation of our work. We'll answer your questions one by one below.\n\n## Response to Weakness 1:  About \"Nonlinear Layer Processing Methods\"\nThank you for your suggestion regarding the treatment of nonlinear layers. In the main text, we have by default adopted simple weight averaging for handling nonlinear layers. Following your advice, we further explored additional strategies for merging nonlinear layers, including Weight Averaging and Task Arithmetic[1]. As shown in the table below, we observe that **the performance differences between these two approaches are minor**, and both significantly outperform baselines such as SWA and OPCM. This is mainly because the parameters of nonlinear layers constitute only a small proportion of the entire model.\n\n| Method | ACC |  \n| :------ | :------: |  \n| SWA | 66.3 $\\pm$ 0.0 | \n| OPCM | 75.5 $\\pm$ 0.5 | \n|  | |   \n| Our DOP (Weight Averaging for NonLinear)  | 78.3 $\\pm$ 1.6 | \n| Our DOP (Task Arithmetic for NonLinear)  | 78.9 $\\pm$ 1.3 | \n\n[1] Editing Models with Task Arithmetic. ICLR, 2023.\n\n\n## Response to Question 1: About \"Advantages of our DOP Compared to OPCM\"\nCompared to OPCM, the proposed DOP method demonstrates clear advantages in several aspects:\n- **Theoretical Analysis**: DOP provides a more comprehensive theoretical framework by analyzing the relationships among data, gradients, and task vectors in the context of continual model merging (CMM). Through this analysis, DOP explicitly defines two key objectives—stability and plasticity—from the perspective of orthogonal projection. In contrast, OPCM does not formally define these objectives nor provide such theoretical analysis.\n- **Stability-Plasticity Trade-off**: DOP simultaneously optimizes both stability and plasticity by formulating the problem as a multi-objective optimization, thus achieving a better balance between adapting to new tasks (plasticity) and retaining knowledge from previous tasks (stability). OPCM, however, focuses only on stability, aiming to minimize interference from new tasks to old knowledge, and overlooks the important aspect of plasticity.\n- **Empirical Performance**: Experimental results across multiple vision and language tasks, four model architectures, and three task scales consistently show that DOP outperforms OPCM, achieving higher average performance and less forgetting.\n\nIn summary, **DOP not only offers a more rigorous theoretical foundation and methodological soundness for continual model merging, but also demonstrates superior empirical performance in practical applications**.\n\n\n## Response to Question 2: About \"Differences in Projection Space Construction\"\nThank you very much for your insightful question. We believe that the row space and column space possess distinct geometric properties in capturing the structure and variations of task data. Specifically, the row space primarily reflects the variation of task vectors in the input data (feature space), while the column space directly reveals the variation of task vectors in the model parameters (output space). As discussed in our main text and supported by previous works on orthogonal projection, traditional orthogonal projection operations in continual learning are typically performed in the input space [1,2], which aligns with our results. Therefore, compared to the output space, projection in the input space may be more effective, as it better captures critical information from the input data and thus impacts the overall model performance. Furthermore, simultaneously capturing the characteristics of both input and output spaces is superior to considering either alone.\n\n[1] Gradient projection memory for continual learning. ICLR, 2021.\n\n[2] TRGP: Trust region gradient projection for continual learning. ICLR, 2022.\n\n\n## Response to Question 3: About \"Visualization of Coefficient Alpha\"\nIn our method, the coefficients $\\alpha = \\{\\alpha_{s,k}, \\alpha_{p,k}\\}$ are adaptively updated across different layers, parameter modules, and merging stages. To further analyze the dynamic evolution of $\\alpha$ during optimization, we visualize the values of $\\{\\alpha_{s,k}, \\alpha_{p,k}\\}$ at various training steps.\n\nFor illustration, we randomly select three linear modules from three different stages (Stage 2 - vision_model.encoder.layers.11.self_attn.k_proj, Stage 4 - vision_model.encoder.layers.0.self_attn.k_proj, and Stage 8 - vision_model.encoder.layers.11.self_attn.v_proj), and record the values of $\\alpha$ at different training steps ($k=0,25,50,75,100,125,150,175,200$). As shown in the following tables, we consistently observe that the stability coefficient $\\alpha_{s,k}$ gradually decreases from the initial value of 0.8, while the plasticity coefficient $\\alpha_{p,k}$ increases from 0.2 to around 0.3. This trend reflects that, in the early stages, the model tends to retain more knowledge from the merged model, and as training progresses, it gradually incorporates new knowledge, resulting in a decrease in $\\alpha_{s,k}$ and an increase in $\\alpha_{p,k}$.\n\nWe will provide detailed plots of the evolution of $\\alpha = \\{\\alpha_{s,k}, \\alpha_{p,k}\\}$ at each iteration step in the revised version.\n\nStage 2 - vision_model.encoder.layers.11.self_attn.k_proj:\n| Iteration (k) | 0 | 25 | 50 | 75 | 100 | 125  | 150 | 175  | 200  |\n| :------ | :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |   \n| $\\alpha_{s,k}$ | 0.80  |  0.79  |  0.78  | 0.78  |  0.77 | 0.77  | 0.76  |  0.76  |  0.75 | \n| $\\alpha_{p,k}$ | 0.20  |  0.21   | 0.22   |  0.22 |   0.23 | 0.23  | 0.24  |  0.24  |  0.25 | \n\nStage 4 - vision_model.encoder.layers.0.self_attn.k_proj:\n| Iteration (k) | 0 | 25 | 50 | 75 | 100 | 125  | 150 | 175  | 200  |\n| :------ | :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |   \n| $\\alpha_{s,k}$ | 0.80  |  0.78  |  0.77  | 0.75  |  0.73 | 0.72  | 0.70  |  0.69  |  0.67 | \n| $\\alpha_{p,k}$ | 0.20  |  0.22   | 0.23   |  0.25 |   0.27 | 0.28  | 0.30  |  0.31  |  0.33 | \n\nStage 8 - vision_model.encoder.layers.11.self_attn.v_proj:\n| Iteration (k) | 0 | 25 | 50 | 75 | 100 | 125  | 150 | 175  | 200  |\n| :------ | :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |   \n| $\\alpha_{s,k}$ | 0.80  |  0.78  |  0.77  | 0.75  |  0.74 | 0.73  | 0.72  |  0.71  |  0.70 | \n| $\\alpha_{p,k}$ | 0.20  |  0.22   | 0.23   |  0.25 |   0.26 | 0.27  | 0.28  |  0.29  |  0.30 | \n\n\n## Response to Question 4: About \"Improvements of DOP in Different Linear Layers\"\nThank you very much for raising this insightful question. We agree that analyzing whether DOP consistently outperforms OPCM across different linear submodules is highly valuable. To this end, we conducted the following ablation experiments:\n- (i) None: Neither OPCM nor DOP is applied to any linear layer.\n- (ii) Q/K/V Projection: OPCM and DOP are applied only to the Q/K/V projection layers.\n- (iii) FFN: OPCM and DOP are applied only to the FFN layers.\n- (iv) ALL: OPCM and DOP are applied to all linear layers.\n\nWe observe that DOP consistently outperforms OPCM. Notably, on the Q/K/V Projection layers, DOP achieves an accuracy of 72.9, compared to 66.6 for OPCM. Furthermore, when optimizing all linear layers, DOP achieves an accuracy of 78.3, surpassing OPCM's 75.5. **These results demonstrate that DOP consistently outperforms OPCM across different linear layers.**\n\n| Method | (i) None |   (ii) Q/K/V Projection |  (iii) FFN |  (iv) ALL |\n| :------ |  :------: |  :------: |  :------: |   :------: |  \n| OPCM   | 66.3 | 66.6 $\\pm$ 0.1 | 75.2 $\\pm$ 0.4 | 75.5 $\\pm$ 0.5 |\n| DOP (Ours)  | 66.3 | **72.9 $\\pm$ 1.8** | **75.8 $\\pm$ 2.0** | **78.3 $\\pm$ 1.6** |"}, {"review_id": "89uVgXnCy2", "reviewer": "Reviewer_1H81", "summary": "This paper proposes a Dual Orthogonal Projection (DOP) method for Continual Model Merging (CMM) without access to original data. DOP formally defines and balances stability and plasticity by projecting parameter updates onto the subspaces of previous and new tasks. The approach is formulated as a multi-objective optimization and solved with multi-gradient descent. Experiments on vision and language models show that DOP outperforms existing methods in both effectiveness and efficiency.", "questions": "- For continual model merging, is it possible to maintain a streamlined model library, where each model covers a certain category, and combinations of models can cover more cases? Could a dynamic merging mechanism based on this model library be used to address the formulation problem?\n- Can the current orthogonal projection analysis be generalized to more complex network architectures?\n- Can the proposed method be extended to networks with different structures?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- Analyzing continual model merging from the perspective of orthogonal projection is interesting.\n- The high-level idea of the method is intuitive: the space spanned by model gradient updates is used to approximate the input space, and by ensuring that the parameter changes for new and old tasks are orthogonally projected onto their respective input spaces, the method achieves both retention of previous knowledge and learning of new information.\n\nWeaknesses\n- The analysis of orthogonal projection is derived under the assumption of linear functions, but neural networks contain many nonlinear components and are highly stacked. It is unclear how the current analysis can be extended to structures that are closer to real scenarios, such as linear layers with activation functions or multiple stacked linear layers.\n- More baseline comparisons could be included in the experimental results.", "was_revised": true, "final_justification": "After reading the author's response, some of my concerns have been solved. I keep my previous justification of borderline acceptance.", "rebuttal": "Thank you very much for your review and affirmation of our work. We'll answer your questions one by one below.\n\n## Response to Weakness 1 \\& Question 2: About the \"Analysis under Complex Network Architectures\"\nWe understand your concerns regarding the linearity assumption, especially given that neural networks typically contain nonlinear activation functions and multiple stacked layers. We would like to clarify that, although modern neural network architectures generally include both linear and nonlinear components, mainstream architectures such as Transformers are in fact dominated by a series of linear layers (e.g., embedding layers, self-attention layers, and feed-forward networks). These linear layers constitute the majority of the parameters in the overall architecture. As shown in the table below, the proportion of linear layer parameters exceeds 90% in both Flan-T5 and ViT architectures, reaching as high as 99% in some cases.\n\nMoreover, all experiments in this paper are conducted on real-world, complex network architectures (including ViT-B/32, ViT-B/16, ViT-L/14, and Flan-T5, which are widely used vision and language models). These architectures contain multiple stacked linear layers, nonlinear activation functions, and other nonlinear components. **Our method consistently outperforms baseline methods across all these real architectures, demonstrating its applicability to complex, practical network settings.**\n\nOn the other hand, to address your concerns regarding the treatment of nonlinear layers, we have implemented and compared different merging strategies for nonlinear layers to observe their impact on final performance. Specifically, we experimented with several approaches, such as directly using the nonlinear weights from the pretrained model, applying weight averaging, and employing task arithmetic to merge the nonlinear weights of expert models. As shown in the table below, the performance differences among these strategies are minimal (79.2% vs. 78.3% vs. 78.9%), and all consistently outperform baselines such as SWA and OPCM. This indicates that **the impact of nonlinear layers on the merged model's performance is limited, as they account for only a small proportion of the total parameters**.\n\n| Architecture | The Proportion of Linear Layer Parameters |  \n| :------ | :------: | \n| ViT-B/32  | 97.21% |  \n| ViT-B/16  | 99.09% |  \n| ViT-L/14  | 99.68% |  \n| Flan-T5-Base   | 90.01% |  \n\n| Method | ACC |  \n| :------ | :------: |  \n| SWA | 66.3 $\\pm$ 0.0 | \n| OPCM | 75.5 $\\pm$ 0.5 | \n|  | |   \n| Our DOP (Pretrained for NonLinear)  | 79.2 $\\pm$ 1.2 |\n| Our DOP (Weight Averaging for NonLinear)  | 78.3 $\\pm$ 1.6 | \n| Our DOP (Task Arithmetic for NonLinear)  | 78.9 $\\pm$ 1.3 | \n\n\n\n## Response to Weakness 2: About the \"More Baselines\"\n\nThank you for your suggestion. We have further included MagMax[1] as a new baseline and evaluated it across three architectures (ViT-B/32, ViT-B/16, and ViT-L/14) and three task scales (8, 14, and 20 tasks). As shown in the table below, we observe that MagMax outperforms the three baselines—SWA, C. Task Arithmetic, and C. Ties-Merging—but still lags behind OPCM, which is specifically designed for continual model merging, and our proposed DOP. Overall, **our DOP consistently achieves the best performance**.\n\n| Method | ViT-B/32 (8 tasks) | ViT-B/32 (14 tasks)  | ViT-B/32 (20 tasks)  | ViT-B/16 (8 tasks) | ViT-B/16 (14 tasks)  | ViT-B/16 (20 tasks)  | ViT-L/14 (8 tasks) | ViT-L/14 (14 tasks)  | ViT-L/14 (20 tasks)  | \n| :------ | :------: |   :------: |   :------ | :------: |   :------: |   :------: | :------: |   :------: |   :------: | \n| Averaging (SWA)  |  66.3 | 65.4 |  61.1 |  72.3   |69.7  | 64.8 |   80.0   |  77.5 |  71.1 |   \n| C. Task Arithmetic   |   67.5 | 66.5 |    60.6 |  77.1 |70.9  | 64.2    |  82.1 |   77.9|  70.3  | \n| C. Ties-Merging    |   49.0 | 66.2 | 59.9 | 66.8    |  70.5| 63.0 | 64.3   | 78.0 | 68.3 |     \n| MagMax    | 70.7  | 67.0  | 61.2 |  76.7   | 67.0 | 62.5 |  83.4   |  71.2 | 71.2 |   \n| OPCM   |   75.5| 71.9 |  65.7|  81.8   | **77.1** |  70.3|   87.0  | 83.5 | 76.0 |   \n| DOP (Ours)   |  **78.3** | **73.0** | **68.5** |  **84.5**   |  76.2 |  **72.2** |  **88.3**   | **85.5** | **80.7** |   \n\n[1] “MagMax: Leveraging Model Merging for Seamless Continual Learning”, ECCV 2024.\n\n\n\n## Response to Question 1: About the \"Dynamic Model Merging Library\"\nWe agree with your perspective that maintaining a model library and leveraging model composition to cover more tasks or categories is indeed a promising direction. This approach is closely related to ensemble learning, which aims to collaboratively solve complex tasks by integrating the capabilities of multiple experts. However, the continual model merging (CMM) setting discussed in this paper is more aligned with multi-task learning or continual learning, where only a single set of model parameters is maintained to accomplish all tasks, resulting in higher efficiency. In summary, dynamic model libraries and unified models have different application scenarios:\n\n- **Expert Library or Ensemble Learning**: The expert library approach is suitable for scenarios with ample memory resources, where each model specializes in different tasks or categories, and model composition is used to enhance overall task coverage. The advantage of this method lies in its ability to handle more complex tasks through the collaboration of multiple expert models. However, it requires significant memory to store multiple models and may lead to inefficiency, especially in resource-constrained environments.\n- **Continual Model Merging**: In contrast, continual model merging is more appropriate for resource-limited scenarios, particularly when memory or computational capacity is restricted. In the continual model merging setting, the goal is to build a single unified model that incorporates the capabilities of all tasks by sequentially merging models for new tasks as they arrive. This approach avoids the need to store a large number of expert models, thereby reducing memory consumption while maintaining adaptability to new tasks and retention of previous knowledge.\n\nTherefore, expert libraries and continual model merging serve different application needs. We believe that **each has its own advantages and limitations, and the appropriate strategy can be chosen based on actual resource constraints and task requirements**.\n\n\n\n## Response to Question 3: About the \"Extended to Networks with Different Structures\"\nIn our experiments, we include four different architectures: three ViT models of varying sizes for vision tasks (ViT-B/32, ViT-B/16, and ViT-L/14) and the Flan-T5-Base model for language tasks. Additionally, our experiments cover different numbers of tasks (8, 14, and 20). As summarized in the table below, **DOP consistently achieves the best or near-best performance across different architectures and task scales, which strongly demonstrates the effectiveness of our method under various complex architectures**.\n\n| Method | ViT-B/32 (8 tasks) | ViT-B/32 (14 tasks)  | ViT-B/32 (20 tasks)  | ViT-B/16 (8 tasks) | ViT-B/16 (14 tasks)  | ViT-B/16 (20 tasks)  | ViT-L/14 (8 tasks) | ViT-L/14 (14 tasks)  | ViT-L/14 (20 tasks)  | Flan-T5-Base (8 tasks)  | \n| :------ | :------: |   :------: |   :------ | :------: |   :------: |   :------: | :------: |   :------: |   :------: |  :------: | \n| SWA  |  66.3 | 65.4 |  61.1 |  72.3   |69.7  | 64.8 |   80.0   |  77.5 |  71.1 |   78.8 |\n| OPCM   |   75.5| 71.9 |  65.7|  81.8   | **77.1** |  70.3|   87.0  | 83.5 | 76.0 |  80.6 |   \n| DOP (Ours)   |  **78.3** | **73.0** | **68.5** |  **84.5**   |  76.2 |  **72.2** |  **88.3**   | **85.5** | **80.7** | **83.6**   |"}, {"review_id": "Y4DhM4mTtO", "reviewer": "Reviewer_4cHS", "summary": "This paper tackles the problem of continual model merging in settings where models are received sequentially, without simultaneous access to all models. It considers the balance between stability and plasticity in continual model merging and investigates the relationships between data, gradients, and task vectors (accumulated gradients). Thus, it proposes a data-free dual orthogonal projection method, using a multi-gradient descent algorithm to derive Pareto-optimal coefficients for stability and plasticity. Experiments validate the effectiveness of the proposed DOP across various models and visual/language tasks.", "questions": "1. The paper consider reducing the influence of old merged model on the new model, which means $\\Delta W_{n}^{*}X_{n}=0$. \n\nBut if the old merged model have similar knowledge or positive knowledge which can be transferred to the new model and make the performance of new model have better performance on the new task data, then $\\Delta W_{n}^{*} X_{n}$ may not be zero in this case. How does DOP handle such situations? Does the method risk discarding positively aligned knowledge that would otherwise enhance performance on the new task?\n\n2. The baselines in the experiments are limited. Can authors compare DOP with MagMax[1]?\n\n[1] “MagMax: Leveraging Model Merging for Seamless Continual Learning”, ECCV2024.\n\n3. In Eq(2), why does DOP use $U_{o}^{\\top}\\Sigma_{o}$ and $\\Sigma_{o}V_{o}$ rather than $U_{o}\\sqrt{\\Sigma_{o}}$ and $\\sqrt{\\Sigma_{o}}V_{o}^{\\top}$, since $U_{o}\\sqrt{\\Sigma_{o}}\\sqrt{\\Sigma_{o}}V_{o}^{\\top}=\\tau_{o}$? Additionally, can authors report the effect of removing singular values (i.e., omitting $\\Sigma_o$ and $\\Sigma_n$) in this representation? A comparison of these variants could clarify the role of singular value scaling in the projection spaces.\n\n4. What’s the value of final $\\alpha_{s,K}$ and $\\alpha_{p,K}$? Can authors show the trending or changes of $\\alpha_{s,K}$ and $\\alpha_{p,K}$ during multi-objective optimization?\n\n5. For computational efficiency, beyond running time, can authors provide additional metrics, such as GPU memory usage or computational complexity, especially in comparison with OPCM?\n\n6. In Eq(3), there appears to be a missing index $k$ in the term  $\\nabla_{W_{*,k}}\\mathcal{L}_{p}$ in $\\min_{\\alpha_{k}\\in[0,1]}\\|\\cdot\\|$.", "limitations": "While DOP is a promising algorithm, it is currently limited to full model merging and may not scale efficiently to large language models in resource-constrained environments. Besides, the method is designed to minimize interference from the old merged model, but may ignore scenarios where positive transfer from the old merged model could benefit the new task. And DOP requires training during merging, thus developing a training-free algorithm remains an open direction for future research.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper proposes DOP, which not only addresses the forgetting issue in continual model merging but also reduces the influence of the old merged model on the new upcoming model.\n\n2. The ablation studies are well-designed and provide clear evidence of the contributions of different components within DOP.\n\n3. The motivation and formulation of DOP are presented clearly, with a well-grounded theoretical explanation.\n\nWeaknesses:\n\n1. The baselines in the experiments are insufficient. For the simultaneous model merging, only three methods are included (SWA, Task Arithmetic, TIES), but there are other recent merging algorithms, such as MagMax[1], which should be considered to strengthen the empirical evaluation.\n\n2. While the paper focuses on proposing an effective algorithm for continual model merging and the experiments also show that the simultaneous model merging algorithms exactly fail in the continual merging setting, it lacks a deeper analysis of why these simultaneous model merging algorithms fail in continual merging. Providing such analysis would reinforce the novelty and necessity of the proposed approach.\n\n3. The proposed DOP requires post-training (gradient-based optimization) when merging, which incurs additional training cost. It would be good to have a training-free merging algorithm instead of additional training.\n\n[1] “MagMax: Leveraging Model Merging for Seamless Continual Learning”, ECCV2024.", "was_revised": true, "final_justification": "I believe the continual merging setting explored in this paper offers a meaningful and under-explored direction that extends beyond traditional assumptions in model merging literature. I keep my positive score.", "rebuttal": "Thank you very much for your review and affirmation of our work. We'll answer your questions one by one below.\n\n## Response to Weakness 1 & Question 2: About the \"More Baselines\"\nThank you for your suggestion. We have added MagMax[1] as a new baseline and evaluated it across three architectures (ViT-B/32, ViT-B/16, and ViT-L/14) and three task scales (8, 14, and 20 tasks). As shown in the table below, we observe that MagMax outperforms the three baselines—SWA, C. Task Arithmetic, and C. Ties-Merging—but still lags behind OPCM, which is specifically designed for continual model merging, and our proposed DOP. Overall, **our DOP consistently achieves the best performance**.\n\n| Method | ViT-B/32 (8 tasks) | ViT-B/32 (14 tasks)  | ViT-B/32 (20 tasks)  | ViT-B/16 (8 tasks) | ViT-B/16 (14 tasks)  | ViT-B/16 (20 tasks)  | ViT-L/14 (8 tasks) | ViT-L/14 (14 tasks)  | ViT-L/14 (20 tasks)  | \n| :------ | :------: |   :------: |   :------ | :------: |   :------: |   :------: | :------: |   :------: |   :------: | \n| Averaging (SWA)  |  66.3 | 65.4 |  61.1 |  72.3   |69.7  | 64.8 |   80.0   |  77.5 |  71.1 |   \n| C. Task Arithmetic   |   67.5 | 66.5 |    60.6 |  77.1 |70.9  | 64.2    |  82.1 |   77.9|  70.3  | \n| C. Ties-Merging    |   49.0 | 66.2 | 59.9 | 66.8    |  70.5| 63.0 | 64.3   | 78.0 | 68.3 |     \n| MagMax    | 70.7  | 67.0  | 61.2 |  76.7   | 67.0 | 62.5 |  83.4   |  71.2 | 71.2 |   \n| OPCM   |   75.5| 71.9 |  65.7|  81.8   | **77.1** |  70.3|   87.0  | 83.5 | 76.0 |   \n| DOP (Ours)   |  **78.3** | **73.0** | **68.5** |  **84.5**   |  76.2 |  **72.2** |  **88.3**   | **85.5** | **80.7** |   \n\n[1] “MagMax: Leveraging Model Merging for Seamless Continual Learning”, ECCV 2024.\n\n\n## Response to Weakness 2: About the \"Why Traditional Model Merging Methods Fail\"\nThank you very much for your insightful question. We explain why traditional model merging methods fail in the continual model merging (CMM) setting from two perspectives:\n\nFirst, traditional methods (such as Ties-Merging) assume simultaneous access to all models and rely on consistent parameter sign alignment across models. However, these approaches face significant limitations in the CMM setting, where tasks arrive sequentially and, at each merging step, only the current model and the previously merged model are accessible. As a result, methods like Ties-Merging **cannot effectively handle scenarios with only local information available at each step**, leading to inaccurate sign alignment and degraded merged model quality.\n\nAnother critical challenge is that **continual model merging does not satisfy the commutative property** [1]:\n$$\n\\mathrm{CMM}\\left(\\mathrm{CMM}\\left(\\theta_{\\text {merged }}^{(i-1)} ; \\theta^{(0)}, \\theta^{(i)}\\right) ; \\theta^{(0)}, \\theta^{(i+1)}\\right) \\neq \\mathrm{CMM}\\left(\\mathrm{CMM}\\left(\\theta_{\\text {merged }}^{(i-1)} ; \\theta^{(0)}, \\theta^{(i+1)}\\right) ; \\theta^{(0)}, \\theta^{(i)}\\right), \n$$\nwhere $\\mathrm{CMM}(\\cdot)$ denotes the continual merging algorithm. This formula highlights that, in continual model merging, the order of merging affects the final result, which explains the performance degradation of traditional model merging methods in this scenario.\n\n[1] \"Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging\", Arxiv, 2025.\n\n\n## Response to Weakness 3: About the \"Training of DOP Algorithm\"\nRegarding the concern that DOP requires gradient-based optimization, we would like to clarify the following points. Although DOP does involve gradient computation during optimization, it offers several key advantages that keep the extra training cost low and ensure high computational efficiency:\n- **No Forward Pass Required**: DOP does not rely on the original task data. Instead, it directly optimizes based on the parameter differences between models. This means that, during merging, there is no need to perform traditional forward passes to compute losses or update gradients.\n- **No Backpropagation Through the Whole Network**: DOP performs optimization layer by layer, i.e., it independently optimizes each linear layer rather than jointly optimizing the entire model. This approach eliminates the need for complex backpropagation through the entire network, resulting in significant savings in both time and GPU memory.\n\nIn summary, while DOP requires gradient computation, **it is highly efficient, does not depend on original data, and avoids the computational burden of full backpropagation by optimizing each layer independently**. As shown in the table below, the optimization time for merging each linear layer is very short, typically only 1 to 3 seconds.\n\n| Architecture | Execution Time|\n| :------ | :------: |  \n| ViT-B/32 |  1.74 $\\pm$ 1.18 | \n| ViT-B/16 |  1.55 $\\pm$ 0.47 | \n| ViT-L/14 |  3.11 $\\pm$ 1.73 | \n| Flan-T5-Base |  1.30 $\\pm$ 1.32 | \n\n## Response to Question 1: About the \"Projection in Scenarios Where the Old Model Helps the New Model\"\nIn our DOP method, although we minimize the parameter differences between the old and new models to reduce interference from previous tasks, we do not completely disregard the beneficial knowledge contained in the old model (in practice, $\\Delta W_{n}^{*}X_n$ does not converge perfectly to zero, but only approaches this extreme value). More importantly, when computing gradients and performing optimization, DOP adjusts the parameter differences via orthogonal projection, thereby avoiding excessive interference with the new task. In this process, if some knowledge from the old model aligns with the objectives of the new task, DOP will retain this information, as it resides in a shared subspace and is thus unaffected by the projection. \n\n## Response to Question 3: About the \"Analysis of Variants of Projection Space\"\nThank you very much for your suggestion. We analyzed the performance of the following two variants on the ViT-B/32 architecture:\n- (i) DOP (w/o $\\Sigma$) refers to directly removing the singular values $\\Sigma$. We observed that its performance drops significantly (although it still outperforms the baseline method SWA), as it discards the projection penalty along the principal directions (those corresponding to larger singular values). \n- (ii) DOP ($\\sqrt{\\Sigma}$) follows your suggestion of applying a square root to the singular values, which still preserves the relative importance across different directions. As shown in the table below, we observe that the results using $\\sqrt{\\Sigma}$ are essentially the same as those using $\\Sigma$ (78.2% vs. 78.3%).\n\nIn summary, **retaining the magnitude of the singular values as the relative penalty strength for each projection direction is important**.\n\n| Method | ACC |\n| :------ | :------: |  \n| SWA | 66.3 $\\pm$ 0.0 | \n| OPCM | 75.5 $\\pm$ 0.5 | \n|  |  | \n| DOP (w/o $\\Sigma$)  |  67.6 $\\pm$ 1.3 |  \n| DOP ($\\sqrt{\\Sigma}$)  | 78.2 $\\pm$ 1.5  |  \n| DOP (Ours)  |  78.3 $\\pm$ 1.6 |  \n\n\n## Response to Question 4: About the \"Variation of Coefficient Alpha\"\nIn our method, the coefficients $\\alpha = \\{\\alpha_{s,k}, \\alpha_{p,k}\\}$ are adaptively updated across different layers, parameter modules, and merging stages. To further analyze the dynamic evolution of $\\alpha$ during optimization, we visualize the values of $\\{\\alpha_{s,k}, \\alpha_{p,k}\\}$ at various training steps.\n\nFor illustration, we randomly select two linear modules from two different stages, and record the values of $\\alpha$ at different training steps ($k=0,25,\\ldots,200$). As shown in the following tables, we consistently observe that the stability coefficient $\\alpha_{s,k}$ gradually decreases from the initial value of 0.8, while the plasticity coefficient $\\alpha_{p,k}$ increases from 0.2 to around 0.3. This trend reflects that, in the early stages, the model tends to retain more knowledge from the merged model, and as training progresses, it gradually incorporates new knowledge, resulting in a decrease in $\\alpha_{s,k}$ and an increase in $\\alpha_{p,k}$.\n\n\nStage 2 - vision_model.encoder.layers.11.self_attn.k_proj:\n| Iteration (k) | 0 | 25 | 50 | 75 | 100 | 125  | 150 | 175  | 200  |\n| :------ | :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |   \n| $\\alpha_{s,k}$ | 0.80  |  0.79  |  0.78  | 0.78  |  0.77 | 0.77  | 0.76  |  0.76  |  0.75 | \n| $\\alpha_{p,k}$ | 0.20  |  0.21   | 0.22   |  0.22 |   0.23 | 0.23  | 0.24  |  0.24  |  0.25 | \n\n\nStage 8 - vision_model.encoder.layers.11.self_attn.v_proj:\n| Iteration (k) | 0 | 25 | 50 | 75 | 100 | 125  | 150 | 175  | 200  |\n| :------ | :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |  :------: |   \n| $\\alpha_{s,k}$ | 0.80  |  0.78  |  0.77  | 0.75  |  0.74 | 0.73  | 0.72  |  0.71  |  0.70 | \n| $\\alpha_{p,k}$ | 0.20  |  0.22   | 0.23   |  0.25 |   0.26 | 0.27  | 0.28  |  0.29  |  0.30 | \n\n## Response to Question 5: About \"More Metrics on Computational Efficiency\"\nThank you for your suggestion. To provide a more comprehensive evaluation, we have included the peak GPU memory usage (Peak GPU Memory Allocated) during both the model merging and inference stages on the ViT-B/32 architecture. As shown in the table below, during the merging stage, our method requires slightly more GPU memory than OPCM due to the optimization performed on each linear layer. However, the overall GPU consumption remains very low, at only 247.3 MB. In the inference stage, DOP and OPCM exhibit comparable GPU memory usage. **While previous results have demonstrated the time efficiency of our method during optimization, these findings further confirm that our approach is also highly efficient in terms of memory usage.**\n\n| Method | Merging Stage (GPU) | Testing Stage (GPU) |\n| :------ | :------: |  :------: |  \n| OPCM  |  71.12 MB |  964.17 MB |  \n| DOP (Ours)  | 247.3 MB  | 973.48 MB  |    \n\n## Response to Q6: About the \"Typo Error in Eq(3)\"\nWe have carefully reviewed and corrected it in the revised version."}, {"review_id": "R3XVJPRCKZ", "reviewer": "Reviewer_fLc1", "summary": "This paper addresses the problem of continuous model merging, where expert models are received sequentially, and the original training data is not accessible. The authors propose a data-free method named Dual Orthogonal Projection (DOP), which aims to balance stability and plasticity by formulating the merging process as a multi-objective optimization problem. Extensive experiments on both vision and language models across multiple task configurations demonstrate superior performance over baselines.", "questions": "See weakness.", "limitations": "No. The author leverages task vectors instead of actual data or gradients to conduct projection. The approximation error remains unsolved and analyzed.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths.\n+ This paper is well-written and easy to follow.\n+ Extensive experiments on various architectures and tasks show the effectiveness of the proposed method.\n+ The proposed method does not rely on data during continual merging.\n\nWeaknesses.\n+ Although the method is grounded in orthogonal projection theory, this author leverages task vectors instead of actual data or gradients. The approximation error remains unsolved and analyzed. This could affect the reliability of the proposed method.\n+ The approach assumes linear layers dominate the parameter space, which is valid for transformers. I suggest the author consider significant non-linear modules, such as bias parameters and parameters of normalization layers.\n+ The paper proposes to solve the continuous model merging problem, where old task vectors cannot be stored. However, in practice, storing task vectors is usually not expensive, making this assumption somewhat questionable. So, I think this problem may not be a significant setting for model merging.\n+ The novelty is also limited; the proposed method seems very close to OPCM. The proposed method projects boththe  new task vector and the old task vector, while OPCM only project the old task vector.", "was_revised": true, "final_justification": "After the rebuttal, some of my concerns were addressed. However, I still think this paper makes less contribution compared with existing methods such as OPCM. Moreover, careful discussion with the authors reveals that CMM may not be a significant problem for the model merging community. Therefore, I would like to give the final justification of Weak Reject.", "rebuttal": "Thank you for your review and affirmation of our work. We'll answer your questions one by one below.\n\n## Response to Weakness 1: About the \"Approximation Error\"\nRegarding your concern about our method relying on task vectors instead of actual data or gradients, we would like to provide a detailed explanation as follows:\n\nFirst, why don't we use real data or true gradients? The continual model merging (CMM) setting [1], like traditional model merging [3,4], assumes that access to the original training data is unavailable. Without access to the data, it is impossible to compute true gradients, so we must approximate them using task vectors to enable effective model merging. \n\nSecond, regarding the ability of task vectors to approximate true gradients and the data subspace, we provide a detailed theoretical analysis in Section 3.2.2. \nSpecifically, the approximation error of task vectors to the input data in DOP can be theoretically bounded as follows: \n Let the task vector for a linear layer be $\\tau_l$, and the input samples be $\\{x_n\\}\\_{n=1}^{N}$, with each sample undergoing $T$ optimization iterations. Suppose the gradient norm of the loss function with respect to $\\tau_l \\cdot x_n$ is upper-bounded by $C_l$. Then, there exists a constant $\\Phi_l$ such that:\n$\n\\| \\tau_l - \\sum_{n=1}^{N} \\alpha_{n,l} (x\\_n^\\top) \\|\\_2 \\leq \\Phi\\_l \\cdot \\left( \\sum_{t=1}^{T} \\sum_{i=t}^{T} \\eta\\_t \\eta_i \\right)\n$,\nwhere $\\eta_t$ denotes the learning rate at iteration $t$, $\\alpha_{n,l}$ are reconstruction coefficients, and $\\| \\cdot \\|_2$ is the Euclidean norm. *Due to space limitations, we will provide the detailed proof of this approximation bound in the revised version*. \n\nFurthermore, from the perspective of empirical error, our experiments cover four different complex architectures—including three ViT models of varying sizes for vision tasks. As summarized in the table below, **DOP consistently achieves the best or near-best performance across different architectures and task scales, which strongly validates the effectiveness of using task vectors as gradient approximations in our method**.\n\nIn summary, although task vectors are an approximation of the true gradients, in scenarios where training data is inaccessible, they provide a reasonable and efficient solution for model merging.\n\n\n| Method | ViT-B/32 (8 tasks) | ViT-B/32 (14 tasks)  | ViT-B/32 (20 tasks)  | ViT-B/16 (8 tasks) | ViT-B/16 (14 tasks)  | ViT-B/16 (20 tasks)  | ViT-L/14 (8 tasks) | ViT-L/14 (14 tasks)  | ViT-L/14 (20 tasks)  | Flan-T5 (8 tasks)  | \n| :------ | :------: |   :------: |   :------ | :------: |   :------: |   :------: | :------: |   :------: |   :------: |  :------: | \n| SWA  |  66.3 | 65.4 |  61.1 |  72.3   |69.7  | 64.8 |   80.0   |  77.5 |  71.1 |   78.8 |\n| OPCM   |   75.5| 71.9 |  65.7|  81.8   | **77.1** |  70.3|   87.0  | 83.5 | 76.0 |  80.6 |   \n| DOP (Ours)   |  **78.3** | **73.0** | **68.5** |  **84.5**   |  76.2 |  **72.2** |  **88.3**   | **85.5** | **80.7** | **83.6**   |  \n\n\n## Response to Weakness 2: About the \"Nonlinear Modules\"\n\nWe agree with your concerns and have supplemented our analysis and experiments on both linear and nonlinear layers to address them.\n\nFirst, while modern neural network architectures do contain certain nonlinear components, in mainstream architectures such as Transformers, linear layers (e.g., self-attention and feed-forward networks) overwhelmingly dominate the parameter space. As shown in Table 1, the proportion of linear layer parameters exceeds 90% in Flan-T5-Base and all three ViT architectures, with ViT-B/16 reaching as high as 99%. This indicates that, although nonlinear modules exist, **linear layers constitute the vast majority of parameters in these architectures**.\n\nFurthermore, to directly address your concerns about the impact of nonlinear layers (such as bias parameters and parameters of normalization layers), we conducted experiments on various nonlinear layer merging strategies and evaluated their effects on the final model performance. Specifically, we explored several common approaches, including directly using the nonlinear weights from the pretrained model, applying weight averaging, and employing task arithmetic to merge the nonlinear weights of expert models. As shown in Table 2, the performance differences among these three strategies are minimal (79.2, 78.3, and 78.9, respectively). This demonstrates that, in mainstream real-world architectures, **the impact of nonlinear layers on the final merged model performance is limited**, primarily because nonlinear layers account for only a small fraction of the total parameters, while the overall model performance is mainly determined by the linear layers.\n\nIn summary, although nonlinear modules play an important role in modern neural networks, the dominance of linear layers in the continual model merging setting means that the merging of nonlinear modules has a relatively limited effect on overall performance. \n\n**Table 1**\n| Architecture | The Proportion of Linear Layer Parameters |  \n| :------ | :------: | \n| ViT-B/32  | 97.21% |  \n| ViT-B/16  | 99.09% |  \n| ViT-L/14  | 99.68% |  \n| Flan-T5-Base   | 90.01% |  \n\n\n**Table 2**\n| Method | ACC |  \n| :------ | :------: |  \n| SWA | 66.3 $\\pm$ 0.0 | \n| OPCM | 75.5 $\\pm$ 0.5 | \n|  |  | \n| Our DOP (Pretrained for NonLinear)  | 79.2 $\\pm$ 1.2 |\n| Our DOP (Weight Averaging for NonLinear)  | 78.3 $\\pm$ 1.6 | \n| Our DOP (Task Arithmetic for NonLinear)  | 78.9 $\\pm$ 1.3 | \n\n## Response to Weakness 3: About the \"Continual Model Merging Setup\"\n\nRegarding your concern about the storage overhead of task vectors, we acknowledge that in certain cases (e.g., when model size is small or the number of tasks is limited), the cost of storing task vectors may not be significant. However, the goal of this work is to provide a continual model merging (CMM) solution for memory-constrained environments where access to data is unavailable. **This CMM setting is not introduced for the first time in our paper**; prior studies have also explored similar scenarios [1,2].\n\nFurthermore, traditional model merging aims to combine multiple expert models into a single unified model, thereby reducing storage overhead and enabling knowledge transfer across tasks—a trend that has gained significant traction in recent research [3,4,5,6]. Compared to conventional model merging, the CMM Settings in this work place greater emphasis on scenarios where the model is gradually available in chronological order. Specifically, we introduce additional constraints to make this setting more realistic:\n- (i) Traditional model merging assumes all models are available simultaneously, whereas in practical scenarios, models may become available sequentially. Thus, CMM is more aligned with real-world applications.\n- (ii) Traditional model merging requires loading all models into memory, while CMM only needs to load the pretrained model, the new model, and the merged model at each step, further reducing resource requirements during merging and enhancing practicality.\n\nOverall, we believe that **avoiding the need to store all historical task vectors will greatly improve the scalability and efficiency of the model, especially as the number of tasks or model parameters increases substantially**. In addition, **given the broad applicability of model merging, further optimizing its use in continual scenarios is of significant practical value**.\n\n\n## Response to Weakness 4: About \"Novelty of the Method\"\n\nThank you for your attention to the novelty of our work. At first glance, our method shares some similarities with OPCM, as both leverage projection operations of task vectors to address the continual model merging (CMM) problem. However, the contribution of our proposed DOP method goes far beyond simply extending OPCM's unidirectional projection to a bidirectional one. Specifically, our approach offers **several significant advantages**:\n- **Theoretical Analysis**: DOP provides a deeper theoretical framework by analyzing the relationships among data, gradients, and task vectors, and elucidates their interactions in CMM. Through this analysis, DOP explicitly defines two key objectives—stability and plasticity—from the perspective of orthogonal projection. In contrast, OPCM does not formally define these objectives nor offer a systematic theoretical analysis.\n- **Stability-Plasticity Trade-off**: DOP simultaneously optimizes both stability and plasticity by formulating the problem as a multi-objective optimization, thus achieving a better balance between adapting to new tasks (plasticity) and retaining knowledge from previous tasks (stability). OPCM, on the other hand, focuses solely on stability—aiming to minimize interference from new tasks to old knowledge—while overlooking the crucial aspect of plasticity. Notably, OPCM merely performs a matrix projection operation without truly balancing stability and plasticity through an explicit optimization objective, whereas our method defines a loss function to jointly optimize both goals.\n- **Empirical Performance**: Experimental results across multiple vision and language tasks, four model architectures, and three task scales consistently demonstrate that DOP significantly outperforms OPCM, achieving higher average performance, less forgetting, and stronger task adaptability.\n\nIn summary, **the DOP method surpasses OPCM in terms of theoretical depth, methodological soundness, and empirical performance**.\n\n# **Reference**\n\n[1] Merging Models on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging. Arxiv, 2025.\n\n[2] How to Merge Your Multimodal Models Over Time?. CVPR, 2025.\n\n[3] Editing Models with Task Arithmetic. ICLR, 2023.\n\n[4] TIES-Merging: Resolving Interference When Merging Models. NeurIPS, 2023.\n\n[5] Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 2025.\n\n[6] Learning from models beyond fine-tuning. Nature Machine Intelligence, 2025."}]

Author rebuttal:

Meta-review: This paper develops an orthogonal projection approach to continual model merging that explicitly balances stability and plasticity, providing strong empirical results. The reviews were generally positive on the clarity of the approach, insights from the subspace analysis, and consistent performance improvements shown in the experiments. The rebuttal addressed most questions, clarifying some details of the approach, and adding comparisons and efficiency metrics. A few minor concerns seem to persist regarding the novelty over OPCM, and the linearity assumptions and ability to handle nonlinear components. The most negative reviewer seems to have focused on the practical necessity of the continual model merging setting, despite other reviewers viewing the setting as useful. The authors are encouraged to incorporate all aspects of the rebuttal and relevant parts of the discussion into their revision; additionally, I'd recommend strengthening the real-world motivation for CMM in the final version.
Decision: Accept (poster)