Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging
Authors: Anke Tang
Enneng Yang
Li Shen
Yong Luo
Han Hu
Lefei Zhang
Bo Du
Dacheng Tao
Venue: NeurIPS 2025 poster ()
Keywords: model merging
continual learning
knowledge transfer
Primary area: deep_learning
TL;DR: We propose a training-free projection-based continual merging method that efficiently combines models incrementally.

Abstract: Deep model merging represents an emerging research direction that combines multiple fine-tuned models to harness their specialized capabilities across different tasks and domains. Current model merging techniques focus on merging all available models simultaneously, with weight interpolation-based methods being the predominant approach. However, these conventional approaches are not well-suited for scenarios where models become available sequentially, and they often suffer from high memory requirements and potential interference between tasks. In this study, we propose a training-free projection-based continual merging method that processes models sequentially through orthogonal projections of weight matrices and adaptive scaling mechanisms. Our method operates by projecting new parameter updates onto subspaces orthogonal to existing merged parameter updates while using an adaptive scaling mechanism to maintain stable parameter distances, enabling efficient sequential integration of task-specific knowledge. Our approach maintains constant memory complexity to the number of models, minimizes interference between tasks through orthogonal projections, and retains the performance of previously merged models through adaptive task vector scaling. Extensive experiments on CLIP-ViT models demonstrate that our method achieves a 5-8% average accuracy improvement while maintaining robust performance in different task orderings. Code is publicly available at https://github.com/tanganke/opcm .

Merging on the Fly Without Retraining: A Sequential Approach to Scalable Continual Model Merging
Anke Tang'
Enneng Yang?
Li Shen Yong Luo' Han Hu
Lefei Zhang'
Bo Du' Dacheng Tao
School of Computer Science, National Engineering Research Center of Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering.
Wuhan University, Wuhan, China
Shenzhen Campus of Sun Yat-sen University, China Beijjng Institute of Technology, Beijing, China *Nanyang Technological University, Singapore
fanketang, luoyong, zhangleifei, dubo )uwhu. edu. yangenn@mail.. sysu, edu. hhu@bit... edu. en (mathshenli, dacheng. tao -agaal. com
Abstract
Deep model merging represents an emerging research direction that combines multiple fine-tuned models to harness their specialized capabilities across different tasks and domains. Current model merging techniques focus on merging all avail- able models simultaneously, with weight interpolation-basee methods being the predominant approach. However, these conventional approaches are not well-suited for scenarios where models become available sequentially, and they often suffer from high memory requirements and potential interference between tasks. In this study, we propose training-fire projection-bele
continual merging method that processes models sequentially through orthogonal projections of weight matrices and adaptive scaling mechanisms. Our method operates by projecting new pa- rameter updates onto subspaces orthogonal to existing merged parameter updates while using an adaptive scaling mechanism to maintain stable parameter distances, enabling efficient sequential integration of task-specific knowledge. Our approach maintains constant memory complexity to the number models, minimizes inter- ference between tasks through orthogonal projections, and retains the performance of previously merged models through adaptive task vector scaling. Extensive exper- iments on CLIP-Vis ViT models demonstrate that method achieves a 5-8% average accuracy improvement while maintaining robust performance different task or- derings. Code is publicly available htthts:://gtt htt///gtttttbbu wwt./oo////aa//e hatp//aanannk///ppe opcm/
Introduction
With the continued expansion of the scale of foundation models, the demand for efficient model development becomes increasingly pressing [104 94). Model merging, as promising approach to address this challenge, aims to combine multiple fine-tuned models to harness their specialized capabilities different tasks and domains [43 effectiveness of model merging in various scenarios, from combining language models with different
35 Recent studies have demonstrated the
task expertise [9] 98 51
to merging vision models trained in distinct domains
86 73
The predominant approach to model merging is weight interpolation, which combines model param- eters through weighted averaging of the individual models' weights 29
82 This method can be
formalized as 0 merged ((0) D (0(()) ((0)), where (()) and ((4) represent the parameters
*Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

of the pre-trained model and the i-th expert model, respectively, and a; are the interpolation coeffi- cients, which can be applied task- or layer-wise [52 96.. Theoretically, the effectiveness of weight interpolation-based merging methods is mainly influenced by two factors: data heterogeneity (the diversity of training set distribution between expert models) and training heterogeneity (differences
in hyperparameter settings and optimization dynamics) [59 78).
However, existing model merging approaches face several fundamental limitations that hinder their practical application and can be revisited from a continual merging perspective. Firstly, current methods typically require concurrent access to all models during the merging process, resulting in significant memory overhead that scales linearly with the number of models being merged [69]. Secondly, the law of commutation in model merging may not hold in continual merging scenarios, where the order of merging models is crucial and can affect the performance of the final merged model. This results in additional challenges for the algorithm design. Third, sequential model availability is a common real-world scenario and lacks attention in existing studies. Last, many current approaches necessitate additional training phases or extensive hyperparameter optimization, introducing substantial computational costs validation demands 33
74
[07 42)
To address these challenges, we propose the Orthogonal Projection-based Continual Merging (OPCM) method which enables a sequential merging as new models become available. Our approach consists of three key features: (1) By maintaining only the current merged and pre-trained models. we sequentially process new models, achieving constant memory complexity O(16)). (2) A projection- based merging mechanism processes parameter updates through orthogonal projections to minimize parameter interference between the incoming and merged models. (3) An adaptive time- varyiva scaling factor that dynamically adjusts the contribution of each model during the merging process to maintain stable performance, making the method robust to different merging orders. Together, these features enable OPCM to achieve training-free and memory-efficiet merging.
Our approach is built upon two key insights: (1) the use of orthogonal projections effectively minimizes interference between different tasks while preserving the specialized capabilities of each model, and (2) an adaptive scaling strategy keeps an approximately constant distance between the merged model and the pre-trained model, ensuring stable performance of the merged model.
In summary, our contributions are (1) We provide a formal mathematical framework for sequential model merging, and discuss its relationship with existing model merging methods. (2) We propose training-free free projection-base continual merging method that processes models sequentially through orthogonal projections and adaptive scaling mechanisms.
(3) We conduct extensive experiments on image classification tasks to demonstrate the effectiveness of our method and show 5-8% better accuracy on average while remaining consistently robust regardless of how tasks are ordered.
2
Related Work
Deep Model Fusion is scalable and effective in various scenarios, including language modeling [106]
79 81, vision modeling 97 27). and multimodal modeling 994 g Following (75), deep model
fusion techniques can be categorized into three types: model ensemble [68 3 48), model merging [43] and model mixing 99 38. In this study, we focus on model merging, which does not change the model architecture or introduce inference costs.
(1) Weight interpolation-baxee model merging is one of the most straightforward and intuitive methods for merging model parameters. This approach is largely based on the observation of mode connectivity, which suggests that different solutions within the parameter space can be connected by paths that maintain a consistently low loss 17 21 [01]. This method assumes that the models being merged lie in similar regions of the loss landscape, allowing for a seamless transition between them without significantly increasing the loss 30 52 86 2 29 677. In addition to its application in multi-task model fusion, weight interpolation can also be utilized during the preference alignment
phase in LLM training [103] 46
611, data-mixing [3 auxiliary-tast learning [31], and deep multi-objective optimization 64
7
25
76
[5]. (2) Alignment-bused model merging represents
another approach that focuses on merging models through the reduction of parameter or feature
disparities. Current research includes activation and weight matching [72
33
95 36
90 54] utilizing
channel-wi wise graph matching [7), and applying permutation invariant
2
44
2

---

Time
(())
(11)
Merge
e!ly 'urree
(00)
((3)
Merge
@ merppii
(12)
Merge
...
...
(
(tT)
Merge
(a) Conventional model merging
(b) Continual model merging
Figure 1: Comparison between conventional and continual model merging approaches. (a) Conven- tional model merging requires simultaneous access to expert models, performing merging in single step. (b) Continual model merging processes models sequentially as they become available.
However, the above methods require all models to be available simultaneously for merging, which limits their upplicability. contrast, our approach enables the continual merging of sequentially arriving models with higher memory efficiency.
Continual Learning addresses scenarios where models must sequentially learn new tasks while
maintaining performance previously acquired knowledge [4] [05]. This challenging paradigm has
led several key approaches: memory-based methods that store past samples 449 [1, architecture- based methods that modify network structures [66]] (20, 19], regularization based methods involve the
addition of regularization loss terms [37 [02), subspace-base based methods that use lower-dimensional projections [19] [6], and Bayesian methods via incorporating uncertainty estimation [56]. However, the aforementioned methods are all based on learning from the original data. contrast, the continual merging approach proposed in this paper allows to directly merge sequence of models (i.e., tasks) arriving in a stream, offering an orthogonal perspective to continual learning.
3 Rethinking Model Merging From a Continual Learning Perspective
In this section, we present novel perspective on multi-task model merging through the lens of continual learning, wherein task-specific models are obtained sequentially rather than simultaneously.
3.1 Problem Setup
Formally, let feos X' - y denote the pre-trained model parameterized by ((0), where X' represents the input space and y represents the output space. Consider 'I' task-specific models: {fam, faca).......... ... forr) ). each fine-tuned from the pre-trained model foo using the corresponding data set D; for task i. Each model may incorporate task-specific head ha y 2,. Our objective to construct a unified model fowmpa X > y that maintains high performance in all tasks while preserving specifce heads {hs, TT Initially, we define the conventional versus continual model merging approaches in formal terms and subsequently explore the challenges and current solutions from a continual learning perspective. In Figure [ we provide a visual comparison between conventional and continual model merging techniques, highlighting their distinctive approaches.
Definition 3.1 (Conventional Model Merging). Conventional model merging approaches predomi- nantly address scenarios where all expert models are available simultaneously. executing the merge operation in single step. This process can be formally expressed as follows:
mergged Merge g(1) ((2) ((T)).
Definition 3.2 (Continual Model Merging). Task-specific models are introduced progressively over time in the continuous model merging scenario. Model merging occurs incrementally in this paradigm as each new task-specing model becomes available. Specifically, at step f, we merge the current merged model parameterized by o((-1) with the newly trained task-specific model parameterized by g(t) @ This process can be formalized as:
g(t), yielding an updated merged model parameterized by
glt) marged = ContinualMerge (((tt g(0), g(0)), t t2 2,
marged
(2)
where typically (1) mergedd ((1) is initialized as the first expert model and ( (()) expected risk merged arg ming E(s.))).,....DD. (fe(z),
merged aims to minimize the
3

---

3.2 Opportunities & Challenges in Continual Merging
Storage and memory efficiency is a significant advantage of continual merging over traditional merg- ing methods. continual merging, at each step I, only a fixed number of models need to be stored: the current merged model, the new model to be merged, and optionally the pre-trained base model. This approach leads to a constant memory complex- ity of O(10)), where [0] represents the size of an in- dividual model. Crucially, this memory requirement remains constant, regardless of the total number of downstream tasks involved. In contrast, as shown in Figure 2 conventional merging methods typically involve maintaining all models, which results in a linear memory complexity of (T
Normalizad Memory Usags
10.0
Task Arilhmetic
1.5
Ours
5.0
2.5
0.0
The law of commutation in model merging refers to the property that the sequence in which models are combined does not influence the outcome the merged model. Formally, this commutative property described by the equation for standard merging techniques the following holds:
2
3
Number of
Figure Memory complexity of task arith- metic 29
and our method.
g(0) () g(ا)) = Merge(B(") ())))( g()), Vi,j.
(3)
However, in continual model merging, this commutative characteristic may not hold. Here, the ordering of models can have an impact on the final performance of the merged model. This is because, in a continual merging setting, each step of merging builds upon the previously merged model, resulting in sequential dependencies that disrupt commutativity. To illustrate, consider the continual merging process ContinualMerging (CM), where the commutative property may not hold:
CM (CM (Cmenedd (0)) (l(o ;0(0) o(++1) - CM (CM (Ommrgad ((0) ofitti)) 000) .00) Vi. (4)
This non-commutative nature introduces additional complexities in the knowledge preservation of preceding models, where the quality of the final merged model hinges not just on the models themselves but critically on the ordering in which they are integrated. Moreover, as earlier parameter information is gradually diminished throughout the sequential process, the final model might not completely encapsulate the knowledge from all individual tasks. Consequently, it becomes essential to develop effective merging strategies that take the non-commutation property into account, balancing sequential dependencies and task knowledge retention.
Existing applications of continual model merging are prevalent in online training settings, where models are updated along the training trajectory. This approach includes techniques such as the latest weight averaging (LAWA) [34], exponential moving average (EMA) [53] 651, and stochastic weight averaging (SWA) 30) These strategies, however, were not originally intended for scenarios with significant distribution shifts between consecutive models, which can be critical challenge. In Dziadzio et al. (18), the authors propose the concept of temporal model merging, where they select models to merge at each step, akin to continual model merging. While most existing model merging approaches assume the simultaneous availability of expert models, research on continual model merging remains limited adapted within the framework of continual model merging. As an example, consider task arithmetic: Baseline 3.3 (Continual Task Arithmetic). This is straightforward extension of the conventional task arithmetic to the continual model merging scenario. The update rule can be expressed as follows:
43 Some conventional model merging techniques can be revised and
g(t)
merged = e (lt-gg + X(B(#) g(0)),
(5)
where À is a scalar hyperparameter that controls the contribution of the new task-specific model.
To summarize, continual model merging methods align with real-world applications where tasks emerge progressively and offer improved memory efficiency. However, it can be non-commutative, which introduces unique challenges in task knowledge retention and interference resolution.
4 Methodology
Our approach to continual model merging is motivated by three key insights: (1) Orthogonality Preservation: To minimize task interference, the parameter updates for new tasks should be approxi- mately orthogonal to the existing merged parameter updates. (2) Magnitude Control: The magnitude

---

of parameter changes should remain stable throughout the merging process to prevent drift from the pre-trained model's 's loss basin. (3) Information Retention: The merged model should preserve task-specific information from all previously seen models while accommodating new tasks efficiently.
Building on these insights, we propose projection- based continual merging approach that (1) Projects new task vectors onto subspaces orthogonal to the current merged model: (2) Employs adaptive time- varying scaling to maintain stable parameter drift from the base model; (3) Preserves task-specific in-
10 tot [ 199
3/2
wetted
Avaf
ampes
oaf[)aoll) (2)
0
formation through careful parameter updates.
The complete procedure is outlined in Algorithm I and illustrated in Figure [] Starting with the first task-specific model facu, our method iteratively in- corporates new models they become available. For each new model, we handle the parameters in two distinct ways: weight matrices in linear layers un- dergo orthogonal projection to minimize interference, whereas other parameters, such as biases and those in -on-einear layers. are averaged directly.
(AB91)
((0)
orthogonal subspace
Figure 3: Subspace view. Geometric inter- pretation of the proposed continual merging approach, illustrating the orthogonal projec- tion and adaptive scaling mechanisms.
Algorithm 1 OPCM
An overview of the update rule is provided below. For weight matrices in linear layers, we propose the following update rule:
1: Initialize menged = g(1) average norm of
the task vectors n. = (1000) l|a, scaling fac- tor (1) 1.
W- merged (0(0)) ( Aw(L-1)
+ 1 AW(()
2: for f = 2 to T do
merged
3: for weight matrices W E mmxx in lin-
A(0)
ear layers do
(6)
4:
AW((-1)
merged <- w W
where W& Rmxn AW(L-1
merged Wellrged (
5:
(W w (
and ( = w - (0) denote the element-wise
6:
AW
p)(-1)
(AW(N);
AW
myj
margel
differences between the model parameters and the pre-trained model, commonly referred to as task vec-
7: end for
8:
for other parameters p
tors or delta parameters
[2 100] P p.(-(()) (.) repre-
9:
Ap(t) -p(*) 0p(
10:
end for
sents a projection mapping that projects the task vec-
11:
Ae(t-))
merged + plt-1) g(D)
tor (W onto the subspace spanned by Aw mergod-
12:
Concatenate AW
peoj and Ap! into
When new task model (shown in solid red arrow in Figure 3 arrives, its difference from the pre-trained model (() is projected onto this orthogonal subspace using the projection operator (((-1) (t-1) denotes vareing scaling factor that controls the con- tribution of the projected task vector to the merged model. By induction, we can expand Eq. to obtain the general term formula, proved in Theorem A.1
Ae(t)
prg
13:
n
+
L1
n + / 11
14:
A
(x(-1) A (p(t-1) Ag(()
/n
pcuj
1 1) s0 +2000
15:
marged
100)
16:
end for
17: return ((T)
marged
(+) merged = (.(0)
1
p(i-1) (aw()).
This formulation can be interpreted as a more generalized form of Task Arithmetic, where the task vectors projected onto subspace spanned by the previous merged model.
For biases in linear layers and other parameters, we simply set (t!- in Eq. as the identity mapping. (t-1) Thus the update rule is simplified as Pinerged = p!) + (1-1 Ap mmrgged + () ) / The general
term formula degenerates to a similar form as Task Arithmetic, i.e., Pinergod (0) + 1 A
Σ Ap()
The projection mapping (!-11) is designed to ensure orthogonality between the incoming task vector (t and the previous merged model Aw((-1) mergd while maintaining proximity between P!! (Aw()) and AW(), We first perform full singular value decomposition (SVD) on the previous merged model AW!(-1) merged ((--))((-1)((-) where (((-1) mm and ((t-1)
5

---

Rax" comprise orthonormal basis vectors, and E(t-1) E R is (rectangular) diagonal matrix containing the singular values (58) The proposed projection mapping p((-1) is then defined as: P!--") (Aw(()) = Lij=ra. (AW()) ue"))) "uu," where vi and Uj denote the i-th column of U((11) and the j-th column of ((t-1), respectively. is the projection threshold hyper-parameter, which controls the balance between retaining existing knowledge and incorporating knowledge from
new tasks. Ta represents the minimal rank of AWil--)) merged such that E 0; 2 Li=1 Em dz.
The adaptive time-varying scaling factor (4) is introduced to maintain a consistent magnitude of the merged model's deviation from the pre-trained model throughout the merging process, specifically 1|2 remains stable over time. The scaling factor (+) is computed as (1) = 1, (()) = 1 -1 merged + (!t-") 1) (AB(0) ||_ Avg (1 [|2): where
ensuring that the Frobenius norm ||W merged
A merged plsaged ( and 20 = ((t) g(0) represent flattened vectors of parameter differences, and Avg( | = Ei= || 1|2 is the average norm. For notational simplicity, we apply P!-"") directly to the task vector rather than individual weight matrices.
Alternatvely, the time-varying scaling factor can be set to ( = Vt. This choice is motivated by empirical ob- servations from prior work indicating that task vectors from different tasks tend to be approximately orthog- i.e., (AO(8), A06)) & dúj generally holds for all i.j E tL. The empirical validation of task vector orthogonality is presented in Figure 4 which analyzes 8 task-specific ViT-B/32 models. For
1.0
SUN397
1.00
0.03
0.01
0.01
Cas
0.02
1.00
0.02
0.01
0.01
0.02
0.02
0.02
0.8
RASISCA
0.03
1:00
0.01
0.02
0.02
onal 29
EundAA
0.02
0.01
0.05
1.00
0.02 0.03
0.01
0.02
0.6
SVIN -.01 0.01
0.01
0.02
1.00
0.07
0.15
0.01
0.4
GTSRB 0.02 0.02
0.07
0.05
more comprehensive analysis across model sizes and
MNIST 0.01 0.02 0.02 0.01 0.15
0.05 2.00 0.02
022
task quantities, we present additional results in the ap-
pendix: F (ViT-B/32)) Figure T0 [ (ViT-B/16), and
DTD 0.00 0.02
0.01
0.02
each evaluated on 20 downstream tasks. This orthogonality property makes /t natural choice for the scaling factor, as it helps maintain the magnitude of parameter changes across merging steps.
SUN399 Cars
RESSCCA
Bundaay
DTT
Figure 4: The cosine similarity between task vectors of ViT-B/32.
Theoretical Analysis
We now present some theoretical analysis of the proposed continual merging method, establishing key properties regarding orthogonality and stability. Proofs are provided in Appendix A. and additional discussion on the computational complexity of OPCM can be found in Appendix B
Theorem 5.1 (Orthogonality of Projected Task Vectors). Given a sequence of task-specific models {face Vi= fine-tuned from pre-trained model fawo, for any time step t, the projected task vector 1) (AW(I) obtained from the update rule in Eq. orthogonal to A Aw((-1) P!--1
merged: i.e.
(p(t-1) (AW()))
Aw((--1) merged = O.
(8)
Similar to [89] we can define the merging gap for i-th task as follows:
(2061))) - L. (010) + p(((111) (AO(i))
G, = Ci ((0) + Σ n pli-11 EnPdL" (Ae(t))
(9)
Where {mili= is sequence of scaling factors, and L, is the loss function for i-th task. Apply Taylor expansion to G, at g(o), we obtain the first-order approximation G, R
V Foror CT (E= p(i-1) (AQ(1)) At time step t, have ni 1/ for all i [1,
G. is the merging gap for 1-th task, which can be simplified G. Voc CT (10-13 X
Ao (naggee
On the other hand, since the task vector A0 represents the cumulative effect of gradient updates throughout the training process, we express the merging gap in terms of the inner product between

---

Table 1: The performance comparison of continual merging methods. We report the average accuracy and backward transfer (BWT) of the merged models. The best results are highlighted in bold. We abbreviate 'Continual' 'C.' in the table to space.
VIT-B/32
VIT-B/16
VIT-L/14
Method
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
Pre-Trained
48.1
56.9
55.6
55.4
62.0
59.8
64.9
69.1
65.6
Fine-Tuned
90.4
89.3
89.8
92.4
91.3
91.6
94.3
93,4
93.5
Fine-Tuned
79.8
67.4
62.6
82.9
72.2
68.2
90.0
70.9
77.7
Average (SWA)
66.3=000
65,42000
61.11000
72.310.0
69.71:000
64.8110.0
80.01:.00
77.510.00
711110.0
ACC1
C. Task Arithmetic
67.5=000
66.5=000
60.6+0.0
77.1+0.0
70.9+0.0
64.2+0.00
82.1
77.9+000
70.3
C. Ties-Merging
49.0:::0.22
66.2:006
59.9+007
66.843.7
70.5+0.8
63.01:..
64.347.00
78.0
68.340.9
OPCM (Ours)
75.5=0.5
71.9=0.3
65.7+002
81,80033
77.100.5
70.3+0.2
87.0
83.5+0.2
76.0
BWTT
Average (SWA)
-11.5:22.2
18.0
-7.12211
-9.721.55
-7.1114
-7341177
-7.3114
-5.41000
16,41..5
C. Task Arithmetic
.9.61:155
01.3=003
-3.1044
4.2100
-13104
-3610.4
-7.10.88
-1.8.0.3
-3,31..3
C. Teee-Mggggg
-15.3188.0
1.916.6
-1510..7
-5.11..4
1410.7
-15122
13.00.5.7
1.100
2..11.
OPCM (Ours)
1..11.11
16.011..
-7811.
-4.8.0.7
-5.1114
-63122
-2.11.0
4.310.77 -6.11..8
task vectors as G2 CA0())
A merged: where C is constant
77
89
Consequently, enforcing
orthogonality between these vectors (Ag(e) A 0) serves to minimize the gap G..
Theorem 5.2 (Bounded Parameter Distance). Under the update rule in Eq. with the empirical choice of scaling factor ( = Vl, the distance belween the merged model and the pre-trained model
remains bounded:
wi(0) merged
W(O)
A
S max 1E
(10)
The proposed continual merging method ensures that the merged model maintains a stable distance from the pre-trained model throughout the merging process, preventing parameter drift that could lead to catastrophic forgetting.
Corollary 5.3 (Preservation of Task Information). For any time step t, the merged model preserves information from all previous tasks in the following sense:
( merged
W(()
1 A(e)
((i-1) (AW()).
(11)
This formulation follows directly from Eq. This corollary demonstrates that our continual merging approach maintains weighted combination of all previously learned task task-specific changes, where each task's 's contribution is preserved through its projection. In Appendix A.2] we further discuss the special case where two adjacent task vectors highly correlated, and our method can automatically filter redundant knowledge in the incoming task vector.
6 Experiments
In this section, we present the experimental results of our continual merging approach and the ablation studies for comprehensive analysis of our method.
Experimental Setup: (1) Models and datasets. We first fine-tune the CLIP-ViT models 63 on up to
20 downstream tasks. Following [ evaluate approach three task sets of increasing size:
a base set of 8 tasks, an extended set of 14 tasks, and the complete set of 20 tasks. This hierarchical grouping allows to analyze how our method scales with increasing task diversity and complexity. Additional experimental details are provided in Appendix C (2) Evaluation metrics. To evaluate our merged models, employ two key metrics: average accuracy (ACC) and backward transfer (BWT) 45), details are provided in Appendix D.2
6.1 Continual Multi Mask Model Merging
7

---

Average accuracy with different number of tasks
VTT-B/32
We evaluate our method on three different CLIP-ViT architectures (ViT-B/32, ViT-B/16, and ViT-L/14) across three task sets of increasing size (8. 14, and 20 tasks). For each experiment, we set a = 0.5 and repeat the experiment 10 times with shuffled task or- der and report the mean and standard deviation of the results. We compare our continual merging approach against four baselines: (1) Continual fine-tuning, (2) Simple Weight Averaging (SWA), which takes the arithmetic mean model parameters, (3) Continual Task Arithmetic [29]. which performs weighted av- eraging based on task-specific scaling factors, and (4) Continual Ties-Merging [91] Details are in Ap- pendix D
VIT-B/16
VIT-L/14
Aucuracy
0.7
19
Number of tasks
Figure 5: Performance comparison of ViT models with different architectures across increasing number of sequential tasks.
As shown in Table [ our method consistently outperforms the baseline methods across all model architectures and task sets. Our method consistently achieves improvements of 5-8% over continual Task Arithmetic and continual Ties-Merging. In addition, the performance gain is maintained as we scale the task set size and model capacity. Furthermore, the backward transfer (BWT) results demonstrate our method's ability to retain knowledge of previously learned tasks. While some negative transfer is inevitable in a continual learning setting, our approach shows better BWT scores compared to the baselines in most cases. The larger ViT-L/14 model exhibits the least forgetting, with BWT of only -2.6% on the 8-task set, suggesting that increased model capacity helps mitigate catastrophic forgetting.
In Figure we show a line plot comparing the average accuracy of three different Vision Transformer (ViT) architectures they handle an increasing number of tasks from to 20 (a = 0.5). For each run, the order of the tasks is shuffted. Each line represents a different model variant, with shaded areas indicating confidence intervals. The plot demonstrates a general downward trend in accuracy as more tasks are added, with ViT-L/14 maintaining the highest performance throughout. Starting from around 90% accuracy for all models, there's 's gradual decline, with ViT-L/14 maintaining approximately 75-80% accuracy by task 20, while the smaller models (ViT-B/32 and ViT-B/16) show steeper degradation, dropping to around 65-70% accuracy.
(1) Robustness to task ordering. The relatively narrow confidence bands in Figure 5 and small standard deviations in Table [ demonstrate that our method is robust to different task orderings, maintaining consistent overall performance regardless of the sequence in which tasks are presented. Even as the number of tasks increases to 20, the performance variation remains contained, suggesting that our orthogonal projection mechanism effectively manages task interference regardless of the order in which tasks are merged. This robustness to task order is crucial practical advantage of a continual merging technique, as it climinates the need for careful task sequence engineering in real-world applications. (2) Model size matters in average performance. We also observe that model size plays a crucial role in the merging performance. The ViT-L/14 consistently achieves better results than ViT-B/32 and ViT-B/16, this indicates that larger models may be better suited for continual multi-task merging, possibly due to their increased dimensionality, which helps manage parameter interference by expanding the orthogonal subspace where task-specific updates can exist without affecting other tasks. This property is particularly beneficial when merging multiple task-specific models. (3) Model size matters in mitigating forgetling. The results in Table and Figure 5 also indicate that model size plays a crucial role in mitigating catastrophic forgetting during continual merging. While all models show some degree of performance degradation as the number of tasks increases. the rate of decline varies significantly with model capacity. The ViT-L/14, our largest model, exhibits the most graceful degradation (-17.5% drops at 20 tasks compared to individuals). In contrast, the smaller VIT-B/32 and ViT-B/16 models show steeper performance drops (24.1 %% and 21.3% average accuracy drops at 20 tasks, respectively).
6.2 Hyper-Parammter Analysis
Ablations on the projection threshold a. We perform ablation studies on the projection threshold a to validate the robustness of proposed method to different hyper-parameter settings. Figure 6 presents comparative ablation study between Task Arithmetic (a) and our method (b) on ViT-B/32 across
prgueetppeesents
8

---

VIT-B/32 on 14, 20 tasks using Task Arithmetic
VIT-B452 on 14, tasks using our mothod
Tasks
0.75
0.6
14 Tisks
20 Tasks
0.70
Accuracy
Accurcc
&
0.65
8 Tasks
Tasks
0.55
0.4
1.0
1.0
Scaling Factor
Projection threshold a
(a) Task Arithmetic
(b) OPCM (Ours)
Figure 6: Ablations on the hyper- pprammer settings. We compare our method with Task Arithmetic.
This figure highlights the robustness of our method to different hyper-parameter settings.
CLIP-VIT models on 20 tasks
10
0.75
0,70
0.65
VIT-B/32
0.60
VIT-D/32
2-
VIT-B/16
VIT-B/16
VIT-L/14
0.55
VIT-L/14
A =
0.6
15
17
19
Projection threshold
Time steps model index
(a)
(b)
Figure Hyper- pareee-rrretters analysis. (a) Performance comparison with varying size of models across different projection threshold on 20 tasks; (b) The adaptive scaling factor ( across different steps.
different task scales. Several key findings emerge from this comparison: (1) Knowledge retention and new task learning trade-off. Across all task sets, the performance curves exhibit a consistent inverted U-shape pattern, with optimal performance achieved the range of a =e 0.4-0.6 This suggests that moderate projection thresholds strike optimal balance between preserving task- information and managing interference between tasks. (2) Hyperparameter robustness and transferability. Our method demonstrates remarkably more stable performance across hyperparameter ranges compared to Task Arithmetic. The optimal a value remains relatively consistent (s00.) regardless of the number of tasks, demonstrating the robustness and transferability across different configurations. This stability and transferability is particularly valuable from practical perspective especially when hyper-parameter tuning expensive, thus minimal tuning of a is required when scaling different numbers of tasks. In Figure 7a we also show the performance comparison of CLIP-ViT models of different sizes with varying and similar trends as the projection threshold changes. More details are provided in Appendix D.4
20 tasks. It is observed that all three models show stable performance
Empirical analysis of the adaptive time-varying scaling factor (1), In our method, the scaling factor (t) is adaptively adjusted based the norm ratio between the combined task vectors and the average task vector norm. This adaptive mechanism helps maintain stable parameter distances throughout the merging process. Figure 7b shows the adaptive scaling factor (1) across different steps. The adaptive scaling factors closely follow a V curve (shown as the dashed black line), empirically validating our hypothesis of an empirical choice of (+) = vt in Section 4 All three model architectures (ViT-B/32, ViT-B/16, and ViT-L/14) exhibit remarkably similar scaling patterns, indicating that this time-varying adaptive scaling mechanism also robust and transferable across different model capacities.
Conclusion and Future Work
In this work, we presented a novel training-free projection-basee continual model merging method for combining multiple fine-tuned models sequentially. Our approach addresses several key challenges in model merging through orthogonal projections of weight matrices and adaptive scaling mechanisms.

---

Through extensive experiments, we demonstrated that our method consistently outperforms baseline approaches. The results show that our approach achieves 5-8% average accuracy improvement while maintaining robust performance across different task orderings. The effectiveness of our method scales well with model capacity, with larger models showing a better ability to mitigate catastrophic forgetting during sequential merging.
While our current work focuses on vision models, the proposed method's principles are generally applicable to other domains. Future work could explore extensions to language models, multi- modal architectures, and other scenarios where sequential model merging is beneficial. Additionally, investigating the relationship between model capacity and merging performance could provide valuable insights.
Broader impacts
This research advances the field of model merging with potential positive impacts on computational efficiency and accessibility of machine learning systems. By enabling effective continual merging of task-specific models. our method can significantly reduce storage requirements and computational costs associated with maintaining multiple specialized models. This efficiency gain could lead to reduced energy consumption and carbon footprint in Al deployments.
Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China (Grant No. 622251 13, U23A20318, U2336211, 1, 62576364 and 62276195), the Foundation for Innovative Research Groups of Hubei Province (Grant No. 2024AFA017), the Science and Technology Major Project of Hubei Province (Grant No. 2024BAB046) and the Shenzhen Basic Research Project (Natural Science Foundation) Basic Research Key Project (NO.
Dr. Tao's research is partially supported by NTU RSR and Start Up Grants. The numerical calculations in this paper have been done on the supercomputing system in the Supercomputing Center of Wuhan University.
References
[1] Arash Ahmadian, Seraphina Goldfarb- Tarrant, Beyza Ermis, Marzich Fadacc, Sara Hooker, et al. Mix data or merge models? optimizing for diverse multi-task learning. arXiv preprint arXiv:2410. 10801, 2024.
[2] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023.
[3] Devansh Arpit, Huan Wang, Yingho Zhou, and Caiming Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. Advonces Neural Information Processing Systems, 35:8265-8227 2022.
[4] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- 10d--011-iin minninnn discriminative components with random forests. In Computer September 12, 2014, proceedings, part VI 13, pages 446-461. Springer, 2014.
2014: 13th European conference, zurich, Switzerland,
[5] Pictro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 10211920--9999 2020.
[6] Arslan Chaudhry, Ranzato Marc' Aurelio, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient life- long learning with a-gem. In 7th International Conference on Learning Representations, ICLR 2019. International Conference on Learning Representations, ICI.R, 2019.
[7] Atoosa Chegini. Hamid Kazemi, Seyed Iman Mirzadeh, Dong Yin, Maxwell Horton, Moin Nabi, Mehrdad Farajtabar, and Keivan Alizadeh. Model soup for better rlhf: Weight space averaging to improve alignment in llss. In NewrIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024.
[8] Chi Chen, Yiyang Du, Zheng Fang, Ziyue Wang. Fuwen Luo, Peng Li, Ming Yan. Zhang, Fei Huang, Maosong Sun, et al. Model composition for multimodal large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
11246 11 12622, 2024.
10

---

[9] Weiyu Chen and James Kwok. You only merge once: Learning the pareto set of preference-aware model merging, arXiv preprint arXiv:2408. 12155, 2024.
[10] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEFE, 105 (65(10):186--1883 2017.
[11] Mircea Cimpoi. Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures the wild. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[12] Tarin Clanuwat, Mikel Bober-Irizat Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint 11120:181,.01718, 2018.
[13] Adam Coates, Andrew Ng. and Honglak Lec, An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
JMLR
[14] Gregory Cohen, Saeed Afshar, Jonathan Tapson. and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference neural networks (IJCN)) pages 2921-2926.
IEEE, 2017.
[15] Nikolaos Dimitriadis, Pascal Frossard, and Francois Fleuret. Pareto low-rank adapters: Efficient multi-task learning with preferences. arXiv preprint arXiv:2407.0805: 2024.
[16] Thang Doan, Mchdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusscau, and Pierre Alquier. theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference on Artificial Intelligence and Statistics, pages 1072-1080. PMLR, 2021.
[17] Felix Draxler. Kambis Veschgini. Manfred Salmhofer. and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pages 1309-1318.
PMLR, 2018.
[18] Sebastian Dziadzio, Vishaal Udandarao, Karsten Roth, Ameya Prabhu, Zeynep Akata, Samuel Albanie, and Matthias Bethge. How to merge your multimodal models over time? arXiv preprint 27X.022412.06711, 2024.
[19] Mehrdad Farajtabar, Navid Azizan, Alex Mou, and Ang Li. Orthogonal gradient descent for continual learning. International Conference on Artificial Intelligence and Statistics, pages 3762-3773, PMLR. 2020.
[20] Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv: 1701.08734, 2017.
[21] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Learning, pages 9259-3269 PMLR, 2020.
[22] C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. In International Conference on Learning Representations, 2017.
[23] Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofaa Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola. Task singular vectors: Reducing task interference in model merging. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18695-18705, 2025.
[24] Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mchdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong- Lee, et al. Challenges in representation lcarning: A report on three machine learning contests. In Neural information processing: 20th international conference, ICONIP 2013, daegu, korea, november 3-7, 2013. Proceedings, Part III 20, pages 117-124. Springer, 2013.
[25] Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. In The Thirteenth International Conference on Learning Representations, 2025.
[26] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2222 2019.
11

---

[27] Yongqi Huang, Peng Ye, Xiaoshui Huang. Sheng Li. Tao Chen, Tong He. and Wanli Ouyang. Ex- perts weights averaging: A new general training scheme for vision transformers. arXiv preprint arXiv:2308. 06093, 2023.
[28] Gabriel Ilharco, Mitchell Wortsman. Samir Yitzhak Gadre. Shuran Song, Hannaneh Hajishirzi, Simon Korblithh, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. Advances in Neural Information Prucessing Systems, 02222222-222777 2022.
[29] Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt. Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference Learning Representations. 2023.
[30] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty Artificial Intelligence 2018, UAI 2018, pages 876-808 688. Association For Uncertainty Artificial Intelligence (AUAT), 2018.
[31] Junguang Jiang, Baixu Chen, Junwei Pan, Ximei Wang. Dapeng Liu, Jie Jiang, and Mingsheng Long. Forkmerge: Mitigating negative transfer auxiliary-task learning. Advances in Neural Information Processing Systems, 36, 2024.
[32] Ruochen Jin, Bojian Hou, Jiancong Xiao, Weijie Su, and Li Shen. Fine-tuning linear layers only is simple yet effective way for task arithmetic. arXiv preprint (xXi:2447700088. 2024.
[33] Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietre and Pengxiang Cheng, Dataless knowledge fusion by merg- ing weights of language models. In The Eleventh International Conference Learning Representations, 2023.
[34] Jean Kaddour. Stop wasting my time! saving days of imagenet and bert training with latest weight averaging. arXiv preprint arXhv:2209, 14981, 2022.
[35] Arham Khan, Todd Nief, Nathaniel Hudson, Mansi Sakarvadia, Daniel Grzenda, Aswathy Ajith, Jordan Pettyjohn. Kyle Chard, and lan Foster. Sok: On finding common ground in loss landscapes using deep model merging techniques. arXiv preprint arXiv:2410. 12927, 2024.
[36] Edan Kinderman, Itay Hubara, Haggai Maron, and Daniel Soudry. Foldable supernets: Scalable merging of transformers with different initializations and tasks. arXiv preprint arXiv:2410.01483, 2024.
[37] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13); 3521-3526, 2017.
et al. Overcoming
[38] Aran Komatsuzaki, Joan Puigcerver, James Lee- Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslic, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-of-expers from dense checkpoints. In The Eleventh International Conference on Learning Representations.
[39] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[40] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[41] Yann LeCun, Lćon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applicd to document recognition. Proceedings of the IEEE, 86(1 1):2278 2324, 1998.
[42] Tao Li, Weisen Jiang, Fanghui Liu, Xiaolin Huang, and James T Kwok. Learning scalable model soup on a single gpu: An efficient subspace training strategy. In European Conference on Computer Vision, pages 342-359. Springer, 2024.
[43] Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey, 2023.
[44] Zexi Li. Zhiqi Li. Jie Lin, Tao Shen, Tao Lin, and Chao Wu. Training-ting time neuron alignment through permutation subspace for improving linear mode connectivity and model fusion. arXiv preprint a0X1:2240..01342, 2024.
[45] Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Beyond not forgtttgtt:nng:: Continual learning with backward knowledge transfer. Advances in Neural Information Processing Systems, 35:16165-1666 16177. 2022.
12

---

[46] Yong Lin, Hangyu Lin, Wei Xiong. Shizhe Diao. Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang. Wenbin Hu, Hanning Zhang, et al. Mitigating the alignment tax of rlhf. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 580-606, 2024.
[47] Chang Liu, Chenfei Lou, Runzhong Wang, Alan Yuhan Xi, Li Shen, and Junchi Yan. Deep neural network fusion via graph matching with applications to model ensemble and federated learning. In International Conference on Machine Learning. pages 13857-13869, PMLR, 2022.
[48] Tian Yu Liu and Stefano Soatto. Tangent model composition for ensembling and continual fine-tuning, In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 18676-18686, 2023.
[49] David Lopez-Pat Pazz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances neural information processing systems, 30, 2017.
[50] Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models. arXiv preprint arXiv22407.06089 2024.
[51] Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin Twin-merging: Dynamic integration of modular expertise in model merging. arXiv preprint arXiv:2406. 15479, 2024.
[52] Michael Matena and Colin A Raffel. Merging models with fisher weighted averaging. Advances in Neural Information Processing Systems, 35: 15:17703-17716 2022.
[53] Daniel Morales-Brotons, Thijs Vogels, and Hadrien Hendrikx. Exponential moving average of weights in deep learning: Dynamics and benefits. Transactions on Machine Learning Research, 2024.
[54] Anshul Nasery, Jonathan Hayase, Pang Wei Koh, and Sewoong Oh. Pleas- models with permuta- tions and least squares. arXiv preprint urXiv:2407.02444, 2024.
[55] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng. et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and insupervised feature learning, volume 201 I, page 4. Granada, 2011.
[56] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard E Turner Variational continual learning. arXiv preprint arXiv: 2017.
[57] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pages 722-729. IEEE, 2008.
[58] Peter J. Olver and Chehrzad Shakiban. Applied Linear Algebra. Undergraduate Texts in Mathematics. Springer International Publishing. Cham. 2018. ISBN 978 978-3-319.91040- 978-3-319--1041-- doi: N0.1107//98-3-311 1--119911041-3 URL http://link. springer www/ 1007/978-3 3-919-1--99 91041-3
[59] Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36, 2024.
[60] Omkar Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar, Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[61] Shiva Kumar Pentyala, Zhichao Wang, Bin Bi, Kiran Rammath, Xiang-Bo Bo Mao, Regunathan Radhakrish- nan, Sitaram Asur, Na, and Cheng. Paft: A parallel training paradigm for effective 1lm fine-tuning. arXiv preprint arXiv:2406. 0tt//066.9922 /7923, 2024. URL battps://arxils org/ aat///bb////// 2406. 17923
[62] Akshara Prabhakar, Yuanzhi Li. Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 644-655, 2025.
[63] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[64] Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean- Caantttp Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
13

---

[65] Karsten Roth, Vishaal Udandarao. Sebastian Dziadzio, Ameya Prabhu, Mehdi Cherti, Oriol Vinyals. Olivier Hénaff, Samuel Albanie, Matthias Bethge, and Zeynep Akata. A practitioner's guide to continual multimodal pretraining. arXiv preprint arXiv:2488.1442 2024.
[66] Andrei A Rusu, Neil C. Rahinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: 04677, 2016.
[67] Ildus Sadrtdinov, Dmitrii Pozdeev, Dmitry P Vetrov, and Ekaterina Lobacheva. To stay or not to stay in the train basin: Insights on ensembling in transfer learning. Advances in Neural Information Processing Systems, 36, 2024.
[68] Omer Sagi and Lior Rokach. Ensemble learning: A survey. Wiley interdisciplinary reviews: data mining and knowledge discovery, 8(4): 2018
[69] Li Shen, Anke Tang, Enneng Yang. Guibing Guo, Yong Luo, Lefei Zhang. Xiaochun Cao, Bo Du, and Dacheng Tao. Efficient and effective weight- ensembling mixture of experts for multi-task model merging. arXiv preprint arXiv:2410.2 20.0000.2.000 1804, 2024.
[70] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empiricul methods in natural language processing, pages 1631-1642, 2013.
[71] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian lgel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 32:323-332, 2012.
[72] G Stoica, D Bolya, Bjorner, P Ramesh, T Hearn, and J Hoffman. Zipit! merging models from different tasks without training. In International Conference on Learning Representations. International Conference on Learning Representations. 2024.
[73] Derek Tam, Mohit Bansal, and Colin Raffel. Merging by matching models in task parameter subspaces. Transactions on Machine Learning Rexearch, 2024.
[74] Anke Tang, Li Shen, Yong Luo, Liang Ding, Han Hu, Bo Du and Dacheng Tao. Concrete subspace learning based interference elimination for multi-tasi task model fusion. arXiv preprint arXiv:2312.06173, 2023.
[75] Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Do, and Dacheng Tao. Fusionbench: A comprehensive benchmark of deep model fusion. arXiv preprint arXiv:2406.03220 032800, 2024.
[76] Anke Tang, Li Shen, Yong Luo, Shiwei Liu, Han Hu, and Bo Du. Towards efficient pareto approxima- tion via mixture of experts hased model fusion. arXiv preprint arXiv:2406.09770, 2024.
[77] Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. Parameter- efficient multi-task model fusion with partial linearization. In The Twelfth International Conference on Learning Representations, 2024.
[78] Zhixu Tao, Ian Mason, Sanjeev Kulkarni, and Xavier Boix, Task arithmetic through the lens of one-shot federated learning. arXiv preprint 241iv:2411. 1. 1.1607, 2024.
[79] Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, Zachary Yahn, and Ling Liu. H3 fusion: Helpful, harmless, honest fusion of aligned lms.. arXiv preprint 2rXiv:2411.1 1.17992 2024.
[80] Bastiaan S Vceling, Jasper I Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. In Medical Image Computing and Computer Assisted Interventio- 2018: s/ International Conference, Granada, Spain, September 16-20, 2018 Proceedings, Part 11, pages 210-218. Springer, 2018
[81] Fanqi Wan, Xinting Huang, Deng Cai,, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Representations, 2024.
[82] Hu Wang, Congbo Ma, Ibrahim Almakky, Ian Reid, Gustavo Cameiro, and Mohammad Yaqub, Rethinking weight-averaged model-merging, arXiv preprini arXiv:2411.0926, 2024.
[83] Ke Wang, Nikolaos Dimitriadis, Guillermo izing task information for improved model merging and compression. arXiv preprint arXiv:2405.08813 2024.
François Fleuret, and Pascal Frossard. Local-
14

---

[84] Zhenyi Wang. Enneng Yang, Li Shen. and Heng Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024.
[85] Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-a natural language processing, arXiv preprint ar a/Xiv:1100.0377,, 2019.
[86] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre. Rebecca Roelofs, Raphael Lonejj--opes, Ari S Morcos, Hongseok Namkoong. Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: aver- aging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pages 23965 23998, PMLR, 2022.
[87] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion mnis:: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:
2017.
[88] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to z0o. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[89] Feng Xiong, Runxi Cheng, Wang Chen, Zhanqiu Zhang, Yiwen Guo, Chun Yuan, and Ruifeng Xu. Multi-task model merging via adaptive weight disentanglement. arXiv preprint arXiv:2411.1872, 2024.
[90] Yichu Xu, Xin-Chun Li, Le Gan, and De-Chuan Zhan. Weight scope alignment: A frustratingly easy method for model merging. In ECAI 2024, pages 1720-1727. IOS Press, 2024.
[91] Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36:7093-7115, 2023.
[92] Prateek Yadav, Colin Raffel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model mocrging: Recycling and routing among specialized experts for collaborative learning. arXiv preprint arXiv:2408.0700 07055, 2024.
[93] Prateek Yadav, Tu Vu, Jonathan Lai, Alexandra Chronopoulou, Manaal Faruqui, Mohit Bansal, and Tsendsuren Munkhdalai. What matters for model merging at scale? arXiv preprint 17X.:22410.33117, 2024.
[94] Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang. Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in Ilms, mllms, and beyond: Methods, thcories, applications and opportunities. arXiv preprint arXiv:2408.00 07666, 2024.
[95] Enneng Yang. Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. Forty-first International Conference on Machine Learning, 2024.
[96] Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. The Twelfth International Conference on Learning Representations, 2024.
[97] Peng Ye, Chenyu Huang, Mingzhu Shen, Tao Chen, Yongqi Huang, Yuning Zhang, and Wanli Ouyang. Merging vision transformers from different tasks and domains. arXiv preprint 1rXiv:2312.162444 2023.
[98] Xin Yi, Shunfan Zheng, Linlin Wang, Xiaoling Wang, and Liang He. A safety realignment framework via subspace-orienter model fusion for large language models. Knowledge- Baseed Systems, 306: 1 12701, 2024.
[99] Jaehong Yoon, Eunho Yang. Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv: 1704.01547 2017.
[100] Le Yu. Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. Forty-first International Conference on Machine Learning, 2024.
[101] David Yunis, Kumar Kshitij Patel, Pedro Henrique Pamplona Savarese, Gal Vardi, Jonathan Frankle, Mathew Walter, Karen Livescu, and Michael Maire. On convexity and linear mode connectivity in neural networks. In OPT 2022: Optimization for Machine Learning (NeurIPS 2022 Workshop), 2022.
15

---

[102] Friedemann Zenke. Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pages 3987-3995. PMLR, 2017.
[103] Chujie Zheng, Ziqi Wang. Heng Ji, Minlie Huang, and Nanyun Peng. Weak-to-strong extrapolation expedites alignment. In ICML 2024 Workshop Models of Human Feedback for Al Alignment, 2024.
URL / net/forum -dbyymbbittDDy
[104] Hongling Zheng, Li Shen, Anke Tang. Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Learn from model beyond tine-tuning: A survey, arXiv preprint arXi::2310008/11 2023,
[105] PaiWei Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: survey. arXiv preprint urXiv:2401.16386 2024.
[106] Yuyan Zhou, Liang Song, Bingning Wang. and Weipeng Chen. Metagpt: Merging large language models using model exclusive task arithmetic. arXiv preprint arXiv:2406. 1135, 2024.
[107] Max Zimmer, Christoph Spiegel, and Sebastian Pokutta. Sparse model soups: A recipe for improved pruning via model averaging. arXiv preprint arXiv:2306. 2023.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state that propose Orthogonal Projection-basee Continual Merging (OPCM), a sequential approach to merging multi- ple fine-tuned models that enables constant memory complexity. These claims are accurately reflected throughout the paper with theoretical analysis and empirical evaluation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses limitations in Section 3.2 regarding the con-commutative nature of continual merging and how task ordering affects final performance. Additional computational complexity considerations of the SVD-based orthogonal projection are dis- cussed in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
17

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:[Yes [Ye]
Justification: All theoretical results presented in Section (including Theorem 5.1 Theo- rem 5.2, and Corollary 5.3) are accompanied by detailed proof's in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper provides detailed information about the experimental setup, includ- ing dataset details, model architectures, and hyperparameters in Section and Appendix [ The implementation details of the continual merging algorithm are clearly described in Algorithm | and code is publically available via anonymous repository.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
18

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The datasets used in the experiments are publicly available, and the code is publically available via anonymous repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https www.oooewwniiinnnoooooooooieeeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides detailed information about the experimental setup, includ- ing dataset details, model architectures, and hyperparameters in Section and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our tables and figures include standard deviations across multiple runs with different task orderings, clearly showing the robustness of our method compared baselines.
Guidelines:
19

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix D.I provides details the computational resources used, including GPU types, memory requirements.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp//////aapppsc/c///////iiiia
Answer: [Yes]
Justification:
Our research adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
20

---

Section 7 discusses the potential positive impacts (enabling more efficient model deployment, reducing computational resources).
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper presents methodology for merging existing models rather than releasing new models or datasets with high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly cite and credit the CLIP models and all datasets used in our experiments, with appropriate references the original papers and licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
21

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code implementation of OPCM is documented with detailed comments, usage examples.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Our research does not involve crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our research does not involve human subjects, so IRB approval was not required.
22

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our research does not LLMs component of our methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy for what should or should not be described.
23

---

The appendix is organized into several sections, each providing additional insights and details related different aspects of the main work.
The appendix is divided into several sections, each giving extra information and details.
A Additional Theoretical Analysis
24
I Proofs of Theorems)
24
A.2 Theoretical Analysis of Correlated Task Vectors
26
Computational Complexity
27
Details of the Fine-Tunee Models
27
Additional Details on Experiments
32
D.1 Computational Resources
32
D.2
Evaluation Metrics
32
D.3
Continual Multi-Task Model Merging
32
D.3.1
Baseline Methods
32
D.3.2
Experimental Results
33
D.4
Ablation Studies
37
E
Comparison With SOTA Conventional Model Merging Methods
38
Additional Theoretical Analysis
A.1 Proofs of Theorems
In this section, we provide detailed proofs for the theorems and corollaries presented the main text.
First, we prove the general term formula of the merged model, which demonstrates that the merged model can be expressed shows how each task's ' contribution is preserved through its projection operator and serves as the foundation for understanding the key properties of our continual merging approach.
weighted combination of all previous task-specific models. This formula
Theorem A.1 (General Term Formula). Given the update rule of our method as defined in Eq. the general term formula of the merged model at time step t can be expressed as:
W(+) merged
W(()
1 A(0)
((t-1) (AW()).
(12)
Proof. We prove the general term formula by mathematical induction. For the base case 1, we have (11))
"merged = W111) = (0)) + p.d) (AWO))
X() where (()) is the identity mapping and (1) 1. For
I = 2, applying the update rule yields:
(1) (1)
merged + P( (AW(2))
(2(2) merged = (0(0) +
(13)
A(2)
11) ! ( (AW(1))
!!!) (AW(2)
(W +
X(1
(14)
X(2)
(/(0) + ( (AW(1) + (1) (AW(2)
(15)
(2)
AW
meseed
24

---

Thus, the formula holds for t S 2. For the inductive step, assume the formula holds for t = k 1. Then for t l:
A(=11) AW + +p. (AW(H)
w merged = ((0) +
merged
(16)
1(k)
A(k-1)
1
Eisl k-1 Pl 1) (AW(1)) + P/" 1((-1)) (AW(s)
-W(0)
(17)
A(4e)
k
Σ p(--1) ((k)
(AW(i).
(18)
By the principle of mathematical induction, the formula holds for all t 1.
fundamental property of our method is the orthogonality between the projected task vectors and the previously merged model. This orthogonality property is essential as it guarantees that new task vectors do not interfere with previously acquired knowledge, thereby preventing catastrophic forgetting.
Proof of Theorem 5.1 By definition of the projection mapping P'-- have:
(((-1)(AW(0)) AWmlrgud (
Σ
(wwi),ut unff) uny. D1--))(--y(--11
morgodd
(19)
F
(20)
Since (1--- represents the SVD of AW merged: where the columns of U(--1) and v(t-1) form orthonormal bases, for any pair (f,j) where * j. we have (u.oz (tt-))(t-
due to the orthogonality of singular vectors. Therefore:
p (AW(()), AW
mergod
0.
(21)
Finally, we prove the bounded distance theorem, which establishes that the distance between the merged model and the base model is bounded by the maximum distance between any task-specific model and the base model. This property provides a theoretical guarantee that the merged model remains within controlled region around the base model, preventing catastrophic drift while allowing for effective adaptation to new tasks.
We proceed by induction on f. For the base case t = 1, we have W W-(1) ( (1) so the bound holds trivially.
Proof of Theorem 5.2
merged
Assume the bound holds for t - 1. For step f. using Eq. 6 and the orthogonality property of the projection operator:
Vt (LW1(-1) merged
p(ll (((-1)(AW(1))
(AW(0) 2
(22)
merged
Vt
(p mergedi
p(t 1(t- (AW(())
+
(23)
1
25

---

By the properties of the projection operator, we know that |!- (Awrlt) 2
/
Applying the induction hypothesis:
wrll merged
AW(i)
/|2 + 1
(24)
t
iE[1,t-1]
Aw
max [E(L.
(25)
This completes the induction, proving the bound holds for all 2 1.
A.2 Theoretical Analysis of Correlated Task Vectors
Another important theoretical consideration in continual model merging how the projection mechanism handles correlated task vectors, where two task-specific models are highly similar each other. We analyze this scenario to demonstrate another advantageous property of our approach. Corollary A.2 (Efficient Handling of Correlated Task Vectors). Consider two task vectors AW((-1) and (W with high correlation. After incorporating AW(t-1) into Weeged the contribution of AW()) the merged model is automatically filtered to include only novel information not present in W (t--1) merged
Proof. Due the high correlation between task vectors, we decompose AW()) follows:
AW = مa ((t-)) merged + ?)
(26)
where p is a scalar representing the correlation coefficient (high when vectors are strongly correlated)
and 1/ is component orthogonal to AW Aw(t-1) i.e., (n, AWrt-gii F 0. Applying our projection
merged
operator, we have:
(((-1) (AW(()) (((-1) (pAW((t-1) + 1))
(27)
merged
-pplt 1) (AW<( merged + p(t-1)
(28)
By Theorem 5.1 (((- (AW mergod
since aw(t-1)
merged is orthogonal to its own projection. Addi-
tionally, since 7j is already orthogonal to Aw((-1)
merged+ we have P!(-1) (1) = 1y. Therefore:
(AW())
(29)
Consequently, the update for the merged model becomes:
(( + ((t-1) Aw
( merged
merged +
(30)
(e)
When the correlation is high, |y/ will be small, resulting in a minimal parameter update from the second task, which is precisely what want for highly correlated task vectors. This demonstrates that the orthogonal projection mechanism of our method:
1. Automatically filters redundant knowledge in AW( that is already captured in wx(t-1)
merged"
2. Incorporates only the novel information (7) into the merged model.
3. Prevents over-representation of similar concepts across multiple tasks, which could lead to catastrophic forgetting overfitting specific patterns.
This result has important practical implications. In scenarios where new task models are highly similar to previously merged models, our projection approach automatically adjusts the magnitude of parameter updates based on the novelty of the information. This automatic filtering mechanism contributes to the stability and effectiveness of the continual merging process, particularly when dealing with sequence of related tasks.
26

---

Fall ANEMET
WEL13
CTARIK
CITARIO
DUIST
CEARIR
DINST
RESSce
12313
G1388
01304
(8) ViT-B/32
(h) ViT-B/16
(c) VIT-1/14
Figure 8: Comparison of test set accuracy between the pre-trained model and individual fine-tuned models different downstream tasks.
B Computational Complexity
We briefly discussed the memory complexity in Section 3.2 showing O((9) constant memory
complexity for continual model merging versus 0 for conventional merging approaches. Here we discuss the computational complexity of OPCM. The computational complexity of OPCM is primarily dominated by the SVD decomposition, for each weight matrix W E IRmx,, computing the full SVD has complexity (mmin(m²) n n²m).. Therefore, for tasks and linear layers, the complexity is approximately OT L min(mn,mau mn)).
Table 2: This table shows the runtime of different methods.
Method
Wall-Clock Time of Merging Wall-Clock Time per Model
Continual Task Arithmetic
0.37s
ec0.44s
Continual Ties Merging
15.65s
</2s
AdaMerging (Conventional Model Merging) 10.47min
#r78.55
AdaMerging (Continual Model Merging)
11.85min
R=88.9 98.9s
OPCM
73.26s
29.1s
In Table 2 show the runtime of different methods, including AdaMerging (96), Continual Task
Arithmetic 29), Continual Ties Merging 191) and OPCM (the details of the compared baseline methods can be found in Appendices D and E). As shown in the Table, method is much faster than test-time adaptation-basee methods Tike AdaMerging, while maintaining runtime in the same order of magnitude training-free methods like Task Arithmetic and Ties-Merging.
C Details of the Fine-Tuned Models
In this section, present the experimental details of model fine-tuning and evaluate the performance of the pre-trained model (zero-shot test accuracy) and our fine-tuned models on the test set of each downstream task. For each downstream task, fine-tuned the visual encoder of the pre-trained CLIP-ViT models using task-specific training data, the classification heads are initialized using the pre-trained text encoder and fixed throughout the fine-tuning process. We employed a standard fine- tuning protocol with cross-entropy loss and the Adam optimizer, using cosine annealing learning rate schedule with maximum learning rate of le-5 and batch size 128, and train for 4000 steps
Models and datasets. We adopt the same experimental setup as described in 83 where we fine-tune three different sizes of CLIP-VIT models V/33 ViT-B/16. and ViT-L/14
20
downstream image classification tasks. All the models and datasets are publicly available on Hugging
Face [85) The 20 downstream tasks are SUN397 (88) Cars 339 RESISC45 [00, EuroSAT 26
SVHN (55), GT'RB (71) MNIST 41), DTD 11.. Flowers 102 57 PCAM [00 FER2013
24
OxfordllITPPe (60). STL10 13), CIFAR100, CIFAR10 40) Food101
4 FashionMNIST
87
EMNIST 14, KMNIST (2), and RenderedSST2 70 63
TThe models tuned and evaluated using the FusionBench 775
27

---

Table 3: Test set accuracy of the pre-trained model and individual fine-tuned models on different downstream tasks.
Model
SUN397
Cars
RESISC45
EuroSAT
SVIN
GTSRB
MNIST
DTD
Flower102
PCAM
CLIP-VIT-B/32
Pre-trained
63.2
59.6
60.3
45.0
31.6
32.5
48.3
44.2
66.4
60.6
Finc-tuned
74.9
78.5
99.1
97.3
98.9
99.6
79.7
88.6
88.0
TLIP-VTTR/16
Pre treitrained
65.5
64.7
66.4
54.1
52.0
43.5
51.7
45.0
71.3
54.0
Fine-tuned 78.9
85.9
96.6
97.6
99.0
99.7
82.3
94.9
90.6
CLIP-VIT-L14
Pre-trained
68.2
77.9
71.3
61.2
58.4
50.5
76.3
55.5
79.2
51.2
Fine-tuned
82.8
92.8
97.4
99.1
97.9
99.2
99.8
85.5
97.7
91.1
Model
FER2013 OxfonilITPPe
STL.10
CIFARIOD CIFARIO Food101 FushionMNIST EMNIST
KMNIST RenderedSST2
CLIP B/3
Pro-trained
41.3
83.3
97.1
63.7
89.8
82.4
63.0
12.0
10.0
58.6
Finc-tuned
71.6
92.5
97.5
88.4
97.6
88.4
94.7
95.6
98.2
71.3
CLIP-VIT-B/16
Pre-trained
67.3
11.2
Fine-tuned
72.8
94.5
98.2
88.8
98.3
91.9
94.5
95.3
98.1
75.7
CTJP-VIT-/114
Pre-trained
50.0
93.2
99.4
75.1
95.6
91.2
67.0
12.3
9.7
68.9
Fine-tuned
In the fine-tuning process, only the vision encoder was adjusted, while the text encoder remained unchanged. The pre pre-trained text encoder was used to generate crrr--shtttssk-speccff classification heads, and the classification heads were also fixed throughout the fine-tuning process. This means that both the pre- pre-trained and fine-tuned models utilize the same classification heads for given task. This approach preserves the opee-locabyllry capability of the model, allowing it to remain applicable any downstream task. Additionally, freezing the classification head does not compromise accuracy as shown in 28
Figure 8 and Table 3 demonstrate the performance comparison between pre-trained and fine-tuned models across 20 downstream tasks. As illustrated, fine-tuning consistently improves model perfor- mance across all architectures (ViT-B/16, VIT-B/32, and ViT-L/14), with particularly notable gains in task-specific accuracy for most tasks.
Figures 9 10 and I1 present the cosine similarity matrices between task vectors for CLIP-ViT models of different sizes (B/32, B/16, and I./14) fine-tuned on 20 downstream tasks. Several key observations emerge from these visualizations: (1) Most off-diagonal elements show very low cosine similarity values (0.01-0.05), indicating that the majority task vectors are largely orthogonal to each other. (2) The similarity patterns remain largely consistent across different model architectures, though ViT-L/14 14 shows slightly lower similarity values overall, suggesting that larger models may learn more specialized representations for each task. (3) Some groups of related tasks show slightly elevated similarity values, indicating that the tasks are more similar to each other. For example,
Digit recognition tasks MSVHN, EMNIST, KMNIST) exhibit higher mutual similari- ties.
Natural image classification tasks (CIFARIO, CIFAR100) show moderate correlations.
In most of the experiments, group the tasks into 8, 14, and 20 tasks, and report the average accuracy (ACC) and backward transfer (BWT) of the merged models. The task groups are as follows:
The 8 tasks: SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD
The 14 tasks: the tasks together with Flowers PCAM, FER2013, OxfordllITPet, STL10, and CIFAR100.
The 20 tasks: the 14 tasks together with CIFAR1O, Food 101, FashionMNIST. EMNIST, KMNIST, and RenderedSST2.
28

---

SUNte9
Heatmap of Task Voctor Cos Simnllarities VCT-B-V
1.00
0.02
0.03
0.02
0.01 0.02 0.01
0.03
0.02
0.01
0.02 0.03 0,02
0.02
0.01
0.01
0.11
Can
0.02 0.02
0.03
0.02
1.00
0.05 0,01 0.02
0,02 0.02
0.01
0.02
0.02 0.01 0.03 0.03 0.03
0,02
0.01
0.01
0.01
0.01
SVEN
0.02
0.05
1,00
0.02
0.03
0.01
0.02
0.02
0.00
0.05
0.02
0.01
0.01
O19K6
0.01
0.04
MINST
0.02
0.02
0.02
0.03
0.07
1.00
0.05 0.02
0.01
0.01
0.02
8.01 0.02
0.04
0.04
0.02
0.03
0.83
0.83
0.01
0.01
0.02 0.00 0.01 0.15 0.05
1,00
0.02
0.01
0.01
0.02
0.01 0.00
0.01
0.03
0.01
0.04
0.09
0.07
0.1
Plewrrst2
0.6
PCAM
0.02 0.02 0.01
0.00
0,00
0.01
0.01
0.03
2.00
0.00 0.01
0.08
0.00
0.01
0.00
0.01
0.01
0.01
0.01
0.00
0.01
0.01 0.02 0.02
0.01 0.01 0.01 0.01
0.00
1.00
0.01
0.00
0.02
0.02
0.01
0.01
0.01 0.01
0.00
STLIO
0.02 0.02 0.01
0.00
0.01 0.01 0.01 0.02 0.03
0.00
0.01
1.00
0.01
0.01
0.00
0.00
0.01
0.01
0.01
0.01
0.4
CIFAR10
0.03 0.02 1.03 0.03 0,02 0.02 0,02 0.02 0.00 0.02 0.02
0.01
1.00
0.05
0.09
0.02
0.02
0.02
0.01
0.01
CIFARI
0.02
0.1
Foodint
0.02 0.02 0.03
0,05 0.04 0.04 0.03 0.01 -0.00 0.02 0.03
0.00
0.11
1.00
0.02
0.04
0.03
0.02
0.00
0.02 D.02 0.00 0.02 0.01 0.02 0.01 0.02 0.01 0.01 0.02 0.02 0.02 0.03
0.02
1.00
0.02
0.01
0.01
0.01
0.2
RANNST
001
KANNEST
0.01 0.00 0.01 0.0L 0.04 0.03 0.09 0.01 0.01 0.01 0.02 0.01 0.02 0.03
0.01
0.04
1.00
0.06
0.00
0.01 D.0L 0.01 U.0L 0.33 0.03 0.07 0.01 0.01 0.11 0.02 0.01 0.01 0.02 0.02 0.01 0.03 0.06 3.00 0011
D.1
0.0 not 0.01
0,01
0.00
SURUM
RESBSCA
Fr
orsRn
MINST
DTD
Flavarttz
PrAM
Oslssilii Pei
$7760
CIBARi
Foudlty
EMIINT
KMISST
R
Figure 9: Cosine similarity between task vectors of CLIP- ViT-B/32 models fine-tuned on different downstream tasks.
29

---

SUNte9
Heatmap of Task Voctor Cos Simillarities (CLIP-VIT-B/1))
1.00
0.02
0,02
0.0L
0.01
0.01 0.01
0.02
0.02
-0.00 0.01 0.01 0.01 0.01 0.01 0.01 0.00 0.01
0.01
0.00
Can
0.01 0.01
0.02
0.00
1.00
0.04 0,01 0.02
0.01 0.02
0.02
0.01
0.01 0.01
0.02 0.02
0.02
0.01
0.01 0.01
0.01
0.01
SVEN
0.01
0.04
1,00
0.02
0.02
0.01
0.01
0.02
0.02
0.01
0.03
0.01
0.01
0.00
O19K6
0.01
0.03
MNNST
0.01
0.01 0.02
0.02
0.07
1.00
0.05 0.02
0.02
0.00
0.02
8.01
0.01 0.00 0.00
0.01
0.02
0.03 0.83
0.01
0.01
0.01 0.01 0.01 0.25 0.05
1,00
0.02
0.01
-0.00
0.02
0.01
0.01 0.00
0.02
0.00
0.03
0.08
0.07
0.01
Plewrrstt
PCAM
0.02
0.02 0,02
0.02
0.01
0.02
0.01
0.02
2.00
0.01
0.01
0.02
0.01
0.02
0.01
0.02
0.00
0.01
0.01
0.00
0..00.M
00000 0,01 0.02 0.00 0.00 40.00 40.00
0.01
1.00
0.02
0.01
0.02
0.03
0.02
0.02
0.01 0.01 0.00
STLIB
0.01 DUDE 0.01 U.0L
0.01 0.01 0.01 0.01 0.02
0.01
0.02
1.00
0.03 0.02
0.03
0.02
0.01
0.01 0.01
0.00
0.4
CIFAR10
0.01 0.00 1102
0,02 0.01 0.01 0.01 0.00 0.01 0.03 0.03 0.03
1.00
0.05 0.008
0.02
0.03
0.02 0.02
0.00
CIFARI
0.11
Foodinn
0.01 0.01 0.00
0.03 0.03 0.02 0.02 0.000 0.01 0.03
0.03
0.11
1.00
0.03
0.05
0.03
0.03
-0.00
0.01 DOD 0.01 0.01 0.01 0.01 0.00 0.00 0.02 0.02 0.02 0.02 0.02 0.03
0.03
1.00
0.02
0.01
0.01
40.00
0.2
RANNST
0.01
0.05
KANNEST
0.01 0.00 0.01 0.0L 0.04 0.03 0.08 0.01 0.01 0.11
0.01 0.02 0.03
0.01
0.05
1.00
0.08
0.00
0.01 D.0L 0.01 U.0L 0.94 0.03 0.07 0,01 0,01 0.11 0.03 0.01 0.02 0.03 0.03 0.01 0.04
0.08
3.00
0001
0.00
0.0 0.00 0.00 .0.00
0.00
0.0
SURUM
RESBSCA
Fr
OTsRn
MINST
DTD
Flavanttz
PrAM
Oslssili Pei
$7760
CIBARi
Foudlty
EMIRNT
KMISST
R
Figure 10: Cosine similarity between task vectors CLIP- ViT-B/16 models fine-tuned on different downstream tasks.
30

---

SUNte9
Heatmap of Task Vector Cos Similaritics (CLIP-VIT-L11))
1.00
0.01
0.02
0.0L
0.01
0.01
0.01
0.01
0.01
0.01
0.01 0.01 0.01 0.01 0.01 0.01 0.01 0.01
0.01
0.01
Can
0.01 0.01
0.02
0.00
1.00
0.05 0.01 0.01 0.01 0.02
0.02
0.02
0.01 8.02
0.02
0.02
0.03
0,02
0.02
0.02 0.01
0.01
SVEN
0.01
0.05
1,00
0.03
0.02
0.01
0.02
0.04
0.02
0.01
0.02
0.03
0.02
0.01
O19K6
0.02
0.01
0.03
MINST
0.01
0.01 0.01
0.02
0.05
1.00
0.03 0.01
0.01
0.01
0.01
8.01 0.01
0.00
0.00
0.01
0.03
0.83
0.04
0.01
0.01
0.01 0.01 0.00 0.09 0.03
1,00
0.01
0.01
0.01
0.01
0.01 0.01 0.00
0.03
0.01
0.03
0.05
0.05
0.11
Plewrrst2
0.01
0.6
PCAM
0.01 0.00 0.02
0.02
0.01
0.01
0.01
0.02
2.00
0.02 0.01
0.08
0.02
0.02
0.02
0.02
0.02
0.01
0.01
0.01
0.01
0.01 0.02
0.04
0.02 0.01 0.01 0.01 0.02
1.00
0.02 0.02
0.02
0.03
0.02
0.03
0.02 0.02
0.00
STLIO
0.01 DUOE 0.02
0.02
0.00 0.01 0.01 0.01 0.03 0.02
0.01
1.00
0.03
0.02
0.03
0.02
0.02
0.02
0,01
0.01
CIFAR10
0.01 D.0 1102
0,02
0.02 0.01 0.01 0.01 0.02 0.02 0.01 0.03
1.00
0.03
0.06
0.02
0.02
0.02
0.02
0.01
CIFARI
0.1
Foodinn
0.01 0.01 0.03
0.05
0.04 0.03 0.03 0.01 0.02 0.03 0.02 0.03
0.09
1.00
0.02
0.05
0.04
0.03
0.01
0.01 D.0 0,02 0.02 0.01 0.01 0.01 0.01 0.02 0.02 0.01 0.02 0.02 0.02
0.02
1.00
0.02
0.01
0.01
0.00
RMNNST
0.01
0.03
0.1
KANNEST
0.01 0.00 0.02 0.03 0.05 0.03 0.05 0.01 0.01 0.02 0.02 0.02 0.02 0.03 0.04 0.01
0.05
1.00
0.08
0.00
0.01 D.0L 0.01 0.02 0.94 0.04 0,05 0,01 0,01 0.02 0.02 0.01 0.02 0.03 0.03 0.01 0.05 0.08 3.00
0.01
D.1
0.01
0.00 0.0 DOt 0.00 0,00
0,00
SURUM
RESBSCA
orsen
MISST
DTD
Flavantz
PrAM
FRRZOL3
Orlssili Pei
$7760
CIBAR
Foudlty
EMIINT
KMISST
R
Figure 11: Cosine similarity between task vectors CLIP- V-T-L/1- models fine-tuned on different downstream tasks.
31

---

D Additional Details on Experiments
D.1
Computational Resources
All experiments were conducted on a single machine with 8 NVIDIA RTX 4090 GPUs, each with 24GB of memory. The code was implemented in PyTorch and Python 3.12.
D.2
Evaluation Metrics
Here we provide the details of the evaluation metries used our experiments, including average accuracy (ACC) of the merged model at the final step and backward transfer (BWT).
T
(WIr)) merged
ACCavy
=
T
ACC:
(31)
1=1
7-1
ACC,
ACC,
marged
BWT
(32)
T
( where merged and 0 ( merged the parameters of the merged model at i-th step and the final merged model, respectively. (-C is the accuracy on the i-th task's test data.
D.3 Continual Multi-Task Model Merging
For our continual multi-task model merging experiments, we evaluated three different CLIP-ViT architectures (ViT-B/32, ViT-B/16, and ViT-L/14) task sets of increasing size (8, 14, and 20 tasks). Each experiment was repeated 10 times with randomly shuffled task orders to ensure robust evaluation. The models were initialized using pre- re-ttrined CLIP checkpoints and subsequently fine- tuned separately for each task before being merged. During fine-tuning, only the vision encoder updated, while the text encoder remained frozen and was employed to generate task-specific classification heads. For further details on the tne-ttunig process, please see Appendix C
D.3.1 Baseline Methods
In our experiments, follow the common practice in model merging literature of using zero-shot classification heads initialized with the pre-trained CLIP model's text encoder. Specifically, we construct task-specific classification heads using CLIP's text encoder to generate zero-shot weights for each task's class names and templates. When fine-tuning the vision encoder, the classification head remains fixed. This is meant preserve the open-vocabulary nature of the CLIP model, it can still handle unseen classes during inference.
Continual Fine-tuning is a common baseline method for continual learning. For sequence of tasks associate with training datasets (Di, D D.), the model fine-tuned on each task sequentially. Denote the model parameters after fine-tuning on the i-th task as 6 The update rule of continual fine-tuning is given by:
( Fine-tune (D)),
(33)
where Fine-tune is the fine-tuning process. For each task, we use the same cosine annealing learning rate schedule with maximum learning rate of le-5 and batch size 128, and train for 4000 steps. For continual fine-tuning, we set the random seed to 42 and shuffle the task order.
Stochastic weight averaging (SWA) is common technique to stabilize the training process and improve generalization in model training 30 The update rule SWA is given by:
(t-1)) (t 1) + ((0)
(1) SWA
1())
(4)) SWA
(34)
where ((t) is the model parameters step t, and ((t) the averaged model parameters at step t.
Expanding the above equation, we have:
(+))
(60
(35)
SWA
i=1
32

---

Therefore, this method is equivalent to averaging the parameters of all the checkpoints, which is also
known as modelsoups in the literature [6 107 62
Task Arithmetic is a simple yet powerful technique for merging multiple task-specific models into a unified multi-task model, as demonstrated in recent studies 29 59 This method linearly combines the parameters of task-specific fine-tuned models with those of the shared pre-trained base model, allowing the merged model to handle multiple tasks effectively at the same time. Mathematically, task arithmetic can be formalized as follows. Given pre-trained model with parameters g(0) and set of T task- task-specific models fine-tuned from the pre-trained model with parameters ((+) for i 1.2..... .T. the parameters the merged model merged are computed as:
! merrged = ( + A Σ (0()) -000)
(36)
where À is a scaling factor that is usually selected on a validation set using a grid search. In our experiments, choose the value of À that achieves the highest average accuracy from the set {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0). The experimental results show that = 0.3 is the optimal value for eight image classification tasks, and À 0.1 is the optimal value for 14 and 20 tasks.
In continual multi-task model merging experiments, we use the following update rule of the merged model at step t
((0)
( mergedd
(g()) -(0))
(37)
merged
where g(t) the model parameters at step f, and g(9) morged the merged model parameters step
Memory complexity of task arithmetic: As explained in Section 3.2 continual model merging algorithm generally requires (())) memory, where [0] represents the size single model. And task arithmetic can be implemented as continual model merging method naturally. However, the most naive implementation of task arithmetic under the conventional model merging setting requires (T)))) memory since it loads all the task-specific models and obtain the merged model as merrged = ((0) + X Σ (g(+)
((0))
Mies-ng further extends the task arithmetic method by incorporating a systematic approach to handle parameter redundancy and sign conflicts during model merging (91). This method ensures that the merged model retains the most relevant and consistent information from the individual task-specific models, thereby improving overall performance and robustness. By trimming redundant parameters, electing the most significant sign for each parameter, and performing a disjoint merge, Ties-Merging effectively reduces interference between tasks, leading more stable and accurate results. Mathematically, Ties-Merging can be formalized as follows:
{
())
T
Tiesingging (1700)
{ Ties :=1
(38)
1=1
T
merged = g(0)
Ties
(i)
(39)
where ((0) is the pre-trained model parameters, 7(0) 000)
((0) is the task vector for the i-th task, Thee is the trimmed and selected task vector for the i-th task. À a scaling factor that is similar to the one used in Task Arithmetic. We use the same scaling factor À as in Task Arithmetie.
and (1)
In our continual multi-task model merging experiments, we use the following update rule of the merged model at step t:
(
merged = ((0) - x(((1) - ((0)),
((t)
marged = 0(0) + > (fooo-iiiood +
Ties
(1)
(40)
(t-1) where
TiesMerging (( (olt-gu) 000), 0(1) - -0(0))
Trees-merged Bies
D.3.2 Experimental Results
We conduct the continual multi-task model merging experiments on the three task groups using different sizes of CLIP-VIT models (ViT-B/32, ViT-B/16, and ViT-L/14), For each task group and
33

---

10
4MG
Dal
18.73
D71
CONAD
4.2
R14
0.57
M4
HAT M44
0.65
274 6:75
KTTH
M.76 4.71 0.1
5:1 6:51
0.90
0.13
5.44
612
62
Mens
Maay
VIT-B/32 on 8 tasks
ViT-B/16 on tasks
ViT-L/14 on tasks
0295
439
drave
10..0
RAY
EXTT
CATT
G4P
0.30
044
024
047
A44
E3T
G47
8:77
8:00
6:7
084
6.5
045
0M7
8:2
EST
(d) VIT-B/32 14 tasks
(e) V/T-B/32 on 20 tasks
Figure 12: Here we show the accuracy matrix of the continual multi- model merging using weight averaging for different model architectures and task sets. The order of the tasks is shuffled for each run.
model size, we repeat the experiment 10 times with different task orders to investigate the effect of task order on the performance of the continual multi-task model merging.
Accuracy matrix analysis. The accuracy matrix is structured follows: each row corresponds to new task-specific model incorporated into the merged model, while each column displays the test accuracy for individual tasks after merging. Specifically, for the i-th row, the j-th column represents the test accuracy of the j-th task for the merged model o i.e. ACC, (011) (Omggg)) The diagonal elements represent the performance on the newly added task, while off-diagonal elements show how well the model maintains performance previously learned tasks.
we show the accuracy matrix of the continual multi-task model merging using weight averaging for different model architectures and task sets. These accuracy matrices reveals several key insights about the continual multi-task model merging process. First, we observe a general trend of accuracy degradation more tasks are added. This suggests some degree of catastrophic forgetting occurs during continual merging. Second, larger models (ViT-L/14) show better resistance to this degradation compared to smaller models (ViT-B/32), maintaining higher accuracies across tasks. Third, when comparing task sets of varying sizes (8, 14, and 20 tasks), it becomes evident that maintaining consistent performance across all tasks grows increasingly challenging the number of tasks expands. This is reflected in the generally lower accuracy values observed in the 14- and 20-task matrices. Finally, note that the performance decline is particularly significant for certain tasks, such as KMNIST and EMNIST. This could be attributed to the fact that these tasks exhibit the greatest divergence from the pre-trained model and the other tasks in the set.
In Figure 12,
Per-task performance analysis. The experimental results in Table 4 and Table demonstrate the effectiveness of different model merging approaches across three CLIP-ViT architectures (B/32,
34

---

Table Test set accuracy of the pre- pre-trained model and individual fine-tuned models on 14 different downstream tasks. For continual fine-tuning, we fix the random seed to 42, shuffle the task order, and report the average accuracy. For other methods, we repeat the experiment 10 times with different task orders. Here we abbreviate 'Continual" as 'C.' to save space.
Method
SUN397
Cars
RESISC4S
EuroSAT
SVHN
GTSRB
MNIST
CLIP-VIT-B/32
C. Finc-Tuned
56.9
56.9
79.2
89.7
81.0
93.4
Average (SWA)
64.8.10.0
60.4.0.0
67.11000
67.010.0
50.710.0
45.6.10.0
76.6.10.0
Task Arithmetic
64.4110.0
59.610.0
67.310.00
67.810.0
$4.01.0.0
50.0.10.0
80.7.10.0
C. Ties-Merging
64.8100.8
59.7.1.7
68.914.77
62.513.1
50.1
53.3.17.2
80.8.5.00
OPCM (Ours)
65.6+:0.4
57.9=100
7.
77.8=2.1
69.9+1.8
62.3#1.0
92.2+10
CLIP-WT-B/16
C. Fine-TTned
64.3
69.9
66.2
93.3
79.6
94.9
Average (SWA)
67.5+0.00
65.91000
71.5
71.120.0
64.6:10.0
54.1::0.0
82.6:10.0
C. Task Arithmetic
67 07.8100.0
65.2=000
71,9-0.00
74,0=0.0
67.020.0
57.2=0.00
88,940.00
Tics-Merging
68.5100.7
67.2-2.7
72.31:55
73.0=7.1
65.611.00
53.81:3.0
88.6:4..1
OPCM (Ours)
68.810.3
6..411.
78.71:.22
84.6119
79.2
70.5::2.5
95.4:0.55
CLJP-VET-I/14
C. Fine-Tumed
53.8
78.7
59.7
73.3
89.0
99.0
98.8
Average (SWA)
71.2+00.0
79.0=0.0
78.7=0.0
80,4=0.0
71.3+0.0
64.6#0.0
94.3++0.00
Task Arithmetic
71.6+0.0
78.4=0.0
79.3
80.3=0.0
72.4+0.0
67.9+0.0
95.3++0.0
C. Tics-Merging
72.2+1.2
78.4=0.8
81.5+4.9
79.4=5.3
72.4+.00
66.1+2288
94.6+1
OPCM (Ours)
74.010..
82.110.4
85.8.10.8
86.2.11.
84.4188
85.1.11.3
97.7.10.2
Method
DTD
10owers102
PCAM
Fer2013
OxfordIITPPet
STL.10
CIFAR100
CLIP-VIT-B/E
C. Finn--Tuned
47.4
38.8
69.4
70.4
72.6
87.8
43.7
Average (SWA)
46.9=0.0
67.4=0.0
65.2=0000
51.6=0.0
84.2+0.0
97.2=0.00
70.4+:0.0
Task Arithmetic
48.0.10.0
66. 16.1000.0
69,8.10.0
53.110.0
84.21000
96.6.10.0
69.2.10.0
Tiee-Merging
48.2.11.6
66.0.12.2
67.6.1288
54.313.1
84.0.0.77
97.1.1..3
69.912.33
OPCM (Ours)
52,310.7
65.6.10.6
81,2.122
60.5.00.5
84.610.2
96.7.10.2
68.5
CLIP-WIT-B/6
C. Finnc-Tuned
46.4
42.0
64.1
72.7
84.1
90.8
56.5
Average (SWA)
47.2+00.0
72.5=0.0
63.2+0.0
54.1=0.0
90.4+00.0
98.3+00.0
73.4+:0.0
Task Arithmetic
47.8.10.0
72.010.0
64.810.00
54.8-0.0
90.6:10.0
98.1100.0
72.810.00
C. Ties-Merging
48.1110.9
71.5100
63.211.77
54.513.5
90,311.0
98.2.10.1
72.542.8
OPCM (Ours)
518810.7
74,5-00.7
83,911.2
62.7-004
92.7-0.2
97.9-00.1
7..8
CIJP-
C. CuneeeTTueed
42.6
60.9
83.1
87.6
Average (SWA)
58.7400.0
81.910.0
74,2+:000
54.8:0.0
94.61:000
99.31:0.0
82.4:10.0
Task Arithmetic
59.8+0.0
81.92000
71.120.0
56.1=0.0
94.820.0
99.0:10.0
82.3+000
C. Tics-Merging
59.3+1.1
81,6=1.5
72.8+199
57.3=188
94.6+0.4
99.2+0022
82.9+1.2
OPCM (Ours)
63.6+1.11
87.6=0.3
80.4+0.9
63.1-0.4
95.8+0.11
99.2+0.00
84.0:t0.5
B/16, and L/14) on 14 and 20 downstream tasks, respectively. It is observed that OPCM (the proposed method) outperforms other continual model merging approaches in majority of tasks. For instance, on ViT-B/32, OPCM achieves best performance on several tasks including SUN397 (64.4%), RESISC45 (66.0%). SVHN (66. %). GTSRB (56.0%). MNIST (90.2%), and PCAM (80.2%). FER2013 (58.5%), CIFARIO (92.8%), and RenderedSST2 (64.6%). The results also reveal that certain tasks, particularly EMNIST and KMNIST, remain challenging for all continual model merging methods, though OPCM generally maintains better performance even on these difficult cases. Furthermore, Continual Ties-Merging shows significant higher standard deviations compared to OPCM across multiple tasks and model arichitectures, indicating that Continual Tees-Megging is much more sensitive to the order of tasks merged, leading to inconsistent performance across different task orderings.
In Table 6 we dive deeper into the comparison between continual fine-tuning and OPCM by ex- amining the average accuracy over 10 different task orders for OPCM. Continual fine-tuning extremely computationally intensive. For instance, in 20-task experiments, each task requires
35

---

Table Test set accuracy of the pre- pre-trained model and individual fine-tuned models on 20 different downstream tasks. For continual fine-tuning, we fix the random seed to 42, shuffle the task order, and report the average accuracy. For other methods, we repeat the experiment 10 times with different task orders. Here we abbreviate 'Continual' as 'C.' to save space.
Method
SUN397
Cars
RESISCAS BuroAAT SVHN
GTSRB
MNIST
DTD
Flowers102
PCAM
CIJP-WT-R/32
C. Fine Tumed
53.9
38.2
64.7
98.7
45.4
34,4
86.7
58.4
57.5
67.7
Average (SWA) 64.2:1:0.0
59.610.00
64.810000
60.91000
47.310.0
43.1:10.
71.81000000
46.4-000
66.510.00
63.9.10.0
C. Task Arithmetic 62.0:20.0
53.7.0.0
60.9:0.00
58. 1:000
48.5+0.0
48.940.0
79.40..0
46.1=0.0
61.100.0
73.4:0.0
Ties-MMrggng 62.5:4222
49.1=4.3
55.8+2277
50,9=5.2
$4.6 49.3:6644
82.0=588
46.7=3.7
58.5+3.7
69.9=4411
OPCM (Ours)
64.4:10.3
11.11:16
66.01122
717742.1
66.1 $6.01122
90,2::0.6
0..4
64.9104
80.2:10.6
C. S.Fin-TTuned 62.7
58.0
67.6
99.1
46.0
29.2
93.9
61.9
64.1
75.2
Average (SWA) 67.1:10.0
6..61.0.0
69.310.00
63.4.10.0
62410.0 $2.010.0
80.7.1..0
4..6..0.0
71.8.0.0
63.1:10.00
Task Arithmetic 65.8:1:0.0
57.510.0
63.800.0
59,5:000
64.720.0 54.0:0..0
88.8=0.0
45.3=000
67.540.00
67.1
C. Ties-Merging 64.2:t1.6
52.9=7.8
60.955.2
$3.0+833
12.8 48.8+5.0
88.42422
45.0=2.5
61.3+3.7
68.5=6.4
OPCM (Ours)
67.9:1:022
1.39
73.4008
775529
74420.9 63.21199
94.1=005
49.2=0.4
72.32:03
39,6:
CLIP V/T
Fine-TTuee 69.5
73.6
78.3
59.3
49.3
Average (SWA)
70.7+00.0
77.7=0.0
76.40.00
75.3+0000
69.5+0.0
62.1=000
93.7=0.0
57,7=0,0
80.0=0.00
73,6=20.0
C. Task Arithmetic
70.4:1:0.0
74.11.0.0
73.9110.0
66.3.0.00
69.9.0.0 65.6.10.0
95.1
56.6
78.6.0..0
70.4.10.0
C. Ties-Merging
69.6+::.5
70.343.5
65.342.9
47.9475
76.1 63.647.11
94.7=0.9
24.4=2.1
77.922.7
72.3+30
OPCM (Ours)
73.1+:0.2
78.3=
82.400.4
80.2:2266
80.8=0.4 80.4
974=0.3
61.6-088
84.8
76.3=119
Method
FER2013 OxfoddlITPPt
STLIO
CIFARIO0 CIFARIO Food101
PashionMNIST EMNIST KMNIST ReaderedSST2
CLIP-WT-8/32
C. Fine Tumed
58.3
68.5
86.7
40.2
70.5
50.0
90.7
72.4
54.5
54.5
Average (SWA)
50.2:4:0.0
97.010.00
69.8:0000
92.740.0 80.41000
71.3-0.0
15.0=0.0
11.5.0.0
61.8:10.0
C. Task Arithmetic
51.4+00.0
82.3=0.0
94.9=0.0
64.6:0000
91 71.9+0.0
73.9=0.0
1788=00
12.2+0.0
59.9=20.0
C. Ties-Merging 49.5:22.9
81.31199
95.2400.5
63.711888
91,2+14 70.2422
73.7=4.5
178=28
16.94::66
59.8:14.1
OPCM (Ours)
58.5:10066
02.9:0.3
95.910.3
67.6.10.7
92810.3 74.010.8
76.3:1
22
18311.
64.61:066
CLJP-WT-B/16
Fine-Tuned
73.6
51.5
Average (SWA) 50.9110.0
89.6.10.0
98.0
72.9.0.0
94.2.000 85.9
73.3
15.6.00.0
12.4.0.00
62.5.10.0
Arithmetic
50.74:0.0
89.3-0.0
97.01000
68.0000
93.1 80.340.0
75.7=0.0
18.1=0.0
16.40000
61.8-0000
C. Ties-Merging
50.4:t6.2
87,9=1.6
96.3=0.9
63.1+48 91.7+199 78.0+22.
75.0=4.0
23.4=8.4
24.9+7.11
61.5=499
OPCM (Ours)
59.5:1:04
9188=022
97.7+0.1
732400
94,7=0.2 83.1::022
81,3=006
26.5=1122
23.4::0.6
66.8:0088
CLIP-WTT
C. Fine-Tuned 68.0
92.1
94.5
60.5
85.7
74.8
93.1
89.0
$9.2
78.8
Average (SWA) $2.7:10.0
94.2:00.0
99.2.0000
97.010.0
90.710.0
16.1.0.0
10.410.0
16.1
C. Task Arithmetic
55.7:100.0
94.2+10.0
98.610000
79.1
9666000 87.6:10.0
80.8-0.0
17.6=0.0
10.6:10.0
63.6-10.0
Ties-Merging 57.6+3..
93.5-088
97.8-00.5
74.0#443
95.6+1.7 84.742.
79.7=2.1
20.2-55
12.6+422
58.4224
OPCM (Ours) 61.8:10.2
95.4=0.1
99.2=000
83.0:10.4 97.8=011 90.9:0022
86.0=0.2
26.4=0.9
14.74222
71.0=1111
Table 6: Test accuracy comparison between continual fine-tuning and OPCM across different sizes of CLIP-VIT models and task sets.
Experimental Setup
Continual Fine-tuning
OPCM (Ours, 10 runs)
VIT-B/32, 8 tasks
79.8
75.5++ Min: 74.8, Max:76.5
ViT-B/16, tasks
82.9
81.8++0.3, Min: 81.3, Max: 82.3
ViT-L/14, tasks
90.0
87.0++0.4, Min: 86.6, Max: 87.7
ViT-B/32, 14 tasks
67.4
71.9+ Min: 71.3, Max: 72.5
ViT-B/16, 14 tasks
72.2
77.1 Min: 76.2, Max: 77.8
ViT-L/14, 14 tasks
70,9
83,5 Min: 83.1, Max: 83.8
ViT-B/32, 20 tasks
62.6
65.7+ Min: 65.3, Max: 66.0
ViT-B/16, 20 tasks
68.2
70,3 Min: 69.8, Max: 70.7
ViT-L/14, 20 tasks
77.7
76.0 Min: 75.6, Max: 76.4
20-30 minutes of fine-tuning. A single complete run, therefore, takes 8-10 hours. Here we report continual fine-tuning results from single runs, the comparison with our method's statistical analysis reveals compelling insights: In 5 out of 9 experimental settings (highlighted in bold), even our method's minimum performance exceeds continual fine-tuning's 's single-run results. the remaining cases, method still achieves competitive results within 1-5% of continual fine-tuning. OPCM
36

---

achieves the performance with orders of magnitude lower computational cost, requiring only a few seconds versus 20-30 minutes per task for continual fine-tuning.
D.4 Ablation Studies
CLIMAIT medds tadk Arthoatic
ronkl taka eday Antiaale
CLIP-VIT raakb take edia Tack Artieaai
VIT-0-32
VIT-8-22
VITRII
VITAAI
VITU/1
VISU14
VITU
Acorway
Zox3bet
02
0.4
02
0.5
0.5
Fattor
Scaling
Factor
Figure 13: The average accuracy of Task Arithmetic with varying scaling factors.
0.74-
0.70
0.625
8.72
9.48
8.66
Our
Strels Average
0.64
Average
0.53
Steple Avcrge
Ted Aitrreds
Ted Nittrrete
Thdl Adtinaatc
0.22
0/4
02
64
04
Prjeciin Eredoki
(a) VTT I/3- on eight tasks
(b) VTT- on 14 tasks
(c) ViT- 20 tasks
NTE-876 Hiraks
Wi en 2hrass
775
0.50
055
0.75
8.60
735
Durt
0.393
Dor
0.74
Single Arcrage
Sizple Arcrage
Antonate
0.555
Antarat:
0.72
1.5
05
1.5
(d) ViT-B/16 on tasks
(e) VIT-B/16 on 14 tasks
(f) ViT-B/16 on 20 tasks
VIBLU14 cight tor
en
0.152
0.34
0.33
0.75
0.2
0.533
Single
0.53
SeepleAvenge
64
0.6
0.5
1.5
02
04
96
05
1.0
62
0.4
06
05
Pojecion feedeili
Projetion Soedkille
(g) ViT-L/14 on 8 tasks
(h) ViT-L/14 on 14 tasks
(i) ViT-L/14 on 20 tasks
Figure 14: The effect of the projection threshold a on the performance of the merged CLIP-VIT models.
In this section, we provide additional details the ablation studies. We perform grid search to systematically evaluate the scaling factor for Task Arithmetic and the projection threshold a for our proposed method. This analysis is conducted across range of model architectures, including VIT-B/32, ViT-B/16, and ViT-L/14, as well as diverse task sets comprising 8, 14, and 20 tasks. The hyperparameter search space is as follows:
Scaling factor for Task Arithmetic: (0.0 (zero-shot), 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0).
Projection threshold a for our proposed method: (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0].
37

---

Figure 13 demonstrates the impact of adjusting the scaling factor in Task Arithmetic across various model architectures (ViT-B/32, ViT-B/16, and ViT-L/14) and task sets (8, 14, and 20 tasks). The findings reveal that model performance is highly dependent on the choice of scaling factor, with distinct optimal values identified for different task sets. Specifically, scaling factor 0.3 yields the best results for eight tasks, while factor of 0.1 proves optimal for both 14 and 20 tasks. These results highlight the importance of carefully selecting the scaling factor to maximize performance in multi-task learning scenarios.
Figure 14 shows the influence of the projection threshold on the merged model performance. revealing several key insights. First, the optimal value of consistently lies within the range of 0.4 0.6, regardless of the number of tasks. Second, performance remains remarkably stable within this range, highlighting the robustness of our method to slight variations in the choice of a. Third, as the number of tasks increases from 20, the optimal range for a remains similar, demonstrating its scalability across different task complexities. Fourth, when compared to baseline methods such as Task Arithmetic, our approach consistently outperforms them across broad spectrum of a values (a < 0.9). These findings underscore the effectiveness of our method in facilitating efficient knowledge transfer and mitigating catastrophic forgetting, making a reliable solution for continual multi-task model merging.
E
Comparison With SOTA Conventional Model Merging Methods
Table 7: Comparison with conventional model merging methods on eight CLIP-ViT-B/32 models.
Method
SUN397
Cars
RESISC4S EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg.
Conventioncl Model Merging
Fisher Merging
66.7
64.0
72.2
91.6
69.0
64.3
83.5
53.7
70.6
RegMcan
67.8
68.9
82.5
94.4
90.6
79.2
97.6
63.2
80.5
Task Arithmetic
57.1
64.9
76.7
77.9
96.1
68.0
Ties-Merging
67.1
64.2
74.1
76.8
77.7
69.4
94.1
54.0
72.2
Task-wise AdaMerging
58.6
56.9
69.8
82.4
70.3
58.9
97.2
55.3
68.7
Layer-wise AdaMerging
67.9
71.3
83.5
92.7
87.4
92.9
98.2
67.0
82.6
TSVM
67.6
71.6
84.7
93.4
92.5
98.9
63.8
83.1
Continual Model Merging
C. Layer-wise AdaMerging
62.2
64.6
71.1
70.6
91.2
94.5
98.4
35.3
73.5
C. TSVM
59.9
59.5
64.0
64.7
87.9
86.1
99.5
34.3
68.5
Ours (10 runs)
66.2:10.5
63.5:10.5
78.0:12.2
86.51:19
82.41:1.3 74.31 1.4 96.9.10.7
7 16.41::.3
75.5.00.5
In this study. we introduce the orthogonal projection-bused continual merging (OPCM) method for scenarios where models become available sequentially over time rather than simultaneously. While conventional Model Merging such AdaMerging assumes all expert models are available simultaneously. For a more comprehensive comparison, we also list the experiments results of some SOTA model merging methods eight results demonstrate that OPCM achieves superior performance compared to most conventional model merging approaches, despite operating under the more challenging continual setting where expert models are not simultaneously available, making it harder to resolve inter-task knowledge conflicts.
models in Table Our experimental
We also conducted experiments AdaMerging (96] and TSVM (3) to compare its performance under the continual model merging setting. Specifically, we adapt the AdaMerging method
and TSVM to the continual model merging setting by setting 0 (00) merged
((0), mergedd
mergea: 1(++1)) and /eereed TSV g(() mergedd ((t++)), respectively. The results are shown in Table 7 The continual layer-wise AdaMerging and TSVM experiment was conducted with a single run using the following task sequence: SUN397, Cars, RESISC45, ATroSAT, SVHN, GTSRB, MNIST, and DTD. The results in the table demonstrate that AdaMerging in the con- tinual merging setting achieves 73.5% average accuracy, which is notably lower than its performance in the conventional model merging setting (82.6%).
AdaMerging(Ous
glt)
38


Reviews:
[{"review_id": "27O6c2BBNx", "reviewer": "Reviewer_2FXy", "summary": "The authors tackle the problem of merging fine-tuned models in a sequential setting where models arrive over time without retraining or revisiting data. Unlike prior approaches that rely on simultaneous access to all models and simple weight averaging, the authors propose a projection-based method that merges each new model by projecting its updates orthogonally to the subspace of previously merged updates, with an adaptive scaling mechanism to control drift. This setup helps mitigate task interference and maintain performance across tasks. The method is simple, memory-efficient (constant in the number of models), and shows strong empirical results on CLIP-ViT, outperforming continual baselines by 5–8% on average", "questions": "How is this different than the works I listed about from the orthogonality contribution. \n\nIn addition, what about this paper:\n\nhttps://arxiv.org/html/2505.12082v1", "limitations": "Does having the weights be projected orthogonal to each other cause limitations - lack of sharing useful information from different tasks?", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "What stands out in this paper is how it reframes model merging as a continual process where models arrive one at a time rather than being merged all at once. That feels like a much more realistic and practical setting, especially for real-world systems where tasks evolve over time. The idea is clean and well-motivated. The method itself is simple, does not require retraining, and keeps memory use low, which makes it easy to apply. The results are strong and consistent across model sizes and task counts, and it's good to see that performance holds up even when task order changes.\n\n\nThe technical novelty is on the lighter side. Using orthogonal projections to reduce interference has been done before in the continual learning literature. Adaptive scaling makes sense in this context but is also a pretty natural move. The method is well-executed, but it mostly repurposes existing ideas for the merging setting rather than introducing something fundamentally new. It would help to better understand what is specific to merging here that is not already covered in related projection-based continual learning approaches.\n\nhttps://arxiv.org/abs/2301.12131\nhttps://openreview.net/pdf?id=hac6DzbMa7", "was_revised": true, "final_justification": "I think due to the lack of clear novelty or strong empirical results. I decide to hold my score.", "rebuttal": "We thank the reviewer for the thoughtful feedback.\nWe will revise the manuscript to include the following clarifications on the points raised, and cite the relevant works listed by the reviewer.\n\n## Response to weaknesses: distinction from existing projection-based continual learning methods\n\n> Using orthogonal projections to reduce interference has been done before in the continual learning literature. Adaptive scaling makes sense in this context but is also a pretty natural move. The method is well-executed, but it mostly repurposes existing ideas for the merging setting rather than introducing something fundamentally new. It would help to better understand what is specific to merging here that is not already covered in related projection-based continual learning approaches. [1,2]\n\nWhile orthogonal projections have been used in continual learning, our approach differs fundamentally from existing methods in the following ways:\n\nProjection of gradients in continual learning [1,2] is different from the projection operation in the proposed method in their objectives and mechanisms.\n\n1. In OPCM (our proposed method), we focus on combining multiple fine-tuned models, each specialized for a specific task, into a single multi-task model. The projection in OPCM operates on the parameter deltas (task vectors) between a fine-tuned model and the base pre-trained model. In contrast, projection-based continual learning methods are used during the training process of a single model to learns multiple tasks sequentially.\n2. The projection in OPCM is performed after the fine-tuning of each model, while continual learning methods apply projections during the training process, which requires access to the original training data and is computationally expensive.\n3. For projection-based continual learning methods such as OGD, due to the requirement of storing the gradients of previous tasks, the memory complexity is $O(Td)$, where $T$ is the number of tasks and $d$ is the size of the model. In contrast, OPCM maintains a single merged model, which represents the cumulative results of merging the pre-trained model with task-specific models up to task $t$. The memory complexity of OPCM is $O(d)$.\n4. The computational mechanism of projection in OPCM is different from that in projection-based continual learning methods. In OPCM, we use SVD on the accumulated merged weights $\\Delta W_{\\text{merged}}^{(t-1)}$ to identify the important subspace, and then project the new task vector $\\Delta W_t^{(t)}$ orthogonally to this subspace to avoid interference with previously learned tasks.\nThe projected weights are computed as:\n$$\\mathcal{P}^{(t-1)}\\_\\alpha\\left(\\Delta W^{(t)}\\right) = \\sum\\_{i,j=r_{\\alpha}, i\\neq j}^{m, n} {\\left\\langle \\Delta W^{(t)}, u_i v_j^T \\right\\rangle}\\_F u_i v_j^T,$$\nIn contrast, projection-based continual learning methods typically rely on the stored gradients or other representations from previous tasks.\nThe projected gradients are computed as:\n$$g_t - \\mathcal{P}\\_{\\text{span}(g_1, \\ldots, g_{t-1})}(g_t)$$\n\n| Aspect | Continual Learning Methods | Our Method (OPCM) |\n| --- | --- | --- |\n| Setting | Model Merging | Continual Learning |\n| Operation | Gradients during training | Post-training, task vectors during merging |\n| Requirements | Training data required | No data access and training-free |\n| Memory | $O(Td)$, where $T$ is the number of tasks | $O(d)$, constant |\n| Mechanism | Uses SVD on the accumulated merged task vector to identify subspace | Orthogonal to past gradients or subspaces |\n\n## Response to questions: how is this different from [3]\n\n> How is this different than the works I listed about from the orthogonality contribution.\n>\n> In addition, what about this paper:\n>\n> https://arxiv.org/html/2505.12082v1 [3]\n\nThe proposed OPCM merges fine-tuned models on different downstream tasks after the training phase, which is entirely post-training. While the method introduced in [3] applies model merging during pre-training, requiring access to sequential checkpoints from the training trajectory.\n\nAnother key difference between these two settings is that OPCM merges fine-tuned models post-training using orthogonal projections to preserve task-specific performance, limiting direct task-specific information sharing to avoid interference.\nWhile for model merging in the pre-training phase, such as simple moving averaging ($M_{avg} = \\frac{1}{N}\\sum_{i=1}^{N} M_i$) and exponential moving averaging ($M_{avg}^{(i)} = \\alpha M_i + (1 - \\alpha) M_{avg}^{(i-1)}$), sequential checkpoints are from the same training trajectory, the divergence between the models is not as large as that for task-specific model merging, thus allowing smoother integration of knowledge across checkpoints. \nAveraging during pre-training smooths the parameter trajectory, which can enhance the robustness and generalization by reducing overfitting to specific training iterations or data batches.\n\n## Response to limitation: The limitation of orthogonal projection\n\n> Does having the weights be projected orthogonal to each other cause limitations - lack of sharing useful information from different tasks?\n\nYes, projecting weights to be orthogonal to each other, as done in OPCM, can sometimes cause limitations in the direct sharing of useful information across tasks. However, this limitation is a deliberate trade-off that brings advantages, such as preventing interference and reducing knowledge redundancy.\n\n1. **Reduced direct information sharing**: In OPCM, the goal is to merge multiple pre-trained models sequentially while preserving performance on earlier tasks. To achieve this, task-specific updates (or task vectors) are projected onto a subspace orthogonal to the important parameter directions of previously merged models.\nThis orthogonality means that a new task cannot adjust the model’s parameters in ways that overlap with or build upon the protected directions of prior tasks. For example, if Task A learns a feature that could also benefit Task B, Task B’s updates cannot directly modify or enhance that feature if it lies in a protected direction. As a result, the model may miss opportunities to leverage task-specific knowledge from one task to improve performance on another.\n\n2. **Preventing knowledge redundancy**: While orthogonality imposes this limitation, it also offers a significant advantage: it prevents knowledge redundancy. In a non-orthogonal approach, different tasks might encode similar features or information in their updates, duplicating effort and wasting the model’s capacity.\nFor instance, if two tasks require recognizing similar low-level patterns, they might redundantly adjust parameters to represent that knowledge independently.\nIn OPCM, the base model provides a foundation of shared features accessible to all tasks, and the orthogonal projection ensures that each task’s updates are unique and keep the merged model lying in a low-loss basin that is common to all tasks.\nBy avoiding duplication, OPCM helps maintain the model’s generalization ability across tasks and prevents it from being pushed into a suboptimal region of the parameter space.\n\nTo summarize, by enforcing orthogonality, OPCM minimizes interference, knowledge redundancy, and catastrophic forgetting, helping to ensure that adding a new task does not significantly degrade performance on earlier ones.\n\n[1] Yang et al. - 2023 - Restricted Orthogonal Gradient Projection for Continual Learning.  \n[2] Wang et al. - 2023 - Continual Learning with Orthogonal Weights and Knowledge Transfer.  \n[3] Li et al. - 2025 - Model Merging in Pre-training of Large Language Models."}, {"review_id": "NgvspK12Am", "reviewer": "Reviewer_kfjD", "summary": "The paper introduces OPCM, a training‑free, continual model merging method. It sequentially integrates new fine‑tuned models into a merged model via orthogonal projections and adaptive scaling, reducing interference and maintaining constant memory per merge.", "questions": "- 1. In classification settings, the method does not consider the classification head, and the tasks used appear to be relatively similar. As a result, for Continual Model Merging, there is a possibility that only a few models dominate the merged result, leading to low variance and insensitivity to task order.\n   - Does the use of simple classification tasks limit the persuasiveness of the method for Continual Model Merging?\n   - Would it be more meaningful to evaluate the method on more diverse tasks with shared objectives, such as language generation, where the differences between tasks are more pronounced?\n- 2. Why is the accuracy of the single EuroSAT task in Figure 12(a) significantly lower than the accuracy when MNIST and EuroSAT are merged?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths: \n- 1. The paper is clearly written and addresses a novel and timely problem in model merging.\n\n- 2. The authors conduct comprehensive experiments to demonstrate the effectiveness of their method.\n\nWeaknesses:\n- 1. In Table 1, the results for C. finetune lack variance/error bars.\n- 2. The extremely low variance of accuracy when using SWA in Table 1 suggests the tasks are highly similar, and possibly only a few models dominate the merged performance. To strengthen the claim of order-robustness, the authors should consider testing on tasks with greater diversity.", "was_revised": true, "final_justification": "The authors addressed my concerns, but limitations raised by other reviewers remain. I keep my score: boardline accept.", "rebuttal": "We thank the reviewer for the thoughtful feedback and suggestions.\n\n## Response to weakness 1\n\n> In Table 1, the results for C. finetune lack variance/error bars.\n\nWe appreciate the reviewer's attention to statistical rigor. We chose not to include variance/error bars for continual fine-tuning (C. finetune) due to computational constraints rather than oversight.\n\nContinual fine-tuning is extremely computationally intensive. For instance, in our 20-task experiments, each task requires 20-30 minutes of fine-tuning. A single complete run, therefore, takes 8-10 hours. To obtain meaningful variance estimates, we would need multiple runs across different task orderings and random seeds.\n\nGiven that we evaluate three different model sizes (ViT-B/32, ViT-B/16, ViT-L/14) across multiple task configurations (8, 14, and 20 tasks), computing variance for continual fine-tuning would require several months (about 3-5 months for 10 runs) of continuous GPU time. This is prohibitively expensive for our current resources.\n\nAlthough we report continual fine-tuning results from single runs, the comparison with our method's statistical analysis reveals compelling insights:\n\n1. In 5 out of 9 experimental settings (highlighted in **bold**), even our method's minimum performance exceeds continual fine-tuning's single-run results.\n2. In the remaining cases, our method still achieves competitive results within 1-5% of continual fine-tuning.\n3. OPCM achieves the performance with orders of magnitude lower computational cost, requiring only a few seconds versus 20-30 minutes per task for continual fine-tuning.\n\nTable: Comparison of Continual Fine-tuning and OPCM Results.\n\n| Experimental Setup | Continual fine-tuning Avg Acc. | OPCM Avg Acc.(10-runs) |\n| --- | --- | --- |\n| ViT-B/32, 8 tasks  | 79.8 | 75.5$\\pm$0.5, Min: 74.8, Max:76.5  |\n| ViT-B/16, 8 tasks | 82.9 | 81.8$\\pm$0.3, Min: 81.3, Max: 82.3 |\n| ViT-L/14, 8 tasks | 90.0 | 87.0$\\pm$0.4, Min: 86.6, Max: 87.7 |\n| ViT-B/32, 14 tasks | 67.4 | 71.9$\\pm$0.3, Min: **71.3**, Max: 72.5 |\n| ViT-B/16, 14 tasks | 72.2 | 77.1$\\pm$0.5, Min: **76.2**, Max: 77.8 |\n| ViT-L/14, 14 tasks | 70.9 | 83.5$\\pm$0.2, Min: **83.1**, Max: 83.8 |\n| ViT-B/32, 20 tasks | 62.6 | 65.7$\\pm$0.2, Min: **65.3**, Max: 66.0 |\n| ViT-B/16, 20 tasks | 68.2 | 70.3$\\pm$0.2, Min: **69.8**, Max: 70.7 |\n| ViT-L/14, 20 tasks | 77.7 | 76.0$\\pm$0.2, Min: 75.6, Max: 76.4 |\n\n## Response to weakness 2\n\n> The extremely low variance of accuracy when using SWA in Table 1 suggests the tasks are highly similar, and possibly only a few models dominate the merged performance. To strengthen the claim of order-robustness, the authors should consider testing on tasks with greater diversity.\n\nThe extremely low variance observed with SWA in Table 1 is due to the mathematical equivalence of SWA and simple averaging, which we clarify below.\nThe equivalence between SWA and simple averaging implies that SWA is inherently order-agnostic, as the simple average does not depend on the order of the models being processed.\n\n**Proof of equivalence of SWA and simple averaging:**\nLet's denote the parameters of task-specific models as $\\theta^{(1)}, \\theta^{(2)}, \\ldots, \\theta^{(n)}$ for $n$ tasks.\nFor stochastic weight averaging (SWA), the update rule is given by:\n$$\n  \\theta_{\\text{SWA}}^{(1)} = \\theta^{(1)}, \\quad \\theta_{\\text{SWA}}^{(t)} = \\frac{\\theta_{\\text{SWA}}^{(t-1)}(t-1) + \\theta^{(t)}}{t},\n$$\nWe processed by induction:\nFor the first task, the SWA update rule gives;\n$$\n    \\theta_{\\text{SWA}}^{(1)} = \\theta^{(1)}.\n$$\nAssume the formula holds for $t-1$ tasks, i.e.,\n$$\n    \\theta_{\\text{SWA}}^{(t-1)} = \\frac{1}{t-1} \\sum_{i=1}^{t-1} \\theta^{(i)}.\n$$\nAccording to the SWA update rule, we have:\n$$\n    \\theta_{\\text{SWA}}^{(t)} = \\frac{\\theta_{\\text{SWA}}^{(t-1)}(t-1) + \\theta^{(t)}}{t} = \\frac{(t-1) \\cdot \\frac{1}{t-1} \\sum_{i=1}^{t-1} \\theta^{(i)} + \\theta^{(t)}}{t} = \\frac{1}{t}\n    \\sum_{i=1}^{t} \\theta^{(i)}.\n$$\nThus, by induction, the SWA update rule holds for all $t$ tasks.\n\n**Similarity of selected tasks:**\nThe tasks used in our experiments are: SUN397, Stanford-Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, DTD, Oxford_Flowers102, PCAM, FER2013, Oxford-IIIT-Pet, STL10, CIFAR100, CIFAR10, Food101, Fashion_MNIST, EMNIST_Letters, KMNIST, and Rendered-SST2.\nThese tasks span a wide range of domains, including:\n\n- Natural scene classification: SUN397\n- Fine-grained object recognition: Stanford-Cars, Oxford_Flowers102\n- Remote sensing: RESISC45, EuroSAT\n- Digit and character recognition: MNIST, SVHN, EMNIST_Letters, and KMNIST\n- Traffic sign recognition: GTSRB\n- Texture classification: DTD\n- Medical imaging: PCAM\n- Facial expression recognition: FER2013\n- Pet breed classification: Oxford-IIIT-Pet\n- General object recognition: CIFAR10, CIFAR100, STL10, Food101\n- Fashion product classification: Fashion_MNIST\n- Sentiment analysis: Rendered-SST2\n\nAs shown in Figures 4,8,9, and Figure 10, the cosine similarity between task vectors is generally low, empirically indicating that the tasks are indeed diverse and the learned representations are approximately orthogonal.\n\n## Response to question 1\n\n> In classification settings, the method does not consider the classification head, and the tasks used appear to be relatively similar. As a result, for Continual Model Merging, there is a possibility that only a few models dominate the merged result, leading to low variance and insensitivity to task order.\n>\n> - Does the use of simple classification tasks limit the persuasiveness of the method for Continual Model Merging?\n> - Would it be more meaningful to evaluate the method on more diverse tasks with shared objectives, such as language generation, where the differences between tasks are more pronounced?\n\n**Classification head in experiments:**\nIn our experiments, we follow the common practice in model merging literature of using zero-shot classification heads initialized with the pre-trained CLIP model's text encoder. Specifically, we construct task-specific classification heads using CLIP's text encoder to generate zero-shot weights for each task's class names and templates.\nWhen fine-tuning the vision encoder, the classification head remains fixed. This is meant to preserve the open-vocabulary nature of the CLIP model, so it can still handle unseen classes during inference.\n\n**On task similarity and robustness:**\nThe tasks we selected for our experiments are diverse, covering various domains such as natural scene classification, as clarified above in the response to weakness 2.\n\nOn the other hand, as shown in Figures 4,8,9, and Figure 10, the cosine similarity between task vectors is generally low, empirically indicating that the tasks are indeed diverse and the learned representations are approximately orthogonal.\n\nFrom the experiments, we observe that (1) the task vectors are roughly orthogonal (Figure 4 and Figure 10), and (2) the proposed method remains robust to task order (Table 1 and Figure 5).\n\n**Experimental results on language generation tasks:**\n\nWe conduct experiments on language generation tasks on Flan-T5-base models on eight tasks from the GLUE benchmark. The results are shown below:\n\nTable: Results on Language Generation Tasks\n\n| Method | CoLA | MNLI | MRPC | QNLI | QQP | RTE | SST2 | STSB | Avg.$\\uparrow$ | BWT$\\uparrow$ |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Fine-tuned | 69.1 | 82.7 | 85.5 | 90.9 | 84.0 | 84.4 | 92.9 | 87.4 | 84.6 | - |\n|\n| Averaging (SWA) | 69.1 | 62.6 | 79.4 | 89.8 | **83.9** | 81.2 | 91.7 | 73.2 | 78.8$\\pm$0.0 | -2.5$\\pm$1.1 |\n| Continual task arithmetic | 70.5 | 57.8 | 78.4 | **90.2** | 83.6 | 80.5 | **92.3** | 77.8 | 78.9$\\pm$0.0 | -1.8$\\pm$1.1 |\n| Continual ties-merging | 68.8 | 50.5 | **79.8** | 89.9 | 83.1 | 79.2 | 91.9 | **79.3** | 77.8$\\pm$1.6 | -0.5$\\pm$1.9 |\n| **OPCM(Ours)** | **69.7** | **72.9** | 78.8 | **90.2** | 83.8 | **82.2** | **92.3** | 74.7 | **80.6**$\\pm$0.3 | -2.1$\\pm$0.6 |\n\nThe results show that OPCM outperforms the other continual merging methods on average performance across the eight tasks.\n\n## Response to question 2\n\n> Why is the accuracy of the single EuroSAT task in Figure 12(a) significantly lower than the accuracy when MNIST and EuroSAT are merged?\n\nI suppose the reviewer is referring to Figure 11(a) instead of Figure 12(a).\nIf there is a misunderstanding, please let us know.\n\nIn Figure 11, each row represents a \"new task\" being merged in the continual merging process, and each column represents a \"test task\" being evaluated. \nThe diagonal entries show the performance of each task when evaluated on itself, and the off-diagonal entries show cross-task performance (previous tasks).\n\nFor example, in Figure 11(a), the first task is SUN397 (at t=1), we evaluate the performance of the merged model ($\\theta\\_{merged}^{(1)}$) on SUN397.\nFor $t>1$, the merged model ($\\theta\\_{merged}^{(t)} =ContinualMerging(\\theta\\_{merged}^{(t-1)}; \\theta^{(0)}, \\theta^{(t)})$, where $\\theta^{(0)}$ is the pre-trained model), is evaluated on the current task and all previous tasks.\n\nTable: Test Accuracy Matrix for ViT-B/32 on 8 Tasks (Figure 11a)\n\n| **step** | New Task | SUN397 | GTSRB | Cars | DTD | RESISC45 | SVHN | MNIST | EuroSAT |\n|---|---|---|---|---|---|---|---|---|---|\n|t=1| SUN397 | 0.75 | -     | -    | -   | -        | -    | -     | -       |\n|t=2| GTSRB | 0.70 | 0.96  | -    | -   | -        | -    | -     | -       |\n|t=3| Cars | 0.69 | 0.87  | 0.69 | -   | -        | -    | -     | -       |\n|t=4| DTD | 0.68 | 0.79  | 0.68 | 0.58| -        | -    | -     | -       |\n|t=5| RESISC45 | 0.67   | 0.67  | 0.66 | 0.55| 0.77     | -    | -     | -       |\n|t=6| SVHN | 0.67 | 0.64  | 0.65 | 0.53| 0.74     | 0.70 | -     | -       |\n|t=7| MNIST | 0.66 | 0.59  | 0.64 | 0.52| 0.71     | 0.70 | 0.89  | -       |\n|t=8| EuroSAT | 0.65 | 0.55  | 0.62 | 0.51| 0.71     | 0.65 | 0.86  | 0.76    |"}, {"review_id": "h17mzWWiie", "reviewer": "Reviewer_coXd", "summary": "The authors propose a method for continual model merging, where new models arrive over time and the old models are not available anymore other than via the current merged checkpoint.\nThe method relies on projections that keep the incoming models' task vector and the current merged model orthogonal as well as adaptive scaling.\nThe authors show that the method outperforms other baselines for continual model merging on a variety of tasks and provide theoretical analyses of their method.", "questions": "- Could you please explain how task information is exactly guaranteed to be preserved in Corollary 5.3? Initially it seems like a trivial rewriting of the merging equation that does not provide any guarantees.\n\n- Could you please expand a bit on the motivation for continual model merging? What are the uses cases where checkpoints could not be kept? I think clarifying this also in the paper could help motivating the method.\n\n- How did you choose the scaling for e.g. Task Arithmetic in Table 6?", "limitations": "The paper discusses some limitations, such as, that the order matters in the method and that SVD adds additional overhead.\nHowever, the limitations could be extended by discussing the results in Appendix E that show that conventional model merging techniques (that are not continual in this works' sense) can outperform the proposed method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- The paper is easy to read and follow.\n\n- The method proposed by the authors outperforms other baseline techniques for continual model merging on a variety of datasets.\n\n## Weaknesses\n\n- The motivation of memory complexity increasing with the number of models is of course true but this can also be simple disk storage which is easy to get by or even cloud storage which makes the task setting overall a bit artificial.\n\n- I'm not sure if claiming \"we present a *novel* perspective on multi-task model merging through the lens of \ncontinual learning\" (L96-97) is fair, as the formalization shares many similarities with Dziadzio et al. 2024, which is nevertheless also discussed and cited. \n\n- The adaptation of Task Arithmetic in Eq. 5 is not consistent with theoretical derivations of Task Arithmetic, e.g., from Daheim et al. 2024, as the task vector is calculated wrt. a different model than the one it is added too, which could be discussed more.\n\n- The work should discuss other papers that use SVD to reduce Task interference more, such as Gargiulo et al. 2025.  Perhaps a comparison could also be added.\n\n- The fact that the method is outperformed by other merging techniques like RegMean if the restriction on continually merging and not keeping all checkpoints is lifted is hidden in Appendix E which is not referenced in the main paper. It would be more appropriate to discuss and reference it there.\n\n## References\n\nDaheim, Nico, et al. \"Model merging by uncertainty-based gradient matching.\" ICLR, 2024.\n\nGargiulo, Antonio Andrea, et al. \"Task singular vectors: Reducing task interference in model merging.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "was_revised": true, "final_justification": "The authors have resolved most of my weaknesses, namely, W2-4.\nIn their response, they provide new experiments and writing as a comparison to prior, very related methods which I believe will find their way into the paper\n\nWhile I believe that W1 is still true and the task setting is a bit artificial I don't think that it warrants recommending rejecting in itself.\nI think it would still be good to address W5, too, which show that there are other methods that can work better in some settings.\nIn my opinion, this is not necessarily a big weakness in itself but should not be hidden in the Appendix.\n\nAlso considering the discussion of the authors with other reviewers I have therefore decided to up my score to 4.", "rebuttal": "We thank the reviewer for their thoughtful feedback and constructive suggestions.\n\n## Response to weakness 1 and question 2\n\n> **W1:** The motivation of memory complexity increasing with the number of models is of course true but this can also be simple disk storage which is easy to get by or even cloud storage which makes the task setting overall a bit artificial.\n\n> **Q2:** Could you please expand a bit on the motivation for continual model merging? What are the uses cases where checkpoints could not be kept? I think clarifying this also in the paper could help motivating the method.\n\nWhile disk storage is indeed available, the continual model merging setting is motivated by practical situations where models are frequently updated—often by different teams or individuals—and earlier versions may not be retained or accessible.\n\nBesides, the $O(T|\\theta|)$ complexity isn't just about storage, it also affects the computational overhead during merging.\n\n## Response to weakness 2\n\n> I'm not sure if claiming \"we present a novel perspective on multi-task model merging through the lens of continual learning\" (L96-97) is fair, as the formalization shares many similarities with Dziadzio et al. 2024, which is nevertheless also discussed and cited.\n\nWe appreciate the reviewer's comment, and we acknowledge that our work shares some similarities with Dziadzio et al. 2024 (\"How to Merge Your Multimodal Models Over Time?\").\nHowever, our work presents a distinct perspective on continual model merging and framework, particularly in the following ways:\n\n1. The framework of Dziadzio et al. 2024 is different from ours, as shown in the following:\n```\nDziadzio et al. 2024 (models are continually trained, and the set of models is growing over time):\n\na set of models  ---(append)--> new set of models --(append)--> new set of models ...\n    🡫                 🡡              🡫                🡡.           🡫\n  merge-1 -(train)-> task-1        merge-2 -(train)-> task-2       merge-3 -(train)-> task-3 ...\n```\n```\nOurs (all models are independently trained and merged in a continual manner):\npre-trained --🡫--------🡫---------🡫 ...\n            task-1  -> task-2  -> task-3 ...\n              🡫        🡫         🡫\n            merge-1 -> merge-2 -> merge-3 ...\n```\n- In Dziadzio et al. 2024, the set of models is continually growing over time, and the merging is performed on the entire set of models at each step.\n- In our work, the models are independently trained and merged in a continual manner, where each task is merged with the previous tasks' merged model.\n2. The theoretical analysis in Dziadzio et al. 2024 and ours are focused on different aspects:\n   - Dziadzio et al. 2024 focus on the initialization of the merging operation, i.e., how to select the set of models to be merged.\n   - In our work, we focus on analyzing the merging operation itself, i.e,. how to reduce task interference during merging. We propose a projection-based continual model merging method that ensures the task vectors remain orthogonal, which is a key aspect of our contribution.\n3. What's more, Dziadzio et al. 2024  is a contemporaneous work; at the time of writing our manuscript, it was not yet available. We have included a discussion of this work in a revised version of the manuscript.\n\n## Response to weakness 3\n\n> The adaptation of Task Arithmetic in Eq. 5 is not consistent with theoretical derivations of Task Arithmetic, e.g., from Daheim et al. 2024, as the task vector is calculated wrt. a different model than the one it is added too, which could be discussed more.\n\nEq.(5) in our manuscript  is equivalent to the original Task Arithmetic formulation, as shown below:\n$$\n\\theta\\_{\\text{merged}}^{(t)} = \\theta\\_{\\text{merged}}^{(t-1)} + \\lambda (\\theta^{(t)} - \\theta^{(0)}), \\quad Eq.(5) (\\text{continual task arithmetic})\n$$\nand the original Task Arithmetic formulation is:\n$$\n\\theta_{\\text{merged}} = \\theta_{\\text{0}} + \\lambda \\sum_{i=1}^{t} (\\theta^{(i)} - \\theta^{(0)}), \n$$\nwhere $\\theta^{(0)}$ is the pre-trained weights.\n\n*proof:*\nWe prove the equivalence by induction:\n1. For the first task, we have:\n$$\\theta_{\\text{merged}}^{(1)} = \\theta_{\\text{0}} + \\lambda (\\theta^{(1)} - \\theta^{(0)}).$$\n2. Assume the formula holds for $t-1$ tasks, i.e.,\n$$\\theta_{\\text{merged}}^{(t-1)} = \\theta_{\\text{0}} + \\lambda \\sum_{i=1}^{t-1} (\\theta^{(i)} - \\theta^{(0)}).$$\n3. According to Eq.(5), we have:\n$$\\theta_{\\text{merged}}^{(t)} = \\theta_{\\text{merged}}^{(t-1)} + \\lambda (\\theta^{(t)} - \\theta^{(0)}) = \\theta_{\\text{0}} + \\lambda \\sum_{i=1}^{t-1} (\\theta^{(i)} - \\theta^{(0)}) + \\lambda (\\theta^{(t)} - \\theta^{(0)}) = \\theta_{\\text{0}} + \\lambda \\sum_{i=1}^{t} (\\theta^{(i)} - \\theta^{(0)}).$$\nThus, by induction, the formula holds for all $t$ tasks.\n\n## Response to weakness 4\n\n> The work should discuss other papers that use SVD to reduce Task interference more, such as Gargiulo et al. 2025. Perhaps a comparison could also be added.\n\nThank you for this valuable feedback. Gargiulo et al. 2025 (\"Task Singular Vectors: Reducing Task Interference in Model Merging\") introduce task singular vectors merging (TSVM) that addresses task interference in model merging through SVD decomposition. TSVM reduces task interference by whitening the TSVs after SVD decomposition across all tasks simultaneously. In contrast, our method reduces interference by projecting new task vectors orthogonally to prior merged vectors at each step. Below is an experimental comparison conducted on the 8-task setting using ViT-B/32 models:\n\nTable: Experimental comparison of TSVM and our method on an 8-task setting.\n\n| Method | Sun397 | Stanford Cars | RESISC45 | EuroSAT | SVHN | GTSRB | MNIST | DTD | Avg. |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| TSVM (conventional merging) | 67.6 | 71.6 | 84.7 | 93.4 | 91.9 | 92.5 | 98.9 |63.8 | 83.1 |\n|\n| TSVM (continual merging) | 59.9 | 59.5 | 64.0 | 64.7 | 87.9 | 86.1 | 99.5 | 34.3 | 68.5 |\n| **OPCM (Ours, 10 runs)** | 66.2±0.5 | 63.5±0.5 | 78.0±2.2 | 86.5±1.9 | 82.4±1.3 | 74.3±1.4 | 96.9±0.7 | 56.4±1.3 | 75.5±0.5 |\n\nFor TSVM under continual merging, we use the task order of SUN397, Stanford Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD. \nAnd the update rule is given by:\n$$\n\\theta^{(1)}\\_{\\text{merged}} = \\theta^{(1)}, \\quad \\theta^{(t)}\\_{\\text{merged}} = \\text{TSVM}(\\theta^{(t-1)}\\_{\\text{merged}}, \\theta^{(t)}; \\theta^{(0)}), \\quad t>1.\n$$\n\n## Response to question 1\n\n> Could you please explain how task information is exactly guaranteed to be preserved in Corollary 5.3? Initially it seems like a trivial rewriting of the merging equation that does not provide any guarantees.\n\n$$\nW^{(t)}\\_{\\text{merged}} - W^{(0)} = \\frac{1}{\\lambda(t)} \\sum_{i=1}^{t} P^{(i-1)}\\_{\\alpha}(\\Delta W^{(i)}), \\quad(\\text{Corollary 5.3})\n$$\nThis corollary guarantees that when a new task-specific model is merged, *it doesn't overwrite the subspace spanned by the old tasks*, their contributions are geometrically independent.\n\n## Response to question 3\n\n> How did you choose the scaling for e.g. Task Arithmetic in Table 6?\n\nWe performed a grid search over the scaling factor for Task Arithmetic, evaluating values in [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9] on the test set of each task.\nThe scaling factor is selected based on the best average performance across all tasks, which is 0.3 for the eight-experiment setting in Table 6."}, {"review_id": "5SqRKeH10l", "reviewer": "Reviewer_Eirv", "summary": "This paper proposes a novel continuous model merging method, named OPCM. Unlike traditional model merging approaches that require access to all models simultaneously, OPCM processes models one-by-one using orthogonal projections and adaptive scaling. The method maintains constant memory complexity and mitigates task interference. Extensive empirical evaluations using CLIP-ViT architectures on up to 20 image classification tasks show that OPCM outperforms strong baselines in both average accuracy and backward transfer (BWT), while remaining robust to task orderings.", "questions": "See weakness.", "limitations": "No. This paper uses the assumption that projecting the task vectors in the orthogonal space can cause less conflict to old tasks, which may not be valid for some scenarios.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths.\n+ This paper is clearly written and easy to follow.\n+ This paper introduces a new setting for model merging by incorporating the idea of continual learning.\n+ Extensive experiments on various architectures and tasks show the effectiveness of the proposed method.\n\nWeaknesses.\n+ The paper proposes to solve the continuous model merging problem, where old task vectors cannot be stored. However, in practice, storing task vectors is usually not expensive, making this assumption somewhat questionable. So, I think this problem may not be a significant setting for model merging.\n+ This paper uses the assumption that projecting the task vectors in the orthogonal space can cause less conflict with old tasks. Does this assumption always hold? Maybe some important dimension of old task vectors is zero just simply because the pretrained weights are good enough.", "was_revised": true, "final_justification": "After the rebuttal, most of my concerns are addressed. Overall, I think this is a good paper, even though the proposed Continual Model Merging problem does not seem significant to the model merging community. However, considering that this paper is the first paper that proposed CMM, I would like to give a Weak Accept.", "rebuttal": "Thank you for your thoughtful feedback. We’re grateful for the time and effort you’ve dedicated.\n\n## Response to Weakness 1\n\n> The paper proposes to solve the continuous model merging problem, where old task vectors cannot be stored. However, in practice, storing task vectors is usually not expensive, making this assumption somewhat questionable. So, I think this problem may not be a significant setting for model merging.\n\nWhile disk storage is indeed available, the continual model merging setting is motivated by practical situations where models are frequently updated—often by different teams or individuals—and earlier versions may not be retained or accessible.\n\nBesides, the $O(T|\\theta|)$ complexity isn't just about storage, it also affects the computational overhead during merging.\n\n## Response to Weakness 2\n\n> This paper uses the assumption that projecting the task vectors in the orthogonal space can cause less conflict with old tasks. Does this assumption always hold? Maybe some important dimension of old task vectors is zero just simply because the pretrained weights are good enough.\n\nThis is an excellent point that deserves deeper analysis. In our experiments across 20 tasks, we empirically observed that the orthogonal projection outperforms the continual ties-meriging method, which does not use orthogonal projection. This suggests that the orthogonal projection is indeed beneficial in reducing task interference.\n\nBesides conflict reduction, the projection operation also offers a significant advantage: it prevents knowledge redundancy. In a non-orthogonal approach, different tasks might encode similar features or information in their updates, duplicating effort and wasting the model’s capacity.\nFor instance, if two tasks require recognizing similar low-level patterns, they might redundantly adjust parameters to represent those knowledge independently.\nIn OPCM, the base model provides a foundation of shared features accessible to all tasks, and the orthogonal projection ensures that each task’s updates are unique and keep the merged model lie in a low-loss basin that is common to all tasks.\nBy avoiding duplication, OPCM helps maintain the model’s generalization ability across tasks and prevents it from being pushed into a suboptimal region of the parameter space.\n\n> Maybe some important dimension of old task vectors is zero just simply because the pretrained weights are good enough.\n\nThank you for raising this point.\nRecall that in OPCM we compute an SVD of the old cumulative update\n$$\n\\Delta W_{\\rm merged}^{(t-1)} = U\\,Σ\\,V^⊤,\n$$\nand then select only the top‑$r$ singular components whose values capture at least an α‑fraction of the total energy:\n$$\n\\sum_{i=1}^r σ_i \\;\\ge\\; α\\;\\sum_{j=1}^{\\min(m,n)} σ_j.\n$$\nAny coordinate *k* for which $\\Delta W_{\\rm merged}^{(t-1)}$ has zero entry will contribute a singular value $σ_k=0$, and so its corresponding singular vectors $(u_k,v_k)$ lie outside our top‑$r$ subspace.  \n\nTherefore, If an “important” dimension *k* is zero in $\\Delta W_{\\rm merged}^{(t-1)}$, then in practice that dimension corresponds to a zero singular value and is not selected into the top‑$r$ subspace.  Hence, our orthogonal projection does not eliminate $\\Delta W^{(t)}$’s component along that axis."}]

Author rebuttal:

Meta-review: The paper introduces a continual model merging setting where fine tuned models arrive sequentially and speed of merging and storage are factors. They propose a method based on orthogonal projects with robustness of results to the task order. Experiments highlight the benefits in this setting. While reviewers noted some limited novelty in the projection based method they agreed the formulation of continual model merging is relevant and the experiments well validated
Decision: Accept (poster)