MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging
Authors: Zihuan Qiu
Yi Xu
Chiyuan He
Fanman Meng
Linfeng Xu
Qingbo Wu
Hongliang Li
Venue: NeurIPS 2025 poster ()
Keywords: Continual model merging
Continual learning
Model merging
Primary area: general_machine_learning
TL;DR: A test-time continual model merging framework that leverages low-rank mixture-of-experts and null-space constrained gating to adapt dynamically to new tasks while preserving prior knowledge, all without access to training data.

Abstract: Continual model merging integrates independently fine-tuned models sequentially without access to the original training data, offering a scalable and efficient solution for continual learning. However, existing methods face two critical challenges: parameter interference among tasks, which leads to catastrophic forgetting, and limited adaptability to evolving test distributions. To address these issues, we introduce the task of Test-Time Continual Model Merging (TTCMM), which leverages a small set of unlabeled test samples during inference to alleviate parameter conflicts and handle distribution shifts. We propose MINGLE, a novel framework for TTCMM. MINGLE employs a mixture-of-experts architecture with parameter-efficient, low-rank experts, which enhances adaptability to evolving test distributions while dynamically merging models to mitigate conflicts. To further reduce forgetting, we propose Null-Space Constrained Gating, which restricts gating updates to subspaces orthogonal to prior task representations, thereby suppressing activations on old tasks and preserving past knowledge. We further introduce an Adaptive Relaxation Strategy that adjusts constraint strength dynamically based on interference signals observed during test-time adaptation, striking a balance between stability and adaptability. Extensive experiments on standard continual merging benchmarks demonstrate that MINGLE achieves robust generalization, significantly reduces forgetting, and consistently surpasses previous state-of-the-art methods by 7–9% on average across diverse task orders. Our code is available at: https://github.com/zihuanqiu/MINGLE

MINGLE: Mixture of Null-Space Gated Low-Rank
Experts for Test-Time Continual Model Merging
Zihuan Qiu
Yi Xu²
Chiyuan He' Fanman Meng!"
Linfeng Xu² Qingbo Wu² Hongliang Li
University of Electronic Science and Technology of China, Chengdu, China Dallan University of Technology. Dalian, China (zihuanqiu@std, cyhe@std., fmmeng @ Ifxu@e, qbwu@, hlli@ Juestc.edu.cn, yxu@ dlu..edu.cn
Abstract
Continual model merging integrates independently fine-tuned models sequentially without access to the original training data, offering a scalable and efficient solution for continual learning. However, existing methods face two critical challenges: parameter interference among tasks, which leads to catastrophic forgetting, and evolving test distributions. To address these issues, we introduce the task of Best-Time Continual Model Merging (TTCMM), which leverages a small set of unlabeled test samples during inference to alleviate pa- rameter confficts and handle distribution shifts. We propose MINGLE, a novel framework for TTCMM. MINGLE employs a mixture-of-experts architecture with parameter-efficient, low-rank experts, which enhances adaptability to evolving test distributions while dynamically merging models to mitigate conflicts. To further re- duce forgetting, we propose Null-Space Constrained Gating, which restricts gating updates to subspaces orthogonal prior task representations, thereby suppressing activations old tasks and preserving past knowledge. We further introduce an Adaptive Relaxation Strategy that adjusts constraint strength dynamically based on interference signals observed during test-time adaptation, striking a balance between stability and adaptability. Extensive experiments on standard continual merging benchmarks demonstrate that MINGLE achieves robust generalization, significantly reduces forgetting, and consistently surpasses previous state-of-the-art methods by 7-9% average across diverse task orders. Our code is available at: https: /tttps://gttaiii //github. com comm/ihhunqiuuMMIGG..
limited adaptability
Introduction
Continual learning aims to incrementally adapt machine learning models to new tasks without forget- ting previously learned knowledge, addressing the critical challenge of catastrophic forgetting [43]. However, conventional continual learning approaches typically require continuous access to original training data, raising significant concerns about privacy and substantial computational overhead due to retraining efforts, thus limiting their applicability in dynamic, data-sensitive environments.
To address these limitations, recent works have explored an alternative paradigm known as continual model merging (CMM), which sequentially integrates independently fine-tuned models directly in parameter space, without revisiting any training data [8, 51 [7]_ CMM typically operates under a "merrg--to-transeer" paradigm: given a pretrained model Og and independently fine-tuned models {0: yr 1. unified model is constructed sequentially by combining tpek-specific weight updates AO, 0. 0o via weighted averaging or projection- -ased strategies 24 81 69
*Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Despite its advantages in scalability, data privacy, and distributed training capabilities 44 14 existing CMM methods still encounter critical issues, notably severe parameter interference between tasks and limited adaptability to evolving test distributions. This parameter interference arises because, as fine-tuned models are incrementally merged, overlapping or conflicting parameter updates accumulate, resulting in severe forgetting of previously learned tasks. To mitigate this interference, recent methods introduce structural constraints such as orthogonal projection [70] 78). model linearization [38 (69,, and pruning-based sparsification [86] 90]. However, their effectiveness diminishes as task count grows and interference becomes increasingly entangled. Moreover, models merged across tasks often fail to generalize effectively, particularly when facing unseen shifting task conditions. These flaws result in severe forgetting of earlier tasks and substantial performance gaps compared to the upper bound achieved by individually fine-tuned models. As shown in Fig. [ TA (24] suffers from large performance gaps and strong forgetting, reflected by low accuracy and negative backward transfer. OPCM 770 improves over 'A via orthogonalized merging but still shows notable degradation.
57]
Upper
To overcome these limitations, we propose novel continual merging paradigm--est- Teme-Timm Continual Model Merging (TTCMM)-whicl explicitly in- troduces the concept test-time adaptation (TTA)
100
90
Bocloward Tracefer (%)
Accuracy (%)
80-
-33%
75 63] into model merging. Unlike prior TTA-based
70
multi-task merging methods [87] (68 [8) which as- sume simultancous availability of models and test data from all tasks, TTCMM utilizes only a small set of unlabeled samples from the current task, making it uniquely suited for realistic continual scenarios where revisiting historical data is often infeasible.
Baa5A1
Ovcrall
Figure 1: After 8-task continual merging: ac- curacy on first four tasks and overall BWT.
In this paper, we propose MINGLE (MIxture Null-Space Gated Low-Rank Experts), a method designed to continually merge independently fine-tuned models at test-time while preserving prior knowledge. MINGLE employs a ceatuure-of-expett architecture [27 [55 composed of lightweight LoRA-based [21] experts, enabling efficient and flexible test-time adaptation. To robustly prevent interference from previously learned tasks, we introduce a novel Null-Space Constrained Gating mechanism, restricting gating updates task-orthogonal subspaces. Additionally, we propose an Adaptive Relaxation Strategy to dynamically modulate constraint strength based on test-time interference feedback during adaptation.
Extensive experiments on standard continual learning benchmarks show that MINGLE consistently outperforms previous state-of-the-as approaches by 7-9% average, achieving robust general- ization and strong resistance to catastrophic forgetting across diverse continual learning scenarios. Remarkably, these improvements achieved entirely without any access to original training data, demonstrating the effectiveness of our TTCMM paradigm and the power of test-time adaptation in continual learning.
Our contributions are summarized as follows:
We formalize test-time continual model merging (TTCMM), novel task that leverages unlabeled test samples to merge independently fine-tuned models.
We propose MINGLE, TTCMM framework with Adaptive Null-Space Constrained Gating to effectively balance stability and plasticity.
Extensive experiments show that MINGLE achieves state-of-thh-aa performance, consistently outperforming prior methods in accuracy, robustness and resistance to forgetting.
2
Related Work
Continual Learning. Continual learning (CL) seeks to mitigate catastrophic forgetting
45 where
learning new tasks overwrites prior knowledge. Regularization- based methods constrain updates with importance weights [31] 92 2
30 82 while distillation aligns outputs to preserve knowledge Replay methods store exemplars or generate surrogates with prompts, prototypes, or
20
12
60
52
generators 55
61 53), and dynamic architectures expand capacity via growth or ensembling Recent work leverages lightweight adapters or prompts pre-trained models for efficient
39
79
35
98
42
transfer
89
23
80 Model merging offers an alternative route. Some methods remain close to

---

Duta arrive stages
Models stages
Models arrive stages
D2
D2
D:
0g
Oz
merged 1-1
Bp
f merged
Without Access to Training Data Y
Training Data
Sced Samples
Training Data
Merging
Merging
Partial unlabuled guide merging
engedd
8
merger emerged
(a) Conventional Continual Learning
(b) Continual Model Merging
(c) Test-Time Continual Model Merging
Figure 2: Comparison of three continual learning paradigms, (a) Conventional Continual Learning trains models sequentially with data arriving in stages, without access to previous task data. (b) Continual Model Merging continually fuses independently trained models, without access to any training data. (c) Test-Time Continual Model Merging improves merging by leveraging few unlabeled test samples from the current task.
conventional CL by sequentially fine-tuning and merging models to reduce forgetting 41
[6].
typically requiring training data. In contrast, continual model merging [28 38 independently fine-tuned models without revisiting training data, enabling greater scalability and privacy.
51
70
merges
Model Merging. Early work merged models via direct parameter averaging 72 58 later refined by linear mode connectivity [L3 1 Wortsman er al. [81] showed that weight averaging can also enhance robustness and out-of -utto--distributionn generalization. Task Arithmetic (TA) 24 views models as task vectors to be summed, but relies on weight disentanglement 49), often violated under standard
fine-tuning, motivating structured training 228 65]. Beyond averaging, interference-awan methods reweight or sparsify parameters [86] 90), or fuse models distillation and clustering [88] 76).
LoRA-based tuning (21 introduces additional entanglement challenges. spurring gradient-free frree strieval-based strategies [22 94 (5]. More recently, dynamic merging [68 40) adapts parameters
conditioned inputs, achieving higher flexibility and performance, but remains limited to multi-task fusion and unexplored in continual settings.
Test-Time Adaptation. TTA adapts models at inference mitigate distribution shift. Early ap- proaches used self-supervised objectives [66], entropy minimization (75), or regularized updates 63
Online TTA adapts continuously (25) while batch-wise variants ignore temporal structure [17] To
enhance stability, later work introduced confidence filtering [48], EMA [I,, partial updates [91].
test-time augmentation [93], and adaptive BatchNorm [56). For vision-language models, TTA often
employs prompts or adapters [59 15 36). We draw on TTA to guide merging, aligning fused models with evolving test distributions.
Relation to Prior Work. Most related to our work are MoE-Adapter 89 and WEMOE (6].
MoF-Adapter follows conventional continual learning, embedding expert modules that are jointly trained across tasks. contrast, we adopt model-merging paradigm, where experts are extracted from independently fine-tuned models and inserted without further training. WEMOE incorporates test-time adaptation but targets multitask learning, assuming simultaneous access to all models and data. By contrast, MINGLE is tailored for the more challenging continual merging setting.
both built on MoE architectures 27 29
3 MINGLE: Mixture of Null-Space Gated Low-Rank Experts
3.1 Preliminaries
Problem Setting. We study continual learning in model merging setting, where sequence of task-specific models .O.,..., 0r) are independently fine-tuned from shared pre-trained model 0g.
each using dataset ID; (0 (())) with label space C, V. The goal is construct unified model orerged that generalizes across the combined label space C1.r U C..

---

Unlike conventional continual learning, we assume No access to training data during merging. All adaptation happens directly parameter space. This paradigm relevant in scenarios where only final fine-tuned models are retained, while original training data is discarded due to privacy, storage, or accessibility constraints.
To contextualize this, we compare with two related paradigms in Fig. 2
Conventional Continual Learning. A single model is sequentially updated on Di...., Dr. discarding previous data. It requires direct training data access and extensive retraining.
Continual Model Merging. sequence of models are merged incrementally in parameter
space without access to training data and earlier models: 0 Mer (0merro 0e).
Test-Time Continual Model Merging, An extension of the above where a small unlabeled subset Dieed C 'Diett (e.g., 5 samples per class) is available at each stage to provide lightweight task-specific guidance. We refer to Di as the seed samples of task t.
Existing Continual Merging Strategies. Let 0g denote the parameters of a pre-trained model. The corresponding task vector defined as A0, 0, Op-
Continual Task Arithmetic (C. TA), simple additive merge 24
0 gmerggad gmerged + AA0,
where À is scalar. While training-free free, it is sensitive À and prone to task interference.
Orthogonal Projection-based Continual Merging (OPCM). Tang et al. [70] propose pro- jecting each 20, onto the orthogonal complement of previous directions: 0. 0g
A A + (AO)) where p((-1) retains components orthogonal to previous updates. This reduces interference but ignores adaptation task distributions.
To address these issues, we present MINGLE, which leverages Diod to modulate the integration of 0 at test-time, enhancing alignment test distribution and mitigating task interference.
3.2 Motivation and Theoretical Analysis
Most existing continual model merging methods combine fine-tuned models via static averaging. where each expert is assigned fixed coefficients, thereby enforcing the same mixing rule across the whole input space. Consequently, it cannot specialize regions where one expert clearly superior. In contrast, Mfxture-of-Exxee Experss (MoE) equips every input with a data-dependent gate g(x) = (gi (zr)... gr (a)) that selects or re-weights experts on-the-fly. We give formal comparison between static averaging and dynamic MoE under noisy-routing scenario. 0
Theorem 1 (Dynamic MoE versus Static Averaging). Let ((Dr,fi))t be T independent tasks with priors P(() and per-task risks Ri).). For any static mixture (statii(x)) = E 04 f((1)) and any hard-routed MoE hT)oE()) fe(a)
with task-specific routing errors Ey-
T
R(hMOR) Rideal
Σ P()) ea Rownngg Re(()),
where Rideal = E. ((t) and Ryrongg A Eiet (1).). Moreover,
1. (Perfect routing) If et = 0 for all t, then inf R(hMMO)) tasks disagree on their best expert.
< inf R(hatatic) whenever at least two
(Noisy routing) If Σ P(t) ((tt)e((Rwony.. - Rd (t) < Ritatic Rideaas where R = inf R( R(Dstaiie),
then the MoE still attains lower risk than any static mixture.
The theory above motivates design that (i) keeps experts specialized and (ii) prevents interference between tasks. Our MINGLE framework achieves both goals by combining
Low-rank experts ft that capture task-specific variations with minimal parameters, and
Null-space constrained gating that projects gradient updates away from subspaces spanned by previously activated features, keeping small without harming earlier experts.
Symbols and proof' deferred to App. A

---

3.3 Low-Rank Expert Mixture for Continual Model Merging
We adopt MoE framework for continual model merging, which each task is equipped with low-rank expert fi and an associated input-dependent gating function 9t- These components are injected into the linear layers of the backbone (e.g., CLIP visual encoder). The gate 9i modulates expert activation based on the input features, allowing for fine-grained, localized task specialization.
Mixture of Low-Rank Expert. When a task arrives, dedicated expert and its gate gr appended to the model. The output of given 1-th layer can be formulated as follows:
@merged.10 (x) = (merped.()) (1) (X) + De (X) (x) = 60 (X) + Σ (X) (X). (2)
where only the gate 9I is adaptable during testing, while all experts {ff and old gates {9.
remain frozen to preserve prior knowledge. To construct expert ft, we first project the task vector Ad, onto the orthogonal complement of previously learned directions, following OPCM [7]
m.R
p(t-))(()) = Σ
(t- (A0,, Du ( put ( (1--1)
(3)
P.gma.praq
(t-1) where (t-1) and ui are the p-th and q-th singular vectors from the singular value decomposition (SVD) of previous experts I fx)), and a denotes the effective rank previous experts. This projection removes previously learned directions mitigate interference. We then apply rank-r truncated SVD for (A6,) to construct a low-rank expert PT::
fo BA = (UÉ)İT,
(4)
where Ú Rdixr, E E Rexr, and V Re retaining the top r singular components. The resulting expert captures the principal directions while significantly reducing parameter overhead.
Each gating function is implemented as linear projection:
9r (X) W x b,))
(5)
where W(() E Rdx1 and b(9) are learnable parameters. The gating function is adapted at test time using small number of unlabeled test data.
Test-Time Adaptation. To encourage the merged model retain task-specific behavior, we minimize the Kullback-Leii divergence between its prediction and that of the corresponding individual fine-tuned model 6g. We define the adaptation objective as:
KL. (p(a;; ommerged) pl:; 0-)]]
L = Drzd
where p(2; f) denotes the predictive distribution
3.4 Adaptive Null-Space Constrained Gating for Interference Mitigation
When merging models continually, the primary challenge of gating is to integrate new experts without disturbing prior task predictions. Consider two experts f1, f2 and their corresponding gates 91. 92- When evaluating on the first task domain X1, the interference from g2 can be quantified as:
(gg) = (X,) (i() + (2())) fe(X,) (X,,)11
(7)
This measures the deviation introduced by 92 on the domain where fi originally dominates. A desirable gating function should suppress (x(),,, ensuring predictions X1 remain unaffected. However, as X2 becomes inaccessible after adaptation, this error becomes unobservable and cannot be minimized directly, resulting in prediction drift and catastrophic forgetting.
Hard Null-Space Projection. After completing task I,, we cache the I-th layer inputs the seed buffer Deeed and estimate their covariance Cov (1) E idxdd Applying truncated SVD yields the
The layer index I is omitted hereafter whenever it does cause ambiguity.
5

---

top-k dominant subspaces 0/00 E Baxk, We then concatenate these with the subspaces from all previous tasks and orthonormalize: U.!) orthonorm -orthonorm[US11 [0 E Rdxttk The hard projector P: -- U() U!uT dRdd To suppress interference from tasks I 1, the gating update for task t is w + w. - - V p! However, this projection may also discard gradient components that are informative for task t whenever its feature support overlaps with span(U;-1).
Adaptive Null-Space Relaxation. To restore plasticity, we replace the all-one eigenvalues of Pt-1 with soft coefficients learned online.
Algorithm 1 MINGLE Procedure.
Input: pre-trained model and fine-tuned mod- els {0e - seed data (Di hyper- parameters k. 3. learning rate r)
(i) Interference statistics. For each column
up of U we measure instantaneous align- ment:
(1)
Output: Merged model (TT @rerredd Init: 6 0 +- fU: 0//) +-
I(V 1121l2 / 1.
(8)
for task = 1 to do
create lou-rank experta (Eqa. and fe = SVDTRUNC. (p(t-1) 1) (8e Bu)
We maintain per-direction interference scores s(m.)) E & (initialized to 0) by applying an exponential moving average at each iteration m.
B,A
use 8 -86 when 1=1
D add expert initialize gate (Eq. gmergged t-1 +gr (W(2),
b.th .5(0)
+ 0
g(m.)) 3 5(m-1.) + (1 (1- B) r())
for m = to total iterations do
which suppresses stochastic gradient noise while preserving the dominant interference directions.
X f batch (Dreed)
{ W L, V 6( L) + VL, 6, merged 0a)
ift then
(ii) Aduptive shrinkage. Each direction attenuated by (mm,)) = exp( ((m))
b project gradient null mpace
(Eqp. 31
(r > O, A(m,d) E 1). Let A,
(mm)
35 + +1-13)
diag(A) (m,.) (lag(A(mm)) ....,, (C-1))) The relaxed pro-
V W L t V W1o Le P
jector becomes:
end
W (g) f- W(()) n V Wi
p(mm)) u!!!") (m,.) U U.T.
(10)
0(0) + ((0) n V b6o L
interpolating smoothly between no protection
(A = 0) and the hard null projector (A = I).
end
b update dominant subspaces
(iii) Update rule. We finally update:
U orthonorm(Ui-1 Üd
w(o.)) w/(')) VL (0) P0o
(11) end
Relaxing the projector inevitably allows more residual interference than the hard null-space variant, yet empirically the increase is minor and is offset by markedly higher plasticity (Tab. 5)) indicating a favorable
balance. The overall procedure is outlined in Algo.
4 Experiments
We describe the experimental setup in Sec. 4.1 followed by the main results in See. 4.2 Sec. and further analysis and ablations in Sec. 4.3 Appendix.
Due to page limitations, detailed results are provided in the
4.1 Experimental Setup
Datasets and Models.
Following 70 we evaluate on maage-lasssifiationn tasks with CLIP-VIT
backbones [54]. We consider 8, 14, and 20-task groups using ViT-B/32, ViT-B/16, and ViT-L/14
models, each fine-tuned on up to 20 downstream tasks, with checkpoints from FusionBench [67]
To assess order sensitivity, we repeat experiments over random seeds (42-51). For comparison
with conventional CI., we use the Multi-domain Task- Learning (MTIL) benchmark [97]
with eleven vision tasks. Beyond vision, we evaluate on eight GLUE language tasks 77 with -a-base backbone (6).

---

Table Comparative results of continual merging methods, reporting average accuracy (ACC) and backward transfer (BWT) over ten task orders (mean DM and DA denote method assumptions: dynamic merging or test data access. Best results are in bold; second-best are underlined. MINGLE" denotes lightweight variant.
Assump-
VIT-R/32
VIT-B/16
VIT-L/14
Method
DM/DA
tasks
14 tasks
20 tasks
8 tasks
14 tasks
20 tasks
8 tasksss
14 tasks
20 tasks
PRE-TRAINED
48.1
56.9
55.6
55.4
62.0
59.8
64.9
69.1
65.6
-/=
FINE-TUNED
90.4
89.3
89.8
92.4
91.3
91.6
94.3
93.4
93.5
C. FINE-TUNED
79.8
67.4
62.6
82.9
72.2
68.2
90.0
70.9
77.7
AVERAGE (SWA) R0
X X
66.3
65.4
61.1
72.3
69.7 7
64.8
80.0
77.5
71.1
C. TASK ARITHMETIC
24
X X
67.5 ao0
66.5
60.0
77.1
70.9 iod
64.2
82. ina
77.9
70.3
MEES-MERRIIG
86
X X
49.0 m0a
66.2 10s
59.9 e13
66.8 157
70.5 sus
63.0 614
64.3 km
78.0 :as
68.3 1m
MAGMAX IND 1
X! X
70.7
67.0
61.2
76.7
67.0
62.5
83.4
71.2
71.2 100
CONSENSUS 'AA
77
X1 X
67.1 104
64.1 1os
43.8 m:3
72.8 ens
69.0 1oo
49.9 113
80.4 eas
75.0 :10
51.3 424
OPCM (70
XI X
71.9 1o3
65.7
77.1 jas
70.3
87.0
83.5
C. LW ADAMERGING 87
XI /
53.4 +:
59.8
59.7
29.9
64.3
61.5
68.8 =
73.1
66.9
C. LORA-WEMOE [68]
VI V
68.8 1:
63.8 14
49.6
72.6
67.9 1
55.0
75
74.0
56.9 m9
MINGLE (Ours)
Vf V
85.8
77.1
84,9
91.8
85.5
MINGLE' (Ours)
VI 85.0
81.7
77.1
87.0
84.7 310
81.6
91.4
89.2
83.6
AVERAGE (SWA) 26
X1 X
-11.5 =2:
-8.0 L13
-7.1 #21
-9.7 s13
-7.1 +14
-1.3 esr
-7.3 =:4
-5.8 =1
-6.4 +1.
TASK ARITHMETIC
XI X
21
-1a3
-3.4
-2
-133
-3.6
-7.1 =:*
-1.8
am
C. TIES-MURGING 86
X1 X
-15.3
1.9
-1.5
-5.5 s94
1,400
-1.5
-13.0
-1.1
-2.9
MAGMAX-IND 41
XI X X1
-8.3
-7.4
-7.2
27
-8.0
-5.0
-6.0
DW7
CONSENSUS
-1.3
2.4
-2.5
-16.5
OPCM 70
XI X
-6.3
26.
-7.8
-4.8 207
-5.1
-6.3g2
22.6
-4.3
-6.5
LW ADAMERGING 37
XI /
-3.55 234
22411 2:3
22.77 443
-2..8 827
-22.1
a14
-21.4
22.3 233
-19.6 917
.21.7 gma
AERM- 6S
VIV
-2044
-30.2
-24.5
18.
-18.8
-25.8
-178
-16.8
-27.9
MINGLE (Ours)
V1V
-0.6 104
-1.1 102
-2.2
-0.4 pa1
-0.9 ta:
-1.9 634
-0.6
8t1
1.0
.2.6 100
MINGLE" (Ours)
5/
-0.1 Pm
404 sm
-1.3 206
0.1 s0
+0
-10
-0.2
220
-0.4 =12
Table 2: Results of continual merging Flan-T5- base models on 8 tasks, ordered alphabetically.
Method
DM/DA
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STSB
1CC
BWT1
PRE-TRAINED
69.1
56.5
76.2
88.4
82.1
80.1
91.2
62.2
75.7
INDIVIDUAL
75.0
85.4
85.9
86.4
TASK ARITHMETIC
X1
69.1
88.9
83.1
79.1
74.0
77.6
-466
TIES-MERGING
X1 X
39.3
70.0
82.4
88.8
81.8
75.8
89.7
76.8
75.6
-6.1
OPCM
K1 K
69.9
72.9
78.7
90.3
83.8
83.0
92.2
73.7
80.6
-2.5
LW ADAMERGING
X1
69.1
88.9
83.1
79.1
77.6
LORA-WEMOR
VIV
71.5
80.6
78.2
90.3
82.7
80.5
91.3
76.2
81.4
0.1
MINGLE (Ours)
J.V.
75.0
78.2
86.0
90.9
84.2
80.5
92.5
78.8
83.3
0.1
Implementation Details. We insert low-rank experts into the CLIP vision encoder. Two variants are used: a full setup modifying all attention and MLP layers, and a lightweight one on attn, qkv and mlp. fc1. All experiments share a single set of global cypper-araamtert across models and task orders. Each expert has rank r 64; the null-space constraint uses 3, 7 1, and 3 = 0.99. Adaptation runs for 50 iterations with Adam (Ir le-4, batch size 16). For vision tasks we use unlabeled samples per class, and for NLP tasks 100 in total, all without access to prior-task data.
Evaluation Metrics. We evaluate using average accuracy (ACC) and backward transfer (BWT) 33 ACC is the mean accuracy of the final merged model across all tasks: ACC = Σ ((("eegge)) where a.))) is accuracy on task BWT measures forgetting by comparing performance on earlier [a [a (mergd, (.(oeeerue))))
tasks before vs. after the final merge: BWT = 7-1 Σ T-1
4.2 Main Results
Overall Performance As shown in Tab. MINGLE substantially outperforms previous continual merging methods on all CLIP backbones and task counts. It achieves the highest accuracy with backward forgetting kept near zero, demonstrating both strong forward learning and long-term stability. The lightweight variant performs on par with the full version, further underscoring the robustness of our approach. On NLP benchmarks (Tab. 2 MINGLE likewise attains the best overall accuracy and non-negative BWT, improving on multiple GLUE tasks while maintaining balanced performance across the suite. Together, these results across vision and language confirm that MINGLE
7

---

Table Comparison last accuracy (%) with conventional CL approaches on MTIL benchmark.
Alnraft
Caltech(0
CIFARIOO
DTD
EuroSAT
Flowers
Foodt!t
MNIST
Pets
Cars
SUN397
Method
Avg.
Conventional (Sequential fine-tuned)
WISE-FT 81
27.2
90.8
68.0
86.9
87.6
77.8
ZSCL 12
40.6
92.2
81.3
70.5
94.8
90.5
91.9
98.7
93.9
85.3
80.2
83.6
ADAPTER
49.8
92.2
86.1
78.1
95.7
94.3
89.5
98.1
89.9
81.6
80.0
85.0
DIKI
45.2
86.3
98.0
89.2
81.6
AWOFORGET 96
42,4
92.7
83.2
73.2
97.0
91.8
92.2
99.1
93.9
87.4
82.6
85.0
DUAL-RAIL
52.5
96.8
83.3
80.1
96.4
99.0
89.9
98.8
93.5
85.5
79.2
86.8
MAGMAX 41
40.2
96.1
81.1
97.8
76.3
88.4
70.5
68.9
80.3
MINGLE
58.7
97.5
87.2
79.7
97.3
87.2
90.1
99.6
93.0
80.4
73.3
85.8
Continual Merging (ndependent fine-tuned)
AVERAGE (SWA) 26
26.5
92.3
74.3
48.4
73.7
74.0
87.1
84.0
91.2
67.5
68.5
71.6
24
26.6
74.3
C. TIES [6
30.5
94.0
74.8
49.8
71.7
73.8
87.3
81.5
90.6
67.0
67.9
71.7
MAGMAX-IB IND
41
29.9
93.7
78.4
46.1
58.3
68. 1
86.8
82.8
91.4
62.7
69.3
69.8
OPCM 70
35.7
95.9
77.0
87.1
20.1
70.5
MINGLE
54.2
97.3
79.7
72.3
96.0
86.7
88.7
99.3
93.9
73.1
71.6
83.0
Table 4: Robustness results ViT-B/32 continually merged across 4 tasks.
Method
Clean
Impulse
Gaussian
Pixelate
Spatter
Contrast
JPEG
Avg.
C. ADAMERGING
43.1
43.3 am4
111
C. WEMOE 68
3.4 sas
3.1 104
4,3
3.4
310
4.0 t.0
3.3
4.0
3.6
C. LORA-WEMET
68
78.7
71.0
55.0
59.4
24.9
60.5 =3
68.5
69.7 =sa
61.0
ARITHMETIC
77.5
66.0
58.9
29.7
63.5
ann
MAGMAX- IND 41
79.1 sso
69.0
60.6
61.5 so0
33.0
66.4
68.6
69.9
63.5
OPCM BD
83.6=ts
72.5 sus
64.7 +12
65.2
35.2
70.5 =a
72.5 eas
74.4 zus
67.3
MINGLE (Ours)
1m1
67.5
70.7 a12
::
ADAMERGING
138.0
.2.22 an0
-8.6
34.77 163
211
C. WEMOE 68
-30.7
-28.7
-22.1 sisa
-25.5
-8.0
-23.4
-27.6 154
-28.6
-24.3
C. LORA-WEMER
68
-13.6
-12.6
-11.3
-10.2 =1
-15.6
-10.8 aoo
-11.2
-16.7 ass
-13.0
BWT
261
-1.6218
-1.6217
-2.7.15
21
-6.1
-5.1
MAGMAX- IND 41
-7.7
-8.1 m:s
-6.1
-5.1
-3.5
-7.3
-8.4 816
-8.2
-688
OPCM FD
=13
-4,5 =28
-6.4em
-2.9
-6,3-2
-4.5
-5.7
-5.1
MINGLE (Ours)
202
40.1
0.7
0.6
0.2
g63
consistently delivers state-of-the-ar accuracy while nearly eliminating forgetting under diverse continual scenarios.
Comparison with Conventional CL. Tab. Tab.B.ev... evaluaaes two CL paradigms on the MTIL benchmark: conventional CL, where each task model is fine-tuned from its immediate predecessor; and continual merging, which fine-tunes each task model independently before fusion, eliminating inter-model dependencies and enabling flexible task ordering and model reuse. Within the merging family, MINGLE sets a new oftte--f-the-artt and when integrated into a sequential fine-tur toning pipeline, matches the performance of SOTA CL methods. This demonstrates both its strength fusion strategy and its versatility across different training regimes.
we evaluate MIN- GLE on seven corruptions (motion blur, impulse noise, Gaussian noise, pixelate, spatter, contrast, JPEG) and report results in Tab, 4 It preserves high accuracy and near-zero or even positive BWT, outperforming all baselines, whereas direct application of SOTA TTA-based merging (WEMOE, AdaMerging) in continual setting fails without tailored designs to continual setup.
Robustness to Test-Time Distribution Shifts. Following prior work 87
68
4.3 Ablation Results and Analysis
Ablation Study. We explore the contribution of each component in Tab. 5 Row shows fixed-weight merging of low-rank experts as our baseline. In Row 2, adding TTA boosts ACC substantially but at the cost of worsening BWT with more tasks. Row demonstrates that freezing earlier gates curbs forgetting while retaining ACC gains. Row 4 then applies null-space constraints, yielding further BWT improvements. Finally, Row presents the full method with adaptive relaxation, which best harmonizes accuracy and long-term stability.

---

Table 5: Ablation study of MINGLE with CLIP ViT-B/16 over 14, and 20 tasks.
Test- Time
Frozen
Null-Space
Adaptive Relaxation
B
Adaptation
Old Gate
Constrained Gate
8 tasks
14 tasks
tasks
8 tasks
14 tasks
20 tasks
78.7 taa
76.4
70.6 ea4
-0.5 sm
+.3
-1.3 c03
86.4 113
81.7
76.7 133
-6.
-7.7
-12.8
87.4 1or
76.2 18s
205
-4.3
-6.8 in8
X
86.0 2/5
83,5
78.3
-0.1 an1
-0.1 201
.0.2
88,3 =03
84.9 =92
81.9 e00
-0.4
+0.9
-1.9
Table 6: Ablation on number of adaptation steps for ViT-B/32 across 8, 14, and 20 tasks.
Steps
ACC (8-task)
BWT BWT (8-takk
(C4-tu4))
BWT (14-takk
ACC (20-task)
BWT (20-task)
5
60.9-14
-0.1=02
62.4s12
-0,3+01
60.2+:s
-0.1
10
68.6=1a
-0.2-uz
68.5+1a
-0.1+0:
63.5+u0
-0,4sux
20
78.4
-0.2:mt
75.5
-0.4
71.0:12
-0.4
50
85.8
-0.6_04
81.6114
-1.1103
77.1
-2.2
Table 7: Efficiency and sample analysis. (a) Expert insertion layers and rank sweep over 8 tasks on CLIP-ViT-B/22. (b) Wall-clock adaptation time across tasks on CLIP-ViT-B/32. (c) Accuracy (%) of CLIP-ViT-B/16 under varying numbers of test samples.
(b) Wall-clock adaptation time across fasks
(a) Expert Insertion layers and rank sweep (8 tasks)
#Tasks
Adaptation steps
Total Time
Avg/lask (s)
Configuration
TTA Time Train. Puram Full Parum ACC(%)
8
50
78
9.8
atta. ppt. (r 64)
s
27.7 k
116.9 M
69.9
50
138
atta.out_pro) (r 64)
4
9.3
97.0
53.9
20
50
211
10.6
aIp-fcl (r
48
9.0
111.1
82.6
(c) Results with varying samples/class
mlp.fe2 (r = 61)
48
36.8 k
111.3 M
70.1
# Samples/class
8-task
14-task
20-task
atta nlp 64)
78
83.0
85.8
D (Static)
78.7
76.4su
70.6:14
gkv fel 32)
65
36.9
113.7
84.5
gev &t fc1 (r- 128)
65
36.9
191.6 M
85.1
4304
84
81.7
giv fcl 768)
70
36.9
710.3 M
83.7
3
88.6104
84.7
81.7au
gkv & rci (r 64)
65
36.9
139.7 M
85.0
5
88. 88.3
84.9
10
88. 6ans
85. 1a11
12.
Ablation on TTA optimization steps. Tab. 6 reports accuracy and backward transfer of CLIP ViT-B/32 under different adaptation steps across 8. 14, and 20 tasks. Accuracy improves steadily with longer schedules, rising from 60-63% at 5 steps to 77-86% at 50 steps. Forgetting remains negligible, with BWT close to zero in all cases and only minor drop of about 2% in the 20-task setting at 50 steps. Notably, as few as 20 steps are sufficient to surpass all baselines in Tab. I, while steps yield the best accuracy with only a modest increase in cost. These results confirm that MINGLE is both effective under tight compute budgets and scalable with additional adaptation.
Computation and Parameter Efficiency. Tab. 7 summarizes the efficiency analysis. In part (a), inserting experts into both attn and mlp layers yields the highest accuracy, 85.8%, but also the longest adaptation time of 78s and 83k trainable parameters. A lighter hybrid scheme qkv & fc1 with rank 64 reaches 85.0% accuracy with 36.9k parameters and 65s, offering better trade-off. The rank sweep shows that raising the rank from 32 to 64 improves accuracy from 84.5% to 85.0%, while larger ranks bring little or even negative gain, e.g., rank 768 drops to 83.7%. Part (b) reports wall-clock adaptation time tasks increase: with 20 tasks the total is 21 Is, averaging about 10s per task. After adaptation the router remains fixed and inference is purely feedforward without TTA, enabling low-latency deployment across all tasks. Overall, MINGLE achieves a strong balance of accuracy, parameter efficiency, and scalability under diverse resource budgets.
Number of Seed Samples. The number of seed samples per class is crucial for TTA reliability and efficiency. As shown Tab. (c), using no samples reduces to the Statie baseline, where LoRA modules are merged with fixed coefficients (0.3) without adaptation, yielding 70-79% accuracy. Introducing a single sample lifts accuracy to 81-88% and adding more samples offers only minor gains. Variance across task orders decreases with more samples, making five samples per class balanced trade-off between performance and efficiency.

---

Activatttioonn
1.0
Gate Activaation
1.0
Cate Activation
1.0
Activattiionn
1.0
0.5
0.5
0.5
0.5
Basttrr
avtl
MNIST
DTD
RESIScUs
Cur
Dasttr
aynn
SNNRT
nm
REstoces
arseo
Clus
Banstar
ava
MNNIT
nrp
aisRu
Cam
Famsar
evns
MNIST
DVD
RESICCH
GTER
shRup
Can
Gate
Gate
Gate ?
Gate 8
Gate Activttion
Cote Acttivaaan
1.0
Catee Activatio
0.5
0.5
0.5
0.5
0.0
0.0
0.0
0.0
Rautsy
MNNT Svrey
unp REsscie
ornn
stnw?
Raster
SWIN
MNVOT
tm
REsIrCe
07300
stmtrr
Cur
Bandar
SYIN
SRUST
Reserca
07360
sum99
Cer
Banstt
SVIN
MNIST
DTD
rusncca
arswn
summ9
Null Space
Figure 3: Gate activations across eight tasks under varying 7- Each subplot shows one gate; curves and shaded areas indicate mean and std across layers. Gray bars mark the gate's 's training task. Lower y leads to stronger suppression on prior tasks.
ACC
BWT
RWT
90.0
0.0
3800
0.0
83.0
DAVT (N)
ACC.50
-0.5
$6.0
-0.1
-0.
1.0
$4.0
-0.3
84.0
-15
$2.0
-0.4
3
10
20
0.
0.95
0.08
0.905
10 20
(a) Vary 7/ B 0.99, 1
(b) Vary 8, y
(c) Vary k 7 =
Figure 4: Sensitivity analysis of the null-space constrained gating w.r.l hyper-parameters, 33, 7, and k.
Visualization on Gate Activations. Fig. 3 shows that the null-space constraint suppresses gate responses on previously seen tasks, reducing forgetting, with smaller giving stronger attenuation. We also observe that gate activations remain below 1.0 across tasks, even in the w/o Null-Space variant (blue curve), showing that under-activation not solely due to the constraint. Instead, reflects the complementary nature of experts: multiple LoRA modules capture overlapping but distinct subspaces, and softly combining them often yields better performance, especially in settings like TTCMM where task boundaries are fuzzy.
Hyper-parameter of Gate. We study the effect of three key hyper- rrper-parameters: 7, 33 and y controls the strength of null-space suppression; smaller values lead to stronger attenuation of activations on prior tasks, reducing forgetting (Fig. 4a) As shown in Fig. 4b S regulates the smoothness of the EMA used to accumulate interference signals. A moderate setting (3 0.99) balances responsiveness and stability. Smaller 3 amplifies noise sensitivity, while larger slows detection of interference. determines the number of principal directions retained per task. The mitigation of forgetting saturates (Fig. 4ch, indicating that a small number of task-specific directions suffices.
Conclusions
In this work, introduced the task of test-time continual model merging (TTCMM) and proposed MINGLE, a novel framework for TTCMM that integrates a mixture-of-experts architecture with adaptive null-space constrained gating. Extensive empirical evaluations show that MINGLE substan- tially improves generalization and mitigates catastrophic forgetting, consistently outperforming prior state-of-the-an approaches. These results establish ТTCMM principled paradigm for addressing both task interference and distribution shift, and highlight the practical potential of MINGLE for scalable and efficient continual learning in real-world applications.
10

---

Acknowledgments This work was supported in part by National Science and Technology Major Project (202IZD01 12001). National Natural Science Foundation of China (No.6227111) 119, 08120002. 62071 0067,066,00 U23A20286), the Key Research and Development Project of Hainan Province (Grant No. ZDYF22244(AALH))000) the Fundamental Research Funds for the Central University China (DUT No. 82232031), I), the Natural Science Foundation of Sichuan Province under Grant 20252NSFSC0475.
We thank all reviewers for taking the time review our paper and give valuable suggestions.
References
[1] S. K. Ainsworth, J. Hayase, and S. Srinivasa. Git re- basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209. 20xx:2209.04836, 2022.
[2] R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars. Memory aware synapses: Learning what (not) forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018.
[3] L. Bossard, M. Guillaumin, and Van Gool. Food-101-mining discriminative components with random forests. In European Conference Computer Vision, pages 446-461. Springer, 2014.
[4] G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865-15)) 2017.
[5] R. Chitale, A. Vaidya, A. Kane, and A. S. Ghotkar. Task arithmetic with loRA for continual learning. In Workshop Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ NeurPS 2023), 2023.
[6] H. W. Chung, Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction--inetune language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
[7] M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1606-3613, 2014.
[8] T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha. Deep learning for classical japanese literature. In NeurlPS Workshop on Machine Learning for Creativity and Design, 2018.
[9] A. Coates, A. Y. Ng. and Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 215-223. JMLR Workshop and Conference Proceedings. 2011.
[10] G. Cohen, S. Afshar, J. Tapson, and A. Van Schaik. Emnist: Extending mnist handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN), pages 2921-2926. IEEE, 2017.
[11] M. Döbler, R. A. Marsden, and B. Yang. Robust mean teacher for continual and gradual test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7704-77 77 14, 2023.
[12] A. Douillard, M. Cord, C. Ollion, T. Robert, and Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. European Conference Computer Vision, 2020.
[13] R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:21 10.0026, 2021.
[14] C. Fang, A. Dziedzic, L. Zhang, L. Oliva, A. Verma, F. Razak, N. Papernot, and B. Wang. Decentralised, collaborative, and privacy-preserving machine learning for multi-houe mosti-hospptal data. EBioMedicine, 101, 2024.

---

[15] C.-M. Feng, K. Yu, Y. Liu. S. Khan, and W. Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2704-2714, 2023.
[16] T. Fukuda, H. Kera, and K. Kawamoto. Adapter merging with centroid prototype mapping for scalable class-incremental learning. arXiv preprint arXiv:2412.182) 2024.
[17] J. Gao, J. Zhang, X. Liu, T. Darrell, E. Shelhamer, and D. Wang. Back to the source: Diffusion- driven adaptation to test-time corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11786-1 11796, 2023.
[18] J. Goodfellow, D. Erhan, P.-L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Tang, D. Thaler, D.-H. Lee, et al. Challenges in representation learning: A report on three machine learning contests. In International Conference on Neural Information Processing, pages 117-124. Springer, 2013.
[19] Helber, Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12()):211-2226, 2019.
[20] S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin. Learning a unitied classifier incrementally via rebalancing. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 831 839, 2019.
[21] E. J. Hu, Y. Shen, P. Wallis, 7.. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
[22] C. Huang, Q. Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin. Lorahub: Efficient cross-task generalization via dynamic loRA composition. In First Conference on Language Modeling, 2024.
[23] L. Huang. X. Cao, H. Lu, and X. Liu. Class-incremental learning with clip: Adaptive represen- tation adjustment and parameter fusion. In European Conference on Computer Vision, pages 214-231. Springer, 2024.
[24] G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing mod- cls with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023.
[25] Y. Iwasawa and Y. Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34:2427-2440, 2021.
[26] Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. In 34th Conference Uncertainty in Artificial Intel- ligence 2018, UAI 2018, pages 876 876-885. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
[27] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
[28] X. Jin, X. Ren, D. Preotiuc-Pietrs Pietro, and Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations 2023.
[29] M. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
[30] S. Jung, H. Ahn, S. Cha, and T. Moon. Continual learning with mode-importanee based adaptive group sparse regularization. Advances in neural information processing systems, 33:3647-3658, 2020.
[31] J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114:3521 3526, 2016.
J. Quan, T. Ramalho, A.
12

---

[32] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fineggraieed categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pages 554-561. IEFE, 2013.
[33] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images, Technical report, University of Toronto, 2009.
[34] Y. LeCun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-base learning applied to document recognition. Proceedings of the IEEE, 86(1 (621))2227-2322 1998.
[35] S.-W. Lee, J.-H. Kim, J. Jun, J.-W. Ha, and B.-T. Zhang. Overcoming catastrophic forgetting by incremental moment matching, Advances in neural information processing systems, 30, 2017.
[36] Y. Lee, D. Kim, J. Kang. J. Bang, H. Song, and J.-G. Lee. RA-TTA: Retrieval-augmented test-time adaptation for vision-language models. In The Thirteenth International Conference on Learning Representations, 2025.
[37] Lin, L. Yang. D. Fan, and J. Zhang. Beyond forgettgtt Continual learning with backward knowledge transfer. Advances Neural Information Processing Systems. 35:16165-16177, 2022.
[38] T. Y. Liu and S. Soatto. Tangent model composition for ensembling and continual fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18676- 18686, 2023.
[39] Y. Liu, Schiele, and Q. Sun. Rmm: Reinforced memory management for insssiincremenall learning, Advances in Neural Information Processing Systems, 34:3478-3499 2021.
[40] Z. Lu, C. Fan, W. Wei, X. Qu, D. Chen, and Y. Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. Advances in Neural Information Processing Systems, 37: 18905-78935, 2024.
[41] D. Marczak, B. Twardowski, T. Trzciński, and S. Cygert. Magmax: Leveraging model merging for seamless continual learning. In European Conference on Computer Vision (ECCV), 2024.
[42] I. E. Marouf, S. Roy, E. Tartaglione, and S. Lathuilière. Weighted ensemble models are strong continual learners, 2024.
[43] M. McCloskey and J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109-165. Elsevier, 1989.
[44] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-effiiicnn learning of deep networks from decentralized data, In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[45] S. Mu and S. Lin. A comprehensive survey of mixture-offexpprrs:: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07133, 2025.
[46] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. ural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Wu, and A. Y. Ng. Reading digits in nat-
[47] M.-F. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
[48] S. Niu, J. Wu, Y. Zhang. Y. Chen, S. Zheng, P. Zhao, and M. Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning, pages 16888- 16905. PMLR, 2022.
[49] G. Ortiz-Jimenez, A. Favero, and P. Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36:66727- 66754, 2023.
13

---

[50] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3498-3505, 2012.
[51] A. Porrello, L. Bonicelli, P. Buzzega, M. Millunzi, S. Calderara, and R. Cucchiara. A second- order perspective on model compositionality and incremental learning. In The Thirteenth International Conference on Learning Representations, 2025.
[52] Z. Qiu, L. Xu, Z. Wang, Q Wu, F. Meng, and H. Li. Ism-net: Mining incremental semantics for class incremental learning. Neurocomputing, 523:130-143, 2023.
[53] Z. Qiu, Y. Xu, F. Meng, H. Li, L. Xu, and Q Wu. Dual-consistency model inversion for non-exene non-exemplar class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20025-24035, 2024.
[54] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 87663. PmLR, 2021.
[55] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 5533-5542, 2016.
[56] S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems, 33:1 1539- 1551, 2020.
[57] Y. Shen, K. Song. X. Tan, D. Li, W. Lu, and Y. Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36:38154 38180, 2023.
[58] K. Shoemake. Animating rotation with quaternio eurves. Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pages 245-254, 1985.
[59] M. Shu, W. Nie, D.-A. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35: 15274-1--44 2022.
[60] C. Simon, P. Koniusz, and M. Harandi. On learning the geodesic path for incremental learning. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition, pages 1591-1600, 2021.
[61] J. S. Smith, L. Karlinsky, V. Gutta, P. Aescantee-onilla D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira. Coda-prompt: Continual decomposed attention- prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11999-1111 11919, 2023.
[62] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng. and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, 2013.
[63] J. Song, J. Lee, I. S. Kweon, and S. Choi. Ecotta: Memory-efficient continual test-time adaptation via self-distileed regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11920-11929, 2023.
[64] Stallkamp, M. Schlipsing, J. Salmen, and Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32:323-332, 2012.
[65] G. Stoica, P. Ramesh, B. Eesedi, L.. Choshen, and J. Hoffman. Model merging with SVD to tie the knots. In The Thirteenth International Conference on Learning Representations, 2025.
[66] Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with self- supervision for generalization under distribution shifts. International conference on machine learning, pages 9229-9248 PMLR, 2020.
14

---

[67] A. Tang, L. Shen, Y. Luo, H. Hu, B. Du, and D. Tao. FusionBench: A Comprehensive Benchmark of Deep Model Fusion, June 2024.
[68] A. Tang, L. Shen, Y. Luo, N. Yin, L. Zhang, and D. Tao. Merging multi-task models via weight-ensembling mixture of experts. In International Conference on Machine Learning, pages 4777784499, PMLR, 2024.
[69] A. Tang, L. Shen, Y. Luo, Y. Zhan, H. Hu, B. Du, Y. Chen, and D. Tao. Parameter-efficient multi-task model fusion with partial linearization. In The Twelfth International Conference on Learning Representations, 2024.
[70] A. Tang, E. Yang, L. Shen, Y. Luo, H. Hu, B. Du, and D. Tao. Merging models on the fly without retraining: A sequential approach to scalable continual model merging. arXiv preprint Stti::2501.095222 2025.
[71] L. Tang, Z. Tian, K. Li, C. He, H. Zhou, H. Zhao, X. Li, and J. Jia. Mind the interference: Retaining pre- trained knowledge in parameter efficient continual learning of vision-languaee models. In European Conference on Computer Vision, pages 346-365. Springer, 2024.
[72] J. Utans. Weight averaging for neural networks and local resampling schemes. In Proc. AAAI-96 Workshop on Integrating Multiple Learned Models. AAAI Press, pages 133-138. Citeseer, 1996.
[73] B. S. Veeling, J. Linmans, J. Winkens, T. S. Cohen, and M. Welling, Rotation equivariant cnns for digital pathology. arXiv preprint arXiv: 1806 03962, 2018.
[74] A. Wang. A. Singh, J. Michael, F, Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019,2019.
[75] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 10726, 2020.
[76] H. Wang, B. Ping, S. Wang, X. Han, Y. Chen, 7. Liu, and M. Sun. Lora-flow: Dynamic lora fusion for large language models in generative tasks. In Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 122812, 2024.
[77] K. Wang. N. Dimitriadis, G. Ortiz-Jimenez, F. Fleuret, and P. Frossard. Localizing task informa- tion for improved model merging and compression. In Forty-first International Conference Machine Learning.
[78] X. Wang, T. Chen, Q Ge, H. Xia, R. Bao, R. Zheng, Q. Zhang, Т. Gui, and X. Huang. Orthogonal subspace learning for language model continual learning. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023.
[79] Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pages 631-648. Springer, 2022.
[80] Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Plister. Learning prompt for continual learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 139-149, 2022.
[81] M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Korblith, R. Roelofs, R. G. Lopes, H. Hajishirzi. A. Farhadi, H. Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7959-7971, 2022
[82] Y. Wu, L.-K. Huang, R. Wang, D. Meng, and Y. Wei. Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth International Conference on Learning Representations, 2024.
[83] H. Xiao, K. Rasul, and R. Vollgraf. mashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.077477 2017.
15

---

[84] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Conference on Computer Vision and Pattern Recognition, pages 3485-3492 IEEE, 2010.
[85] Y. Xu, Y. Chen, J. Nie, Y. Wang. H. Zhuang, and M. Okumura. Advancing cross-domain discriminability in continual learning of vision-language models. In The Thirty-eighth Anmual Conference Neural Information Processing Systems, 2024.
[86] P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal. Ties-merging: Resolving in- terference when merging models. Advances in Neural Information Processing Systems, 36: 7093-7115, 2023.
[87] E. Yang, Z. Wang, L. Shen, S. Liu, G. Guo, X. Wang, and D. Tao. Adamerging: Adaptive model merging for multi-task learning, In The Twelfth International Conference on Learning Representations.
[88] E. Yang, L. Shen, Z. Wang, G. Guo, X. Chen, X. Wang, and D. Tao. Representation surgery for multi-task model merging, Forty-first International Conference Machine Learning, 2024.
[89] J. Yu, Y. Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y. He. Boosting continual learning of vision-language models via mixture-of-expeee adapters. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 23219-23230 2024.
[90] L.. Yu, B. Yu, H. Yu, F. Huang, and Y. Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024.
[91] L. Yuan, B. Xie, and S. Li. Robust test test-time adaptation in dynamic scenarios. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 15922-15932,, 2023.
[92] Zenke, B. Poole, and Ganguli. Continual learning through synaptic intelligence. Interna- tional conference on machine learning, pages 3987-3995, 2017.
[93] M. Zhang, S. Levine, and C. Finn. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems, 35:38629 35848622938662, 2022.
[94] Z. Zhao, L. Gan, G. Wang, W. Zhou, H. Yang. K. Kuang, and F. Wu. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild. In Findings of the Association for Computational Linguistics ACL 2024, pages 4447-4462, 2024.
[95] 7. Zhao, T. Shen, D. Zhu, 7. L., Su, X. Wang, and F. Wu. Merging loRAs like playing LEGO: Pushing the modularity of loRA to extremes through rank-wise clustering. In The Thirteenth International Conference on Learning Representations, 2025.
[96] M. Zheng. Y. Tang. Z. Hao, K. Han, Y. Wang, and C. Xu. Adapt without forgetting: Distill proximity from dual teachers in vision-language models. In European Conference on Computer Vision, pages 109-125 129.. Springer, 2024.
[97] Z. Zheng, M. Ma, K. Wang, Z. Qin, X. Yue, and Y. You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEF/CVF international conference on computer vision, pages 19125-19136, 2023.
[98] D.-W. Zhou, H.-L. Sun, H.-J Ye, and D.-C. Zhan. Expandable subspace ensemble for pre- trained based class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23554-23564, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the main contributions, which consistently supported by both theoretical insights and empirical evaluations presented throughout the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the proposed approach are explicitly discussed in Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification: All theoretical claims are accompanied by clear assumptions and complete
proofs in Appendix, with theorem formally stated and rigorously derived.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Detailed descriptions of datasets, training protocols, and evaluation metrics are provided in Section 4.1 and Appendix, enabling faithful reproduction of the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code and data are not provided at submission time, but the authors state they will release them upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper includes detailed descriptions of datasets, data splits, training procedures, hyperparameters, and evaluation protocols in both the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The paper reports mean and standard deviation across multiple runs with different random seeds and elearly states the sources of variability, ensuring the statistical reliability of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies the type GPUs used, training time per experiment, and overall compute requirements, providing sufficient details to assess reproducibility and resource demands.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research adheres to the NeurlPS Code of Ethics, with no identified ethical concerns related to data usage, human subjects, or societal impact.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper includes Broader Impacts section stating that the work aims to advance machine learning and does not raise specific societal concerns requiring detailed discussion.
20

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not introduce models datasets with high risk of misuse, so safeguards are not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-uss should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and models used in the paper properly cited, and their licenses and terms of use are respected and included where applicable.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
21

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsiiiie thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release any datasets, models, code assets, this question is not applicable.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects, so this question is not applicable.
Guidelines:
The answer NA means that the paper clees not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve research with human subjects, so IRB approval is not applicable.
Guidelines:
22

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question:
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
The core methods in this research do not involve LLMs in any important, original, or non-standard way, so this question is not applicable.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy httt::///eerrpssccc/ooorrrrrer/s for what should or should not be described.
23

---

Appendix
A
Theoretical Risk Comparison Additional Descriptions
24
26
B.1
Details of Dataset and Task Settings
26
B.2
Details of Downstream Models
B.3
Details of Baselines
B.4
Details Baseline Syyper-arameteny
29
B.5 Comparison of Assumptions and Requirements
29
C Additional Results
30
C.1 Detailed Overall Performance Results
30
C.2 Accuracy Trends Across Sequential Tasks
C.3 Detailed Results Under Distribution Shifts
30
C.4 Inference Efficiency and Parameter Overhead
34
C.5 Forward Transfer Analysis
34
C.6 Additional Visualizations of Gate Activations and Relaxation Effect
35
Discussions
38
D.I Use of Unlabeled Adaptation Samples
38
D.I Relation Rehearsal- Free Continual Learning
D.3
Limitations
38
D.4
Broader Impacts
38
A Theoretical Risk Comparison: Dynamic MoE vs. Static Averaging
Problem Setup and Definitions Consider T' independent tasks, each associated with data distri- bution D, for = 1,..., T. For each task f, pre-trained expert model fx(x) outputs probability distribution over classes, trained specifically on D,- The overall data distribution is mixture of these tasks, where example (æ, is drawn from task t with prior probability P(t), and then
(x. y) ~ D,. The expected risk a predictive model h(x) is defined as:
R(h) = (((()))(((((()))))))))))))))))") v)] = Σ P(t) Bea.))nn (((())))
(12)
where E(h(x), y) is loss function (e.g., cross-entropy or 0-1 loss).
We compare methods combine the experts into a final prediction l(r)):
ii axff( where (a aT) is fixed weight vector independent of x, typically with a; 2 and E, a; = for probability outputs.
Static Averaging:
Defined as Astatic (x)
Dynamic Lixture-offExpeerty (MoE): Defined hMOB(1)) (z) where i (')))
arg max; 9=(x:) and g(x:) (g1 (x),.... ... ......... or (x)) is gating function that selects expert per input (hard routing). The gating is subject to routing noise, modeled below.
Routing Noise Model For each input drawn from Dr, the true task is t, and the ideal expert is ft. The gating selects the correct expert i (x) = 1 with probability 1 1- 1- ēr, and an incorrect expert (x:) o with probability e, P(i (x) A L x Dz), the task-specific routing enror rate. On error, the gating selects a random expert from (1....,T) \ (1) uniformly. Define:
Ri( = B(x,o)- (,((E)))) v)]. the risk of expert i on task t.
Riideal = Σr P(t) (t),, the risk with perfect routing.
Rownnggt T-I Eist (i), the average risk of incorrect experts on task t.
= Et= P(t) the overall routing erorr rate.

---

Theorem A.1 (Dynamic MoE versus Static Averaging). Let {(D,, ft) be T indepen- dent tasks with priors P(t) and per-task risks (i). For any static mixture (static(:) Σ ag fe(z)) and any hard-routed MoE (x fin(x)) with task task-specific routing errors Ex:
T
R()
= Rideal +
Σ P(t) E (Broong.t - Re(t)).
(13)
where Rideal P(t) and Rwrong,t T-. Eist Re(i). Moreover,
1. (Perfect routing) If = f all t, then inf R(hMOR) < inf R(Matatic) whenever least two tasks disagree on their best expert.
9.
a
2. (Noisy routing) If Σ P(t)E( (Rwrongt Rt) < R Rideal, where R
Raaii
infa R(haaatic) then the MoE still attains lower risk than any static mixture.
The proof proceeds in three parts: (I) deriving the MoE risk with routing noise, (2) proving the optimal gating case, and (3) establishing the condition for MoE superiority under routing noise.
Proof.
Step 1: MoE Risk with Routing Noise
The MoE prediction is (x) for (x) (z)), where (x arg max; (x)) The expected risk is:
T
R(hMoE) -(..))-
Σ P(t) D,
(14)
For task condition on routing correctness:
Correct routing (i* = t): Probability - Et, risk Rt(
Incorrect routing (i*
(i*(x) (x) * t): Probability &t, selects a random expert from (1,...,7) T) \ (t).
with average risk Rwonngt
Σijt Ri(1).
The expected risk on task is:
E (. = Re)) Et Rwoong.
(15)
Thus, the total risk is:
R(hMOB)
Ex) + e, Rwrong.
(16)
1=1
Rewrite:
T (()) +
T
R(hMot)
Σ 1=1
- R2(t)) = Rydeat +
(17)
where 6, = Rwronggt (t) > is the risk increase due to misrouting on task t.
Step 2: Optimal Gating (No Routing Noise)
Assume an oracle gating function with E, for all so io for all x D.. Then:
T
R(HMAR) Σ P())R((!)
Riteal-
(18)
Define hypothesis classes:
T x + Σ
Hatatic
(af(()) a
(19)
i=1
25

---

HMME fI++ (i)()) (x) arg max max g X + =
(20)
Any static model Matanic (2) Et= (x)( can be approximated by MoE with g(x) assigning
constant weights, so Hatatic S HMOE. Thus:
RM inf (hhMat) I inf R(hautic) R
(21)
9(-)
Under task heterogeneity (Re(t) < (s(s) and Rg(s) < (t) for some t + s), the ideal MoE routes each æ Dr to ft, achieving:
Rideal Σ (t))
(22)
For static averaging:
T (a)) E (anl(t)))))
R(ssatic)
(23)
Since e is convex (e.g., cross-entropy), Jensen's inequality implies:
ED, Σ anfilat) }
2 Re(t),
(24)
with strict inequality unless a, 1 and a; for i o t, which cannot hold for all tasks simultane- ously under heterogeneity. Thus:
static
Rideal MoE
(25)
Step MoE Superiority with Routing Noise
Let 7
Rideal > 0 under task heterogeneity, The MoE outperforms the static model if:
R(hMaE) < R
(26)
Substitute:
T
Rideal + P(t)er < Rideal + 7,
(27)
Thus:
Σ P(t)eed <y= Raitic Rideal
(28)
Since R.rrogg,
Re (t), the condition is:
T
Σ
P(troo(Rwongg Re(t))
Riitie
R
(29)
this holds, the MoE's risk, despite routing noise, remains below the best static risk.
The MoE risk is Rideal + Σt, ((t))x((wwong. (t),),, and it outperforms static averaging when routing noise is sufficiently small relative to the static model's suboptimality. The optimal gating case confirms RMoE < R with strict inequality under task heterogeneity.
Conclusion:
B Additional Descriptions
B.1
Details of Dataset and Task Settings
Dataset Details Following prior works (70), we evaluate continual model merging on twenty
publicly available image classification datasets, including SUN397 84), Stanford Cars (32 RE-
SISC45 4 EuroSAT [19] SVHN [46). GTSRB [64] MNIST 34 DTD 7 Flowers
47
PCAM 73 FER2013 [88. Oxford-IIIT Pet [50], STI.-10 9. CIFAR- and CIFAR-10
33
101 3
MaShion-MNIST
[83] EMNIST [10] KMNIST and Rendered SST-2
62).
26

---

Table 8: Extended downstream datasets used in our experiments.
Dataset
#Classes
#Train (k)
#Test (k)
Task
SUN397
287
19.9
19.9
Scene category
Stanford Cars
196
8.1
Car series
RESISC45
45
18.9
6.3
Remote somote-sensing scene
EuroSAT
10
21.6
2.7
Satellite land-use
SVHN
10
73.3
26.0
Digit recognition
GTSRB
43
39.2
12.6
Traffic sign
MNIST
10
60
Hand-written digit
DTD
47
3.8
1.9
Texture recognition
Flowers
102
1.0
6.1
Flower species
PCAM
2
262
32.8
Tumour detection
FER2013
7
28.7
3.6
Facial emotion
Oxford IITT Pet
37
3.7
3.7
Animal species
STL10
10
5
8
Object recognition
CIFAR-100
100
50
10
Natural object
CIFAR-IO
10
50
Natural object
Food101
101
75.8
25.3
Food type
Fashion-MNIST
10
60
10
Fashion product
EMNIST (digits)
10
60
Hand-written digit
KMNIST
10
60
10
Kuzushiji character
Rendered SST-2
2
6.9
1.8
Rendered sentiment
Task Grouping group the 20 datasets into three progressive task sets and evaluate the merged models using average accuracy (ACC) and backward transfer (BWT) metrics. For each task group, we perform experiments using different task sequences (listed Tab. [ and report both the mean and standard deviation of the results to ensure robustness and consistency.
8-task group: (1) SUN397, (2) Stanford Cars, (3) RESISC45, (4) EuroSAT, (5) SVHN, (6)
GTSRB, (7) MNIST, (8) DTD.
14-task group: (1) SUN397, (2) Stanford Cars, (3) RESISC45, (4) EuroSAT, (5) SVHN, (6)
GTSRB, (7) MNIST, (8) DTD, (9) Flowers (10) PCAM, (11) FER2013, (12) OxfordIlIPPet, (13) STL10, (14) CIFAR
20-task group: (1) SUN397, (2) Stanford Cars, (3) RESISC45, (4) EuroSAT, (5) SVHN, (6)
GTSRB, (7) MNIST, (8) DTD, (9) Flowers102, (10) PCAM. (11) FER2013, (12) OxfordIlITPet. (13) STL10, (14) CIFARI00, (15) CIFARIO, (16) Food101, (17) FashionMNIST, (18) EMNIST,
(19) KMNIST, (20) RenderedSST2.
B.2 Details of Downstream Models
In this section, we present the evaluation setup for pre-trained and fine-tuned models. As shown in Tab. I0, we evaluate the zero-shot accuracy of the original CLIP- VIT models and the performance of fine-tuned models on the test sets of various downstream tasks. The fine-tuned checkpoints co/tanganke), where each model
obtained directly from Hugging Face (https://hugging: has been fine-tuned task-specific training data using standard protocol. The visual encoder is updated during fine-tuning, while the classification head fixed and initialized from the pre-trained text encoder. The fine-tuning setup follows standard configuration: cross-entropy loss, Adam optimizer, cosine annealing learning rate schedule with a peak learning rate of le-5, batch size 128, and 4000 training steps.
B.3 Details of Baselines
Our experiments involve the following comparison methods and our method:
Stochastic Weight Averaging (SWA). A simple model averaging technique to stabilize opti- mization and improve generalization 26 At each step I,, the model parameters are averaged across previous checkpoint: 8.S,
0 @sWA 1) + 0SWA]. This approach can be interpreted
27

---

Table 9: Dataset orderings used for experiments in each task group.
Group
Order
Dataset Order (by ID)
000000000000000000000000----00-0
2
000-008-00---000020000----0
3
(03 -02-0-0---00---0
5
-40
000-022-00-00---5-------000000
7 -00-001--000000---------00-
8
001-000-00--02--0----------1---0
10
(08 080-18880000-02--0000000-0000000
(09 -00.0000..000.00---------------0
2 (09 01111:1....11.0011111140000:444.
3 11-.0-1.11666..11100000100444411
5 (08 11010000.......11110000001111110
14 tasks
6 1.1.2...13300111---0000001111113
8 11
9 10.10000000000001111111111111113
1
3 "1111111111100011100010000011411
4 1121111....1.0011111111004414443
5 111.,011.........01.1"1111111114
20 tasks
6 (,h_1//00....1.1.111111100444444
7 "190009...0111111000000031113113
9 [1g_331...0111111111100000101111
10
(01 -
as a form uniform model ensembling. While conceptually straightforward, SWA treats all checkpoints equally and does not account for inter-task conflicts.
Continual Task Arithmetic (C. TA). A training-free merging strategy that linearly combines task-specific fine-tuned models with shared pre-trained model [24]. It computes the merged parameters as @.nerged @merged + 2(6, 0a). where À scaling factor. TA is computationally efficient and easy to apply, but sensitive to À and prone to destructive interference when merging dissimilar tasks.
Continual Mees-Mg (C. Ties). An extension of Task Arithmetic that reduces parameter- level redundancy and sign conflicts during model merging [86]. For task the difference vector A0, Og trimmed and sign-normalized obtain A0liss Ties AOfles A0,), and the
merged model is given by 8 merged merrged + Tes.
Continual Merging (OPCM), A projection- based scheme to mitigate task interference by enforcing orthogonality between parameter updates [70]. Specif- ically, each Ad, is projected onto the orthogonal complement of the subspace spanned by
Orthogonal Projection-
previous updates: @.merged d X1 orthogonal projection.
1x-1A0
merged + + where p(t-1) denotes the
Maximum Magnitude Selection (MagMax). An extension of Task Arithmetic that, for each parameter dimension, selects the update with the larger absolute value: MagMaa MagMax MaMMax A0.). and the merged model given by 0merggdd 0merrgdd MagMox
28

---

Table 10: Test set accuracy of the pre-trained model and individual fine-tuned models on different downstream tasks.
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Flovers 102
PCAM
Model
CLIP-VIT-BB32
Pre-trained
63.2
59.6
60.3
45.0
31.6
32,5
48.3
44.2
66.4
Fine-tuned
74.9
78.5
95.1
97.3
98.9
99.6
79.7
88.0
CLIP-VTT-B/16
Pre-trained
65.5
64.7
66.4
54.1
52.0
43.5
51.7
45.0
71.3
54.0
Fine-tuned
78.9
85.9
96.6
99.0
97.6
99.0
99.7
82.3
94.9
90.6
CLIP-VIT-L/14
Pre-trained
68.2
77.9
71.3
58.4
50.5
76.3
55.5
$1.2
Fine-tuned
92.8
97.4
97.9
99.2
85.5
91.1
OxfordlliTPet
FashioMNIST
RonderedSST2
FER2013
STL10
CIFAR100
CIFAR10
Food101
EMNIST
KMNIST
Model
CLIP-ViT-B/EE
Pre-trained
41.3
83.3
97.1
63.7
89.8
82.4
63.0
12.0
10.0
58.6
Fine-tuned
71.6
92.5
97.5
88.4
97.6
88.4
94.7
95.6
98.2
71.3
CLIP-VYT-B/16
Pre-trained
46.4
88.4
98.3
66.3
90.8
87.0
67.3
12.4
60.6
Fine-tuned
72.8
94.5
98.2
88.8
98.3
91.9
94.5
95.3
98.1
75.7
CLIP-ViT-L/14
Pre-trained 50.0
93.2
99.4
75.1
95.6
91.2
67.0
12.3
9.7
68.9
Fine-tuned
75.9
95.7
99.2
93.0
99.1
94.8
95.3
95.4
98.3
80.5
B.4 Details of Baseline Hyper-parametes
Tab. TI summarizes the hyper-parametent
all baseline methods under different task configurations (8, 14, 20 tasks). Top-k denotes the pruning ratio, TALL the TALL mask threshold, and Cons. the consensus mask threshold. The column LR is the learning rate, while Steps indicates the number of adaptation steps. represents the LoRA rank, and the last column jointly reports the null dimension (k:), EMA decay (3), and relaxation coefficient (7).
Table 11: hyper-ppee parameter settings for all baselines.
Method
Tasks
A
Top-k
TALL
Cons.
LR
Steps
k/3/7
8
0.3
TASK ARITHMETIC
14/20
0.1
0.3
20
TIES HEESGNGGGING
14/20
0.1
CONSENSUS TA
8/14/20
0.1
0.2
LW. ADAMERGING
8/14/20
0.3
le-4
50
WEMOE
8/14/20
0.3
le-4
50
LORA-WEMOR
8/14/20
0.3
lc-4
50
MINGLE-STATIC
8/14/20
0.3
MINGLE (Ours)
8/14/20
lc-4
50
64
3/0.99/1.0
B.5
Comparison of Assumptions and Requirements
Tab. [2 summarizes the assumptions and resource requirements of all baseline methods. We report whether each method requires storing intermediate activations, introduces additional parameters (for storage or inference), and incurs extra test-time computation. Our method only maintains fixed-size covariance matrix instead of full activations, leading to constant memory regardless of test set size. Although LoRA experts are stored, the router merges them into a single model per input, so the effective inference cost matches that of standard individual model.

---

Table 12: Comparison of baseline assumptions and requirements.
Extra Parameters
Extra Parameters (Inference)
Method
Save Activations
Test-time Compute
(Storage)
TASK ARITHMETIC
No
No
TIES-MERGING
No
No
No
No
MAGMAX-IND
No
No
No
No
OPCM
No
No
No
No
CONSENSUS TA
No
Yes
No
No
LW. ADAMERGING
No
No
No
Yes
WEMOE
Yes
MINGLE-STATIC
No
MINGLE (Ours)
No!
Yes²
No
Yes
Only fixed-size covariance matrix is maintained, resulting in constant memory regardless of test size. LoRA experts stored, but the router merges them into single model per input, making the effective inference size equivalent to standard individual model.
C Additional Results
In this section, we provide additional experimental results to support the findings reported the main paper. Specifically, we include: (1) detailed overall performance results (C.T): : (2) accuracy trends
across sequential tasks (C.2 (3) detailed results under distribution shifts (C.3) and (4) extended
visualizations of gate activations and hyper-parameter effects (C.6)
C.1 Detailed Overall Performance Results
Tab. 13 expands on the average results in Tab. [ by reporting per-task average accuracy after continually merging 20 tasks. We compare six methods, SWA, Task Arithmetic, Ties-Merging, MagMax-INe IND, OPCM, and our proposed MINGLE across three CLIP- ViT backbones (B/32, B/16, L/14). MINGLE achieves the highest accuracy on most tasks. These fine-grained results reinforce the main paper's findings, highlighting MINGLE's ability to improve performance on continual model merging.
C.2 Accuracy Trends Across Sequential Tasks
Fig. provides a detailed view of accuracy throughout the continual merging process across different settings, showing both the performance on the current task and on previously encountered ones. The progressive accuracy drop across columns illustrates the degree of forgetting over time. Notably, MINGLE consistently alleviates this degradation, demonstrating markedly reduced forgetting across the full task sequence. Fig. Gig.Gra further compares the average accuracy curves of MINGLE and baseline methods on previously seen tasks after each new model is merged, using the CLIP ViT-B/16 backbone. Results are averaged over random task orderings. MINGLE consistently achieves the highest performance throughout the merging process, with its accuracy curve clearly dominating those of competing methods. Moreover, the narrower standard deviation bands indicate that MINGLE is more robust the task orders.
C.3 Detailed Results Under Distribution Shifts
Tab. 14 expands on Tab, 4 by reporting per-dataset accuracy under both clean test conditions and seven common corruption types: motion blur, impulse noise, Gaussian noise, pixelation, spatter, contrast shift, and JPEG compression. We evaluate six merging methods, across four downstream tasks: Cars, FuroSAT, RESISC45, and GTSRB. This detailed breakdown complements the average results in the main paper, providing more comprehensive assessment of robustness under test-time distribution shifts.
30

---

Table 13: Test set accuracy comparisons on different downstream tasks,
SUN397
Cars
RESISCAS
EunSAT
SYHN
GTSRB
MNIST
DTD
Flowers102
PCAM
Model
VII-B/32
TUNED
53,9
38.2
64.7
45.4
34.4
58.4
57.5
67.7
AVERAGE (SWA)
64.2
59.6
64.8
60,9
47.3
43.1
71.8
46.4
66.5
63.9
C.TA
62.0
51.7
60.9
58.1
48.5
48.9
79.4
46.1
61.1
73.4
C.TIES
62.5
49.1
55.8
50.9
54.6
49.3
82.0
46.7
58.5
69.9
MAGMAX IND
63.6
53.1
39.7
49.1
53.8
53.1
79.8
43.2
56.9
75.1
CONSENSUS TA
37.0
25.2
35.2
36.7
37.3
44,1
80.6
30.3
33.5
89.2
C. LW ADAMERGING
63.1
60.0
63.5
60.1
33.6
32.1
51.8
45.4
66.6
61.2
C. WERME
51,4
45.8
63.3
43.5
42.9
34,6
58.9
46.5
47.5
60.1
OPCM
64.4
51.1
66.0
71.7
66.1
56.0
90.2
40.4
64.9
80.2
MINGLE (OURS)
67.8
58.3
83.5
90.0
82.9
91,8
98.0
65.3
74.0
66.9
MINGLE" (OURS)
68.8
64.2
83.8
91.1
82.4
89.0
96.9
62.8
76.7
72.8
VIE-B/16
C. FIND-TUNED
62.7
58.0
67.6
99.1
46.0
29.2
93.9
61.9
64.1
75,2
AVERAGE (SWA)
67.1
64.6
69.3
63.4
62.4
$2.7
80.7
46.6
71.8
63.1
C.TA
65.8
57.5
63.8
59.5
64.7
54.0
88.0
45.3
67.5
67.1
C.TIES
64.2
52.9
60.9
53.0
62.8
48.8
88.4
45.0
61.3
68.5
MAGMAX IND
65.8
51.8
57.8
42,6
54.4
43.7
83.0
42.8
60.4
69.8
CONSENSUS TA
42.6
24.8
30.4
34.4
47.6
42.2
79.9
30.6
36.2
74.3
C. LW ADAMERGING
65.5
65.7
69.8
59.4
50.1
44.2
47.1
71.8
57.9
C. C.LARAA AERMO
62.7
60.2
69.4
37.7
52.1
39.9
63.1
45.3
64.3
51.7
OPCM
67.9
73.7
74.4
63.2
72.3
79.6
MINGLE (OURS)
71.5
64.9
85.3
90.0
87.5
90.1
97.1
62.7
82.6
80.6
MINGLE" (OURS)
72.0
72.1
87.9
87.1
89.2
97.4
62.5
86.8
76.4
VIE-L/14
C. FINE-TUNED
69.5
73.6
78.3
99.2
59.3
49.3
69.7
83.2
78.3
AVERAGE (SWA)
70.7
77.7
76.4
75.3
69.5
62.1
93.7
57.7
80.0
73.6
C.TA
70.4
74.1
73.9
66.3
69.9
65.6
95.1
36.6
78.6
70.4
C.TIES
69.7
70.3
65.3
47.9
76.1
63.6
94.7
54.4
77.9
72.3
MAGMAX- IND
73.1
73.7
75.6
64.6
73.7
68.8
94.6
56.1
78.0
71.7
CONSENSUS TA
50.7
39.1
31.7
36.4
39.4
44,9
88.5
33.8
45.7
62.5
C. LW ADAMERGING
68.8
78.6
75.9
65.7
58.3
$1.6
79.9
57.4
80.6
52.4
L.RA--E
62.1
68.1
68.7
53.2
47.5
49.4
66.2
54.2
OPCM
73.1
78.3
82.4
80.2
80.8
80.4
97.4
61.6
84.8
76.3
MINGLE (OURS)
75.9
83.4
87.8
88.7
91.1
94.5
98.4
70.8
94.8
75.3
MINGLE" (OURS)
74.3
85.9
00.5
92.5
90.1
92.7
98.1
69.2
95.7
74.0
OxfordUUPPet
FashiomMISST
RenderedSS12
FER2013
STL.10
CIFAR100
CIFARIO
Food101
EMNIST
KMNIST
Model
VIT-R/32
C. FIND- TONED
58.3
86.7
40.2
70.5
50.0
72.4
54.5
54.5
AVERAGE (SWA)
50.2
84.1
97.0
69.8
92.7
80.4
71.3
15.0
11.5
61.8
C.TA
51.4
94.9
64.6
91.4
71.9
73.9
17.8
12.2
59.9
C.TIES
49.5
81.3
95.2
91.2
70.2
73.7
17.8
16.9
59.8
MAGMAX-IND
56.5
94.6
91.9
73.8
18.5
15.4
63.9
CONSENSUS TA
41.7
58.8
81.8
41.5
78.1
29.8
72.6
17.4
18.5
$4.1
C. LW ADAMERGING
43.2
83.7
96.8
67.0
89.9
81.6
63.7
16.8
10.7
59.1
C. LORA--RREOO
44.6
72.5
86.1
63.8
63.8
48.1
10.3
12.8
55.7
OPCM
58.5
82.9
95.9
67.6
92.8
74.0
76.3
22.4
18.3
64.6
MINGLE (OURS)
65.0
85.5
97.0
72.6
81.5
85.4
50.4
65.2
67.1
MINGLE" (OURS)
65.3
88.5
97.7
73.9
94.7
83.7
39.3
56.1
68.7
VIT-R/16
C. FINE-TUNED
60.5
84.5
90.5
38.8
73.6
61.9
89.7
83.3
$1.5
72.8
AVERAGE (SWA)
50.9
89.6
98.0
15.6
12.4
C.TA
50.7
89.3
97.0
68.0
93.1
80.3
75.7
18.1
16.7
61.8
C.TIS
50.4
87.9
96.3
63.1
91.7
78.0
75.0
23.4
24.9
61.5
MAGMAX-IND
57.7
88.8
97.5
71.5
94.4
81.3
77.2
24.5
25.0
59.4
CONSENSUS
45.6
76.8
87.7
44.4
82.2
38.4
72.7
18.8
30.0
58.6
C. LW ADAMERGING
46.8
88.9
98.1
91.4
86.6
67.2
17.2
11.0
59.2
C. WEMOE
45.6
91.2
92.3
41.3
64.3
78.1
48.0
23.5
16.6
$2.7
OPCM
59.5
97.7
73.2
94.7
83.1
81.3
26.5
23.4
66.8
MINGLE (OURS)
67.6
92.7
97.4
74.0
95.3
87.7
87.4
73.5
79.9
74.0
MINGLE" (OURS)
67.9
93.5
98.4
96.4
89.7
87.8
56.6
64.5
75.3
VIT-L/14
C. FINE-TUNEE
68.0
92.1
94.5
60.5
85.7
74.8
93.1
89.0
59.2
78.8
AVERAGE (SWA)
52.7
94.2
99.2
81.7
97.0
90.7
77.4
16.1
10.4
66.1
C.TA
55.7
94.2
98.6
79.1
96.6
87.6
80.8
17.6
10.6
63.6
C/TIES
57.6
93.5
97.8
74.0
95.6
8A.7
79.7
20.2
12.6
58.4
MAGMAX-ING
$2.9
93.9
98.7
82.1
97.3
89.5
81.6
19.2
11.1
68.4
CONSENSUS TA
50.3
81.2
89.7
47.5
86.2
43.5
75,3
14.5
10.4
53.4
C. LW ADAMERGING
49.2
93.5
99.3
77.2
95.8
91.1
68.2
18.6
9.8
66.6
C. LORA WEMOE
46.3
84.5
87.6
52.1
70.5
73.3
18.7
10.9
56.5
OPCM
61.8
95.4
99.2
83.0
97.8
90.9
86.0
26.4
14.7
71.0
MINGLE (OURS)
67.7
96.0
98.7
81.4
97.1
90.6
90.6
60.7
88.6
74.8
MINGLE" (OURS)
67.9
96.0
99.4
84.7
97.8
92.4
88.8
53.0
57.1
75.5
31

---

129
(a) ViT-B/32 on 8 Tasks
(b) ViT-B/16 8 Tasks
(c) ViT-L/14 on 8 Tasks
10..0
120
K:H
KNT VOS
415
150001
(d) ViT-B/16 on 14 Tasks
(c) ViT-L/14 on 14 Tasks
CBU
5
5
5
VTL-
5
S
5
5
5
$
3
5
8
AST
6.99
XM
Inp
Tea Butk
(I) VIT-L/14 on 20 Tasks
Figure 5: Accuracy matrices of MINGLE (VTT-B/32. ViT-B/16, and ViT-L/14) under different task settings.
32

---

Mag
1.0
0.9
Actueyy
0.8
0.7
2
(a) 8 tasks
TA
C. Tice
Mag
OPCM
MINGLE
1.00
0.85
Aceinxy
0.90
0.70
0.65
10
Tosk Number
(b) 14 tasks
MagMax
1.0
0.9
0.5
Asuiny
0.5
Task Numder
(c) 20 tasks
Figure 6: Sequential test accuracy curves of MINGLE and baselines (C.TA, C.Ties, MagMax, OPCM) under different task settings. Shaded regions indicate standard deviation across task orders.
33

---

Table 14: Robustness results when merging ViT-B/31 models on four tasks.
Clean Test Set RESISC45 GTSRB
Corruption: Motion Blur
Method
Cars
Avg ACC
Cars
EuroSAT
RESISC45
GTSRB
Avg
C. LAW ADAMERGING
65.3
49.7
65.4
43.8
56.0
64.2
25.6
62.5
37.5
47.5
C. WEMOE
0.5
2.6
2.5
3.4
0.5
3.1
KORA-WEMONT
78.7
64.8
71.0
TASK ARITHMETIC
64.6
80.2
74.8
77.5
62.3
59.4
78.5
65.9
MAGMAX-IND
63.1
81.7
82.5
79.1
61.4
80.0
72.6
69.0
OPCM
65.7
92.3
85.7
90.5
83.6
62.8
62.5
83.7
82.2
72.8
MINGLE (Ours)
74.4
96.5
91.5
97.3
89.9
73.2
70.5
91.9
95.8
82.9
Corruption: Impulse Noise GTSRB
Corruption: Gaussian Noïse
Method
Cars
EuroSAT RESISCAS
Avg ACC
Cars
FurSAAT RESISC45
GTSRB
Avg ACC
C. LW ADAMERGING
60.5
30.1
56.3
25.5
43.1
62.3
25.6
59.7
25.6
43.3
C. WEMOR
0.5
11.2
2.3
3.2
4.3
0.5
8.1
2.4
2.8
3.4
C. LORA-WEMOR
62.2
23.4
69.9
64.6
55.0
64.9
31.7
77.8
63.4
59,4
C. TASK ARITHMETIC
59.9
57.7
72.9
45.0
58.9
61.8
51.4
75.1
50.1
59.6
MAGMAX-ING IND
59.2
56.3
74.3
52.5
60.6
60.6
51.7
77.0
56.5
61.5
OPCM
61.1
57.1
78.5
62.0
64.7
63.0
52.4
80.7
64.9
65.2
MINGLE (Ours)
69.6
28.0
86.1
86.1
67.5
72.0
38.5
89.4
82.9
70.7
Corruption: Pixelate
Corruption: Spatter
Method
Cars
EuroSAT RESISC4S GTSRB
Avg ACC
Cars
EuroSAT
RESISC45
GTSRB
Avg ACC
C. LW ADAMERGING
3.4
16.5
13.5
39.2
18.1
61.3
34.1
58.2
32,8
46.6
C. WEMOE
0.5
6.3
2.5
2.5
3.0
0.5
10.1
2.7
2.6
4.0
C. LORA-WEMOE:
WEMOE
0.8
26.0
5.8
67.0
24.9
62.4
35.4
71.2
73.0
60.5
C. TASK ARITHMETIC
2.5
31.7
19.1
65.6
29.7
61.2
72.7
63.5
MAGMAX-IND
2.6
19.3
74.0
33.0
60.0
66.1
66.4
OPCM
35.2
61.5
70.5
MINGLE (Ours)
2.3
35.6
18.5
95.1
37.9
70.1
57.8
86.2
93.9
77.0
Corruption: Contrast
Corruption: JPEG Compression
Method
EuroSAT
RESISCAS
GTSRB
Avg ACC
Cars
EuroSAT RESISC45
GTSRB
Avg ACC
C. LW ADAMERGING
61.8
63.1
44.8
48.9
65.1
29.6
65.4
36.4
49.1
C. WEMOE
0.5
2.3
3.3
0.5
2.4
2.7
4.0
WEMOE
64.3
68.5
65.5
C. TASK ARITHMETIC
62.5
55.2
75.3
70.8
66.0
64.1
66.2
80.0
67.8
MAGMAX-IND
61.3
58.0
76.9
78.2
68.6
62.5
67.7
81.1
68.5
69.9
OPCM
63.8
57.5
81.3
87.4
72.5
65.0
68.0
85.4
79.3
74.4
MINGLE (Ours)
72.4
60.1
90.4
97.3
80.1
73.7
73.5
92.0
92.4
82.9
C.4
Inference Efficiency and Parameter Overhead
Tab. compares the inference efficiency and parameter overhead of all baselines on the CLIP V/T-B/32 model after merging eight tasks. We report the total number of parameters, additional storage and inference parameters, throughput (images per second), and accuracy. The results show that most static merging methods (e.g., Task Arithmetic, Ties-Merging MAGMAX-Ind, OPCM) incur no extra storage or inference overhead, but typically achieve limited accuracy. Consensus TA and WEMOE introduce significant storage overhead, while WEMOR also scales up inference parameters considerably. By contrast, MINGLE achieves a favorable trade-off: although introduces additional parameters for LoRA experts and the router, the effective inference overhead remains small, and throughput is only marginally reduced compared to static baselines. This efficiency advantage comes while delivering substantially higher accuracy.
C.5 Forward Transfer Analysis
Forward transfer (FWT) an important metric continual learning, as it quantifies how effectively prior knowledge facilitates the learning of future tasks. We adopt the standard definition:
T
(0phooo) - az
FWT T
(30)
where ae ( denotes the test accuracy task / using the merged model after task 6, and ã the accuracy of the individually fine-tuned model for task Positive FWT indicates beneficial transfer, while negative values suggest interference.
34

---

Table 15: Comparison of inference efficiency and parameter overhead on CLIP ViT-B/32 model after
eight tasks merging.
Method
Total Params (M)
Extra Storage (M)
Extra Inference (M)
Throughput (img/s)
ACC (%)
TASK ARITHMETIC
87.5
0.0
~910
67.5
TIES-MERGING
~910
49.0
MAGMAX-IND
87.5
0.0
~910
70.7
OPCM
87.5
0.0
9910
75.5
CONSENSUS TA
87.5
87.5
910
69.0
LW. ADAMERGING
87.5
0.0
9
52.9
WEMOE
540,9
453.4
858
WEMOE-LORA
103.7
16.2
0.07
-88
66.6
MINGLE
173.1
85.6
0.6
-41
85.8
MINGLE
113.7
26.2
0.3
862
85.0
Table 16: Forward transfer (FWT) results on 8-task continual merging with CLIP ViT-B/16.
Method
ACC (%)
BWT (%)
FWT (%)
TASK ARITHMETIC
77.1 = 0.0
-4.2 1.0
-13.4 1 0.0
TTES-MERGING
66.8 3.7
-5.5 0.4
-30.7 + 9.9
OPCM
81.8
-4.8
-9.0 0.4
MINGLE (Ours)
88.3 0.6
-0.4 0.1
-388 +: 0.8
Tab. 16 reports the results for the 8-task continual merging setup on CLIP ViT-B/16. The results
demonstrate that MINGLE achieves nearly zero forgetting (BWT ss while obtaining the highest forward transfer among all baselines, showing that our adaptive gating and merging strategy not only preserves past knowledge but also enhances feature utility for future tasks.
C.6 Additional Visualizations of Gate Activations and the Relaxation Effect
We provide an extended ablation study on gate hyper-paramette including visualizations of gate activations under 14-task (Fig. 7 and 20-task (Fig. ( configurations, complementing the 8-task results presented the main paper. The visualizations demonstrate that the null-space constraint remains effective the number of tasks increases, consistently suppressing gate responses inputs from previously seen tasks and thereby mitigating forgetting.
35

---

Gat
Cone A-t-----
Cane Acto alee
Casec Auct aien
Madry
Orteaiity
ofteee
Resscce
Faodlyy
Powerttd
cTE
Aater
chtec
Pooud
Fanelay
sints
Sllvyy
Can
s arttor
Can
Cane Setetaatt
Capc
Calcketirain
9
cressso
can Potae
Prventto
pmm
sints
can
Plemertt
art
dak
Gatccettvaii dee
DiD
araay SENoy
Devestta
pm
coneros
con
Demeree
Git Up
Gae 13
Gat 12
aitie
DrD Aastr
Faeiier
pm Anster pit
Geec satvaies
9
9
arrie
dTaa
kooice
Feetarr
Cao
Rewerti
aizi
lsuer
Con
NplleSpace
Figure 7: Visualization of gate activations across 14 tasks under varying y values. Each subplot corresponds to a gate, with curves and shaded regions denoting the mean and standard deviation of activations across layers. Gray bars mark the training dataset for each gate. Smaller values result in stronger suppression of activations on previously learned tasks.
36

---

www.
5
www.
Figure 8: Visualization of gate activations across 20 tasks under varying y values.
37

---

D Discussions
D.1 Use of Unlabeled Adaptation Samples
In our experiments, we simulate a realistic deployment setting by randomly sampling 5 unlabeled examples per class from the test split, which serve as adaptation samples for model merging. Such small unlabeled buffers are practical in real-world applications and can be obtained from various sources, including incoming test-time data such as recent user queries or model inputs, (ii) held-out validation inputs small training subsets (if available), (iii) user neer-provide samples (e.g., few-shot examples) that do not raise privacy concerns, (iv) synthetically generated data, or (v) manually curated public data. Importantly, our method does not depend precise sample selection, and the buffer size remains fixed and small, ensuring feasibility and robustness in deployment scenarios.
D.2 Relation to Rehearsal-Free Continual Learning
Test-time continual model merging (TTCMM) is closely related to the paradigm of rehearsal-free continual learning (RFCL). as both approaches share two fundamental constraints: (i) they do not retain past training data, and (ii) they avoid storing previous task models. The key distinction lies in the information available each stage. RFCL assumes access to the training data of the current task and incrementally fine-tunes single model over time. In contrast, TTCMM assumes access to independently fine-tuned models for each new task and focuses on merging these expert models rather than training them from scratch. Additionally, TTCMM relies on small unlabeled buffer at test time (e.g.. 5 samples per class) to guide the merging process.
From a privacy perspective, TTCMM provides stronger guarantees. Since it does not require access full training sets, it only depends on a small set of unlabeled samples, which can be user-provided without risk, synthetically generated, or curated from public data. By comparison, RFCL requires access to large-scale labeled datasets for every task, raising more significant concerns regarding privacy, storage, and legal constraints (e.g., medical images, personal data, or copyrighted corpora). The reliance on tiny unlabeled buffer makes TTCMM more practical scenarios where data privacy is a primary consideration.
D.3 Limitations
As with many model merging methods, our approach assumes that all independently fine-tuned models originate from shared pretrained initialization. The extent to which this assumption influences merging performance remains unclear and warrants further investigation. In addition. our current experiments focus on merging models with identical backbone architectures (e.g., CLIP VřT-B/16). Although our use of LoRA-based expert offers some structural uniformity, which could potentially accommodate heterogeneous backbones, we have not yet explored this setting. Extending our framework to support diverse initialization points or architectural variants remains an open direction for future work.
D.4 Broader Impacts
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
38


Reviews:
[{"review_id": "YCsde8hVvu", "reviewer": "Reviewer_cHnS", "summary": "The paper introduces a novel paradigm called Test-Time Continual Model Merging (TTCMM), which uses a small set of unlabeled test samples from the current task to guide the merging process at test time. Then the paper introduces a method for this setting, called MINGLE, designed to integrate independently fine-tuned models sequentially without needing the original training data. The method uses MoEs composed of LoRAs and proposes a “Null-Space Constrained Gating” mechanism that restricts updates to subspaces orthogonal to the representations of previous tasks. It also introduces “Adaptive Relaxation Strategy” that dynamically modifies the constraint's strength based on interference signals detected during adaptation. The authors include experiments on CLIP ViTs for sequences of 8, 14 and 20 tasks.", "questions": "See weaknesses. Also:\n\nWhat is the task-specific routing error $\\epsilon_t$? Is it the error of not selecting the correct expert for an input X? If yes, doesnt this defeat the purpose of L143-144 (“it cannot specialize to regions where one expert is clearly superior.”)?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "*Strengths*\n\n1. The paper is well-written and easy to follow. The motivation for using a dynamic Mixture-of-Experts approach over static averaging is clearly articulated. The same applies to core ideas, such as null-space projection, which are intuitive and well-explained.\n\n2. Strong performance: MINGLE consistently outperforms a wide range of baselines across multiple model backbones (ViT-B/32, B/16, L/14) and on task sequences of varying lengths (8, 14, and 20 tasks). The method also shows strong robustness to various data corruptions. See weaknesses for some issues.\n\n3. The paper successfully combines multiple ideas such as MoE, LoRA, and orthogonal projections. The ablation study effectively demonstrates each component’s contribution to the final performance.\n\n\n*Weaknesses*:\n\n\n1. Questionable Problem Setting: The studied setting is somewhat difficult to justify in a practical scenario. The paper assumes access to the current task's model ($\\\\theta_t$) and its unlabeled test data, but *no* access to the data or even the task vectors from previous tasks. It is unclear why the previous task vectors, which are just model parameters, would be unavailable while the models themselves are being merged. Furthermore, since the seed data is unlabeled, it is not obvious why similar unlabeled data from previous tasks could not also be stored or accessed. Overall, the setup feels contrived and needs stronger motivation.\n\n2. Potentially Unfair Baseline Comparisons: The comparison to other methods may be unfair due to the additional memory and computational overhead (additional parameters, saving activations, access to data) of the proposed method. The work notes similarity to WEMOE, but it is tailored to the continual settin–is the performance gap due to differing assumptions rather than strong methodological contributions?\n\n3. High Number of Hyperparameters: The proposed method introduces a considerable number of new hyperparameters, including the LoRA rank $r$, the null-space dimension $k$, the EMA decay $\\\\beta$, the relaxation coefficient $\\\\gamma$, the number of adaptation steps. This effectively limits the applicability of the method (compounded by the next point)\n\n4. Limited Architectural Scope: The experiments are exclusively focused on CLIP-ViT models. It is unclear how the setting applies to LLMs, such as T5, Llama, mistral etc, which are commonly used in the model merging literature.\n\n5. Minor issues wrt writing: explain $\\\\alpha$ in Equation 3, missing \\[a\\] for the benchmarks of 8,14 and 20 tasks. Better differentiate in the related work section and introduction works like TA, wise FT etc which focus on the static setup but are sometimes grouped with more continual papers.\n\n\\[a\\] Wang, K., Dimitriadis, N., Ortiz-Jimenez, G., Fleuret, F., & Frossard, P. (2024). Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813.", "was_revised": true, "final_justification": "Given the authors' rebuttal, I have updated my score to 4 maintaining some concerns on the practicality of the setting.", "rebuttal": "# W1 — Problem setting motivation\n\nWe thank the reviewer for prompting us to better position our setting within established paradigms. \n\nOur proposed setting (TTCMM) is directly inspired by the well-known **rehearsal-free continual learning (RFCL)** framework [1,2], sharing key constraints:\n\n-  **No access to prior task data**\n-  **No storage of past task-specific models**\n\nThis setup is practical, motivated by two real-world constraints:\n\n-  **Storage**: Retaining full models—or even **task vectors**, which are equally large—leads to linear memory growth, which is impractical for edge or long-horizon deployments\n-  **Privacy**: Even unlabeled data (e.g., medical scans, GPS logs) can be sensitive and must be discarded post-training due to laws like HIPAA or GDPR\n\nThus, TTCMM offers a natural extension of RFCL into model merging. We will clarify this link in Sec. 1 & 2 and include examples (e.g., mobile agents, medical federations) to highlight practical relevance.\n\n[1] Dualprompt: Complementary prompting for rehearsal-free continual learning (ECCV2022)\n\n[2] Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning (CVPR2022)\n\n# W2 – Potentially Unfair Baseline Comparisons\n\nWe acknowledge that our setup is not identical to every baseline. To promote transparency, we include the following table comparing each method on their data access, extra parameters, and test-time compute:\n\n**Table S.1: Comparison of baseline assumptions and requirements**\n\n| Method            | Save Activations | Extra Parameters (Storage) | Extra Parameters (Inference) | Test-time Compute |\n| ----------------- | ---------------- | -------------------------- | ---------------------------- | ----------------- |\n| Task Arithmetic   | No               | No                         | No                           | No                |\n| Ties-Merging      | No               | No                         | No                           | No                |\n| MAGMAX-Ind        | No               | No                         | No                           | No                |\n| OPCM              | No               | No                         | No                           | No                |\n| Consensus TA      | No               | Yes                        | No                           | No                |\n| LW. AdaMerging    | No               | No                         | No                           | Yes               |\n| WEMOE             | No               | Yes                        | No                           | Yes               |\n| MINGLE- Static    | No               | No                         | No                           | No                |\n| MINGLE (Ours) | No*¹             | Yes*²                      | No*²                         | Yes               |\n\n*¹: Our method does not store activations. Only a fixed-size covariance matrix is maintained (Line 195), resulting in constant memory regardless of test set size.\n\n*²: While LoRA experts are stored, the router merges them into a single model per input. Thus, the effective inference size matches a standard individual model.\n\n**Table S.2: More comparative results  (ACC%) for continual merging performance.**\n\n| Method              | ViT-B/32 (8)  | ViT-B/32 (14) | ViT-B/32 (20) | ViT-B/16 (8)  | ViT-B/16 (14) | ViT-B/16 (20) |\n| ------------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n| LW. AdaMerging      | 52.9 ±3.5     | 60.1 ±1.7     | 57.5 ±0.7     | 59.8 ±3.2     | 64.8 ±1.6     | 61.4 ±0.8     |\n| WEMOE               | 4.9 ±0.6      | 8.2 ±0.3      | 10.4 ±0.2     | 4.0 ±0.7      | 8.3 ±0.4      | 10.1 ±0.5     |\n| WEMOE-LoRA          | 66.6 ±6.6     | 63.4 ±2.2     | 44.0 ±19.5    | 72.1 ±4.6     | 55.8 ±27.0    | 41.3 ±22.8    |\n| Consensus TA        | 69.0±1.1      | 64.0 ±0.9     | 45.6 ±1.6     | 73.3±0.3      | 69.0 ±1.1     | 50.0 ±2.4     |\n| MINGLE-Static       | 74.3±0.3      | 72.9±0.8      | 67.3±0.8      | 78.7 ±0.1     | 76.4 ±1.0     | 70.6 ±0.4     |\n| MINGLE (Ours)  | **85.8 ±0.8** | 81.6 ±1.4     | **77.1 ±2.0** | **88.3 ±0.6** | **84.9 ±0.8** | **81.9 ±0.9** |\n| MINGLE*  (Ours) | 85.0 ±0.5     | **81.7 ±1.0** | **77.1 ±1.3** | 87.0 ±0.6     | 84.7 ±1.0     | 81.6 ±1.3     |\n\n**On TTA & MoE:** TTA and MoE are not entirely new assumptions in the model merging literature (e.g., WEMOE (TTA+MoE) [ICML 2024], AdaMerging (TTA) [ICLR 2024], Twin-Merging (TTA+MoE) [NeurIPS 2024]). \n\n**To ensure fair comparisons,** we have added results for **WEMOE** and **AdaMerging** in Table 1 (also shown in Table S.2). Additionally, we report a **MINGLE-Static** baseline (Table 4, Row 1), where LoRA modules are combined using fixed coefficients (e.g., 0.3) without any adaptation or routing. We will also include **Table S.1** to summarize the assumptions and requirements of all baselines. These additions ensure clarity and transparency.\n\n\n\n**Why the gap vs. WEMOE?**\n\nTo clarify: **both our method and WEMOE rely on the same MoE and TTA assumptions**. In our experiments, we carefully re-implemented **WEMOE under TTCMM setting**. This ensures that any observed performance gap arises **not from differences in assumptions**, but purely from **methodology**.\n\nWEMOE uses full MLP blocks as experts, which are parameter-heavy and hard to route with limited seed data.\nOur method uses lightweight LoRA adapters, which are easier to gate and effectively regularized via a null-space constraint.\n\nTo isolate this factor, we introduced a \"**WEMOE-LoRA**\" variant, replacing WEMOE’s MLP experts with LoRA. As Table S.2 shows, this closes the gap and supports that:\n\n*  heavy experts degrade routing in low-data regimes, and\n*  our gains primarily stem from constrained gating + LoRA, not from relaxed assumptions.\n\nWe deeply appreciate the reviewer’s question. We will clarify this explicitly in revision.\n\n# W3 – High number of hyperparameters \n\nWe sincerely thank the reviewer for this thoughtful and valid concern. While our method introduces five scalar hyperparameters, each plays a minimal but necessary role. \n\nImportantly, our method **requires no tuning**. We apply a **single fixed configuration** across **all experiments**: all task counts (8/14/20), all 10 task orders, and all model backbones (ViT-B/32, B/16, L/14)—**without any modification**. This deliberate design ensures **strong generalization and ease of deployment**.\n\nWe’ll clarify this global configuration strategy in the main text.\n\n# W4 – Limited Architectural Scope\n\nWe thank the reviewer for raising this point. While our main experiments focus on CLIP model, **our method is not restricted to vision models**.\n\nTo show generality, we applied MINGLE to Flan-T5 (T5-base) on 7 GLUE tasks (merged alphabetically), using only 100 unlabeled test samples for adaptation. The results below show that our method remains effective in non-visual, sequence-based encoder-decoder architectures, confirming that MINGLE generalizes beyond CLIP-style backbones.\n\n**Table S.3: Accuracy (%) / Spearman’s ρ  of continual merging performance using Flan-T5 (T5-base) on 7 GLUE benchmark tasks.**\n\n| Model                 | GLUE-COLA | GLUE-MNLI | GLUE-MRPC | GLUE-QNLI | GLUE-QQP | GLUE-SST2 | GLUE-STSB | Average |\n| --------------------- | --------- | --------- | --------- | --------- | -------- | --------- | --------- | ------- |\n| Pre-trained           | 69.1      | 56.5      | 76.2      | 88.4      | 82.1     | 91.2      | 62.2      | 75.1    |\n| Individual fine-tuned | 75.0      | 83.4      | 87.5      | 91.5      | 85.4     | 93.6      | 88.7      | 86.4    |\n| C. Task Arithmetic    | 70.0      | 65.1      | 80.4      | 90.1      | 83.0     | 92.4      | 77.7      | 79.8    |\n| C. Ties-Merging       | 49.5      | 82.7      | 83.3      | 89.9      | 82.3     | 90.8      | 78.2      | 79.5    |\n| LW. AdaMerging        | 69.1      | 58.1      | 77.9      | 88.9      | 83.1     | 90.7      | 74.8      | 77.5    |\n| WEMOE-LoRA            | 64.5      | 80.8      | 79.7      | 90.8      | 80.3     | 90.6      | 80.0      | 81.0    |\n| **MINGLE**            | 74.8      | 80.0      | 87.5      | 90.5      | 80.9     | 92.8      | 87.0      | 84.8    |\n\n\n# W5 – Minor Writing Issues\n\n**Equation 3 — Clarifying $\\alpha$**:\nWe will clarify that $\\alpha$ denotes the effective rank of the merged task vector, computed as:\n$$\n\\alpha = \\left\\lfloor \\exp\\left( -\\sum_i p_i \\log p_i \\right) \\right\\rfloor, \\quad \\text{where } p_i = \\frac{s_i}{\\sum_j s_j}\n$$\n\nand ${s_i}$ are singular values from SVD. This will be added near Equation 3.\n\n**Missing [a] in benchmarks**:\nThanks for catching this. We’ve reproduced baseline [a] (see Table S.2), will include it in Table 1, and cite it in Sec. 2 for completeness.\n\n**Static vs. Continual baselines**:\nWe’ll revise the Introduction and Related Work to clarify that TA and WiSE-FT are static model merging methods—not MoE-based or continual—and ensure they are properly grouped.\n\n\n\n# Q1 – On the task-specific routing error $\\varepsilon_t$\n\nWe thank the reviewer for raising this point and will clarify the definition in the main text.\n\nWe define the task-specific routing error as:\n$$\n\\varepsilon_t = \\Pr\\big(i^*(x) \\ne t \\mid x \\sim D_t\\big),\n$$\n\nwhere $t$ is the ground-truth task and $i^*(x)$ is the expert selected by the router. Thus, $1 - \\varepsilon_t$ reflects correct routing probability. A **lower $\\varepsilon_t$** indicates **better expert-task alignment**, with $\\varepsilon_t = 0$ meaning perfect routing. This is defined in Appendix (Line 602) and will be referenced in the main text.\n\nRegarding Lines 143–144, the critique targets **static baselines**, which use fixed or implicit routing. In contrast, our method **actively minimizes $\\varepsilon_t$** via TTA, enabling expert specialization.\n\nWe sincerely thank the reviewer again, and hope our clarifications will support a higher evaluation."}, {"review_id": "9qYjLuag7r", "reviewer": "Reviewer_Y4Kk", "summary": "MINGLE integrates test-time adaptation into model merging (Test-Time Continual Model Merging, TTCMM). It uses a small buffer of unlabeled current-task samples to guide the merge process at inference. The model has a mixture-of-experts design: each task adds a lightweight low-rank expert module and an input-dependent gating function. When a new task is added, its expert and gate are appended; only the new gate is tuned on the current task’s data while earlier experts and gates remain frozen. This lets the unified model adapt to the new task’s distribution on the fly, without retraining or accessing old data.", "questions": "1. Provide at least one experiment on a non-vision backbone (e.g., a RoBERTa model fine-tuned on text tasks or a wav2vec-based speech model) or justify why the method is inherently vision-only.\n2. Regarding efficiency: (i) report wall-clock adaptation time for 14- and 20-task runs; (ii) add an ablation that sweeps 5/10/20 steps to show accuracy/forgetting trade-offs; (iii) discuss possibilities for step-freezing or fast first-order updates.\n3. Add a consolidated table in the appendix listing all baselines’ key settings (learning rates, adaptation steps, projection ranks, etc.) and clarify whether you performed any per-method tuning on your task orders.\n4. You consider five unlabeled samples per class, selected from the test split: (i) Clarify how this buffer would be obtained in practice; (ii) extend Fig. 5 by including the “zero-sample” condition (pure parameter-space merge) to examine robustness.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths \nIn terms of quality, it presents a solid theoretical justification (via Theorem 1) for using a dynamic mixture-of-experts rather than static averaging, accompanies that with a clear algorithm (Algorithm 1) and detailed equations, and backs it up with a comprehensive empirical study—covering three CLIP backbones, task sequences of 8/14/20 tasks, seven corruption benchmarks, ablations, and efficiency analyses. Clarity is strong: the narrative flows logically from motivation through theory to experiments; Figures 1-2 quickly convey what distinguishes Test-Time Continual Model Merging (TTCMM) from prior paradigms; notation is consistent; and hyper-parameter roles are later dissected. The significance is notable because the work tackles a practical scenario—merging independently fine-tuned models without sharing data—and achieves 7–9 percentage-point gains in average accuracy while almost eliminating backward forgetting. Finally, on originality, the paper introduces the TTCMM setting and proposes Adaptive Null-Space Constrained Gating, which relaxes hard orthogonality on-the-fly based on measured interference; together, these represent a fresh and principled combination of ideas that goes beyond existing continual model-merging approaches.\n\n\nWeaknesses \nThe main limitations revolve around scope, experimental breadth, and some presentation details. Regarding quality, all experiments focus on vision tasks with CLIP backbones; the method’s applicability to NLP, speech, or non-contrastive architectures is untested. The test-time adaptation loop requires roughly 50 gradient steps per task, which, while measured, could be too heavy for latency-critical or edge deployments, and baseline hyper-parameter parity is not fully transparent. Concerning significance, the method still assumes that each task ships its own LoRA weights plus a small buffer of unlabeled samples, constraints that may not hold in truly federated or on-device scenarios. Finally, in originality, while the adaptive gating mechanism is clever, its components—LoRA experts, MoE routing, orthogonal projection, EMA statistics—are individually incremental; the novelty lies more in their particular integration than in a new algorithmic primitive.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns", "rebuttal": "# W1 – Limited Architectural Scope\nWe thank the reviewer for this point. While our main experiments use CLIP-ViT, MINGLE is not vision-specific. We applied it to Flan-T5 (T5-base) on 7 GLUE tasks (merged alphabetically) using only 100 unlabeled test samples, confirming strong performance on encoder-decoder models and demonstrating generality.\n\n**Table X.1: Accuracy (%) / Spearman’s ρ  of continual merging performance using Flan-T5 (T5-base) on 7 GLUE benchmark tasks.**\n\n| Model                 | GLUE-COLA | GLUE-MNLI | GLUE-MRPC | GLUE-QNLI | GLUE-QQP | GLUE-SST2 | GLUE-STSB | Average |\n| --------------------- | --------- | --------- | --------- | --------- | -------- | --------- | --------- | ------- |\n| Pre-trained           | 69.1      | 56.5      | 76.2      | 88.4      | 82.1     | 91.2      | 62.2      | 75.1    |\n| Individual fine-tuned | 75.0      | 83.4      | 87.5      | 91.5      | 85.4     | 93.6      | 88.7      | 86.4    |\n| C. Task Arithmetic    | 70.0      | 65.1      | 80.4      | 90.1      | 83.0     | 92.4      | 77.7      | 79.8    |\n| C. Ties-Merging       | 49.5      | 82.7      | 83.3      | 89.9      | 82.3     | 90.8      | 78.2      | 79.5    |\n| LW. AdaMerging        | 69.1      | 58.1      | 77.9      | 88.9      | 83.1     | 90.7      | 74.8      | 77.5    |\n| WEMOE-LoRA            | 64.5      | 80.8      | 79.7      | 90.8      | 80.3     | 90.6      | 80.0      | 81.0    |\n| **MINGLE**            | 74.8      | 80.0      | 87.5      | 90.5      | 80.9     | 92.8      | 87.0      | 84.8    |\n\n# W2 –  Efficiency\n### (i) Wall-clock adaptation time\nWe report the wall-clock adaptation time in Table X.2. Adaptation runs once per task (~10s on RTX 4090, Table X.2). Afterward, the router is fixed and inference is feedforward with no TTA, including prior tasks—supporting low-latency deployment.  (see Table Z.1 for Reviewer np8P for inference efficiency).\n\n**Table X.2: Wall-clock adaptation time for ViT-B/32**\n\n| #Tasks | Adaptation Steps | Total Wall-clock Time (s) | Average Per-task Wall-clock Time (s) |\n| ------ | ---------------- | ------------------------- | ------------------------------------ |\n| 8      | 50               | 78                        | 9.8                                  |\n| 14     | 50               | 138                       | 9.9                                  |\n| 20     | 50               | 211                       | 10.55                                |\n\n### (ii) Ablation on TTA optimization steps\nWe performed an ablation by varying the number of adaptation steps. The table below shows the trade-off between accuracy and forgetting:\n\n**Table X.3: Ablation on number of adaptation steps for ViT-B/32 across 8, 14, and 20 tasks**\n\n| Steps | ACC (8-task) | BWT (8-task) | ACC (14-task) | BWT (14-task) | ACC (20-task) | BWT (20-task) |\n| ----- | ------------ | ------------ | ------------- | ------------- | ------------- | ------------- |\n| 5     | 60.9±1.4     | -0.1±0.2     | 62.4±1.7      | -0.3±0.1      | 60.2±1.5      | -0.1±0.2      |\n| 10    | 68.6±1.6     | -0.2±0.2     | 68.5±1.6      | -0.1±0.2      | 63.5±1.0      | -0.4±0.3      |\n| 20    | 78.4±0.6     | -0.2±0.1     | 75.5±1.3      | -0.4±0.1      | 71.0±1.2      | -0.4±0.4      |\n| 50    | 85.8±0.8     | -0.6±0.4     | 81.6±1.4      | -1.1±0.3      | 77.1±2.0      | -2.2±0.8      |\n**Findings**:\n-  Increasing the number of steps consistently improves accuracy across all task counts.\n-  While longer schedules introduce slightly more forgetting, the degradation is **minor** (e.g., BWT drop of only ~2% in 20-task setting).\n\nThese results confirm that MINGLE achieves strong performance even under tight compute budgets—for example, using only 20 steps, it already outperforms all baseline methods in Table 1. Longer adaptation improves accuracy with only minor forgetting\n\n### (iii) Discussion: Step-freezing and fast first-order updates\nWe thank the reviewer for the insightful suggestion. Both step-freezing and first-order adaptation are promising directions to improve TTA efficiency.\n-  **Step-freezing**: Our current method updates all router parameters across layers during adaptation. Step-freezing (e.g., freezing early layers once stabilized) may reduce compute. Preliminary results show early gates converge quickly (<10 steps). \n-  **Fast first-order updates** can benefit from first-order strategies such as Reptile, FOMAML, or linearized single-step updates, which avoid full optimization but enable quick convergence. These methods are especially attractive in low-latency or edge settings,\n\nWe see both directions as highly complementary to our framework. We will include this discussion in the final version to highlight efficient variants as a promising area for future work.\n# W3 –  Hyper-parameter transparenty\nWe thank the reviewer for the suggestion. As requested, we have added a consolidated table (see Table X.4) listing the key hyperparameter settings for all baseline methods and task configurations.\n\nWe also clarify that none of the baselines were tuned per task order. For our method, we emphasize that we used a **single fixed hyperparameter configuration** across **all experiments**, including:\n*  all backbones (ViT-B/32, B/16, L/14),\n*  all task counts (8, 14, 20), and\n*  all 10 task orders.\n\nThis highlights the robustness and generality of our method, and ensures that performance gains are not due to fine-grained tuning. \n\n**Table X.4: Hyperparameter settings for all baselines across different task configurations.**\n\n| Method            | #Tasks  | Scale Factor ($\\lambda$) | Top-k (%) | TALL mask threshold | Consensus mask threshold | LR   | Adapt Steps | LoRA Rank ($r$) | Null Dim ($k$) | EMA Decay ($\\beta$) | Relax Coeff ($\\gamma$) |\n| ----------------- | ------- | ------------------------ | --------- | ------------------- | ------------------------ | ---- | ----------- | --------------- | -------------- | ------------------- | ---------------------- |\n| Task Arithmetic   | 8       | 0.3                      | —         | —                   | —                        | —    | —           | —               | —              | —                   | —                      |\n|                   | 14/20   | 0.1                      | —         | —                   | —                        | —    | —           | —               | —              | —                   | —                      |\n| Ties-Merging      | 8       | 0.3                      | 20        | —                   | —                        | —    | —           | —               | —              | —                   | —                      |\n|                   | 14/20   | 0.1                      | 20        | —                   | —                        | —    | —           | —               | —              | —                   | —                      |\n| Consensus TA      | 8/14/20 | 0.1                      | —         | 0.2                 | 2                        | —    | —           | —               | —              | —                   | —                      |\n| LW. AdaMerging    | 8/14/20 | 0.3                      | —         | —                   | —                        | 1e-4 | 50          | —               | —              | —                   | —                      |\n| WEMOE             | 8/14/20 | 0.3                      | —         | —                   | —                        | 1e-4 | 50          | —               | —              | —                   | —                      |\n| WEMOE-LoRA        | 8/14/20 | 0.3                      | —         | —                   | —                        | 1e-4 | 50          | 64              | —              | —                   | —                      |\n| MINGLE- Static    | 8/14/20 | 0.3                      | —         | —                   | —                        | —    | —           | —               | —              | —                   | —                      |\n| **MINGLE (Ours)** | 8/14/20 | —                        | —         | —                   | —                        | 1e-4 | 50          | 64              | 3              | 0.99                | 1.0                    |\n\n# W4 – On the use of unlabeled adaptation samples\n**(i) How is the buffer obtained in practice?**\nWe thank the reviewer for raising this important question. In our experiments, we simulate a realistic scenario by randomly sampling **five unlabeled examples per class from the test split**—serving as a proxy for real-world deployment data.\n\nIn practical settings, such small unlabeled buffers are easily obtainable from a wide range of sources, including:\n*  incoming test-time data (e.g., recent user queries or model inputs),\n*  held-out validation inputs or small training subsets (if available),\n*  user-provided samples without privacy risks (e.g., few-shot examples),\n*  synthetically generated,\n*  or manually curated from public data.\n\nWe will include this discussion in the revision, making it practical for real-world deployment.\n\n**(ii) Zero-sample condition (“pure parameter-space merge”)**\n\nWe thank the reviewer for this suggestion. As shown in Table 4 (Row 1), we already include a zero-sample baseline where LoRA adapters are merged with fixed weights (e.g., 0.3) without any adaptation data. (refer to MINGLE-Static in Table S.2 for Reviewer chHs). \n\nSurprisingly, this simple strategy still outperforms classical baselines like Task Arithmetic and Ties-Merging. We attribute this to the low-rank, near-orthogonal structure of LoRA modules, which reduces interference and preserves task-specific features in disjoint subspaces.\n\nTo improve clarity, we will (i) note in the main text that Table 4’s first row reflects the zero-sample case, and (ii) add it to Fig. 5 to better visualize the trade-off between adaptation-free and adaptive merging across buffer sizes.\n\nWe sincerely thank the reviewer again, and hope our clarifications will support a higher evaluation."}, {"review_id": "QZtST5wYiO", "reviewer": "Reviewer_9ZjH", "summary": "This paper presents MINGLE, a framework that integrates test-time adaptation into continual model merging.  MINGLE inserts low-rank LoRA experts into a pretrained CLIP backbone and uses a Null-Space Constrained Gating mechanism, relaxed adaptively to combine experts based on incoming inputs.  At each task stage, a small seed buffer of unlabeled test samples guides light adaptation of the new task’s router network, mitigating interference with prior tasks.  Extensive experiments over multiple CLIP sizes, task orders, and distribution‐shifted benchmarks are presented.", "questions": "Overall, I find this paper is well-written and easy to follow. But I am extremely concerned about the usage of MoE in the merging paradigm. I hope the authors could address my concerns, and I am happy to raise the score if all my concerns are addressed.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The idea of adapting the MoE router network during test-time is interesting\n2. The paper is well-written and easy to follow\n3. The TTA is robust to the number of samples\n4. Comprehensive experiments \n\n\nWeakness:\n1. MAJOR CONCERN: My major concern with this paper is whether the MoE framework used in this paper counts as model merging. Cause essentially, the merged parameters are static, but in your moe framework, the model parameter can be thought of as dynamically adjusted based on the input. The usage of this MoE structure will be naturally stronger than traditional merging methods. This raises concerns about the fairness of comparisons to other merging methods in the experiments.\n2. Based on (1), in Table 3 (MTIL benchmark), MINGLE’s continual merging variant only marginally beats MoE-based baselines MoE-Adapter and DIKI , and underperforms RAIL. \n3. In  Fig3, except for Gate 6, most gate-task activation peaks remain ≤ 0.5, despite each LoRA adapter being trained specifically on its task (which should resemble Gate 6’s behavior).  This under-activation suggests either: (a) the gating network fails to learn strong specialization, or (b) the null-space constraints overly suppress new task signals.  Either way, it underactivates the correct experts for a given task.", "was_revised": true, "final_justification": "Thank the authors' rebuttal. I will raise my score.", "rebuttal": "# **W1– Clarifying MoE as a Form of Model Merging**\n\nWe thank the reviewer for this insightful concern. Your point about the potential advantages of MoE over static merging is valid and important.\n\nWe clarify that **MoE is increasingly adopted in recent model merging works**, such as **WEMOE** (ICML’24) and **TwinMerging** (NeurIPS’24), which use MoE-style routing to merge fine-tuned experts. These methods treat MoE as a scalable and valid alternative to static merging.\n\nWe fully acknowledge that **MoE introduces dynamic parameter selection**, which may appear stronger than static approaches at first glance. However, our method operates under **strict continual learning constraints** that make the router tuning significantly more challenging.\n\nTo ensure fairness and transparency, we have taken several steps:\n\n* **We add comparison with WEMOE** in Table 1. Results are shown in Table Y.1.\n* We include a **static MINGLE baseline** (Table 4, Row 1), where LoRA modules are combined with fixed coefficients, without any adaptation or routing. Results are shown in Table Y.1.\n* We will **clearly separate static and adaptive methods** in Table 1, and explicitly compare all baselines' assumptions regarding data access, activation storage, and test-time compute to ensure transparency.(see Table Y.2).\n\nIn summary, although our method uses a MoE-style mechanism, it remains fully aligned with the goals of model merging: combining multiple experts into a **single  model**. Crucially, we address **continual merging**, which is not covered in prior MoE-based merging works. We will clarify this distinction more explicitly in Sec. 2 and Sec. 4.2 of the main paper.\n\n**Table Y.1: More comparative results  (ACC%) for continual merging performance.**\n\n| Method              | ViT-B/32 (8)  | ViT-B/32 (14) | ViT-B/32 (20) | ViT-B/16 (8)  | ViT-B/16 (14) | ViT-B/16 (20) |\n| ------------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- |\n| LW. AdaMerging      | 52.9 ±3.5     | 60.1 ±1.7     | 57.5 ±0.7     | 59.8 ±3.2     | 64.8 ±1.6     | 61.4 ±0.8     |\n| WEMOE               | 4.9 ±0.6      | 8.2 ±0.3      | 10.4 ±0.2     | 4.0 ±0.7      | 8.3 ±0.4      | 10.1 ±0.5     |\n| WEMOE-LoRA          | 66.6 ±6.6     | 63.4 ±2.2     | 44.0 ±19.5    | 72.1 ±4.6     | 55.8 ±27.0    | 41.3 ±22.8    |\n| Consensus TA        | 69.0±1.1      | 64.0 ±0.9     | 45.6 ±1.6     | 73.3±0.3      | 69.0 ±1.1     | 50.0 ±2.4     |\n| MINGLE-Static       | 74.3±0.3      | 72.9±0.8      | 67.3±0.8      | 78.7 ±0.1     | 76.4 ±1.0     | 70.6 ±0.4     |\n| MINGLE (Ours)  | **85.8 ±0.8** | 81.6 ±1.4     | **77.1 ±2.0** | **88.3 ±0.6** | **84.9 ±0.8** | **81.9 ±0.9** |\n| MINGLE*  (Ours) | 85.0 ±0.5     | **81.7 ±1.0** | **77.1 ±1.3** | 87.0 ±0.6     | 84.7 ±1.0     | 81.6 ±1.3     |\n\n**Table Y.2: Comparison of baseline assumptions and requirements**\n\n| Method            | Save Activations | Extra Parameters (Storage) | Extra Parameters (Inference) | Test-time Compute |\n| ----------------- | ---------------- | -------------------------- | ---------------------------- | ----------------- |\n| Task Arithmetic   | No               | No                         | No                           | No                |\n| Ties-Merging      | No               | No                         | No                           | No                |\n| MAGMAX-Ind        | No               | No                         | No                           | No                |\n| OPCM              | No               | No                         | No                           | No                |\n| Consensus TA      | No               | Yes                        | No                           | No                |\n| LW. AdaMerging    | No               | No                         | No                           | Yes               |\n| WEMOE             | No               | Yes                        | No                           | Yes               |\n| MINGLE- Static    | No               | No                         | No                           | No                |\n| MINGLE (Ours) | No               | Yes                        | No                           | Yes               |\n\n# **W2 – On MTIL Results and Comparison to RAIL / MoE-based Baselines**\n\nWe thank the reviewer for the thoughtful comment.\n\nWe would like to clarify that **MINGLE was not specifically designed for the conventional continual learning setup used in the MTIL benchmark (Table 3)**. Instead, it was developed for the **continual model merging** scenario.\n\nNonetheless, we applied MINGLE directly to the MTIL benchmark in order to **demonstrate its generality and robustness** beyond its original design scope. We adapted it to this benchmark by merging models obtained via **Sequential Finetuning (Sequential FT)**. This yields **MINGLE-SEQ**, a direct application of our merging strategy on sequentially trained models.\n\nAs shown in Table Y.3, MINGLE-SEQ outperforming several CL baselines on early and mid-stage tasks (e.g., Aircraft, Caltech101, CIFAR100), but underperforms on the last two tasks (Cars, SUN397). Importantly, this drop is **not due to MINGLE itself**, but rather a consequence of the **declining quality of the sequential FT models** used as inputs.\n\nTo verify this, we examined the **Sequential FT (Immediate Eval)** baseline, which evaluates each task immediately after learning it. It exhibits the same degradation on the final tasks. We attribute this to the **loss of general pretraining knowledge** caused by continued finetuning, which results in **suboptimal task-specific performance on later tasks**, even before merging. Since MINGLE operates on these models, this limits its upper-bound performance in this setting.\n\nThis insight suggests that **MINGLE-SEQ could benefit from upstream continual learning techniques**. For instance, using regularization-based methods like EWC during sequential finetuning may better preserve pretraining knowledge and improve the quality of task models fed into MINGLE, potentially leading to stronger results.\n\n**Table Y.3: Comparison of last accuracy (%) with conventional CL approaches on MTIL benchmark.**\n\n| Method                         | Aircraft | Caltech101 | CIFAR100 | DTD  | EuroSAT | Flowers | Food101 | MNIST | Pets | Cars | SUN397 | Avg  |\n| ------------------------------ | -------- | ---------- | -------- | ---- | ------- | ------- | ------- | ----- | ---- | ---- | ------ | ---- |\n| Sequential FT (Immediate Eval) | 63.2     | 97.6       | 89.3     | 80.1 | 99.2    | 91.8    | 91.7    | 99.6  | 93.1 | 85.9 | 78.9   | 88.2 |\n| MoE-ADAPTER                    | 49.8     | 92.2       | 78.1     | 78.1 | 95.7    | 94.4    | 89.5    | 91.8  | 89.9 | 81.4 | 80.0   | 85.0 |\n| DIKI                           | 45.2     | 95.7       | 83.6     | 72.9 | 98.0    | 97.0    | 90.4    | 99.4  | 94.2 | 81.6 | 76.6   | 85.1 |\n| DUAL-RAIL                      | 52.5     | 96.5       | 84.0     | 80.1 | 96.9    | 97.6    | 90.5    | 99.0  | 95.5 | 85.2 | 86.8   | 86.8 |\n| **MINGLE-SEQ**                 | 58.7     | 97.5       | 87.2     | 79.7 | 97.3    | 87.2    | 90.1    | 99.6  | 93.0 | 80.4 | 73.3   | 85.8 |\n\n\n# **W3– On Under-activation and Gating Specialization**\n\n\nWe sincerely thank the reviewer for the insightful observation regarding the under-activation behavior in Fig. 3.\n\nTo investigate this, we provide a new comparison (Table Y.4) between MINGLE and individual fine-tuned models across all tasks. The results show that our method achieves **per-task performance close to that of individually fine-tuned models**, indicating that the gate indeed selects meaningful experts as intended. This empirical evidence helps to refute concern (a)—that the gate fails to specialize.\n\n**Table Y.4: Accuracy (%) of continual merging performance using CLIP ViT-B/16 on the 8-task Order-1 sequence.**\n\n| Model                 | EuroSAT | SVHN | MNIST | DTD  | RESISC45 | GTSRB | SUN397 | Cars | Average |\n| --------------------- | ------- | ---- | ----- | ---- | -------- | ----- | ------ | ---- | ------- |\n| Individual fine-tuned | 99.0    | 97.6 | 99.7  | 82.3 | 96.6     | 99.0  | 78.9   | 85.9 | 92.4    |\n| C. Task Arithmetic    | 84.5    | 88.9 | 98.1  | 54.0 | 75.5     | 82.0  | 65.9   | 68.3 | 77.1    |\n| C. Ties-Merging       | 97.0    | 93.9 | 98.5  | 46.1 | 49.1     | 40.4  | 47.4   | 48.9 | 65.2    |\n| OPCM                  | 70.7    | 74.0 | 83.8  | 93.5 | 88.9     | 84.9  | 97.9   | 60.8 | 81.8    |\n| LW. AdaMerging        | 54.1    | 51.7 | 61.0  | 46.0 | 65.2     | 45.0  | 65.1   | 75.9 | 58.0    |\n| WEMOE-LoRA            | 38.6    | 70.1 | 69.2  | 51.0 | 80.7     | 89.7  | 72.2   | 78.7 | 68.8    |\n| MINGLE (Ours)         | 97.3    | 95.9 | 99.1  | 78.2 | 91.1     | 95.9  | 74.0   | 81.0 | 89.1    |\n\nRegarding (b)—that null-space constraints may overly suppress task signals—we refer to the w/o Null-Space variant already shown in Fig. 3 (blue curve). While this variant does exhibit higher gate activations, most values **still do not approach** 1.0, suggesting that under-activation is not solely caused by the null-space constraint. In fact, this variant also shows worse performance in accuracy, indicating that the constraint is helpful overall, rather than overly suppressive.\n\n**As to why gate activations rarely reach 1.0**, we believe this may reflect the complementary nature of experts rather than a failure of specialization. In many cases, multiple LoRA experts may capture overlapping yet distinct subspaces of knowledge. As a result, softly combining several related experts—rather than activating only a single one—can lead to equal or even better performance. This behavior is especially reasonable in settings like TTCMM, where task boundaries may be fuzzy and information from adjacent tasks remains useful.\n\nWe sincerely thank the reviewer again, and hope our clarifications will support a higher evaluation."}, {"review_id": "0CB96cJvGT", "reviewer": "Reviewer_np8P", "summary": "This paper proposes mingle for test-time continual model merging, which alleviates the conflict between tasks through mixtures of lora experts and adaptive null-space constrained gating, bringing significant improvement compared to the existing continual model merging methods.", "questions": "+ Can you provide a comparative experiment on efficiency with baselines?\n+ Is there any forward transfer?", "limitations": "+ Extending their framework to support diverse initialization points or architectural variants remains an open direction for future work.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "+ Strength\n\n    + Through a lightweight seed dataset, the model merging effect is significantly improved with slightly increased computational overhead.\n    + Clearly illustrate the advantages of the moe structure over traditional continual model mergin.\n\n+ Weakness\n\n    +  Lack of comparative experiments with baselines on efficiency, storage overhead.\n    + The relationship between continual model merging and rehearsal-free continual learning needs to be discussed. Because both continual model merging and rehearsal-free continual learning do not have training data of previous tasks when learning task t, both require training data of the current task and need to have learned models of previous tasks. I do not understand why continual model merging brings about improved privacy. \n    + If I understand correctly, the advantage of continual model merging may be that it can train models for t tasks separately and solve any permutation and combination of these t tasks, which is the flexibility. The models for each task must be saved to achieve this flexibility. Given conditions above, if test-time adaptation is added to continual model merging, I have the following concerns: doing **independent** test-time adaptation for **the downstream task currently being tested** may achieve better results because there is no need to consider conflicts between tasks.", "was_revised": true, "final_justification": "I maintain my score for the following reasons:\n+ I agree that the proposed scenario test-time continual model merging is reasonable and meaningful to some extent.\n+ However, the assumption of the proposed scenario is too strong and the application scenario is limited. The proposed test-time continual model merging must assume a scenario like some pre-trained models are released by someone and then must be deleted. It is reasonable to delete data due to privacy, but it is kind of strange to delete a model to be merged.\n+ From my point of view, test-time adaptation is to use some current unlabeled test samples to adapt the model, thus the model can always be adapted if you need to test or use it. Thus **the authors do not adapt the model for task t-1 test samples after merging task t** is kind of strange as well, especially when the experts of previous tasks are preserved.", "rebuttal": "# **W1 – Comparison of Efficiency and Storage Overhead with Baselines**\n\n### **Table Z.1: Comparison of inference efficiency and parameter overhead on CLIP ViT-B/32 model after eight tasks merging**\n\n| Method          | Total Params (M) | Extra Storage Params (M) | Extra Inference Params (M) | Throughput (img/s) | ACC (%) |\n| --------------- | ---------------- | ------------------------ | -------------------------- | ------------------ | ------- |\n| Task Arithmetic | 87.5             | 0.0                      | 0.0                        | ~910               | 67.5    |\n| Ties-Merging    | 87.5             | 0.0                      | 0.0                        | ~910               | 49.0    |\n| MAGMAX-Ind      | 87.5             | 0.0                      | 0.0                        | ~910               | 70.7    |\n| OPCM            | 87.5             | 0.0                      | 0.0                        | ~910               | 75.5    |\n| Consensus TA    | 87.5             | 87.5                     | 0.0                        | ~910               | 69.0    |\n| LW. AdaMerging  | 87.5             | 0.0                      | 0.0                        | ~910               | 52.9    |\n| WEMOE           | 540.9            | 453.4                    | 0.07                       | ~858               | 4.9     |\n| WEMOE-LoRA      | 103.7            | 16.2                     | 0.07                       | ~848               | 66.6    |\n| MINGLE          | 173.1            | 85.6                     | 0.6                        | ~841               | 85.8    |\n| MINGLE*         | 113.7            | 26.2                     | 0.3                        | ~862               | 85.0    |\n\n\n\nWe sincerely thank the reviewer for raising this point—efficiency and overhead are indeed critical factors for real-world deployments, and we greatly appreciate the suggestion to expand this comparison.\n\nIn response, we have included a consolidated table (Table Z.1) comparing **inference speed** and **storage overhead** across all baselines, which demonstrates the practical efficiency of **MINGLE**. Despite employing MoE-style routing, MINGLE incurs **minimal additional inference overhead**, with only a small number of extra parameters used for the gating mechanism (i.e., the MoE routing gates).  The **inference throughput is comparable to static merging methods**, only slightly lower due to the additional MoE parameters, **while providing substantial accuracy improvements**.\n\n\n\n\n# **W2 – Discussion to Rehearsal-Free Continual Learning (RFCL)**\n\nWe thank the reviewer for the insightful question, and for prompting a more precise comparison between TTCMM and **rehearsal-free continual learning (RFCL)**—a widely studied paradigm.\n\nIndeed, TTCMM is inspired by RFCL and shares important constraints:\n\n* both **do not retain past training data**, and\n* both **avoid storing previous task models**.\n\nHowever, the **core difference lies in what is received at each stage**:\n\n* **RFCL** assumes access to the **training data** of the current task, and incrementally fine-tunes a single model over time.\n* **TTCMM**, by contrast, assumes access to an **already independently fine-tuned model** for each new task—thus focusing on **merging** existing expert models rather than learning each from scratch. Additionally, TTCMM requires a **small unlabeled test-time buffer** (e.g., 5 samples per class) to guide the merge.\n\n**On privacy benefits:**\nWhile both RFCL and TTCMM operate without historical data, TTCMM offers improved privacy guarantees for the following reasons:\n\n1. **TTCMM does not require access to full training sets.** Instead, it only uses a **tiny** buffer (e.g., 5 **unlabeled** samples per class) from the **test or calibration set**.\n2. These small unlabeled samples can be:\n*  incoming test-time data (e.g., recent user queries or model inputs),\n*  held-out validation inputs or small training subsets (if available),\n*  user-provided samples without privacy risks (e.g., few-shot examples),\n*  synthetically generated,\n*  or manually curated from public data.\n\nIn contrast, RFCL relies on **large-scale labeled** training data for each task, which raises significant privacy, storage, and legal concerns (e.g., medical images, user data, copyrighted materials). In comparison, small unlabeled samples are easier to obtain while maintaining privacy protection.\n# **W3 – On Flexibility and Test-Time Adaptation in TTCMM**\nWe sincerely thank the reviewer for their insightful reflection. Your interpretation of TTCMM as offering high flexibility through the ability to merge independently trained task-specific models in arbitrary orders is absolutely correct—and we deeply appreciate your articulation of this strength.\n\nYou also raise an important point: that *independently applying TTA to each downstream task* might yield better performance in some cases, as it avoids reconciling conflicts across tasks. We agree that this is indeed a powerful strategy—but only under the assumption that **all task-specific models are available at inference time**.\n\nHowever, we respectfully emphasize that this assumption does not hold in the **TTCMM setting**, which imposes strict continual learning constraints:\n\n-  **Previous expert models are not retained**—once merged, earlier models are discarded.\n-  The protocol is **strictly incremental**, with one expert model arriving at a time; merging must maintain knowledge across tasks *without* re-accessing past models or data.\n\nThus, the setting envisioned in your comment—while valid and practically useful—**requires simultaneous access to all expert models**, and is closer in nature to **multi-task model merging  with task-specific adaptation**, which lies **outside the scope of TTCMM**.\n\nWe fully agree that this contrast is important and worth clarifying more explicitly. In the revision, we will revise Sec. 1 and Sec. 3.2 to:\n\n* More clearly distinguish TTCMM from settings that assume full model access.\n* Emphasize that TTCMM’s flexibility stems from supporting *arbitrary task sequences*, **but only when models are provided incrementally**, not jointly.\n\n# **Q2 – On Forward Transfer**\n\nWe sincerely thank the reviewer for raising this insightful question. Forward transfer is indeed a critical capability in continual learning, measuring how well prior knowledge aids the learning of future tasks. Your suggestion meaningfully strengthens the completeness of our evaluation.\n\nTo quantify forward transfer (FWT), we adopt the standard definition:\n$$\n\\text{FWT} = \\frac{1}{T-1} \\sum_{t=2}^{T} \\left[ a_t(\\theta_t^{merged}) - \\bar{a}_t \\right]\n$$\n\nwhere $a_t(\\theta_t^{merged})$ denotes the test accuracy on task $t$ using the  model merged after task $t$, and $\\bar{a}_t$ is the test accuracy of task $t$ individual finetuned model. Positive FWT indicates beneficial transfer, while negative values suggest interference.\n\nWe report the following results for the 8-task continual merging setup with CLIP ViT-B/16:\n\n| Method            | ACC (%)     | BWT (%)     | FWT (%)     |\n| ----------------- | ----------- | ----------- | ----------- |\n| Task Arithmetic   | 77.1 ± 0.0  | -4.2 ± 1.0  | -13.4 ± 0.0 |\n| Ties-Merging      | 66.8 ±  3.7 | -5.5 ± 0.4  | -30.7 ± 9.9 |\n| OPCM              | 81.8  ± 0.3 | -4.8  ± 0.7 | -9.0 ± 0.4  |\n| **MINGLE (Ours)** | 88.3 ± 0.6  | -0.4 ± 0.1  | -3.8 ± 0.8  |\n\nThese results show that **MINGLE achieves both zero forgetting (BWT ≈ 0) and the highest forward transfer**, indicating that our adaptive gating and merging pipeline not only preserves knowledge but also improves representation utility for downstream tasks.\n\nWe will include these results and the FWT definition in the final version for completeness. \n\nWe sincerely thank the reviewer again, and hope our clarifications will support a higher evaluation."}]

Author rebuttal:

Meta-review: MINGLE is a test-time continual model merging method that uses LoRA experts with null-space gated routing. It adapts a router on a small unlabeled buffer to add tasks while likely preserving prior ones, and reports roughly 7–9% gains across CLIP backbones. Reviewers generally find the presentation clear with extensive experiments and some strong gains. Concerns seem to center on the realism of the TTCMM setting, fairness vs. static merging, hyperparameter load, and whether MoE truly counts as “merging,” with one reviewer still borderline-reject.

Recommendation: Accept. The method appears simple and effective. Results look consistent and strong, with seemingly fair comparisons to WEMOE and AdaMerging. Adaptation seems fast (~10 s per task) and gates are fixed after merging. The rebuttal appears to address most issues and suggests some non-vision generality, while the setting may still be debated under plausible privacy and storage constraints.
Decision: Accept (poster)