Title: MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging
Venue: NeurIPS 2025
Accepted: yes
Keywords: Continual model merging

Continual learning

Model merging
Abstract: Continual model merging integrates independently fine-tuned models sequentially without access to the original training data, offering a scalable and efficient solution for continual learning. However, existing methods face two critical challenges: parameter interference among tasks, which leads to catastrophic forgetting, and limited adaptability to evolving test distributions. To address these issues, we introduce the task of Test-Time Continual Model Merging (TTCMM), which leverages a small set of unlabeled test samples during inference to alleviate parameter conflicts and handle distribution shifts. We propose MINGLE, a novel framework for TTCMM. MINGLE employs a mixture-of-experts architecture with parameter-efficient, low-rank experts, which enhances adaptability to evolving test distributions while dynamically merging models to mitigate conflicts. To further reduce forgetting, we propose Null-Space Constrained Gating, which restricts gating updates to subspaces orthogonal to prior task representations, thereby suppressing activations on old tasks and preserving past knowledge. We further introduce an Adaptive Relaxation Strategy that adjusts constraint strength dynamically based on interference signals observed during test-time adaptation, striking a balance between stability and adaptability. Extensive experiments on standard continual merging benchmarks demonstrate that MINGLE achieves robust generalization, significantly reduces forgetting, and consistently surpasses previous state-of-the-art methods by 7–9% on average across diverse task orders. Our code is available at: https://github.com/zihuanqiu/MINGLE

MINGLE: Mixture of Null-Space Gated Low-Rank Experts for Test-Time Continual Model Merging

Abstract

Continual model merging integrates independently fine-tuned models sequentially without access to the original training data, offering a scalable and efficient solution for continual learning. However, existing methods face two critical challenges: parameter interference among tasks, which leads to catastrophic forgetting, and limited adaptability to evolving test distributions. To address these issues, we introduce the task of Test-Time Continual Model Merging (TTCMM), which leverages a small set of unlabeled test samples during inference to alleviate parameter conflicts and handle distribution shifts. We propose MINGLE, a novel framework for TTCMM. MINGLE employs a mixture-of-experts architecture with parameter-efficient, low-rank experts, which enhances adaptability to evolving test distributions while dynamically merging models to mitigate conflicts. To further reduce forgetting, we propose Null-Space Constrained Gating, which restricts gating updates to subspaces orthogonal to prior task representations, thereby suppressing activations on old tasks and preserving past knowledge. We further introduce an Adaptive Relaxation Strategy that adjusts constraint strength dynamically based on interference signals observed during test-time adaptation, striking a balance between stability and adaptability. Extensive experiments on standard continual merging benchmarks demonstrate that MINGLE achieves robust generalization, significantly reduces forgetting, and consistently surpasses previous state-of-the-art methods by 7-9% on average across diverse task orders. Our code is available at: https://github.com/zihuanqiu/MINGLE

1 Introduction

Continual learning aims to incrementally adapt machine learning models to new tasks without forgetting previously learned knowledge, addressing the critical challenge of catastrophic forgetting [43]. However, conventional continual learning approaches typically require continuous access to original training data, raising significant concerns about privacy and substantial computational overhead due to retraining efforts, thus limiting their applicability in dynamic, data-sensitive environments.

To address these limitations, recent works have explored an alternative paradigm known as continual model merging (CMM), which sequentially integrates independently fine-tuned models directly in parameter space, without revisiting any training data [38, 51, 70]. CMM typically operates under a "merge-to-transfer" paradigm: given a pretrained model θ 0 and independently fine-tuned models { θ t } T t =1 , a unified model is constructed sequentially by combining task-specific weight updates ∆ θ t = θ t -θ 0 via weighted averaging or projection-based strategies [24, 81, 69].

Despite its advantages in scalability, data privacy, and distributed training capabilities [44, 14, 57], existing CMM methods still encounter critical issues, notably severe parameter interference between tasks and limited adaptability to evolving test distributions. This parameter interference arises because, as fine-tuned models are incrementally merged, overlapping or conflicting parameter updates accumulate, resulting in severe forgetting of previously learned tasks. To mitigate this interference, recent methods introduce structural constraints such as orthogonal projection [70, 78], model linearization [38, 69], and pruning-based sparsification [86, 90]. However, their effectiveness diminishes as task count grows and interference becomes increasingly entangled. Moreover, models merged across tasks often fail to generalize effectively, particularly when facing unseen or shifting task conditions. These flaws result in severe forgetting of earlier tasks and substantial performance gaps compared to the upper bound achieved by individually fine-tuned models. As shown in Fig. 1, TA [24] suffers from large performance gaps and strong forgetting, reflected by low accuracy and negative backward transfer. OPCM [70] improves over TA via orthogonalized merging but still shows notable degradation.

To overcome these limitations, we propose a novel continual merging paradigmTest-Time Continual Model Merging (TTCMM) -which explicitly introduces the concept of test-time adaptation (TTA) [75, 63] into model merging. Unlike prior TTA-based multi-task merging methods [87, 68, 88], which assume simultaneous availability of models and test data from all tasks, TTCMM utilizes only a small set of unlabeled samples from the current task, making it uniquely suited for realistic continual scenarios where revisiting historical data is often infeasible.

EuroSAT

SVHN

MNIST

DTD

Overall BWT

50

60

70

80

90

100

Accuracy (%)

BWT

ACC

TA

OPCM

MINGLE (ours)

0%

-1%

-2%

-3%

-4%

In this paper, we propose MINGLE ( MI xture of N ull-Space G ated L ow-Rank E xperts), a method designed to continually merge independently fine-tuned models at test-time while preserving prior knowledge. MINGLE employs a mixture-of-experts architecture [27, 45] composed of lightweight LoRA-based [21] experts, enabling efficient and flexible test-time adaptation. To robustly prevent interference from previously learned tasks, we introduce a novel Null-Space Constrained Gating mechanism, restricting gating updates to task-orthogonal subspaces. Additionally, we propose an Adaptive Relaxation Strategy to dynamically modulate constraint strength based on test-time interference feedback during adaptation.

Extensive experiments on standard continual learning benchmarks show that MINGLE consistently outperforms previous state-of-the-art approaches by 7-9% on average, achieving robust generalization and strong resistance to catastrophic forgetting across diverse continual learning scenarios. Remarkably, these improvements are achieved entirely without any access to original training data, demonstrating the effectiveness of our TTCMM paradigm and the power of test-time adaptation in continual learning.

Our contributions are summarized as follows:

Weformalize test-time continual model merging (TTCMM), a novel task that leverages unlabeled test samples to merge independently fine-tuned models.

We propose MINGLE, a TTCMM framework with Adaptive Null-Space Constrained Gating to effectively balance stability and plasticity.

Extensive experiments show that MINGLE achieves state-of-the-art performance, consistently outperforming prior methods in accuracy, robustness and resistance to forgetting.

2 Related Work

Continual Learning. Continual learning (CL) seeks to mitigate catastrophic forgetting [43], where learning new tasks overwrites prior knowledge. Regularization-based methods constrain updates with importance weights [31, 92, 2, 30, 82], while distillation aligns outputs to preserve knowledge [20, 12, 60, 52]. Replay methods store exemplars or generate surrogates with prompts, prototypes, or generators [55, 39, 79, 61, 53], and dynamic architectures expand capacity via growth or ensembling [35, 98, 42]. Recent work leverages lightweight adapters or prompts in pre-trained models for efficient transfer [89, 23, 80]. Model merging offers an alternative route. Some methods remain close to

(a) Conventional Continual Learning

𝜃

!

𝜃

"

𝜃

!#$

%&'(&)

FT

Merging

𝜃

!

%&'(&)

Models arrive at stages

Seed Samples

(c) Test-Time Continual Model Merging

𝜃

!

𝜃

"

𝜃

!#$

%&'(&)

FT

Merging

𝜃

!

%&'(&)

Partial unlabeled

test data is used to

guide merging

Models arrive at stages

Training Data

Without Access

to Training Data

𝒟

*

𝒟

$

𝒟

!

𝜃

!

No Access to Previous

Training Data

Data arrive at stages

⋯

conventional CL by sequentially fine-tuning and merging models to reduce forgetting [41, 42, 16], typically requiring training data. In contrast, continual model merging [28, 38, 5, 51, 70] merges independently fine-tuned models without revisiting training data, enabling greater scalability and privacy.

Model Merging. Early work merged models via direct parameter averaging [72, 58], later refined by linear mode connectivity [13, 1]. Wortsman et al. [81] showed that weight averaging can also enhance robustness and out-of-distribution generalization. Task Arithmetic (TA) [24] views models as task vectors to be summed, but relies on weight disentanglement [49], often violated under standard fine-tuning, motivating structured training [28, 65]. Beyond averaging, interference-aware methods reweight or sparsify parameters [86, 90], or fuse models via distillation and clustering [88, 76]. LoRA-based tuning [21] introduces additional entanglement challenges, spurring gradient-free or retrieval-based strategies [22, 94, 95]. More recently, dynamic merging [68, 40] adapts parameters conditioned on inputs, achieving higher flexibility and performance, but remains limited to multi-task fusion and unexplored in continual settings.

Test-Time Adaptation. TTA adapts models at inference to mitigate distribution shift. Early approaches used self-supervised objectives [66], entropy minimization [75], or regularized updates [63]. Online TTA adapts continuously [25], while batch-wise variants ignore temporal structure [17]. To enhance stability, later work introduced confidence filtering [48], EMA [11], partial updates [91], test-time augmentation [93], and adaptive BatchNorm [56]. For vision-language models, TTA often employs prompts or adapters [59, 15, 36]. We draw on TTA to guide merging, aligning fused models with evolving test distributions.

Relation to Prior Work. Most related to our work are MoE-Adapter [89] and WEMOE [68], both built on MoE architectures [27, 29]. MoE-Adapter follows conventional continual learning, embedding expert modules that are jointly trained across tasks. In contrast, we adopt a model-merging paradigm, where experts are extracted from independently fine-tuned models and inserted without further training. WEMOE incorporates test-time adaptation but targets multitask learning, assuming simultaneous access to all models and data. By contrast, MINGLE is tailored for the more challenging continual merging setting.

3 MINGLE: Mixture of Null-Space Gated Low-Rank Experts

3.1 Preliminaries

Problem Setting. We study continual learning in a model merging setting, where a sequence of task-specific models { θ 1 , . . . , θ T } are independently fine-tuned from a shared pre-trained model θ 0 , each using a dataset D i = { ( x ( i ) j , y ( i ) j ) } with label space C i ⊂ Y . The goal is to construct a unified model θ merged T that generalizes across the combined label space C 1: T = ⋃ T i =1 C i .

Unlike conventional continual learning, we assume no access to training data during merging. All adaptation happens directly in parameter space. This paradigm is relevant in scenarios where only final fine-tuned models are retained, while original training data is discarded due to privacy, storage, or accessibility constraints.

To contextualize this, we compare with two related paradigms in Fig. 2:

Conventional Continual Learning. A single model θ is sequentially updated on D 1 , . . . , D T , discarding previous data. It requires direct training data access and extensive retraining.

Continual Model Merging. A sequence of models are merged incrementally in parameter space without access to training data and earlier models: θ merged t = Merge ( θ merged t -1 , θ t ) .

Test-Time Continual Model Merging. An extension of the above where a small unlabeled subset D seed t ⊂ D test t ( e.g. , 5 samples per class) is available at each stage to provide lightweight task-specific guidance. We refer to D seed t as the seed samples of task t .

Existing Continual Merging Strategies. Let θ 0 denote the parameters of a pre-trained model. The corresponding task vector is defined as ∆ θ t = θ t -θ 0 .

Continual Task Arithmetic (C. TA). A simple additive merge [24]: θ merged t = θ merged t -1 + λ ∆ θ t , where λ is a scalar. While training-free, it is sensitive to λ and prone to task interference.

Orthogonal Projection-based Continual Merging (OPCM). Tang et al. [70] propose projecting each ∆ θ t onto the orthogonal complement of previous directions: θ merged t = θ 0 + 1 λ t [ λ t -1 ∆ θ merged t -1 + P ( t -1) (∆ θ t ) ] , where P ( t -1) retains components orthogonal to previous updates. This reduces interference but ignores adaptation to task distributions.

To address these issues, we present MINGLE, which leverages D seed t to modulate the integration of θ t at test-time, enhancing alignment to test distribution and mitigating task interference.

3.2 Motivation and Theoretical Analysis

Most existing continual model merging methods combine fine-tuned models via static averaging, where each expert is assigned fixed coefficients, thereby enforcing the same mixing rule across the whole input space. Consequently, it cannot specialize to regions where one expert is clearly superior. In contrast, a Mixture-of-Experts (MoE) equips every input with a data-dependent gate g ( x ) = ( g 1 ( x ) , . . . , g T ( x )) that selects or re-weights experts on-the-fly. We give a formal comparison between static averaging and dynamic MoE under a noisy-routing scenario. 2

Theorem 1 (Dynamic MoE versus Static Averaging) . Let { ( D t , f t ) } T t =1 be T independent tasks with priors P ( t ) and per-task risks R t ( i ) . For any static mixture h static ( x ) = ∑ T i =1 α i f i ( x ) and any hard-routed MoE h MoE ( x ) = f i ⋆ ( x ) ( x ) with task-specific routing errors ε t :

̸

where R ideal = ∑ t P ( t ) R t ( t ) and R wrong ,t = 1 T -1 ∑ i = t R t ( i ) . Moreover,

(Perfect routing) If ε t = 0 for all t , then inf g R ( h MoE ) < inf α R ( h static ) whenever at least two tasks disagree on their best expert.

(Noisy routing) If ∑ t P ( t ) ε t ( R wrong ,t -R t ( t ) ) < R ∗ static -R ideal , where R ∗ static = inf α R ( h static ) ,

then the MoE still attains lower risk than any static mixture.

The theory above motivates a design that (i) keeps experts specialized and (ii) prevents interference between tasks. Our MINGLE framework achieves both goals by combining

Low-rank experts f t that capture task-specific variations with minimal parameters, and

Null-space constrained gating that projects gradient updates away from subspaces spanned by previously activated features, keeping ε t small without harming earlier experts.

3.3 Low-Rank Expert Mixture for Continual Model Merging

We adopt MoE framework for continual model merging, in which each task i is equipped with a low-rank expert f i and an associated input-dependent gating function g i . These components are injected into the linear layers of the backbone ( e.g. , CLIP visual encoder). The gate g i modulates expert activation based on the input features, allowing for fine-grained, localized task specialization.

Mixture of Low-Rank Expert. When a new task t arrives, a dedicated expert f t and its gate g t are appended to the model. The output of a given l -th layer 3 can be formulated as follows:

where only the gate g t is adaptable during testing, while all experts { f i } t i =1 and old gates { g i } t -1 i =1 remain frozen to preserve prior knowledge. To construct expert f t , we first project the task vector ∆ θ t onto the orthogonal complement of previously learned directions, following OPCM [70]:

̸

where u ( t -1) p and v ( t -1) q are the p -th and q -th singular vectors from the singular value decomposition (SVD) of previous experts ∑ t -1 i =1 f i ( X ) , and α denotes the effective rank of previous experts. This projection removes previously learned directions to mitigate interference. We then apply a rankr truncated SVD for P ( t -1) (∆ θ t ) to construct a low-rank expert [21]:

where ˜ U ∈ R d 1 × r , ˜ Σ ∈ R r × r , and ˜ V ∈ R d 2 × r , retaining the top r singular components. The resulting expert captures the principal directions while significantly reducing parameter overhead.

Each gating function is implemented as a linear projection:

where W ( g ) t ∈ R d × 1 and b ( g ) t ∈ R are learnable parameters. The gating function is adapted at test time using a small number of unlabeled test data.

Test-Time Adaptation. To encourage the merged model to retain task-specific behavior, we minimize the Kullback-Leibler divergence between its prediction and that of the corresponding individual fine-tuned model θ t . We define the adaptation objective as:

where p ( x ; θ ) denotes the predictive distribution

3.4 Adaptive Null-Space Constrained Gating for Interference Mitigation

When merging models continually, the primary challenge of gating is to integrate new experts without disturbing prior task predictions. Consider two experts f 1 , f 2 and their corresponding gates g 1 , g 2 . When evaluating on the first task domain X 1 , the interference from g 2 can be quantified as:

This measures the deviation introduced by g 2 on the domain where f 1 originally dominates. A desirable gating function should suppress g 2 ( X 1 ) , ensuring predictions on X 1 remain unaffected. However, as X 1 becomes inaccessible after adaptation, this error becomes unobservable and cannot be minimized directly, resulting in prediction drift and catastrophic forgetting.

Hard Null-Space Projection. After completing task t , we cache the l -th layer inputs in the seed buffer D seed t and estimate their covariance Cov ( l ) t ∈ R d × d . Applying truncated SVD yields the

topk dominant subspaces ˜ U ( l ) t ∈ R d × k . We then concatenate these with the subspaces from all previous tasks and orthonormalize: U ( l ) t = orthonorm [ U ( l ) t -1 | ˜ U ( l ) t ] ∈ R d × tk . The hard projector is P t = I -U ( l ) t U ( l ) ⊤ t ∈ R d × d . To suppress interference from tasks ≤ t -1 , the gating update for task t is W ( g,l ) t ← W ( g,l ) t -η ∇ L ( l ) t P ( l ) t -1 . However, this projection may also discard gradient components that are informative for task t whenever its feature support overlaps with span( U t -1 ) .

Adaptive Null-Space Relaxation. To restore plasticity, we replace the all-one eigenvalues of P t -1 with soft coefficients learned online.

(i) Interference statistics. For each column u ( l ) p of U ( l ) t -1 we measure instantaneous alignment:

We maintain per-direction interference scores S ( m,l ) ∈ R k (initialized to 0 ) by applying an exponential moving average at each iteration m :

which suppresses stochastic gradient noise while preserving the dominant interference directions.

(ii) Adaptive shrinkage. Each direction is attenuated by λ ( m,l ) = exp( -γ S ( m,l ) ) ( γ > 0 , λ ( m,l ) ∈ (0 , 1] ). Let Λ ( m,l ) t -1 = diag( λ ( m,l ) 1 , . . . , λ ( m,l ) ( t -1) · k ) . The relaxed projector becomes:

interpolating smoothly between no protection ( Λ = 0 ) and the hard null projector ( Λ = I ).

(iii) Update rule. We finally update:

Algorithm 1 MINGLE Procedure.

Relaxing the projector inevitably allows more residual interference than the hard null-space variant, yet empirically the increase is minor and is offset by markedly higher plasticity (Tab. 5), indicating a favorable stability-plasticity balance . The overall procedure is outlined in Algo. 1.

4 Experiments

We describe the experimental setup in Sec. 4.1, followed by the main results in Sec. 4.2 and further analysis and ablations in Sec. 4.3. Due to page limitations, detailed results are provided in the Appendix.

4.1 Experimental Setup

Datasets and Models. Following [70], we evaluate on image-classification tasks with CLIP-ViT backbones [54]. We consider 8, 14, and 20-task groups using ViT-B/32, ViT-B/16, and ViT-L/14 models, each fine-tuned on up to 20 downstream tasks, with checkpoints from FusionBench [67]. To assess order sensitivity, we repeat experiments over 10 random seeds (42-51). For comparison with conventional CL, we use the Multi-domain Task-Incremental Learning (MTIL) benchmark [97] with eleven vision tasks. Beyond vision, we evaluate on eight GLUE language tasks [74] with a Flan-T5-base backbone [6].

Implementation Details. We insert low-rank experts into the CLIP vision encoder. Two variants are used: a full setup modifying all attention and MLP layers, and a lightweight one on attn.qkv and mlp.fc1 . All experiments share a single set of global hyper-parameters across models and task orders. Each expert has rank r = 64 ; the null-space constraint uses k = 3 , γ = 1 , and β = 0 . 99 . Adaptation runs for 50 iterations with Adam (lr 1 e-4, batch size 16). For vision tasks we use 5 unlabeled samples per class, and for NLP tasks 100 in total, all without access to prior-task data.

Evaluation Metrics. We evaluate using average accuracy (ACC) and backward transfer (BWT) [37]. ACC is the mean accuracy of the final merged model across all tasks: ACC = 1 T ∑ T i =1 a i ( θ merged t ) , where a i ( · ) is accuracy on task i . BWT measures forgetting by comparing performance on earlier tasks before vs. after the final merge: BWT = 1 T -1 ∑ T -1 i =1 [ a i ( θ merged T ) -a i ( θ merged i ) ] .

4.2 Main Results

Overall Performance As shown in Tab. 1, MINGLE substantially outperforms previous continual merging methods on all CLIP backbones and task counts. It achieves the highest accuracy with backward forgetting kept near zero, demonstrating both strong forward learning and long-term stability. The lightweight variant performs on par with the full version, further underscoring the robustness of our approach. On NLP benchmarks (Tab. 2), MINGLE likewise attains the best overall accuracy and non-negative BWT, improving on multiple GLUE tasks while maintaining balanced performance across the suite. Together, these results across vision and language confirm that MINGLE

consistently delivers state-of-the-art accuracy while nearly eliminating forgetting under diverse continual scenarios.

Comparison with Conventional CL. Tab. 3 evaluates two CL paradigms on the MTIL benchmark: conventional CL, where each task model is fine-tuned from its immediate predecessor; and continual merging, which fine-tunes each task model independently before fusion, eliminating inter-model dependencies and enabling flexible task ordering and model reuse. Within the merging family, MINGLE sets a new state-of-the-art, and when integrated into a sequential fine-tuning pipeline, it matches the performance of SOTA CL methods. This demonstrates both its strength as a fusion strategy and its versatility across different training regimes.

Robustness to Test-Time Distribution Shifts. Following prior work [87, 68], we evaluate MINGLE on seven corruptions (motion blur, impulse noise, Gaussian noise, pixelate, spatter, contrast, JPEG) and report results in Tab. 4. It preserves high accuracy and near-zero or even positive BWT, outperforming all baselines, whereas direct application of SOTA TTA-based merging (WEMOE, AdaMerging) in a continual setting fails without tailored designs to continual setup.

4.3 Ablation Results and Analysis

Ablation Study. We explore the contribution of each component in Tab. 5. Row 1 shows a fixed-weight merging of low-rank experts as our baseline. In Row 2, adding TTA boosts ACC substantially but at the cost of worsening BWT with more tasks. Row 3 demonstrates that freezing earlier gates curbs forgetting while retaining ACC gains. Row 4 then applies null-space constraints, yielding further BWT improvements. Finally, Row 5 presents the full method with adaptive relaxation, which best harmonizes accuracy and long-term stability.

Ablation on TTA optimization steps. Tab. 6 reports accuracy and backward transfer of CLIP ViT-B/32 under different adaptation steps across 8, 14, and 20 tasks. Accuracy improves steadily with longer schedules, rising from 60-63% at 5 steps to 77-86% at 50 steps. Forgetting remains negligible, with BWT close to zero in all cases and only a minor drop of about 2% in the 20-task setting at 50 steps. Notably, as few as 20 steps are sufficient to surpass all baselines in Tab. 1, while 50 steps yield the best accuracy with only a modest increase in cost. These results confirm that MINGLE is both effective under tight compute budgets and scalable with additional adaptation.

Computation and Parameter Efficiency. Tab. 7 summarizes the efficiency analysis. In part (a), inserting experts into both attn and mlp layers yields the highest accuracy, 85.8%, but also the longest adaptation time of 78s and 83k trainable parameters. A lighter hybrid scheme qkv & fc1 with rank 64 reaches 85.0% accuracy with 36.9k parameters and 65s, offering a better trade-off. The rank sweep shows that raising the rank from 32 to 64 improves accuracy from 84.5% to 85.0%, while larger ranks bring little or even negative gain, e.g. , rank 768 drops to 83.7%. Part (b) reports wall-clock adaptation time as tasks increase: with 20 tasks the total is 211s, averaging about 10s per task. After adaptation the router remains fixed and inference is purely feedforward without TTA, enabling low-latency deployment across all tasks. Overall, MINGLE achieves a strong balance of accuracy, parameter efficiency, and scalability under diverse resource budgets.

Number of Seed Samples. The number of seed samples per class is crucial for TTA reliability and efficiency. As shown in Tab. 7 (c), using no samples reduces to the Static baseline, where LoRA modules are merged with fixed coefficients (0.3) without adaptation, yielding 70-79% accuracy. Introducing a single sample lifts accuracy to 81-88%, and adding more samples offers only minor gains. Variance across task orders decreases with more samples, making five samples per class a balanced trade-off between performance and efficiency.

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUB397

Cars

0.0

0.5

1.0

Gate Activation

Gate 1

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUB397

Cars

0.0

0.5

1.0

Gate Activation

Gate 2

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUB397

Cars

0.0

0.5

1.0

Gate Activation

Gate 3

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUB397

Cars

0.0

0.5

1.0

Gate Activation

Gate 4

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUB397

Cars

0.0

0.5

1.0

Gate Activation

Gate 5

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUB397

Cars

0.0

0.5

1.0

Gate Activation

Gate 6

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUB397

Cars

0.0

0.5

1.0

Gate Activation

Gate 7

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUB397

Cars

0.0

0.5

1.0

Gate Activation

Gate 8

w/o Null-Space

°

=

0

°

=

1

°

=

3

0

1

3

5

10

20

°

84.0

86.0

88.0

90.0

ACC (%)

-2.0

-1.5

-1.0

-0.5

0.0

BWT (%)

ACC

BWT

(a) Vary

γ

,

β

= 0

.

99

,

k

= 3

0.9

0.95

0.99

0.995

0.999

¯

86.0

87.0

88.0

89.0

ACC (%)

-1.5

-1.0

-0.5

0.0

BWT (%)

ACC

BWT

(b) Vary

β

,

γ

= 1

,

k

= 3

1

3

5

10

20

k

82.0

84.0

86.0

88.0

ACC (%)

-0.4

-0.3

-0.2

-0.1

0.0

BWT (%)

ACC

BWT

(c) Vary

k

,

γ

= 0

Visualization on Gate Activations. Fig. 3 shows that the null-space constraint suppresses gate responses on previously seen tasks, reducing forgetting, with smaller γ giving stronger attenuation. We also observe that gate activations remain below 1.0 across tasks, even in the w/o Null-Space variant (blue curve), showing that under-activation is not solely due to the constraint. Instead, it reflects the complementary nature of experts: multiple LoRA modules capture overlapping but distinct subspaces, and softly combining them often yields better performance, especially in settings like TTCMM where task boundaries are fuzzy.

Hyper-parameter of Gate. Westudy the effect of three key hyper-parameters: γ , β , and k . γ controls the strength of null-space suppression; smaller values lead to stronger attenuation of activations on prior tasks, reducing forgetting (Fig. 4a). As shown in Fig. 4b, β regulates the smoothness of the EMA used to accumulate interference signals. A moderate setting ( β = 0 . 99 ) balances responsiveness and stability. Smaller β amplifies noise sensitivity, while larger β slows detection of interference. k determines the number of principal directions retained per task. The mitigation of forgetting saturates at k = 3 (Fig. 4c), indicating that a small number of task-specific directions suffices.

5 Conclusions

In this work, we introduced the task of test-time continual model merging (TTCMM) and proposed MINGLE, a novel framework for TTCMM that integrates a mixture-of-experts architecture with adaptive null-space constrained gating. Extensive empirical evaluations show that MINGLE substantially improves generalization and mitigates catastrophic forgetting, consistently outperforming prior state-of-the-art approaches. These results establish TTCMM as a principled paradigm for addressing both task interference and distribution shift, and highlight the practical potential of MINGLE for scalable and efficient continual learning in real-world applications.

Acknowledgments This work was supported in part by National Science and Technology Major Project (2021ZD0112001), National Natural Science Foundation of China (No.62271119, 08120002, 62071086, U23A20286), the Key Research and Development Project of Hainan Province (Grant No. ZDYF2024(LALH)003), the Fundamental Research Funds for the Central University of China (DUT No. 82232031), the Natural Science Foundation of Sichuan Province under Grant 2025ZNSFSC0475.

We thank all reviewers for taking the time to review our paper and give valuable suggestions.

References

S. K. Ainsworth, J. Hayase, and S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836 , 2022.

R. Aljundi, F. Babiloni, M. Elhoseiny, M. Rohrbach, and T. Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European conference on computer vision (ECCV) , pages 139-154, 2018.

L. Bossard, M. Guillaumin, and L. Van Gool. Food-101-mining discriminative components with random forests. In European Conference on Computer Vision , pages 446-461. Springer, 2014.

G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE , 105(10):1865-1883, 2017.

R. Chitale, A. Vaidya, A. Kane, and A. S. Ghotkar. Task arithmetic with loRA for continual learning. In Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@NeurIPS 2023) , 2023.

H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research , 25(70):1-53, 2024.

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3606-3613, 2014.

T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha. Deep learning for classical japanese literature. In NeurIPS Workshop on Machine Learning for Creativity and Design , 2018.

A. Coates, A. Y. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

G. Cohen, S. Afshar, J. Tapson, and A. Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN) , pages 2921-2926. IEEE, 2017.

M. Döbler, R. A. Marsden, and B. Yang. Robust mean teacher for continual and gradual test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7704-7714, 2023.

A. Douillard, M. Cord, C. Ollion, T. Robert, and E. Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. European Conference on Computer Vision , 2020.

R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296 , 2021.

C. Fang, A. Dziedzic, L. Zhang, L. Oliva, A. Verma, F. Razak, N. Papernot, and B. Wang. Decentralised, collaborative, and privacy-preserving machine learning for multi-hospital data. EBioMedicine , 101, 2024.

C.-M. Feng, K. Yu, Y. Liu, S. Khan, and W. Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2704-2714, 2023.

T. Fukuda, H. Kera, and K. Kawamoto. Adapter merging with centroid prototype mapping for scalable class-incremental learning. arXiv preprint arXiv:2412.18219 , 2024.

J. Gao, J. Zhang, X. Liu, T. Darrell, E. Shelhamer, and D. Wang. Back to the source: Diffusiondriven adaptation to test-time corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11786-11796, 2023.

I. J. Goodfellow, D. Erhan, P.-L. Carrier, A. Courville, M. Mirza, B. Hamner, W. Cukierski, Y. Tang, D. Thaler, D.-H. Lee, et al. Challenges in representation learning: A report on three machine learning contests. In International Conference on Neural Information Processing , pages 117-124. Springer, 2013.

P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 12(7):2217-2226, 2019.

S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin. Learning a unified classifier incrementally via rebalancing. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 831-839, 2019.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. ICLR , 1(2):3, 2022.

C. Huang, Q. Liu, B. Y. Lin, T. Pang, C. Du, and M. Lin. Lorahub: Efficient cross-task generalization via dynamic loRA composition. In First Conference on Language Modeling , 2024.

L. Huang, X. Cao, H. Lu, and X. Liu. Class-incremental learning with clip: Adaptive representation adjustment and parameter fusion. In European Conference on Computer Vision , pages 214-231. Springer, 2024.

G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations , 2023.

Y. Iwasawa and Y. Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems , 34:2427-2440, 2021.

P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018 , pages 876-885. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.

R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural computation , 3(1):79-87, 1991.

X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations , 2023.

M. I. Jordan and R. A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation , 6(2):181-214, 1994.

S. Jung, H. Ahn, S. Cha, and T. Moon. Continual learning with node-importance based adaptive group sparse regularization. Advances in neural information processing systems , 33:3647-3658, 2020.

J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences , 114:3521 - 3526, 2016.

J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops , pages 554-561. IEEE, 2013.

A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

S.-W. Lee, J.-H. Kim, J. Jun, J.-W. Ha, and B.-T. Zhang. Overcoming catastrophic forgetting by incremental moment matching. Advances in neural information processing systems , 30, 2017.

Y. Lee, D. Kim, J. Kang, J. Bang, H. Song, and J.-G. Lee. RA-TTA: Retrieval-augmented test-time adaptation for vision-language models. In The Thirteenth International Conference on Learning Representations , 2025.

S. Lin, L. Yang, D. Fan, and J. Zhang. Beyond not-forgetting: Continual learning with backward knowledge transfer. Advances in Neural Information Processing Systems , 35:16165-16177, 2022.

T. Y. Liu and S. Soatto. Tangent model composition for ensembling and continual fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1867618686, 2023.

Y. Liu, B. Schiele, and Q. Sun. Rmm: Reinforced memory management for class-incremental learning. Advances in Neural Information Processing Systems , 34:3478-3490, 2021.

Z. Lu, C. Fan, W. Wei, X. Qu, D. Chen, and Y. Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. Advances in Neural Information Processing Systems , 37: 78905-78935, 2024.

D. Marczak, B. Twardowski, T. Trzci´ nski, and S. Cygert. Magmax: Leveraging model merging for seamless continual learning. In European Conference on Computer Vision (ECCV) , 2024.

I. E. Marouf, S. Roy, E. Tartaglione, and S. Lathuilière. Weighted ensemble models are strong continual learners, 2024.

M. McCloskey and N. J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation , volume 24, pages 109-165. Elsevier, 1989.

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

S. Mu and S. Lin. A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07137 , 2025.

Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning , 2011.

M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing , pages 722-729. IEEE, 2008.

S. Niu, J. Wu, Y. Zhang, Y. Chen, S. Zheng, P. Zhao, and M. Tan. Efficient test-time model adaptation without forgetting. In International conference on machine learning , pages 1688816905. PMLR, 2022.

G. Ortiz-Jimenez, A. Favero, and P. Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems , 36:6672766754, 2023.

O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. 2012 IEEE Conference on Computer Vision and Pattern Recognition , pages 3498-3505, 2012.

A. Porrello, L. Bonicelli, P. Buzzega, M. Millunzi, S. Calderara, and R. Cucchiara. A secondorder perspective on model compositionality and incremental learning. In The Thirteenth International Conference on Learning Representations , 2025.

Z. Qiu, L. Xu, Z. Wang, Q. Wu, F. Meng, and H. Li. Ism-net: Mining incremental semantics for class incremental learning. Neurocomputing , 523:130-143, 2023.

Z. Qiu, Y. Xu, F. Meng, H. Li, L. Xu, and Q. Wu. Dual-consistency model inversion for non-exemplar class incremental learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 24025-24035, 2024.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PmLR, 2021.

S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5533-5542, 2016.

S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances in neural information processing systems , 33:11539-11551, 2020.

Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems , 36:38154-38180, 2023.

K. Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques , pages 245-254, 1985.

M. Shu, W. Nie, D.-A. Huang, Z. Yu, T. Goldstein, A. Anandkumar, and C. Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems , 35:14274-14289, 2022.

C. Simon, P. Koniusz, and M. Harandi. On learning the geodesic path for incremental learning. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , pages 1591-1600, 2021.

J. S. Smith, L. Karlinsky, V . Gutta, P. Cascante-Bonilla, D. Kim, A. Arbelle, R. Panda, R. Feris, and Z. Kira. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11909-11919, 2023.

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing , pages 1631-1642, 2013.

J. Song, J. Lee, I. S. Kweon, and S. Choi. Ecotta: Memory-efficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11920-11929, 2023.

J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks , 32:323-332, 2012.

G. Stoica, P. Ramesh, B. Ecsedi, L. Choshen, and J. Hoffman. Model merging with SVD to tie the knots. In The Thirteenth International Conference on Learning Representations , 2025.

Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with selfsupervision for generalization under distribution shifts. In International conference on machine learning , pages 9229-9248. PMLR, 2020.

A. Tang, L. Shen, Y. Luo, H. Hu, B. Du, and D. Tao. FusionBench: A Comprehensive Benchmark of Deep Model Fusion, June 2024.

A. Tang, L. Shen, Y. Luo, N. Yin, L. Zhang, and D. Tao. Merging multi-task models via weight-ensembling mixture of experts. In International Conference on Machine Learning , pages 47778-47799. PMLR, 2024.

A. Tang, L. Shen, Y. Luo, Y. Zhan, H. Hu, B. Du, Y. Chen, and D. Tao. Parameter-efficient multi-task model fusion with partial linearization. In The Twelfth International Conference on Learning Representations , 2024.

A. Tang, E. Yang, L. Shen, Y. Luo, H. Hu, B. Du, and D. Tao. Merging models on the fly without retraining: A sequential approach to scalable continual model merging. arXiv preprint arXiv:2501.09522 , 2025.

L. Tang, Z. Tian, K. Li, C. He, H. Zhou, H. Zhao, X. Li, and J. Jia. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models. In European Conference on Computer Vision , pages 346-365. Springer, 2024.

J. Utans. Weight averaging for neural networks and local resampling schemes. In Proc. AAAI-96 Workshop on Integrating Multiple Learned Models. AAAI Press , pages 133-138. Citeseer, 1996.

B. S. Veeling, J. Linmans, J. Winkens, T. S. Cohen, and M. Welling. Rotation equivariant cnns for digital pathology. arXiv preprint arXiv:1806.03962 , 2018.

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019 , 2019.

D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726 , 2020.

H. Wang, B. Ping, S. Wang, X. Han, Y. Chen, Z. Liu, and M. Sun. Lora-flow: Dynamic lora fusion for large language models in generative tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 12871-12882, 2024.

K. Wang, N. Dimitriadis, G. Ortiz-Jimenez, F. Fleuret, and P. Frossard. Localizing task information for improved model merging and compression. In Forty-first International Conference on Machine Learning .

X. Wang, T. Chen, Q. Ge, H. Xia, R. Bao, R. Zheng, Q. Zhang, T. Gui, and X. Huang. Orthogonal subspace learning for language model continual learning. In The 2023 Conference on Empirical Methods in Natural Language Processing , 2023.

Z. Wang, Z. Zhang, S. Ebrahimi, R. Sun, H. Zhang, C.-Y. Lee, X. Ren, G. Su, V. Perot, J. Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision , pages 631-648. Springer, 2022.

Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, and T. Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 139-149, 2022.

M. Wortsman, G. Ilharco, J. W. Kim, M. Li, S. Kornblith, R. Roelofs, R. G. Lopes, H. Hajishirzi, A. Farhadi, H. Namkoong, et al. Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7959-7971, 2022.

Y. Wu, L.-K. Huang, R. Wang, D. Meng, and Y. Wei. Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction. In The Twelfth International Conference on Learning Representations , 2024.

H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 , 2017.

J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE Conference on Computer Vision and Pattern Recognition , pages 3485-3492. IEEE, 2010.

Y. Xu, Y. Chen, J. Nie, Y. Wang, H. Zhuang, and M. Okumura. Advancing cross-domain discriminability in continual learning of vision-language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems , 36: 7093-7115, 2023.

E. Yang, Z. Wang, L. Shen, S. Liu, G. Guo, X. Wang, and D. Tao. Adamerging: Adaptive model merging for multi-task learning. In The Twelfth International Conference on Learning Representations .

E. Yang, L. Shen, Z. Wang, G. Guo, X. Chen, X. Wang, and D. Tao. Representation surgery for multi-task model merging. Forty-first International Conference on Machine Learning , 2024.

J. Yu, Y. Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y. He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23219-23230, 2024.

L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning , 2024.

L. Yuan, B. Xie, and S. Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15922-15932, 2023.

F. Zenke, B. Poole, and S. Ganguli. Continual learning through synaptic intelligence. International conference on machine learning , pages 3987-3995, 2017.

M. Zhang, S. Levine, and C. Finn. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems , 35:38629-38642, 2022.

Z. Zhao, L. Gan, G. Wang, W. Zhou, H. Yang, K. Kuang, and F. Wu. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild. In Findings of the Association for Computational Linguistics ACL 2024 , pages 4447-4462, 2024.

Z. Zhao, T. Shen, D. Zhu, Z. Li, J. Su, X. Wang, and F. Wu. Merging loRAs like playing LEGO: Pushing the modularity of loRA to extremes through rank-wise clustering. In The Thirteenth International Conference on Learning Representations , 2025.

M. Zheng, Y. Tang, Z. Hao, K. Han, Y. Wang, and C. Xu. Adapt without forgetting: Distill proximity from dual teachers in vision-language models. In European Conference on Computer Vision , pages 109-125. Springer, 2024.

Z. Zheng, M. Ma, K. Wang, Z. Qin, X. Yue, and Y. You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF international conference on computer vision , pages 19125-19136, 2023.

D.-W. Zhou, H.-L. Sun, H.-J. Ye, and D.-C. Zhan. Expandable subspace ensemble for pretrained model-based class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23554-23564, 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the main contributions, which are consistently supported by both theoretical insights and empirical evaluations presented throughout the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the proposed approach are explicitly discussed in Appendix. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical claims are accompanied by clear assumptions and complete proofs in Appendix, with theorem formally stated and rigorously derived.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Detailed descriptions of datasets, training protocols, and evaluation metrics are provided in Section 4.1 and Appendix, enabling faithful reproduction of the main results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification:The code and data are not provided at submission time, but the authors state they will release them upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper includes detailed descriptions of datasets, data splits, training procedures, hyperparameters, and evaluation protocols in both the main text and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper reports mean and standard deviation across multiple runs with different random seeds and clearly states the sources of variability, ensuring the statistical reliability of the results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper specifies the type of GPUs used, training time per experiment, and overall compute requirements, providing sufficient details to assess reproducibility and resource demands.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research adheres to the NeurIPS Code of Ethics, with no identified ethical concerns related to data usage, human subjects, or societal impact.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper includes a Broader Impacts section stating that the work aims to advance machine learning and does not raise specific societal concerns requiring detailed discussion.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not introduce models or datasets with a high risk of misuse, so safeguards are not applicable.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All datasets and models used in the paper are properly cited, and their licenses and terms of use are respected and included where applicable.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release any new datasets, models, or code assets, so this question is not applicable.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing or research with human subjects, so this question is not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects, so IRB approval is not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core methods in this research do not involve LLMs in any important, original, or non-standard way, so this question is not applicable.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

A Theoretical Risk Comparison: Dynamic MoE vs. Static Averaging

Problem Setup and Definitions Consider T independent tasks, each associated with a data distribution D t for t = 1 , . . . , T . For each task t , a pre-trained expert model f t ( x ) outputs a probability distribution over classes, trained specifically on D t . The overall data distribution D is a mixture of these tasks, where an example ( x, y ) is drawn from task t with prior probability P ( t ) , and then ( x, y ) ∼ D t . The expected risk of a predictive model h ( x ) is defined as:

where ℓ ( h ( x ) , y ) is a loss function ( e.g. , cross-entropy or 0-1 loss).

We compare two methods to combine the experts into a final prediction h ( x ) :

Static Averaging : Defined as h static ( x ) = ∑ T i =1 α i f i ( x ) , where α = ( α 1 , . . . , α T ) is a fixed weight vector independent of x , typically with α i ≥ 0 and ∑ i α i = 1 for probability outputs.

Dynamic Mixture-of-Experts (MoE) : Defined as h MoE ( x ) = f i ∗ ( x ) ( x ) , where i ∗ ( x ) = arg max i g i ( x ) and g ( x ) = ( g 1 ( x ) , . . . , g T ( x )) is a gating function that selects one expert per input (hard routing). The gating is subject to routing noise, modeled below.

̸

Routing Noise Model For each input x drawn from D t , the true task is t , and the ideal expert is f t . The gating selects the correct expert i ∗ ( x ) = t with probability 1 -ε t , and an incorrect expert i ∗ ( x ) = t with probability ε t = P ( i ∗ ( x ) = t | x ∼ D t ) , the task-specific routing error rate. On error, the gating selects a random expert from { 1 , . . . , T } \ { t } uniformly. Define:

̸

R t ( i ) = E ( x,y ) ∼ D t [ ℓ ( f i ( x ) , y )] , the risk of expert i on task t .

R ideal = ∑ T t =1 P ( t ) R t ( t ) , the risk with perfect routing.

̸

R wrong ,t = 1 T -1 ∑ i = t R t ( i ) , the average risk of incorrect experts on task t .

ε = ∑ T t =1 P ( t ) ε t , the overall routing error rate.

Theorem A.1 (Dynamic MoE versus Static Averaging) . Let { ( D t , f t ) } T t =1 be T independent tasks with priors P ( t ) and per-task risks R t ( i ) . For any static mixture h static ( x ) = ∑ T i =1 α i f i ( x ) and any hard-routed MoE h MoE ( x ) = f i ⋆ ( x ) ( x ) with task-specific routing errors ε t :

̸

where R ideal = ∑ t P ( t ) R t ( t ) and R wrong ,t = 1 T -1 ∑ i = t R t ( i ) . Moreover,

(Perfect routing) If ε t = 0 for all t , then inf g R ( h MoE ) < inf α R ( h static ) whenever at least two tasks disagree on their best expert.

(Noisy routing) If ∑ t P ( t ) ε t ( R wrong ,t -R t ( t ) ) < R ∗ static -R ideal , where R ∗ static = inf α R ( h static ) , then the MoE still attains lower risk than any static mixture.

Proof. The proof proceeds in three parts: (1) deriving the MoE risk with routing noise, (2) proving the optimal gating case, and (3) establishing the condition for MoE superiority under routing noise.

Step 1: MoE Risk with Routing Noise

The MoE prediction is h MoE ( x ) = f i ∗ ( x ) ( x ) , where i ∗ ( x ) = arg max i g i ( x ) . The expected risk is:

For task t , condition on routing correctness:

Correct routing ( i ∗ ( x ) = t ): Probability 1 -ε t , risk R t ( t ) .

̸

Incorrect routing ( i ∗ ( x ) = t ): Probability ε t , selects a random expert from { 1 , . . . , T } \ { t } , with average risk R wrong ,t = 1 T -1 ∑ i = t R t ( i ) .

The expected risk on task t is:

Thus, the total risk is:

Rewrite:

where δ t = R wrong ,t -R t ( t ) > 0 is the risk increase due to misrouting on task t .

Step 2: Optimal Gating (No Routing Noise)

Assume an oracle gating function with ε t = 0 for all t , so i ∗ ( x ) = t for all x ∼ D t . Then:

̸

Define hypothesis classes:

Any static model h static ( x ) = ∑ T i =1 α i f i ( x ) can be approximated by an MoE with g ( x ) assigning constant weights, so H static ⊆ H MoE. Thus:

̸

Under task heterogeneity ( R t ( t ) < R t ( s ) and R s ( s ) < R s ( t ) for some t = s ), the ideal MoE routes each x ∼ D t to f t , achieving:

For static averaging:

Since ℓ is convex ( e.g. , cross-entropy), Jensen's inequality implies:

̸

with strict inequality unless α t = 1 and α i = 0 for i = t , which cannot hold for all tasks simultaneously under heterogeneity. Thus:

Step 3: MoE Superiority with Routing Noise

Let γ = R ∗ static -R ideal > 0 under task heterogeneity. The MoE outperforms the static model if:

Since δ t = R wrong ,t -R t ( t ) , the condition is:

If this holds, the MoE's risk, despite routing noise, remains below the best static risk.

Conclusion : The MoE risk is R ideal + ∑ T t =1 P ( t ) ε t ( R wrong ,t -R t ( t )) , and it outperforms static averaging when routing noise is sufficiently small relative to the static model's suboptimality. The optimal gating case confirms R ∗ MoE ≤ R ∗ static , with strict inequality under task heterogeneity.

B Additional Descriptions

B.1 Details of Dataset and Task Settings

Dataset Details Following prior works [70], we evaluate continual model merging on twenty publicly available image classification datasets, including SUN397 [84], Stanford Cars [32], RESISC45 [4], EuroSAT [19], SVHN [46], GTSRB [64], MNIST [34], DTD [7], Flowers102 [47], PCAM [73], FER2013 [18], Oxford-IIIT Pet [50], STL-10 [9], CIFAR-100 and CIFAR-10 [33], Food-101 [3], Fashion-MNIST [83], EMNIST [10], KMNIST [8], and Rendered SST-2 [62].

Substitute:

Thus:

Task Grouping We group the 20 datasets into three progressive task sets and evaluate the merged models using average accuracy (ACC) and backward transfer (BWT) metrics. For each task group, we perform 10 experiments using different task sequences (listed in Tab. 9), and report both the mean and standard deviation of the results to ensure robustness and consistency.

8-task group : (1) SUN397, (2) Stanford Cars, (3) RESISC45, (4) EuroSAT, (5) SVHN, (6) GTSRB, (7) MNIST, (8) DTD.

14-task group : (1) SUN397, (2) Stanford Cars, (3) RESISC45, (4) EuroSAT, (5) SVHN, (6) GTSRB, (7) MNIST, (8) DTD, (9) Flowers102, (10) PCAM, (11) FER2013, (12) OxfordIIITPet, (13) STL10, (14) CIFAR100.

20-task group : (1) SUN397, (2) Stanford Cars, (3) RESISC45, (4) EuroSAT, (5) SVHN, (6) GTSRB, (7) MNIST, (8) DTD, (9) Flowers102, (10) PCAM, (11) FER2013, (12) OxfordIIITPet, (13) STL10, (14) CIFAR100, (15) CIFAR10, (16) Food101, (17) FashionMNIST, (18) EMNIST, (19) KMNIST, (20) RenderedSST2.

B.2 Details of Downstream Models

In this section, we present the evaluation setup for pre-trained and fine-tuned models. As shown in Tab. 10, we evaluate the zero-shot accuracy of the original CLIP-ViT models and the performance of fine-tuned models on the test sets of various downstream tasks. The fine-tuned checkpoints are obtained directly from Hugging Face ( https://huggingface.co/tanganke ), where each model has been fine-tuned on task-specific training data using a standard protocol. The visual encoder is updated during fine-tuning, while the classification head is fixed and initialized from the pre-trained text encoder. The fine-tuning setup follows a standard configuration: cross-entropy loss, Adam optimizer, cosine annealing learning rate schedule with a peak learning rate of 1e-5, batch size 128, and 4000 training steps.

B.3 Details of Baselines

Our experiments involve the following comparison methods and our method:

Stochastic Weight Averaging (SWA). A simple model averaging technique to stabilize optimization and improve generalization [26]. At each step t , the model parameters are averaged across previous checkpoint: θ SWA t = 1 t [ θ SWA t -1 ( t -1) + θ SWA t ] . This approach can be interpreted

as a form of uniform model ensembling. While conceptually straightforward, SWA treats all checkpoints equally and does not account for inter-task conflicts.

Continual Task Arithmetic (C. TA). A training-free merging strategy that linearly combines task-specific fine-tuned models with a shared pre-trained model [24]. It computes the merged parameters as θ merged t = θ merged t -1 + λ ( θ t -θ 0 ) , where λ is a scaling factor. TA is computationally efficient and easy to apply, but sensitive to λ and prone to destructive interference when merging dissimilar tasks.

Continual Ties-Merging (C. Ties). An extension of Task Arithmetic that reduces parameterlevel redundancy and sign conflicts during model merging [86]. For task t , the difference vector ∆ θ t = θ t -θ 0 is trimmed and sign-normalized to obtain ∆ θ Ties t = Ties ( ∆ θ Ties t -1 , ∆ θ t ) , and the merged model is given by θ merged t = θ merged t -1 + λ ∆ θ Ties t .

Orthogonal Projection-based Continual Merging (OPCM). A projection-based scheme to mitigate task interference by enforcing orthogonality between parameter updates [70]. Specifically, each ∆ θ t is projected onto the orthogonal complement of the subspace spanned by previous updates: θ merged t = θ 0 + 1 λ t [ λ t -1 ∆ θ merged t -1 + P ( t -1) (∆ θ t ) ] , where P ( t -1) denotes the orthogonal projection.

Maximum Magnitude Selection (MagMax). An extension of Task Arithmetic that, for each parameter dimension, selects the update with the larger absolute value: ∆ θ MagMax t = MagMax ( ∆ θ MagMax t -1 , ∆ θ t ) , and the merged model is given by θ merged t = θ merged t -1 + λ ∆ θ MagMax t .

B.4 Details of Baseline Hyper-parameters

Tab. 11 summarizes the hyper-parameters for all baseline methods under different task configurations (8, 14, 20 tasks). Top-k denotes the pruning ratio, TALL the TALL mask threshold, and Cons. the consensus mask threshold. The column LR is the learning rate, while Steps indicates the number of adaptation steps. r represents the LoRA rank, and the last column jointly reports the null dimension ( k ), EMA decay ( β ), and relaxation coefficient ( γ ).

B.5 Comparison of Assumptions and Requirements

Tab. 12 summarizes the assumptions and resource requirements of all baseline methods. We report whether each method requires storing intermediate activations, introduces additional parameters (for storage or inference), and incurs extra test-time computation. Our method only maintains a fixed-size covariance matrix instead of full activations, leading to constant memory regardless of test set size. Although LoRA experts are stored, the router merges them into a single model per input, so the effective inference cost matches that of a standard individual model.

C Additional Results

In this section, we provide additional experimental results to support the findings reported in the main paper. Specifically, we include: (1) detailed overall performance results (C.1); (2) accuracy trends across sequential tasks (C.2); (3) detailed results under distribution shifts (C.3); and (4) extended visualizations of gate activations and hyper-parameter effects (C.6).

C.1 Detailed Overall Performance Results

Tab. 13 expands on the average results in Tab. 1 by reporting per-task average accuracy after continually merging 20 tasks. We compare six methods, SWA, Task Arithmetic, Ties-Merging, MagMax-IND, OPCM, and our proposed MINGLE across three CLIP-ViT backbones (B/32, B/16, L/14). MINGLE achieves the highest accuracy on most tasks. These fine-grained results reinforce the main paper's findings, highlighting MINGLE's ability to improve performance on continual model merging.

C.2 Accuracy Trends Across Sequential Tasks

Fig. 5 provides a detailed view of accuracy throughout the continual merging process across different settings, showing both the performance on the current task and on previously encountered ones. The progressive accuracy drop across columns illustrates the degree of forgetting over time. Notably, MINGLE consistently alleviates this degradation, demonstrating markedly reduced forgetting across the full task sequence. Fig. 6 further compares the average accuracy curves of MINGLE and baseline methods on previously seen tasks after each new model is merged, using the CLIP ViT-B/16 backbone. Results are averaged over 10 random task orderings. MINGLE consistently achieves the highest performance throughout the merging process, with its accuracy curve clearly dominating those of competing methods. Moreover, the narrower standard deviation bands indicate that MINGLE is more robust to the task orders.

C.3 Detailed Results Under Distribution Shifts

Tab. 14 expands on Tab. 4 by reporting per-dataset accuracy under both clean test conditions and seven common corruption types: motion blur, impulse noise, Gaussian noise, pixelation, spatter, contrast shift, and JPEG compression. We evaluate six merging methods, across four downstream tasks: Cars, EuroSAT, RESISC45, and GTSRB. This detailed breakdown complements the average results in the main paper, providing a more comprehensive assessment of robustness under test-time distribution shifts.

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUN397

Cars

Test Task

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUN397

Cars

New Task

0.98

0.98

0.96

0.98

0.96

0.99

0.98

0.96

0.99

0.76

0.98

0.96

0.99

0.75

0.89

0.98

0.96

0.99

0.75

0.89

0.96

0.98

0.96

0.99

0.74

0.89

0.95

0.69

0.98

0.96

0.99

0.74

0.89

0.95

0.69

0.71

0.0

0.2

0.4

0.6

0.8

1.0

(a) ViT-B/32 on 8 Tasks

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUN397

Cars

Test Task

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUN397

Cars

New Task

0.98

0.98

0.96

0.98

0.96

0.99

0.98

0.96

0.99

0.79

0.98

0.96

0.99

0.78

0.92

0.97

0.96

0.99

0.78

0.91

0.96

0.97

0.96

0.99

0.78

0.91

0.96

0.74

0.97

0.96

0.99

0.78

0.91

0.96

0.74

0.81

0.0

0.2

0.4

0.6

0.8

1.0

(b) ViT-B/16 on 8 Tasks

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUN397

Cars

Test Task

EuroSAT

SVHN

MNIST

DTD

RESISC45

GTSRB

SUN397

Cars

New Task

0.99

0.99

0.97

0.99

0.97

0.99

0.99

0.97

0.99

0.79

0.98

0.97

0.99

0.79

0.93

0.98

0.97

0.99

0.79

0.93

0.97

0.98

0.97

0.99

0.79

0.93

0.97

0.77

0.98

0.97

0.99

0.79

0.93

0.97

0.77

0.90

0.0

0.2

0.4

0.6

0.8

1.0

(c) ViT-L/14 on 8 Tasks

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

Test Task

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

New Task

0.95

0.94

0.99

0.94

0.99

0.77

0.94

0.99

0.77

0.99

0.94

0.99

0.77

0.99

0.81

0.93

0.99

0.75

0.99

0.81

0.94

0.93

0.99

0.75

0.99

0.80

0.94

0.95

0.93

0.98

0.75

0.99

0.80

0.94

0.95

0.92

0.93

0.98

0.74

0.99

0.80

0.94

0.95

0.92

0.81

0.93

0.98

0.74

0.99

0.80

0.94

0.94

0.92

0.81

0.93

0.93

0.98

0.74

0.99

0.80

0.94

0.94

0.92

0.81

0.93

0.90

0.93

0.98

0.74

0.99

0.80

0.94

0.94

0.92

0.81

0.92

0.90

0.72

0.93

0.98

0.73

0.99

0.80

0.94

0.94

0.92

0.81

0.92

0.90

0.72

0.75

0.92

0.98

0.73

0.99

0.80

0.94

0.94

0.92

0.82

0.92

0.90

0.72

0.75

0.67

0.0

0.2

0.4

0.6

0.8

1.0

(d) ViT-B/16 on 14 Tasks

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

Test Task

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

New Task

0.98

0.98

1.00

0.98

1.00

0.79

0.98

1.00

0.79

0.99

0.98

1.00

0.79

0.99

0.87

0.98

1.00

0.78

0.99

0.87

0.97

0.98

1.00

0.77

0.99

0.87

0.97

0.96

0.98

1.00

0.77

0.99

0.87

0.97

0.96

0.95

0.98

1.00

0.77

0.99

0.87

0.97

0.96

0.95

0.74

0.98

1.00

0.77

0.99

0.87

0.97

0.96

0.95

0.75

0.96

0.98

1.00

0.77

0.99

0.87

0.97

0.96

0.95

0.76

0.96

0.93

0.98

1.00

0.77

0.99

0.87

0.97

0.96

0.95

0.75

0.96

0.93

0.77

0.98

1.00

0.76

0.99

0.87

0.97

0.96

0.95

0.73

0.96

0.93

0.77

0.88

0.98

1.00

0.76

0.99

0.87

0.97

0.96

0.95

0.74

0.96

0.93

0.77

0.88

0.70

0.0

0.2

0.4

0.6

0.8

1.0

(e) ViT-L/14 on 14 Tasks

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

Test Task

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

New Task

0.85

0.84

0.98

0.84

0.98

0.98

0.84

0.98

0.98

0.93

0.84

0.98

0.98

0.93

0.83

0.83

0.97

0.98

0.93

0.83

0.85

0.82

0.97

0.98

0.93

0.83

0.85

0.93

0.82

0.97

0.98

0.93

0.82

0.84

0.93

0.70

0.82

0.97

0.98

0.93

0.81

0.85

0.93

0.67

0.98

0.81

0.97

0.98

0.93

0.81

0.84

0.93

0.66

0.98

0.99

0.81

0.97

0.98

0.92

0.81

0.84

0.93

0.60

0.98

0.99

0.53

0.81

0.97

0.98

0.92

0.81

0.84

0.93

0.60

0.98

0.99

0.53

0.68

0.81

0.97

0.98

0.92

0.82

0.85

0.93

0.67

0.97

0.99

0.50

0.67

0.86

0.81

0.97

0.98

0.91

0.82

0.85

0.93

0.67

0.98

0.99

0.50

0.67

0.86

0.96

0.81

0.97

0.98

0.91

0.82

0.85

0.93

0.68

0.97

0.99

0.50

0.67

0.86

0.96

0.92

0.81

0.96

0.98

0.91

0.82

0.85

0.93

0.67

0.97

0.99

0.49

0.67

0.86

0.96

0.92

0.88

0.81

0.96

0.98

0.91

0.82

0.85

0.93

0.67

0.97

0.99

0.49

0.66

0.86

0.96

0.92

0.88

0.70

0.81

0.96

0.98

0.91

0.82

0.85

0.93

0.68

0.97

0.99

0.49

0.66

0.86

0.96

0.92

0.88

0.69

0.95

0.80

0.96

0.98

0.91

0.82

0.85

0.93

0.67

0.97

0.99

0.48

0.66

0.86

0.96

0.92

0.88

0.69

0.95

0.74

0.81

0.96

0.98

0.91

0.82

0.84

0.93

0.67

0.97

0.99

0.48

0.66

0.86

0.96

0.92

0.88

0.68

0.95

0.74

0.87

0.0

0.2

0.4

0.6

0.8

1.0

(f) ViT-L/14 on 20 Tasks

1

2

3

4

5

6

7

8

Task Number

0.6

0.7

0.8

0.9

1.0

C. Ties

MagMax

OPCM

MINGLE

(a) 8 tasks

2

4

6

8

10

12

14

Task Number

0.65

0.70

0.75

0.80

0.85

0.90

0.95

1.00

C. Ties

MagMax

OPCM

MINGLE

(b) 14 tasks

2

4

6

8

10

12

14

16

18

20

Task Number

0.5

0.6

0.7

0.8

0.9

1.0

C. Ties

MagMax

OPCM

MINGLE

(c) 20 tasks

C.4 Inference Efficiency and Parameter Overhead

Tab. 15 compares the inference efficiency and parameter overhead of all baselines on the CLIP ViT-B/32 model after merging eight tasks. We report the total number of parameters, additional storage and inference parameters, throughput (images per second), and accuracy. The results show that most static merging methods ( e.g. , Task Arithmetic, Ties-Merging, MAGMAX-Ind, OPCM) incur no extra storage or inference overhead, but typically achieve limited accuracy. Consensus TA and WEMOE introduce significant storage overhead, while WEMOE also scales up inference parameters considerably. By contrast, MINGLE achieves a favorable trade-off: although it introduces additional parameters for LoRA experts and the router, the effective inference overhead remains small, and throughput is only marginally reduced compared to static baselines. This efficiency advantage comes while delivering substantially higher accuracy.

C.5 Forward Transfer Analysis

Forward transfer (FWT) is an important metric in continual learning, as it quantifies how effectively prior knowledge facilitates the learning of future tasks. We adopt the standard definition:

where a t ( θ merged t ) denotes the test accuracy on task t using the merged model after task t , and ¯ a t is the accuracy of the individually fine-tuned model for task t . Positive FWT indicates beneficial transfer, while negative values suggest interference.

Tab. 16 reports the results for the 8-task continual merging setup on CLIP ViT-B/16. The results demonstrate that MINGLE achieves nearly zero forgetting (BWT ≈ 0 ) while obtaining the highest forward transfer among all baselines, showing that our adaptive gating and merging strategy not only preserves past knowledge but also enhances feature utility for future tasks.

C.6 Additional Visualizations of Gate Activations and the Relaxation Effect

We provide an extended ablation study on gate hyper-parameters, including visualizations of gate activations under 14-task (Fig. 7) and 20-task (Fig. 8) configurations, complementing the 8-task results presented in the main paper. The visualizations demonstrate that the null-space constraint remains effective as the number of tasks increases, consistently suppressing gate responses to inputs from previously seen tasks and thereby mitigating forgetting.

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 1

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 2

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 3

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 4

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 5

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 6

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 7

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 8

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 9

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 10

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 11

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 12

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 13

Flowers102

STL10

DTD

MNIST

CIFAR100

OxfordIIITPet

GTSRB

RESISC45

PCAM

EuroSAT

SVHN

SUN397

Cars

FER2013

0

1

Gate Activation

Gate 14

w/o Null-Space

°

=

0

°

=

1

°

=

3

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 1

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 2

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 3

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 4

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 5

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 6

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 7

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 8

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 9

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 10

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 11

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 12

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 13

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 14

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 15

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 16

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 17

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 18

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 19

RenderedSST2

GTSRB

CIFAR10

SVHN

PCAM

CIFAR100

Food101

KMNIST

MNIST

STL10

EMNIST

FER2013

Cars

OxfordIIITPet

RESISC45

FashionMNIST

DTD

Flowers102

SUN397

EuroSAT

0.0

0.5

1.0

Gate Activation

Gate 20

w/o Null-Space

°

=

0

°

=

1

°

=

3

D Discussions

D.1 Use of Unlabeled Adaptation Samples

In our experiments, we simulate a realistic deployment setting by randomly sampling 5 unlabeled examples per class from the test split, which serve as adaptation samples for model merging. Such small unlabeled buffers are practical in real-world applications and can be obtained from various sources, including (i) incoming test-time data such as recent user queries or model inputs, (ii) held-out validation inputs or small training subsets (if available), (iii) user-provided samples ( e.g. , few-shot examples) that do not raise privacy concerns, (iv) synthetically generated data, or (v) manually curated public data. Importantly, our method does not depend on precise sample selection, and the buffer size remains fixed and small, ensuring feasibility and robustness in deployment scenarios.

D.2 Relation to Rehearsal-Free Continual Learning

Test-time continual model merging (TTCMM) is closely related to the paradigm of rehearsal-free continual learning (RFCL), as both approaches share two fundamental constraints: (i) they do not retain past training data, and (ii) they avoid storing previous task models. The key distinction lies in the information available at each stage. RFCL assumes access to the training data of the current task and incrementally fine-tunes a single model over time. In contrast, TTCMM assumes access to independently fine-tuned models for each new task and focuses on merging these expert models rather than training them from scratch. Additionally, TTCMM relies on a small unlabeled buffer at test time ( e.g. , 5 samples per class) to guide the merging process.

From a privacy perspective, TTCMM provides stronger guarantees. Since it does not require access to full training sets, it only depends on a small set of unlabeled samples, which can be user-provided without risk, synthetically generated, or curated from public data. By comparison, RFCL requires access to large-scale labeled datasets for every task, raising more significant concerns regarding privacy, storage, and legal constraints ( e.g. , medical images, personal data, or copyrighted corpora). The reliance on a tiny unlabeled buffer makes TTCMM more practical in scenarios where data privacy is a primary consideration.

D.3 Limitations

As with many model merging methods, our approach assumes that all independently fine-tuned models originate from a shared pretrained initialization. The extent to which this assumption influences merging performance remains unclear and warrants further investigation. In addition, our current experiments focus on merging models with identical backbone architectures ( e.g. , CLIP ViT-B/16). Although our use of LoRA-based expert offers some structural uniformity, which could potentially accommodate heterogeneous backbones, we have not yet explored this setting. Extending our framework to support diverse initialization points or architectural variants remains an open direction for future work.

D.4 Broader Impacts

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes mingle for test-time continual model merging, which alleviates the conflict between tasks through mixtures of lora experts and adaptive null-space constrained gating, bringing significant improvement compared to the existing continual model merging methods.\n\nSTRENGTHS AND WEAKNESSES:\n+ Strength\n\n    + Through a lightweight seed dataset, the model merging effect is significantly improved with slightly increased computational overhead.\n    + Clearly illustrate the advantages of the moe structure over traditional continual model mergin.\n\n+ Weakness\n\n    +  Lack of comparative experiments with baselines on efficiency, storage overhead.\n    + The relationship between continual model merging and rehearsal-free continual learning needs to be discussed. Because both continual model merging and rehearsal-free continual learning do not have training data of previous tasks when learning task t, both require training data of the current task and need to have learned models of previous tasks. I do not understand why continual model merging brings about improved privacy. \n    + If I understand correctly, the advantage of continual model merging may be that it can train models for t tasks separately and solve any permutation and combination of these t tasks, which is the flexibility. The models for each task must be saved to achieve this flexibility. Given conditions above, if test-time adaptation is added to continual model merging, I have the following concerns: doing **independent** test-time adaptation for **the downstream task currently being tested** may achieve better results because there is no need to consider conflicts between tasks.\n\nQUESTIONS:\n+ Can you provide a comparative experiment on efficiency with baselines?\n+ Is there any forward transfer?\n\nLIMITATIONS:\n+ Extending their framework to support diverse initialization points or architectural variants remains an open direction for future work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MINGLE, a framework that integrates test-time adaptation into continual model merging.  MINGLE inserts low-rank LoRA experts into a pretrained CLIP backbone and uses a Null-Space Constrained Gating mechanism, relaxed adaptively to combine experts based on incoming inputs.  At each task stage, a small seed buffer of unlabeled test samples guides light adaptation of the new task’s router network, mitigating interference with prior tasks.  Extensive experiments over multiple CLIP sizes, task orders, and distribution‐shifted benchmarks are presented.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The idea of adapting the MoE router network during test-time is interesting\n2. The paper is well-written and easy to follow\n3. The TTA is robust to the number of samples\n4. Comprehensive experiments \n\n\nWeakness:\n1. MAJOR CONCERN: My major concern with this paper is whether the MoE framework used in this paper counts as model merging. Cause essentially, the merged parameters are static, but in your moe framework, the model parameter can be thought of as dynamically adjusted based on the input. The usage of this MoE structure will be naturally stronger than traditional merging methods. This raises concerns about the fairness of comparisons to other merging methods in the experiments.\n2. Based on (1), in Table 3 (MTIL benchmark), MINGLE’s continual merging variant only marginally beats MoE-based baselines MoE-Adapter and DIKI , and underperforms RAIL. \n3. In  Fig3, except for Gate 6, most gate-task activation peaks remain ≤ 0.5, despite each LoRA adapter being trained specifically on its task (which should resemble Gate 6’s behavior).  This under-activation suggests either: (a) the gating network fails to learn strong specialization, or (b) the null-space constraints overly suppress new task signals.  Either way, it underactivates the correct experts for a given task.\n\nQUESTIONS:\nOverall, I find this paper is well-written and easy to follow. But I am extremely concerned about the usage of MoE in the merging paradigm. I hope the authors could address my concerns, and I am happy to raise the score if all my concerns are addressed.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMINGLE integrates test-time adaptation into model merging (Test-Time Continual Model Merging, TTCMM). It uses a small buffer of unlabeled current-task samples to guide the merge process at inference. The model has a mixture-of-experts design: each task adds a lightweight low-rank expert module and an input-dependent gating function. When a new task is added, its expert and gate are appended; only the new gate is tuned on the current task’s data while earlier experts and gates remain frozen. This lets the unified model adapt to the new task’s distribution on the fly, without retraining or accessing old data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \nIn terms of quality, it presents a solid theoretical justification (via Theorem 1) for using a dynamic mixture-of-experts rather than static averaging, accompanies that with a clear algorithm (Algorithm 1) and detailed equations, and backs it up with a comprehensive empirical study—covering three CLIP backbones, task sequences of 8/14/20 tasks, seven corruption benchmarks, ablations, and efficiency analyses. Clarity is strong: the narrative flows logically from motivation through theory to experiments; Figures 1-2 quickly convey what distinguishes Test-Time Continual Model Merging (TTCMM) from prior paradigms; notation is consistent; and hyper-parameter roles are later dissected. The significance is notable because the work tackles a practical scenario—merging independently fine-tuned models without sharing data—and achieves 7–9 percentage-point gains in average accuracy while almost eliminating backward forgetting. Finally, on originality, the paper introduces the TTCMM setting and proposes Adaptive Null-Space Constrained Gating, which relaxes hard orthogonality on-the-fly based on measured interference; together, these represent a fresh and principled combination of ideas that goes beyond existing continual model-merging approaches.\n\n\nWeaknesses \nThe main limitations revolve around scope, experimental breadth, and some presentation details. Regarding quality, all experiments focus on vision tasks with CLIP backbones; the method’s applicability to NLP, speech, or non-contrastive architectures is untested. The test-time adaptation loop requires roughly 50 gradient steps per task, which, while measured, could be too heavy for latency-critical or edge deployments, and baseline hyper-parameter parity is not fully transparent. Concerning significance, the method still assumes that each task ships its own LoRA weights plus a small buffer of unlabeled samples, constraints that may not hold in truly federated or on-device scenarios. Finally, in originality, while the adaptive gating mechanism is clever, its components—LoRA experts, MoE routing, orthogonal projection, EMA statistics—are individually incremental; the novelty lies more in their particular integration than in a new algorithmic primitive.\n\nQUESTIONS:\n1. Provide at least one experiment on a non-vision backbone (e.g., a RoBERTa model fine-tuned on text tasks or a wav2vec-based speech model) or justify why the method is inherently vision-only.\n2. Regarding efficiency: (i) report wall-clock adaptation time for 14- and 20-task runs; (ii) add an ablation that sweeps 5/10/20 steps to show accuracy/forgetting trade-offs; (iii) discuss possibilities for step-freezing or fast first-order updates.\n3. Add a consolidated table in the appendix listing all baselines’ key settings (learning rates, adaptation steps, projection ranks, etc.) and clarify whether you performed any per-method tuning on your task orders.\n4. You consider five unlabeled samples per class, selected from the test split: (i) Clarify how this buffer would be obtained in practice; (ii) extend Fig. 5 by including the “zero-sample” condition (pure parameter-space merge) to examine robustness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel paradigm called Test-Time Continual Model Merging (TTCMM), which uses a small set of unlabeled test samples from the current task to guide the merging process at test time. Then the paper introduces a method for this setting, called MINGLE, designed to integrate independently fine-tuned models sequentially without needing the original training data. The method uses MoEs composed of LoRAs and proposes a “Null-Space Constrained Gating” mechanism that restricts updates to subspaces orthogonal to the representations of previous tasks. It also introduces “Adaptive Relaxation Strategy” that dynamically modifies the constraint's strength based on interference signals detected during adaptation. The authors include experiments on CLIP ViTs for sequences of 8, 14 and 20 tasks.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n\n1. The paper is well-written and easy to follow. The motivation for using a dynamic Mixture-of-Experts approach over static averaging is clearly articulated. The same applies to core ideas, such as null-space projection, which are intuitive and well-explained.\n\n2. Strong performance: MINGLE consistently outperforms a wide range of baselines across multiple model backbones (ViT-B/32, B/16, L/14) and on task sequences of varying lengths (8, 14, and 20 tasks). The method also shows strong robustness to various data corruptions. See weaknesses for some issues.\n\n3. The paper successfully combines multiple ideas such as MoE, LoRA, and orthogonal projections. The ablation study effectively demonstrates each component’s contribution to the final performance.\n\n\n*Weaknesses*:\n\n\n1. Questionable Problem Setting: The studied setting is somewhat difficult to justify in a practical scenario. The paper assumes access to the current task's model ($\\\\theta_t$) and its unlabeled test data, but *no* access to the data or even the task vectors from previous tasks. It is unclear why the previous task vectors, which are just model parameters, would be unavailable while the models themselves are being merged. Furthermore, since the seed data is unlabeled, it is not obvious why similar unlabeled data from previous tasks could not also be stored or accessed. Overall, the setup feels contrived and needs stronger motivation.\n\n2. Potentially Unfair Baseline Comparisons: The comparison to other methods may be unfair due to the additional memory and computational overhead (additional parameters, saving activations, access to data) of the proposed method. The work notes similarity to WEMOE, but it is tailored to the continual settin–is the performance gap due to differing assumptions rather than strong methodological contributions?\n\n3. High Number of Hyperparameters: The proposed method introduces a considerable number of new hyperparameters, including the LoRA rank $r$, the null-space dimension $k$, the EMA decay $\\\\beta$, the relaxation coefficient $\\\\gamma$, the number of adaptation steps. This effectively limits the applicability of the method (compounded by the next point)\n\n4. Limited Architectural Scope: The experiments are exclusively focused on CLIP-ViT models. It is unclear how the setting applies to LLMs, such as T5, Llama, mistral etc, which are commonly used in the model merging literature.\n\n5. Minor issues wrt writing: explain $\\\\alpha$ in Equation 3, missing \\[a\\] for the benchmarks of 8,14 and 20 tasks. Better differentiate in the related work section and introduction works like TA, wise FT etc which focus on the static setup but are sometimes grouped with more continual papers.\n\n\\[a\\] Wang, K., Dimitriadis, N., Ortiz-Jimenez, G., Fleuret, F., & Frossard, P. (2024). Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813.\n\nQUESTIONS:\nSee weaknesses. Also:\n\nWhat is the task-specific routing error $\\epsilon_t$? Is it the error of not selecting the correct expert for an input X? If yes, doesnt this defeat the purpose of L143-144 (“it cannot specialize to regions where one expert is clearly superior.”)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}