Enhanced Continual Learning of Vision-Language Models with Model Fusion
Authors: Haoyuan Gao
Zicong Zhang
Yuqi Wei
Linglan Zhao
Guilin Li
Yexin Li
Bo Wang
Linghe Kong
Weiran Huang
Venue: ICLR 2026 Poster (2026)
Keywords: continual learning
multi-domain task incremental learning
vision-language models
model fusion
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We propose a novel continual learning approach for VLMs by introducing model fusion, which supports both parameter-efficient and full fine-tuning paradigms.

Abstract: Vision-Language Models (VLMs) represent a significant breakthrough in artificial intelligence by integrating visual and textual modalities to achieve impressive zero-shot capabilities. 
However, VLMs are susceptible to catastrophic forgetting when sequentially fine-tuned on multiple downstream tasks. Existing continual learning methods for VLMs face various limitations, often relying on additional reference datasets, compromising zero-shot performance, or being restricted to parameter-efficient fine-tuning scenarios.
In this paper, we propose a novel Continual Decoupling-Unifying (ConDU) approach that pioneers the use of model fusion for continual learning in VLMs. 
Specifically, ConDU maintains a unified model along with task triggers and prototype sets, employing an iterative process of decoupling task experts for previous tasks and unifying them with the task expert for the newly learned task. 
Additionally, we introduce an inference strategy for zero-shot scenarios by aggregating predictions from multiple decoupled task experts.
Extensive experiments on the MTIL benchmark show that ConDU achieves up to a 2\% improvement in average performance across all seen tasks compared to state-of-the-art baselines, while also enhancing zero-shot capabilities relative to the original VLM. 
Our code is available at https://github.com/zhangzicong518/ConDU.

Published as conference paper at ICLR 2026
MIFA LAB
ENHANCED CONTINUAL LEARNING OF VISION-LANGUAN MODELS WITH MODEL FUSION
Haoyuan Gaol Zicong Zhang'*, Yuqi Wei', Linglan Zhao'
Guilin Li*, Yexin Li", Bo Wang', Linghe Kong', Weiran Huang! 1.2.1
Shanghai Jiao Tong University Shanghai Innovation Institute
3 State Key Laboratory of General Artificial Intelligence, BIGAI 4 Tencent
ABSTRACT
Vision Vision-Language Models (VLMs) represent significant breakthrough in artifi- cial intelligence by integrating visual and textual modalities to achieve impres- sive zero-shot capabilities. However, VLMM are susceptible to catastrophic for- getting when sequentially fine-tuned on multiple downstream tasks. Existing continual learning methods for VLMs face various limitations, often relying on additional reference datasets, compromising zero-shot performance, or being re- stricted to parameter-efffcient fine-tuning scenarios. In this paper, propose Cocpuuliin-Unnifitii (ConDU) approach that pioneers the use
novel Continual of model fusion for continual learning in VLMs. Specifically, ConDU maintains a unified model along with task triggers and prototype sets, employing an iter- ative process of decoupling task experts for previous tasks and unifying them with the task expert for the newly learned task. Additionally, we introduce an inference strategy for zero-shot scenarios by aggregating predictions from mul- tiple decoupled task experts. Extensive experiments on the MTIL benchmark show that ConDU achieves up to 2% improvement in average performance across all seen tasks compared to state-of-the-art baselines, while also enhanc- ing zero-shot capabilities relative to the original VLM. Our code is available https: / g thub. com / eecanngzicongi. B/CoDU..
INTRODUCTION
Artificial Neural Networks (ANNs) often suffer significant performance drop on earlier tasks when learning sequentially. This issue, known as catastrophic forgetting (McCloskey & Cohen, 1989; Ramasesh et al., 2020), limits the adaptability of ANNs in dynamic environments. To overcome this challenge, continual learning (also referred to as lifelong learning) (Zenke et al., 2017: Kirkpatrick et al., 2017; Verwimp al., 2023; Shi et al., 2024) has been developed. This paradigm aims to enable machine learning models to acquire new knowledge over time while preserving previously learned information, thus mimicking the adaptability of the human brain.
Recently, Vision-Language Models (VLMs) such as CLIP (Radford et al., 2021) have made a ma- jor breakthrough in artificial intelligence by integrating visual and textual modalities to achieve impressive zero-shot capabilities. However, despite their demonstrated success (Shen et al., 2021; Zhao et al., 2023b; Fan et al., 2024), VLMs remain susceptible to catastrophic forgetting when fine- tuned for multiple downstream tasks. Conventional continual learning approaches are insufficient for VLM fine-tuning, as they struggle to maintain the crucial sero-shot capabilities that make these models valuable (Zheng et al., 2023).
In contrast to the extensive research on conventional continual learning, relatively few methods (Zheng et al., 2023; Yu et al., 2025; Park, 2024; Xu et al., 2024) have been proposed for continual learning of VLMs. Some methods, such as (Zheng et al., 2023) and (Yu et al., 2025), require ad-
'Hayyuan (gaohaoyuan edu.cn) and Zicong contributed equally to this work. This work was con- ducted at MIFA Lab (members from SJTU & SII). 1 CCoresponding author.

---

Published as conference paper at ICLR 2026
MIFA LAB
ditional reference datasets for distillation from pre-trained models, and their performance is highly sensitive to the choice of the dataset (Zheng et al., 2023). Moreover, these methods require careful tuning of multiple handcrafted hyperparameters to balance different optimization objectives: miti- gating catastrophic forgetting, preserving zero-shot capabilities, and optimizing performance on the current task. Alternative methods (Yu et al., 2024; Park, 2024; Xu et al., 2024) focus exclusively on parameter-efficient fine-tuning (Ding et al., 2023) employing modules such as adapters or LoRA (Hu al., 2021), but struggle adapt to full fine-tuning scenarios.
To overcome these limitations, we propose the Continual Decoupling- Uniifyin (ConDU), a novel continual learning approach for VLMs that is the first to introduce model fusion for this purpose. Model fusion (Ilharco et al., 2022; Yang et al., 2023; Huang et al., 2024) is a technique that com- bines multiple models into a single unified model without requiring access to the original training data. This property is particularly well-suited for the sequential learning scenario, as it allows one to maintain a single unified model that can be decoupled into multiple task experts to handle different tasks. However, direct application of model fusion (iteratively merging new task experts into a sin- gle unified model) is unsuitable for continual learning as it causes severe performance degradation. Therefore, we carefully designed rating new task experts at the individual model level. Furthermore, ConDU is inherently compatible with both parameter-efficien and full fine-tuning paradigms, offering flexible solution for diverse continual learning scenarios.
caapoupliinguniffiiin framework that avoids this issue by incorpo-
ConDU maintains unified delta model and set of task triggers throughout the continual learning process. ConDU handles each new task by fine-tuning the pre-trained VLM to obtain its task ex- pert, decoupling to obtain past task experts via task triggers, and unifying all task experts into an updated unified model with new task triggers. We remark that the decoupling and unifying proce- dures introduced in ConDU are training-free, and thus their running time is much shorter than the time required for model fine-tuning, Moreover, compared to previous continual learning methods for VL.Ms mentioned earlier, ConDU eliminates the need for adjusting trade-off hyperparameters, incorporating reference datasets, and maintaining replay exemplars.
After the above continual learning process, our method supports multiple inference scenarios. If the test sample belongs to a previously seen task and its task ID is known, we can directly reconstruct the corresponding task expert and use it for prediction. When the task ID is unknown or the test sample comes from unseen task (i.e., the zero-shot scenario), we can instead reconstruct multiple task experts relevant to the test sample's 's domain and make a prediction by aggregating their results. Evaluated widely used benchmarks across diverse settings, including Multi-domain Task Incre- mental Learning (MTIL), few-shot MTII,, and task-agnostic MTII,,, ConDU achieves up to a 2% improvement in average performance across all seen tasks compared to state-of-the-art baselines, demonstrating the effectiveness of incorporating model fusion. Moreover, ConDU exhibits strong zero-shot capabilities of VLMs, outperforming the original pre-trained VLM and other state-of-the- art continual learning methods.
The contributions of this work can be summarized as follows:
We introduce model fusion into continual learning for VLMs and propose novel Continual Decoupling- -nnifiin (ConDU) framework, which is compatible with both parameter-efficieet and full fine-tuning paradigms.
We propose aggregating the predictions of multiple decoupled models for zero-shot scenarios.
Through extensive experiments, we demonstrate that ConDU effectively learns new knowledge while preserving previously acquired knowledge and enhancing zero-shot capabilities.
RELATED WORK
Continual Learning for VLMs. Conventional continual learning has been extensively studied, including architecture-baser methods (Rusu et al., 2016; Mallya & Lazebnik, 2018; Serra et al., 2018), replay-based methods (Riemer et al., 2018; Buzzega et al., 2020; Boschini et al., 2022; Gao & Liu, 2023; Kim et al., 2024), and regularization based methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Li & Hoiem, 2017; Zhao et al., 2023a; Lu & Sun, 2024). However, these methods cannot be directly applied to recently developed Vision-Language Models (VLMs), as they struggle to maintain the crucial zero-shot capabilities (Zheng al., 2023).
2

---

Published as conference paper at ICLR 2026
MIFA LAB
Recently, continual learning methods specifically designed for VLMs have been introduced. These methods can be broadly classified into parameter-efficient fine-tuning based approaches ( (Wang et al., 2022: Yu et al., 2024; Park, 2024; Li et al., 2024; Xu et al., 2024) and distillation-basee methods (Ding et al., 2022; Zheng et al., 2023; Yu et al., 2025). However, these methods either require reference datasets the careful adjustment of trade-off hyperparameters, or they not suitable for full fine-tuning, In contrast, our method eliminates these drawbacks by introducing model fusion.
Model Fusion. Model fusion combines multiple models into single unified model, retaining the strengths of its constituent models without requiring additional training data. Fisher Merg- ing (Matena & Raffel, 2022) and RegMean (Jin et al., 2022) use Fisher information matrices (Fisher, 1922) and inner-product matrices (Jin et al., 2022), respectively, to compute fusion coefficients for weighted model fusion. Task Arithmetic (Ilharco et al., 2022) introduces a fusion technique that combines models by summing delta models, where a delta model is defined as the difference be- tween the parameters of a fine-tuned model and its pre-trained counterpart. Other approaches, such as TIES Merging (Yadav et al., 2024), Ada Merging (Yang et al., 2023), DARE (Yu et al., 2023), and EMR Merging (Huang et al., 2024), focus on enhancing delta model-based fusion in various ways.
PROBLEM FORMULATION
In this paper, we focus on continual learning for Vision-Language Models (VLMs).. Given pre- trained VLM (e.g., CLIP (Radford al., 2021)), a sequence of I'' tasks arrives incrementally, where each task is associated with a training dataset D These tasks may involve distinct classes, different domains, or exhibit significant variation in sample sizes. After seeing each task, the VLM can be updated with access to a limited memory storing essential information (e.g., selected past data or parameters). The goal is to develop a method that incrementally updates the VLM while achieving high performance all previously encountered tasks and retaining its zero-shot capabilities for unseen tasks. Additionally, we aim for the proposed continual learning method to support both aarameter-efficeen fine-tuning (e.g., LoRA or adapters) and full fine-tuning.
Under the continual learning setting, the system is permitted to retain only a single VLM throughout training. Yet, if multiple models were allowed, one could simply fine-tune a separate model from the pre-trained VLM for each task and choose the corresponding model at test time whenever the task identity is known. In addition, a defining property of VLMs is their zero-shot ability, which ideally should be preserved-on even improved-after continual learning. For test samples from unseen tasks (the zero-shot case), using several specialized models fine-tuned on different domains and aggregating their predictions would naturally outperform relying on single VLM.
Motivated by this observation, if the shared components across these individually fine-tuned models could be extracted and merged into one maintained VLM, while the task-specific differences are stored in limited memory, then single main VLM plus small auxiliary memory could effectively mimic the behavior of multiple task- mask-specialized models. Moreover, this idea is inherently compatible with both cefumettr-effccenn fine-tuning and full fine-tuning approaches.
4 CONDU: CONTINUAL DECOUPLING-UNIFVIN
We propose Contimual Aecoupling-nnifyiii (ConDU), a novel continual learning approach for VLMs that leverages model fusion. Figure/ shows the overall framework of ConDU. ConDU maintains a unified model, a set of task triggers, and a series of prototype sets throughout the con- tinual learning process. Our framework includes five modules. We will introduce three modules for training in Section 4.1 and two modules for inference in Section 4.2.
4.1 DELTA MODELS CONTINUALLY FUSION AT TRAINING STAGE
At each session of the continual learning process, ConDU implements three steps: Tuning Indi- vidually, Decoupling Unified Model, and Unifying Models. The time spent Decoupling Unified Model and Unifying Models is nearly 1% of Tuning Individually (see Appendix I for detailed anal- ysis). Since the process of Decoupling Unified Model relies the task triggers produced during
3

---

Published as conference paper at ICLR 2026
MIFA LAB
Training Stage: Delta Models Continually Fusion
Task Triggers 1: 1:1-1
Task Triggers
Task Expert
Unified Model 1: t-1
Unified Model 1:t
Task Expert
Decouple
Unity
Task Expert 1
Pre-trained VLM
Task Expert
Fine-tune
Inference Stage: Semantic-based Aggregating Mechanism
Task
Task
Max
Weight
Task
Task
Task
Max
Weight Task
Proto
Proto
Sim
Test Sample
Pre-trained VLM
Image Feature
Final Prediction
lask Expert
Logits 1
Task Triggers
lask Expert
Logits
Unified
Decouple
Figure I: Overall framework of the proposed method. This framework includes designs for both the training stage and the inference stage. The upper part of the figure corresponds the training stage of session t, with the relevant components detailed in Section 4.1. The colored points "unify" and "decouple" illustrate the corresponding operations, which are explained in Figure 2a and Figure 2b, respectively. During the training stage, ConDU handles each new task by fine-tuning the pre- trained VLM to obtain its task expert, decoupling to obtain past task experts via task triggers, and unifying all task experts into an updated unified model with new task triggers. The lower part corresponds to the inference stage after session t, with its components detailed in Section 4.2. During the inference stage, ConDU calculate the cosine similarity between the image feature of the test sample and prototypes of each category in the feature space pre-trained VI.M, then choose the maximum similarity in each task as the weight of the corresponding task expert.
Unifying Models, we first introduce the process of Unifying Models before detailing process of Decoupling Unified Model.
Tuning Individually. We denote a VLM as f(: where 0 represents only the learnable pa- rameters of the VLM, excluding the frozen parameters for clarity. At session f, by fine-tuning the pre-trained VLM 8 on task we obtain a task expert 6r We defined delta model f, the parameter offsets of task expert relative to the pre-trained VLM, as 5' = 0¹ - f0. We will unify delta models instead of directly unifying task experts following the setting of advanced model fusion methods (Ilaarco et al., 2022; Yadav et al., 2024; Huang et al., 2024).
Unifying Models. The process of Unifying Models is illustrated in Figure 2a. When task t ar- rives, first decouple the current unified delta model glit-1 to obtain the approximation of 5 denoted as 8 for 2,2....,t (this process will be introduced in the paragraph of Decoupling Unified Model). Then let 5i 8 1,2,... .2....,,. 1, the calculation of unified delta model is
4

---

Published as conference paper at ICLR 2026
MIFA LAB
Models
Triggers
Triggers
Delta Models
Positive
Sum
-2
Task
Task
Absolute Maximum
Task
Different
Symboos
Mask:
Just
Symbol
Task
Task
Rescaler:
Copy
Copy
3
Unified Delta Model
Unified Delta Model
a) Unifying Models
b) Decoupling Unified Model
Figure 2: The process of Unifying Models (a) and Decoupling Unified Model (b) transformed to unifying delta models (a) and decoupling unified delta model (b), respectively. Unified Model Unified Delta Model + Pre-trained VLM. Task Expert Delta Model + Pre-trained VLM. a) When unifying delta models, the unified model is obtained by election process. Each task's 'ss task trigger is calculated according to the difference between the delta model and the unified delta model. b) When decoupling the unified delta model, we use the task trigger on the unified delta model to reconstruct the delta model i.
81 = unify([6',,'' ..... }), where 81# is the unified delta model, and the j-th dimension of is Σ -
calculated 8. max; (6)) if
This process means choosing the j-th parameter
(59) if
Σ
of all to t delta models with the largest absolute value and has the same sign as Σ 8). retaining the largest magniude and consistent sign information shared across the delta models.
Then the unified delta model is added to the pre-trained VLM to construct the unified model 01:t 8 g1:t
Except unified model, other productions of Unifying Models at session t are task triggers. For i 1, ........ .t, each task trigger will be used on the unified model reconstruct delta model in the future, composed of mask MZ with the same dimension as the delta model, and a rescaling scalar X The binary number of M position indicates whether the delta model has the same
if 8 gl:t
sign as the unified delta model at position j. that is Mj
if 83 81: 0. The rescaler is to
preserve the average magnitude of elements in 5 and M @ 5ist defined as X
sum(abs (8 sum(abst M (5E)))
The final productions of Unifying Model at session is unified model and task triggers. Then introduce how to use task triggers to decouple the unified model.
Decoupling Unified Model. The process of Decoupling Unified Model is illustrated in 2b. This process is needed both the beginning of a training session and the inference stage. If there are t seen tasks, task triggers are applied to the unified delta model to obtain delta models 3 X MM @ d+t, which are then added to the pre-trained VLM 6 to obtain task experts 0 6 + At a training session, 8 will attend the unifying instead of 8 At inference stage, the output logits of all reconstructed task expert f(. Ø*) will aggregated to predict the test samples. We will introduce this Aggregating Predictions in the next section.

---

Published as conference paper at ICLR 2026
MIFA LAB
4.2 SEMANTIC-BASED AGGREGATING MECHANISM AT INFERENCE STAGE
During inference, we propose Semantic-basee Aggregating Mechanism to predict sample without task ID or sample from unseen tasks. Specifically, the unified model is decoupled into task experts by task triggers at the inference stage. For test sample from a seen task with a known task ID, we choose the corresponding task expert to predict. For test sample from an unseen task or without task ID, we input the sample into all task experts, and the output logits are added with weights calculated by semantic matching, using memory-store prototypes computed during the training stage as the next paragraph shows.
Computing Prototypes. The process of Computing Prototypes illustrated in 3. For each cat- egory in each task, we save its prototype during training. The prototype of the k-th category in the i-th task is the mean of the image feature vectors plus the text feature vector for that category, extracted by the pre-trained VLM, that is P 0") + dataset of the k-th category in the i-th task, y is the text of this category, and Im is the m-th image of D. Then will introduce how the aggregating weights are calculated by these prototypes and how they are utilized to aggregate predictions.
Eme [ 00), where D is the
DD
Aggregating Predictions. The process of Aggregating Predictions is illustrated in Figure 1. For a test image x, we use f(: (0) to extract the image feature. We then calculate the cosine similarity between the test image feature and the learned prototypes of different tasks. For each task, we select the highest similarity score as the weight of this task expert for the test sample. We compare the weights across different task experts and reassign the weights of the K-highest tasks 1, and the others to O. The output logits of all task experts are added with these weights to determine the final prediction result. This value K is the only hyperparameter that needs be determined in our method, and the ablation study of K is in Appendix F, which shows that the performance of our method is very insensitive to the choice of K. The inference time of Aggregating Predictions is very close to that of inference by single model, since the time taken by the model selection phase is almost negligible, and the forward propagation of multiple task experts can be computed in parallel (see Appendix I for a detailed analysis of inference time).
Task
Proto
Proto
Appand
Protoype
Class K
Image Mean
Text Feature
Feature
Average
Image Feature
Pre-trained VLM
"A photo of a dog"
5 EXPERIMENTS
The process of Computing Prototypes: The prototype of each category is the mean of the image feature vectors plus the text feature vec- tor for that category, all extracted by the original pre-trained VLM.
Figure 3:
5.1 EXPERIMENT SETTING
We apply ConDU to two fine-tuning scenarios. ConDU (FT): Apply ConDU to full parameter fine-tuning. ConDU (LoRA): Freeze the pre- trained model parameters and only fine-tune the parameters of LoRA modules during the ConDU process. LoRA is one of the most com- monly used PEFT modules, and its detailed introduction is in Appendix L. We test our method on three benchmarks, including Multi-domain Task Incremental Learning (MTIL) (Zheng al., 2023), task-agnostic MTHL, and few-shot MTIL. The MTIL (Zheng et al., 2023) extends task incremental learning to a cross-domain setting, where each task is derived from a distinct domain, including 11 individual tasks. As in (Zheng et al., 2023; Park, 2024; Yu et al., 2024; Xu et al., 2024: Zhang et al., 2024), the tasks are arranged alphabetically, and we further report how ConDU outperforms SOTA methods under another task order in Appendix C. The task-agnostic MTIL is the variant of the MTIL bench- mark, where the task ID is unknown during inference for each test sample. The few-shot MTIL variant involves training with only five train samples per category for each task. We follow the exist- ing works (Zheng et al., 2023; Park, 2024; Yu et al., 2024; Li et al., 2024) to use three key metrics for evaluation. The "Transfer" metric evaluates the model's zero-shot transfer performance quent tasks. The "Average" metric evaluates the model's performance on specific task averaged over all sessions, regardless of whether the task has already been encountered in particular session. The
subse-

---

Published as conference paper at ICLR 2026
MIFA LAB
Table 1: Comparison with SOTA methods on MTIL benchmark in terms of "Transfer", "Average", and "Last" scores (%) We label the best methods on average of all datasets with bold styles. The lines with background color represent our methods. The results of more baselines can be found in Appendix B.
Aireraft
Calteh101
CIEARIO
EWoSAT
Flowass
OxTordPa
Method
DTD
ANNSS
Cars
Averuge
Zero-shot
88.4
59,4
65.2
Individual FT
62.0
95.1
89:6
79.5
98.9
97.5
92.7
99.6
94.7
89.6
61.8
89.2
ZSCL
86.0
67.4
45.4
50.4
69.1
87.6
61.8
86.8
60.1
66.8
68.1
Transfer
Dual-RAIL DPeCLIP
88.4
68.2
44.6
54.9
71.0
88. 5
59.6
89.0
64.7
65.2
69.4
88.2
67.2
44.7
54.0
70.6
88.2
59.5
89.0
64.7
64.8
69.1
87.8
ConDU (LoRA)
88.1
68.9
45.7
57.0
71.3
$8.8
61.2
89.1
65.
67.8
70.3
ConDU (PT)
88.1
68.9
46.4
57.1
71.4
$8.7
65,5
89.3
65.0
67.8
70,8
ZSCL
45.1
92.0
80.1
64.3
79.5
81.6
75.2
88.9
68.0
75.4
Average
Dual-RAIL
96.0
81.3
89.1
73.9
90.2
66.5
77.8
DPeCLIP
49.9
94.9
82.4
69.4
82.2
84.3
90.0
74.0
90.4
68.3
66.3
77.5
MulKI
52.5
93.6
79.4
67.0
79.8
83.9
89.6
77.
91.2
67.1
69.1
77.3 3
CoaDU (LoRA)
94.9
84.4
81.1
84.4
77.3
89.55
69.3
78.3
CouDU (PT)
39.6
93.4
83.7
68.1
83.4
83.7
900.1
76.7
90.6
68.6
68.6
78.8
ZSCL
40.6
92.2
81.3
70.5
94.8
90.5
91.9
9%.7
93.9
85.3
M0.2
83.6
Dual-RAIL
52.5
96.8
83.3
80.1
96.4
99.0
98.8
93.5
85.5
79.2
86.8
Last
DReCLIP
95.6
98-
99.4
81.7
MulKI
49.7
93.0
82.8
73.7
96.2
92.1
90.4
99.0
94.8
85.2
78.9
85.1
ConDU (LoRA)
48.9
95.2
87.8
78.5
96.3
95.2
91.7
97.6
92.0
85.3
78.8
86.2
ConDU
93.7
86.6
76.1
98.2
93.4
99.6
94.8
80.5
87.1
"Last" metric reflects the model's average performance at the end of the continual learning process. In the task-agnostic MTIL setting, we omit the "Transfer" and focus solely on the "Average" and "Last". More implementation details and description of baselines are provided in Appendix A. To explore the generality of the ConDU framework beyond vision-language models, we also conducted additional experiments on single-modalie class-incremental learning in Appendix D.
5.2 COMPARISON WITH STATE-OF-THE-ART METHODS
Multi-Domain Task Incremental Learning. Table presents the detailed comparison results of our proposed ConDU and the baselines on the MTIL benchmark. As seen, our method outperforms all baseline methods across all three metrics. The "Transfer" metric of our method is 70.8% for FT and 70.3% for LoRA, which exceeds the best bascline by 0.7% and surpasses the pre- praitrd VLM by 5.5%. The "Averrge" metric of our method is 78.8% for FT and 78.3% for LoRA, which exceeds the best baseline by 1.5% and surpasses the pre-trained VLM by 13.5%. The "Last" metric of our method is 87.1% for FT and 86.2% for LoRA, which exceeds the best baseline by 0.2% and surpasses the pre-trained VLM by 21.9%. These results highlight our approach's effectiveness in mitigating catastrophic forgetting while progressively incorporating new knowledge.
Task-Agnostic MTIL. Table presents the detailed comparison results of our proposed ConDU and the baselines the task-agnostic MTIL benchmark. As seen, our method outperforms all baseline methods across all two metrics. The "Average" metric of our method is 78.1 % for FT and 78.0% for LoRA, which exceeds the best baseline by 2% and surpasses pre-trained VLM by 20.3%. The "Last" metric of our method is 86.4% for FT and 85. % for LoRA, which exceeds the best baseline by 1.8% and surpasses the pre-trained VLM by 28.6%. These results highlight our approach's 's effectiveness in mitigating catastrophic forgetting while progressively incorporating new knowledge even without task ID.
Few-Shot MTIL. Table presents the comparison results of ConDU and the baselines on the few- shot MTIL benchmark. The "Transfer" metric of our method is 70.0% for FT and 70.3% for LoRA, which exceeds the best baseline by 1.4% and surpasses the pre-trained VLM by 4.7%. The "Aver- age" metric of our method is 72.3% for FT and 72.7% for LoRA, which exceeds the best baseline by 1.3% and surpasses the pre- trained VLM by 7.4%. The "Last" metric of our method 76.6% for FT and 77.4% for LoRA, which exceeds the best baseline by 1.3% and surpasses the pre-trained VLM by 12.1%. These results highlight our approach's 's effectiveness in mitigating catastrophic forgetting while progressively incorporating new knowledge even with very few samples.

---

Published as conference paper at ICLR 2026
MIFA LAB
Table Comparison with SOTA methods on task-agnostic MTIL benchmark in terms of "Tranffer", "Average", and "Last" scores (%). We label the best methods on average of all datasets with bold styles. The lines with background color represent our methods. Individual FT can not be utilized on task-agnostic MTIL, the Individual FT results here is the prediction with task ID while other methods cannot know the task ID.
Aironft
Catech101
CIPAR 100
EmroAAT
Flowar
OxfondPet
Method
DTD
Cum
Average
Zero-sbot
24.4
63.7
41.0
39.5
53.0
70.0
858.4
39.6
88.9
64.5
63.3
57.8
Individual
97.5
81.8
89.2
Continual FT
25.5
81.5
59.1
53.2
61.7
51.8
63.2
64.3
69.7
31.8
49.7
55.9
ZSCL
MoB
37.2
65.3
79.5
67.6
19.7
83.1
80.5
74,0
88.5
67.5
65.3
66.2
Average
Primal--RA RAIL
12.4
88.5
57.1
55.7
6-.77
80.7
$3.0
62.9
84.8
68.7
63.7
68.4
Dual RAIl-
45.0
88.8
57.8
56.8
66.2
81.0
85.2
63.4
87.8
68.9
64.7
69.6
CoLeCLIP
DPeCLIP
49.9
85.3
81.5
68.3
81.6
84.3
89.9
74.0
90.4
68.3
66.2
76.1
ConDU (LoRA)
51.8
94.4
84.2
68.8
80.0
84.1
90.0
77.1
88.9
68.8
69.3
78.0
ConDl
75.0
Crarinual
71.1
92.9
77.3
ZSCL
42.5
64.4
67.2
$4.8
89.7
90.4
91.7
95.8
93.4
85.2
78.3
77.6
Mols
34.1
47.6
80.9
75.5
0.0
93.0
70.8
99..4
86.4
79.8
68.9
66.9
Primal-RAIL
Laat
Dual-RAIL
45.2
94.4
74.7
70.7
87.3
97.9
86.5
92.8
91.9
81.7
76.7
81.8
CoLeCLIP
48.1
73.1
65.2
69.6
84.0
96.2
90.9
94.6
92.5
$2.6
79.3
79.7
DPeCLIP
19.9
84.2
83.2
97.0
95.8
99.1
93.9
$1.5
80.2
$1.6
(LoRA)
CoaDU (PT)
58.6
90.8
86.3
74.0
96.3
93.4
91.9
99.6
94.7
84.9
80.1
86.4
Table 3: Comparison with SOTA methods few-shot MTIL benchmark in terms of "Transfer", "Average", and "Last" scores (%). We label the best methods on average of all datasets with bold styles. The lines with background color represent our methods. The results of more baselines can be found in Appendix B.
Aircaft
Catech101
CIPAR100
EurSAT
Flowess
MNIST
OxfondPel
SUN395
Method
D70
Cas
Average
Zero-shot WISE-FT ZSCL
88.4
68.2
54.9
89.0
96.6
84.9
71.3
77.5
77.6
60.0
39.4
53.0
76.6
75.5
58.2
57.7
Tranderr
81.0
68.1
41.8
46.8
63.6
$1.9
61.1
81.4
55.5
62.2
65
MolB
87.9
68.2
44.1
48.1
64.7
SS..5
69.0
89.1
64.5
65.1
68.9
CoaDU (LoRA)
88.1
68.5
56.-1
71.2
39.0
6/100
88.8
61.9
66.1
70.3
Coarinual FT
28.1
$6.4
59.1
$2.8
55.8
62.0
70.2
64.7
75.5
35.0
$4.0
58.5
WT
32.0
87.7
61.0
68.1
69.3
76.8
71.5
77.6
42.0
59.3
63.7
Average
ZSCL
88.6
66.5
56.3
73.4
56.4
MoE:
30.0
89.6
73.9
$8.7
69.3
79.3
88..1
76.55
89.1
65.3
65.8
71.4
COnDU (FT)
33.1
90.5
74.1
58.3
76.2
81.0
85.99
73.4
88.0
64.8
67.1
72.3
CouDi
75.4
89.3
67.0
Continual
27.8
86.9
60.1
56.6
75.7
73.8
93.1
82.5
57.0
66.8
67.1
WiSE-FT
30.8
88.9
59.6
60.3
80.9
81.7
77.1
94,9
83.2
62.8
70.0
71.9
Last
ZSCL
26.8
88.5
63.7
55.7
60.2
82.1
82.6
$8.6
85.9
66.7
70.4
67.4
ConDU (PT)
90.7
75.0
63.1
88.8
88.6
87.0
91.8
85.6
66.5
71.9
7666
ConDU (LoRA)
31.8
92.4
76.7
63.4
86.8
91.8
85.6
93.9
90.3
68.1
70.9
77.4
5.3 ABLATION STUDY
PTM Task Expert Features. We compare prototype-sane sample similarity computed via (a) shared PTM features and (b) task-specific expert features. the latter variant, prototypes and test samples mapped into the feature space of the corresponding task expert for similarity computa- tion. Results show that our PTM--hased strategy consistently outperforms the expert-based approach. unified and reliable representation for cross-task
This indicates that the frozen PTM provides similarity compared to disjoint expert-specific spaces.
The Efffec of Rescalers. We compared ConDU (FT) with its no-rescaler variant. The variant per- forms substantially worse, confirming the necessity of rescaling. Importantly, without the rescaler,
8

---

Published as conference paper at ICLR 2026
MIFA LAB
the reconstructed task experts will produce features with significantly mismatched magnitudes com- pared to the original task experts, motivating the inclusion rescalers.
Table Comparison of "Transfer" and "Average" metrics on MTIL benchmark between PTM-based and Expert-base feature extraction approaches.
Method
Altrait
Catochi
CIFARIOO
DTD
EooSAT
Flowors
Food
MNNST
OxtontPet
Can
SUNDY
Arerage
Transfer
PIM-based based
Espurt-biae based
88.2
$8.1
68.9
08.9
40.3
46.4
57.0
49.1
69.7
71.3
88.7
86.4
65.5
62.9
84.6
89.3
65.0
59.7
67.4
07.8
67.7
70.8
Expert- based
69.6
93.4
83.7
67.5
81.7
82.7
77.0
88.2
65.1
68.3
77.8
PTM-based
Table 5: Comparison of "Transfer", "Average" and "Last" metries on MTIL benchmark between ConDU (FT) and its no-rescaler variant.
Method
Aiterfft
Calechi01
CIFAR/00
DTD
ESOSAT
Flowess
Food
MNISST
SUND9
Awergee
Transter
ConDU
$8.1
68.9
46.4
57.0
21.3
88.7
65.5
89.3
65.0
67.8
70.8
rescalers
88.2
68.7
46.1
56.5
71.4
88.7
63.8
89.1
66.7
70.1
Average
ConDU
59.6
83.4
78.8
Lust
ConDU
w/o rescalers
58.6
57.8
93.7
$9.9
86.6
83.1
66.2
98.2
96.9
30.3
93.4
91.9
88.7
99.6
99.6
94.8
84.4
84.9
74.3
77.1
73.1
87.1
DISCUSSION
Hardware Robustness. We conducted experiments (ConDU FT) on NVIDIA RTX 4090 and Huawei Ascend 910B to verify the hardware robustness. As shown Table the discrepancy remains negligible.
Computational Cost and Storage Analysis. We evaluate the efficiency of ConDU by comparing learnable parameters with SOTA methods (Fig- ure 4). ConDU (LoRA) requires significantly fewer parameters while achieving superior performance, whereas ConDU (FT) delivers the best performance among full-parameter update methods with a compa- rable parameter count. Regarding storage, ConDU significantly alleviates the overhead of Individual FT, with efficiency gains scaling as tasks and fine- tunable parameters increase (Appendix H). Furthermore, ConDU maintains competitive training and inference times (Appendix I): it matches Continual FT in training efficiency, saves 62% time compared ZSCL (Zheng et al., 2023), and retains inference speeds comparable to single model.
Table 6: Performance comparison across dif- ferent hardware platforms.
Transfer
Average
Last
Ascend 910B
70.2
79.0
87.0
RTX 4090
70.8
78.8
87.1
t-SNE Visualization of Feature Space. To compare the change of feature space of task experts from initial fine-tuning to the end of all sessions, we perform t-SNE visualization of the features extracted from the training data of Task 1 (AirCraft). Figure 5a illustrates that after session 1, the fine-tuned task expert 1 shows significantly better data discrimination on Task 1. Figure 5b demon- strates that throughout continual learning process by ConDU (FT), task expert 1 undergoes multiple rounds unifying and decoupling, but its feature space changes very little, almost undetectable by t-SNE. This indicates that the task expert reconstructed by ConDU closely matches the representa- tion ability of the model obtained through initial fine-tuning.
Convergence of Delta Models. We further theoretically analyzed that in the ConDU process, when the number of sessions approaches infinity, the change of each task expert parameter is mono- tonically non-increasing. The proof of this theorem is in Appendix G.
Theorem (Convergence of Delta Models). Suppose the relative order of rescalers remains invari- throughout the continual learning process. For any session & ' have t - delta models

---

Published as conference paper at ICLR 2026
MIFA LAB
Transfer Accuracy vs, Parameter Size
Accuracy vs. Parameter Size
70
ConDU (LoRA)
MIE Apopters
ConDu
Frima-RAIL
85
ConDU (LoRA)
DadlRALL
NoE VoE-Adapters
ConDU (FT)
Frimg/ KAIL
DusH-RAIL
25CL
65
WISF-FT
75
LVF
UWF-VB (LORA)
Accuracy
WISE-FT
WISE-FT (1984)
UWF-gR
65
55
WISCET (LORA) LWFF-R (LoRA)
50
LMF (OFA)
55
LWF (L084)
10
20
150
170
10
50
150
170
Paramaters (Million)
Paramaters (Million)
Transfer Acc
Last Acc
Figure 4: Comparison of parameter-accuracy trade-off with SOTA continual learning methods. The vertical axis represents the evaluation metrics of different continual learning methods (namely "Transfer" and "Lastt in the subplots), while the horizontal axis indicates the logarithm of the number of learnable parameters during the training process for each method.
cless
closs
class
(a)
class
class
cless
Tre-Trained
Session
class
class
class
class
class
(b)
class
closs
class
class
Session
Session
Session 11
class
Figure 5: We perform i-SNE visualization of the features extracted from the training data of Task (AirCraft). We use four models in total, including the pre-trained VLM and the task expert fine-tuned the end of sessions and reconstructed by ConDU (FT) at the end of sessions 6 and 11, to extract features of 10 randomly sampled data categories in Task 1. (a) We perform the t- SNE visualization on pre-trained model and task expert 1 independently to provide fair comparison. (b) We concatenate the features extracted from task expert after session 1, 6, and 11 for t-SNE visualization. The enlarged area to show the slight changes features.
(t))....................) (t)))() decoupled from bounded unified delta model, along with the latest delta model (t). If all delta models are independently and identically distributed, and the parameter signs of all delta models are identical for each dimension, then for each i f1,2....,f 1}. the following property holds.
lim E 810000
1) 1.
More Discussion. We comprehensively compared ConDU (FT) and ConDU (LoRA) in Appendix to show distinct advantages of full-fietuning and PEFT under different conditions. We also comprehensively compared ConDU and Individual FT in Appendix to show the effectiveness of decoupling unrppiiiggunfyiigggg mechanism. The accuracy of each tasks during each session shown in Ap- pendix K to demonstrate the reconstruction of our method for task experts.
10

---

Published as conference paper at ICLR 2026
MIFA LAB
ACKNOWLEDGMENTS
This project supported by the National Natural Science Foundation of China (No. 62406192), Shanghai Municipal Special Program for Basic Research on General Al Foundation Models (Grant No. 2025SHZDZXX25G003 Opening Project of the State Key Laboratory of General Artificial In- telligence (No. SKLAG120240P22, the Tencent WeChat Rhino-Bird Focused Research Program, Kuaishou Technology, and the SJTU Kunpeng & Ascend Center of Excellence.
REPRODUCIBILITY STATEMENT
Our code is available https: : /tt///ttttas com/ wtt.ccc///aaaaa httaaaagg..cnnn11////C1 nations of the experimental setup can be found in Section 5.1 and Appendix A, where provide the values all hyperparameters used during training. Consulting these sections may help repro- ducing the results and better understanding our released code.
Detailed expla-
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. Our research focuses algorithmic and method- ological contributions continual learning and does not involve human subjects, sensitive personal data, information that raises direct privacy security concerns. The datasets used in experi- ments (e.g., MTIL, TinylmageNet) are widely adopted public benchmarks released under permissive licenses, and we follow standard usage practices without modification that could introduce ethical risks. The proposed methods are intended for advancing machine learning research and have no fore- seeable harmful applications. We are not aware of any conflicts of interest, and the study complies with established principles of fairesss, transparency, and research integrity.
REFERENCES
Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class- incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 45(5), 2022.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. 1ood-101-mining discriminative compo- nents with random forests. In Computer Vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461. Springer, 2014.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark ex-
perience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33: 11550-119993 2020.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web). IEEE signal processing magazine, (9)6:14411142 2012.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen. Chi-Min Chan, Weize Chen, et al. Parameter-efffeient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-22 2350 2023.
Yuxuan Ding. Lingqiao Liu, Chunna Tian, Jingyuan Yang, and Haoxuan Ding, Don't stop learning: Towards continual learning for the clip model. arXiv preprint arXiv:2207.09224, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Mathhis Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929. 2020.
Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36, 2024.

---

Published as conference paper at ICLR 2026
MIFA LAB
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Ronald A Fisher. On the mathematical foundations of theoretical statistics. Philosophical transac- tions of the Royal Society of London. Series A, containing papers of a mathematical or physical 1922.
character, 2226594-604)3309-33-3
Rui Gao and Weiwei Liu. Ddgr: Continual learning with deep diffusion-based generative replay, In International Conference on Machine Learning, pp. 10744-10763, PMLR, 2023.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Observ. Remote Sens., 12(7):2217-22266 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint (rXiv:2106.09685. 2021.
Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free free high-pere sesthpeeffrmacce model merging. arXiv preprint arXiv:240S. 174611, 2024.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089, 2022.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2022.
Junsu Kim, Hoseong Cho, Jihyeon Kim, Yihalem Yimolal Tiruneh, and Seungryul Back. Sddgr: Sta- ble diffusion-based deep generative replay for class incremental object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28772-28781, 2024.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske Barwinska, al. Overcom- ing catastrophic forgetting in neural networks. Proc. Nat. Acad. Sci. USA, 1 114((3):552-33526 2017.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Univ. Toronto, Toronto, ON, Canada, Tech. Rep., 2009.
Yukun Li, Guansong Pang, Wei Suo, Chenchen Jing, Yuling Xi, Lingqiao Liu, Hao Chen, Guoqiang Liang, and Peng Wang. Coleclip: Open-domain continual learning via joint task prompt and vocabulary learning. arXiv preprint arXiv:2403.10225, 2024.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-294) 29947 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
arXiv:1711.01101, 2017.
Junxin Lu and Shiliang Sun. Pamk: Prototype augmented multi-teacher knowledge trans-
fer network for continual zero-shot learning. IEEE Trans. on Image Process., 2024. doi: 100.1109/IIP2224444000000
Yadong Lu, Shitian Zhao, Boxiang Yun, Dongsheng Jiang, Yin Li, Qingli Li, and Yan Wang. Boost- ing open-domain continual learning via leveraging intra-domain category- prototype. arXiv preprint arXiv:2408.0990 2024.
12

---

Published as conference paper at ICLR 2026
MIFA LAB
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151,, 2013.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018.
Michael S Matena and Colin A Raffel, Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703-17716, 2022.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics image processing, pp. 722-729. IEEE, 2008
Sejik Park.
Learning more generalized experts by merging experts in mixture-of-experts. arXiv preprint arXiv:2405. 2024.
Omkar M Parkhi, Andrea Vedaldi, CV Jawahar, and Andrew Zisserman. The truth about cats and dogs. In 2011 International Conference on Computer Vision, pp. 1427-1434. IEEE, 2011.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Vinay V. Ramasesh, Ethan Dyer, and Maithra Raghu, Anatomy of catastrophic forgetting: Hidden representations and task semantics. In Advances in Neural Information Processing Systems, 2020.
Sylvestre-AINe Alliise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interfer- ence. In International Conference on Learning Representattions, 2018
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04677 04670, 2016.
Joan Serra, Didac Suris. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International conference on machine learning, pp- 4548 4557. PMLR, 2018
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and- lann--aa--a tasks? arXiv preprint arXiv:2107.06388, 2021.
Haizhou Shi, Zihao Xu, Hengyi Wang. Weiyi Qin, Wenyuan Wang, Yibin Wang, and Hao Wang. Continual learning of large language models: A comprehensive survey. arXiv preprint arXiv:2404.16789, 2024.
Eli Verwimp. Rahaf Aljundi, Shai Ben-David, Matthias Bethge, Andrea Cossu, Alexander Gepperth, Tyler L. Hayes, Eyke Hüllermeier, Christopher Kanan, Dhireesha Kudithipudi, Christoph H. Lam- pert, Martin Mundt, Razvan Pascanu, Adrian Popescu, Andreas S. Tolias, Joost van de Weijer, Bing Liu, Vincenzo Lomonaco, Tinne Tuytelaars, and Gido M. van de Ven. Continual learning: Applications and the road forward. arXiv preprint arXiv:2311.1 1908, 2023.
13

---

Published as conference paper at ICLR 2026
MIFA LAB
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre- pre-trained transformers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of sero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7959-7971, 2022.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to 2o0. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Yicheng Xu, Yuxin Chen, Jiahao Nie, Yusong Wang, Huiping Zhuang, and Manabu Okumura. Ad- vancing cross-domain discriminability in continual learning of vision-language models. arXiv preprint arXiv:2406. 188.688888.8888... 2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. mees-meg Re- solving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang. and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint a0X00:2310002575. 2023.
Jiazuo Yu, Yunzhi Zhuge. Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You Hc. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23219 -32300, 2024.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yonghin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. arXiv preprint arXiv:2311.0300 2023.
Yu-Chu Yu, Chi-Pin Huang, Jr-Jen Chen, Kai-Po Chang, Yung-Hsuan Lai, Fu-En Yang, and Yu- Chiang Frank Wang. Select and distill: Selective dual-teacher knowledge transfer for continual learning on vision-language models. In European Conference on Computer Vision, pp. 219-236. Springer, 2025.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference on machine learning, pp. 3987-3995. PMLR, 2017.
Hongsheng Zhang. Zhong Ji, Jingren Liu, Yanwei Pang, and Jungong Han. Multi-staee knowledge integration of vision- languaee models for continual learning. arXiv preprint arX1v:2411.06764, 2024.
Lingan Zhao, Jing Lu, Yunlu Xu, Zhanzhan Cheng, Dashan Guo, Yi Niu, and Xiangzhong Fang. Few-shot incremmaattt learning via class-aware bilateral distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11838-11847, 2023a.
Zihao Zhao, Yuxiao Liu, Han Wu, Mei Wang, Yonghao Li, Sheng Wang, Lin Teng, Disheng Liu, Zhiming Cui, Qian Wang, et al. Clip in medical imaging: A comprehensive survey, arXiv preprint arXiv:2312.07353, 2023b.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of lasiun-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19125-19136, 2023.
14

---

Published as conference paper at ICLR 2026
MIFA LAB
Appendix
A DETAILED EXPERIMENT SETTING
Dataset. We test our method three benchmarks, including Multi-domain Task Incremental Learning (MTIL) (Zheng et al., 2023), task-agnostic MTIL, and few-shot MTIL.
The MTIL (Zheng al., 2023) extends task incremental learning cross-domain setting, where each task derived from a distinct domain. The MTIL framework comprises 11 individual tasks, each associated with a separate dataset, collectively representing total of 1201 classes. In alignment with previous works, we adopt the following datasets: Aircraft (Maji et al., 2013), Caltech101 (Fei- Fei al., 2004), CIFARI00 (Krizhevsky al., 2009), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), Flowers (Nilsback & Zisserman, 2008), Food (Bossard et al., 2014), MNIST (Deng, 2012), OxfordPet (Parkhi et al., 2011), 1). StanfordCars (Krause al., 2013), and SUN397 (Xiao al., 2010). The lask-agnostic MTIL is the variant of the MTIL benchmark, where the task ID is unknown during inference for each test sample. The few-shot MTIL variant involves training with only five train samples per category for each task.
Protocol. In our experiments, all evaluation protocols follow the existing works (Zheng et al., 2023; Park, 2024; Yu et al., 2024; Li et al., 2024) for fair comparison. We utilize a pre-trained CLIP model with a ViT-B/16 (Dosovitskiy et al., 2020) image encoder. We perform 1000 iterations of training for each task in both MTIL and task-agnostic MTIL. For few-shot MTIL, train each task for 500 iterations. We use AdamW (Loshchilov Hutter, 2017) as the optimizer and set the batch size 32 across all experiments.
Metric. For evaluating the MTHL, task-agnostic MTII,, and few-shot MTIL, we follow the exist- ing works (Zheng al., 2023; Park, 2024; Yu et al., 2024; Li et al., 2024) to use three key metries: "Average", "Last", and "Transfer". The "Average" metric computes the average accuracy across all seen tasks. The "Transfer" metric evaluates the model's zero-shot transfer performance on subse- quent tasks. The "Last" metric reflects the model's average performance at the end of the continual learning process. the task agnossic MTIL setting, we omit the "Transfer" metric and focus solely on the "Average" and "Last" metrics.
Baseline. We compare our method with several state-of-the-arr (SOTA) approaches, including:
(1) Zero-shot, (2) Individual FT, (3) Continual FT, (4) LwF (Li & Hoiem, 2017). (5) iCaRL (Rebuffi et al., 2017). (6) LwF-VR (Ding et al., 2022). (7) WiSE-FT (Wortsman et al., 2022) (8) ZSCL (Zheng et al., 2023), (9) MoE (Park, 2024), (10) MA (Yu et al., 2024), (11) Primal-RAll. (Xu et al., 2024), (12) Dual-RAIL (Xu et al., 2024), (13) CoLeCLIP (Li et al., 2024), (14) DPcCLIP (Lu et al., 2024), (15) MulKI (Zhang al., 2024).
The results of baselines (2)-8) and (15) in this paper are FFT-based. The results of baselines (9)-(14) in this paper are PEFT-based.
Zero-shot denotes directly using the pre-trained VLM for prediction on each task without additional fine-tuning. The results of Individual FT represent the performance of using a fully fine-tuned model. trained independently on each task based on the pre-trained VLM, for prediction. Continual FT refers to incrementally fine-tuning the VLM on new tasks without employing any forgetting mitigation strategies.
B MORE RESULTS OF COMPARISON WITH STATE-OF-THEE-AR METHODS
MTIL. Table 7 presents the full version of Table 1.
Few-shot MTIL., Table 8 presents the full version of Table 3.
15

---

Published as conference paper at ICLR 2026
MIFA LAB
Table 7: Comparison with SOTA methods on MTIL benchmark in terms of "Transfer", "Average.",
and "Last" scores (%). We label the best methods on average of all datasets with bold styles. The lines with background color represent our methods.
Methed
Airenft
Calech111
CIPAR100
DTD
EuoSAr
Flowers
MNIST
OxfendPa
Cars
SUX39
Average
Zero-shot
24.3
88.4
68.2
44.6
54.9
71.0
88.5
59.4
89.0
64.7
65.2
65.3
Continual
35.0
60.8
74.5
39.1
75.1
ICaRL
56.6
44.6
32.7
39.3
46.6
68.0
46.0
77.4
31.9
61..5
50.4
LoF-R
77.1
61.0
40.5
45.3
54.4
74.6
47.9
76.7
36.3
58.6
57.2
WISE-FT
Transfer
McE
ZSCL
86.0
88.2
66.9
67.4
44.7
45.4
$4,1
50.4
70.6
69.1
S7.6
$8.4
61.8
59.5
89.0
86.8
64.7
60.1
65.0
66.8
68.
69.1
MA
87.9
68.2
41.4
49.9
70.7
S5.7
59.7
89.1
65.5
68.9
Primal-RAIL
Dual RAIL
88.4
68.2
44.6
$4,9
71.0
$8.5
59.6
89.0
64.7
65.2
69.4
CoLeCCIP
88.2
65.1
44.7
$4.1
68.8
33..5
59.5
89.0
64.7
65.1
68.8
DPeCLIP
88.2
67.2
$4.0
70.6
88.2
59.5
89.0
64.8
69.1
71.3
89.7
ConDU (LoRA)
88.1
68.9
45.7
$7.0
71.3
83.8
61.2
89.3
65.1
67.8
70.3
ConDU (FT)
88.1
68.9
46.4
57,1
71.4
858.7
65.5
89.3
65.0
67.8
70,8
Continual
73.7
ICaLL
35.5
89.2
72.2
60.6
68.8
70.0
78.2
62.3
8LS
41.2
62.5
65.7
LeFVR
29.6
87.7
74.4
39.5
72,4
63.6
77.0
66.7
81.2
43.7
60.7
65.1
ZSCL
45.1
92.0
80.1
61.3
79.5
81.6
89.6
75.2
64.7
68.0
75.4
Averipe
McE
37.4
93.9
80.5
68.3
$1.9
BALL
90.0
74.0
90.6
67.7
66.4
75.9
MA
50.2
91.9
83.1
78.9
84.0
89.
73.7
89.1
66.9
76.7
Primal-RAIL
77.6
Dual RAIL
$2.5
96.0
80.6
70.4
B1.3
86,3
89
73.9
90.2
68.5
66.5
77.8
CoLeCCLJP
48.7
94.3
76.6
69.2
79.0
83.8
89.7
73.3
90.5
68.0
66.5
76.3
DPeCLIP
82.4
82.2
84.3
74,0
90.4
68.3
66.3
77.5
93.6
79.4
79.8
77.1
91.2
77.3
CouDU (LoRA)
$1.9
94.9
84.4
69.8
81.1
84.4
9000
77.3
89.5
69.0
69.3
78.3
ConDU (FT)
39.6
93.4
83.7
68.1
83,4
83.7
90.1
76.7
90.6
68.6
68.6
78.8
LwF
26.3
87.5
71.9
66.6
79.9
66.9
83.8
99.6
92.1
66.1
80.4
74.6
ICRL
35.8
93.0
77.0
70.2
83.3
88.5
90.4
86.7
93.2
$1.2
81.9
S0.1
LoF-VR
20.5
89.8
72.3
67.6
85.5
73.8
85.7
99.6
93.1
73.3
80.9
76.6
WISE-FT
2SCL.
40.6
92.2
81.3
70.5
94.8
90.5
91.9
98.7
93.9
85.3
80.2
83.6
Jast
McE
34.6
94.7
82.7
76.9
97.7
94.8
91.9
99.4
94.7
$0.9
80.5
84.4
MA
86.1
MILD
85.0
Primal-RAIL
98.7
78.7
Dual-RAlL
$2.5
96.8
83.3
N0.1
96.4
99.0
89.9
98.8
W3.5
K3.5
79.2
86.8
Col CoLP
48.7
94.9
78.8
78.4
88.9
96.3
91.1
97.6
94.4
82.7
80.2
84.7
DReCLIP
93.0
96.2
92.3
90.8
94.8
78.9
ConDU (LoRA)
48.9
95.2
87.8
78.5
96.3
95.2
91.7
97.6
93.0
85.3
78.8
86.2
ConDU (FT)
58.6
93.7
86.6
76.1
9%.2
93.4
91.9
99.6
94.8
84.9
80.5
87.1
16

---

Published as conference paper at ICLR 2026
MIFA LAB
Table 8: Comparison with SOTA methods on few-shot MTIL benchmark in terms of "Transfer", "Average.", and "Last" scores (%). We label the best methods on average of all datasets with bold styles. The lines with background color represent our methods.
Aitoraft
Cattch100
CIEARTO0
EunSAT
Flowess
MNIST
OxlondA
90N80
DTD
Car
Averuge
Zero-shot Individual FT
24.3
88.4
68.2
44.6
9
71.0
88.5
59.6
89.0
64.7
65.2
65.3
30.6
93.5
76.8
65.1
91.7
92.9
83.3
96.6
84.9
65.4
71.3
77.5
Coarinual FT
72.8
53.0
36.4
35.4
43.3
68.4
47.4
72.6
30.0
52.7
$1.2
LwF-VR WISE-FT
K2.2
62.5
40.1
56.3
6019
77.6
6018
Transfer
77.6
60.0
11.3
39.1
$3.0
76.6
58.1
75.5
37.5
58.2
57.7
ZSCL
84.0
68.1
44.8
46.8
63.6
84.9
61.4
81.4
55.5
62.2
65.5
Primal--RAi RAII.
88.4
68.2
51.9
71.0
$8.5
59.6
89.0
61.7
65.2
69.1
Dual RAIL
88.4
68.2
44.6
$4.9
71.0
$8.5
59.6
89.0
64.7
65.2
69.4
ConDU (PT)
$8.0
69.5
$4.4
71.1
$8.7
62.2
88.9
66.6
70,0
(LoRA)
Coarinual
28.
$6.4
59.1
$2.8
55.8
62.0
70..2
64.7
75.3
35.01
$4.0
58.5
LwF
23.5
77.4
43.5
41.7
43.5
52.2
$4.6
63.4
68.0
2.1.3
$2.6
49.2
LWEVE
62.5
Averagge
77.6
ZSCL
28.2
88.6
66.5
53.5
$6.3
73.4
853.
$6.4
82.4
$7.5
62.9
64.4
Molt
30.0
89.6
73.9
$8.7
69.3
79.3
76.5
89.1
65.3
65.8
71.4
Primal-RAIL
Dual-RAIL
94.2
70.9
$8.88
70.6
84.3
70.3
89.7
65.8
72.3
CoaDU (FT)
33.1
90.5
74.1
58.3
76.2
BLLDD
$7.9
73.4
88.0
61.8
67.1
72.3
ConDU (LoRA)
32.4
92.1
75.4
38.8
75.1
82.9
$7.3
74.0
89,3
65.1
67.0
72.7
LwF
22.1
58.2
17.9
32.
28.1
66.7
46.0
84.3
64.1
31.5
60.1
46.5
LWF-VR
22.9
89.9
19.3
$7.1
$7.6
79.2
78.3
71.7
83.6
60.1
69.8
66.9
WISE.PT
88.9
59.6
80.9
81.7
77.1
83.2
62.8
70.0
71.9
Last
ZSCL
63.7
85.9
70.4
67.4
Mol:
30.1
89.3
74.9
64.0)
82.3
89.4
87.1
89.0
89.1
69.5
72.5
76.1
Primal-RAIL
32.9
95.1
70.3
63.2
81.5
95.6
85..5
89.7
89.0
72.5
71.0
77.2
Dual-RAl.
CoaDU (FT)
90.7
75.0
88.8
88.6
91.8
85.6
71.9
76.6
ConDU (LoRA)
31.8
92.4
76.7
63.4
86.8
91.8
85.6
93.9
90.3
68.1
70.9
77.4
Table 9: Performance (%) Comparison of of state-of-the-art CI. methods on MTIL benchmark in
Order II
Method
Transfer
Average
A
Last
A
CLIP shro (Radford et al., 2021)
65.4
0.0
65.3
0.0
65.3
0.0
Continual FT
46.6
-18.8
56.2
-9.1
67.4
2.1
LWF (Li Hoiem, 2017)
53.2
-12.2
62.2
-3.1
71.9
6.6
iCaRL (Rebuffi et al., 2017)
50.9
-14.5
56.9
-8.4
71.6
6.3
LwF-VR (Ding al., 2022)
53.1
-12.3
60.6
-4.7
68.3
3.0
WiSE-FT (Wortsman et al., 2022)
51.0
-14.4
61.5
-3.8
72.2
6.9
ZSCL (Zheng et al., 2023)
64.2
-1.2
74.5
9.2
83.4
18.1
MoE (Park, 2024)
64.3
-1.1
74.7
9.4
84.1
18.8
MulKI (Zhang et al., 2024)
65.6
0.2
75.0
9.7
84.2
18.9
ConDU (FT)
66.5
1.1
75.9
10.6
85.6
20.3
C OTHER ORDER OF MTIL
To further validate the effectiveness our approach, we refer to existing works (Zheng et al., 2023; Zhang al., 2024) and present the MTIL experimental results based another task order in Table 9, with all other experimental settings unchanged. Order is StanfordCars, Food, MNIST, OxfordPet, Flowers. SUN397, Aircraft, Caltech101, DTD, EuroSAT, CIFAR100. Only few baselines have reported results for this task order, and we present the results these baselines in the table. Even with different task order, still achieve performance beyond the state-of- ofte-off
17

---

Published as conference paper at ICLR 2026
MIFA LAB
Table 10: Comparison of different methods under 10-step and 20-step settings of TinylmageNet Inss-Incremental learning.
Method
10 Steps
20 Steps
ZSCL
71.62
68.30
LwF
44.00
42.26
LwF-VR
67.05
63.89
iCaRL
65.97
64.48
Continual FT
41.54
44.55
CLIP
65.59
65.30
COnDU (FT)
71.74
71.49
ConDU (LoRA)
68.80
68.28
D EXPERIMENTAL RESULTS ON CLASS-INCREMENTAL LEARNING
To explore the generality of the ConDU framework beyond vision- languaee models, we conducted additional experiments on single- modality dataset. Specifically. we evaluated ConDU on Tinylm- ageNet under class-incrementas settings with 10, and 20 tasks. The results show that ConDU con- sistently outperforms competitive baselines, indicating that is also effective for class-incremental learning in moggllty scenarios.
In the first session, we train the model on 100 categories. Then we split the remaining 100 cate- gories of TinyImageNe dataset into 10 tasks (each with 10 categories) and 20 tasks (each with 5 categories). Following the standard class-incremental learning setup, we train on all tasks sequen- tially and report the average inference accuracy across all tasks after all sessions (task ID agnostic). The results are in Table 10.
E COMPARISON BETWEEN CONDU (FT) AND CONDU (LORA)
Comparison between the ConDU (FT) and ConDU (LoRA). When hardware conditions limited, we recommend using ConDU (LoRA). When hardware conditions can support full fine- tuning, ConDU (FT) and ConDU (LoRA) has have distinct advantages under different conditions.
Table (Standard MTIL): For the "Transfer" metric, ConDU (FT) outperforms ConDU (LoRA) on tasks, underperforms on 2, and ties the rest. Average accuracy favors ConDU (FT), For the "Average" metric, ConDU (FT) outperforms ConDU (LoRA) on 4 tasks, and underperforms 7. However, again, the average accuracy is higher for ConDU (FT). For the "Last" metric, ConDU (FT) outperforms on tasks and underperforms on 5, with higher average accuracy. Overall, ConDU (FT) achieves higher mean accuracy across all three metrics in standard MTIL, and shows better per-task performance in two metrics out of the three.
Table (Task-Agnostic MTIL): For the "Average" metric, ConDU (FT) outperforms ConDU (LoRA) 4 tasks and underperforms on 7, but still achieves a higher overall average. For the "Last" metric, FT outperforms on 7 tasks and underperforms on 4, again with higher overall average. Overall, ConDU (FT) achieves higher mean accuracy across all two metrics in task- agnostic MTIL, and shows better per-task performance in one metric out of the two.
Table 3 (Few-shot MTIL): ConDU (LoRA) is better when there is less fine-tuning data for single task as shown in Table 3), because too little data will lead to overfitting during full fine-tuning.
Then, also conducted experiments the TinylmageNet maaseinccrmentll benchmark, using both 10-task and 20-task splits. The results in Table 10 show that ConDU (FT) significantly outper- forms ConDU (LoRA) under this setting.
In summary, full fine-tuning and PEFT each have distinct advantages under different conditions in continual learning.
18

---

Published as conference paper at ICLR 2026
MIFA LAB
Table 11: Comparison of "Transfer" metric on MTIL with different choice of K by ConDU (FT).
Airerft
Caltch110
CIFAR 100
DTD
BurOAAT
Howers
Food
MNNS
OxfordPet
SUNWS
Average
K
88.4
68.2
44.7
55.3
71.0
88.5
59.5
89.0
64.7
65.4
69.5
K
88.1
68.6
45.7
57.8
71.1
88.7
63.5
89.1
64.8
66.5
70.4
K
57.4
88.7
89.4
67.2
70.6
K
88.1 1
68.9
46.4
56.4
71.6
88.7
64.4
89.3
64.8
67.7
70.6
K
88.1
68.9
57.0
71.3
88.7
65.5
89.3
65.0
67.8
70.8
K
88.1
68.9
46.4
57.0
71.5
88.5
64.0
88.6
65.0
67.9
70.6
K
88.1
68.9
46.4
57.0
71.5
88.1
64.5
88.2
64.5
67.9
70.5
K
68,9
57.0
88.1
87.8
64.5
67.7
70.3
K 9
88.1
68.9
46.4
37.0
71.5
88.1
62.7
87.6
64.4
67.7
70.2
88.1
87.6
K 11
88.1
68.9
46.4
57.0
71.5
88.1
62.7
87.6
64.3
67.7
70.2
Unified
66.7
44.6
29.3
43.6
47.7
60.1
48.7
53.5
28.0
49.1
47.1
F THE CHOICE OF K OF AGGREGATING
Table 11 presents the inference performance of the "Transfer" metric of ConDU (FT) on the MTIL benchmark with different values of K. The last row "Unified" denotes directly using unified model to predict unseen tasks. From the table, we can observe that when K is set to 1, the Avg performance the lowest 69.5%. As K increases, the Avg performance gradually improves, reaching the best result of 70.8% when K is 5. However, after this point, the Avg performance begins to decrease as K increases further. Overall, the performance of the aggregating mechanism is relatively insensitive to the choice of K. The results in the last indicate that directly using the unified model for predicting unseen tasks yields significantly worse performance compared to our proposed aggregat- ing prediction approach. Within our framework, the unified model can be viewed storage mechanism designed to avoid storing all task-specific models individually. Its primary pur- pose is interact with task triggers to reconstruct all task-specific models, rather than serving standalone model for inference. This because the objective of model fusion was never intended to make the unified model itself directly applicable for inference.
compressed
G PROOF OF THEOREM
G.1 DEFINITIONS
One iteration of unifying and decoupling is the same as Section 4, but some symbols need to be slightly changed. Therefore, we redefine the unifying and decoupling process here. At session if we have n delta models o' (t), 82 (t), .aa....,"(( (t), the j-th position unified delta model 8(t) (which
denoted 81m in Section 4 and is simplified here) calculated
( (t) if ΣР E 0
6,()) min, (6)
if -i=1 ()())
The j-th position of mask Mt(t) calculated as Mt(t) = { (t)
The rescaler x²()) is calculated as X²(()
sum(abs( (+)))
The rescaler Xf(1) ensures the
sum (() (B)))
[|6" (t)]) remains unchanged while increases.
The reconstructed delta model i is calculated 6' (t+ (t) (t o 8(t).
In the proof below, there are two settings, and we will mark in each lemma and thcorem whether holds in setting or 2.
Setting 1:
This setting is the same as the normal continual learning process in the main paper. After the decoupling phase of each session t, have 1 delta models 5' (t), 82 (t),..., 5' (t), and then newly fine-tuned delta model (²()) is added into the sequence and all the above t delta models unified.
19

---

Published as conference paper at ICLR 2026
MIFA LAB
This setting is transitional setting used in the proof. Unlike the normal continual learning process, in this setting. we only have n delta models during all sessions. After the decoupling phase of each session t, we have delta models 8' (t), 8 (t),..., (t) to be unified, and no newly fine- tuned delta model will be added in the sequence.
Setting 2:
G.2 PROOF
Before proving Theoreml, we prove Lemmas 1, Lemmas 2 and Corollary Lemma 1, Let X1, X2..... Xn be ii.d. random variables. Then
P( max Xn)) n
Proof. Assume that X1, X2,...... X,, are random variables with common probability density func- tion f and cumulative distribution function F. We compute
P(X,. max(Xi...... Xn3)
=P(,n > . > Xn-1)
P(x > XI...., (nd))
Since X1,..., , are i.i.d. with CDF F, we have
PP(1 < ..... Xn-1 <x) [F(x)]*
Hence
pCc [F(x)] f(x) dr
P(,,, max (X.....,,.)))
[F(x)] dF(x)
u=- du
This completes the proof.
Lemma 2 (Sign Preservation of 8)). setting 1, for a task 8 in a continual learning session, the parameter at any position is guaranteed preserve its sign. Specifically, if a position in 8 8'(t') (denoted as a' (i')) is positive (or negative) after an iteration, then Vl. 2 L', (1) Z 0 (or S 0).
Moreover, if a² (C')
then VL > (t) 0.
Proof. We observe that (t) = (8 1) o 8(t)). Therefore, if the signs remain consistent after an iteration, then Mt
and if the signs change, then (t) 0. As result, positive (or negative) values in 8 during the iteration process will either retain their signs or become zero. From
the definition of (t),), if 5' (t- 1) = 0, it implies M' 0, and thus 8 (t) 0. Hence, the latter part of the lemma is also proved.
Lemma 3 (Convergence of Iteration). In the setting 2, if the relative order of (t) values remains unchanged and vi + k, Vt, {j Mt(t) = and M (t) = 1} + Ø, then these n delta models will converge to a uniquely determined of n delta models.
Proof. First, we show that Vi E [1,...,n],. O, =
Indeed, from the proof of Lemma 2, know that the number of zeros in M does not decrease during iterations. Since no new delta models are introduced, the number of zeros M² cannot increase either. Thus, the sign of each position remains fixed during iterations.
20

---

Published as conference paper at ICLR 2026
MIFA LAB
Let (t) denote the value at the j-th position of the i-th delta model after t iterations, and let d(t) denote the delta model obtained by taking the absolute value of each position of ö(t). Note that 8' (1) is obtained by scaling ((1) )) with (1) and setting certain positions to O:
65 = (1()) max(x | Mj(1) = 13.
(1)
From Eq. (1), it follows that the delta model 8' (1) with the largest (1) contributes all its non- zero values to 6(2). Let im denote the index of the m-th largest scaling factor (1).. Since the relative order of (t) values remains unchanged, for > 1, the real i corresponding im remains in (1,2....,,)) n...
For t 2 2, show that (t) 1. By assumption, X max = 1, :,.... n) for all t.
Consider the part of 8 (1) where M (1) 1. These positions remain unchanged in e5(() during iterations. Hence, 8 (t) M (t) O ((t) = 8 (t 1), and 1.
Next, we consider '2 and divide (+) into two parts. Let xxt) denote the sum of absolute values in (t) positions where M (t) 1 during the t-th iteration, and let y(t) represent the sum of absolute values at positions where (l) = O. By the definition of (1). we know that sx)) + all I, where is constant. During each iteration, the values in v({) are incorporated y(() for
into ptt). Let denote the sum of absolute values in 8th (t) at positions where M (t) 1, which is also a constant. Then, we have:
(+) = s(t)+y(t)
6
sty(t) s-u(1))
x(1+1) (1) sty(())
u((++) 1) = (t) y(t) (+v(())
ct(t)
Since iterations do not alter the relative order of X (t), we have (ta < (²) = 1. Consequently,
< yt + 1) < y(), which implies that both s() and yt) converge, and (1) monotonically converges to a stable solution.
Using a similar analysis for ip...., bns can be shown that all (()) eventually converge to unique solutions.
Corollary 1. Under the same conditions as Theorem 3, for each = 1,2,..., for sufficiently larget t, the following inequality holds:
110 (t+++1 5 < - -8( (0(())))) 1)1l1
Proof of Theorem 1.
Proof. Without loss of generality, assume that each position of (t) Vi E fl.....f] {} is positive. Since we assume that 5(() is bounded, and the absolute value at each position of 5²(() is less than 5(1), follows that all (1) are bounded. Therefore, AI and E (1....,(). (1)] bounded, there exists constant S > 0 such that
(15( < S, and t.
Denote by d_ (t) the unified delta model selected in the iteration of (o) and let X (t) be the corresponding scaling factor for (1). The post-iteration results are (1 + 1)).
According to Theorem 1, when iterating over the delta model set (8 (1) | {1,.... O}, for any positional dimension p. we have
(t) # 5x() (1) max (t)) 1
(2)
1sist
Since (t) mmxi 57 (1) and (t. = maxigiii---- 5),
let ft) E [max] (t)] and similarly f(t 1) E 15 then
lim r(1)= = 1, 1-000
21

---

Published as conference paper at ICLR 2026
MIFA LAB
where r(t)
f(0) 1(1-1)
Therefore,
E [maax (85(0) i = 1,...,11........ 0
f(1)
E 6p(0) (9)
8(t 1) r(t)-
(3)
E max (t) i == 1, 1,....t t 1}
Combining (2) and (3), it follows that
|
= F(6p(t) = 6ep(t)) + +((b((t) * )tp(t)) ET 5p(0)
6.1l
(t)
1-1 1
+ t r())
t
- 1 + r(t)
t
and hence, for i E (1,..., 1},
E X1
E 100 (01l, |0((0
= E [ل.0
t
xt(t)
||(())]1 (t)l
1600111
t rtt())
Since
|10 (())) S - ()
and suppose the dimension (t + 1) is d. then
[[ + - 1)
= (1) = (() (x()) - 00000 ((1 + 52 1)[la]
t- 1 r(t) - x (0) = =[0)((++++
1)1l (tp(t) # (t)
(t 1) (r(t) 1) [((((()) + { 1 5t + 1) dup(t) # [2(
82
(t 1) (r(t) 1) {(10(0))] [[[x(()))(00]] -
12
1) (r(t) 1) e[(8(()) + ( (())-) B[1lo. (t)ll
B2
d
1) (r(1) - 1)
(t ()-()()(() - 1)
S ((t-141 ((t-1+r(t)) S.
Hence
lim (1 (t + 1]]11] = O. t1000
Moreover, by Corollary 1, it follows that
1[1b0 (t 1) 8 [t)] < (1)) - 8'(t- - 1)]l
Therefore, for i E f1....,t-1 1}. we conclude
lim 10000
[[(15((L+1) 5t (1)b) - B (1) - 5²(1 - 1)1b s o.
This finishes the proof.
H STORAGE ANALYSIS
The masks align the unified delta model's direction with that of each delta model, and the rescalers ensure that the unified model's parameter magnitude matches that of each delta model. Although the masks share the structure of the delta model, their binary nature ensures they require much less storage than the delta models, and the rescalers are sealars whose storage is negligible, We compare the storage size of model parameter files saved in Python between method and Individual FT,
22

---

Published as conference paper at ICLR 2026
MIFA LAB
The results of Individual FT represent the performance of using fully fine-tuned model, trained independently on each task based on the pre-trained VLM, for prediction.
First, look the comparison in the full fine-tuning scenario. After training all 11 tasks, the storage size for ConDU as follows: Clip Model (570.86 MB) + Unified Delta Model (570.86 MB) Masks 20 MB) Rescalers (747 KB) 1377.9 .22 MB. In contrast, Individual FT's storage size Task-specific Model (570.86 MB) x 11 = 6279.46 MB, saving a total of 4901.54 MB storage.
Next, compare in the LoRA (Rank=64) scenario. After training all 11 tasks, the storage size for ConDU is as follows: Clip Model (570.86 MB) Unified Delta Model of LoRA (37.53 MB) + Masks of LoRA (12.89 MB) Rescalers (747 KB) 621.28 .28 MB. For Individual FT, the storage size is Clip Model (570.86 MB) + LoRA (37.53 MB) x 11 983.51 MB, saving a total of 362.23 MB storage.
As shown, our method significantly alleviates the excessive storage requirement of Individual FT, and the storage reduction more pronounced as the proportion of fine-tunable parameters increases and the number of tasks grows.
I TRAINING AND INFERENCE TIME
We compare the training and inference time between ConDU and a SOTA method ZSCL (Zheng et al., 2023). Both methods are evaluated on single GPU with computational power equivalent to an GeForce 4090. "s" denotes "seconds" here.
Training Time. To demonstrate concrete performance, we analyze the SUN397 dataset from the last training session. For a fair comparison, maintain the settings, where the model is trained for 1000 iterations across all methods. The training pipeline of our method comprises four distinct phases: decoupling the unified model, tuning individually, computing prototypes, and uni- fying models.
For ConDU (LoRA), the total training time of this session is decoupling (0.64s) + unifying (3.72s) + tuming (401.72s) + computing proto (303.63s) = 709.71s. In constrast, the total training time of ConDU (FT), 7SCL. Continual Fine-tuning 738.91s, 1504s and 443s. According to the results, ConDU does not significantly increase training time compared to Continual Fine-tuning and saves approximately 52% time relative ZSCL.
Inference Time. At the inference stage, if a test sample comes from a known task with provided task ID, the corresponding task-specific model is selected for prediction, making the inference time equivalent to that of a single model. When dealing with test samples from unseen tasks or without task ID, our ConDU (LoRA) method involves two phases: task-specific model selection based on computing cosine similarity, and aggregating the predictions of the selected models. We take the SUN397 dataset in the last session as an example. The model selection phase takes 0.27s. Since the forward propagation of the selected task mask-specffic models can be computed in parallel, the whole aggregating phase takes nearly the same time as the prediction of single model, which just 30.22s. In contrast, ZSCL takes 29.88s at the inference stage. The additional 1.58s consumed by our method due to extra algorithmic steps accounts for only 5.16% of the total inference time. Since ConDU (FT) requires higher computational resources for parallel inference, recommend using ConDU (LoRA) for scenarios where inference speed is a concern.
J COMPREHENSIVE COMPARISON BETWEEN CONDU AND INDIVIDUAL FT
We now provide an integrated comparison between Individual FT (i.e., training and storing a sepa- rate expert model for each task without unification) and ConDU in Table 12.
"IF" denotes Individual FT. "CD" denotes ConDU. The results show that:
Accuracy (on standard MTIL, using the "Last" metric): Individual FT achieves slightly better accuracy than ConDU.
23

---

Published as conference paper at ICLR 2026
MIFA LAB
Table 12: Comprehensive Comparison between ConDU and Individual FT.
Aireraft
Caftech
CIFARI00
DTD
EuroSAT
Flowers
Food
MNIST
OxfordPet
Cars
SUN397
Arg
FMS
(s)s)
Method
IF (FT)
62.0
95.1
89.6
79.5
98.9
97.5
92.7
99.6
94.7
89.6
81.8
89.2
6279.46
28.34
IF (LoRA)
59.5
97.3
89.0
79.9
98.6
97.7
92.8
99.4
94.3
90.5
81.9
89.2
983.51
29.95
CD (FT)
58.6
93.7
86.6
76.1
98.2
93.4
91.9
99.6
94.8
84.9
80.5
87.1
1377.92
28.75
CD (LoRA)
48.9
95.2
87.8
78.5
96.3
95.2
91.7
97.6
93.0
85.3
78.8
86.2
621.28
30.22
Table 13: Performance across sessions on different datasets.
Aireratt
Caltech 101
CIFAR100
EuroSAT
Flowers
MNIST
OxfordPet
SUN397
DTD
Food
Cars
Average
Session 1
61.3
61.3
Session 2
60.5
94.6
77.6
Session 3
93.9
87.9
Session 4
60.0
94.0
87.5
77.1
79.7
Session 5
59,9
93.9
87.2
76.7
98.4
83,2
Session 6
59.8
93.9
87.2
76.8
98.3
95.0
85,2
Session 7
59.5
93.9
87.1
76.5
98.3
94.5
92.3
86.0
Session 8
58.7
93.8
86.5
75.9
98.1
93.4
92.0
99.6
87.3
Session 9
58.8
93.8
86.5
76.0
98.2
93.4
91.9
99.6
94.9
88.
Session 10
58.5
93.8
86.5
76.0
98.2
93.4
91.9
99.6
94.8
85.7
87.8
Session 11
58.6
93.7
86.6
76.1
98.2
93.4
91.9
99.6
94.8
84.9
80.5
87.1
Storage: "FMS" (Final Model Size) measures the size the model after all sessions are com- pleted. Individual FT incurs significantly higher storage costs than ConDU, as all expert models are stored independently.
Inference Time: "IT" denotes "Inference Time". While ConDU introduces a lightweight expert- weight computation step, overall inference latency is nearly unchanged. The inference time here refers to the inference time of the last session the SUN397 test set.
Generality: Importantly, Individual FT does not support zero-shot inference or task-agnostic set- tings, limiting its applicabilityy Thus, the comparsion only includes the "Last" metric on standard MTIL.
K ACCURACY DURING SESSIONS
Table 13 shows the test accuracy across all tasks at the end of each session. Based on our exper- imental observations, the performance of all previous tasks decreases than other sessions due to the significant differences between the handwritten digit recognition task (MNIST) and other tasks that focus object classification. However, the magnitude of the drop mains small, demonstrating ConDU's strong robustness even when encountering highly OOD tasks.
after MNIST session
L INTRODUCTION OF LORA
Below revisit LoRA, in which g is the module that PEFT attached to, and h are input and output of the original and h' is output attached with PEFT.
LoRA (Hu et al., 2021) assumes the change of parameters is in a low-rank space when tuning the pre-trained model downstream task. For linear layer with weight W Rdxd', the weight
24

---

Published as conference paper at ICLR 2026
MIFA LAB
updates A W can be decomposed into the multiplication of two small matrices:
W
W down
W
up
where W down RRdx and W E -R" For the convolution layer, the updates can reshaped
into the kernel shape. Finally, LoRA modifies the forward pass of the adapted layer into the follow- ing form:
h' h (W down W W
where * is matrix multiplication or convolution operation, the bias and reshape operation are omitted for conciseness. Since LoRA adapts the weight of 9. the weight updates can be merged into g to reduce the inference latency.
M
THE USE OF LARGE LANGUAGE MODELS (LLMS)
The LLMs are used only to help polishing writing in this work.
25


Reviews:
[{"review_id": "jRGNVmY16H", "reviewer": "Reviewer_BJYc", "summary": "This paper proposes a novel Continual Decoupling-Unifying approach to handle the fusion of delta model in VLM continual learning. Specifically, it contains a unified model along with task triggers and a set of prototypes. It derives the new unified model utilizing the old unified model with new task trigger and prototype. Extensive experimental results show the effectiveness of this method.", "questions": "The previous question was raised in Weaknesses. Regarding the performance of the experiments, the authors have compared it with some state-of-the-art (SOTA) methods and achieved a significant improvement. My main concern is the algorithm design, because this kind of hard model editing problem often leads to a huge shift in the model's parameters in its space during continual learning. Simply put, sometimes a poorly edited layer parameter can cause a huge drop in continual learning, which seriously impairs the learning of subsequent stages. Therefore, an editing method that can ensure that multiple stages do not collapse is very interesting.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is quite novel. The unified delta model it utilizes is a very novel design, and the decoupling and unifying operations are dual, which is quite interesting.\n\n2. In continual learning, it's not necessary to store only one model. Combining a general model with multiple task-specific substructures is a good decoupling approach, which also alleviates the forgetting problem to some extent and ensures the model's ability to learn new content.", "weaknesses": "1. The proposed method doesn't seem to provide a metric for the error of the model after unification and decoupling relative to the original model. My understanding is that while the unification and decoupling in this algorithm are dual, it shouldn't guarantee a complete reconstruction of the model corresponding to task t. If I'm mistaken, please point it out (and offer my apologies). Otherwise, could you provide a rough experimental analysis to assess the error of this algorithm after unification and decoupling relative to the original model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ffkXGUoAtK", "reviewer": "Reviewer_WuML", "summary": "The paper proposed a novel continue learning framework called ConDU, which introduces model fusion into the continue learning framework for VLM and is capable of both Full-finetuning and Parameter-efficient tuning. In the framework, the newly fine-tuned model will be merged with the old model through parameter delta based on trigger output.", "questions": "- When applying such a paradigm in the real world, as the number quickly increases, does such a paradigm still work well with the growing requirement for trigger saving?\n- Considering the current VLM paradigm that involves LLM, would such a fusion strategy also work? The benchmark is kind of old and is hard to view as a real continuous learning setting.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The trigger framework enables the model to deal with both old tasks and unknown tasks.\n- The framework enables test-time scaling, which aggregates prediction results from multiple decoupled models.", "weaknesses": "- Though improvement on the benchmark is higher on average, compared to the former method, there are always certain subsets that perform worse than the former method. And the average gain is growing smaller as they continue learning and progress to the end.\n- The similarity used for prototypes and task selection is computed entirely from the pre-trained VLM’s image/text features, rather than from the features of each specialized task model. This can cause a mismatch in “task selection” when a task’s decision boundary shifts substantially after fine-tuning—especially under severe domain shift. While the paper reports insensitivity to K, it does not compare “pre-trained vs. expert features.”\n- The theorem assumes conditions such as “each delta has the same sign per dimension” and “i.i.d.,” which conflict with the directional conflicts commonly seen in real multi-domain/multi-task fine-tuning; therefore, its convergence claims have limited applicability to practical training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EnjPvznEqq", "reviewer": "Reviewer_1yyf", "summary": "This paper proposes ConDU, a continual learning framework for VLMs that introduces training-free decoupling–unifying model fusion. A single unified model is maintained; per-task “triggers” (sign mask + rescaler) reconstruct task-specific deltas, and a prototype-based semantic aggregator combines multiple reconstructed experts for zero-shot or unknown-task inference. ConDU supports both full fine-tuning and LoRA. On MTIL (plus task-agnostic and few-shot variants), it improves Average by up to ~2% over SOTA while preserving/boosting zero-shot transfer, with low fusion overhead (~1% of fine-tuning time).", "questions": "1. How would performance change if the unify step used Fisher/TIES/EMR-style merging or soft masks/rescalers; which component (masking, rescale, signed-extrema) drives gains? \n\n2. Can learned routers or temperature-calibrated cosine improve task weighting, and how robust is aggregation to prompt wording or domain shift in the prototype bank? \n\n3. What happens when the theorem’s sign/i.i.d. assumptions are violated—e.g., conflicting signs across tasks—and can relaxed bounds be established empirically or analytically? \n\n4. What are end-to-end latency/throughput and memory footprints for K-expert aggregation on commodity GPUs/edge devices, and how does this compare to MoE or per-task checkpoints?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**1. Model-fusion continual learning, broadly compatible.** ConDU formalizes a decouple–unify routine with task triggers, works with full FT and PEFT, and keeps only one deployable model plus small metadata—no replay or reference dataset. \n\n**2. Zero-shot/unknown-task inference.** Prototype-weighted aggregation of reconstructed experts enables robust prediction without task IDs, aligning with VLM zero-shot goals. \n\n**3. Consistent empirical gains and efficiency.** Across MTIL variants, ConDU raises Transfer/Average/Last and reports modest storage/time costs; decoupling/unifying are training-free and ~1% of tuning.", "weaknesses": "**1. Limited novelty.** The contribution largely integrates known model-merging ideas into a continual pipeline; the unify rule (signed extreme per parameter) and trigger design, while clever, are incremental versus established fusion schemes (Fisher, TIES, EMR). Stronger positioning against these would clarify originality. \n\n**2. Lacking sufficient ablation study.** Beyond an insensitivity check on the top-K selector, there is little analysis of design choices: unify rule vs. Fisher/TIES/EMR, trigger rescaling, mask binarization, prototype variants, or alternative gating (learned vs. cosine). Such ablations would substantiate the mechanism-level claims. \n\n**3. Benchmark scope.** Results focus on MTIL and its task-agnostic/few-shot variants (11 domains, classification); broader VLM tasks (detection, OCR, medical, video, captions) aren’t covered, limiting external validity. \n\n**4. Assumptions/theory.** Convergence analysis assumes identical parameter signs and i.i.d. deltas—conditions unlikely across heterogeneous tasks—so theoretical guarantees may not reflect practice. \n\n**5. Aggregation sensitivity.** Zero-shot prediction hinges on prototypes from the pre-trained VLM; domain shifts or text prompt bias could mis-weight experts, and calibration is not discussed. \n\n**6. Efficiency reporting.** While fusion is fast and memory-light, end-to-end latency under multi-expert aggregation and edge constraints is only briefly discussed; more granular profiles would aid adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X4DAGXOOjV", "reviewer": "Reviewer_j5xv", "summary": "This paper introduces ConDU (Continual Decoupling–Unifying), a novel framework for continual learning (CL) of vision-language models (VLMs) such as CLIP. The core idea is to employ model fusion—traditionally used for combining independently fine-tuned models—to enable continual learning without catastrophic forgetting or reliance on replay data. Extensive experiments on MTIL, task-agnostic MTIL, and few-shot MTIL benchmarks show that ConDU outperforms existing methods by up to 2% in average accuracy while maintaining or improving zero-shot transfer performance.", "questions": "1.\tScalability.\nHow does ConDU behave as the number of tasks increases significantly (e.g., >20)? Does the unifying process saturate, or does performance plateau as the unified delta model grows?\n2.\tMemory usage.\nSince ConDU stores task triggers and prototype sets, how does the memory footprint scale with the number of tasks? Could the method benefit from pruning or compressing old triggers to maintain constant memory?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. Strong empirical gains across multiple benchmarks and settings.\n2. Novel use of model fusion for continual learning of VLMs.\n3. Fast inference and training efficiency due to training-free decoupling/unifying steps.", "weaknesses": "While the technical contribution is solid, the presentation quality should be improved to make the paper easier to follow. If these issues are addressed, I would be inclined to raise my evaluation.\nKey suggestions:\n1.\tFigure clarity.\nThe text in figures (e.g., Fig. 1, 3, 4) is too small to read. Redrawing them with larger labels and more space would improve readability.\n2.\tFigure redundancy.\nFig. 1 is not clearly explained and highly overlaps with Fig. 3; consider merging the key information into Fig. 3 and removing Fig. 1 to streamline the narrative.\n3.\tStructure of the paper.\nThe introduction currently contains too much methodological detail. The core method description should be moved entirely into Section 4 (“ConDU”), leaving the introduction focused on motivation and contribution.\n4.\tLanguage and phrasing.\nSeveral passages could benefit from linguistic polishing for clarity and conciseness. Improving transitions and consistent terminology would greatly enhance readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)